Proving Document Quality

You say you produce high-quality document products? Great, now prove it! Although many can recognize quality when they read it, the prospect of measuring documentation quality seems dubious to many.

I work with a great many clients where authors say they generate high quality documents, but they can only show that their documents are accurate. That is, accurate in terms of template, source data, and grammar conformance. Beyond these three inspection parameters, many regard documentation quality as inherently immeasurable. I argue that these three parameters by themselves are wholly insufficient measures of document quality and that important aspects of how documents convey meaning can be measured.

Surely they must see other metrics of value, especially when considering factors like the number of regulatory questions elicited by errors of omission and commission in their very accurate submission documents or the number of protocol amendments generated over the life of their very accurate research protocol.

Some see additional metrics as meaningful, but not for their type of documentation. In the realm of clinical development documentation, many consider their output unique and beyond measurement. I hear more than a few medical writers say, “We know our own field intimately, and everywhere we look we see shades of gray and unique situations, so measuring is not useful.” and others in clinical development say “We are unique, and your notions of quality do not apply to us.”

Still others find the prospect of measuring output impractical or suggest that even if they had data, “Against what standards could we make comparisons?”

Even easily measured attributes like accuracy are not effectively tracked. Documents are routinely verified/corrected by quality control operators, yet the output is rarely tabulated and statistically analyzed to provide a measure of writer performance. The intention is merely to ensure the document is archived as accurate in terms of template, source data, and grammar conformance.

I do not know of any pharmaceutical or medical device company that makes use of any statistical quality control techniques to track their document quality. It would be easy to apply similar principles as used in the auto industry or pharmaceutical manufacturing. The auto manufacturers turn out millions of units every year, and each company tracks manufacturing defects, but not by examining every car. Instead, they pull a few cars off the assembly line and tear them apart. Pharmaceutical manufacturers test an appropriate sample from each manufacturing lot. In both cases the products are measured against a set of well-defined acceptance criteria.

Such acceptance criteria exist for technical and scientific documents (in our work at McCulley/Cuppan we have developed such a set of criteria that we use during our training and consulting work for clients) so such quality control methods could easily be deployed.


Originally published on our Knowledge Management blog