How the Most Sophisticated Documentation Groups Operate

At the apex of our version of the Documentation Capability Maturity Model are the Level 6 "Optimizing" groups. These are very sophisticated writing groups that are continually looking at ways to enhance work practices and processes so as to better serve their customer needs.

At this level, the job descriptions for all the subject matter experts contain extensive descriptions regarding their roles and responsibilities in the development of high quality document products. Work performance is not merely judged on how well they execute design and conduct of studies, but also on how effective they documents are in supporting organizational strategies and economic objectives.

At this level, the writing groups rely upon carefully defined document quality standards that reach well beyond style guides and template preferences. These groups articulate detailed guidance for executing effective strategic reviews. All understand the importance of working to shared standards over individual preferences. Reviewers authenticate and sign off for documents meeting strategic intentions and communication quality. If problems arise downstream, then the reviewers are held as culpable for the problems.

Documentation project management inside a Level 6 writing group tracks the amount of time along with other parameters applied by individuals to planning, authoring, and reviewing documents. Performance is always reviewed for "lessons-learned" at the end of all major writing projects.

At this level, there is a clear commitment to the assessment of document usability for the target audience and even testing of  document designs for certain types of documents (such as clinical study protocols)  early in the document life cycle.

These writing groups take full advantage of authoring tools to assure information is effectively generated once and then repurposed to other documents as a drug or device asset moves forward in the development life cycle.

Lastly, these very sophisticated groups make time in their busy schedules for innovation both in terms of work practice and work tools.

I do not know of any groups in the pharma or med device industries who have the above mentioned attributes. Do you?

More on Sophistication of Writing Groups

In our McCulley/Cuppan version of a Documentation Capability Maturity Model, the fourth level is called Organized and Repeatable as suggested by JoAnn Hackos in her various books. But perhaps a better term to use in place of repeatable is consistent, as at the fourth level, the application of well-defined work practices is much more consistent across documentation projects. In these organizations, the majority of team members operate by the credo: we recognize some of our processes and work practices represent "best of craft" and we know they will get us through any crisis.

At this level, the writing group does keep project tracking data in a simple database. Unfortunately,  most of the data only tracks time and draft iterations. These remain the only parameters used to create project milestones.

At this level there is some recognition that the role of the medical writer involves more than "just writing" and on some teams writers are seen as knowledge managers and they are actively involved in team meetings well before data base lock. However there remains credibility issues for the writing group in the broader organization where writers are often seen as necessary evil and only "just write" the reports.

A fourth level writing organization routinely uses pre-writing planning and project kick-off meetings to shape team expectations. At this level, the writers are more aware of document design considerations that impact usability, but little effort goes into mounting discussions with teams about document design during the pre-writing planning. This remains a discussion item for draft review.

Little attempt is made to formally collect information from document end-users about readability and usability of the documents submitted to them. Any information collected happens on a casual basis and is largely applied in ineffective ways.

Some of the belief statements found in the Level 4 Writing Group are as follows:

• We are surprised and even sometimes mad at our document end-users when we get questions from them regarding information that was incorporated into submitted documents.
• We recognize that we cannot just have meetings where we talk about the data, that we need to have meetings where we plan how and what we are going to say about the data in the reports, but this does not always happen.
• We have good pre-writing planning and review tools, but it is a struggle to get the subject matter experts to actually use them.
• Reviewers still spend too much time editing and not reviewing because they believe editing style and word choice help to make a document significantly better.
• During the review process many reviewers still feel compelled to revisit sections already reviewed in an earlier draft version of the document.
• Team members recognize best practice review calls for different roles and points of focus during the review process, but many still do not follow the guidance.
• Sometimes we get stuck in our processes and still like to make all documents "look just like the last one that got approved."

Designing the architecture of the argument in development reports

Kirk Livingston, a teacher and a medical writer working largely in the medical device industry, as well as a fellow blogger at LivingstonContent, shared this comment on my previous post regarding poor rhetorical shaping of arguments in research reports.

There’s a lot of work involved with producing solid, well-reasoned conclusions. Can it even be accomplished as an “authoring team” or is it the work of an individual? Recent research about medical device companies in Minnesota suggests communication teams are chronically understaffed. So–who has time to come to the right conclusions? Thanks for the thoughtful post.

I agree there is considerable work that goes into producing solid, well-reasoned conclusions. I am certain the work can indeed be accomplished by an authoring team. The caveat here is, it can be accomplished as long as the team engages in truly collaborative authorship work practices and makes use of pre-writing planning tools to help shape the argument.

I am not so sure that writing teams are chronically understaffed. I think the real issue here is the limits of interest and skill that team members may have towards the task of writing. As I reflect on 17 years of work associated with the authorship of regulatory documentation, I am convinced adding numbers to the equation will have little bearing on the rhetorical qualities of any given document. Larger writing teams will likely yield only emotional comfort--the notion of safety in numbers.

Producing high quality documents is a function of knowing what you want the document to do for you, a sense of where arguments must be played out in a document, and what writing tools to use in order to get true collaboration and sharpen everyone's focus to achieve the objectives you want to document to support.

Producing high quality documents in the forum of pharmaceutical and medical device research requires understanding how to build out the red thread of logic in a research report. In pre-writing planning it starts with something as simple as building a table that is to be filled in by the authoring team. The table has three columns to be completed by the team:

Primary & Secondary Objectives | Conclusions | Key Data

You then have one row in the table for each objective.

The team's task is then is to build out conclusions about achievement of each objective and what data warrants that conclusion.  A simple but powerful writing tool that helps a team to lend considerable shape to the architecture of the argument that must be represented within a report.


Originally published on our Knowledge Management blog

Why documentation work practices in pharma and medical device industries need enhancement

I recently taught a workshop for a client where I gave the participants a good news/bad news scenario. The good news, I suggested, is the documentation work practices I observed in their company were like what I saw in other pharma companies. I then suggested the bad news is their work practices were like what I saw in other companies. My point to them is that the bar for sophisticated work practices is set pretty low in the pharmaceutical industry.  I used as my line of reasoning our McCulley/Cuppan modification of JoAnn Hackos’ Document Capability Maturity Model (DCMM) she presents in her book: “Managing Your Documentation Projects.”

To this group I suggested their documentation work practices would come out at about a 2 on our 6 point scale, which suggests also the point on the scale I will place most of the documentation groups in the pharma industry. Most were surprised at the low ranking and a few were even offended I think. By the end of our workshop, all were feeling different about what it takes to create high quality technical documentation in the life sciences.

I’ll characterize here a documentation project undertaken by this group I had in the workshop and suggest such work is emblematic as to why I give them (and the industry) a low DCMM score.
The group generated a briefing document to be submitted to a regulatory health agency in advance of a request (a standard practice across the industry.) The briefing document is intended to inform the health agency as to what topics the company wanted to receive feedback from the health agency regarding an ongoing drug development project.
The company had an hour meeting with the agency and ended up wanting to address 8 topics within the hour (an ambitious agenda.)

The documentation team went through a three draft process to create the briefing package where the number of questions and the actual questions changed w/ each draft. I mean they were truly using a “fire; aim; now we are ready” approach to create this briefing package. The approach of: "activity is everything, so we are way too busy to plan."

Each draft looked dramatically different than the previous version and required massive investment of time and energy from the team. In the end the briefing package was 83 pages—that is a little over 10 pages per question. The first question did not appear until page 25 (which means there were 24 pages of introductions and throat clearing before we get to the meat. Curiously to me, none of the questions appeared in the Table of Contents of the briefing package and the document did not have an Executive Summary (we want you to work as hard in reading this document as we did in crafting it.)

Then I asked the authoring team what they did as a documentation project post-mortem for lessons learned, such as solicit usability feedback from the customer using the document or to identify how the documentation work practices impacted the process and the product. The answer was, “Oh we did not do anything like what you suggested. The document must have worked because we had our meeting with the agency. We just moved on to the next document.” I asked if they got what they wanted out of the meeting. The answer was a categorical no. “We did not get an answer to our most important question as we had placed it as 8 of 8 in the queue. We ran out of time to address that question.”

I asked how uniform their documentation practices are for the creation for briefing packages (they will generate such packages numerous times in a given year for various development projects). Response was “We have no set approach, so each briefing package will be generated utilizing variable methods and processes.”

How they could be surprised with a low rating for the sophistication of their documentation practices is the interesting question for me. Where comes the comfort with such outcomes? In part, I think, it is due to erroneously assigning the sophistication in conduct and analysis of science as carrying over to documentation work practices.

Originally published on our Knowledge Management blog

The importance of understanding the reader and the need to be informed by reading theory

In thinking about the ExLPharma conference where I spoke two weeks ago on review, I talked for some time about the importance of reviewers understanding their readers and the need to be informed by reading theory so as to become a truly good reviewer. My comments were largely well received by the group attending the conference and, for most of the points I raised regarding reading theory, were truly novel for them. I came away from the conference with the notion that few people actively engaged in creating business documents have invested time in thinking about what regulatory readers actually “do” with their documents and how they actually read these mission-critical documents.

Given that most people inspect a document for accuracy tells me that the prevalent view regarding reading is “if we get the numbers and the words right, then we are good to go and surely everyone will understand what we mean.”

I remain perplexed as to why this view is so uniformly applied. I am even more perplexed why some people I cross paths with in my client settings struggle to accept the notion that readers construct a meaning that they personally create from a text, so that "what a text means" can differ from reader to reader. Readers construct meaning based not only on the visual cues in the text (the words and format of the page itself) but also based on the knowledge readers already have stored in their memory. This pre-existing knowledge readers bring with them as they encounter a text is very potent and not much appreciated by many I work with across the life sciences.


Originally published on our Knowledge Management blog

Proving Document Quality

You say you produce high-quality document products? Great, now prove it! Although many can recognize quality when they read it, the prospect of measuring documentation quality seems dubious to many.

I work with a great many clients where authors say they generate high quality documents, but they can only show that their documents are accurate. That is, accurate in terms of template, source data, and grammar conformance. Beyond these three inspection parameters, many regard documentation quality as inherently immeasurable. I argue that these three parameters by themselves are wholly insufficient measures of document quality and that important aspects of how documents convey meaning can be measured.

Surely they must see other metrics of value, especially when considering factors like the number of regulatory questions elicited by errors of omission and commission in their very accurate submission documents or the number of protocol amendments generated over the life of their very accurate research protocol.

Some see additional metrics as meaningful, but not for their type of documentation. In the realm of clinical development documentation, many consider their output unique and beyond measurement. I hear more than a few medical writers say, “We know our own field intimately, and everywhere we look we see shades of gray and unique situations, so measuring is not useful.” and others in clinical development say “We are unique, and your notions of quality do not apply to us.”

Still others find the prospect of measuring output impractical or suggest that even if they had data, “Against what standards could we make comparisons?”

Even easily measured attributes like accuracy are not effectively tracked. Documents are routinely verified/corrected by quality control operators, yet the output is rarely tabulated and statistically analyzed to provide a measure of writer performance. The intention is merely to ensure the document is archived as accurate in terms of template, source data, and grammar conformance.

I do not know of any pharmaceutical or medical device company that makes use of any statistical quality control techniques to track their document quality. It would be easy to apply similar principles as used in the auto industry or pharmaceutical manufacturing. The auto manufacturers turn out millions of units every year, and each company tracks manufacturing defects, but not by examining every car. Instead, they pull a few cars off the assembly line and tear them apart. Pharmaceutical manufacturers test an appropriate sample from each manufacturing lot. In both cases the products are measured against a set of well-defined acceptance criteria.

Such acceptance criteria exist for technical and scientific documents (in our work at McCulley/Cuppan we have developed such a set of criteria that we use during our training and consulting work for clients) so such quality control methods could easily be deployed.


Originally published on our Knowledge Management blog

More on Topic: Is Medical Writing Really a Profession?

I posed this question on a medical writing affinity group I belong to on Linked In (Professional Medical / Scientific Writers). The shared commentary, all from medical writers, has been quite interesting. In this post I want to share a few of the commentaries and my responses.

It was stated by one commentator: “I do feel, because of the way this question is worded, there is a need to defend my profession.” Another said: “It seems to me that whether or not one describes medical writing as a profession is partly down to semantics, and what any individual person understands by the term profession.”  In response, I am not asking you to defend your work, but I am asking you to defend that the work of medical writers as a class is indeed a profession. I shared some thoughts regarding how the scholars characterize the concept of a profession, as I think much more than semantics is involved here.

Another commented “Is medical writing a profession? Well, some parts of it are more professionalized than others, but I think.... as a whole? No, currently it's a job description.” I wholly concur with this position for reasons I have already expressed and additional reasons I will add in this discussion entry. I modified slightly another comment made in the Linked In discussion to help shape a sense of the needed attributes for applying the moniker of professional to medical writing: “That practitioners are expected to understand the complexities of language and communication we've seen in this discussion so far and medical writers use workplace research and academic theory to push the practice of medical writing forward.”

Another interesting comment posed as a question is the following: “On another level, the question is: are you hiring a profession or an individual?” My observations find writing approaches within organizations and across the industry are idiosyncratic and outcomes unpredictable, so therefore the answer is clearly you are hiring individuals. By the way—this variability contributes to why so many enterprises struggle to identify “markers” to use to judge capabilities of medical writing candidates.
This same person said: “which part do we emphasize: the medical or the writer?” The answer I suggest is neither—the emphasis is on reporting first and foremost (and reporting in a manner that is accurate and compliant), then medical insight, and a distant third is writing (in terms of sharing information and shaping knowledge).

Lastly, I want to highlight the following comments: “Currently, I don't know that medical writers are seen as the experts on Medical Writing -- most of the public discourse seems to think we are unreflective cheerleaders for whoever pays our salaries....” and “ We do a lot of the applied science stuff already. But we have not come together to share that as professionals, and I think that's the biggest barrier to professionalism .”  Harsh commentaries both, but I suggest each is valid based on my observations of the ways of working in this industry. Unfortunately, I will suggest medical writers who do “applied science stuff” in terms of writing practice and document products are far and few between.

Yes, I want medical writing to be considered a profession. I did not start this post to raise a beef with the notion of medical writing. I opened in the discussion in the hopes of fomenting a thought-filled argument about what it takes to call oneself a professional medical writer. An aspect of professionalism is intelligent practice as defined by scholars like Schon. Intelligent practice is the application of knowledge to the body of work at hand. In this case, documents. I am suggesting that such intelligent practice is not a common attribute within or across communities of medical writing. I say this because of a significant amount of data that suggests the vast majority of documents produced by medical writers fail. These documents fail in terms of purpose, conveyance of logic, and meeting reader’s needs.

In defense of my claim I submit to you that there is considerable data in the public record regarding target-audience usability for the output of medical writing. A survey of keen interest to me shows that FDA reviewers cite poor document quality as a significant impediment to effective, efficient reviews; a comprehension study of clinical study protocols found that over 50% of test subjects (clinical investigators and study sight managers) failed to achieve a passing score in identifying roles and decision making responsibilities in the conduct of a clinical trial. I also suggest that much has been and continues to be written about the poor quality of manuscripts submitted to journals for publications. Additionally, analyses of the writing output in various clinical research genre suggests the documents are written for a persona that has a high command of the English language, enjoys reading long, dense narrative passages, and has an exquisite memory (I am pretty sure this persona does not match the broad audience these document products are intended to serve.)

When I raise these above points with medical writers, which I do all the time, I can say nearly all are unaware of any elements of the work I generalized above. Further, the vast majority of medical writers I encounter are unaware of any aspects of reading theory. Also I suggest to you that few medical writers can distinguish the differences between the concepts of data, versus information, versus knowledge; few of the hundreds of medical writers I have spoken to in the past 10 years have heard of the concept of plain language; most have not heard of the concept of document usability; and even fewer attempt to apply such concepts to their work (why would you when you see your work is to report versus transfer knowledge.) There are certainly individuals and even groups of medical writers who operate in a manner that warrant use of the term professional, but I am unable to find as appropriate, the broad application of the term to the various communities within medical writing.

I am suggesting that the intelligent practice of seeing clinical research documents as tools to be used by others outside of an enterprise is largely absent from the “ways of working” mindset of most medical writers. I am also suggesting that the task of document production—at all levels and for all forms of documentation in the clinical research community—is largely inefficient. Few know what it really costs the organization per page to produce a final version document (many are afraid to know that number); few consider techniques and tools to really transform the document production process (much of what I see done to improve work streams is at best incremental, but is largely “feel good” activity that has no impact whatsoever); few engage in meaningful post work reflection where performance is analyzed consistently across document projects by well-described standards. I could go on, but I think this is enough detail to support my position.

A hallmark of professionals as defined by Donald Schon and others is a continual reflection on practice, both during work and post-mortem and then application of what is learned to transform ways of working and product. If reflection is occurring, then I suggest little fruit is born by the activity for I hear all of the same criticisms of medical writers and clinical documents today as I did in 1994.


Originally published on our Knowledge Management blog

More On Why Do So Many Feel Complex Language Is Needed To Have Good Scientific Writing

I made post a few weeks back addressing the topic of why some authors in the life sciences feel compelled to construct documents that are dense and difficult to read. Here's a link to that post.  The post generated quite a bit of discussion in a medical writing discussion group I belong to on LinkedIn.  The discussion there has motivated me to do some further investigation on this topic. I want to share with you some interesting pieces of information that I hope impacts how you think about what constitutes "good writing."

I came across a series of papers by J. Scott Armstrong, who is at the Wharton School of Business at the University of Pennsylvania. In these papers, Armstrong considers the broad question that academic communications should enhance knowledge and, therefore, researchers should invest energy in developing understandable ways to present their findings.

In his paper, Unintelligible Management Research and Academic Prestige, Armstrong explores the notion that if the goal of successful communication is to share information with others and, if science places a premium on successful communication, then, all other things being equal, journals should prefer articles that are clearly written to those that are not. Armstrong concluded from the studies presented in this paper that clear communication of one's research is not the norm in the prestigious journals he examined, nor is it widely appreciated in his small test of the academic community at three universities.

In a another paper, Research on Scientific Journals:Implications for Editors and Authors Armstrong states the following: "A review of editorial policies of leading journals and of research relevant to scientific journals revealed conflicts between 'science' and 'scientists'. Owing to these conflicts, papers are often weak on objectivity and replicability. Furthermore, papers often fall short on importance, competence, intelligibility, or efficiency."

In yet another paper, Barriers to Scientific Contributions: The Author’s Formula, published in the journal Behavioral and Brain Sciences, Armstrong, with tongue in cheek, describes a set of rules that authors can use to increase the likelihood and speed of acceptance of their manuscripts. Authors should: (1) not pick an important problem, (2) not challenge existing beliefs, (3) not obtain surprising results, (4) not use simple methods, (5) not provide full disclosure, and (6) not write clearly.


Originally published on our Knowledge Management blog

How Do You Measure Communication Quality?

One of the truisms we see in our McCulley/Cuppan consulting work is that rounds of document review tend to go until the point when the document must be sent somewhere. That's why we say that in the pharmaceutical industry, the opportunities for making changes to a document are virtually limitless. The problem driving this situation is most people involved with authoring and reviewing process do not have good markers to inform them of the overall communication quality of a document.  So without good markers they are left to utilize really poor markers to help them measure document quality. Markers like: grammatical soundness; how many people have reviewed the document; how many rounds of review; and how many comments leveled on the text and data in the document. Unfortunately, these markers have little correlation in the case of grammatical soundness and, for the other three, no correlation whatsoever to the communication quality of a document.

To paraphrase Steve Jong in his paper (you can read it here)  You Get What You Measure—So Measure Quality: "if you don't measure it, you'll never get it."  This is so true with document communication quality.  In order to measure communication quality you have to employ meaningful markers. We find our clients typically employ only two markers that are useful: accuracy and compliance. Unfortunately, neither of these do much to measure the quality of argument, soundness of logic, or overall usability of a document for the end-user. There are some useful markers to consider for measuring these document attributes. More on these markers in my next post.


Originally published on our Knowledge Management blog

How Do We Get People to Apply Improved Work Practices?

I am always thinking about how we can get people to really apply improved work practices. In our work situation, people readily accept the ideas and principles we share regarding ways of working on knowledge elicitation and document development, but then I see people routinely retreat to old ways of working. Lots can be done to help counter these tendencies.

A couple ideas come from  a great post I read this morning by Elizabeth Harrin on the blog Project Management Tips.  Her post addresses the very interesting question of  "how do you make lessons-learned stick?"  Two points in her article stand out for me:

1. Do not wait until the end of the project to review lessons learned.
2. Make it difficult for people to do things the old way.

Great concepts and unfortunately, concepts I rarely see implemented.
We get involved in client projects that carry on for months or years--frankly, it is really too late to wait until the project is over to collect lessons learned. I will endeavor to apply Point One in all of my project work with clients. I think we take this approach at McCulley/Cuppan on projects, but not in a consistently applied formal way.

Point Two is a really interesting concept shared by Ms. Harrin, but one that takes steely determination and solid support from the highest levels of the organization. In my work, I find the underlying issue behind this point to be that people just hate leaving their comfort zones, even if they recognize their personal ways of working are suboptimal. Point Two is all about making a retreat to the recesses of comfort zones an uncomfortable process. I recall a number of years ago we had a client who wanted the organization to stop printing and filing paper versions of reports and memos. So they did two things: took away the personal file cabinets and limited access to departmental printers. Initially the situation was like being on the famous British Naval vessel, The Bounty, but all quickly learned to reinvent their personal ways of working, those that could not tolerate the demanded change in habits soon left the organization.

These two points connect back  to another important concept I picked up on a post by Wendy Wickham on the blog: In the Middle of the Curve.  She suggests that business training typically is centered on developing knowledge (she refers to it as brainpower) and takes little time to actually help people understand how to manage it. This is so true. It is one thing to intellectualize ideas and concepts, but the challenge is to actually apply the ideas and concepts. Pausing mid-stream in a project can help and changing work environments to prevent a retreat to old habits can too (though this approach comes pre-loaded with the potential for encouraging bad morale and other social issues that can sink a project real fast.)


Originally published on our Knowledge Management blog

Just What Do We Mean by Collaborative vs Cooperative?

I spent considerable time working the past two weeks with three different clients developing three different business platforms with three different work cultures in three different geographic locations. Yet they all do their development work , especially mission-critical development work and associated documentation (such as regulatory submissions) in essentially the same way. All three companies engage in cooperative work with very little collaboration occurring at any time in the process. Generally when collaboration does happen it is very, very late in the process and occurs only during meetings to address review comments on documents.  That is it for collaboration, a thin veneer very late in the process.

In all three instances, the people I work with believe they are indeed operating in a very collaborative work environments. They sit in dismay listening  to my characterization of their work practices as  described above. But after we the peel the onion, they begin to appreciate the distinctions of working merely at the level of cooperation versus the level of collaboration. My starting point with clients is to provide an effective working definition of what we at McCulley/Cuppan mean by collaborative versus cooperative work practices.

As a starting point for any discussion we have to examine the fundamental difference between collaboration and cooperation. The line of demarcation is the level of formality in the relationships between departments or stakeholders in the conduction of work to support a common goal, which in the pharmaceutical industry is to bring a new drug or line extension to the market. I tell my clients that collaboration involves these departments or stakeholders coming together and fundamentally changing their individual approaches to sharing of resources and responsibilities as well as ways of working and information sharing. Cooperation on the other hand is where departments or stakeholders maintain their separate mandates and responsibilities, engage in doing most work as they see appropriate (and generally isolated from others on the project)  but may agree to do some work together or present work for review by other stakeholders or departments in order to meet a common goal.

To help drive further discussion on just what do we mean by collaborative versus cooperative, I suggest you read this David Eaves post discussing his perspective on cooperating versus collaborating.

Originally published on our Knowledge Management blog