More on Sophistication of Writing Groups

In our McCulley/Cuppan version of a Documentation Capability Maturity Model, the fourth level is called Organized and Repeatable as suggested by JoAnn Hackos in her various books. But perhaps a better term to use in place of repeatable is consistent, as at the fourth level, the application of well-defined work practices is much more consistent across documentation projects. In these organizations, the majority of team members operate by the credo: we recognize some of our processes and work practices represent "best of craft" and we know they will get us through any crisis.

At this level, the writing group does keep project tracking data in a simple database. Unfortunately,  most of the data only tracks time and draft iterations. These remain the only parameters used to create project milestones.

At this level there is some recognition that the role of the medical writer involves more than "just writing" and on some teams writers are seen as knowledge managers and they are actively involved in team meetings well before data base lock. However there remains credibility issues for the writing group in the broader organization where writers are often seen as necessary evil and only "just write" the reports.

A fourth level writing organization routinely uses pre-writing planning and project kick-off meetings to shape team expectations. At this level, the writers are more aware of document design considerations that impact usability, but little effort goes into mounting discussions with teams about document design during the pre-writing planning. This remains a discussion item for draft review.

Little attempt is made to formally collect information from document end-users about readability and usability of the documents submitted to them. Any information collected happens on a casual basis and is largely applied in ineffective ways.

Some of the belief statements found in the Level 4 Writing Group are as follows:

• We are surprised and even sometimes mad at our document end-users when we get questions from them regarding information that was incorporated into submitted documents.
• We recognize that we cannot just have meetings where we talk about the data, that we need to have meetings where we plan how and what we are going to say about the data in the reports, but this does not always happen.
• We have good pre-writing planning and review tools, but it is a struggle to get the subject matter experts to actually use them.
• Reviewers still spend too much time editing and not reviewing because they believe editing style and word choice help to make a document significantly better.
• During the review process many reviewers still feel compelled to revisit sections already reviewed in an earlier draft version of the document.
• Team members recognize best practice review calls for different roles and points of focus during the review process, but many still do not follow the guidance.
• Sometimes we get stuck in our processes and still like to make all documents "look just like the last one that got approved."

So How Sophisticated is Your Writing Group?

From time to time I have talked about the sophistication of writing groups in the pharma and medical device industries. My position is that for the most part, writing work practices in the life sciences are well removed from "best of craft" work practices.

In my authoring workshops, I offer the portrayal that most writing groups in pharma and medical device companies rate only a 2 or 3 for sophisticated work practices on a six point scale. I argue that most are rudimentary at best in terms of sophistication.

This usually gets me a couple of the desired guffaws from the people in the room. I remind them, that just because you are really, really sophisticated in the conduct of science does not mean you are equally sophisticated in the tasks associated with reporting on this science.

The six levels in our writing sophistication system are based on the parameters as created by JoAnn Hackos, The scale of sophistication is as follows:

1. Oblivious
2. Ad-hoc
3. Rudimentary
4. Organized and repeatable
5. Managed and sustainable
6. Optimizing

We have created criteria for each level that differs from what Hackos did for the software world. Our criteria for Rudimentary, where we think most writing groups fall, is characterized as follows:

• We use style guides and templates for all of our documents and routinely make decisions on what to do based upon previous documents "approved" by senior management.
• We always coordinate on design and basic messages and worry about writing style across documents in a development program so that we can assure consistency in terms of appearance, style, and common messages.
• We make use of documentation project management to  assign resources and ensure documentation projects meet timelines and budgets.
• We recognize that documentation team performance varies across teams and we DO NOT know the performance factors having the greatest influence.
• We DO NOT systematically track user feedback regarding readability and usability of our documents. 

The credo for rudimentary groups is: "We always follow our routines except when we panic."

The belief statements at this level would include the following:

• We are supposed to develop information strategies for our reports before we write them, but we can never get the Subject Matter Experts to take the process seriously.
• We have lots of meetings to talk about what the data means, but we rarely have a meeting to talk about how we will represent the data in our report and never talk ahead of time on how we will represent the implications for what we see or fail to see in the data.
• We don't have time to talk about how we want to design arguments in our reports. We have more important things to do.
• We have no idea how big a document will be until after it is written.
• Just write everything you have to say and we'll fix it during review.
• Anybody with the similar professional training as I have will want to read a report in exactly the manner I choose to read it. 

So where does your group stack up?

More on Review--Surveys Show Performance Has a Long Ways to Go

In a poll on the McCulley/Cuppan website  we posed the question: "How satisfied are you with the review performance in your organization?"

The response revealed what we've consistently seen in our consulting work at McCulley/Cuppan:  
87% of respondents saw room for improvement in the review performance of their organization.

Though this was an informal poll with 38 respondents,  23 of the respondents (60%) were either Unsatisfied or Very Unsatisfied and only 3 respondents were Very Satisfied. 

Here's the breakdown:

• 3 (8%) Very Satisfied
• 2 (5%) Satisfied
• 10 (26%) Somewhat Satisfied
• 11 (29%) Unsatisfied
• 12 (32%) Very Unsatisfied

In a survey at a pharma company we asked 139 individuals to identify their personal successes and frustrations with review performance as experienced in that company. The results were quite telling:

• Listed frustrations outnumbered successes 2.5 to 1
• 13.5% of respondents cited poor reviewer discipline as their greatest frustration
• 19.5% of respondents cited collaborative pre-writinng planning of key messages as the root cause of successful document review projects.

Visualizing Argumentation

I want to share with you a book that is definitely worth exploring: Visualizing Argumentation: Software Tools for Collaborative and Educational Sense-Making.  Here's a link to the book reference page on Amazon.

The point of these tools is to support user decision-making with visual prompts that summarize the "pro" and "con" arguments on any given topic. In our McCulley/Cuppan consulting we've been huge advocates of this type of approach for years now. If you intend to make a regulatory submission document message-focused and issue-driven, then you have to create carefully crafted arguments.

Constructing arguments and at the same time understanding them is not easy, especially when working in a collaborative environment. A good argument in any research or regulatory report is a structure of messages linked in inferential or evidential relationships that supports your conclusions. Getting all the pieces and underlying propositions pulled together is not an easy task. Hence using visualization tools.

Visualization of arguments is well known in the research community as the most effective means to help foster understanding and improve critical thinking. The concept of argument mapping goes back to J. H. Wigmore and the approach of mapping remains a routine authoring tool in the legal community. I am suggesting it needs to become a more common tool in the pharmaceutical and medical device writing communities.

So back to the book: the book talks about an interesting software-based approach that is light years ahead of the tabular approach we have used for years.

Be a Reader to Be an Effective Reviewer

When I first started working at McCulley/Cuppan, Greg gave me the task of reviewing a clinical study report, not only as a way to gauge my reviewing skills, but as a starting place for lessons on judging document quality using the McCulley/Cuppan Document Standards. The first instruction he gave me was "Read before you edit."

Read before you edit.

Sounds simple, but when asked to review a document, most people start off by looking for mistakes. I remember sitting there, trying to pay attention to the document when all I could see was a misspelled word. All too often we see ourselves as that teacher of grade school with the red pen in hand, correcting verb tense and punctuation errors. We don't see the document at the meta-level so we miss the message and logic of the document.

Think about how you read a book, an article, or this blog. You read for meaning. If I leave a TyPo or too, you still understand that this post is about editing and reviewing documents. Now consider your habits when someone asks you to look at a document. Even if they say they want you to make sure that a paragraph makes logical sense, your eye probably catches all the typos and then proceeds to analyze the logic.

The truth is that inspection is easier than reviewing and is a quick way to show your contribution to a document. A way to say Look, see all these good edits I made? The reality is that lots of time is wasted, especially on first or second drafts, fixing nit-picky mistakes in grammar and punctuation on sentences that will be cut out of the document before the third draft. If you read a document (or section of a document) once all the way through, you will avoid re-reading sentences again and again just to get the meaning and avoid the need to revert your changes back to the original sentence.

During the last few months, I've been assessing a number of documents to judge reviewer performance as part of my work at McCulley/Cuppan. What I see all too often is that a reviewer will correct a sentence, changing a word or two, only to go back and change the sentence again within that same review session or in a subsequent draft. It is apparent many reviewers read to fix grammatical and stylistic errors. They are reading and analyzing each word individually versus parsing the sentence or paragraph for meaning. So a given word might seem to be the wrong choice at first glance, but if the reviewer were to read the entire paragraph, then the author's chosen word may be seen as appropriate. Style is a much overrated, overwrought topic during reviews. This is especially important when you read documents written by multiple people, where writing style and word usage is likely to vary widely.

But whether you are inspecting (editing for punctuation and grammar) or reviewing (assessing for purpose, logic, and content), reading a document in its entirety before making revisions places you closer to how the reader is going to engage with the document. In the assessments I have done, I have seen so many reviewers miss the forest because they were busily engaged inspecting branches on trees.

Editing When You Should Be Reviewing Costs Serious Money

The idealized model of document project management describes an iterative process of information planning, content specification, authorship, review, and publishing. Working to develop best-of-craft work practices aligned with this model can help organizations meet their goals efficiently and at lower costs.

Most organizations in the life sciences do not carefully consider their true costs to produce a document. When all the hidden costs associated with review are added in, the cost-per-page to produce a final version document becomes significant. If you add in opportunity costs for staff time dedicated to additional rounds of review (even if a document meets a specified deadline for the final version) then the cost-per-page skyrockets. Many organizations choose to ignore this point, which is why in my presentations on strategic review I have a slide with the Great Pyramids of Giza and the caption: “We can do anything we want as long as we have a 24/7 schedule and an expendable supply of labor.”

The goal for review is to improve document communication quality—not grammatical accuracy or data integrity, which is done via inspection. To be most effective and efficient, reviews need to be strategic and orchestrated. This means the primary focus of review by subject matter experts should be on the intended messages and arguments and testing the document to ensure it minimizes the prospects for a qualified reader to construct alternative meanings. Tasks not accomplished if you are editing a document for grammar and style. 

Our systematic analysis of review practices at numerous companies strongly suggests that while significant time and energy is expended on document review, the collective effort is generally over-sized in relation to the improvements made in the communication quality of the documents. That is, there are more people and more rounds of review than should be needed to get to a final version document. Another way to look at it is that reviews do not really move the communication quality meter very far given how much time and resources are thrown at the task. 

Our analysis suggests that review performance is hampered by poorly defined expectations for what the team wants to accomplish in a document, that reviewers do not apply systematic means of analysis to their review (meaning team reviews are largely ad-hoc feeding frenzies), and that reviewers stray from attending to the messages and arguments of a document, instead attending to matters truly related to publishing standards. In other words, reviewers stop being reviewers and become editors. Editing when you should be reviewing comes with serious costs.

Build Documents Following Logic of What the Reader is Trying to "Do"

I ended my previous post stating that documentation teams have to recognize their documents must be built following the underlying logic of what the reader is trying to "do" with the document. By this I mean there are different design choices for document content depending on what the reader is intending to accomplish through accessing the document. 


If your reader is making decisions versus merely being informed about a body of work, then this greatly impacts document content and a major difference between regulatory medical writing and the medical writing associated with the development of manuscripts.


For instance, if you are creating a summary-level regulatory submission document, the logic of what this document is to "do" for the user will dictate design decisions. A summary-level document is not intended to report results; it is intended to synthesize study findings in a way so as to build the information base for the therapeutic activity and safety of a drug or device in various patient populations. The summary documents are meant to address issues associated with development work and candidly represent limits of current understanding. It is in the summary documents that one must argue that observed differences in the data (or lack of differences) are determined reliably and provide a low probability of alternative results upon further study. In creating summary documents authoring teams must recognize that successful arguments need to establish warranted claims that respond to the known or theorized questions or concerns of the reader. 


A second example, if you are writing a risk management plan, like a RiskMAP for FDA, then the document must be designed to clearly demonstrate the logic underlying the risk assessment and the quantitative and qualitative assignment of risk. The document must clearly lay out the strategy to monitor and characterize the nature, frequency, and severity of the risks associated with the use of a product. Ultimately the document has to characterize why the proposed plan and the applied tools are the most appropriate to minimize a health outcome risk potential on a go forward basis. 


My final example, if you are writing a clinical research protocol, then you must recognize the logic of this document is not only to demonstrate study design and when assessments will be made in or out of the clinic. It is essential the document also be seen as a vehicle to represent potential benefit in relation to risk (what an institutional review board is looking to assess when they consider the merits of a research study.) Also a protocol must clearly establish the agents and actors engaged in the conduct of the research trial and the assignment of decision-making to the actors (all users of the document need to know this information.) In my review of protocols, I routinely find these documents generally obscure actors (who will do what) to varying degrees of opacity and often leave assignment of decision-making as implied versus explicit. In such instances, the document is clearly not written reflecting the underlying logic of what the document is intended to “do” in the hands of the users of the document.