Need a New Mental Model for Regulatory Documents

Wow……I have been away from this blog a whole lot longer than intended. Those competing interests….like you all understand….are the bane of my existence.

For the past couple months I have been looking closely at how people think about the “vehicles” used to communicate with regulatory health agencies. I am using the word vehicle here because I am trying to divorce myself from the notion of document, in particular, the notion of “a document.” In the modern times of on-screen reading and linked files, what is a document anyhow? To me it is the entire corpus somebody may be able to access, not just one slice of that body.

In my consulting/training interactions at McCulley/Cuppan, I find that the majority of people I interface with in the client setting operate within the mind set of individual documents (some have even smaller boundaries and operate by document sections) that are stand alone with well defined boundaries (pages and page counts.)

I want to argue that the vehicle of communication for regulatory submissions is not a document. It is the full and complete dossier submitted by the sponsor. Documents are just placeholders where I go to get a piece or pieces of information that help answer my questions. I want to argue that the regulatory reader does not see a dossier as a set of documents. Rather they see a dossier as a corpus of information that they will use to answer question and make decisions. The contents are just vehicles they peruse to get what they want.

Applying my working model means you stop seeing documents as “stand alone” and stop saying “this document has to tell a story.” I’d also like you to stop using the word document. That word has baggage I am trying to jettison. Instead I want people to view their work at least as “modules” and preferably as vehicles that help a user to answer very specific questions. Bottom line, a research report is just a part of the constellation that tells the stories. Note the plural form as we have many stories to tell in a dossier, not just one.

Applying my working model means you stop seeing your work as being like a novella—something to be read from page 1 to page n. Applying my working model means you see your body of work as something that is read in a coordinate manner that is defined by very narrowly defined aspect rules of inclusion/exclusion. Applying my model means you stop seeing pages and sections and you start seeing concepts and topics.

My argument is that the selective professional reader at regulatory health agencies cares little about documents, sections, pages, and data tables. I am suggesting such readers care solely about making informed decisions and where in the submission dossier they find vehicles that can get answers to their concept and topic questions.

How the Most Sophisticated Documentation Groups Operate

At the apex of our version of the Documentation Capability Maturity Model are the Level 6 "Optimizing" groups. These are very sophisticated writing groups that are continually looking at ways to enhance work practices and processes so as to better serve their customer needs.

At this level, the job descriptions for all the subject matter experts contain extensive descriptions regarding their roles and responsibilities in the development of high quality document products. Work performance is not merely judged on how well they execute design and conduct of studies, but also on how effective they documents are in supporting organizational strategies and economic objectives.

At this level, the writing groups rely upon carefully defined document quality standards that reach well beyond style guides and template preferences. These groups articulate detailed guidance for executing effective strategic reviews. All understand the importance of working to shared standards over individual preferences. Reviewers authenticate and sign off for documents meeting strategic intentions and communication quality. If problems arise downstream, then the reviewers are held as culpable for the problems.

Documentation project management inside a Level 6 writing group tracks the amount of time along with other parameters applied by individuals to planning, authoring, and reviewing documents. Performance is always reviewed for "lessons-learned" at the end of all major writing projects.

At this level, there is a clear commitment to the assessment of document usability for the target audience and even testing of  document designs for certain types of documents (such as clinical study protocols)  early in the document life cycle.

These writing groups take full advantage of authoring tools to assure information is effectively generated once and then repurposed to other documents as a drug or device asset moves forward in the development life cycle.

Lastly, these very sophisticated groups make time in their busy schedules for innovation both in terms of work practice and work tools.

I do not know of any groups in the pharma or med device industries who have the above mentioned attributes. Do you?

Still More on What Sophisticated Writing Groups Do

Continuing on the discussion of the last two posts regarding how we categorize the level of sophistication of writing groups. The next stop on the documentation capability sophistication chain is Level 5--Managed and sustainable.

The Level 5 writing organization applies a broad range of sustainable best-of-craft work practices.

At this level, the planning of business critical documents occurs in parallel with the planning of clinical research studies. Writing teams always deploy document prototyping techniques, such as populating sections of a clinical study report after the study protocol has been completed and planning the report results sections once the statistical analysis plan is finalized. Level 5 writing teams can tell you how many pages will be in their study report even before LPLV because there has been so much planning. Level 5 writing teams always map arguments across the sections of a clinical study.

Level 5 writing teams are aware of agile authoring techniques, but have not yet deployed these work practices. Level 5 writing teams clearly understand that repurposing information involves a whole lot more than merely cut and paste.

Level 5 writing groups clearly understand what strategic review means. The teams articulate and rely upon defined document quality standards and guidance for executing effective reviews. They always define reviewer roles and responsibilities acknowledging unique and strategic expertise.They recognize that the problems of word choice, style preferences, transcription accuracy, and format should be passed onto the writer/editors and not made a focus of review.

Level 5 writing teams routinely solicit document user information and maintain databases to help them track and understand usability and readability statistics on all of their documents. At Level 5, teams engage in root cause analysis to ascertain why questions were received from regulatory agencies. Level 5 writing teams apply standards and measures to the task of document authorship and review that are well down the highway from the simple metrics of time and draft numbers. Level 5 writing teams always engage in a lessons learned session at the end of each documentation project and such sessions are not seen as merely an activity to be filed and forgotten. Process and practice is tweaked and refined for the next time.

More on Sophistication of Writing Groups

In our McCulley/Cuppan version of a Documentation Capability Maturity Model, the fourth level is called Organized and Repeatable as suggested by JoAnn Hackos in her various books. But perhaps a better term to use in place of repeatable is consistent, as at the fourth level, the application of well-defined work practices is much more consistent across documentation projects. In these organizations, the majority of team members operate by the credo: we recognize some of our processes and work practices represent "best of craft" and we know they will get us through any crisis.

At this level, the writing group does keep project tracking data in a simple database. Unfortunately,  most of the data only tracks time and draft iterations. These remain the only parameters used to create project milestones.

At this level there is some recognition that the role of the medical writer involves more than "just writing" and on some teams writers are seen as knowledge managers and they are actively involved in team meetings well before data base lock. However there remains credibility issues for the writing group in the broader organization where writers are often seen as necessary evil and only "just write" the reports.

A fourth level writing organization routinely uses pre-writing planning and project kick-off meetings to shape team expectations. At this level, the writers are more aware of document design considerations that impact usability, but little effort goes into mounting discussions with teams about document design during the pre-writing planning. This remains a discussion item for draft review.

Little attempt is made to formally collect information from document end-users about readability and usability of the documents submitted to them. Any information collected happens on a casual basis and is largely applied in ineffective ways.

Some of the belief statements found in the Level 4 Writing Group are as follows:

• We are surprised and even sometimes mad at our document end-users when we get questions from them regarding information that was incorporated into submitted documents.
• We recognize that we cannot just have meetings where we talk about the data, that we need to have meetings where we plan how and what we are going to say about the data in the reports, but this does not always happen.
• We have good pre-writing planning and review tools, but it is a struggle to get the subject matter experts to actually use them.
• Reviewers still spend too much time editing and not reviewing because they believe editing style and word choice help to make a document significantly better.
• During the review process many reviewers still feel compelled to revisit sections already reviewed in an earlier draft version of the document.
• Team members recognize best practice review calls for different roles and points of focus during the review process, but many still do not follow the guidance.
• Sometimes we get stuck in our processes and still like to make all documents "look just like the last one that got approved."

So How Sophisticated is Your Writing Group?

From time to time I have talked about the sophistication of writing groups in the pharma and medical device industries. My position is that for the most part, writing work practices in the life sciences are well removed from "best of craft" work practices.

In my authoring workshops, I offer the portrayal that most writing groups in pharma and medical device companies rate only a 2 or 3 for sophisticated work practices on a six point scale. I argue that most are rudimentary at best in terms of sophistication.

This usually gets me a couple of the desired guffaws from the people in the room. I remind them, that just because you are really, really sophisticated in the conduct of science does not mean you are equally sophisticated in the tasks associated with reporting on this science.

The six levels in our writing sophistication system are based on the parameters as created by JoAnn Hackos, The scale of sophistication is as follows:

1. Oblivious
2. Ad-hoc
3. Rudimentary
4. Organized and repeatable
5. Managed and sustainable
6. Optimizing

We have created criteria for each level that differs from what Hackos did for the software world. Our criteria for Rudimentary, where we think most writing groups fall, is characterized as follows:

• We use style guides and templates for all of our documents and routinely make decisions on what to do based upon previous documents "approved" by senior management.
• We always coordinate on design and basic messages and worry about writing style across documents in a development program so that we can assure consistency in terms of appearance, style, and common messages.
• We make use of documentation project management to  assign resources and ensure documentation projects meet timelines and budgets.
• We recognize that documentation team performance varies across teams and we DO NOT know the performance factors having the greatest influence.
• We DO NOT systematically track user feedback regarding readability and usability of our documents. 

The credo for rudimentary groups is: "We always follow our routines except when we panic."

The belief statements at this level would include the following:

• We are supposed to develop information strategies for our reports before we write them, but we can never get the Subject Matter Experts to take the process seriously.
• We have lots of meetings to talk about what the data means, but we rarely have a meeting to talk about how we will represent the data in our report and never talk ahead of time on how we will represent the implications for what we see or fail to see in the data.
• We don't have time to talk about how we want to design arguments in our reports. We have more important things to do.
• We have no idea how big a document will be until after it is written.
• Just write everything you have to say and we'll fix it during review.
• Anybody with the similar professional training as I have will want to read a report in exactly the manner I choose to read it. 

So where does your group stack up?

Not knowing when good is good enough in writing regulatory documentation has a huge cost

We do not talk much on this blog regarding the use of language or the application of terms in science writing. Principal reason is that much of what we see in regulatory submission documents is genuinely “good enough.” However, others do not necessarily see it that way. I want to share with you how discussions in review roundtables can end up getting focused at really absurd levels of detail with a misapplied sense of establishing quality communication. 

In our consulting work, we try to be disciplined during our document reviews and only comment on language when it truly obscures or alters meaning. Being grammatically perfect in regulatory submission documents is a nice notion, but in practice consumes way too much time and organizational energy and will yield little in terms of outcomes.

We share this point with people all the time...but at times the advice goes unheeded and even worse…at times people just do not know when to move on and address real big concerns in their documents.

A case in point is a situation I observed regarding a long winded discussion in a review meeting over the use of the term “very critical”. The term “critical” in a medical sense means: of a patient's condition having unstable and abnormal vital signs and other unfavorable indicators. In theory, the meaning of critical is a black-or-white proposition without qualifications regarding gradation. Something is either critical or it is not. Therefore there should be no adverbs, like “very” in front of the term “critical” to connote a measureable degree of criticality. In this roundtable review the team got caught up in a 30 minute discussion that involved only two people arguing whether to use the term “very critical” or change it to “critical.”

Being pragmatic, I’d have to say: “Guys what are you thinking? You hold a team hostage for 30 minutes to argue over grammatical accuracy? To argue over something that will not matter when and if read by a regulatory reviewer. There were 10 professionals sitting in the room and 8 did nothing for 30 minutes. Cost of salaries alone is enough argument to say “Forget about it, let’s move on…we cannot afford to argue over such insignificant detail.” When we add in the opportunity cost (what these 10 people collectively could have been doing with their time), then for sure you have to make the argument.

This above episode gets played out time and time again in review sessions all over the pharma and medical device industries and is the reason why I am steadfast in my position that the vast majority of people involved in authorship and review do not know the answer to the question “How do you know when is good, good enough?” The end result is inordinate amounts of time can be applied at the wrong level of detail in reports and submission documents.

More thoughts on the limited sophistication of documentation practices in the life sciences

I mentioned in this post that I consider the documentation practices for creation of regulatory submission documents in most pharma and medical device enterprises to rather unsophisticated. My position is largely driven by comparing observations of documentation practices to descriptions of varying levels of documentation maturity we have developed. Our descriptors have their roots in the work presented by Joann Hackos in her book: Managing Your Documentation Projects.

Like Hackos, our documentation practices-maturity model is a six point scale ranging from: Level 0 — Oblivious to Level 5 — Optimizing. My observations suggest that the vast majority of regulatory writing falls into the 2.5 range, which is between Rudimentary and Organized/Repeatable.
The Rudimentary documentation organization is one where the vast majority of effort is placed to ensure documentation consistency. All documents are generated utilizing well characterized templates. Document strategic review is largely absent as energy is applied to ensuring structural (that is, grammar and format) accuracy and consistency. Work practices are highly individualized and there is little application of meaningful estimating of document size or time lines.

The belief statements of organizations working at the rudimentary level include the following:

• All writers manage their own projects
• We would like to know more about our reading audience, but nobody takes the time to learn more….so we “suppose” what they want in our documents
• Our users are just like us…….I am a subject matter expert, so every other person educated like me will read documents just like I do
• We talk about what the data means, but we rarely talk about how to represent this meaning in our documents until we are into round table reviews
• We have little concern with how many rounds of review go into creating the final version of a document….we strongly endorse the credo “the end justifies the means”
• Planning document content before actually writing a report is just busy work
• We routinely reverse engineer document development timelines from the stated publication deadline and not from the scale and scope of the intended document
• We care what the customer thinks of our documents, but we do not use any organizationally applied quality standards. Standards are principally driven by teams and their senior reviewers
• File and forget–we do not take time to collectively reflect on documentation work practices practices

The belief statements of organizations working at the organized and repeatable level include the following:

• We have begun to study our document users but see little value in a concerted effort to collect information on how well our documents “satisfice” their needs
• We are surprised or even indignant when we get questions from our regulatory user looking for information that we included in our submission package
• The quality of our document project management is inconsistent, but we are okay with that because that is reality and each project is unique
• We do not see meaningful metrics beyond time for documentation projections…if we meet the deadline, then ways of working had to be good
• Nobody knows what others are doing in the process of review…the only way they know is via what may be discussed during a round table review
• We believe in our described “ways of working” until faced with unexpected situations, then we panic and call for “all hands on deck”
• We can easily get caught up in “process” at the expense of “product”

Now let’s contrast the above with belief statements of organizations working at the optimized level which include the following:

• We always engage in collaborative pre-writing planning of documents to make sure we fulfill strategic purpose of any given document
• We are thoroughly committed to understanding our document users and we work to systematically collect information from them
• We know how big a document will be even before we write it
• We always do end-of-project analysis to collect lessons learned and then disseminate this information across the organization
• We maintain a database of critical documentation work practice parameters and benchmark all documentation projects
• We recognize that accuracy and consistency are just the start of ensuring quality….we have expanded focus to quality of argument and document usability
• We are learning how to be innovative and not let the process control us

Originally published on our Knowledge Management blog

Why documentation work practices in pharma and medical device industries need enhancement

I recently taught a workshop for a client where I gave the participants a good news/bad news scenario. The good news, I suggested, is the documentation work practices I observed in their company were like what I saw in other pharma companies. I then suggested the bad news is their work practices were like what I saw in other companies. My point to them is that the bar for sophisticated work practices is set pretty low in the pharmaceutical industry.  I used as my line of reasoning our McCulley/Cuppan modification of JoAnn Hackos’ Document Capability Maturity Model (DCMM) she presents in her book: “Managing Your Documentation Projects.”

To this group I suggested their documentation work practices would come out at about a 2 on our 6 point scale, which suggests also the point on the scale I will place most of the documentation groups in the pharma industry. Most were surprised at the low ranking and a few were even offended I think. By the end of our workshop, all were feeling different about what it takes to create high quality technical documentation in the life sciences.

I’ll characterize here a documentation project undertaken by this group I had in the workshop and suggest such work is emblematic as to why I give them (and the industry) a low DCMM score.
The group generated a briefing document to be submitted to a regulatory health agency in advance of a request (a standard practice across the industry.) The briefing document is intended to inform the health agency as to what topics the company wanted to receive feedback from the health agency regarding an ongoing drug development project.
The company had an hour meeting with the agency and ended up wanting to address 8 topics within the hour (an ambitious agenda.)

The documentation team went through a three draft process to create the briefing package where the number of questions and the actual questions changed w/ each draft. I mean they were truly using a “fire; aim; now we are ready” approach to create this briefing package. The approach of: "activity is everything, so we are way too busy to plan."

Each draft looked dramatically different than the previous version and required massive investment of time and energy from the team. In the end the briefing package was 83 pages—that is a little over 10 pages per question. The first question did not appear until page 25 (which means there were 24 pages of introductions and throat clearing before we get to the meat. Curiously to me, none of the questions appeared in the Table of Contents of the briefing package and the document did not have an Executive Summary (we want you to work as hard in reading this document as we did in crafting it.)

Then I asked the authoring team what they did as a documentation project post-mortem for lessons learned, such as solicit usability feedback from the customer using the document or to identify how the documentation work practices impacted the process and the product. The answer was, “Oh we did not do anything like what you suggested. The document must have worked because we had our meeting with the agency. We just moved on to the next document.” I asked if they got what they wanted out of the meeting. The answer was a categorical no. “We did not get an answer to our most important question as we had placed it as 8 of 8 in the queue. We ran out of time to address that question.”

I asked how uniform their documentation practices are for the creation for briefing packages (they will generate such packages numerous times in a given year for various development projects). Response was “We have no set approach, so each briefing package will be generated utilizing variable methods and processes.”

How they could be surprised with a low rating for the sophistication of their documentation practices is the interesting question for me. Where comes the comfort with such outcomes? In part, I think, it is due to erroneously assigning the sophistication in conduct and analysis of science as carrying over to documentation work practices.

Originally published on our Knowledge Management blog