More on What is a Document?

So what is a document?

In response to my last blog post, I have been asked by several individuals—"so then what is a document?"

My short answer—"I do not know for sure."

Now for the long answer.

The widely accepted definition for a document is as a textual record. This definition served us well in the past. But now with digital records, semiotics, and information retrieval tools; I am not sure the definition meets the needs of how we communicate in 2012.

As early as the 1930s Paul Otlet, an Information Scientist of considerable renown, suggested that the definition of documents also include digital images and even three dimensional objects. I am not prepared to toss all the elements Otlet describes into the mix. But I am prepared to suggest that documents are organized physical evidence and as such the organization transcends the classic definition for a document as this vehicle is a less relevant communication medium in 2012 than it was is 1982. I do not have a preferred term, I wish I did, but I do suggest we attempt to move away from the term document as it suggests a domain for organized physical evidence that does not match the reality of the digital age.

Suzanne Briet suggested a definition some time ago that a document is evidence in support of a fact. I rather like this notion. She makes the point that documents should not be viewed as being concerned with texts, but with access to the evidence. I suggest this is the essence of all regulatory writing that I talk about often in this Blog. If one considers the models in place for electronic drug submissions, thinking in the classic terms of 8.5 x 11 and A4 is really not very useful.

Rather it is better to be thinking in terms of taxonomies of information or perhaps even semiotics. Semiotics is the study of signs, indication, designation, signification, and communication. Semiotics is closely related to the field of linguistics. I look at semiotics as a valid attribute for this discussion because the life sciences are driven by numbers and what are numbers, but signs and the significance of those signs.

Then there is Michael Buckland who talks about how a key characteristic of “information-as-knowledge” is that it is intangible: one cannot touch it or measure it in any direct way. Knowledge, belief, and opinion are personal, subjective, and conceptual. Therefore, to communicate them, they have to be expressed, described, or represented in some physical way, as a signal or communication.

What we are really talking about happening in regulatory submission packages is the conveyance of knowledge.  This conveyance often transcends the boundaries of a traditional text, that is, a document as it is generally defined. The Briet notion of "evidence in support of a fact" works well as a definition of a document especially if we change the quote to read "evidence is support of a claim."

Need a New Mental Model for Regulatory Documents

Wow……I have been away from this blog a whole lot longer than intended. Those competing interests….like you all understand….are the bane of my existence.

For the past couple months I have been looking closely at how people think about the “vehicles” used to communicate with regulatory health agencies. I am using the word vehicle here because I am trying to divorce myself from the notion of document, in particular, the notion of “a document.” In the modern times of on-screen reading and linked files, what is a document anyhow? To me it is the entire corpus somebody may be able to access, not just one slice of that body.

In my consulting/training interactions at McCulley/Cuppan, I find that the majority of people I interface with in the client setting operate within the mind set of individual documents (some have even smaller boundaries and operate by document sections) that are stand alone with well defined boundaries (pages and page counts.)

I want to argue that the vehicle of communication for regulatory submissions is not a document. It is the full and complete dossier submitted by the sponsor. Documents are just placeholders where I go to get a piece or pieces of information that help answer my questions. I want to argue that the regulatory reader does not see a dossier as a set of documents. Rather they see a dossier as a corpus of information that they will use to answer question and make decisions. The contents are just vehicles they peruse to get what they want.

Applying my working model means you stop seeing documents as “stand alone” and stop saying “this document has to tell a story.” I’d also like you to stop using the word document. That word has baggage I am trying to jettison. Instead I want people to view their work at least as “modules” and preferably as vehicles that help a user to answer very specific questions. Bottom line, a research report is just a part of the constellation that tells the stories. Note the plural form as we have many stories to tell in a dossier, not just one.

Applying my working model means you stop seeing your work as being like a novella—something to be read from page 1 to page n. Applying my working model means you see your body of work as something that is read in a coordinate manner that is defined by very narrowly defined aspect rules of inclusion/exclusion. Applying my model means you stop seeing pages and sections and you start seeing concepts and topics.

My argument is that the selective professional reader at regulatory health agencies cares little about documents, sections, pages, and data tables. I am suggesting such readers care solely about making informed decisions and where in the submission dossier they find vehicles that can get answers to their concept and topic questions.

Why do we still organize clinical research reports by the IMRAD convention?

Here is Part 2 of the questions I posed in my previous blog post.

I have been thinking about this question for some time now. I remain curious as to why we still organize clinical research reports by the long-standing convention of Introduction, Objectives, Methods, Results, Discussion, and Conclusion? The IMRAD form of organization. The year 1665 is often cited as the origin of the platform commonly referred to as scientific papers. It was not until the second half of the 1800s that these documents moved in the direction of “theory – experiment – discussion.” Then starting in the early 1950s the IMRAD structure became the prominent norm for the structure of a scientific paper. 

The supposed reason for the IMRAD structure is that this organizational approach facilitates literature review, allowing readers to navigate articles more quickly to locate material relevant to their purpose. It has been suggested that the IMRAD structure effectively supports a reordering that eliminates unnecessary detail, and allow the reader to assess a well-ordered and noise free presentation of the relevant and significant information. I can see how this argument may apply to research manuscripts published in Journals, but it certainly does not apply to the kinds of study reports submitted to regulatory agency health authorities.  

Unfortunately, the supposedly neat order of the IMRAD arrayed report rarely corresponds to how regulatory readers consider using documents to help them make decisions. That is, decisions about how study results support or fail to support broad or narrowly defined questions/arguments. People are surprised to learn that health authority review agents rarely read study reports. They certainly engage with study reports and make use of the content. But read these documents? No, that is an approach not normally taken by the health authority review agent. 

Frankly, I cannot blame them. After all, the IMRAD form of organization does little to help them get answers to their questions or support their approach to making decisions.

Over the past 50 years, the idealized sequence of the IMRAD structure has on occasion been criticized for being too rigid and simplistic. Perhaps the most famous castigation was made by Peter Medawar in the early 60s. He criticized the IMRAD design for not giving a realistic representation of the thought processes of the writing scientist. And then in the mid 80s A. G. Gross wrote a paper titled: “The form of an experimental paper: A realization of the myth of induction” that challenged the notion of the IMRAD structure as well. 

I am right there too in questioning the merits of this approach. At least when it comes to study reports submitted to health authorities.

I believe the reason for rigid adherence to the IMRAD approach for research document organization is the precedence of the past. Not that it affords the most effective stucture for the reader to get what they want from a document.

As for me—I’d prefer a Question & Answer approach—something along the lines of organizing the report by study objectives with the study methods appropriately subordinated at the back of the report. Give me a report organized as follows: Objective—Results—Discussion—Assignment of Significance. Each section of the report deals with only one study objective and information is presented in the sequence I have suggested. Then the last section of the report would be Conclusions where you integrate and contextualize the study findings to answer the really big question of so what do all these findings mean?

So what do you think is the most effective form of organization for a regulatory clinical study report?

Why do we do some of the things we do in clinical study reports? Part 1

Over the past three months I have been doing a series of workshops focused on the authorship of clinical research reports for a couple clients. This work has me posing questions about two very specific notions

1. Why do people still demand that a clinical research report intended for incorporation into regulatory submission packages must be written as “stand-alone” documentation?
2. Why do we still organize clinical research reports by the long-standing convention of Introduction, Objectives, Methods, Results, Discussion, and Conclusion?

In this post let’s consider the notion of the “stand-alone” clinical study report.

First question I have is why must these documents be stand alone? The term “stand alone” generally refers to a document or device that is self-contained and does not require any other document or device to fully function. I argue that a clinical study report submitted to a drug regulatory agency is part of a larger corpus and as such the report should never be construed as a stand-alone document.

It is a given that most submissions made to drug regulatory agencies contain multiple research studies. So you will never have a “stand-alone” research study. So an individual study is part of a corpus of clinical research that best operates collectively. Also regulatory submissions have multiple sections as mandated by guidance that are topoi (places of argument) for different and often integrated attributes of clinical research drawn from this corpus. So clearly the clinical study report is not a stand-alone document in the instance of interpretation and argumentation.

Another point, a clinical study report incorporated into a regulatory submission will have numerous appendices including the final version research protocol and the statistical analysis plan. So I struggle to understand the premise that says you must incorporate extensive elements of the protocol and stats plan into the body of the clinical study report in order to assure the report is a stand-alone document. I truly fail to see the merit of this premise.

My last point is why do so many feel it is necessary to write the Introduction Sections to clinical study reports in the classic inductive form of rhetoric that moves from describing the disease condition, to the therapeutic void, to the chemical or biological construct of the drug under study and how this drug’s pharmacological action addresses a therapeutic void. The standard response is, “well we want to make this a stand-alone document.” Why? All the rhetorical moves you are trying to make in the Introduction will happen for sure elsewhere in the drug submission package in the appropriate topos. So why waste your time here, at the low level of the clinical study report? In my way of thinking the study report Introduction is intended to convey only two points:

1. Why are you doing the study (the answers to what questions do you want at the end of the study that you do not have now?)
2. What previous work has informed the design of this study?

Designing Regulatory Submission Documents for Decision Making

I have made blog posts from time to time talking about users of various document genre represented in communication about life science research. I believe that for many, my portrayal of how people engage with documents presents a different picture than what they hold in their imagination. At least I hope it is the case that they are designing documents to a meaningful mental model, as that helps to explain why so many documents I examine are rather ineffectual communication vehicles. Unfortunately I believe most documents are generated according to the model of precedence--"what did we do last time?" I think this is generally the only question consistently asked in the life sciences regarding document design.

In this post I want to talk about designing documents to satisfy readers who must make decisions, like a regulatory health agency reviewer. These reviewers must decide whether a drug, biologic, or device can be marketed as desired by the sponsoring company.

There is nice body of work in the cognitive sciences that describes the mind set and reading style of the selective professional reader who is reading documents in order to make a decision or a set of decisions. It is clear from this research that these readers are not empty cups waiting to be filled up with whatever you want to send their way in your documents. Not only is the professional reader very selective in what they will read, they are also very critical. That is, they read against the information you have submitted in your documents looking for insufficiencies and weaknesses.

Remember that the selective professional reader working in the the health regulatory agencies is a very sophisticated reader and in many instances they know what type of information they need right after they review the requested marketing claims for the drug or the medical device.

Here is the part that many people miss in creating their documents. No matter how logical and detailed your information may be, if you have not correctly anticipated what the selective professional reader needs in terms of information and the necessary level of detail, then you have failed. It is that simple.

Research shows clearly that decision-making is a function of schema-based cognitive processes deployed by the selective professional reader. The process is really well understood now and suggests that all documents  to be used in a decision-making environment should be designed to support a schema-based reading style versus the approach of reporting information or summarizing findings.

Schema activation is at the heart of decision making. Schema provide the interpretive framework for a reader to pass judgement on data and written discourse. In the process of reviewing drug and device submissions, the selective professional regulatory reader will make use of many, many schemata during the course of the decision-making process. Each schema they deploy will have multiple slots for information. They will want to fill every slot and they search documents and data bases looking for the details associated with the various slots in a particular schema. For example, a specific schema can be characterized by this question: "will the indicated drug dose lead to an undesirable toxicity profile in the elder, under weight, and hepatically compromised patient?" They will then examine the documents presented to find all the information they desire in making the decision whether the drug label language regarding dosing is acceptable or needs to be modified. And they read only looking for what they believe are the necessary details. In essence, they read at the virtual exclusion of anything within the document(s).

This is why we talk about  the selective professional reader scanning documents looking for very specific information and now using some sophisticated search tools to navigate through documents looking for specific pieces of information and data.

Much of what the selective professional reader at regulatory agencies do is further characterized as an attribute-based search. The attributes are often compared against known standards, but may also be subjective standards. The attributes they are considering are largely tied into the product label claims the sponsor wants to make.

So the net-net here is that life science research professionals need to reconsider what they believe to be acceptable document design parameters. It is essential to build out effective working models for how the selective professional reader using your documents makes decisions and what schemata are being deployed in this process and then design documents that support these specific reading methods.

Presentation: Why Most Document Reviews are Not Really Reviews

View Greg's presentation slide set from the recent DIA Clinical Forum:

The Good, The Bad, and The Ugly of Review Practices in The Clinical Sciences

View more presentations from McCulley/Cuppan LLC

Notes on Managing Reviews Presentation and Discussions at DIA Clinical Forum

We had an interesting set of presentations and discussions yesterday at the DIA (Drug Information Association) Clinical Forum in Basel, Switzerland related to managing reviews. Here are my summary notes on the session.

I spoke along with Barry Drees from Trilogy Medical Writing and Rhys Stoneman from GlaxoSmithKline. We each covered aspects of improving review outcomes for clinical research documentation.

I addressed the topic that to affect change you must look at the root causes of poor review outcomes not merely the symptoms of poor outcomes. Barry and Rhys talked about how their organizations--a contract writing house  and a large multinational pharmaceutical company, respectively--look to improve review outcomes.

I talked about the underlying root cause of most poor review outcomes fit into two boxes

1. No formal training in review methods
2. Poor reviewer discipline

Note: Jessica will get my presentation slide set posted to the M/C web site.

I stressed that without training most people are left to inspect and edit documents and generally are not sensitive to how rhetorical situations vary with audience and document genre.  I talked about reviewer discipline is difficult to modify without  making process changes and applying performance mandates.

Barry  talked about methods he uses to influence behavior of review teams. He had a great line on one of his slides addressing lengthy review cycles: "We do not have enough time or money to do it right, but there is always time and money to do it over."

Rhys talked about the steps taken to transform work practices and belief systems related to the authorship and review of clinical documents. We did a lot of support and training work at GSK on this project, so I have seen firsthand the changes. It is fair to use the descriptor, dramatic, to characterize how they have reformed work practices and outcomes.

Not surprisingly, the topic of greatest interest to the audience was issues related to reviewer discipline and what can be done to modify or blunt the impact of bad behavior.