When I first started working at McCulley/Cuppan, Greg gave me the task of reviewing a clinical study report, not only as a way to gauge my reviewing skills, but as a starting place for lessons on judging document quality using the McCulley/Cuppan Document Standards. The first instruction he gave me was "Read before you edit."
Read before you edit.
Sounds simple, but when asked to review a document, most people start off by looking for mistakes. I remember sitting there, trying to pay attention to the document when all I could see was a misspelled word. All too often we see ourselves as that teacher of grade school with the red pen in hand, correcting verb tense and punctuation errors. We don't see the document at the meta-level so we miss the message and logic of the document.
Think about how you read a book, an article, or this blog. You read for meaning. If I leave a TyPo or too, you still understand that this post is about editing and reviewing documents. Now consider your habits when someone asks you to look at a document. Even if they say they want you to make sure that a paragraph makes logical sense, your eye probably catches all the typos and then proceeds to analyze the logic.
The truth is that inspection is easier than reviewing and is a quick way to show your contribution to a document. A way to say Look, see all these good edits I made? The reality is that lots of time is wasted, especially on first or second drafts, fixing nit-picky mistakes in grammar and punctuation on sentences that will be cut out of the document before the third draft. If you read a document (or section of a document) once all the way through, you will avoid re-reading sentences again and again just to get the meaning and avoid the need to revert your changes back to the original sentence.
During the last few months, I've been assessing a number of documents to judge reviewer performance as part of my work at McCulley/Cuppan. What I see all too often is that a reviewer will correct a sentence, changing a word or two, only to go back and change the sentence again within that same review session or in a subsequent draft. It is apparent many reviewers read to fix grammatical and stylistic errors. They are reading and analyzing each word individually versus parsing the sentence or paragraph for meaning. So a given word might seem to be the wrong choice at first glance, but if the reviewer were to read the entire paragraph, then the author's chosen word may be seen as appropriate. Style is a much overrated, overwrought topic during reviews. This is especially important when you read documents written by multiple people, where writing style and word usage is likely to vary widely.
But whether you are inspecting (editing for punctuation and grammar) or reviewing (assessing for purpose, logic, and content), reading a document in its entirety before making revisions places you closer to how the reader is going to engage with the document. In the assessments I have done, I have seen so many reviewers miss the forest because they were busily engaged inspecting branches on trees.
22 February 2011
15 February 2011
Editing When You Should Be Reviewing Costs Serious Money
The idealized model of document project management describes an iterative process of information planning, content specification, authorship, review, and publishing. Working to develop best-of-craft work practices aligned with this model can help organizations meet their goals efficiently and at lower costs.
Most organizations in the life sciences do not carefully consider their true costs to produce a document. When all the hidden costs associated with review are added in, the cost-per-page to produce a final version document becomes significant. If you add in opportunity costs for staff time dedicated to additional rounds of review (even if a document meets a specified deadline for the final version) then the cost-per-page skyrockets. Many organizations choose to ignore this point, which is why in my presentations on strategic review I have a slide with the Great Pyramids of Giza and the caption: “We can do anything we want as long as we have a 24/7 schedule and an expendable supply of labor.”
The goal for review is to improve document communication quality—not grammatical accuracy or data integrity, which is done via inspection. To be most effective and efficient, reviews need to be strategic and orchestrated. This means the primary focus of review by subject matter experts should be on the intended messages and arguments and testing the document to ensure it minimizes the prospects for a qualified reader to construct alternative meanings. Tasks not accomplished if you are editing a document for grammar and style.
Our systematic analysis of review practices at numerous companies strongly suggests that while significant time and energy is expended on document review, the collective effort is generally over-sized in relation to the improvements made in the communication quality of the documents. That is, there are more people and more rounds of review than should be needed to get to a final version document. Another way to look at it is that reviews do not really move the communication quality meter very far given how much time and resources are thrown at the task.
Our analysis suggests that review performance is hampered by poorly defined expectations for what the team wants to accomplish in a document, that reviewers do not apply systematic means of analysis to their review (meaning team reviews are largely ad-hoc feeding frenzies), and that reviewers stray from attending to the messages and arguments of a document, instead attending to matters truly related to publishing standards. In other words, reviewers stop being reviewers and become editors. Editing when you should be reviewing comes with serious costs.
04 February 2011
Build Documents Following Logic of What the Reader is Trying to "Do"
I ended my previous post stating that documentation teams have to recognize their documents must be built following the underlying logic of what the reader is trying to "do" with the document. By this I mean there are different design choices for document content depending on what the reader is intending to accomplish through accessing the document.
If your reader is making decisions versus merely being informed about a body of work, then this greatly impacts document content and a major difference between regulatory medical writing and the medical writing associated with the development of manuscripts.
For instance, if you are creating a summary-level regulatory submission document, the logic of what this document is to "do" for the user will dictate design decisions. A summary-level document is not intended to report results; it is intended to synthesize study findings in a way so as to build the information base for the therapeutic activity and safety of a drug or device in various patient populations. The summary documents are meant to address issues associated with development work and candidly represent limits of current understanding. It is in the summary documents that one must argue that observed differences in the data (or lack of differences) are determined reliably and provide a low probability of alternative results upon further study. In creating summary documents authoring teams must recognize that successful arguments need to establish warranted claims that respond to the known or theorized questions or concerns of the reader.
A second example, if you are writing a risk management plan, like a RiskMAP for FDA, then the document must be designed to clearly demonstrate the logic underlying the risk assessment and the quantitative and qualitative assignment of risk. The document must clearly lay out the strategy to monitor and characterize the nature, frequency, and severity of the risks associated with the use of a product. Ultimately the document has to characterize why the proposed plan and the applied tools are the most appropriate to minimize a health outcome risk potential on a go forward basis.
My final example, if you are writing a clinical research protocol, then you must recognize the logic of this document is not only to demonstrate study design and when assessments will be made in or out of the clinic. It is essential the document also be seen as a vehicle to represent potential benefit in relation to risk (what an institutional review board is looking to assess when they consider the merits of a research study.) Also a protocol must clearly establish the agents and actors engaged in the conduct of the research trial and the assignment of decision-making to the actors (all users of the document need to know this information.) In my review of protocols, I routinely find these documents generally obscure actors (who will do what) to varying degrees of opacity and often leave assignment of decision-making as implied versus explicit. In such instances, the document is clearly not written reflecting the underlying logic of what the document is intended to “do” in the hands of the users of the document.
If your reader is making decisions versus merely being informed about a body of work, then this greatly impacts document content and a major difference between regulatory medical writing and the medical writing associated with the development of manuscripts.
For instance, if you are creating a summary-level regulatory submission document, the logic of what this document is to "do" for the user will dictate design decisions. A summary-level document is not intended to report results; it is intended to synthesize study findings in a way so as to build the information base for the therapeutic activity and safety of a drug or device in various patient populations. The summary documents are meant to address issues associated with development work and candidly represent limits of current understanding. It is in the summary documents that one must argue that observed differences in the data (or lack of differences) are determined reliably and provide a low probability of alternative results upon further study. In creating summary documents authoring teams must recognize that successful arguments need to establish warranted claims that respond to the known or theorized questions or concerns of the reader.
A second example, if you are writing a risk management plan, like a RiskMAP for FDA, then the document must be designed to clearly demonstrate the logic underlying the risk assessment and the quantitative and qualitative assignment of risk. The document must clearly lay out the strategy to monitor and characterize the nature, frequency, and severity of the risks associated with the use of a product. Ultimately the document has to characterize why the proposed plan and the applied tools are the most appropriate to minimize a health outcome risk potential on a go forward basis.
My final example, if you are writing a clinical research protocol, then you must recognize the logic of this document is not only to demonstrate study design and when assessments will be made in or out of the clinic. It is essential the document also be seen as a vehicle to represent potential benefit in relation to risk (what an institutional review board is looking to assess when they consider the merits of a research study.) Also a protocol must clearly establish the agents and actors engaged in the conduct of the research trial and the assignment of decision-making to the actors (all users of the document need to know this information.) In my review of protocols, I routinely find these documents generally obscure actors (who will do what) to varying degrees of opacity and often leave assignment of decision-making as implied versus explicit. In such instances, the document is clearly not written reflecting the underlying logic of what the document is intended to “do” in the hands of the users of the document.
02 February 2011
Regulatory Writing Must Be "Fit for Function" Not Perfect
Documents for the medical regulatory audience do not have to be perfect. Rather, they have to be "fit for the intended function."
I suggest to you that the majority of people associated with drug and device development do not either understand or appreciate this concept. I also suggest to you that the majority do not understand that you apply different document designs to different document genres. Which in turn means you have different standards for different genre. Fit for function means you do not work to one standard.
Fit for function means a document is built and judged by standards that are determined by audience and the task or role the document will take when in the hands or on the computer screen of the specified audience.
A few weeks back I was working with a development team that judges all of their work by one standard. No matter what they are writing, the team judges their handiwork by their personal standards for a research manuscript. This one standard is applied to clinical study reports, protocols, and investigator brochures. I know this to be the case because review comments are often predicated with statements like: "Well, when I was writing manuscripts we always did.............in the Discussion Section. We need to do that in this report." or "I think the reviewers at the agency would find this very interesting to read. So let's expand the discussion here." and "You have a hyphen break at the end of the line in this paragraph. We cannot have mistakes like this in our documents."
Fit for function places focus squarely on the elements that matter, that is, the underlying logic of what the document is to convey or enable the reader to do. So in the world of drug and device development, this means authoring teams have to move away from the notions that when creating documents for regulatory readers, they are either simply reporting data or writing for an audience interested in knowledge acquisition. Working to either of those standards guarantees that your documents will not be fit for function. Rather, documents must be built following the underlying logic of what the reader is trying to do with the document.
More on how we characterize the logic of document genres in my next post.
I suggest to you that the majority of people associated with drug and device development do not either understand or appreciate this concept. I also suggest to you that the majority do not understand that you apply different document designs to different document genres. Which in turn means you have different standards for different genre. Fit for function means you do not work to one standard.
Fit for function means a document is built and judged by standards that are determined by audience and the task or role the document will take when in the hands or on the computer screen of the specified audience.
A few weeks back I was working with a development team that judges all of their work by one standard. No matter what they are writing, the team judges their handiwork by their personal standards for a research manuscript. This one standard is applied to clinical study reports, protocols, and investigator brochures. I know this to be the case because review comments are often predicated with statements like: "Well, when I was writing manuscripts we always did.............in the Discussion Section. We need to do that in this report." or "I think the reviewers at the agency would find this very interesting to read. So let's expand the discussion here." and "You have a hyphen break at the end of the line in this paragraph. We cannot have mistakes like this in our documents."
Fit for function places focus squarely on the elements that matter, that is, the underlying logic of what the document is to convey or enable the reader to do. So in the world of drug and device development, this means authoring teams have to move away from the notions that when creating documents for regulatory readers, they are either simply reporting data or writing for an audience interested in knowledge acquisition. Working to either of those standards guarantees that your documents will not be fit for function. Rather, documents must be built following the underlying logic of what the reader is trying to do with the document.
More on how we characterize the logic of document genres in my next post.
12 January 2011
Something to consider: using intranets as Knowledge Management tool to support collaborative pre-writing planning
Jakob Nielsen is a name some of you may recognize. He is a self-appointed guru on all matters regarding usability of web sites and web-mediated work tools. You can find much of his work summarized on his web site titled Useit.
Last week Nielsen came out with his list of Top 10 designed business Intranets for 2011. Here's the link.
I noted with interest Nielsen's comment "If there's anything that has been overused, abused, and hyped beyond the level of cliché, it's "knowledge management." Thus, it might be better to say that many of this year's winners were strong in "managing knowledge" on their intranets."
Managing knowledge. This is a concept where I see drug and medical device teams struggle all the time. One reason for the struggle is the slow uptake of tools that can help foster an effective work environment and change deeply ingrained cultural practices.
I believe intranets, such as WIKIs are great tools that should be deployed at the project team level. These platforms become incredibly valuable from the moment a clinical development plan is written until well after a dossier is submitted to the regulatory authorities.
While it is true that knowledge management is not a technology issue, effort must still be spent in providing a suitable environment to facilitate knowledge capture and sharing.
I am suggesting the use of team specific intranets as a way to promote cultural change in an organization, at the level of knowledge-sharing activities and also for shaping broader work behaviors. In most companies where I train or consult, little has really changed since the 1960s in how people approach the planning and authoring of documents. Using intranets, like a WIKI, can quickly get a team applying best practices in terms of pre-writing planning.
Last week Nielsen came out with his list of Top 10 designed business Intranets for 2011. Here's the link.
I noted with interest Nielsen's comment "If there's anything that has been overused, abused, and hyped beyond the level of cliché, it's "knowledge management." Thus, it might be better to say that many of this year's winners were strong in "managing knowledge" on their intranets."
Managing knowledge. This is a concept where I see drug and medical device teams struggle all the time. One reason for the struggle is the slow uptake of tools that can help foster an effective work environment and change deeply ingrained cultural practices.
I believe intranets, such as WIKIs are great tools that should be deployed at the project team level. These platforms become incredibly valuable from the moment a clinical development plan is written until well after a dossier is submitted to the regulatory authorities.
While it is true that knowledge management is not a technology issue, effort must still be spent in providing a suitable environment to facilitate knowledge capture and sharing.
I am suggesting the use of team specific intranets as a way to promote cultural change in an organization, at the level of knowledge-sharing activities and also for shaping broader work behaviors. In most companies where I train or consult, little has really changed since the 1960s in how people approach the planning and authoring of documents. Using intranets, like a WIKI, can quickly get a team applying best practices in terms of pre-writing planning.
09 January 2011
Minimal Time and Effort Should be Applied to the Creation of the Clinical Study Report Synopsis
How much time and effort to apply to the creation of the clinical study report synopsis?
This is another question I am asked on a regular basis and a line of discussion that repeatedly comes up when I am working with clients to help streamline work practices. I usually draw slack jaws accompanied by an incredulous stare as I give my answer:
"The amount of time should be minimal, involve no more than three people, the level of effort better be next to nothing, and the time should be no more than an hour to create and a whole lot less time to review."
My reasoning is very simple and straightforward.........."You apply time and effort to the development of a product in relationship to the product's strategic value. The value of the CSR Synopsis to the regulatory reader is virtually zero."
Think about it. The CSR Synopsis affords little utility to what the reviewers are looking to accomplish when they choose to enter the framework (that is, the document) of an individual clinical study. If a regulatory reviewer wants a “snapshot” of a study, they will likely take a contextualized snapshot at a higher level of a drug submission dossier. That is, the documents in Module 2. They do not enter the framework of the study report to get generalized or summarized information. They are at the Module 5 level and embedding themselves in a clinical study report because they are seeking answers to narrowly defined questions.
At McCulley/Cuppan we have queried regulatory reviewers about how they "use" a study report synopsis. Their responses support the premise I have laid out for you in the above paragraph.
So this gets us back to the question on time and effort. Why generate a study report synopsis with every draft? Why allow the full team to look at the document? Talk about wasting time and energy.
In our assessment of review practices at pharmaceutical and medical device companies we see the same review pattern played out time and time again. The study report synopsis is generated with the first draft and in the review process it consistently garners the attention of the full review team (many of whom never make it all the way through the results sections during the course of their reviews.) Same thing happens on each subsequent draft.
Given the synopsis is but a summary of the body of work presented in the study report, it should not be generated until that body of work is completed and signed off as "good to go." The synopsis does not even warrant review. It should be critiqued. Critique is a comparative read. A reading to ensure that the synopsis accurately and appropriately portrays the sum total of key details of the study. The critique process requires at best two people and certainly no more than three. All being subject matter experts drawn from the key clinical disciplines represented in the research study.
04 January 2011
Importance of language and writing style in a clinical study report
How important is language and writing style in a clinical study report? I was recently asked this question by a medical writer working for one of my McCulley/Cuppan clients. The writer is dealing with a team that seems to obsess over every word in every draft and the writer is looking for some help in how to address the situation.
Here is my response to the question:
You are asking about lexical and syntactical elements of writing (the third element of writing is grammatical.)
Lexical pertains to the words (vocabulary) of a language. In the context of clinical research we need to talk about several applied lexicons of scientific phraseology that apply broadly to science and then narrowly to a specific therapeutic area. The admittedly most distinctive feature of any clinical study report is the application of specific scientific and technical prose. So, language is very important in a CSR to avoid lexical ambiguity (why I so love statisticians and their demands for careful use of language when describing statistical observations) in order to allow the reader to derive the intended meaning.
My experience suggests that many people in Pharma think attention to syntactical elements (style) means they are either eliminating ambiguity or improving clarity of message. Rarely is this the case.
You have heard me say before that style does not matter in the type of writing represented in clinical study reports submitted to regulatory authorities in the US and elsewhere.
My position is supported by current discourse theory. Discourse theory states that, as a rule in scientific writing, meaning is largely derived from the precise use of key scientific words, not how these words are strung together. It is the key words that create the meta-level knowledge of the report. Varying style does little to aid or impede comprehension.
What happens is people often chase and play around with the style of document. Largely they are looking to manipulate an advanced set of discourse markers specific for clinical science writing or some subset specific to a therapeutic discipline. Discourse markers are the word elements that string together the key scientific words and help signal transitions within and across sentences. These discourse markers are the elements that provide for style. There are macro markers (those indicating overall organization) and micro markers (functioning as fillers, indicating links between sentences, etc.) Comprehension studies show that manipulating discourse markers--that is, messing with style--in most instances does not influence reader comprehension. It is worth noting that manipulation of macro markers appears to have some impact on comprehension for non-native speakers of English (why it is worth using textual advanced organizers to help with document readability.)
So the net-net is: there is little fruit to be picked from messing with style in a clinical study report. Put review focus on the use and placement of key terms.
This is a bit of a non-sequitur to the question, but a concept I’d like to share. To derive meaning from scientific text, readers will rely on their prior knowledge, and cues provided by the key terms and data they encounter or fail to find in a sentence, paragraph, table, or section of a clinical study report. So what I’d really prefer to get people thinking about is the semantical elements of their documents. Semantics is fundamentally about encoding knowledge and how you as an author enable the reader to process your representation of knowledge in a meaningful way. Semantics is about how much interpretive space you provide to the reader in a document by what you say and equally important, by what you do not say. Of course you cannot get to the point of thinking about semantics unless you see clinical study reports as something more than just a warehouse for data.
Here is my response to the question:
You are asking about lexical and syntactical elements of writing (the third element of writing is grammatical.)
Lexical pertains to the words (vocabulary) of a language. In the context of clinical research we need to talk about several applied lexicons of scientific phraseology that apply broadly to science and then narrowly to a specific therapeutic area. The admittedly most distinctive feature of any clinical study report is the application of specific scientific and technical prose. So, language is very important in a CSR to avoid lexical ambiguity (why I so love statisticians and their demands for careful use of language when describing statistical observations) in order to allow the reader to derive the intended meaning.
My experience suggests that many people in Pharma think attention to syntactical elements (style) means they are either eliminating ambiguity or improving clarity of message. Rarely is this the case.
You have heard me say before that style does not matter in the type of writing represented in clinical study reports submitted to regulatory authorities in the US and elsewhere.
My position is supported by current discourse theory. Discourse theory states that, as a rule in scientific writing, meaning is largely derived from the precise use of key scientific words, not how these words are strung together. It is the key words that create the meta-level knowledge of the report. Varying style does little to aid or impede comprehension.
What happens is people often chase and play around with the style of document. Largely they are looking to manipulate an advanced set of discourse markers specific for clinical science writing or some subset specific to a therapeutic discipline. Discourse markers are the word elements that string together the key scientific words and help signal transitions within and across sentences. These discourse markers are the elements that provide for style. There are macro markers (those indicating overall organization) and micro markers (functioning as fillers, indicating links between sentences, etc.) Comprehension studies show that manipulating discourse markers--that is, messing with style--in most instances does not influence reader comprehension. It is worth noting that manipulation of macro markers appears to have some impact on comprehension for non-native speakers of English (why it is worth using textual advanced organizers to help with document readability.)
So the net-net is: there is little fruit to be picked from messing with style in a clinical study report. Put review focus on the use and placement of key terms.
This is a bit of a non-sequitur to the question, but a concept I’d like to share. To derive meaning from scientific text, readers will rely on their prior knowledge, and cues provided by the key terms and data they encounter or fail to find in a sentence, paragraph, table, or section of a clinical study report. So what I’d really prefer to get people thinking about is the semantical elements of their documents. Semantics is fundamentally about encoding knowledge and how you as an author enable the reader to process your representation of knowledge in a meaningful way. Semantics is about how much interpretive space you provide to the reader in a document by what you say and equally important, by what you do not say. Of course you cannot get to the point of thinking about semantics unless you see clinical study reports as something more than just a warehouse for data.
Subscribe to:
Posts (Atom)