I have been away from the blog for most of the summer. It has not been an intended absence, rather I have been caught in a whole lot of work and personal obligations that kept me away from making posts here.
We have been doing several projects this summer related to the planning, writing, and reviewing of clinical research protocols. In this post I want to share some thoughts about review.
We believe that protocol reviewers need a more a refined
mental model for how to review drafts of their documents—how to concentrate on
the big-picture rhetorical concerns and set aside attending to edits and
stylistic corrections during the early rounds of review and then attend
exclusively to these elements in the final rounds of review.
In work with several clients, we find that reviewers confuse the roles of reviewers and editors and reviewers also confuse the timing of when edits should be made. By this I mean we see many reviewers operating at the sentence and word level in the initial review of the protocol concept document. The protocol concept document is where you have to get all the big picture details ironed out regarding study design and conduct before you sweat the details at the paragraph and sentence level. Based on our observations and interviews, we suggest that better guidance is needed to define the roles for reviewers and editors and time points for application.
We recommend that performance criteria and subsequent
measurement should be enacted to help review teams better understand and
appreciate the need to attend to the intellectual attributes of a planned research trial. When I ask people sitting in my workshops how many protocol amendments they average per study the response is always very similar: a collective groan followed by the retort, "we average way too many." So it seems like it is time to change methods. It is at this point in the conversation with clients that I invoke the great Albert Einstein quote regarding the working definition for insanity: to continue to deploy the same methods, but then expect different outcomes.
Our assessment of work practices at a number of pharmaceutical companies suggests that reviewer discipline is poor in that reviewers fail to
participate at all or in meaningful ways in early draft reviews and also extend reviews of certain protocol
sections without really improving communication quality. When a protocol goes through 6 rounds of review, then something is likely not right and when the Background section in the protocol is actively reviewed in all 6 rounds, then for sure something is not right.
We recommend that late in the protocol development process guidance regarding the line-level review of text should be enacted to ensure precision and consistency. We find huge inconsistencies and ambiguities with three heavily used modal
verbs: may, should, and must. We also find that many protocols do a poor job of characterizing aspects of agency for instance, who has responsibility for decisions or ensurance of patient data integrity. We also find that reviewers rarely consider the temporal frame of reference for information in a protocol. We find huge inconsistencies where in a given paragraph the language may start in present tense, move to future tense, and then back to the present tense.
