Peer Review Revisited: It Really Needs to Change

Back in June of 2010 I made a posting here on our blog that considered peer review to be a largely bankrupt way to screen and validate research worthy of publication. This post extends the discussion I started in that post.


An important point to consider is that an increasing number of studies and journal editors are suggesting that peer review is a failure as currently practiced in the life sciences.


As food for thought consider the following:

1. This is old data, but worthy of attention. In a survey, only 8% of the members of Sigma Xi, the Scientific Research Society, agreed that peer review works well as currently applied. (Chubin and Hackett, 1990).
2. As a tool to filter out science worthy of publishing, peer review may be blocking the flow of innovation. (Horrobin, 2001) Horrobin spoke strongly when he suggested the peer review system in a non-validated charade. Here's a link to his article  http://www.nature.com/nbt/journal/v19/n12/full/nbt1201-1099.html
3. Richard Smith's article: "Classical peer review: an empty gun" cites the Drummond Rennie quote: "If peer review was a drug it would never be allowed onto the market because we have no convincing evidence of its benefits but a lot of evidence of its flaws." (Rennie is a deputy editor of the Journal of the American Medical Association and a supporting force behind the international congresses of peer review.)
4. In late April 2012 Carl Zimmer reported in the New York Times that according to a study made by PubMed that the number of articles retracted from scientific journals increased from 3 in 2000 to 180 in 2009. Here is link to his piece: "Sharp Rise in Retractions Prompts Calls for Reform"
5. A couple more articles worth checking out
1. "What's wrong with peer review"
2. "Peer review is f***ed up -- let's fix it"

Article Published: Missed Opportunities in the Review and Revision of Clinical Study Reports

It has been some time since the last blog post. I am not taking time off from posts. Rather, my workload and travel schedule have created a hectic environment where I have found it difficult to carve out time to create and upload useful posts.

This post is an unabashed personal plug for an article I coauthored with my colleague Stephen Bernhardt. The paper: Missed Opportunities in theReview and Revision of Clinical Study Reports appears in the April issue of the Journal of Business and Technical Communication. 

In the paper we look to further the understanding of review practices as applied to large and complex technical documentation. Specifically, the paper describes several case studies where we examined closely the review efforts of teams in the process of finalizing clinical study reports.  Our title suggests we see considerable room for improvement of review practices.

We see our work as an extension of other studies of review practices in professional settings. 

• Paradis, Dobrin, and Miller (1985) first established document review as a critical site of contentious interaction in a research and development environment (Exxon). These authors contrasted document review from the opposing perspectives of managers and employees, detailing the ways that individuals often worked without shared purpose or expectations, resulting in frustrating differences of perception, feelings of resentment, and the need to substantially rework documents during successive reviews. 
• Van der Geest and van Gemert (1997) pinpointed a general sense of frustration with review processes in several Dutch companies, noting that “both writers and reviewers find reviewing the most cumbersome stage in the process of text production, given that many parties are involved in it and that it is loaded with different expectations” (p. 445).
• Henry (2000), working with data gathered by many interns at various professional sites, found that reviews were “fraught with second guessing” and required “interpretations of organizational culture to the ends of adequately and appropriately delivering discursive products” (p. 65).
• And several studies (Henry, 2000; Katz, 1998; Paradis et al., 1985) discuss the activity of “document cycling,” an activity in which documents pass through multiple and sometimes conflicting reviews, as various reviewers weigh in with commentary.

A theme throughout the literature is that document review frequently brings into play conflicting or competing purposes. In addition to improving a document, review sometimes functions to evaluate worker performance (Couture & Rymer, 1991) or to discipline individuals (Henry, 2000, p. 81). Thus, interpersonal dynamics are frequently in play. Katz (1998), in particular, highlighted how review processes can function positively as a way to socialize new workers, helping them learn how to both write and work successfully within the local culture. Henry (2000) shared this concern for how interns come to understand the ways that organizations perform work.

The position we take in this paper is that changing nonproductive, conditioned, inefficient practices is NOT an easy matter, or companies would have already done so. We suggest that recognizing nonproductive review practices and understanding the causes for such practices should be an object of focus for more organizations. We understand that collaborating to develop complex documents with sound arguments involves difficult cognitive and social practices. But if a company establishes the goal of producing quality documentation through efficient and effective review practices, it will find that it must do a lot of work to counter the ingrained tendencies of review teams to focus on low-level stylistic edits as opposed to high-level rhetorical concerns.

Inefficient Meeting Practices Cost Money

As I read the meeting manifesto by Al Pittampalli, Read This Before Our Next Meeting (now available to read for free from Amazon), I noticed many similarities between what Pittampalli writes and what Greg has been advising clients for years. Mainly, that there is a high cost to inefficient and ineffective work practices.

Anyone working in corporate America will be able to relate to the book, and, hopefully, learn from it. For us at McCulley/Cuppan, the focus is, as always, on how those in biomedical R&D can improve work practices.

Below are a few highlights from the book that relate to what we consider best practices (bolded text is quoted from the book).


The Modern Meeting moves fast and ends on schedule.
"Traditional meetings seem to go on forever, with no end in sight." How often have you felt this way? And how often do these meetings accomplish the intended purpose of the meeting, if there was a clearly stated purpose?

What we've seen over the years is that review meetings, those scheduled to discuss a document in person, last for hours, going from page 1 to page n through a document, with hours wasted on word choice and often leaving the author feeling overwhelmed. When the time limit is up on this type of meeting, more meetings are scheduled that will follow the same pattern.

The Modern Meeting limits the number of attendees.
More often than not, executives are involved in these document review meetings, even on meetings focused on editing and not strategic review, along with anyone who ever had anything to do with the project. How many man-hours are wasted sitting in a meeting instead of working on discovery or development? As Greg mentioned in his post "Editing When You Should be Reviewing Costs Serious Money", "when all the hidden costs associated with review are added in, the cost-per-page to produce a final version document becomes significant."

Meetings should be scheduled for a set amount of time, a time limit that is short enough to prevent repetitive attacks on one word or phrase (nitpicking), but long enough to actually allow for the goal of the meeting to be accomplished. Only the people who are directly responsible for a decision or must act on that decision should be in the meeting. Just because a person provided a line of text to the document does not mean that they should attend the meeting. If a person feels they must be included, provide them with a meeting outline and/or a meeting recap.

The Modern Meeting rejects the unprepared. 
Often meetings are scheduled to review a draft of a document, but the meeting ends with only a few pages of that 100 page document having been marked up. Then more meetings are scheduled and the cycle continues.

Meetings should have a clearly defined purposed. If a meeting has a clearly declared purpose, the leader of the meeting should be able to provide a list of items to accomplish for the meeting and the time allotted for each action item. This helps the scheduler of the meeting weed out people whose presence is unnecessary. This also ensures everyone is prepared for the meeting. According to Pittampalli, if you aren't prepared, you shouldn't attend.

The Modern Meeting produces committed action plans.
At the end of a well-organized meeting, there should be a committed action plan, not just a deadline. Too often with our clients we've seen meetings that produce reams of notes for the authors of the document with a deadline that keeps moving as more notes are piled into the author's inbox.

If meetings, and work practices in general, are efficient and focused only on the contributions of those directly involved, there is less opportunity for circular arguments and nitpicking and less contradictory comments for the author to wade through. Plus there is more time to actually work.


For more tips on improving meetings, follow Al Pittampalli on Twitter at @Pittampalli or view his blog.

More on What is a Document?

So what is a document?

In response to my last blog post, I have been asked by several individuals—"so then what is a document?"

My short answer—"I do not know for sure."

Now for the long answer.

The widely accepted definition for a document is as a textual record. This definition served us well in the past. But now with digital records, semiotics, and information retrieval tools; I am not sure the definition meets the needs of how we communicate in 2012.

As early as the 1930s Paul Otlet, an Information Scientist of considerable renown, suggested that the definition of documents also include digital images and even three dimensional objects. I am not prepared to toss all the elements Otlet describes into the mix. But I am prepared to suggest that documents are organized physical evidence and as such the organization transcends the classic definition for a document as this vehicle is a less relevant communication medium in 2012 than it was is 1982. I do not have a preferred term, I wish I did, but I do suggest we attempt to move away from the term document as it suggests a domain for organized physical evidence that does not match the reality of the digital age.

Suzanne Briet suggested a definition some time ago that a document is evidence in support of a fact. I rather like this notion. She makes the point that documents should not be viewed as being concerned with texts, but with access to the evidence. I suggest this is the essence of all regulatory writing that I talk about often in this Blog. If one considers the models in place for electronic drug submissions, thinking in the classic terms of 8.5 x 11 and A4 is really not very useful.

Rather it is better to be thinking in terms of taxonomies of information or perhaps even semiotics. Semiotics is the study of signs, indication, designation, signification, and communication. Semiotics is closely related to the field of linguistics. I look at semiotics as a valid attribute for this discussion because the life sciences are driven by numbers and what are numbers, but signs and the significance of those signs.

Then there is Michael Buckland who talks about how a key characteristic of “information-as-knowledge” is that it is intangible: one cannot touch it or measure it in any direct way. Knowledge, belief, and opinion are personal, subjective, and conceptual. Therefore, to communicate them, they have to be expressed, described, or represented in some physical way, as a signal or communication.

What we are really talking about happening in regulatory submission packages is the conveyance of knowledge.  This conveyance often transcends the boundaries of a traditional text, that is, a document as it is generally defined. The Briet notion of "evidence in support of a fact" works well as a definition of a document especially if we change the quote to read "evidence is support of a claim."

Need a New Mental Model for Regulatory Documents

Wow……I have been away from this blog a whole lot longer than intended. Those competing interests….like you all understand….are the bane of my existence.

For the past couple months I have been looking closely at how people think about the “vehicles” used to communicate with regulatory health agencies. I am using the word vehicle here because I am trying to divorce myself from the notion of document, in particular, the notion of “a document.” In the modern times of on-screen reading and linked files, what is a document anyhow? To me it is the entire corpus somebody may be able to access, not just one slice of that body.

In my consulting/training interactions at McCulley/Cuppan, I find that the majority of people I interface with in the client setting operate within the mind set of individual documents (some have even smaller boundaries and operate by document sections) that are stand alone with well defined boundaries (pages and page counts.)

I want to argue that the vehicle of communication for regulatory submissions is not a document. It is the full and complete dossier submitted by the sponsor. Documents are just placeholders where I go to get a piece or pieces of information that help answer my questions. I want to argue that the regulatory reader does not see a dossier as a set of documents. Rather they see a dossier as a corpus of information that they will use to answer question and make decisions. The contents are just vehicles they peruse to get what they want.

Applying my working model means you stop seeing documents as “stand alone” and stop saying “this document has to tell a story.” I’d also like you to stop using the word document. That word has baggage I am trying to jettison. Instead I want people to view their work at least as “modules” and preferably as vehicles that help a user to answer very specific questions. Bottom line, a research report is just a part of the constellation that tells the stories. Note the plural form as we have many stories to tell in a dossier, not just one.

Applying my working model means you stop seeing your work as being like a novella—something to be read from page 1 to page n. Applying my working model means you see your body of work as something that is read in a coordinate manner that is defined by very narrowly defined aspect rules of inclusion/exclusion. Applying my model means you stop seeing pages and sections and you start seeing concepts and topics.

My argument is that the selective professional reader at regulatory health agencies cares little about documents, sections, pages, and data tables. I am suggesting such readers care solely about making informed decisions and where in the submission dossier they find vehicles that can get answers to their concept and topic questions.

Why do we still organize clinical research reports by the IMRAD convention?

Here is Part 2 of the questions I posed in my previous blog post.

I have been thinking about this question for some time now. I remain curious as to why we still organize clinical research reports by the long-standing convention of Introduction, Objectives, Methods, Results, Discussion, and Conclusion? The IMRAD form of organization. The year 1665 is often cited as the origin of the platform commonly referred to as scientific papers. It was not until the second half of the 1800s that these documents moved in the direction of “theory – experiment – discussion.” Then starting in the early 1950s the IMRAD structure became the prominent norm for the structure of a scientific paper. 

The supposed reason for the IMRAD structure is that this organizational approach facilitates literature review, allowing readers to navigate articles more quickly to locate material relevant to their purpose. It has been suggested that the IMRAD structure effectively supports a reordering that eliminates unnecessary detail, and allow the reader to assess a well-ordered and noise free presentation of the relevant and significant information. I can see how this argument may apply to research manuscripts published in Journals, but it certainly does not apply to the kinds of study reports submitted to regulatory agency health authorities.  

Unfortunately, the supposedly neat order of the IMRAD arrayed report rarely corresponds to how regulatory readers consider using documents to help them make decisions. That is, decisions about how study results support or fail to support broad or narrowly defined questions/arguments. People are surprised to learn that health authority review agents rarely read study reports. They certainly engage with study reports and make use of the content. But read these documents? No, that is an approach not normally taken by the health authority review agent. 

Frankly, I cannot blame them. After all, the IMRAD form of organization does little to help them get answers to their questions or support their approach to making decisions.

Over the past 50 years, the idealized sequence of the IMRAD structure has on occasion been criticized for being too rigid and simplistic. Perhaps the most famous castigation was made by Peter Medawar in the early 60s. He criticized the IMRAD design for not giving a realistic representation of the thought processes of the writing scientist. And then in the mid 80s A. G. Gross wrote a paper titled: “The form of an experimental paper: A realization of the myth of induction” that challenged the notion of the IMRAD structure as well. 

I am right there too in questioning the merits of this approach. At least when it comes to study reports submitted to health authorities.

I believe the reason for rigid adherence to the IMRAD approach for research document organization is the precedence of the past. Not that it affords the most effective stucture for the reader to get what they want from a document.

As for me—I’d prefer a Question & Answer approach—something along the lines of organizing the report by study objectives with the study methods appropriately subordinated at the back of the report. Give me a report organized as follows: Objective—Results—Discussion—Assignment of Significance. Each section of the report deals with only one study objective and information is presented in the sequence I have suggested. Then the last section of the report would be Conclusions where you integrate and contextualize the study findings to answer the really big question of so what do all these findings mean?

So what do you think is the most effective form of organization for a regulatory clinical study report?

Why do we do some of the things we do in clinical study reports? Part 1

Over the past three months I have been doing a series of workshops focused on the authorship of clinical research reports for a couple clients. This work has me posing questions about two very specific notions

1. Why do people still demand that a clinical research report intended for incorporation into regulatory submission packages must be written as “stand-alone” documentation?
2. Why do we still organize clinical research reports by the long-standing convention of Introduction, Objectives, Methods, Results, Discussion, and Conclusion?

In this post let’s consider the notion of the “stand-alone” clinical study report.

First question I have is why must these documents be stand alone? The term “stand alone” generally refers to a document or device that is self-contained and does not require any other document or device to fully function. I argue that a clinical study report submitted to a drug regulatory agency is part of a larger corpus and as such the report should never be construed as a stand-alone document.

It is a given that most submissions made to drug regulatory agencies contain multiple research studies. So you will never have a “stand-alone” research study. So an individual study is part of a corpus of clinical research that best operates collectively. Also regulatory submissions have multiple sections as mandated by guidance that are topoi (places of argument) for different and often integrated attributes of clinical research drawn from this corpus. So clearly the clinical study report is not a stand-alone document in the instance of interpretation and argumentation.

Another point, a clinical study report incorporated into a regulatory submission will have numerous appendices including the final version research protocol and the statistical analysis plan. So I struggle to understand the premise that says you must incorporate extensive elements of the protocol and stats plan into the body of the clinical study report in order to assure the report is a stand-alone document. I truly fail to see the merit of this premise.

My last point is why do so many feel it is necessary to write the Introduction Sections to clinical study reports in the classic inductive form of rhetoric that moves from describing the disease condition, to the therapeutic void, to the chemical or biological construct of the drug under study and how this drug’s pharmacological action addresses a therapeutic void. The standard response is, “well we want to make this a stand-alone document.” Why? All the rhetorical moves you are trying to make in the Introduction will happen for sure elsewhere in the drug submission package in the appropriate topos. So why waste your time here, at the low level of the clinical study report? In my way of thinking the study report Introduction is intended to convey only two points:

1. Why are you doing the study (the answers to what questions do you want at the end of the study that you do not have now?)
2. What previous work has informed the design of this study?