30 March 2008

Bad Advice on Peer Review of Grant Proposals

The blog Entertaining Research posted some comments on the question I posed regarding formal training for peer review. One piece posted cited work by Alan Jay Smith the Task of the Referee. I think this paper is a good reference point for discussion of peer review of proposed journal articles. However, it is not a good reference regarding the task of assessing grant proposals. I find Smith encouraging some of the elements that contribute to poor review performance and variable outcomes as described by Nancy Mayo, et al. at the 2005 JAMA Conference on Peer Review: “Peering at Peer Review: Harnessing the Collective Wisdom to Arrive at Funding Decisions About Grant Applications” or their associated article: “Peering at peer review revealed high degree of chance associated with funding of grant applications” published in the Journal of Clinical Epidemiology [the conclusion taken by Mayo and her co-authors is there is a lack of concordance among reviewers on the relative merits of individual research grants, which indicates there is a risk that funding outcomes will depend on who is assigned as reviewers rather than the merits of the project] and by Cole, Cole, and Simon: Science, 1981, Vol 214, Issue 4523, 881-886 “Chance and consensus in peer review” [the paper covers an experiment in which 150 proposals submitted to the National Science Foundation were evaluated independently by a new set of reviewers. Results showed little correlation between sets of reviewers and suggest that getting a research grant depends to a significant extent on chance. The implication here is that the method of grant proposal assessment is very idiosyncratic].

In Smith's work, he suggests that subjective assessment of grants is acceptable. That is, judge merit of future work based on prior output even if proposal is sloppy or contains insufficient detail and judge the merit of proposed work based on where one is educated. Here is a quote from his paper:
A major difference between a research proposal and a paper is that a proposal is speculative, so you must evaluate what is likely to result. Therefore, when you evaluate a proposal by a well-known investigator, a substantial fraction of that evaluation should depend on the investigator's reputation. People with a consistent history of good research will probably do good work, no matter how sloppy or brief their proposal. People with a consistent history of low-quality research will probably continue in the same manner, no matter how exciting the proposal, how voluminous their research, or how hot the topic. However, you must also consider the possibility that a well-regarded researcher may propose poor research or that a researcher noted for poor-quality work has decided to do better work. It is important that you do not discriminate against newcomers who have no reputation, either good or bad. In this case, you must rely much more heavily on the text of the proposal and such information as the investigator's PhD institution and dissertation, academic record, host institution, and comments by his or her advisor or others.
The need is for a well-characterized set of standards with associated set of criteria for judging the merits of research proposals. Then all assessors must base their decisions within the framework of the standards and they must make their arguments regarding merit based on the criteria.


Originally published on our Knowledge Management blog

13 March 2008

The Role of the Professional Writer in the World of Drug and Medical Device Research

We at McCulley/Cuppan (M/C) begin this new blog discussing a topic that is important to the life sciences industry and yet does not receive enough attention: the role of the professional writer in the world of drug and medical device research. Over time we will touch on several related themes, including Quality, Knowledge Management, and Review Practices. But we'll start by thinking about professional writers.

What is a professional writer? Definitions vary depending on perspective. Those employed as professional writers may have a view of their role that is different from someone who employs them or works alongside them. Our goal here is to provide a meaningful description of the professional writer in the life sciences. We encourage those engaged in such work to pause and consider comparisons between how they approach their work and the descriptions we lay out here and equally to consider how they want others within their respective organizations to view the professional writer in the life sciences. As the starting point for this discussion we want to talk about how professional writers working in the sciences are generally defined, or, to say it more accurately, undefined. In going to Wikipedia we find the definition for a science writer is the following:
A science writer is a journalist who specializes in writing about science topics, and thus practices science journalism. Due to this combination, a science journalist needs to be proficient in two areas: as a journalist who can write well enough for magazines and newspapers, and as a scientist, or at least as a dedicated learner capable of following complex science and explaining it in simple terms.
As a point of comparison the term technical writer has the following definition:
A technical writer is a professional writer who designs, creates, maintains, and updates technical documentation—including online help, user guides, white papers, design specifications, and other documents. Engineers, scientists, and other professionals may also produce technical writing, often handing their work to a professional technical writer for editing and formatting.Technical writers work any technical subject requiring explanation to a particular audience. A technical writer usually is not a subject matter expert (SME). Companies, governments and other institutions hire technical writers not for their expertise in a particular subject, but for their expertise in gathering information, analyzing the subject and the audience, and producing clear documentation. (See technical writing.)The technical writer's primary responsibility is to gather information and produce documentation tailored to a particular audience. A good technical writer creates documentation that is accurate, complete, and as concise as possible. Technical writers communicate in many ways: printed documentation, Web-based or other electronic documentation, training materials, and industrial film scripts.
Searching Wikipedia for the term medical writer finds no definition at all. But the term medical writing yields the following definition:
Medical writing is the activity of writing scientific documentation by someone who is a specialized writer (a medical writer) and is generally not one of the scientists or doctors who performed the research. A medical writer is anyone engaged in communication in the medical or allied professions and sciences. The purpose of medical writing is to have a writing specialist work together with the people who produce the scientific data in order to create documents that effectively and clearly express the messages the data have to tell. The medical writer also serves to make sure that the documents comply with any regulatory, journal or other guidelines in terms of content, format and structure.Medical writing is a well-established function in the pharmaceutical world, due to the fact that people have begun to recognize that not everyone can write a well-structured document that presents information and tells a message clearly and concisely. It is one thing to put a bunch of data onto a page, it is another thing to do it in such a way that someone not familiar with the data can immediately understand the meaning of the data. With an increasing demand for getting new drugs to the market as quickly as possible, writing documents that can be read and understood quickly and easily has become an ever higher priority. Hence, the growth of professional medical writing.
The American Medical Writers Association describes a medical writer in the following manner:
Although successful medical writers come from many backgrounds, we share certain attributes:
  • Knowledge of medicine or an aptitude for understanding it
  • Ability to write (most medical writing does not require us to write like Hemingway, but medical writers must be able to write clearly at a level appropriate to the audience and the project)
  • Education: College degree in science, pharmacy, medicine, journalism, or English. Scientific medical writing generally requires a scientific degree (science, pharmacy, medicine).
  • Marketing medical writing generally requires a degree in journalism or English
  • Deadline orientation
Characteristics of good medical writer include:
  • Thorough research
  • Accuracy
  • Logical organization
  • Clear thinking and writing
  • Readability
None of these definitions begins to approach our description of what professional writers do and can do in the task of codifying knowledge into highly usable reports and summary documents. At M/C we see the work of the professional writer in the life sciences as indeed the task of knowledge management.

In 2004, McCulley/Cuppan consultants Stephen A. Bernhardt, Julie Dyke Ford and Gregory P. Cuppan wrote an article for the American Medical Writers Association "From Medical Writer to Communication Specialist: Expanding Roles and Contributions in Pharmaceutical Organizations." (American Medical Writers Association. June 2004.) This article describes our vision of the expanding roles of the medical writer working the pharmaceutical drug development industry. We revisited this article and extended some discussion points to help advance the argument presented in this piece.

Our vision of the definition of the professional writer working in the life sciences is more aligned with the position taken by Sara Smith in her paper Staking a Claim: Positioning Technical Communication in Knowledge Management. Here Smith makes the case that technical writers are knowledge management practitioners (a concept we heartily endorse and apply to medical writers as well). We find it useful to look at how some other technical writers have attempted to define their work. Corey Wick, in his article “Knowledge Management and Leadership Opportunities for Technical Communicators” (2000), suggests that technical writers operate as knowledge managers whereby they extract and synthesize information from people with specialized knowledge and develop it into well-structured and understandable forms that are then usable by others (readers). Amanda Metz Bemer, in her article Technically Its All Communication: Defining the Field of Technical Communication, suggests that knowledge management is a sub-discipline of technical communication (an interesting position that we want to pursue in future postings). She further argues that technical communication should become the area of the organization that “people turn to when they need their knowledge managed.” This vision, as suggested by Metz Bemer, resonates well with us here at M/C and provides a highlight for how we see the professional writer operating in support of research and development projects in the life science industries. In subsequent installments of this discussion we will provide our arguments for the expansion of the role typically held by professional writers in the life sciences and, in the process, we will provide our working definition for that role.

We look forward to the thoughts and ideas of our readers on this topic as well.


Work Cited

Bemer, Amanda Metz. Technically Its All Communication: Defining the Field of Technical Communication. http://orange.eserver.org/issues/6-2/bemer.html/document_view

Smith, Sara. Staking a Claim: Positioning Technical Communication in Knowledge Management. http://orange.eserver.org/issues/6-2/smith.html/document_view

Wick, Corey. Knowledge Management and Leadership Opportunities for Technical Communicators. Technical Communication. 47.4 (2000): 515-29.


Originally published on our Knowledge Management blog