Topics: Peer Review

Introduction

What is Research Ethics

Why Teach Research Ethics

Evaluation

Topics

Overview

Animal Subjects

Authorship

Biosecurity

Collaboration

Conflicts of Interest

Data Management

Human Subjects

Mentoring

Peer Review

Publication

Research Misconduct

Social Responsibility

Stem Cell Research

Whistleblowing

Educational Settings

Descriptions of educational settings, including in the classroom, and in research contexts.

Discussion Tools

Case Studies

Debates

Literature

Textbooks

Videos

Other Discussion Tools

About the RCREC

Information about the history and authors of the Resources for Research Ethics Collection

Summary

• The integrity of science depends on effective peer review
  A published paper reflects not only on the authors of that paper, but also on the community of scientists. Without the judgment of knowledgeable peers as a standard for the quality of science, it would not be possible to differentiate what is and is not credible.
   
• Effective peer review depends on competent and responsible reviewers
  The privilege of being part of the research community implies a responsibility to share in the task of reviewing the work of peers.

Background

For much of the last century, peer review has been the principal mechanism by which the quality of research is judged. In general, the most respected research findings are those that are known to have faced peer review. Most funding decisions in science are based on peer review. Academic advancement is generally based on success in publishing peer-reviewed research and being awarded funding based on peer review; further, it involves direct peer review of the candidate's academic career. In short, research and researchers are judged primarily by peers.

The peer-review process is based on the notion that, because much of academic inquiry is relatively specialized, peers with similar expertise are in the best position to judge one another's work. This mechanism was largely designed to evaluate the relative quality of research. However, with appropriate feedback, it can also be a valuable tool to improve a manuscript, a grant application, or the focus of an academic career. Despite these advantages, the process of peer review is hampered by both perceived and real limitations.

Critics of peer review worry that reviewers may be biased in favor of well-known researchers, or researchers at prestigious institutions, that reviewers may review the work of their competitors unfairly, that reviewers may not be qualified to provide an authoritative review, and even that reviewers will take advantage of ideas in unpublished manuscripts and grant proposals that they review. Many attempts have been made to examine these assumptions about the peer review process. Most have found such problems to be, at worst, infrequent (e.g., Abby et al., 1994; Garfunkel et al., 1994; Godlee et al., 1998; Justice et al., 1998; van Rooyen et al., 1998; Ward and Donnelly, 1998). Nonetheless, problems do occur.

Because the process of peer review is highly subjective, it is possible that some people will abuse their privileged position and act based on unconscious bias. For example, reviewers may be less likely to criticize work that is consistent with their own perceptions (Ernst and Resch, 1994) or to award a fellowship to a woman rather than a man (Wennerds and Wold, 1997). It is also important to keep in mind that peer review does not do well either at detecting innovative research or filtering out fraudulent, plagiarized, or redundant publications (reviewed by Godlee, 2000).

Despite its flaws, peer review does work to improve the quality of research. Considering the possible failings of peer review, the potential for bias and abuse, how can the process be managed so as to minimize problems while maintaining the advantages?

Regulations and Guidelines

Most organizations reviewing research have specific guidelines regarding confidentiality and conflicts of interest. In addition, many organizations and institutions have guidelines dealing explicitly with the responsibilities of peer reviewers, such as those of the American Chemical Society (2015), the Society for Neuroscience (1998, and the Council of Biology Editors (CBE Peer Review Retreat Consensus Group, 1995). And, there had been a federal requirement that made discussion of peer review part of instruction in the responsible conduct of research (National Institutes of Health, 2009).

Peer review is governed by federal regulations in two respects. First, federal misconduct regulations can be invoked if a reviewer seriously abuses the review process, and second, peer review for the grant process prohibits review by individuals with conflicts of interest.

Despite these regulations, much of peer review is not directly regulated. It is governed instead by guidelines and customs.

Discussion

Case Study 1

Dr. George Adams receives a manuscript for ad hoc review from the editor of a scientific journal. George gives the manuscript to Al Nance, his senior postdoctoral fellow. He asks Al to read the manuscript and prepare some written comments critiquing it. One week later, Al provides to Dr. Adams one page of comments. Al also provides Dr. Adams with an extensive verbal critique of the paper. Dr. Adams then prepares a written review which is submitted to the editor of the scientific journal. A few weeks later, Dr. Adams learns that Al made photocopies of the entire literature citation section of the manuscript because it contained "some useful references". Dr. Adams proceeds to verbally reprimand Al, telling him that no part of a manuscript received for review should be copied. Comment on the behavior of both the faculty member and the postdoctoral fellow in this scenario.

Case Study 2

Dr. Taylor, an expert in the area of aging and mental health, agreed to review an unpublished manuscript for a leading journal. Although Dr. Taylor has limited time outside of his teaching and research activities, he found the article so interesting that he gathered some of his colleagues together to share the findings with them.

Case Study 3

Alana is a medical student researcher in the laboratory of Prof. Hayes. Prof. Hayes has received a manuscript for review for possible publication in a biomedical journal and asks Alana to review the manuscript. Alana knows that the review process is intended to be confidential, so she asks if the journal editor has been notified of this request. Prof. Hayes says that this is not necessary. Alana asks for your advice.

Is Professor Hayes' answer (that notification is not necessary) ethical? Why or why not?

Questions

1. Based on your own experience, or on discussion with someone who is an experienced reviewer, which of the following are common practice? Which of the following should not be acceptable practice? 
       a. The reviewer is not competent to perform a review but does so anyway.
       b. Reviewer bias results in a negative review that is misleading or untruthful.
       c. The reviewer delays the review or provides an unfairly critical review for the purpose of personal advantage.
       d. The reviewer and his or her research group take advantage of privileged information to redirect research efforts.
       e. The reviewer shares review material with others (for the purpose of training or scientific discussion) without notifying or obtaining approval from the editor or funding agency.
2. What are the advantages and disadvantages of having a reviewer blinded to the identity of manuscript authors, a grant applicant, or a candidate for academic advancement?
3. What are the advantages and disadvantages of having manuscript authors, a grant applicant, or a candidate for academic advancement blinded to the identity of a reviewer?
4. What are the ethical responsibilities of peer reviewers?
5. List and describe federal regulations relevant to peer review.
6. Should reviewers working in the same field of research be excluded from reviewing each others' work? How can the risks of bias and the advantages of expertise be reconciled in the selection of peer reviewers?
7. What are the responsibilities of a reviewer to preserve the confidentiality of work under review? What protections, if any, help to prevent the loss of confidentiality?

The purpose of peer review is not merely to evaluate the submitted work, but also to promote better work within the scientific community. As such, there are several essential responsibilities for peer reviews.

• Provide a timely response
  Reviewers should make every effort to complete a review in the time requested. If it is not possible to meet the conditions for the review, then the reviewer should promptly decline or see if some accommodation is possible. Research reports, grant applications, and academic files submitted for review all represent a significant investment of time and effort, and frequently the documents under review contain timely results that will suffer if delayed in the review process.
   
• Ensure Competence
  Reviewers who realize that their expertise is limited have a responsibility to make their degree of competence clear to the editor, funding agency, or academic institution asking for their expert opinion. A reviewer who does not have the requisite expertise is at risk of approving a submission that has substantial deficiencies or rejecting one that is meritorious. Such errors are a waste of resources and hamper the scientific enterprise.
   
• Avoid Bias
  Reviewers' comments and conclusions should be based on a consideration of the facts, exclusive of personal or professional bias. To the extent possible, the system of review should be designed to minimize actual or perceived bias on the reviewers' part. If reviewers have any interest that might interfere with an objective review, then they should either decline a role as reviewer or declare the conflict of interest to the editor, funding agency, or academic institution and ask how best to manage the conflict.
   
• Maintain Confidentiality
  Material submitted for peer review is a privileged communication that should be treated in confidence. Material under review should not be shared or discussed with anyone outside the designated review process unless approved by the editor, funding agency, or academic institution. Authors, grant applicants, and candidates for academic review have a right to expect that the review process will remain confidential. Reviewers unsure about policies for enlisting the help of others should ask.
   
• Avoid unfair advantage
  A reviewer should not take advantage of material available through the privileged communication of peer review. One exception is that if a reviewer becomes aware on the basis of work under review that a line of her or his own research is likely to be unprofitable or a waste of resources, then they may ethically discontinue that work (American Chemical Society, 2006; Society for Neuroscience, 1998. In such cases, the circumstances should be communicated to those who requested the review. 
  Beyond this exception, every effort should be made to avoid even the appearance of taking advantage of information obtained through the review process. Potential reviewers concerned that their participation would be a substantial conflict of interest should decline the request to review.
   
• Offer constructive criticism
  Reviewers' comments should acknowledge positive aspects of the material under review, assess negative aspects constructively, and indicate clearly the improvements needed. The purpose of peer review is not to demonstrate the reviewer's proficiency in identifying flaws but to help the authors or candidates identify and resolve weaknesses in their work.

Resources

• Best Practices for Publishing Your Research
  An excellent online tutorial that discusses all aspects of preparing research for publication, including issues of peer review.
   
• OEC Publication Ethics Bibliography
  A bibliography of materials on the ethics of publication, including peer review. Includes guidelines, websites, books, and journal articles. 
   
• Publication Ethics Subject Aid
  A short guide to some key resources and readings on the topic of publication ethics, including authorship, peer review, and collaboration.

Cited Resources

• Abby M, Massey MD, Galandiuk S, Polk HC (1994): Peer review is an effective screening process to evaluate medical manuscripts. JAMA 272: 105-107. 
• American Chemical Society (2015): Ethical guidelines to publication of chemical research. ACS Publications
  http://pubs.acs.org/userimages/ContentEditor/1218054468605/ethics.pdf
• CBE Peer Review Retreat Consensus Group (1995): Peer review guidelines - A working draft. CBE Views 18(5): 79-81. 
• Ernst E, Resch KL (1994): Reviewer bias: a blinded experimental study. Journal of Laboratory and Clinical Medicine 124(2): 178-82. 
• Garfunkel JM, Ulshen MH, Hamrick HJ, Lawson EE (1994): Effect of institutional prestige on reviewers' recommendations and editorial decisions. JAMA 272: 137-138. 
• Godlee F, Gale CR, Martyn CN (1998): Effect on the quality of peer review of blinding reviewers and asking them to sign their reports. JAMA 280: 237-240. 
• Godlee F (2000): The ethics of peer review. In (Jones AH, McLellan F, eds.): Ethical Issues in Biomedical Publication. Johns Hopkins University Press, Baltimore, MD, pp. 59-84. 
• Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D, PEER Investigators (1998): Does masking author identity improve peer review quality? JAMA 280: 240-242. 
• National Institutes of Health (2009): Update on the Requirement for Instruction in Responsible Conduct of Research.
  https://grants.nih.gov/grants/guide/notice-files/not-od-10-019.html
• Society for Neuroscience (1998): Responsible Conduct Regarding Scientific Communication.
  https://www.jneurosci.org/content/20/1/F5
• van Rooyen S, Godlee F, Evans S, Smith R, Black N (1998): Effect of blinding and unmasking on the quality of peer review. JAMA 280: 234-237. 
• Wennerds C, Wold A (1997): Nepotism and sexism in peer review. Nature 307: 341.