Chapter 5: Peer Review

Section 5.1: Introduction

The primary focus of this chapter is peer review in the context of academic journals. However, many of the principles apply to other forms of peer review, including those related to grant proposals and career advancement.

The APS Guidelines on Ethics provides a good structure for a discussion of peer review. The ethical principle describes the role of peer review: “Peer review provides advice concerning the merit of research proposals, the publication of research results, and career advancement of colleagues. It is an essential component of the scientific process.”[1] Peer review also plays a role in the development of some textbooks.

An article by Baldwin provides a history of peer review, with a focus on journals and grant proposals.[2] The Letters to the Editor that follow up on the article add different perspectives to a few of Baldwin’s points.[3] For a more in-depth look at the nineteenth century origins of the publication system, see this book review (and, of course, the accompanying book, if interested).[4]

Section 5.2: Fairness

From the APS Guidelines on Ethics,¹ “Peer review can serve its intended function only if the members of the scientific community are prepared to provide thorough, fair, and objective evaluations based on requisite expertise.”

In the academic world as well as in some national labs and industry settings, career advancement can be heavily influenced by one’s ability to attract research funding. One’s ability to publish papers can also be a factor in career advancement, either directly or indirectly (such as through its influence on grant applications). Decisions made regarding career advancement, grant applications, and acceptance of manuscripts for publication all commonly involve peer review. To treat members of the physics community fairly requires a peer review system that operates with fairness.

Physical Review Letters has a web page that provides information to its referees.[5] In particular the link to PRL Editorial Policies and Practices provides a fairly detailed overview of the manuscript review process and of expectations for referees. Also worth reading is the Guidelines for Referees, which is briefer and focuses on how a report is put together. For an alternate introduction to how a careful referee report is constructed, see the 1995 Letter to the Editor in Physics Today by Rothman, who wrote from his experience as former editor of the Journal of Applied Physics.[6]

Most physics journals have a single blind system for peer review, meaning the reviewers know the names of the authors but the authors do not know the names of the reviewers.  The reports written by the reviewers are also considered non-public documents, being passed back to the authors but not released to a wider audience. For an argument in support of listing the names of the reviewers and releasing their reports, see this Letter to the Editor.[7]

In the life sciences, more information on the issue of peer review is available, some of it from surveys and some of it from simple experiments to test for reviewer bias. Resnik summarizes these in a 2011 article.[8]

Discussion Prompts

1. Some journals use a double-blind peer review system: the authors are not told who the referees are and the author’s names are deleted from the manuscript before it is sent to the referees. This approach is intended to reduce the possibility of bias on the part of the reviewers. Under what circumstances do you think that the double-blind system would fail because the reviewers could guess the names of one or more of the authors?
2. In almost all cases, it is clearly not feasible for a reviewer of a manuscript about an experiment to reproduce the experiment as part of the review. Given realistic limitations like this, what are some reasonable expectations for a thorough review?

Section 5.3: Participation

From the APS Guidelines on Ethics,¹ “Although peer review can be difficult and time consuming, scientists have an obligation to participate in the process.”

Aside from concise statements such as the one above, not much has been written on the obligation of scientists to participate in the peer review process. To highlight the importance of participation, the American Physical Society now recognizes outstanding referees of submissions to its journals.[9]

Discussion Prompts

1. Discuss the benefits to the community of scientists as a whole when scientists participate in the peer review system.
2. Discuss the benefits to an individual for being a peer reviewer.

Section 5.4: Timeliness

From the APS Guidelines on Ethics,¹ “All steps in the peer review process should be executed as expeditiously as possible by reviewers, editors, and authors.”

Resnik’s paper⁸, mentioned in the previous section, includes a little bit of survey information on the issue of intentional delay by referees. Since the study had limitations (including focusing on the life sciences), one can at best conclude that such delays probably take place on occasion in physics, but it is not clear how often they happen.

Deliberate delays can give referees or their colleagues a competitive advantage in getting their results published first. Regardless of whether or not the delay was intentional, there can be negative consequences for the authors of the submission, including having less evidence to put forth in an application for funding or in an application for career advancement.

Discussion Prompts

1. Peer reviewers are usually at the level of advanced graduate students or above. Estimates for the length of time involved in being a peer reviewer of a manuscript are typically in the range of about five hours. How hard do you think it is for such a person to carve out five hours from their schedule in a three-week window of time, a typical timeline for reviewing a manuscript? Do you think it would be easier or harder for you to find five hours in the next three weeks of your time?
2. Sometimes, when an individual reads a report from a reviewer, they can guess the identity of the reviewer. If you have reason to believe that Dr. A was intentionally slow in providing a peer review of one of your manuscripts, is it ethical for you to level the playing field by being intentionally slow in peer reviewing one of Dr. A’s manuscripts, if you are given the opportunity for such a review?

Section 5.5: Confidentiality

From the APS Guidelines on Ethics,¹ “Privileged information or ideas that are obtained through peer review by reviewers and editors must be kept confidential and not used for competitive gain.”

A good practical introduction to rules that arise related to both confidentiality and conflict of interest can be found in the National Science Foundation document geared towards grant application reviewers.[10] Physical Review Letters has a succinct statement on confidentiality in the Guidelines for Referees section of its Information for Referees.[11]

Discussion Prompts

1. After you review a paper, at what point, if any, can you freely discuss the contents of that paper with your colleagues? [At some point during the discussion, instructors may wish to point out that if a preprint has been posted on a site like arXiv, then the content may already be public knowledge. A reviewer planning to take advantage of that fact should check the posted version to make sure it is the same as the version being reviewed.]
2. The National Science Foundation and other agencies often assemble review panels who meet in person to discuss a pool of applications for funding under a particular program. It is common for members of the panel to be asked not to disclose to their colleagues that they are members of that panel. (If necessary, panel members may say that they are participating in a panel without identifying which one.) What reasons might these agencies have for this request?

Section 5.6: Conflicts of interest

From the APS Guidelines on Ethics,¹ “Reviewers should disclose conflicts of interest resulting from direct competitive, collaborative, or other relationships with any of the authors, candidates, or proposers, and avoid cases in which such conflicts preclude an objective evaluation (see Conflicts of Interest and Commitment in Section IV).”

Conflicts of interest can result not only in bias in peer review, but also in the appearance of bias, which would then undermine confidence in the peer review system. In many contexts, such as when reviewing a journal submission or a grant application, the primary responsibility of the peer reviewer is to disclose possible conflicts of interest to the person or group who solicited the review. It is then the duty of the journal or granting agency representative to make the final decision on whether or not the conflict is significant enough that the review might be compromised. It is worth pointing out to students, however, that it is often difficult to avoid all conflicts of interest in a peer review. For instance, in a highly specialized field, there may be a limited number of peer reviewers with appropriate expertise, and they may be in direct competition for resources. While a granting agency would not allow someone who has a grant application to provide a peer review on another grant application in that same funding cycle and applicant pool, a reviewer without an application in that pool might be in competition in the broader sense with another applicant.

Resnik and Elmore have a short article on conflict of interest involving journal peer reviewers.[12] While the statistics reported are largely drawn from the medical field, almost all of the issues raised are relevant to physics, and the article provides a nice, concise summary of those issues.

Universities receiving federal funding are required to have conflict of interest policies related to federally funded research. A simple exercise for students would be to look up the policy at their own university. These policies tend to focus on financial conflicts, unlike the broader policies that also treat non-financial conflicts related to manuscript reviews¹¹ and grant reviews¹⁰.

Discussion Prompts

1. Journal policies generally emphasize disclosure of conflicts of interest that authors may have while agencies involved in processing grant applications tend to emphasize nonparticipation in reviews by people with conflicts of interest. Why is there a difference in approaches?
2. Read through your institution’s conflict of interest policy and discuss aspects, if any, that you had not expected to find there. Were there any provisions that you did not find that you had expected to?

Section 5.7: Career advancement

In this context, career advancement decisions refer to decisions physicists make or provide input on regarding the careers of other physicists, including decisions to hire, promote, and, in the case of academic positions, grant tenure. One common challenge in these decisions is the limited time and information. A job applicant, for instance, supplies a resume or CV and supporting materials. There may be dozens of application packets to review in a task that is likely to be an add-on to the reviewer’s usual workload. The volume of applications that need to be reviewed can lead to a tendency to rely on “gut reactions,” which in turn opens the door to inappropriate bias. The APS Guidelines on Ethics provides a good starting point to discussing these issues in Section III Treatment of Colleagues and Subordinates: Explicit, Systemic, and Implicit Bias.¹ In addition, most large institutions have a human resources department that provides guidance on many career advancement decisions. While the guiding procedures may at times seem overly constraining, they are designed to increase the objectivity of those with decision-making responsibilities. Students may be able to locate materials on their university’s human resources web page that pertain to career advancement decision-making.

Discussion Prompts

1. What types of systemic bias and implicit bias do you think commonly show up in college admissions systems? What changes to the process would you recommend to reduce or eliminate these biases?
2. Provide examples of implicit and systemic bias that might show up in career advancement decisions.
3. What sorts of conflicts of interest might arise for individuals participating in career advancement decisions, such as a faculty member participating in a tenure decision?

Section 5.8: Textbooks

Textbooks often undergo various forms of peer review. A publisher that is interested in a textbook proposal may send that proposal out for feedback from physicists (usually in academia) on the need for and content of the proposed book. A textbook that is under contract with a publisher may have some or all of its chapters reviewed by other physicists. Reviews requested by publishers generally involve some form of compensation. Sometimes they are open, not blind, in that the authors know who wrote each review. Finally, unlike reviewers of journal submissions, a textbook reviewer typically earns an acknowledgment in the book.

Another form of prepublication peer review that sometimes takes place is an author asking a colleague to look over some or all of a manuscript as a favor. These reviews are obviously open and usually result in an acknowledgment in the final product.

While the preceding forms of review are mostly hidden from public view (outside of the acknowledgments), book reviews can be considered a form of public, open peer review. Physics Today has at least two book reviews in almost every issue. Any of these could be easily accessed for a short reading assignment.

Discussion Prompt

Do you think that the open nature of book reviews in Physics Today makes them less reliable since the authors may not write with complete candor?

 

Continue to Chapter 6: Underrepresented Groups in Physics

Acknowledgment

The author is grateful for the time and effort of the anonymous reviewers of this work, and for their numerous helpful suggestions.

---

[1] American Physical Society Guidelines on Ethics (19.1) (2019). https://www.aps.org/policy/statements/guidlinesethics.cfm

[2] Melinda Baldwin, “In referees we trust?,” Physics Today 70 (2) 44-49 (2017). https://doi.org/10.1063/PT.3.3463

[3] Yves Gingras et al., “Peer review as conflict,” Physics Today 70 (10) 16-18 (2017).  https://doi.org/10.1063/PT.3.3712, https://doi.org/10.1063/PT.3.3713, https://doi.org/10.1063/PT.3.3714, https://doi.org/10.1063/PT.3.3715

[4] Matthew Wale, Writing the record of scientific knowledge,” Physics Today 72 (3) 59 (2019). https://doi.org/10.1063/PT.3.4167

[5] Physical Review Letters, “Information for Referees’” https://journals.aps.org/prl/referees (accessed October 8, 2019).

[6] Steven J. Rothman, “A Review of Peer Review,” Physics Today 48 (9) 124-125 (1995). https://doi.org/10.1063/1.2808189

[7] Tai-Yin Huang, “Peering into peer review,” Physics Today 61 (12) 14 (2008). https://doi.org/10.1063/1.3047647

[8] David Resnik, “A Troubled Tradition,” American Scientist 99 (1) 24ff (2011). DOI: 10.1511/2011.88.24

[9] American Physical Society, “Outstanding Referees Program,” https://journals.aps.org/OutstandingReferees (accessed October 8, 2019).

[10] National Science Foundation, “Conflict-of-Interest and Confidentiality Statement for NSF Panelists,”  (NSF Form 1230P, 10/2017) https://www.nsf.gov/pubs/forms/nsf1230p.pdf (accessed October 9, 2019).

[11] Physical Review Letters, “Guidelines for Referees,” https://journals.aps.org/prl/referees/guidelines-for-referees (accessed October 9, 2019).

[12] David B. Resnik and Susan A. Elmore, “Conflict of Interest in Journal Peer Review,” Toxicology Pathology 46 (2) 112-114 (2018). https://doi.org/10.1177/0192623318754792