Choosing Reviewers

Whether one is selecting reviewers in an editorial role (journal, conference, proposal, etc.) or for commentary on one’s own work, it is important to realize that there are both substantive and ethical concerns to weigh. Fortunately, the substantive concerns can inform the ethical concerns if this critical point is borne in mind: the decision of what to do with the comments made by reviewers rests with the editor or author. Reviewing is not an election or a vote: sometimes the editor or author concludes that the reviewers are wrong, and goes ahead with what he or she originally had in mind. But of course, the whole point of getting reviews is to improve the work. In practice, few editors or authors ignore the comments of good reviewers.

Peer review is not perfect, but it beats just about all the alternatives. It does not make substantive or ethical sense to send something for review to people who, it is clear in advance, will trash it because they hate the author or the work, or will praise it because they love the author or the work. People whose views are “known in advance” do not contribute much to the process. It is ethically inappropriate to send work to people who are sure to trash it or laud it. This cannot be considered “fair” to either the author or the principles of peer review. The harder cases lie in the “gray zone” between the extremes of hate and love. In selecting reviewers it can make good substantive sense to send the work to those known to be critical of the author or the work because critical views sometimes reveal the flaws in the work. Whether those flaws are fatal, can be fixed, or are inherent in the work itself is a judgment call.

A critic who responds with nothing more than the usual dislike of the author or the work might not be helpful to the author or the review process, and possibly should be passed over for such reviews in the future. Similarly, those known to like the author or the work can provide helpful comments, and are often motivated to make the work better through their criticism. A fan of the author or work will not contribute simply by responding with a testimonial saying that the work is great, any more than a hater will help by simply saying the work is bad. In all cases, the objective should be to filter out work that fails to make a contribution, and to improve work that does make a contribution.

The corollary to the selection of reviewers is how to use reviewer comments. It puts authors in an impossible bind to ask them to meet the requests of contradictory reviews. If one reviewer says “do X” and another says “do not do X” it is doubtful the author can satisfy both reviewers. The editor must decide what he or she wishes to see. Process can help (e.g., requiring a certain minimum number of reviews, using various “blind” techniques to lower the likelihood that reviewers know authors or vice-versa, sending work out for additional review). Ultimately, however, the decision of what to do is a judgment made by the editor and/or author. Moral hazards exist in the selection of reviewers because publication in good venues has value in terms of promotion, compensation, reputation, etc. It is important to behave in an ethical manner in selection of reviewers. In most cases, sticking close to the ideals of the peer review process will provide good ethical guidance.