Raymond Goldstein provides a valuable commentary on the peer-review process (Physics Today, December 2016, page 10), in which he finds that “with few exceptions, there is no mechanism for the referees and the editor to discuss the paper and arrive at a consensus recommendation before reviews are sent to the authors. Instead, the initial recommendation is based on the editor’s implicit averaging of the reports.” Goldstein then discusses the journal eLife, in which the “review process is an online discussion between the referees and the handling editor of a paper so that they arrive at a single consensus report … that is sent to the authors.”
I agree entirely with Goldstein’s assessment that scientists who work with the “deeply flawed process … deserve better” than the averaging he describes. At the same time, as founding editor, now emeritus, of Physical Review Applied, I offer some insight into the process that I was not aware of before taking on that position.
As an author, I, like most of my colleagues, have often been frustrated by conflicting referee reports such as the example Goldstein recounts. The canonical outcome of the review process is one review that, to varying degrees, pans the paper and a second that does the opposite. As an editor, I have found—to my surprise—precisely the same thing: I expected I would be able to either choose better referees or direct the referees in a way that would avoid quite so many diametrically opposed reviews. Alas, that has not been the case. It seems that we scientists are an opinionated and often contrary bunch.
That said, I have also learned that conscientious full-time editors with whom I have worked expend much time and energy working to reconcile differences between reviewers. Indeed, the process is similar to what Goldstein describes at eLife: When reviewers strongly disagree, the editors typically communicate the key points of the reviews back to the reviewers and ask whether a consensus view can be reached. It often can, in which case one or both reviewers modify their original report. When it cannot, we seek mediation from a third reviewer or from an editorial board member who is an expert in the field.
Much of this process occurs behind the scenes, and far more often than anyone would like, disagreements between reviewers persist despite anything that editors can do. The process used by eLife sounds highly worthwhile, and insofar as mediation results in consensus, its editors deserve all possible credit and support. Meanwhile, the editors at the Physical Review journals (and I am sure elsewhere) annually meet to propose, refine, and from time to time implement modifications to the peer-review process.
I will be interested to follow the progress of the eLife model and open, online, and other review approaches.1 Ultimately, however, I believe that conflicting reviews may be inevitable and that peer review, like democracy, is the worst system imaginable, except for the alternatives.
