Preprints Mean Peer Review Can Be Reimagined as it Should Always Have Been

By Stuart RF King (Research Culture Manager, eLife) <s.king@elifesciences.org> ORCID: 0000-0003-4374-3587

There are perhaps few topics in scholarly publishing that elicit such mixed opinions as peer review. Academics and publishers alike will recognize it as both an essential yet imperfect part of the scientific process, while some acknowledge it as the “least worst” option that we have when it comes to evaluating research (Smith, 2010). Preprints, versions of scholarly works that are shared prior to being peer reviewed, are another subject that can divide opinion (Teixeira da Silva, 2018). The recent boom in preprints, however — particularly within the life and medical sciences — may now present an unprecedented opportunity to reimagine peer review for the better.

The most commonly listed shortcomings of peer review include it being slow, opaque, inconsistent and biased. It has also been criticised for being wasteful — in that it requires much time and effort frequently with little to show in return (Heeson and Bright, 2021) — and for perpetuating a power imbalance that disproportionately impacts those already most systemically disadvantaged by academia, including early-career researchers and scholars from historically underrepresented groups (Silbiger and Stubler, 2019).

Preprints provide both impetus and opportunity to rethink how peer review should be done. They have uncoupled and effectively inverted the review and dissemination stages that occur in most traditional journal publishing. No longer needed to determine whether and when a piece of scholarly work can be published, peer review is instead free to be reimagined into a more open, efficient and equitable process.

Fortunately, this change is not just possible but is already happening. Over recent decades, multiple journals started to make their peer review more transparent by posting the reviewers’ comments alongside the articles they publish (Polka et al., 2018). This approach, adopted by eLife since its inception over ten years ago (Schekman et al., 2012) and in part pioneered by BioMed Central, is also used at Biology Direct, The EMBO Journal , F1000 Research , PeerJ and several Nature-branded journals, among others (Nature, 2020). Now, an increasing number of organisations and initiatives — like Review Commons and PREreview — are building upon this foundation of open reviews, the wider Open Scholarship movement, and the growing momentum around preprints to reform peer review even further (see PREreview’s contribution “Tackling bias and inequity in the peer review process with PREreview” in this issue of Against the Grain).

eLife, for example, has focused on only reviewing articles that are first published as preprints since July 2021 (Eisen et al., 2020). In October 2022, we announced that, as of last month, we will be eliminating “accept/reject” decisions after peer review and instead instructing our reviewers and editors to prepare public reviews and a concise assessment for every preprint we review. These will then be published alongside it to transform it into a “Reviewed Preprint” (Eisen et al., 2022). Every “eLife assessment” will use a common vocabulary to summarise the significance of the findings and the strength of the evidence reported in the preprint; the hope is that providing a more nuanced appraisal of the work in a clear and consistent manner will help academia to move away from its obsession with journal titles. A group of nine funders and other research organisations have since committed to including reviewed preprints from eLife and others involved in preprint review in their evaluation processes.

This is a call for other publishers to seriously consider open review, particularly of preprints, to overcome the shortcomings of the closed, pre-publication peer review that has been the norm for the last fifty years or so. Adopting such a model of peer review could benefit almost all stakeholders across the research ecosystem: from the wider research community to those who fund research and researchers, and from the reviewers who assess research articles to the authors who publish them.

Publishing reviewers’ comments in general means that the wider community can benefit from them, too. During peer review, reviewers spend time reading, thinking carefully about, and then commenting on the work of another. In the process, they typically generate rich, in-depth, detailed appraisals of the work. Too often, however, all of that time and effort is largely used to reach a binary decision of accept or reject, after which it is effectively lost. By instead making reviewers’ reports public, either alongside published articles or reviewed preprints, readers may gain a more nuanced understanding of the work’s specific strengths and weaknesses. They would also be less reliant on crude proxies to determine its quality, such as assuming every work reviewed and published by a particular journal shares identical qualities.

This reimagined approach would also offer pedagogical benefits. With access to a wider sample of reviewers’ reports, early-career researchers would have additional opportunities to learn how to write their own peer reviews and, if the authors’ responses are published too, how to effectively respond to reviewers’ comments about their work. Both are essential skills that are rarely taught to budding academics. These skills instead form part of a hidden curriculum that is not taught and where unequal access to this knowledge serves to perpetuate inequalities in research communities (Goodman, 2022).

Importantly, publishing reviews openly for all articles would not preclude journals from helping readers identify works of interest to them. This is often cited as a valuable service provided by journals (Kelly et al., 2014) and could easily continue in a more sophisticated way than simply deciding whether a study can be published in a specific journal or not. For example, journal editors could curate lists or collections of reviewed preprints that they feel are exemplary of particular qualities, such as rigour or reproducibility, while the accompanying reviews could include prominent recommendations for which groups of researchers might be most interested in the article.

Publishing reviews would also help stop the work of peer reviewers from being invisible and thus easily undervalued. While peer review is often regarded as a community service, there is nothing to say that individual reviewers cannot also be recognised and rewarded for the skill and effort involved. For example, if made publicly available, a researcher’s peer reviews could provide evidence of their broader contributions to their field. This would mean, in turn, that funders and institutions could assess a more complete range of activities undertaken by researchers, beyond merely their publication records (which is also a key recommendation in the San Francisco Declaration on Research Assessment). Additionally, as those who are in large part financing peer review by paying researchers’ salaries (because peer review remains a largely unpaid activity (Aczel et al., 2021)), funders and institutions would see more return on their investment if their researchers’ peer reviews were made visible.

This kind of recognition would require that reviewers publish their reviews under their names. However, reviewer anonymity can be important under certain circumstances and should probably always be given as an option. That said, even if published anonymously, making reviews public demonstrates more respect for reviewers’ time and efforts. For example, in traditional journal peer review where valid criticisms lead to a rejection, there is nothing that prevents the rejected authors from simply ignoring the reviewer’s concerns and resubmitting their manuscript unchanged to another journal with the hopes that other reviewers will not identify its flaws. In this situation, the first reviewer’s time has largely been wasted, possibly serving only to delay publication and bump the paper down the perceived hierarchy of journals. This behaviour could also potentially waste the time of future reviewers by making them unknowingly assess a manuscript that has already been thoroughly reviewed. If instead, the authors knew that the reviewer’s comments were going to be added to the preprinted version of their article, they would be incentivised to ensure that valid concerns are addressed. And if the authors still opt to ignore the concerns, the public reviews would nevertheless serve to alert anyone reading the preprint to its potential flaws.

Finally, making peer review more transparent has the potential to improve the experience of authors too. The immediate benefits for those who publish preprints have already been discussed by others (Berg et al., 2016, McKiernan et al., 2016, Sarabipour et al., 2019). If open peer review of preprints were to become the norm, authors may also benefit from reviews that are more constructive. There would be little reason for reviewers to write superficial reviews that merely serve as a brief justification for their opinion that an article should be rejected. Instead, when peer review is no longer used to gatekeep publications and when the publication of reviewers’ reports becomes commonplace, there should be a greater incentive for reviewers and publishers alike to make sure that the reviews they produce are fair and balanced. Additionally, with more transparency, there should also be more accountability for publishers to make sure that their reviewers’ reports, which they will then publish, treat authors with civility and respect, lest they be judged by their community and potential future authors.

The remaining barrier to the wider adoption of this “preprintfirst” model of transparent peer review is not technological but cultural. The organisations already carrying out this type of review are evidence of this. As such, what is now needed is for more publishers to join them, capitalise on the opportunities presented by preprints and reimagine how they conduct peer review. A good first step could involve publishers simply publishing the peer reviews that are already being written alongside the related manuscripts, while also encouraging authors to post preprints. In time, with this direction of travel, it will become easier for further changes to transform peer review from the “least worst” option to the open, efficient and equitable process that it should always have been.

References

Smith R. Classical peer review: an empty gun. Breast Cancer Res 12 (Suppl 4), S13 (2010). https://doi.org/10.1186/bcr2742

Teixeira da Silva JA. The preprint debate: What are the issues? Med J Armed Forces India. 2018 Apr;74(2):162-164. doi: 10.1016/j.mjafi.2017.08.002

Heesen R, Bright LK. Is Peer Review a Good Idea? The British Journal for the Philosophy of Science. Volume 72, Number 3 (2021) doi: https://doi.org/10.1093/bjps/axz029

Silbiger NJ, Stubler AD. Unprofessional peer reviews disproportionately harm underrepresented groups in STEM. PeerJ. 2019 Dec 12;7:e8247. doi: 10.7717/peerj.8247.

Polka JK, Kiley R, Konforti B, Stern B, Vale RD. Publish peer reviews. Nature. (2018). 560:545-547. https://www.nature.com/ articles/d41586-018-06032-w#correction-0

Schekman R, Patterson M, Watt F, Weigel D. (2012). Scientific Publishing: Launching eLife, Part 1 eLife 1:e00270. https://doi. org/10.7554/eLife.00270

Nature will publish peer review reports as a trial. Nature 2020 Feb;578(7793):8. doi: 10.1038/d41586-020-00309-9

Michael B Eisen, Anna Akhmanova, Timothy E Behrens, Diane M Harper, Detlef Weigel, Mone Zaidi (2020) Peer Review: Implementing a “publish, then review” model of publishing eLife 9:e64910. https://doi.org/10.7554/eLife.64910

Michael B Eisen, Anna Akhmanova, Timothy E Behrens, Jörn Diedrichsen, Diane M Harper, Mihaela D Iordanova, Detlef Weigel, Mone Zaidi (2022) Scientific Publishing: Peer review without gatekeeping eLife 11:e83889. https://doi.org/10.7554/ eLife.83889

Goodman S. (2022). The Scholarly Skill Amost No One Is Teaching. The Chronicle of Higher Education https://www. chronicle.com/article/the-scholarly-skill-almost-no-one-isteaching

Kelly J, Sadeghieh T, Adeli K. Peer Review in Scientific Publications: Benefits, Critiques, & A Survival Guide. EJIFCC. 2014 Oct 24;25(3):227-43. PMID: 27683470; PMCID: PMC4975196.

Aczel, B., Szaszi, B. & Holcombe, A.O. A billion-dollar donation: estimating the cost of researchers’ time spent on peer review. Res Integr Peer Rev 6, 14 (2021). https://doi.org/10.1186/ s41073-021-00118-2

Berg JM, Bhalla N, Bourne PE, Chalfie M, Drubin DG, Fraser JS, Greider CW, Hendricks M, Jones C, Kiley R, King S. Preprints for the life sciences. Science. 2016 May 20;352(6288):899-901.

Erin C McKiernan, Philip E Bourne, C Titus Brown, Stuart Buck, Amye Kenall, Jennifer Lin, Damon McDougall, Brian A Nosek, Karthik Ram, Courtney K Soderberg, Jeffrey R Spies, Kaitlin Thaney, Andrew Updegrove, Kara H Woo, Tal Yarkoni (2016) Point of View: How open science helps researchers succeed eLife 5:e16800. https://doi.org/10.7554/eLife.16800

Sarabipour S, Debat HJ, Emmott E, Burgess SJ, Schwessinger B, et al. (2019). On the value of preprints: An early career researcher perspective. PLOS Biology 17(2): e3000151. https:// doi.org/10.1371/journal.pbio.3000151

<https://www.charleston-hub.com/media/atg/>