Tackling Bias and Inequity in the Peer Review Process with PREreview

By Daniela Saderi, Ph.D. (Co-founder and Director of PREreview, PREreview / fiscally sponsored by Code for Science & Society) <daniela@prereview.org> <contact@prereview.org>

Preprints — versions of scholarly manuscripts that are freely published online by the authors before journal publication1 — offer the unprecedented opportunity to challenge and reform the way the research community engages with the evaluation of each other’s work. Traditional peer review is an opaque, biased, and antiquated process that for too long has remained in the hands of a few, for-profit publishers that have used it to effectively control the fate of knowledge production and dissemination for centuries.2 Some have (understandably) argued that peer review is a failure that cannot be repaired.3 At PREreview, a community-focused organization using preprints to improve peer review, we see a way forward, an alternative future for peer review that centers around openness, collaboration, and equity. In this short article we summarize our approach towards making that vision a reality.

The Long-standing Problems

Academic journal publication originated in Europe and North America, where both producers and publication gatekeepers were disproportionately white and male, typically from wellfunded, prestigious institutions. 4 Despite efforts to open scientific publication and make it more accessible, inclusive, and welcoming to diverse communities of researchers, much remains unchanged.

Traditionally, research is evaluated by a handful of reviewers hand-picked by a relatively homogenous group of journal editors.5, 6, 7 Behind closed doors, these editors and reviewers decide the fate of each article, free to make their decisions based on subjective criteria. Rarely is research assessed by a diverse pool of reviewers who can provide multiple, disparate viewpoints to check one another’s biases and provide a more comprehensive and context-appropriate evaluation of the work.

This lack of diversity in editorial boards and reviewer pools often makes it more difficult for journals to find reviewers from different backgrounds with the relevant expertise and experience even when they want to do so. This overburdens academics who receive multiple requests for peer review and prevents others from ever being asked to review.

Being asked to conduct peer review is seen as a mark of expertise and can open doors to opportunities for career progression.8 By not engaging researchers from marginalized groups and diverse geographic backgrounds, editors and journals perpetuate the existing system and disadvantage both themselves and the communities they serve.

Existing inequities, lack of diversity, and exclusionary practices within peer review are a manifestation of much larger structural and cultural problems maintained by traditional peer review practices. These oppressive systems include, but are not limited to, patriarchal systems, white supremacy culture, heteronormativity, and the practice of colonialism. All of these systems connect and overlap with one another. Because they are all human endeavors, they are part of who we are collectively and are embedded into everything we create.9 It is therefore imperative that we, as part of the community dedicated to improving the accessibility, inclusivity, and openness of the scholarly publication process, spend time and effort in understanding and recognizing the roots of these issues in order to offer solutions that are more likely to bring transformative change.

The PREreview Approach

PREreview was founded in 2017 by three women, early career academics who shared both frustrations with the current research system and a belief that research evaluation could be made more equitable, transparent, and collaborative.10

PREreview’s mission is to bring more equity and openness to the scholarly evaluation process, centering the needs and expectations of researchers who have been traditionally excluded.11 To us, equity and openness are two inextricable ingredients needed to bring positive change to how research is shared and evaluated. Without centering equity, approaches that solely focus on openness risk recapitulating the same oppressive systems and practices that have dominated scholarship for centuries.

Our approach can be framed around what we often refer to as the three “pillars” of PREreview: training, community, and the technological infrastructure needed to hold it all together.

• Training: PREreview has developed multiple preprint review training formats that can be tailored to participants’ needs.12 We have run everything from cohort-based programming — for those taking a deep dive into equitable review practices — to half-day workshops that stress the importance of understanding the root causes of biased scholarly evaluation processes stemming from centuries of systemic oppression. While we most often partner with journals, funders, and research institutions to facilitate trainings tailored to their communities, we are excited to explore programming for individual researchers in 2023.

• Community: PREreview also helps facilitate livestreamed preprint journal clubs13 in partnership with journals, labs and research groups who are starting their preprint review journeys. These events offer researchers from around the world the opportunity to collaborate on providing constructive feedback to improve preprints, whilst also connecting with peers and honing their peer review skills. In 2023, we plan to overhaul our community features to make it easier than ever to organize, schedule, and facilitate preprint journal clubs on PREreview.org.

• Infrastructure: PREreview.org also hosts an intuitive, welcoming platform for publishing preprint reviews onsite. Reviewers can either copy and paste their review or compose one on the website itself, and then publish it using either a pseudonym — to preserve their privacy and safety — or their real name, which can be linked to their ORCID iD for attribution. We are looking forward to further improving our review platform based on feedback and input collected from our communities. Through this multifaceted approach, we hope to foster a community of belonging for researchers who are interested in participating in peer review, shaping and owning the process, and maximizing the value it brings to their lives.

Next Challenges and Opportunities

Influencing culture change takes time; however, the shift is already well underway. The publication of preprints has rocketed in recent years, and the conversation has now moved on to not only getting research results out quickly, but gathering feedback quickly, openly, and from a global community of reviewers.14

Influencing culture change also takes a coordinated effort driven by globally distributed communities. We recognize that our work is limited by the perspectives, expertise, and experiences we have as a team and that our ideas are influenced by the communities with which we are most closely connected. We therefore value partnership building and try to approach collaborations from an opportunity perspective, offering our support, knowledge and resources to individuals, groups, and organizations who need it to advance goals and missions that align with ours.

In the upcoming months, we hope to build on some pioneering work done in collaboration with AfricArXiv, Eider Africa, eLife, and Training Centre in Communication Africa where we worked together to adapt our existing training to a Train-of-Trainer model that has the potential to scale beyond what we can do as individual organizations and empower the next generation of African peer reviewers.15

Looking forward to the next couple of years, we also aim at implementing engagement and technological strategies to more effectively connect preprint authors and reviewers at the point when that feedback is most needed. In collaboration with eLife and with the support of a recent award from the Chan Zuckerberg Initiative,16 we will implement new technological standards for authors to solicit constructive feedback on their preprint at the time of its publication on a preprint server (see eLife’s contribution “Preprints mean peer review can be reimagined as it should always have been” in this issue of Against the Grain). Furthermore, as a way to incentivize early sharing of preprints and facilitate public community feedback, in collaboration with ASAPbio, we will run a series of live-streamed journal clubs specifically on early preprints, i.e., preprints shared by authors well in advance of journal submission.

These are just a few examples of goals and activities that we are planning to pursue in the near future. Our work is our attempt at a single approach. It is imperfect and incomplete. And that is okay. We believe a future in which knowledge is shared and evaluated openly and equitably exists, but no one knows what that future looks like because we have never seen it. All we can do is to allow for mistakes, work together, and be open to feedback and change.

Calls to Action and Ways to Stay Connected

If you are a researcher interested in providing your input and feedback to our preprint review platform and programming, contact us at community@prereview.org

If you are the author of an early preprint and are interested in having your work showcased and constructively reviewed by your peers, you can learn more and request to be part of this pilot here

If you are a publisher or an organization sharing our vision of a more participatory and open peer-review process and are interested in working together, reach out to us at contact@ prereview.org. We offer ways to incorporate community-driven preprint reviews into journal-organized review processes, as well as tailored peer review training.

If you are a funder interested in learning more about our ideas and potentially supporting us, contact us at contact@ prereview.org.

For bimonthly updates from our team, sign up to our newsletter.

Follow us on Twitter, Mastodon, and LinkedIn.

Endnotes

1. https://en.wikipedia.org/wiki/Preprint

2. https://lps.library.cmu.edu/ETHOS/article/id/38/

3. https://experimentalhistory.substack.com/p/the-riseand-fall-of-peer-review

4. Alperin, Juan Pablo (2018): World scaled by number of documents with authors from each country in Web of Science: 2016. figshare. Figure. https://doi.org/10.6084/ m9.figshare.7064771.v1

5. Salazar JW, Claytor JD, Habib AR, Guduguntla V, Redberg RF. Gender, Race, Ethnicity, and Sexual Orientation of Editors at Leading Medical and Scientific Journals: A Crosssectional Survey. JAMA Intern Med. 2021;181(9):1248–1251. doi:10.1001/jamainternmed.2021.2363

6. Dada S, van Daalen KR, Barrios-Ruiz A, Wu K-T, Desjardins A, Bryce-Alberti M, et al. (2022) Challenging the “old boys club” in academia: Gender and geographic representation in editorial boards of journals publishing in environmental sciences and public health. PLOS Glob Public Health 2(6): e0000541. https://doi.org/10.1371/ journal.pgph.0000541

7. Altman, M., & Cohen, P. N. (2021, June 29). Openness and Diversity in Journal Editorial Boards. https://doi. org/10.31235/osf.io/4nq97

8. Squazzoni F, Bravo G, Farjam M, Marusic A, Mehmani B, Willis M, Birukou A, Dondio P, Grimaldo F. Peer review and gender bias: A study on 145 scholarly journals. Sci Adv. 2021 Jan 6;7(2):eabd0299. doi: 10.1126/sciadv. abd0299. PMID: 33523967; PMCID: PMC7787493.

9. PREreview Equity, Diversity, and Inclusion Statemen. https://content.prereview.org/edi

10. https://osaos.codeforscience.org/personal-perspectivethe-origin-story-of-prereview

11. https://content.prereview.org/mission

12. https://content.prereview.org/openreviewers

13. https://content.prereview.org/liveprejcs

14. https://www.science.org/content/article/researcherspush-preprint-reviews-improve-scientificcommunications

15. https://content.prereview.org/open-peer-reviewers-inafrica-a-train-of-trainer-program

16. https://elifesciences.org/for-the-press/35695590/ prereview-and-elife-welcome-chan-zuckerberginitiative-s-support-to-boost-community-engagementin-public-preprint-review

<https://www.charleston-hub.com/media/atg/>