Elliot T. Berkman is professor of psychology at the University of Oregon. His Social and Affective Neuroscience Lab researches the motivational and cognitive factors that contribute to success and failure of real-world goals, as well as the neural systems that support goal pursuit. He actively communicates the societal impact of psychological research on Psychology Today’s blog, The Motivated Brain, and on Twitter as @Psychologician.

I’m too embarrassed to give you an exact date, but some decades ago I submitted my first research paper to a journal for peer review. The process was even slower in those days because of how the paper got distributed. I mailed—not emailed, not submitted via a website, but actually physically mailed—three copies of the paper along with a cover letter to the journal’s editor in chief. Upon deciding that the topic of the paper fit within the journal’s scope, the editor in chief then mailed all three copies along with my cover letter to an associate editor, who identified three reviewers and—yup, you guessed it!—mailed it to them. Countless weeks later, the reviewers mailed their reviews back to the associate editor, who drafted a decision, then mailed that back to the editor in chief. Several months after I dropped my envelope thick with manuscript pages into the mailbox, I received a letter back from the editor in chief requesting that I revise the paper in response to the reviewers’ critiques. Weeks later, I’d draft another cover letter to my pen pals describing the changes I made, starting the whole cycle over again.

I tell this tale not just to date myself or complain about the bad old days, but also to explain how one key part of the scientific process—peer review—is shaped by the technology of the day. The peer review process operated the way it did not because it was optimal but because it was the best we had at the time. When the internet emerged, the review process shifted to online submission portals. But progress is incremental, so the initial changes were superficial. Even with online submission, the process still mirrored the old structure of one editor in chief passing a paper to one associate editor, who passes it along to a small number of reviewers.

The open science movement currently permeating the field of psychology represents a breakthrough in scientific technology because it is fundamentally transforming how the peer review process operates. The changes in how psychologists review each other’s work represent the latest step in the evolution of the scientific method. Peer review in the open science era is more transparent, more accessible, and more democratic than it ever has been. 

I find that my students enjoy learning about the “behind the scenes” process of our science and how it has improved over time. Let’s look more closely at these new features of peer review in the open science era and how we might teach their importance to students.

Transparency. When scientists write a paper describing a research project, they now often submit the paper to a special website called a preprint server, such as PsyArXiv. An important part of preprint servers is version tracking, which enables authors and readers to see the “paper trail” of previous versions of a manuscript. Papers can be posted as drafts and updated as changes are made. That means anyone—not only editors and reviewers—can see the development of a manuscript and how it has been modified over time. Preregistrations are an important companion to preprints. Preregistrations are time-stamped documents that describe a study’s rationale and hypotheses, as well as the steps researchers plan to take in gathering, analyzing, and reporting data. By comparing a preprint to a preregistration, readers can know how closely researchers followed their plan and whether and how they deviated from it. Together, preprints and preregistrations provide an unprecedented level of visibility into the scientific process. The accountability they provide is good for science and it is a good way for our students to learn about the realities of conducting psychological research. 

Accessibility. Many preprints are public. This feature is afforded by the internet, but the computer technology is not the main reason why researchers are now sharing their papers. Public preprints reflect a deeper shift in the mindset of researchers brought about by the open science movement. It was once the norm in the field to hide iterative drafts of a paper from view, as though making revisions to a document in response to reasoned feedback were a source of shame. Not long ago, it was standard to add the words “DRAFT – DO NOT CIRCULATE” to the title pages of papers. Now, many scholars proudly post drafts of their work, knowing that input from the research community will only improve it. In teaching this topic, you might encourage your students to consider how they would feel sharing a draft of a paper they are writing for class. What are the primary pros and cons of sharing in that way? Another major accessibility feature is that preprint servers provide an alternative to for-profit publishers, who limit access to science behind paywalls. Thanks to preprint servers, students of psychology can now access our science wherever they might be.

Democracy. In the “old” peer review system, only a small number of gatekeepers control what gets published and what gets rejected. One editor in chief decides if a paper is relevant to the journal, and three reviewers and maybe one associate editor give the paper a close read. Editors and reviewers are hand-selected to be people with long-established expertise in the field, and they were (and still are) predominantly White men from North America. Which papers get published or rejected was determined by people with status and power in the field and a vested interest in the status quo. Psychologists would never come to a scientific conclusion based on an N = 5 sample from such an unrepresentative population, so why would we accept a small sample when it comes to peer review? Peer review in the open science era is more democratic because anyone can weigh in on a manuscript, not just the anointed elite. We’re not there just yet, but it is easy to foresee a “Wikipedia” model of psychological research where manuscript review and editing can be crowdsourced to a large number of scholars operating with very few barriers to access. 

What I’ve presented here is an idealized vision of how peer review operates in the open science era. We’re presently in an in-between state where many papers are posted to preprint servers and simultaneously submitted for traditional (closed) peer review. On occasion, input provided from peers on a preprint gets included in a revision alongside changes suggested by journal-requested peer reviews. However, the norm is still traditional peer review even as we move toward a more open, accessible, and democratic system. In addition to old-fashioned academic inertia, the main barriers to broader adoption of open peer review are questions about the quality of review, the high value placed on publication in traditional journals by hiring and promotion committees, and the lack of incentives for people to review preprints.

Despite these barriers, I am convinced that open review is the future of peer review. The advantages of open review are too many and the barriers can be overcome. For example, many people have questions about the quality of review in the traditional system as it is now, and hiring and promotion committees are beginning to recognize the increasingly diverse means of scientific knowledge dissemination and peer review. Perhaps most importantly, current trainees who will be the next generation of psychological scientists are enthusiastic about open science. They know the downsides of the present system all too well and are leading this revolution in how science is conducted. This generation of scientists will only accelerate the adoption of open science as they take up leadership positions in the field.