Pre-Results Review Services and Resources

About Pre-Results Review

Research yielding unexpected or null results often goes unpublished, even when it has applied rigorous research design and used high-quality data. As a result, bodies of published literature tend to be comprised of papers that tell “tidy” stories but may not be wholly accurate representations of the world. Such publication bias can be costly to funders, to researchers attempting to build on past work, and to anyone seeking to base their decisions on published evidence.

In response to this problem, over 220 journals across the social and life sciences have adopted a publishing format called “registered reports” (also referred to as “pre-results review”), in which papers are selected for publication based on the merits of their research questions and methodological quality before the results are known. Pending successful implementation of the pre-specified research design, journals publish these papers regardless of the nature of their results (see figure below). Insights from early adoption of the process show that 55% of articles published based on pre-results review across the scientific disciplines have reported null results, while estimates for conventionally reviewed papers yielding null findings range from 5 to 20%.

Editorial Workflow

Pre-results review splits the peer review process into two stages. At Stage 1, authors submit an introduction, methods, and analysis plan for a prospective empirical study for which data have yet to be collected (or accessed) and analyzed. This submission is evaluated based on the importance of the research question(s), the soundness of the theoretical reasoning, and the credibility and feasibility of the research design.

The publishing process for pre-results review submissions at the JDE

Positively evaluated submissions are accepted based on pre-results review. This constitutes a commitment by the journal to publish the full paper, regardless of the nature of the empirical results. Authors then collect and analyze their data, and submit the full paper (including results and analysis) for final review and publication (Stage 2). The final Stage 2 review provides quality assurance and ensures alignment with the research design peer reviewed in Stage 1.

Services and Resources

Leveraging lessons learned and resources developed as part of our collaboration with the Journal of Development Economics, BITSS offers the following services and resources to journals interested in adopting pre-results review:

  1. Development of author guidelines and submission templates — or adaptation of existing guidelines — considering discipline-specific conventions in scholarly communication, as well as each journal’s methodological and topical focus. All resources developed by BITSS and the JDE are available for public use with attribution at https://osf.io/f4jx6/.
  2. Targeted outreach and promotion, leveraging our networks, media presence, and lists of contacts to deliver targeted messages and encourage uptake. In addition to email campaigns, our communication channels include the BITSS website (www.bitss.org) and blog (with 7,500+ monthly visitors), Twitter account (with 3,200+ followers), newsletter (with 1,300+ subscribers), and the Catalyst network – over 130 influential research transparency advocates representing 14 disciplines.
  3. Swift on-demand support and consultations for authors preparing submissions, as well as guidance on the use of transparency practices such as pre-registration and pre-analysis plans. We also develop promotional materials to clearly outline the review process and answer questions from authors and referees.
  4. Process monitoring, evaluation based on feedback from authors, reviewers, and editors to facilitate improvements in the submissions and peer review processes.

This implementation checklist outlines the basic steps that journals should take when considering adopting pre-results review.

Benefits for authors

Pre-results review offers several benefits to authors forgoing conventional peer review. Authors can:

  1. Add a prospective publication to their resume before collecting and analyzing data based on the in-principle acceptance granted by the journal, which usually comes sooner than acceptance in conventional peer review.
  2. Receive constructive feedback from peer reviewers, including helpful suggestions for research design before they begin data collection. Based on the JDE’s experience, many authors have found pre-results review helpful in improving the quality of their research papers, regardless of the outcome of the peer review process.
  3. Publish their work irrespective of the results of the study! Acceptance at Stage 1 is a commitment by the journal to publish the work regardless of the nature of its results.

Contact

Contact Grace Han at grace.han24[at]berkeley[dot]edu with any questions.