Accelerating Computational Reproducibility (ACRe)

Replication, robustness checks, and extension of published work are possible to the extent that published findings are first computationally reproducible. In other words, one must be able to reproduce tables and figures within a reasonable margin of error using the available data, code, and materials. In July 2019, to improve computational reproducibility in economics, the American Economic Association (AEA) updated its Data and Code Availability Policy to include pre-publication verification of computational reproducibility by the AEA Data Editor. Similar policies have been adopted in political science, particularly at the American Journal of Political Science.

In collaboration with Dr. Lars Vilhuber, the current Data Editor for the journals of the American Economic Association (AEA), BITSS launched the Accelerating Computational Reproducibility (ACRe) project. ACRe builds capacity for social science researchers and is developing the Social Science Reproduction Platform, an online tool for systematically sourcing and recording the results of attempts to verify the computational reproducibility of published work.

Training

ACRe supports social science researchers in meeting journal editorial expectations of pre-publication verification of computational reproducibility. Capacity-building activities include:

  • Training events for participants of major association meetings in economics (e.g., Allied Social Sciences Association, the Association for Public Policy Analysis and Management, the Western Economics Association International, etc.). Learn about past and upcoming ACRE training events here.
  • Guidance resources for improved workflow reproducibility for AEA authors.

Social Science Reproduction Platform

The Social Science Reproduction Platform (SSRP) is an openly licensed platform that facilitates the sourcing, cataloging, and review of attempts to assess and improve the computational reproducibility of social science research. Sign up for a free account now to get started in improving computational reproducibility—one claim at a time!

The platform allows users to:

  • Record the results of reproductions using a standardized survey, creating real, citable scientific contributions;
  • Review, comment, and collaborate on reproductions submitted by other users on the platform; and
  • Access aggregate metrics of reproducibility across publications, journals, sub-disciplines, timespans, etc.
The stages of reproductions conducted based on the SSRP approach.

The SSRP is based on the Guide for Accelerating Computational Reproducibility (ACRe Guide), collaboratively developed by BITSS and members of the open social science community. The ACRe Guide contains detailed steps and criteria for assessing and improving computational reproducibility, as well as resources for constructive and efficient communication between reproducers and original authors.

Using the SSRP in the Classroom

Instructors can use the SSRP in combination with the ACRe Guide to facilitate reproduction assignments in applied social science courses at the graduate and undergraduate levels. Students can use these materials with little to no supervision, covering learning activities such as:

  • Assessing and improving the reproducibility of published work;
  • Applying good coding and data management practices;
  • Engaging in constructive exchanges with authors; and
  • Developing a deep understanding of commonly used methods and computational techniques.

Find more information about how to use the SSRP in the classroom here.

Contact

Reach out to acre@berkeley.edu if you are interested in using the SSRP as part of your course or would like to contribute to the ACRe Guide or the SSRP.