Pre-registration is a great tool for prospective experimental studies, but how useful or feasible is it for observational studies, which make up the vast majority of social science research? In this video, Professor Miguel introduces two articles discussing this question and representing two very differing views. The first, written by Lucas Coffman and Muriel Niederle in 2015, presents an argument against using pre-analysis plans for laboratory experiments, while the second, written by Rafael Dal-Ré and colleagues in 2014, argues for the use of pre-analysis plans in observational studies.

If you have time, both articles mentioned in this paper are worth reading. We discuss them further below.

---

In the first article “Pre-analysis plans have limited upside, especially where replications are feasible”, economists Lucas Coffman and Muriel Niederle discuss the benefits and costs of using pre-analysis plans, study registries, and performing replications.

They argue that pre-analysis plans, while valuable for very large, unique studies, are less useful for studies in which multiple hypotheses are tested, when null results are likely to be unreported, or when replication is feasible and likely. They also argue that “recent empirical literature suggests the behavioral problems that pre-analysis plans attenuate are not a pervasive problem in experimental economics” and that “pre-analysis plans may discourage the use of novel research designs and hence inhibit studies of robustness of previous findings.”

They are especially concerned with how the requirement of a pre-analysis plan might discourage exploratory research: “…the costs for exploratory work may be increased relative to somewhat more derivative work as a researcher may be reluctant to head into uncharted territory if the researcher has to commit to a rigid pre-analysis plan beforehand.”

On the other hand, Rafael Dal-Ré, John Ioannidis, and six of their colleagues argue in “Making prospective registration of observational research a reality” that the registration of a pre-analysis plan can facilitate future meta-analyses and address the “file drawer problem” we discussed earlier:

“There are several postulated benefits in systematically registering all OSs [observational studies]: increasing transparency and credibility, improving the peer-review process and ethical conduct of studies, and ensuring that the totality of evidence is publicly available. Moreover, registration of OSs may enhance communication regarding explored, but not published, hypotheses, facilitate systematic reviews and research collaborations, and reduce redundancy and funding committed to research questions for which adequate studies have already been conducted or are being performed, allowing published evidence to be better placed in context.”

These authors also address Coffman and Niederle’s concerns that pre-analysis plans discourage exploratory research:

“A potential disadvantage of registering highly exploratory, hypothesis-generating research with complex, meandering analyses is the burden of ongoing serial amendments and the resulting hindrance of new idea generation, as well as reduction in the analyses of end points not prespecified because they were conceived after the study started. However, there is no compelling reason why new concepts should be hindered; they just need to be identified as post hoc observations. Such disclosure allows others to fully understand and openly debate the nature and merit of the analyses. There is no evidence that registering CTs [clinical trials] has led to fewer hypotheses being tested or a decline in secondary analysis of trial data. Conversely, there is greater recognition that hypotheses and analyses for testing them need to be specified a priori; without such delineation, study results can lead to biased reframing of the hypothesis or cherry-picking among unspecified end points.”

The authors call for journal editors, as well as research funders, to endorse registration in the same way that they have required Institutional Review Board (IRB) approval. Moreover, if IRB approval is required, a pre-analysis plan does not require much additional work.

What do you think? Do the benefits of using and registering a pre-analysis plan outweigh the costs of not doing so?

You can read the entirety of both papers by clicking on the links in the SEE ALSO section at the bottom of this page.

---

References

Coffman, Lucas C., and Muriel Niederle. 2015. “Pre-Analysis Plans Have Limited Upside, Especially Where Replications Are Feasible.” Journal of Economic Perspectives 29 (3): 81–98.

Dal-Ré, Rafael, John P. Ioannidis, Michael B. Bracken, Patricia A. Buffler, An-Wen Chan, Eduardo L. Franco, Carlo La Vecchia, and Elisabete Weiderpass. 2014. “Making Prospective Registration of Observational Research a Reality.” Science Translational Medicine 6 (224): 224cm1-224cm1. doi:10.1126/scitranslmed.3007513.