By Fernando Hoces de la Guardia
This post was originally published on the CEGA Blog on May 25, 2021.
The emergence of “alternative facts” and post-truth politics is usually associated with the rise of populism in western democracies. While there is an element of truth to this association, democratic governments have struggled with estimating the expected costs and benefits of policy options even before the rise of new authoritarian trends. In other words, long before the emergence of alternative facts, we have allowed for multiple policy reports to co-exist, often representing vastly diverging “facts.” For example, in 2010 two reputable analysts made diverging claims about the Affordable Care Act — the Congressional Budget Office (CBO) estimating $138 billion in net savings while its former Director claimed it would cost $562 billion. These disagreements are more subtle than blatantly disagreeing on the size of crowds; however, they are similar in nature as they legitimize several (often conflicting) positions, where analysis using the best available methods and data should typically favor a limited range of policy estimates.
The traditional justification for differences across policy reports can be summarized in two famous words: “it depends.” Policy (and academic) economists often justify the existence of multiple reports quantifying the same policy estimate by claiming that they rely on different assumptions and modeling choices. The fundamental problem is that these assumptions and modeling choices can be easily buried in opaque reports that in turn leave plenty of latitude for analysts to obtain their desired results. Multiple opaque analytical choices give way to multiple reports, and multiple reports allow policymakers to choose the most convenient set of “facts.”
This is where open science tools and practices, which emerged in response to a credibility crisis in academic research, can be used to shed light on policy analysis. Transparent and reproducible analyses allow us to see under the hood, making it possible to distinguish between policy reports that contain the most accurate representation of the facts and those that are closer to post-hoc justifications of a desired result.
The Open Policy Analysis (OPA) initiative at CEGA, led by BITSS aims to accelerate the adoption of open science practices by policy analysts. We have developed a framework (ungated) to guide this transformation and identify areas where more work needs to be done, drawing a parallel between the problems that open science addresses in academic research and their less discussed counterparts in policy analysis. We proposed basic principles for OPA and piloted the application of this framework in an existing policy analysis of US Senator Elizabeth Warren’s wealth tax proposal with Emmanuel Saez (UC Berkeley) and Gabriel Zucman (UC Berkeley), providing a tool for both advocates and skeptics of the policy to use in the debate. More recently, we have begun to develop a series of OPA projects, where we collaborate with policy analysts and researchers across domains to help them “open up” highly consequential policy reports.
“Transparent and reproducible analyses allow us to see under the hood, making it possible to distinguish between policy reports that contain the most accurate representation of the facts and those that are closer to post-hoc justifications of a desired result.”
We just released our second OPA project, this time in the domain of deworming interventions. We collaborated with the authors of original research and a key policy organization looking to open up a policy analysis that compares the costs and benefits of mass deworming interventions for children. The best available research had, until now, primarily focused on policy analyses for rural Kenya where the original study took place. With this new OPA, analysts and policymakers can assess the relevant costs and benefits behind mass deworming across a wide array of contexts using this interactive plot to enter geographic-specific costs of treatment and prevalence of worm infections.
Even if supporters and opponents of a given proposed policy might arrive at the table with some fundamental differences in key parameters of the policy analysis (see worm wars), the OPA framework allows for different parties to play around with the key components of the analysis and specifically pin down where their differences exist, while agreeing on structure and methodology of the overall analysis. See the OPA project page for more information about the initiative, and this discussion of its role in the deworming policy and other development programs.
We hope that the policy analysis and research community will accept this OPA on deworming as the current most accessible and complete representation of the facts around the costs and income benefits of mass deworming across several settings. We recognize that new data and methods are constantly generated. As they are integrated into our deworming OPA — whether this implies light updates or a complete overhaul — we hope that the underlying rationale and implementation will remain fully transparent and reproducible to facilitate productive policy debates.
We look forward to deepening collaborations with researchers and policy analysts in opening up policy reports and to further supporting an ever-growing OPA community. We envision OPA reaching a critical mass in the near future such that full transparency by default is seen as a signal of credibility and superior quality in policy analysis. In this future we see a much clearer path to agreeing on what the facts are, enabling adversaries to pinpoint the origins of potential differences ensuring that debates and decision making (often impacting many lives where official statistics are limited) can move forward constructively and efficiently.