If social science is, in fact, experiencing a revolution, what changes have we seen? Economists Joshua Angrist of MIT and Jörn-Steffen Pischke of the London School of Economics identified some recent shifts in how economists and other social scientists do research. In general, they found that different subfields tend to use very different methods.

---

I’d like to dive a little deeper into Joshua Angrist and Jörn-Steffen Pischke’s article “The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con out of Econometrics,” which I discussed in the video. In the article, Angrist and Pischke explore Ed Leamer’s view of econometrics as explained in his well-known 1983 paper “Lets take the con out of econometrics” and seek to determine “whether the quality and the credibility of empirical work has increased since Leamer’s pessimistic assessment.”

In 1983, Leamer argued that contemporary empirical work “suffer[ed] from a distressing lack of robustness to changes in key assumptions – assumptions he called ‘whimsical’ because one seemed as good as another.” The ideal research design, he believed, was randomized trials and the “best way to use nonexperimental data to get closer to the experimental ideal [was] to explore the fragility of nonexperimental estimates.” However, he didn’t actually advocate searching for credible natural experiments or even performing randomized trials, and none of the central figures in his debate had much to say about research design.

To confront the “whimsical nature” of the key assumptions made in regression analysis, Leamer proposed that a process of sensitivity analysis should be implemented, featuring an “explicitly Bayesian agenda” and “extreme bounds analysis” — an estimation of regressions using different sets of covariates as controls.

Angrist and Pischke argue that empirical econometric work has vastly improved since Leamer’s article was published in 1983 and that “a clear-eyed focus on research design is at the heart of the credibility revolution in empirical economics.” They list other key factors contributing to this improvement, including the availability of more and better data and advances in theoretical econometric understanding, as well as sensitivity analysis for which Leamer advocated. They offer explanations for the improvements in empirical work today:

“Better and more robust estimation methods are part of the story, as is a reduced emphasis on econometric considerations that are not central to a causal interpretation of the main findings. But the primary force driving the credibility revolution has been a rigorous push for better and more clearly articulated research designs.”

They also state that a “well-done observational study can be more credible and persuasive than a poorly executed randomized trial.”

Further expressing the importance of research design, Angrist and Pischke argue that “good designs have a beneficial side effect: they typically lend themselves to a simple explanation of empirical methods and a straightforward presentation of results.” They go on to say that they “find the empirical results generated by a good research design more compelling than the conclusions derived from a good theory…”

In response to this “revolution” in credibility and research designs, concerns have arisen, especially with regards to external validity — whether or not the evidence from an experiment is applicable to situations beyond those of the original study. Angrist and Pischke respond to such concerns saying that “empirical evidence on any given causal effect is always local, derived from a particular time, place, and research design” and that a constructive response would be to “look for more evidence, so that a more general picture begins to emerge.”

In conclusion, Angrist and Pischke argue that improvement in empirical work in econometrics is not due to the use of extreme bounds sensitivity analysis, as Leamer suggests, but due to better research design and “careful implementation of quasi-experimental methods.” The progress that has been made in this field has produced a “credibility revolution” in fields as varied as labor, public finance, and development economics.

If you want to dive deeper into the material, you can read the entirety of each paper by clicking on the links in the SEE ALSO section at the bottom of this page.

---

Reference

Angrist, Joshua D, and Jörn-Steffen Pischke. 2010. “The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics.” Journal of Economic Perspectives 24 (2): 3–30. doi:10.1257/jep.24.2.3.