Why most published research findings are false
In 2005, John Ioannidis, well known for his research on the validity of studies in the health and medical sciences, wrote an essay titled “Why Most Published Research Findings are False.” The blunt title and Ioannidis’s provocative and compelling arguments have made this paper one of the foundational pieces of literature in the areas of meta-science and research transparency. You’d be hard-pressed to find an article on these topics – published in a journal or the popular media – that doesn’t mention it.
In this video, I introduce you to the different types of errors that can occur in research, their probabilities, and the concept of statistical power. We will also learn about Positive Predictive Value, or the believability of a study’s findings, as well as how biases can impact results. The last part of the video lays out six corollaries that characterize scientific research and what scientists can do to improve the validity of research. We go into more depth about these corollaries below.
In the article, Ioannidis lays out a framework for demonstrating:
- the probability that research findings are false,
- the proportion of findings in a given research field that are valid,
- how different biases affect the outcomes of research, and
- what can be done to reduce error and bias.
Ioannidis first defines bias as “the combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced.” He goes on to say that “bias can entail manipulation in the analysis or reporting of findings. Selective or distorted reporting is a typical form of such bias”.
With increasing bias, the chances that findings are true decreases. And reverse bias – the rejection of true relationships due to measurement error, inefficient use of data, and failure to recognize statistically significant relationships – becomes less likely as technology advances.
Another important point Ioannidis makes is that, while multiple research teams often study the same or similar research questions, it is the norm that the scientific community as a whole tends to focus on an individual discovery, rather than on broader evidence.
He goes on to list corollaries about the probability that a research finding is indeed true:
Corollary 1: “The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.” He refers here to sample size. Research findings are more likely to be true with larger studies such as randomized controlled trials.
Corollary 2: “The smaller the effect sizes in a scientific field, the less likely the research findings are to be true.” Also remember that effect size is related to power. An example of a large effect that is useful and likely true is the impact of smoking on cancer or cardiovascular disease. This is more reliable than small postulated effects like genetic risk factors on disease. Very small effect sizes can be indicative of false positive claims.
Corollary 3: “The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.” If the pre-study probability that a finding is true influences the post-study probability that is true, it follows that findings are more likely to be true in confirmatory research than in exploratory research.
Corollary 4: “The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true.” “Flexibility”, Ioannidis tells us, “increases the potential for transforming what would be ‘negative’ results into ‘positive’ results”. To combat this, efforts have been made to standardize research conduct and reporting with the belief that adherence to such standards will increase true findings. True findings may also be more common when the outcomes are universally agreed upon, whereas experimental analytical methods may be subject to bias and selective outcome reporting.
Corollary 5: “The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.” Conflicts of interest may be inadequately reported and may increase bias. Prejudice may also arise due to a scientist’s belief or commitment to a theory or their own work. Additionally, some research is conducted out of self-interest to give researchers qualifications for promotion or tenure. These can all distort results.
Corollary 6: “The hotter a scientific field (with more researchers and teams involved), the less likely the research findings are to be true.”When there are many players involved, getting ahead of the competition may become the priority, which can lead to rushed experiments or a focus on obtaining flashy and positive results that are more publishable than negative ones. Additionally, when teams focus on publishing “positive” results, others may want to respond by finding “negative” results to disprove them. What results then, is something called the Proteus phenomenon, which describes rapidly alternating extreme research claims and opposite refutations.
Using his framework for determining Positive Predictive Value and the corresponding corollaries, Ioannidis concludes that “most research findings are false for most research designs and for most fields.”
While the wide extent of biased and false research findings may seem a harsh reality, the situation can be improved in a few ways. First, higher powered and larger studies can lower the proportion of false findings in a literature, with the caveats that such studies are more helpful when they test questions for which the pre-study probability is high and when they focus on broader concepts rather than specific questions. Second, rather than focusing on significant findings from individual studies, researchers should emphasize the totality of evidence. Third, bias can be reduced by enhancing research standards, especially by encouraging pre-study registration. Finally, Ioannidis suggests that, instead of only chasing statistical significance, researchers should focus on understanding pre-study odds.
After reading this, what are your reactions? Are you surprised? How, if at all, does this change your perception about research in general? How might the individual factors described in the corollaries influence each other to exacerbate bias?
Read the full essay on PLOS.org here. You can also find this link in the SEE ALSO section at the bottom of this page.
Reference
Ioannidis, John P. A. 2005. “Why Most Published Research Findings Are False.” PLoS Med 2 (8): e124. doi:10.1371/journal.pmed.0020124.
© Center for Effective Global Action.