Can the existence of a registry help us deal with hidden findings?
The medical sciences have led the way in requiring pre-registration, leading to substantial growth in their use. But how effective is the registry in shifting the reporting of actual results? In 2009, Sylvain Mathieu and his colleagues set out to answer this question by comparing collected data and results reported in medical journals. Their findings demonstrate ample room for improvement.
We’ll explore pre-registration and pre-analysis plans further next week.
Registries were developed to address the “file drawer problem” and encourage more complete reporting of results. But because they are a relatively recent platform – the American Economics Association (AEA) RCT registry was recently established in 2012 – Sylvain Mathieu and four of his colleagues in the medical sciences decided to examine the impact of registries on reporting.
In 2005, the International Committee of Medical Journal Editors (ICMJE) began requiring the registration of clinical trials. As a precondition for publication in a journal, investigators were required to disclose information about their trial before enrolling participants.
In order to assess the prevalence of primary outcome reporting bias, the authors selected and examined 323 cardiology, rheumatology, and gastroenterology RCTs published in medical and specialty journals. Their main objective was “to help achieve transparency in results and to make information about the existence and design of clinical trials publicly available,” with the prediction that knowledge of these trials would reduce the risk of selective reporting.
Their main questions were:
- How many registered studies were eventually published in high impact journals?
- Of those studies, how many pre-registered primary outcomes were reported in the published articles?
- What was the proportion of statistically significant reported primary outcomes?
They also sought to compare general medical journals and specialty journals in terms of the proportion of registered trials and differences in primary outcomes.
Using the MEDLINE online medical database for studies and RCTs, the authors selected studies from high impact journals and for each study, established the funding sources, the number of reported outcomes, and their registration status.
Mathieu and his colleagues found that, while nearly half of the selected trials were adequately registered, 27.6% of trials had never been registered, 14% were registered only after the study was complete, and 12% were registered without any clear description of the study’s primary outcomes. They found no significant differences in reporting between general and specialty journals, though there were more adequately registered trials within general medical journals.
So does this mean that registries aren’t as useful as we may have hoped? Consider the AEA RCT registry. Though it was established less than five years ago, there are already over 1,000 registered trials on it – trials that can be accessed by readers looking to make more informed decisions regarding a study’s credibility or researchers attempting to accurately replicate a study.
If executed correctly, trial registration can improve accountability in planning and reporting, as well as safeguard against publication bias. Mathieu and his colleagues believe that the scientific community should endorse and adhere to these requirements, being cognizant of deadlines and using clarity when instructing authors on submission guidelines. They suggest quality control procedures and conclude that, with careful implementation, trial registration may ensure more robust results and unbiased reporting.
How can we inform more people of the significance of registries? Why might people be hesitant to register their studies?
You can read the whole paper here.
Reference
Mathieu S, Boutron I, Moher D, Altman DG, and Ravaud P. 2009. “Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials.” JAMA 302 (9): 977–84. doi:10.1001/jama.2009.1242.
© Center for Effective Global Action.