How NOT to Interpret p-values

Your dose of BITSS humor, via xkcd.



Now Accepting Applications for Summer Institute

BITSS is pleased to announce it is now accepting applications to attend its 2015 Summer Institute.

This year’s workshop entitled “Transparency and Reproducibility Methods for Social Science Research” will be held in Berkeley, June 10-12. The intensive course will provide participants with a thorough overview of best practices for open, reproducible research, allowing them to remain in the vanguard of new scientific frontiers.

The 2014 Cohort of the BITSS Institute

The 2014 Cohort of the BITSS Institute

Topics covered include:

  • Ethics in Experimental Research
  • False-positives, P-hacking, P-curve, Power Analysis
  • Data Management & Statistical Analysis in R
  • Theory and Implementation of Pre-analysis Plans
  • Approaches to the Replication of Research
  • Meta-analyses: New Tools & Techniques
  • Next Steps in Changing Scientific Research Practices


Research Transparency Meeting with CGD

By Garret Christensen (BITSS)

Though BITSS hopes to increase research transparency across the social sciences, several of us, myself included, have a background in development economics. So we were happy to take part in a meeting last week at the Center for Global Development (CGD) in Washington, DC. In addition to BITSS and CGD, representatives from the International Initiative for Impact Evaluation (3ie), Inter-American Development Bank, InterAction, Innovations for Poverty Action Lab (IPA), Millennium Challenge Corporation (MCC), World Bank research group, United States Agency for International Development (USAID), and the US Treasury were present.

I was impressed by how much agreement there was, and how interested these large, sometimes slow-moving, organizations seemed to be, but I should probably temper my enthusiasm a bit: the people in the room were not randomly selected from their respective agencies, and even if they had been, we may still be far from actual policy changes and wider adoption. Regardless, we had a fruitful discussion about some of the roadblocks on the way to increased transparency.

Here are a few of the themes we discussed, mostly obstacles to increased transparency:


The Disturbing Influence of Flawed Research on Your Living Habits

Last year, we featured a story on our blog about the so-called cardiovascular benefits of fish oil, largely based on a seminal research study that had more to do with hearsay than with actual science. After your diet, flawed research is now trying to meddle with your sports life.

A Danish study published in the Journal of the American College of Cardiology recently made the headlines for suggesting that too much jogging could have a negative impact on life expectancy. In a recent post in the New York Times, economist Justin Wolfers carefully analyses the study and provides a brilliant response discrediting this overly-confident claim:

The researchers asked Danish runners about the speed, frequency and duration of their workouts, categorizing 878 of them as light, moderate or strenuous joggers. Ten years later, the researchers checked government records to see how many of them had died […] Happily, only 17 had. While this was good news for the surviving runners, it was bad news for the researchers, because 17 was clearly too few deaths to discern whether the risk of death was related to running intensity.

Nonetheless, the study claimed that too much jogging was associated with a higher mortality rate […] The evidentiary basis for this claim is weak. It is based on 40 people who were categorized as “strenuous joggers” — among whom only two died. That’s right: The conclusions that received so much attention were based on a grand total of two deaths among strenuous joggers. As Alex Hutchinson of Runner’s World wrote, “Thank goodness a third person didn’t die, or public health authorities would be banning jogging.”

Because the sample size was so small, this difference is not statistically significant. You may have heard the related phrase “absence of evidence does not equal evidence of absence,” and it is particularly relevant here […] In fact, the main thing the study shows is that small samples yield unreliable estimates that cannot be reliably discerned from the effects of chance.

Wolfers goes on highlighting other weaknesses in the Danish study. This new case of unreliable research finding receiving wide media coverage brings out a couple of important points that are central to our work at BITSS:


Announcing New Grants for Data Publication!

The Berkeley Initiative for Transparency in the Social Sciences (BITSS), the Center for Effective Global Action (CEGA), and the Alfred P. Sloan Foundation are pleased to announce the Sloan Grants for Data Publication.

Regardless of how transparent or rigorous a study design may be, if the study materials (datasets, code, metadata, etc.) are not made publicly available, the research community cannot verify the study’s results. This has prompted research funding institutions, BITSS and its allies to push for data publication.

Unfortunately, the incentives to publish studies materials remain limited. Consequently, few researchers have the bandwidth to take on the burdensome task of preparing study materials for public access. This results in the loss of valuable resources for the research community and the public alike.

We would like to help CEGA and BITSS affiliated researchers get through the last hurdles in finalizing replication sets by giving out grants up to $5,000 to help increase the “transparency footprint” of our community within the next year. We hope these small grants will aid in the hiring of RAs to create and finalize replication sets or support researchers own workflow in a way that increases their ability to make data and study materials available online. Please fill in this short expression of interest by March 15, 2015 to apply.

We look forward to seeing just how many datasets we can make publicly available!

Please contact BITSS Program Coordinator Alex Wais with any questions.

UC Press Launches New Open Access Publications

The University of California’s publishing house, UC Press, has announced the launch of two new publications, Collabra and Luminos. Collabra will publish academic articles across many academic disciplines including the life, environmental, social and behavioral sciences. Luminos will publish monographs across all fields of study. As the UC Press website indicates, the publications will “not only [share] the research but also the value created by the academic community.”

With low up-front APCs [Article Processing Charges], a sponsorship fund for authors unable to pay, and sharing actual revenue with editors and reviewers, Collabra builds a fair and welcoming ecosystem. (Collabra Website)

Luminos is also based on an innovative publishing model:

[It] shares the cost burden of publishing in manageable amounts across the academic community. For each title, UC Press makes a significant contribution, augmented by membership funds from supporting libraries. Authors will then be asked to secure a title publication fee to cover the remaining costs. Additional revenue from supporting libraries and print sales will help to support an author waiver fund. (UCSD Library Blog)

The publication of the two initiatives coincides with a well needed push towards greater access and openness in academic publishing and will hopefully increase the benefits of the public goods provided by researchers and their affiliated academic institutions.

For more information on either publication, contact Lorraine Weston at

Science Magazine Releases Special Issue on Digital Privacy

Yesterday, January 29th, Science Magazine released a new Special Issue entitled The End of Privacy. In line with its theme, the edition will be made available online at no cost for the first week following publication. Take this chance to look through!

For scientists, the vast amounts of data that people shed every day offer great new opportunities but new dilemmas as well. New computational techniques can identify people or trace their behavior by combining just a few snippets of data. There are ways to protect the private information hidden in big data files, but they limit what scientists can learn; a balance must be struck.

Boldly declaring “Privacy as we have known is ending and we’re only beginning to fathom the consequences,” the Special Issue deals with a host of topics related to different facets of data deposition, use, and confidentiality. Included in the publication are formal reports and new articles.


All the latest on research transparency

Here you can find information about the Berkeley Initiative for Transparency in the Social Sciences (BITSS), read and comment on opinion blog posts, learn about our annual meeting, find useful tools and resources, and contribute to the discussion by adding your voice.

Enter your email address to follow this blog and receive notifications of new posts by email.



Get every new post delivered to your Inbox.

Join 290 other followers