Data Management and Sharing, and Open Evaluation Workshop Catalyst Training Project
Emmanuel Orkoh
In recent years, there has been a growing concern about a significant proportion (between 51% and 89%) of published research that are non-reproducible, largely due to lack of rigor in data management and analysis. This phenomenon contributes to losses of millions of funds invested in research. As a result, funding agencies and journals are increasingly requiring researchers to share, archive, and plan for the management of their data to foster the validation of published analyses and results of research studies. As research and information professionals respond to these new requirements, data curation knowledge is necessary for the effective management, long-term preservation, and reuse of data. This course will provide participants with knowledge of data management, best practices throughout the research life cycle from the planning stage to when data will be shared and made available within a trustworthy repository. This will increase researchers’ impact and support for open scientific inquiry.
Transparent piloting: Understanding researcher practices and attitudes towards reporting pilot studies in psychology Catalyst Training Project
Isaac Handley-Miner
Piloting—the practice of conducting preliminary studies or trial runs to refine research designs, procedures, or instruments—is common in psychology. Yet, pilot studies are rarely reported in the literature. Publishing methods and results from pilot studies alongside the full studies these pilots precede could have many benefits, including: (1) what was learned during piloting could help other researchers better replicate, extend, and repurpose study designs; (2) pilot results could shed light on the generalizability of the full study’s findings (e.g., reveal boundary conditions); (3) the field could better assess common piloting practices and identify areas of improvement. This project will survey researchers about their piloting practices, attitudes towards reporting pilots, and perceived barriers to reporting pilots. This survey data will inform three downstream projects: a meta-assessment of current piloting reporting practices, a tool to facilitate the reporting of pilot studies, and a perspectives article discussing the merits of reporting pilot studies.
Reproducible Data Analysis in the Social Sciences Catalyst Training Project
Anne-Katherin Kleine
This project aims to bridge the gap between the demand for research adhering to reproducibility criteria and the availability of quality training. The project employs a Training of Trainers (ToT) approach to foster engagement. Specifically, experienced Master Trainers coach Junior Trainers in the principles of reproducible data analysis. Over the course of two months, Junior and Master Trainers collaborate to develop a workshop series covering data analysis with R and Quarto, version control with Git and GitHub, and the publication of reproduction packages. The workshop is designed to address the needs of master students, PhD Candidates, and post-docs within the social sciences. The materials developed will be publicly shared to promote widespread and ongoing access. This initiative is set to boost the growth of the open science community and empower dedicated individuals to actively promote reproducibility in the social sciences.
REPKIT: Automated reproducibility tools for Stata (REPRUN) Catalyst Training Project
Benjamin Daniels
This project will expand the existing (open beta) IEDOREP Stata command to automate checking for reproducibility issues in do-files, with the goal of inclusion in the new REPKIT Stata package to be published by DIME (World Bank) and presented at the 2024 Stata Conference. Building on the World Bank’s new program requiring reproducibility packages for its Policy Research Working Paper series, IEDOREP helps integrate automated reproducibility checks into routine workflows. This project will enhance IEDOREP’s capabilities for complex code, produce expanded debugging options, and develop publication and dissemination for the tool. Deliverables include an enhanced reproducibility tools package, documentation, and educational materials to promote adoption. By translating lessons learned from manual reproducibility checking into a widely accessible automated tool, this project will advance capacity for reproducible research across the social sciences.
Research Transparency Overview and Guidelines (in Arabic) Catalyst Training Project
Heba Abou-El-Sood
One of the fundamental challenges towards transparency and reproducibility of research in the MENA region is a cultural barrier. Information regarding data input for research and outcomes (not yet publishable) have been always viewed in the lenses of secrecy, confidentiality, and competitive advantage. The research community in the MENA region needs to have content in their own language (Arabic), to be culturally embedded, and not superficially imposed or borrowed from other cultural backgrounds. This project aims to achieve this end goal by providing culturally adapted concepts, guidelines, and translation of material to appeal to the research community and facilitate/motivate adoption and execution.
Promoting Research Ethics through Research Transparency Techniques Catalyst Training Project
Mercyline Kamande
Research transparency is an approach that promotes open practices including registering studies, sharing study data, and publicly reporting research findings. By pre-registering research methodology and intended analyses, researchers tend to be more accountable by ensuring that they report the actual findings of the research, which in turn increases research credibility. By openly sharing data, it is possible to replicate studies, which allows the extension of research work.
This project builds on a previous project in 2018 whose objective was to share knowledge of research transparency techniques and their role in addressing the file drawer problem and publication bias. The project was implemented in a series of four workshops in two leading universities in Rwanda. The current study seeks to extend this study to Mount Kenya University in Kenya and incorporate the research transparency and ethics into an existing staff development program currently being rolled out at the university.
Improving Research Transparency, Reproducibility and Open Science Research in Burkina Faso Catalyst Training ProjectEconomicsPolitical SciencePublic Policy
Idrissa Ouedraogo Alassane Koulibaly
This training aims to contribute to empowering the next generation with the appropriate tools and the road map used to conduct transparent, reproducible, and open research. We will train master’s students, Ph.D. candidates, postdocs, junior faculty, and researchers in Economics, Political Sciences, and Public policy in methods for transparent, open, and reproducible research. The training will be held at the Thomas Sankara University and the Centre d’Etudes, de Documentation et de Recherche Economiques et Sociales (CEDRES) in Ouagadougou, Burkina Faso. Topics will include the theory of Transparency, Reproducibility and Open Science Research, pre-registration and pre-analysis, data and code sharing, data de-identification, data mining, fishing, p-hacking, selective reporting outcomes, and multiple inference correction, replication, and dynamic documents and reproducible workflows. We will tailor and promote the materials and tools used since the RT2 2017 trainings and other resources developed by BITSS.
Project outputs:
Brazilian Open Science Training: principles and tools Catalyst Training ProjectPolitical Science
Amanda Domingos Matheus Ferreira Marcus Torres Rodrigo Lins
Search results that cannot be reproduced are not trustworthy. Despite recent moves towards a more open and transparent social science, Brazil faces challenges both in the belief in the importance of principles and in the implementation of transparency and reproducibility practices. We argue that the way to change this paradigm is through the training of young researchers. To do so, we propose conducting training workshops for undergraduate and graduate students. The project will provide an overview of principles, tools, and practices for building more transparent social research and will allow the exchange of feedback and receiving support in building a reproducible workflow in the medium and long term.
Project outputs:
Making research more tangible to study participants and communities: templates for dissemination of study findings Catalyst Training ProjectPublic Health
Larissa Da Silva Daniel Umpierre
This project aims to amplify open science and research transparency by fostering the communication of research findings for study participants and the overall community. Although several studies in public health rely on the participation of human individuals who give their time and share their data, many times their knowledge about research findings or on their own individual data/results is unwarranted. Therefore, we will promote discussions with Brazilian stakeholders on how to disseminate research findings to study participants and the overall public. Thereafter, dissemination packages (templates in Portuguese, English, and Spanish) will be generated to facilitate results dissemination in two broad disciplines (clinical sciences and epidemiology). By collective construction and strong advocacy, we expect this project may foster as much attention on ethical issues as exist at the beginning of a research study (which are associated with regulatory requirements needed to study approval), valuing public engagement and transparent research.
Doing Open Science: Reproducibility, Transparency, and Ethics in Social Sciences Catalyst Training ProjectEconomics
Ignacio Sarmiento-Barbieri
Social science should be widely available for critical interpretation and contribution to foster open societies. The “Haciendo Ciencia Abierta” workshop aims to strengthen research practices by providing the Spanish-speaking community with approaches and free, open-source tools for transparent and reproducible social sciences. The 3-day synchronous remote workshop will take place at the Economics Department at the University of Los Andes in Bogotá. The workshop will provide an overview and hands-on experience on using the Declare Design R package for experimental research, GitHub for version control, and data sharing and management strategies.
Project outputs:
Payments and Governance Research Program reproducible workflow onboarding Catalyst Training ProjectPolitical Science
Thomas Brailey
The Payments and Governance Research Program conducts large-scale Randomized Control Trials (RCTs) and provides independent, rigorous, and scientific evidence on India’s social policies. We are committed to going beyond transparency on the front end and want to ensure that our entire research pipeline—from power calculations to data analysis—is transparent and reproducible. To reach this standard, PGRP will transition our entire team towards a version-controlled research pipeline with GitHub. The training needs for such a transition are substantial. The Catalyst grant will facilitate the development of onboarding workshops and resources tailored for PGRP and eventually the whole of JPAL SA. We believe our team’s size and geographical spread are part of the opportunity. We can leverage this grant to diffuse good research norms across our global network of collaborators. We are excited to make this crucial investment in a transparent and reproducible workflow and a Catalyst grant will expedite that process.
Find all training materials developed as part of this project here.
Advancing research transparency, reproducibility, and ethical research at Busara Center and partner organizations. Catalyst Training ProjectEconomics
Joel Mumo Kelvin Kihindas
Reproducibility is important for research because it ensures transparency and clarity on what was exactly done by researchers. This project aims to support Busara Center in developing or improving the existing open science protocols and will involve the development of training materials and protocols as well as holding training sessions with staff members to guide them on how to create reproducible project documents and use Open Science Framework. We will also spend time curating and making some of our datasets open to the public.
Research Transparency and Reproducibility: A Complementary Module for All Catalyst Training ProjectPublic Health
Shaon Lahiri
Engagement with Research Transparency and Reproducibility can be challenging in a classroom, where students are more likely to worry about an upcoming quiz rather than a system of perverse incentives and questionable practices that characterize academic research. Creating a full-length course on the subject is wonderful where possible, but may not be prioritized in graduate student curricula, particularly if curriculum committees are not enthusiastic about the subject. This project aims to develop a complementary module that can be drawn upon in existing social and behavioral science methods graduate courses. It emphasizes four main components: a) history and ethics of RT2, b) reproducibility crises and the scope of the problem, c) systemic and behavioral solutions, and d) hands-on activities and applications. By emphasizing practical application, this project aims to ignite positive emotional experiences with RT2, which will extend from pedagogy to practice.
Teaching materials: https://osf.io/qw52b/
Starting from Scratch: Training Teachers and Students of Tribhuvan University, Nepal on Fundamentals of Research Ethics and Transparency Catalyst Training ProjectEconomics
Bishal Chalise
At Tribhuvan University, Nepal, graduate, and undergraduate students conduct a mini-research project or write a thesis in the last year of study. They receive training in research methods and related coursework. However, the course curricula do not have research ethics and transparency component. As a result, students are missing out on learning the issue, techniques, and objective of research ethics. Hence, proposed activities combine the development and piloting of training resources and orient teachers and students about the basics of research ethics. The specific activities include developing training and workshop materials on issues of data integrity, research misconduct, informed consent, conflict of interest, etc., conducting training sessions for final year graduate and undergraduate students, and workshop for lecturers teaching research methods and supervising students’ research.
Pre-Specification and Reproducibility – Applications for Academic and Private Sector Research Catalyst Training Project
Ann Furbush
Open science best practices are not universally adopted in the social sciences and are often not included in graduate-level curricula, despite the potential for pre-specification and reproducibility practices to enhance credibility in both academic and private-sector research. This project seeks to introduce graduate students to the many avenues through which open science concepts can improve transparency and reproducibility in both academic and professional settings. Through a series of workshops, I will facilitate discussion with my peers about how to apply open science principles to bolster credibility in current and future research. Specifically, I will discuss the benefits of pre-specifying both academic and professional analyses and introduce tools to enhance research reproducibility.
Project outputs:
Open Science in Agricultural Economics: Promoting Replicability, Reproducibility, and Transparency in Research Agricultural EconomicsCatalyst Training ProjectEconomics
Anna Josephson Jeffrey Michler Talip Kilic
Agricultural and applied economists (AAE) play a key role in delivering information and insight that contributes to policy in agriculture, the environment, and development. The success of these policies depends on the quality and transparency of the underlying research. However, standard MSc and PhD curricula in the field do not cover many of the best practices related to open science.
With this project, we will develop materials, which will be used as part of a virtual workshop on transparency, replicability, and reproducibility in AAE research. The workshop will provide participants with an overview of the best practices and applied tools for open science. Materials will be made available via Open Science Framework and GitHub.
Public Materials:
- Lectures: https://osf.io/3vtng/, available under Files > BITSS: Open Science in Agricultural Economics
- LSMS-ISA Handbook on Data Processing: https://github.com/jdavidm/lsms-isa_data_handbook
Filling in the gaps: a workshop to promote, improve, and implement transparency and reproducibility tools in health economics research Catalyst Training ProjectEconomicsHealth Economics
Andrea Salas Ortiz
This project aims to improve open science (OS) practices in health economics research (HER) through the promotion of transparency and reproducibility (TR) tools. Although HER has a huge impact on the policy-making process, transparent and reproducible research has not been entirely fulfilled. The practice of OS standards within the community has been taken for granted. Even though many tools for OS research are available, barriers to their implementation still exist.
Catalyst Andrea Salas Ortiz organized a transparency and reproducibility workshop for health economics researchers at the University of York on Sep. 22-23, 2021. This workshop’s goals were to understand the perceived norms within the community and identify how interests could be harmonized, and operations aligned to decrease the probability of reluctance to improve transparency and to identify the leadership role each researcher should play in adopting transparency tools and reproducibility standards. All materials can be found here.
Improving research transparency and visibility of null findings in observational studies of secondary data Catalyst Training ProjectDemographyInterdisciplinary
Jordan Weiss
Publication bias is well documented in the social sciences. Only a fraction of scholarship reporting null results is made accessible to the public. One way to increase the visibility of this work is to post a pre-analysis plan (PAP) which has the added benefit of improving research transparency. However, despite the rising importance of PAPs, the majority of available resources for PAPs in the social sciences are dedicated to experimental studies which do not necessarily translate well to observational studies of secondary data. The objectives of this project are to (1) improve research transparency and (2) increase the visibility of studies that report null findings by developing a framework for PAPs that is specifically designed for observational studies of secondary data.
Development of Research Transparency and Reproducibility Curriculum and Training Modules for Debre Markos University Catalyst Training ProjectPublic Health
Getachew Kassa
Research Transparency and Reproducibility (RT2) curriculum and training manuals are essential to improve knowledge and skill towards RT2 like research ethics, scientific misconduct, pre-registration, and pre-analysis plans, computational reproducibility and approaches to replication, and data management. However, the lack of standard curriculums and training manuals in the areas of RT2 has been the main challenge, particularly in academic institutions located in low-and-middle-income countries.
This project, therefore, plans to develop an RT2 curriculum and training manual for Debre Markos University, Ethiopia. The project will include rapid technical review and need assessment of the RT2 curriculum and training manual content. The documents will be prepared in accordance with global standards using available books and researches on RT2. Developed materials will guide, govern, and improve research activities and practice in Ethiopian universities.
Open Science in Language Research Catalyst Training ProjectLinguisticsPsychology
Esra Ataman Bilal Kirkici Ozan Can Çağlar
Esra Ataman, Bilal Kirkici, and Ozan Can Çağlar developed a module titled ‘Open Science in Language Research’ and incorporated it into the graduate course “Quantitative Research Methods and Ethics in Language Research.” The goal of the module was to raise awareness in language researchers, who are underserved compared to other disciplines, to become more mindful consumers of scientific output and introduce a toolbox to facilitate reproducible, transparent, and rigorous scientific practice. Find all materials here.
Transparency and Equity in Pre-Doctoral Research Catalyst Training ProjectEconomics
Jun Wong Dominic Russel Coly Elhai
Pre-doctoral research positions are becoming increasingly common in economics and related fields. While these pre-docs are often responsible for data collection, cleaning, and analysis, there are no open science initiatives geared toward those who will fill these positions. Furthermore, pre-doctoral researchers in economics tend to have similar backgrounds in terms of demographics and undergraduate institutions.
This workshop aimed to diversify the set of applicants to pre-doctoral fellowships and provide them with the tools to conduct transparent and reproducible research. To do so, the project invited economics students, particularly those from underrepresented backgrounds, interested in pursuing graduate studies to a workshop. This workshop featured panels of current pre-doctoral researchers discussing their experiences, collaborative coding practice in small groups, as well as presentations on open science from other BITSS Catalysts. Slides, coding materials, and recordings of the workshop are available here.
New Branches of the Student Initiative for Open Science Catalyst Training ProjectPsychology
Maike Dahrendorf
The Student Initiative for Open Science (SIOS) organizes events, workshops, and discussions to promote transparent and open research among students. As SIOS is based in Amsterdam, it has not been possible to accommodate interested students from other universities. The proposed project aims to encourage and enable interested students to start a new branch of SIOS at their own university. The development of a Step-by-Step guide, promotional materials, and guest lectures through the Open Science communities in the Netherlands will help grow the awareness of SIOS among students. By offering materials and a (virtual) workshop, the initial establishment of a new SIOS branch will be made as easy as possible. By creating more student-led initiatives for transparent and open science, students can better learn from each other and learn how to conduct and produce high-quality, reliable, and replicable research already from the beginning of their career.
Developing Content and Training for Transparent, Reproducible, and Ethical Research in Brazil Catalyst Training ProjectEconomics
Raquel Guimaraes
The objective of this project is to promote the dissemination of methods and practices for the transparency and reproducibility of research in Social Sciences at the Federal University of Parana, and also to provide materials for training in Brazilian Portuguese language. The project is structured according to the book by Christensen et al. (2019), and the activities will be divided into four parts: Introduction and Motivation, Problems, Solutions, and Practices. In a series of workshops to be held during the year 2021, all the aspects of open science will be explored with students, faculty, and staff in a hands-on and applied fashion.
Advancing Transparency and Reproducibility in Constrained Contexts in Public Health Catalyst Training ProjectPublic Health
Ahwaz Akhtar
Public health practitioners frequently find themselves in a bind: faced with high volumes of data, frequent demands for non-standard analyses, and quick turnaround times, even the most rigorous and transparent workflows are not safe from analysis breakdown and miscommunication, something seen all-too-frequently during the COVID-19 crisis. Therefore, there is a need to make workflows resistant to such “short-circuiting” with fail-safe checks that strike the right balance between the need for speed and ensuring transparency and reproducibility.
In this project, we will try to deal with thorny questions e.g. how to maintain privacy when managing contact-tracing data, ensuring rigor even when timelines are short and how to communicate results to non-literate in when trust between researchers and the broader community is scarce. This project will be executed at the Global Health Directorate of Indus Health Network located in Karachi, Pakistan.
PhD Toolkit on Transparent, Open, and Reproducible Research Catalyst Training ProjectInterdisciplinary
Ada Gonzalez-Torres
This project will train PhD students in Economics, Political Science, and other empirical social sciences in methods for transparent, open and reproducible research. There is no course offering such training at the EUI currently and this project will provide an opportunity for PhD students to develop an essential toolkit on research transparency and reproducibility. Topics will include pre-registration and pre-analysis plans and, ethical issues regarding the protection of human subjects in experimental work, data de-identification, data mining, fishing, p-hacking, selective reporting outcomes and multiple inference correction, meta-analysis, and reproducible workflows.
Policy Researcher Reproducibility and Transparency in Colombia Catalyst Training ProjectPublic Policy
Oscar Barriga-Cabanillas Aleksandr Michuda
Colombian law dictates that information or data gathered during public policy evaluations, after being anonymized, must be publicly available. However, this is usually not the case in practice. Methods exist to rectify this situation, but there is low adoption of these methods. This project involves two workshops in Bogotá and Cartagena targeted at policymakers, a consultancy firm that works with the government, and graduate students, focusing on methods and tools to make their work more transparent and easier to reproduce. The training will help them develop smoother workflows, as well as foster more transparency within their organizations and with their citizenry.
Events:
April 17, 2018: Workshop in Bogotá
A University Goes Open – An Interdisciplinary Open Science Curriculum for all Graduate Schools Catalyst Training ProjectInterdisciplinary
Felix Schönbrodt
This project will create reusable course materials for open science and reproducibility topics in a modular structure. The basic material will be the same across disciplines. However, discipline-specific add-ons are planned with specific reading lists and homework assignments. A key part of the project is to use existing infrastructures at the host university, such as the LMU Graduate Center and the LMU Open Science Center, to disseminate and promote the material to over 3,000 PhD students and the program coordinators of more than 30 graduate programs at Germany’s largest university.
Transparency in Economic Research – An Opportunity for Ecuador Catalyst Training ProjectEconomics
Jan H. Höffler
The project will expose bachelors and masters students to open science, a topic that has not yet been included in curricula at Universidad de Cuenca. With small practical tasks, participants will be guided through a hands-on experience of how to make empirical research transparent, why it is so important, the frustration that accompanies engaging with non-transparent research, the challenges involved in making one’s own research transparent, how manageable it can be when following established guidelines, and the rewards of conducting reproducible research. Course materials in Spanish as well as data from Latin America about the transparency of empirical literature will be made available to the general public through the Open Science Framework and the ReplicationWiki.
Incorporating Reproducibility and Transparency in an Undergraduate Economics Course Catalyst Training ProjectEconomics
Jeffrey Naecker
This project developed lectures and hands-on materials to teach key reproducibility, replication, and transparency concepts for an undergraduate economics audience. The goal is to translate existing BITSS materials for students who have no research experience, setting the stage for them to develop good research habits as early as possible. The output includes 4.5 hours of lecture materials and three problem sets/lab activities. Materials will be field-tested in Jeffrey Naecker‘s experimental economics course at Wesleyan University.
Curricular Materials can be found on the OSF here.
Improving Reproducibility and Transparency Practices in Public Health Research in Mexico: A Working Group and Short Course Catalyst Training ProjectPublic Health
Sergio Bautista Private: Nerissa Nance Gina La Hera Fuentes Lily Alexander
The Catalysts developed Standard Operating Procedures and established an institutional hub with resources for research transparency and reproducibility at the National Institute of Public Health (INSP) in Mexico. The group also developed modular coursework for MSc in Public Health students, incorporating open science tools and practices in research design, data collection and analysis, and dissemination.
Introducing Transparency, Reproducibility and Open Science Research in South Africa, Ivory Coast and Burkina Faso Catalyst Training ProjectEconomicsPolitical SciencePublic Policy
Dief Reagen Nochi Faha Idrissa Ouedraogo Tafadzwa Maramura
This training aims to contribute to empowering the next generation of African researchers with the appropriate tools to conduct transparent research. The Catalysts will introduce the concepts of transparency, reproducibility, and Open science at three universities in Sub-Saharan Africa. Topics will include pre-registration and pre-analysis plans, data sharing, code sharing, replication, and dynamic documents. The audience will consists of masters students, PhD candidates, faculty, and researchers in Economics, Political Science, and Public Policy.
AP Psychology Workshop: Embedding Transparent Practices in Psychological Research for High School Students Catalyst Training ProjectPsychology
Shane McCarty
As non-traditional researchers, high school students represent a potential Trojan horse for changing scientific practices and norms. This 4-week project-based learning course aims to complement the existing Advanced Placement (AP) psychology curriculum for high school students, which does not currently include openness or transparency in research. The Catalyst will collaborate with an AP Psychology teacher or PhD student to train over 125 Yorktown High School students to conduct their own psychological experiments using transparent, open, and reproducible research practices, applying the scientific method to answer research questions regarding student character development and school climate, using the OSF for pre-registration, collaboration, and data sharing.
Practical Tools for an Open Science Project, from Literature Review to Accessible Publication Catalyst Training ProjectCognitive ScienceLinguisticsPsychology
Sho Tsuji Christina Bergmann
This project will develop a modularized, intensive workshop that takes undergraduate and graduate students in cognitive science, experimental psychology, and linguistics through a full research cycle (from literature search and hypothesis generation to publication) with a focus on reproducibility. The 4-day workshop will be spread over three thematic blocks: (1) Literature review and meta-analysis; (2) Experimental planning and pre-registration; and (3) Data management and accessible publication. There will also be practical sessions wherein participants will work through tailored examples. The content will be organized into 8 half-day modules, which makes the length and scope easy to adapt to shorter or more focused formats.
Research Transparency Workshop Catalyst Training ProjectInterdisciplinary
Zacharie Dimbuene
This workshop will familiarize social science researchers with research transparency in both teaching practices and research. Workshop activities will include (1) organizing file structures, (2) creating understandable variable labels and value codes, as well as connecting variables to survey instruments through consistent labels and codebook creation, (3) version control of code and data, and (4) creating and maintaining documentation files about surveys and data, as well as data cleaning steps. An Open Discussion will also be organized, bringing together researchers from various disciplines. Topics to be addressed include, pre-registration, pre-analysis plans, reporting guidelines, sharing data and code, and other practices (e.g., Open Science Framework) that enable reproducible research.
PhD Level Course: Transparent, Open, and Reproducible Policy Research Catalyst Training ProjectPublic Policy
Sean Grant
This course aims to train graduate students at the Pardee RAND Graduate School in Policy Analysis and researchers at the RAND Corporation in the essential theories and methods of transparent, open, and reproducible research. This elective can be taken by graduate students with an analytic concentration in “Social and Behavioral Science” as well as RAND staff eligible to audit Pardee courses. After introductory sessions on the research transparency and open science movement, enrolled students will discuss and learn the essentials of the open science toolkit, namely: pre-registration of study methods and analysis plans; reproducible workflows; open research materials, code, data, and access; reporting guidelines for preparing research manuscripts; and replication.
Reproducible Research Computational Tools for the Next Generation of Social, Behavioral and Policy Scholars Catalyst Training ProjectInterdisciplinary
Jose M. Magallanes
In developing countries, accepted applicants to pursue PhD studies in highly-developed countries in the Social, Behavioral and Policy (SBP) fields will face several challenges. It will be particularly challenging to develop abilities to carry out research in a more transparent way, considering that this topic may not be present in the curriculum. There is enough time for them to learn the basic tools of reproducibility before they leave their countries, so as to encourage their adoption of transparent practices.
To master these tools, this workshop motivates and trains accepted applicants who are about to start their SBP PhD studies. The content will consider the background of SBP students in Latin America, and offer several workshops in Peru. The workshop will guide the students into the steps to ensure their research in shareable and replicable via the use of tools such as LaTeX, R, RStudio, Zotero, GitHub/AWS, Sharelatex/Overleaf, and Docker. This project will also provide curricular material in Spanish, which can be used for future activities in the region. It is also a great opportunity to involve universities and government agencies funding international higher education; which may increase the interest in the introduction of these topics in the higher education SBP curriculum.
Southeast Asian Social Sciences Workshops for Transparency and Reproducibility Catalyst Training ProjectInterdisciplinary
Ben Marwick
Reproducibility, transparency and openness of research has received a lot of attention and resources in Western countries, but the global South and East have had fewer opportunities to participate in these important developments in many research areas. This project will bring to five mainland Southeast Asian institutions with whom the Catalyst has collaborated a two-day workshop to motivate and update social scientists in the global East about current issues relating to the reproducibility, transparency and openness of research, and provide them with skills to improve their research along these lines.
The Catalyst will draw on BITSS materials and his own publications and presentations using the R programming language to teach general social science skills to enhance the reproducibility, transparency and openness of research. The workshops will be optimized for low-bandwidth internet, drawing on previous experience teaching computational methods in these locations. Workshop summaries will be generated in local languages and freely distributed online and in hard copy to more easily transmit the contents to students and other scholars.
Knowledge Sharing on Best Practices in Research Transparency and Reproducibility to Address the File Drawer Problem and Publication Bias in Rwanda Catalyst Training ProjectInternational Development
Mercyline Kamande
This project seeks to share knowledge and skills to motivate open, transparent, and reproducible research in order to address the file drawer problem and publication bias in Rwanda, as well as to improve the usability of research findings for policy-making. The Catalyst will first conduct exploratory research on the extent of the file drawer and publication bias problems in Rwanda. The project will then introduce to scholars in the areas of economics, business studies, social science and public health the tools that exist to address the problem, as well as the benefits of using best practices in open, transparent, and reproducible research. These practices will include pre-registration and pre-analysis plans; sharing materials, data, and code using platforms like the Open Science Framework; and replication studies and meta-analysis.
Research Transparency and Reproducibility workshops will be held at two of the leading universities in Rwanda – the College of Business and Economics at University of Rwanda and Mount Kenya University Rwanda. Independent researchers from the Economic Policy Research Network will also be included. To ensure learning, weekly peer coaching meetings will be conducted by IERD to extend the peer learning experience realized by those enrolled in an online course currently in progress. The project will also include policy dialogue sessions with other stakeholders including Rwandan policymakers.
Research Transparency and Reproducibility in the Social Sciences and Impact Evaluation Workshop Catalyst Training ProjectImpact Evaluation
Elise Wang Sonne
In order to raise the quality of academic manuscripts written by Niger’s graduate students to allow them to compete in the demanding arena of research, the Catalyst has partnered with the UC Berkeley OASIS Initiative to organize a three-day intensive bootcamp on research transparency and reproducibility in social science research and impact evaluation.
The aim of this activity is to (i) raise the awareness of Masters and PhD students in public health, sociology, psychology, political sciences, economics, and statistics of the Abdou Moumouni University in Niamey (Niger) of common academic research misconduct issues and present solutions to avoid and address them; and (ii) train researchers working for the UC Berkeley-OASIS Initiative in Niamey on data management and organization, developing pre-analysis plans, pre-registering studies, and dynamic documents, which will be extremely useful for their day-to-day tasks pertaining to ongoing impact evaluation projects.
Fostering Transparency in Government Institutions and Higher Education: A Research and Teaching Initiative Catalyst Training ProjectInterdisciplinary
Nicole Janz Dalson Figueiredo
Locations: University of Nottingham, UK; Recife, Brazil; Brasilia, Brazil
We find research findings resulting from data that is not publicly accessible to not be credible. Similarly, governments withholding administrative information should not be trusted. We argue that the lack of government and research transparency are connected, and can be tackled in by offering training on reproducibility. This project aims to foster transparency in scholarly research and in government institutions. In particular, we will conduct educational workshops that will leverage insights that have been used to increase governmental and research transparency in the UK to improve transparency in Brazil. Our target groups are 100 undergraduate and graduate students, 20 scholars, and 20 bureaucrats. The project will strengthen research skills and transparency norms that can contribute to scientific innovation, development, and social welfare.
The first workshop was part of a day-long conference: “The Gold Standard of Reproducible Research” at the University of Nottingham on March 9, 2017.
Best Practices of Openness for African Researchers and Research Transparency Workshops at Three Social Science Conferences Catalyst Training ProjectInterdisciplinary
Dief Reagen Nochi Faha Elise Wang Sonne
Locations: LSE-Africa Summit, London School of Economics, London, UK; Population Association of America, Chicago, IL; University of Dschang, Cameroon; UNU-MERIT, Maastricht, Netherlands
This project will communicate best practices for openness and reproducibility in research. We will hold a workshop for African researchers at the University of Dschang in Cameroon, focusing on sensitizing researchers to the necessity of avoiding academic research misconduct such as p-hacking, publication bias, and failure to replicate, but also on data management practices in Stata. A series of workshops will also be held for public policy graduate students, demographers, sociologists, economists, and public health professionals in the Netherlands, USA, and the UK. In addition to sensitizing researchers to the necessity of avoiding academic research misconduct and data management practices, these workshops will also include trainings on Github, the Open Science Framework, Project TIER, and Dynamic Documents using StatTag and Markdoc.
The first workshop “Research Transparency and Reproducibility in the Social Sciences” will be held on March 31, 2017 at the 2017 LSE Africa Summit Research Conference in London.
Improving transparency of complex interventions through the facilitation of process evaluation training Behavior ChangeCatalyst Training Project
Elaine Toomey
Locations: National University of Ireland, Galway (NUIG), Health Behaviour Change Research Group (HBCRG)
Process evaluation is a way of investigating how well an intervention, programme or treatment was implemented as intended. It is crucial for facilitating transparency in the development, conduct and reporting of interventions in numerous research fields, including psychology, social science and public health, as it helps to increase confidence that changes in study outcomes are due to the influence of the intervention being investigated, and not due to differences or variability in the implementation of the intervention. It is particularly important within complex interventions (interventions with several interacting components such as multiple providers or intervention sites) due to the number of components that can be implemented variably and thus influence outcomes separately. This increases scientific confidence in the results of the intervention, enhances the replication and implementation of effective interventions, facilitates the refinement or de-implementation of ineffective interventions and overall serves to reduce research waste. However, at present specific process evaluation training is not available in Ireland which represents a significant barrier to the transparency of Irish intervention research and the implementation of best quality evidence into practice.
At present, the gold-standard for process evaluation training in Europe is run by the Centre for Development and Evaluation of Complex Interventions for Public Health Improvement (DECIPHer) in the UK, an International Centre of Excellence. This project aims to facilitate world-class DECIPHer process evaluation training in Ireland to improve the transparency of complex interventions in psychology, public health or social science settings and overall enhance the quality and impact of this research. Subsequent dissemination of the workshop proceedings and materials will also promote further understanding of this work amongst the wider BITSS community.
This project is supported in part by the John Templeton Foundation.
Research Transparency in the Social Sciences Workshop, Second Edition Catalyst Training ProjectInterdisciplinary
Zacharie Dimbuene
Location: University of Kinshasa (The Democratic Republic of the Congo)
Research Transparency is gaining attention in the scientific community around the world, including the United States, European countries, and Anglophone countries in sub-Saharan Africa; yet the concept is quite a “new world” in Francophone Africa. In my efforts to advance the movement in Francophone Africa, I successfully delivered the first Research Transparency Workshop at the University of Kinshasa (Democratic Republic of the Congo). This project is intended to sustain previous efforts to set up “Research Transparency in the Social Sciences” as a culture in the next generation of social scientists in Francophone Africa.
I will offer a training workshop for 60 graduate students, research staff, and professors at the University of Kinshasa to promote best practices for reproducible research with concrete guidance about how to make materials understandable for publication. The activities to be addressed during the workshop will include (1) organizing file structure; (2) creating understandable variable labels and value codes, as well as connecting variables to survey instruments through consistent labels and codebook creation; (3) version control of code and data; and (4) creating and maintaining documentation files about the survey and data, as well as data cleaning steps.
This workshop is supported in part by the John Templeton Foundation.
Introducing open science to researchers and policymakers Catalyst Training ProjectInterdisciplinary
Arnaud Vaganay
Locations: New Delhi, India; Lausanne, Switzerland; London, England; Delft, Netherlands
This project involved a number of workshops, seminars, a summer school, and public testimony at the UK Parliamentary Office of Science and Technology inquiry on research integrity. Events reached over 250 students, researchers, faculty, librarians, and policymakers. Arnaud also served as a faculty member at three Research Transparency and Reproducibility Trainings (RT2) in Berkeley, London, and Amsterdam.
Presntations and other materials developed for this project aimed to (i) highlight the benefits of open science to social scientists and (ii) provide practical recommendations to enhance the transparency and reproducibility of social research. Materials can be found on the following event pages, as well as on RT2 event pages.
Events:
April 28 – May 1, 2017: Transparency and Reproducibility of Impact Evaluation: A Turning Point for the Evaluation Community, 3ie, New Delhi, India
September 25-29, 2017: Open Science in Practice Short Course, École polytechnique fédérale de Lausanne (EPFL), Lausanne, Switzerland
January 30, 2018 – Geography and Environment Research Transparency and Reproducibility Training, London School of Economics, London, England
April 4, 2018 – Making Research Reproducible: A Call for Action, TU Delft, Delft, Netherlands
Ongoing – Parliamentary Inquiry into Research Integrity at the House of Commons of the UK, London, England
Disseminating Research Transparency in Perú, Bolivia, and Chile Catalyst Training ProjectInterdisciplinary
Fernando Hoces
Locations: Peru; Bolivia; Chile (final locations TBA)
The goal of this project is to bring to the attention of the academic communities in Peru, Bolivia, and Chile the recent developments in science regarding transparency and openness. This will be done in a two-fold format. First, a seminar-style talk will present the key issues (the reproducibility crisis, specification searching, and publication bias) and its solutions (pre-registrations, the TOP guidelines, and other tools for reproducible research). Second, a day-long workshop aimed at students will present the main tools for reproducible research (including R, Dynamic Documents, Git, and OSF). Increasing the scope of the research transparency community to this region can have additional benefits as it would bring highly talented researchers and students to elements of frontier research that are usually undisclosed in published papers.
Introducing the Transparent and Reproducible Research Paradigm in Ugandan Higher Institutions of Learning Catalyst Training ProjectInterdisciplinary
Jayne Tusiime Saint Kizito Omala
Locations: Universities across Uganda (final locations TBA)
The concept of transparent and reproducible research is not known, nor appreciated, by researchers in Uganda and many other developing countries. There is usually delayed adoption of new knowledge and technologies in developing countries because of the slower flow of information in these regions. The concept of transparent and reproducible research is still relatively new even in the developed world and almost unknown in the developing world. As Catalysts of this paradigm shift, we wish to introduce this concept in Uganda. Groundbreaking research, especially in health has been done in Uganda and much more research continues to be done. However, the practice of transparent and reproducible is non-existent. We thus propose to start in Uganda a project to train 500 established and upcoming researchers in conducting transparent and reproducible research.
Our target population will be graduate and undergraduate students from 30 universities in Uganda, as well as faculty of both public and private universities that train students in research disciplines where theses, dissertations or journal article publications are required for either degree award or promotion. The objective of the project will be to sensitize and create awareness about conducting transparent and reproducible research. Specifically, we shall conduct ten regional workshops across Uganda. Following this introduction, we plan to design course modules to incorporate into existing academic programs at the participating universities.
Development of a Graduate Public Health Online Course in Research Integrity, Transparency, and Reproducibility Catalyst Training ProjectEpidemiology
Dennis M. Gorman
Location: Texas A&M University, USA
There is now a growing recognition within the scientific community that flexibility in study design, data analysis, and the reporting of research findings is increasingly leading to the publication of misleading results that capitalize on chance and cannot be replicated. It has been suggested that the use of such practices, if not made apparent in a manuscript describing the results of a study, is a form of research misconduct. There is little doubt that the widespread use of such practices undermines the integrity of a scientific field as they produce a body of non-reproducible, random findings. Both epidemiology and general public health are among the fields of research in which questions have been raised about research integrity, transparency, and reproducibility.
This course will examine various threats to the integrity of research, the professional and organizational factors that produce these threats, and the solutions that have been suggested to improve research quality (such as registered reports, open data, and team of rivals). Upon completion of the course, students should have the ability to differentiate research that is conducted with integrity and capable of producing valid and reproducible findings from research that is conducted without integrity and produces chance results that are trivial and non-reproducible. Students should also have the ability to incorporate practices into their own research that will increase its transparency and ensure it is conducted with integrity.
Creating Pedagogical Materials to Enhance Research Transparency at UCSD Catalyst Training ProjectInterdisciplinary
Scott Desposato Craig McIntosh
This project involved creating a teaching module by the Policy Design and Evaluation Lab (PDEL) at UC San Diego (UCSD). The module is intended for use in graduate-level social science methodology courses—including those in political science, economics, sociology, and psychology—at UCSD and beyond. It covers the main causes of the credibility crisis in social science, as well as practical solutions for individual researchers to address these issues in their own work.
All materials are available online at PDEL’s GitHub page.