by Coly Elhai, Dominic Russel, Jun Wong

Introduction from BITSS: Today on the BITSS blog, Coly Elhai, Dominic Russel, and Jun Wong reflect on their Catalyst training project entitled “Transparency and Equity in Pre-Doctoral Research,” which featured a large online workshop on research transparency for pre-doctoral economics students.

Full-time pre-doctoral research experiences are now common for incoming Ph.D. students in economics and related fields. In the top fifty economics Ph.D. programs in the US, for example, around forty percent of Americans who went on the job market in 2016-17 had full-time research assistant positions prior to starting graduate school. In recent years, the growth of formal pre-doc programs at a number of universities has only furthered this trend.

Pre-docs are often responsible for key aspects of empirical research including data collection, cleaning, and analysis — yet few are trained in open science and reproducible research methods. Additionally, while pre-doctoral positions might diversify the pathway to a Ph.D., evidence suggests that those who attain these positions have demographics and educational backgrounds similar to current Ph.D. graduates.

We hoped to tackle these two challenges through the Predoctoral Research in Economics (PRE) Workshop, which we hosted with the support of BITSS and the PREDOC consortium. Our goals were to help (1) diversify the set of applicants for pre-doctoral fellowships, and ultimately economics PhDs, and (2) provide them with the tools to conduct transparent and reproducible research. 

“Pre-docs are often responsible for key aspects of empirical research including data collection, cleaning, and analysis — yet few are trained in open science and reproducible research methods”.

The online workshop attracted hundreds of individuals from around the world interested in applying for pre-doctoral positions and learning about open science. It included talks on research transparency from Fernando Hoces de la Guardia, quantitative standards from Anna Josephson, and pre-analysis plans from Jeffrey Michler, as well as panels with current pre-docs and faculty supervisors (all slides and recordings of these sessions are available here). It also included a coding exercise to help prepare participants to be competitive for pre-doc positions.

One of the biggest challenges we faced was how to scale up the workshop without losing value. We had envisioned the coding exercise as a central piece of the workshop, but when we received over 450 applications, we realized that we wouldn’t be able to involve everyone in a meaningful way. Our solution was to select 85 participants for the coding portion — with an eye toward creating a diverse and highly engaged cohort — while leaving the rest of the workshop open to all applicants. We then divided these selected participants into small groups of 6-8 and assigned each group to a current pre-doctoral researcher mentor.

The mentors were key to achieving the workshop’s goals. They met with their groups before the workshop to answer any questions and help them get their coding environments set up. During the workshop, mentors coached participants through the coding exercise. Our hope was that mentors could give participants a more individualized experience, despite the Zoom format. Participant feedback after the workshop suggests that this worked. One participant wrote, “My mentor was really good at explaining what should be done/what challenges she faced with STATA in the past and tips so we don’t have to go through the same thing.”

We were also especially happy with how our panels with current pre-doctoral researchers turned out. Participants consistently rated these panels, which focused on the panelists’ paths to pre-doc positions and their day-to-day on the job, as one of their favorite parts of the workshop. One participant told us “Receiving insight from the current predocs was extremely helpful…hearing them share the bumps they went through along the way in securing said pre-doc positions helped to humanize the experience.”

“We look forward to seeing the social sciences continue to become more inclusive, transparent, and reproducible”.

An area of potential improvement in the future is outreach. Our outreach strategy relied mainly on Twitter and an email list we compiled of economics departments in public universities and historically black colleges and universities in the US. As the number of applications we received demonstrates, we were able to reach a large number of people. While we were able to reach some students from under-represented backgrounds, many of those who found out about the workshop and applied were the same people who tend to apply to pre-doc positions already: people from elite universities with plenty of research experience. We hope that future iterations of this workshop engage in continued outreach and more direct communication with students in public universities across the U.S.

In all, we appreciated the opportunity to put together this workshop and to learn from our participants, pre-doc mentors, and speakers. We are especially grateful to our partners PREDOC and BITSS, who provided invaluable guidance and support along the way. In post-workshop surveys, respondents reported that the workshop deepened their understanding of careers in research, coding fundamentals, and tools of open science. We have also heard from several participants who have received and accepted pre-doc job offers in the fall. We’re excited to follow their progress in the coming years. PREDOC plans to continue this workshop in some form. We look forward to seeing the social sciences continue to become more inclusive, transparent, and reproducible.

---