By Esra Ataman, Ozan Can Çağlar, and Bilal Kırkıcı

Introduction from BITSS: In this post, Esra Ataman, Bilal Kirkici, and Ozan Can Çağlar reflect on their project supported by one of our 2020 Catalyst grants, which involved developing an open science module for their graduate course “Quantitative Research Methods and Ethics in Language Research.” Read a Turkish translation of this post on the Open Science Community Turkey Blog.

The Open Science movement has spurred discussions about making research more open, transparent, reproducible, replicable, and rigorous (see here for relevant readings). Even though these discussions have at first pertained to some disciplines in particular (e.g., psychology, cancer biology, economics, etc.), they are becoming increasingly widespread in many others, including language research. Because language research is closely intertwined with many other fields such as philosophy, psychology, sociology, and biology, as language researchers, we’ve been able to learn a great deal from them. For instance, some replication calls and replication attempts have been made in some sub-disciplines (e.g., phonetics, sentence processing). Also, many articles have emerged focusing on how certain practices like pre-registration could be applied to different sub-disciplines of language research such as infant research, hypothesis-driven bilingualism research, and applied linguistics.

Many different initiatives promote open science in Turkey, such as those initiated by the Turkish Council of Higher Education (YÖK) and Turkey’s Scientific and Technological Research Council (TÜBİTAK). These initiatives include but are not limited to establishing committees responsible for awareness-raising activities on open science and open access at universities and encouraging open access and open course materials. All of these inspired us to design a small course module for graduate students with our Catalyst Grant. The module’s goal was to raise awareness about open science, including both problems and proposed solutions, focusing on the needs of graduate students conducting language research. We decided to focus on graduate students because they are at the beginning of their research journey. They can bring about changes in the local research culture (e.g., at their labs, research groups) and their research practices. Our graduate student cohort consisted of 28 language researchers (14 Master’s and 14 PhD students) interested in different sub-fields of language studies such as language acquisition, language processing, theoretical linguistics, discourse and corpus studies, and language education. 

Given that open science is relatively new to language researchers, we expected a low level of awareness. Thus, we first introduced students to the motivation for open science, like the replication crisis and cases of academic misconduct. Moreover, we developed language research-specific examples (i.e., scenarios) of researcher degrees of freedom and questionable research practices (see here for the materials). We then held a session on the suggested solutions, including pre-registration, pre-print, reproducible workflow, and data & material sharing. The reaction we got from the students was quite positive. They learned a lot from these sessions since they had not heard about these discussions before. Further, learning about some questionable research practices (e.g., hypothesizing after the results are known) appeared to be very enlightening for the students since they had been innocently thinking that there was nothing wrong with such practices.

The session on possible solutions was quite informative for us as the instructors of this module. First, we were unsure whether our attendees would be willing to use the open science tools and practices in their research. Second, we were also unsure how existing open science tools and practices would apply to students’ research, which was quite varied methodologically—15 students used predominantly quantitative methods, 9 used qualitative methods, and 4 used mixed methods. However, it was good to see that at the end of the module, almost all of them were enthusiastic about using the solutions in their own research and sharing them with their colleagues and students. 

Pre-registration and data sharing were perhaps the most concerning open science practices for the students. Some of our attendees were unsure whether pre-registration would be viable or helpful for corpus studies (testing language-related hypotheses relying on a corpus) and qualitative research. Luckily, we were able to point the students to a template for qualitative pre-registrations, and we talked about the importance of pre-registration in different types of research. In terms of data sharing, some students were worried about transforming their data into a shareable version without revealing the participants’ sensitive information. They were also concerned about other ethical issues and legal restrictions in data sharing. We responded to these concerns by acknowledging that each data sharing situation is unique and deserves careful consideration by referring to rules, regulations, and ethics boards of the institutions, and by weighing pros and cons before sharing. 

These concerns convinced us that there is a pressing need for comprehensive training specifically designed for data sharing in language research with the involvement of research ethics committees. For our next project, we plan on designing a more advanced module, possibly also encompassing a session on qualitative research. In the meantime, we invite other language instructors to use our module materials to discuss open science practices in their classrooms. We also welcome feedback and suggestions!

---

About the authors