Improving open and rigorous science: ten key future research opportunities related to rigor, reproducibility, and transparency in scientific research

Participants

We invited participants based on prior contributions to RRT research. Participants included representatives from several leading organizations and Indiana University (IU) faculty, staff, and graduate students who were invited to participate in the meeting and proceedings (Table 1). For their participation in the meeting, invited guests who were not federal employees or IU employees received a $1,000 honorarium.

Meeting format

The two-day meeting was comprised of nine prepared research talks, moderated panel discussions, and small-group open-forum style sessions related to each of the three previously stated goals.

Day one. On the first day, participants presented 10–12 minute research talks, each of which was followed by a moderated question-and-answer period. Participants discussed questions pertaining to RRT and sought to identify emerging areas of research including novel approaches, testable outcomes, and potential limitations. During the afternoon session, participants were divided into three small-groups to discuss potential research opportunities, moderated by an IU faculty representative charged with compiling notes for record keeping and dissemination.

Day two. On the second day, one representative from each group summarized major points through a brief presentation, which was followed by a question-and-answer session with all participants. This dialogue was intended to clarify ideas raised and to identify fundable research opportunities. The meeting concluded with a call to action by the Dean of the School of Public Health-Bloomington and Co-Principal Investigator of the project (DA), to continue promoting interdisciplinary RRT Science.

Subgroup 1: improving education & training in RRT

We asked the first subgroup to discuss research opportunities related to implementing and testing RRT-guided academic curricula. The group identified elements of current undergraduate and graduate education that contribute to problematic data practices, including possible underlying causes and potential solutions (see Table 2). Three primary education-related questions guided the discussion:

With respect to each question the existing and entrenched practices, feasibility of change, and proper audience for interventions were discussed.

1. Can RRT-focused statistics and mathematical modeling courses improve statistical practice?

Incorrect analyses are some of the most common, preventable errors in science (Resnik, 2012). Scholars attribute mistakes to gaps in statistics education (Thompson, 2006). With the rise in data science as a component of scientific exploration, students need more exposure to evidence-based pedagogical approaches to statistics and mathematical modeling (GAISE, 2016; GAIMME, 2016, and NASEM, 2018). Many introductory data science courses include topics from statistics (e.g., contingency tables [chi-square tests], multiple regression, analysis of variance, and the broader general linear model) (Gorsuch, 2015), as well as mathematical modeling approaches and computational algorithms. These topics can be reframed through an RRT lens as modules/domains within existing mathematics or data-science courses or structured as new data-driven courses entirely.

Indeed, participants noted that to improve RRT practices, there are opportunities to design new courses with a direct RRT focus at the undergraduate, graduate and postdoctoral levels (Willig et al., 2018). Courses could include modules related to the identification of errors in published research, proposing solutions to these errors, addressing real-world contexts and demonstrating the importance of careful methodological decision-making (Peng, 2015). Specific assignments could test for and reinforce RRT principles, such as research compendia (i.e. sharable electronic folders containing code, and other exploratory information to validate reported results) (Ball & Medeiros, 2012; King, 1995; Stodden et al., 2015), workflows, which are described later in this paper, and other research projects related to communication and computational reproducibility. The learning practices could be assessed to ensure that students appropriately apply concepts rather than demonstrate rote-formula memorization (Thompson, 2002; Ware et al., 2013). Integrating learning into stages of education where students are concurrently engaged in research can help improve both retention and transfer of the RRT ideas to future scientific settings.

2. Can specialized training in scientific writing improve transparency?

Clear scientific writing is necessary to reproduce and build on research findings. To facilitate better writing, scholars have developed curricula to help academics improve writing practice and quality (e.g., Goodson, 2016). However, many academic writing programs focus on personal habit building and development of linguistic mechanics to craft more powerful prose (Elbow, 1998; Kellogg & Whiteford, 2009). In such courses, RRT-related dimensions of writing (such as writing transparently or minimizing ‘spin’) may not be emphasized. Thus, the subgroup discussed how existing writing curricula could incorporate RRT principles, what new writing courses guided by RRT would entail, and research opportunities to test the efficacy of new writing curricula.

Participants identified several RRT-specific writing principles and discussed how a deeper understanding of the extent to which writing and research are intertwined may increase transparency. Examples included learning about methodological reporting guidelines, writing compelling post-publication peer reviews, and other transparent writing practices. The group also discussed how courses could be developed or redesigned specifically to center on RRT principles. One theme of the discussion was the need for rigorous testing of student learning outcomes associated with novel writing content. However, a primary concern was the identification of the appropriate outcome measures for writing-specific interventions (Barnes et al., 2015) given the subjective and nebulous nature of constructs like writing quality, individual improvement, and writing-related self-efficacy.

3. Does modality affect the efficacy of RRT-related education?

Another research opportunity discussed by the subgroup related to instructional modality, which refers to the manner in which a curriculum or intervention is experienced by the learner (Perry & Pilati, 2011). These may include traditional face-to-face instruction, synchronous or asynchronous online meetings/trainings, and various hybrid formats (Beall et al., 2014). Understanding the relative benefits of each modality is important in choosing an appropriate intervention. Indeed, educational needs vary among learner groups; for example, what is most effective for undergraduate students may not be effective or feasible for post-doctoral researchers with full-time professional commitments. Broad research questions identified by the group included:

In the context of previously discussed coursework in statistics and writing, participants explored the strengths and weaknesses of various modalities and how interventions could be conducted to test them empirically. There are logistical considerations, such as cost, space, and faculty time, that further complicate the feasibility of these interventions. For example, a face-to-face intervention may offer more tailored instruction to individual learners, while an online intervention may better deliver content to a wider audience. Thus, the subgroup identified several areas for future research, including comparisons of student learning across modalities, strategies for scaling educational content to institutional constraints, and the moderating effects of learner demographics on intervention efficacy.

Subgroup 2: reducing statistical errors and increasing analytical transparency

Errors are “actions or conclusions that are demonstrably and unequivocally incorrect from a logical or epistemological point of view” (Brown et al., 2018). Despite the adage that science is self-correcting, uncorrected errors are prevalent in the scientific literature (Brown et al., 2018; Ioannidis, 2012). Subgroup 2 discussed questions related to reducing and mitigating such errors, including:

The costs and benefits associated with each question were also discussed (see Table 2).

4. Can automation help identify errors more efficiently?

Various automated and manual methods have been developed and applied to assess analytic inconsistencies, statistical errors and improbabilities, and other errors (e.g., Anaya, 2016; Baggerly & Coombes, 2009; Brown & Heathers, 2017; Georgescu & Wren, 2018; Labbé et al., 2019; Monsarrat & Vergnes, 2018). An increase in automation (i.e., producing more user-friendly tools and algorithms) has the potential for surveilling the prevalence, prevention, and correction of errors. However, more work is needed to determine the most efficient use of such tools, including their collective abilities to detect field-specific issues that require subject matter expertise (Lakens & Debruine, 2020). For example, the automatic recomputation of some p-values is possible using the program ‘Statcheck’, but only for articles that utilize the American Psychological Association’s (APA) in-text citation style for statistical reporting (Nuijten et al., 2017). Other examples require statistical ratios (Georgescu & Wren, 2018), or integer-based data and sample sizes (e.g., Brown & Heathers, 2017; Heathers et al., 2018), which are both challenging to automate and not recurrent across all fields.

Automated error detection is currently limited to a narrow range of errors. Other types of errors might be detected by careful readers, such as the ignoring of clustering in cluster-randomized trials (Brown et al., 2015; Heo et al., 2018), misinterpretation of differences in nominal significance, and post-hoc fallacies (Brown et al., 2019; George et al., 2016). The subgroup discussed opportunities to define, and possibly automate, diagnostic checklists, advanced natural language processing, or other computational informatics approaches that would facilitate the detection of these errors. These novel automated measures could be tested empirically for effectiveness.

5. What is the prevalence and impact of errors?

Different errors will have varying impacts on study conclusions. While some errors can be easily corrected and reported, some fundamentally invalidate study conclusions. Some general statistical errors have occurred repeatedly across disciplines for decades (e.g., mistaken differences due to “regression to the mean” since at least 1886 [Thomas et al., 2020] and “differences in nominal significance” for decades [Altman, 2002; Thompson, 2002]). Automated methods, such as those outlined above, have been used almost exclusively to illuminate problems but not necessarily correct them (Georgescu & Wren, 2018; Monsarrat & Vergnes, 2018; Nuijten et al., 2017).

To achieve the goal of error reduction, one must first know how pervasive errors are. Yet, it remains challenging to generalize the detection and correction of scientific errors across disciplines because of field specificity (i.e. the unique nuances and methodological specificities inherent to a specific field of study) (Lohse et al., 2020), the various terminologies used for describing the same models (e.g. ‘Hierarchical Linear’ models vs ‘Multilevel’ models), as well as the seeming need to repackage the same problem as new disciplines arise (e.g. ongoing multiple comparison issues raised anew with the advent of genome-wide association studies, microarray, microbiome, and functional magnetic resonance imaging methods). Thus, this subgroup discussed the value of longitudinal, discipline-specific error surveillance and error frequency estimation to collect empirical evidence about error rate differences among disciplines. Other issues discussed were the identification of better prevalence estimates across fields, and how simulation studies can modify our confidence in the understanding of the prevalence of errors and their generalizability across disciplines.

6. Do error prevention workflows reduce errors?

Workflows are the various approaches for accomplishing scientific objectives, usually expressed as tasks and dependencies (Ludäscher et al., 2009). The implementation of clear, logical workflows can potentially prevent errors and improve research transparency. Workflows may be of value to catch errors at various stages of the research process, from planning, to data collection and handling procedures, and reporting/manuscript screening (Cohen-Boulakia et al., 2017). Error detection processes within scientific workflows may serve as mechanisms to prevent errors before publication, akin to how text duplication software (e.g. iThenticate) is used prophylactically to catch inadvertent plagiarism. Separately, some research groups implement workflows that require two independent scientists to verify data, analyses, and statistical reporting prior to manuscript publication, with at least one of those individuals being a professional statistician (George et al., 2016). A similar workflow is to establish “red teams”, consisting of methodologists, statisticians, and subject-matter experts, to critique the study design and analysis for errors, offering incentives akin to “bug bounty” programs in computer software development (Lakens, 2020).

The development and dissemination of research workflows could be modeled after those outlined above, or in other ways such as the use of checklists to complete work systematically. Registrations, reporting guidelines, and other workflow approaches essentially serve as checklists of the plan for a study and what should be reported. Although this subgroup agreed about the importance of preventive versus post-publication workflows and integration of automated methods to detect errors, questions regarding their efficacy remained. For example, how might workflows be generalized across academic disciplines? At what level should standardizing data collection and handling be taught to scientists to maintain data provenance (e.g. Long, 2009)? And can workflows be tested empirically? What is the cost of automated versus manual workflows, versus none at all, at detecting and preventing errors? How do workflows impact productivity?

7. How do we encourage post-publication error correction?

Science cannot self-correct without processes that facilitate correction (Firestein, 2012). Unfortunately, errors in science may be tied with perceived reputation costs, yet it is unclear that correcting errors actually harms a researcher’s reputation (Azoulay et al., 2015). Thus, destigmatizing error correction and likewise embracing the importance of scientific failures may be of value for individual scientists and editors overseeing content through the peer-review process (Teixeira da Silva & Al-Khatib, 2019). Journals and their editors, as gatekeepers of science, are key stakeholders in this culture shift. They may also require practical guidelines to facilitate judgement-free corrections that would be acceptable to editors and reviewers.

Error correction should be done in a fair and efficient manner (e.g., Vorland et al., 2020). Although there are several existing standards for publication ethics and norms (e.g., Committee on Publication Ethics [COPE], and the International Committee of Medical Journal Editors [ICMJE]), few have been tested empirically. The subgroup debated how journals and their editors could be part of empirically tested trials on best approaches to facilitate correction and minimize the incurring of additional costs. For example, based on our experiences, journals have few procedures for handling errors separate from typical scholarly dialogue. We believe it is important to examine which procedures are more efficient and fair to authors, whether such procedures can be standardized to enable editors to handle different types of errors consistently and transparently, whether correction mechanisms are sufficient or require additional innovation (e.g. retraction and republication is sufficient or versioning), and how authors can be supported and encouraged in the process. Three such costs that require further study include the actual cost of post-publication error correction across all parties involved (e.g. page charges, salary), how those costs to the scientific enterprise compare to implementing prevention strategies, and the cost-benefit of salvaging a publication containing an error depending on the quality of the collected data versus simply retracting.

Subgroup 3 - looking outward: increasing truthfulness and accuracy of research communications

The third working group discussed opportunities for research related to research reporting and dissemination, primarily highlighting the importance of accuracy and truthfulness when communicating research findings (see Table 2). Specifically, this group identified research opportunities tied to the following questions:

8. How does “spin” in research communication affect stakeholders’ understanding and use of research evidence?

In addition to conducting research rigorously, investigators should describe their research comprehensively and interpret their findings by balancing the strengths and limitations of their methods and results (Brown et al., 2017). By contrast, researchers might ‘spin’ their results through misleading reporting, misleading interpretation, and inappropriate extrapolation (Fletcher & Black, 2007; Yavchitz et al., 2016). Some evidence suggests that spin is common in reports of clinical trials and meta-analyses (Boutron et al., 2019; Lazarus et al., 2015) and that authors in a variety of research disciplines often draw inappropriate causal inferences (Bleske-Rechek et al., 2015; Casazza et al., 2013; Chiu et al., 2017; Knight et al., 1996; Ochodo et al., 2013). Moreover, spin in popular media (e.g., newspapers) appears to stem from spin in scientific reports (e.g., journal articles) and associated press releases (de Semir et al., 1998; Schwartz et al., 2012; Schwitzer, 2008).

Spin is unscientific, and could have implications for policy and practice (Adams et al., 2016; Boutron et al., 2019; Matthews et al., 2016). Workshop participants discussed the need for more evidence to determine whether and how spin in scientific reports affects other stakeholders such as healthcare and social service providers, service users, policymakers, and payers. Evidence concerning the ways in which stakeholders use and interpret research evidence could inform future efforts to improve research communication (Boutron et al., 2019; Lazarus et al., 2015).

9. Do tools to aid writing research reports increase the comprehensiveness and clarity of research reports?

Research reports (e.g., journal articles) should describe what was done and what was found (von Elm et al., 2007). Stakeholders need comprehensive and accurate information about research methods and results to assess risk of bias, interpret the generalizability of study results, and reproduce the conditions (e.g., interventions) described (Moher et al., 2011). Reporting guidelines describe the minimum information that should be included in reports of different types of research, yet much evidence suggests that scientific reports do not include this information (e.g., Grant et al., 2013). Some tools to help authors write better reports have been developed, such as the consort-based web tool (COBWEB) (Barnes et al., 2015); some preliminary evaluations suggest that these tools could help authors write better reports.

Workshop participants identified a need for research to develop and to test tools that could help authors write reports that adhere to existing guidelines. Some tools could be used when writing scientific manuscripts (Turner et al., 2012) while other tools could be used in graduate education (e.g. class assignments, dissertation writing) or continuing education. Guidelines designed to increase authors’ and reviewers’ knowledge of reporting requirements are not commonly adhered to and, thus, have minimal impact on reporting quality (Capers et al., 2015). Participants emphasized the need for new interventions and implementation research that promote guideline adherence.

10. Is it possible to inculcate scientific values and norms related to truthful, rigorous, accurate, and comprehensive scientific reporting?

In the 1940s, Robert Merton proposed that communism/communality, universalism, disinterestedness, and organized skepticism constitute the ethos of modern science (Merton, 1942). As the National Research Council stated in their report “Scientific Research in Education”, these fundamental principles are enforced by the community of researchers that shape scientific understanding (Shavelson & Towne, 2003). Evidence suggests that most scientists endorse these positive values and norms, but fewer scientists believe that their colleagues behave in accordance with these positive norms (Anderson et al., 2007). Better incentives (Begley et al., 2017; Fanelli, 2010; Nosek et al., 2012) and better methods for detecting scientific errors, might improve scientific practice and communication; yet fundamentally, we will always have to place some trust in the veracity of our fellow scientists (Jamieson et al., 2017).

Participants agreed that ethics and responsibility are vital across scientific disciplines, yet graduate research often neglects the philosophy of science and the formation of professional identity as a scientist. Instead, training tends to focus on the technical skills needed to conduct experiments and analyze data in specific disciplines (Bosch, 2018; Bosch & Casadevall, 2017). Technical skills are essential to produce good science; to apply them ethically and responsibly, however, it is paramount that scientists also endorse scientific values and norms. Participants identified a need for research to determine how these scientific values could be inculcated in scientists and how scientists should be taught to enact those values in their research.