Researchers on research integrity: a survey of European and American researchers

Ethics

Ethical approval for conducting the survey was obtained from the University of Essex, UK, Faculty of Social Sciences Ethics Committee (ETH2021-0441). The approval document can also be found on OSF (DOI 10.17605/OSF.IO/XB9RK). Informed consent was obtained by the provision of information to participants before they agreed to respond by clicking the survey link.

International Research Integrity Survey (IRIS)

The survey was conducted as part of a larger study on RI. Standard Operating Procedures for Research Integrity (SOPs4RI) is a four-year project funded by the European Union under the ‘Science with and for Society’ programme (SOPs4RI – Promoting excellent research, n.d.). IRIS is based on a systematic, stratified probability sample of the authors of research articles published between 2016 and 2020 included in Clarivate’s Web of Science citation database. The full sample from 34 countries is over 60,000 respondents. The focus of this paper is 2300 researchers based in the US and 45,000 in Europe (including UK, Norway, Iceland and Switzerland). Few, if any, previous surveys on RI and related issues have been based on probability designs or cover such a wide range of research fields. Since 2004, a number of surveys have asked researchers about RI (Titus et al., 2008; Van den Eynden et al., 2016; Woolston, 2020), most recently a national study in the Netherlands (Gopalakrishna et al., 2022) but IRIS is the first survey to feature such a broad, representative international sample.

The online questionnaire focuses on the behaviors of researchers but also on what they think about RI practices and policies in their organizations. Respondents were asked about their self-confidence in supporting research with integrity; if they consider their organization is effective in delivering best RI practices; how much institutional oversight of RI is desirable; whether they had engaged in selected questionable (or detrimental) research practices (QRPs); how closely their organization aligns with best practice; what are the dominant channels of communication on RI topics, what would motivate them to engage with RI policies and procedures and on which topics they would welcome additional support.

The sample

The population of interest for this study was active researchers in the humanities, social sciences, natural sciences (including technical science), and medical sciences (including bio-medicine), who hold at least a master’s level degree and produce research for commercial or academic institutions within the EU, EFTA, U.K. and the US. The way we divide researchers into these broad fields follows the OECD Frascati Manual. We show a detailed breakdown of how subfields are allocated within each of the four main fields, along with their frequency distributions in the Extended data (Allum et al., 2022). The sampling frame was the Clarivate Web of Science bibliographic database, which contains details of publications produced by researchers in 21,894 scientific journals, books and conference proceedings. The sample was constructed from a background population of academics, identified in the bibliographic database, Web of Science (WoS). WoS contains article metadata for more than a million research articles annually. From these records we extracted information on author names, affiliations and e-mail addresses, for all articles published in the period 2016-2020, where at least one author had an affiliation to an institution in one of the target countries. We downloaded 8,159,772 metadata records and retrieved 3,929,283 e-mail addresses, from which we were able to create 3,759,814 author profiles. Of these 3,072,372 were from our countries of interest.

Our objective was to obtain a sample that was both representative of the WoS population and contained sufficient numbers of observations within all countries and fields to enable robust comparisons to be made. To accomplish this, we generated a systematic sample with unequal selection probabilities with explicit and implicit stratification. We aimed to increase the precision of comparisons across four scientific fields by each country combinations through aiming for a similar effective sample size within each such combination. This naturally led to an unequal selection probability sample design with lower selection probabilities in those field-country combinations that have a larger number of publications in WoS. The explicit stratification categories were fully-crossed country by scientific field (natural, medical, social sciences and humanities) combinations. Within each such stratum a systematic sample was drawn, additionally using implicit stratification by a more granular indicator of scientific field and an indicator of the number of papers published by each author. The exceptions to this procedure include those countries, or fields within some countries, where the total number of authors was smaller than that required to achieve the planned effective sample size. In such situations all authors were included in the issued sample.

The survey

The survey was conducted entirely online, in English, using the Qualtrics platform. The survey rationale was developed and agreed in consultation with project partners as detailed in the survey protocol document (Reid and Allum, 2021). Survey questions were developed between November 2020 and April 2021 by authors NA and AR, guided by topic experts within the group of project partners. The full survey included sections covering: structural or demographic variables; values, beliefs and attitudes in relation to science practices, research integrity policies and the role of organisations in implementing them; the current research integrity landscape, including awareness of and satisfaction with current research integrity arrangements; personal efficacy and behaviour; and receptivity towards research integrity policies including specific examples of standard operating procedures. A set of items asking about questionable research practices (QRPs) was drawn from past research on the following criteria: breadth of coverage across research integrity areas, applicable regardless of discipline and ranging from common to rare behaviors (Gopalakrishna et al., 2022; Schneider et al., In preparation).

Cognitive testing of the questionnaire was conducted via online meetings due to COVID-19 during February and March 2021. A simple random sample of 5000 people were invited to take part in pilot testing from April to May 2021, following which some changes were made. Invitations and reminders to take part in the survey were distributed from 22^nd June – 28^th July 2021 and the survey closed on 14^th September 2021.

The invitation emails included information about the project and funder, with links to the Qualtrics survey and to instructions for how to opt out from further communication. In addition, it included information about how the individual had been selected, the scope and purpose of the research for which personal data about them would be collected, how their personal data would be used, who would have access to it, the benefits of participation, and their right to withdraw at any time, including instructions on how to do so. Respondents were told that starting the survey after reading the information supplied implied written consent to participate. The full text of the emails and questionnaire can be found in the Extended data (Allum et al., 2022).

73,757 people responded to the survey. Of these, 1,602 were ineligible due to their country of employment being outside our target countries, which were EU, UK, Norway, Switzerland, USA and Canada. A further 6,391 were excluded as they completed less than 25 % of the survey, which gave no information beyond demographics. Lastly, those who did not state that they were trained to at least master’s level were removed. The remaining 64,074 cases were retained. The overall response rate, computed using the American Association for Public Opinion Research’s standard definitions, was 7.2% (AAPOR Response Rate 2) (American Association for Public Opinion Research, 2016). For this study, we selected only those who had completed 75% of the survey or more and who were employed in Europe or the United States of America at the time of completing the survey.

Weighting

We computed weights that we apply in our analyses to correct for the unequal selection probabilities of cases inherent in the sample design and for biases caused by differential non-response. Not all the authors in WoS had the same initial probability of selection, depending on the sizes of the WoS sub-populations used in the stratified design. We aimed to gather approximately 500 responses in each scientific field in each country. Hence those authors in smaller countries that had few authors in WoS had a higher probability of selection than those in countries that had much greater representation. The weighting reflects these relative selection probabilities. In addition to design-related adjustments, we used the information about the WoS authors that we included in the sample design to estimate the overall probability of responding, adjusting for both the study design and non-response. We modelled this using logistic regression. A binary variable that indicated whether a sample member provided a usable response to the survey (i.e. answered more than 25% of the questions) was specified as the dependent variable. The independent variables were country, field, country x field, number of papers and granular subfield. The model therefore takes into account simultaneously the unequal selection probabilities and the differential non-response propensity. The weight variable we derive from estimating this model this was computed as the inverse of the predicted response probability for each respondent and normalised so that the final weighted sample size matched the unweighted sample size. We also trimmed this weight at the 99^th percentile so as not to over-inflate the design effect.

Analysis/statistical methods

Our analysis is largely descriptive and was not preregistered. We present mainly two-way cross-tabulations and bar charts for outcome variables of interest, split by region, scientific field, sex, employment contract, industrial sector and career stage. In some sections, we also present OLS multiple regression analyses to adjust for potential confounding relationships. In some cases, we show 95% confidence intervals around estimates. In many cases we do not, as tables would be unwieldy, but we include unweighted sample sizes. For all data tables, with confidence intervals for all of our estimates, see Extended data (Allum et al., 2022). Standard errors were estimated using Taylor linearization. All analyses were carried out using Stata 17 software with the svy prefix command to adjust for the sample design and weighting. Tables showing all of the weighted and unweighted frequencies for all variables are included in the Extended data (Allum et al., 2022). Details oƒ recodes and derived variables are described in the relevant sections alongside the results. All oƒ the Stata code used for preparation and analysis included in the Extended data along with the full dataset, permitting all of our results to be reproduced independently.

Confidence in integrity of own research

We asked respondents how confident they are that they are currently meeting high RI standards. While more than 95% of researchers report at least ‘some’ confidence, 74% of researchers in the US say they are ‘very confident’, compared to just 52% of European researchers (see Table 1).

This transatlantic confidence gap persists across researchers of different career stages, employment contracts and scientific fields (see Table 2). More established researchers, both in length of career and security of contract, report greater levels of confidence in the integrity of their research, across Europe and in the US.

In the US, high confidence in meeting RI standards rises from 64% among early career researchers to 82% in later career. The comparative figures for Europe are 43% (early career) and 62% (later career). In the US, those with permanent positions register 76% compared to 66% for those with temporary positions. The respective figures in Europe are far lower; 54% and 48%.

The same pattern emerges across the disciplinary fields. A comparison between the US and Europe in the natural, medical, social sciences and humanities shows approximately 20% fewer reporting high confidence in Europe across all fields.

Questionable research practices

Questionable research practices (QRPs) (often referred to as ‘detrimental research practices’) are a recognized challenge to RI (National Academies of Sciences and Medicine, 2017). They fall short of outright misconduct but represent transgressions of best practice, improper use of data and ethically questionable behavior. QRPs potentially diminish the quality and trustworthiness of scientific findings. We selected eight QRPs, based mainly on work by Schneider et al. (In preparation). Figure 1 shows short descriptions of the QRPs and the full wordings are presented in Extended data (Allum et al., 2022).

Figure 1. Percentage of researchers admitting QRPs have occurred in last 3 years.

To introduce the issue of QRPs, respondents were presented with the following statement:

“The next few questions are about questionable research practices (QRPs). These are less than ideal research practices which might happen unintentionally. They are not research misconduct (i.e., fabrication, falsification, or plagiarism). We will present you with a set of research practices and ask you to what extent you have engaged in them when working towards producing your publications over the last three years.

Thinking about research carried out for your publications over the last three years, how often has the following occurred?”

The response alternatives were: often, sometimes, rarely, never, does not apply in my case.

Figure 1 shows the admitted occurrence of each applicable QRP in the US and Europe (error bars represent 95% confidence intervals). ‘Rarely’, ‘sometimes’ or ‘often’ are counted as an admission. For each item respondents could select ‘does not apply’, which excludes them from the results for the corresponding QRP.

The most common QRPs are including authors who hadn’t contributed, not conducting a thorough review and inadequate supervision, to which at least half of respondents admitted. Occurrence of remaining QRPs is acknowledged by between 10 and 20% of researchers. It is worth noting that the most frequent QRP – inappropriate authorship – does not necessarily implicate the respondent directly. For some, an affirmative response could mean that other investigator(s) may have been responsible. Hence, we might expect to see this QRP reported more often as it could include both individual and third-party decisions.

Turning to the contrast between US and European researchers, we find a consistent difference, with the latter admitting more QRPs on average than their US counterparts. The only exception to this is inadequate supervision, which US researchers are more likely to acknowledge. The largest disparities between regions are found for carrying out research without ethical approval and failing to disclose conflicts of interest. In both cases, the rate of admission is twice as high for European researchers.

In Figure 2, we present the mean number of QRPs admitted out of the total applying to each researcher, expressed as percentages. For instance, a researcher who admitted three QRPs, and said that two out of the eight did not apply to them, would be assigned a score of 50% (i.e., three admitted out of six applicable). We show these for the US and Europe, with breakdowns by field, sector, employment contract, career stage and sex. Only one QRP – carrying out research without ethical approval – is indicated as not applicable by more than 20% of respondents, and the non-applicable percentage for most QRPs is in single digits (see Extended data for further detail).

Figure 2. Percentage of applicable QRPs admitted to in last 3 years (n=40,317).

The percentage of QRPs reported by Europeans exceeds that of US researchers, although in some cases the difference is within sampling error. Overall, researchers admit to between 20 to 35% of applicable QRPs. The largest differences are between fields. For example, European medical researchers admit to just under 35% of QRPs while humanities researchers admit to less than 25%. Natural and medical sciences display the greatest tendency for questionable practice, while social sciences and humanities researchers in both US and Europe show a lower self-reported prevalence. University researchers admit fewer QRPs than those in non-academic sectors.

An interesting disparity emerges in relation to career stage; in Europe, early career researchers admit more QRPs than researchers in mid or late career, although the differences are not large. In the US, the pattern is reversed; they are reporting fewer QRPs than their more senior colleagues. Figure 3 presents results from an OLS multiple regression indicating the partial associations of the explanatory variables in Figure 2 with QRPs. Region, contract type, sector and field all show robust differences, after adjusting for all other factors.

Figure 3. Regression of mean number of QRPs admitted, on country group, field, contract, sector, career stage and sex.

Perceptions of RI in organizations

What are the most important topics in organizational policies to support RI? In a series of focus groups and co-creation workshops with researchers from a wide variety of backgrounds, the SOPs4RI study found broad consensus on what these topics are and what counts as good practice (for full descriptions of these features, as presented to respondents, see Extended data) (Mejlgaard et al., 2020; Sørensen et al., 2021). IRIS respondents were asked the extent to which their organization aligns with these policies and practices.

In a fully functioning research system, it might be expected that a high percentage of researchers would say their organization closely or very closely resembles best practice. In both US and Europe, researchers perceive a significant gap between best practice and their organization’s arrangements. In Table 7, we present the percentage of respondents who think that their organization ‘closely’ or ‘very closely’ resembles the ideal on the nine features of RI, broken down by field. Looking first at the bottom row of Table 7, the mean assessments across the nine RI features show that only 52% of researchers in the US and 45% in Europe consider their organizations to be close to the ideal.

Considering transatlantic disparities within each field, only in the case of humanities is there no mean overall difference, although this masks variation within each of the elements. Only 38% in the US and 21% in Europe think integrity training is close to the ideal. Here then, is an area where organizations have considerable scope to improve the situation. Only a minority of researchers in both the US and Europe regard organizational practice around supervision, integrity breaches and publications and communications to be close to ideal. In all but the latter topic, US organizations are viewed as more closely matching the ideal than those in Europe.

Influencers

We have shown that the amount of support provided by researchers’ organizations is regarded by many as less than ideal. At the same time, some researchers do not think that the responsibility for RI lies with their organization at all. This raises the question of who are the actors, communities and organizations whose opinions are most valued? For the many researchers, it is their scholarly community – those that publish in the same journals and attend the same conferences – whose opinions researchers value the most. This is true for 76% of Americans and 62% of Europeans. Researchers’ departments, organizations and professional bodies are each cited by no more than 12% of respondents as having the most important opinions (see Table 8).

Communication channels

Where do researchers gain information about RI? Figure 4 shows the percentages who say they obtain ‘some’ or ‘a lot’ of knowledge from eleven sources. Both in the US and in Europe it is scholarly communities and research collaborators from whom the most knowledge is acquired. Perhaps unsurprisingly, there is a gradient in the amount of knowledge acquired from these more proximal sources (more opportunities, more contact) to more distant sources, such as funding bodies and academies committed to RI. Researchers’ organizations and departments are endorsed by around half of respondents as significant sources of information on RI. Thus, it appears that organizations have ready channels of information transmission to exploit.

Figure 4. Percentage of researchers gaining knowledge from sources of information (n=35,930).

Enthusiasts and skeptics

Drilling deeper into the roles of organizations and researchers, and what the scope for action might be, respondents were asked the following. Firstly, whether they thought RI policies would help to improve the quality of their own research; secondly, whether they considered these policies to be genuinely intended for the purposes of improvement or simply for meeting bureaucratic requirements (‘box-ticking’), and finally how positive they would feel about receiving RI training. Responses to the three questions were combined into a categorical indicator to identify those who are more enthusiastic and likely to be more engaged and those who are generally skeptical towards organizational measures (see Extended data, Allum et al., 2022).

Figure 5 shows the percentages of enthusiasts for personal and organizational RI measures. Overall, researchers are significantly more enthusiastic, at 54%, than skeptical, at 30% (with neutrals at 16%). In absolute terms (chi square test), consistently across the demographic breakdowns, Europeans are more positive than American researchers, although the differences in most cases are not statistically significant. The highest levels of positivity are found amongst women, medical scientists and those working outside of the university sector. Figure 6 shows the results of a multivariate analysis showing the associations of the explanatory variables in Figure 5 considered as a whole. Region, field, sector and sex show robust differences, after adjusting for all other factors.

Figure 5. Percentage of researchers showing positivity towards RI measures (n=41,040).

Figure 6. Regression of RI enthusiast score on country group, field, contract, sector, career stage and sex.

Motivation

For organizations to successfully implement appropriate procedures and protocols for ethical and responsible research, researchers must be motivated to engage with them. Based on previous qualitative research (Sørensen et al., 2021; Sorensen and Ravn, 2020) we presented respondents with a list, shown in Figure 7, of possible motivations for complying with RI practices. We wanted to understand what researchers consider to be the benefits of engaging with these policies, and what kind of incentives could be important to them.

Figure 7. Percentage of respondents very or extremely motivated by benefits of following RI procedures (n=44,713).

The top incentives for scientists in complying with RI procedures concern intrinsic scientific benefits, ‘truth’ and ‘trust’, specifically: more reliable scientific knowledge; more trust from colleagues; more trust from the general public; and a better reputation in my field. It is of note that the extrinsic benefits of promotion and salary prospects are reported to be the least motivating.

There are some differences between researchers based in US and Europe. In the US, the possibility of higher salary is more motivating than it is for Europeans. Europeans are more likely to be motivated by enhanced opportunities to collaborate with others and to gain more self-confidence.

Additional support

How do these overall attitudes and motivations translate into welcoming additional support in each of the nine RI areas? Figure 8 shows that between 20% and 35% of researchers in US and Europe would welcome further support on the majority of the nine RI topics, with European researchers showing greater demand than researchers in the US. Particularly striking is the fact that twice as many Europeans (28%) as Americans (14%) would welcome additional RI training. Additional support in cases of integrity breaches is also an issue where European researchers appear much more open than their American counterparts. This suggests that formal professional development programs in RI would be welcomed by many European researchers, although a majority in both regions remain unpersuaded.

Figure 8. Percentage of researchers welcoming support in nine RI areas (n=47,906).