Search of stakeholder websites

Search of the literature (MEDLINE)

With the help of an experienced research librarian (NB) we designed a comprehensive search strategy using MeSH terms and text words (see S1 Text for full search strategy) and conducted a systematic literature search in MEDLINE using the Ovid interface from database inception to February 27, 2015. We did not impose any language restrictions.

Data extraction

We designed standardized extraction sheets suitable for qualitative data extraction (S2 Table) accompanied by an instruction manual. Before starting data extraction, the data extraction forms were piloted and teams of reviewers conducted calibration exercises to ensure consistency. We extracted text sections on the definition, concept, or criteria of quality from both literature and internet sources independently and in duplicate. Data synthesis of included articles involved categorization by overall topic, author, year of publication, article citation index (as retrieved in ISI Web of Science by 11 January 2016), and journal name. Internet sources were categorized by stakeholder group, country, and name of organization.

Data analysis

We performed descriptive and qualitative explanatory analyses following the Framework Method [31] on definitions, concepts, and criteria of clinical research quality stratified by stakeholders and on evaluation criteria of funding agencies for clinical studies. The Framework Method belongs to a family of qualitative approaches termed thematic or content analysis, which identify commonalities and differences in qualitative data, and eventually seek to draw descriptive and/or explanatory conclusions clustered around themes. Its defining feature is the matrix output, i.e. rows (cases), columns (codes) providing a structure into which the researcher can systematically reduce the data in order to analyze it [32]. We therefore applied codes to excerpts of raw data and added or modified as new responses emerged. Codes were then grouped into clusters around similar and interrelated ideas to identify common and stakeholder-specific quality themes in an iterative process until consensus between the three investigators (BvN, MB, CPM) was reached. Themes were named after the most frequently recurring terms within the same clusters (e.g. generalizability, relevance, high quality data etc.) and were not created or imposed by the investigators.

Definitions or concepts of clinical research quality in different stakeholder groups

We screened publicly available websites and linked documents of 155 stakeholders. Concepts of how to assure quality of clinical research or quality criteria were commonly reported among most stakeholder groups (66.4% (103/155); i.e. in 86.1% (31/36) of pharmaceutical companies or contract research organizations (CROs), 72% (18/25) of academic research organizations or initiatives, 63.6% (14/22) of international and governmental organizations, 61.9% (13/21) of regulatory agencies, 57.9% (11/19) of ethics committees, and 63.2% (12/19) of funding agencies, respectively), but this was relatively uncommon for patient organizations (31%; 4/13). However, only 12 of 155 (7.7%) institutions provided an explicit definition of the term ‘clinical research quality’ (pharmaceutical companies or CROs: 3/36; academic research organizations and initiatives: 3/25; international and governmental organizations: 3/22; regulatory agencies: 2/21; patient organizations: 1/13; ethics committees and funding agencies: 0/38) (S3 Table).

Qualitative analysis of the 12 definitions and the 103 quality concepts or criteria resulted in both common and stakeholder-specific quality themes often focusing on different stages of clinical research (planning/feasibility, conduct, dissemination; Table 1). Common quality themes amongst stakeholder groups included the adherence to all applicable national and international laws and regulations (e.g. ICH GCP), a scientific and methodologically rigorous approach allowing for an efficient and effective answer to the research question, credible and high quality data, the inclusion of trained study personnel, and the presence of Standard Operating Procedures (SOPs) and monitoring.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Qualitative analysis of common and stakeholder-specific quality themes in the context of clinical research.

https://doi.org/10.1371/journal.pone.0180635.t001

Stakeholder-specific emphasis on quality themes ranged from “high quality data” (pharmaceutical industry and CROs); “adherence to guidelines” (regulatory agencies); “patient involvement and applicability of research” (patient organizations); “absence of bias, relevance, and transparency” (academic research and/or initiatives); to “feasibility, generalizability, and objectivity of research” (funding agencies). The terminology used by the stakeholders to describe these themes (e.g. relevance, transparency, feasibility), was no less open to definition than the overarching concept of “quality” and as well depends on the perspective of the observer. In general, priorities within stakeholder groups were similar across different countries. However, for national funding agencies we found considerable variation in quality criteria that were particularly emphasized as relevant for funding decisions across countries (S4 Table).

Definitions or concepts of clinical research quality in the medical literature

Our systematic MEDLINE search yielded 8’289 titles and abstracts, of which we reviewed 90 articles in full text (Fig 1). We excluded 43 full text articles from detailed analysis, because they did not discuss a definition, concept, or criteria of quality (n = 18), they were systematic overviews/summaries of existing quality assessment checklists, instruments, or scores, with or without critical discussions of their validity and/or reliability (n = 5), or they discussed specific measurement instruments of a single dimension of quality (n = 20).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Article flow diagram.

https://doi.org/10.1371/journal.pone.0180635.g001

We included the remaining 47 articles for more detailed analysis (S2 Text). These provided concepts on how to assure or improve overall clinical research quality in specific contexts (e.g. at an academic institution, in a specific country, in a specific industry setting, or in a specific medical field; n = 18), or how to improve quality assessment (e.g. of RCTs, in radiology or hepatology research; n = 6). Measurement instruments or checklists that spanned more than one quality dimension were reported in 23 articles. A large proportion of these tools provided indicators on how to assess bias (n = 21). Almost half of them covered indicators on precision (n = 16), external validity (n = 16), or reporting quality (n = 14). Some tools additionally covered innovation aspects (n = 8) or ethical considerations (n = 4). None of the reviewed articles provided a definition or concept of clinical research quality spanning the encountered range of quality dimensions reflected by stakeholder perspectives such as ethical conduct, patient safety, patient values and preferences, absence of bias, precision, external validity, relevance, generalizability, transparency, infrastructure, and sustainability. Furthermore, we could not identify a definition or concept simultaneously covering several dimensions and differentiating between consecutive stages of research (e.g. study planning, conduct and dissemination), independent of a specific medical field or study setting.

Overall, we identified eight (8.9%) of 90 articles that provided an explicit definition of the term ‘clinical research quality’ (Table 2). The definitions therein span quality from methodological dimensions such as internal validity, external validity, or precision, and operational criteria including adherence to guidelines and applicable regulations (ethical conduct), to the effect of research at the societal level (relevance). None of the definitions appeared to be comprehensive either in terms of quality dimensions, research stages, or stakeholder perspectives. Five of the eight articles were cited less than 10 times in ISI Web of Science^™ by 11.01.2016 (Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Characteristics of articles providing an explicit definition of clinical research quality; by author (n = 8).

https://doi.org/10.1371/journal.pone.0180635.t002

Summary of findings

Our systematic review of stakeholder websites and the medical literature showed that quality of clinical research is frequently discussed, but rarely defined. Although stakeholder groups seem to agree on a basic concept of quality, their emphasis in the conceptualization of clinical research quality varies widely. The medical literature contains many articles discussing approaches to measurement or assessments of quality without prior definition of the term itself, and without reflecting the diversity of stakeholder needs, interests and expectations. A major proportion of these identified quality assessments aim to evaluate the “methodological rigor of randomized controlled trials”. The definition of “methodological rigor” in itself, however, varies substantially between the reported tools. Most authors suggested assessing methodological quality based on the presence or absence of measures to prevent bias. Others included dimensions such as external validity, reporting, or relevance of the study in question. We did not, however, identify a definition or concept including multiple dimensions or differentiating between consecutive stages of research across medical fields, or study settings. Although a comprehensive “definition” of quality may be difficult, a “concept” or “framework” of research quality, rather than a “definition”, could span all research stages and include more than an assessment focused on one aspect of quality. A more comprehensive approach to quality assessment, i.e. ranging from conceptualization to dissemination of a study as proposed by the authors of the Lancet series on “increasing value, reducing waste” [1, 4–7], rather than evaluation of the final published product, would assist in identification of errors that matter at earlier stages, and therefore support reduction of research “waste” more efficiently.

Strengths and limitations

To our knowledge this is the first systematic survey addressing definitions and concepts of clinical research quality. Our systematic approach was suited to detect knowledge gaps, and to examine overlap and differences in perspectives of clinical research quality across stakeholder groups. Further strengths of this study include our consideration of websites and any linked documents from a large number of stakeholders in 13 different countries in addition to a Medline search. Methodologically trained investigators screened articles and websites in duplicate following a pre-specified instruction manual and undergoing a calibration process.

We acknowledge the following limitations: Although we consider our approach comprehensive, we searched only Medline as electronic database and relied on search terms in the title, abstract or other records. Articles in journals not indexed in Medline or providing some definition of research quality in the main text only might have been overlooked. However, Medline covers the most impactful journals and articles in current medical research and articles specifically focusing on quality of clinical research most likely mention this prominently. We may have missed definitions on websites despite screening these in duplicate. We would, however, expect stakeholder groups in clinical research to be transparent and proactive in defining such an important cornerstone of their activities, similarly to efforts in the field of clinical care quality. When coding our findings from the website search as well as from the Medline search we felt that we reached saturation, i.e. the last excerpts from websites or journal articles on aspects of clinical research quality did not bring new insights. The coding and qualitative analysis naturally involved subjective judgments of investigators, which we controlled by performing analyses in triplicate (BvN, CPM, MB) and comparing codes, findings, and interpretations until we reached consensus. Further, we acknowledge that our survey solely portrays perceptions on the quality of research in high income nations that may not necessarily overlap with those of low- or middle-income countries. With a growing percentage of clinical research being conducted in these geographies, a further study investigating quality perceptions of local stakeholders taking into consideration societal aspects and beliefs would be of importance. Finally, we did not conduct a detailed survey of experts or stakeholder groups—except for national funding agencies—nor did we conduct interviews with representatives of these groups to explore reasons for the paucity of explicit definitions.

Comparison with other studies and implications

We are not aware of any other systematic survey on definitions, concepts, or criteria of overall clinical research quality. A similar approach has been taken by other authors to develop a framework for excellence, however with a distinct focus on translational cancer research [41]. In the field of health care quality, the focus of assessments has more and more shifted from process-based measurements towards the evaluation of patient outcomes and patient satisfaction [42–44]. Clinical research conducted in this context of “patient-centered” care would explicitly warrant the engagement and involvement of patients in setting priorities. However, patients (or their representatives) are only rarely considered when discussing the quality of research that might impact their care [45–47]. In our analysis, we also found patients to be surprisingly underrepresented. First, patient organizations were among the last in providing definitions or concepts of clinical research quality. Second, only six (14%) of a total of 43 quality measurement tools or assessments covered an item on patient safety and/or rights (Fig 1). Most efforts in quality assessments so far were taken to ensure compliance with guidelines, methodologically rigorous designs and valid study results. While this may ultimately serve the treatment of disease, we were expecting the clinical research enterprise to put unmet medical need and applicability to the patient population first. Compared with medical care, the clinical research machinery still seems to function with relatively low engagement of the end user (patients) of the product.

Furthermore, while we expected variations in the perception of quality across stakeholder groups, we were surprised how different and vaguely defined some of the concepts were. For example, an explicit definition of “high quality data” may be as dependent on the perspective of the observer as the definition of “high quality research”. It may be linked to concepts such as relevance of the data and absence of errors in the data or the way the data is collected. Similarly, the quality criteria used by funding agencies such as “impact”, “relevance”, or “feasibility” varied in their clarity and elaboration. Public funding agencies have a major role in terms of defining what and how research topics are investigated. Those who use these criteria to evaluate proposals are still left with subjective interpretation, while applicants may aim to provide the readers with these buzzwords with not much reflection on their meaning.

There remains considerable ambiguity in the use of current quality criteria across and within stakeholder groups. Unless carefully explained, these concepts can be easily misinterpreted by the stakeholders. Finding consensus on a common definition or concept of clinical research quality across national borders, stakeholder groups, and study types may therefore seem arduous; assessments of methodological quality do not, however, suffice. Existing quality guidelines such as ICH GCP have not been developed based on consensus across the full range of stakeholder groups, but only between regulatory experts and industry [15]. Existing quality assessment tools predominantly cover single aspects of quality, or particular research stages. Furthermore, there is a lack of approaches tailored to stakeholder requirements in assessing the quality of clinical research, e.g. from a patient’s perspective on how to choose a “good trial”, or from a funding agency’s perspective on how to assess the quality of studies before, during, and after the funding period. The authors of a follow-up study to the 2014 Lancet series reported that academic institutions in particular had paid only little attention to their recommendations on how to increase value in research. Practical guidance on how to implement these recommendations is so far lacking and urgently needed to increase value of academic research at all stages.