INTRODUCTION

Accurately describing the effects of medical treatments using plain language and narrative statements is a key component of systematic reviews and critical to broad dissemination and successful implemention of research findings. Systematic reviews include narrative statements of varying details and lengths in different parts of the report, such as the abstract, results, discussion, and sometimes the title. Narrative statements may also be the primary means of communicating summaries of results, for example, via interviews, lay press, social media, blogs, or as policy/management briefs. When narrative statements accurately summarize research results with easy-to-understand language, they can have an important impact on the understanding and application of findings.

A scoping review on the barriers and facilitators to the uptake of systematic reviews by policy makers and health care managers found need for more consistent approaches to report effect sizes in systematic reviews, and also that key messages should have a decision-making focus and be easy to understand.1 In a mixed-methods study that assessed the format and content of systematic reviews, challenges in interpreting results and their implications were reported as a barrier to uptake by decision-makers.2

Despite the importance of effective narrative statements, there has been no clear or consistent guidance on how to develop them. For decades, statisticians and methodologists have advised against making conclusions solely based on statistical significance and also suggested commenting on the magnitude of the effect and the clinical importance of the findings.3 Two recent approaches were proposed by the Cochrane Effective Practice and Organization of Care (EPOC) group4 and by the Grading Recommendations for Assessing Determining Evidence (GRADE) Working Group.5 These approaches advise authors to use specific terminology that reflects judgments about the magnitude of effect and certainty in evidence.

A workgroup from the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) program developed a roadmap for making narrative statements to guide the EPC program in making narrative statements for findings from their reviews.6 During the development of this roadmap, the workgroup found that the existing guidance uses different underlying constructs to make narrative statements, but these constructs were not explicitly identified. The group conceptualized five constructs that are described in this paper with illustrative examples. We believe the explicit identification of these constructs makes the process of developing narrative statements more transparent and may reconcile differences between existing approaches.

METHODS

The EPC program established a workgroup that included methodologists with expertise in evidence synthesis and research methods to develop guidance on writing narrative statements for systematic reviews. Using several systematic review examples, the workgroup started by piloting the EPOC approach.4 The workgroup also sought feedback from the AHRQ Learning Health System Panel, comprised of 11 decision-makers from large health systems in the USA, to gauge their opinions on the use of plain language for key results in an Evidence Summary. Finally, the workgroup thematically analyzed the EPOC4 and GRADE5 approaches to identify the underlying constructs that can underpin a narrative statement about treatment effects. Experts in systematic reviews or health policy were the primary contributors to this project and patients or members of the public were not engaged in the design or conduct of this work. Additional details about the methodology, workgroup, and roadmap are published elsewhere.6

RESULTS AND DISCUSSION

Five constructs are depicted in Figure 1. For illustration, we proposed narrative statements based on results from two EPC systematic reviews7, 8 using various combinations of the five constructs (see Table 1). The examples include underlined text that refers to the constructs applied.

Figure 1
figure 1

Overlapping constructs that systematic reviewers can use when developing narrative statements about treatment effects.

Full size image
Table 1 Examples of Narrative Statements Using the Different Constructs
Full size table

Constructs That Can Be Used to Describe Treatment Effects

When crafting narrative statements, several domains of an effect can be considered:

  1. 1.

    Direction: whether the intervention increases or decreases the risk of the outcome. The direction of effect could also be described as “no or very small effect.” This statement, however, requires making a judgment that the effect is very small and not clinically important, thus overlapping with the constructs of size of effect and clinical importance.

  2. 2.

    Size: a quantitative estimate of the size of the effect that is independent of the outcome’s importance. For example, relative risks larger than 2 and 5 have been suggested to indicate a large and a very large effect size; respectively.9 A standardized effect size of 0.2, 0.5, and 0.8 may indicate a small, moderate, and large effect size, respectively.10 Therefore, an effect can have a numerically large size even if the outcome was infrequent or not important (e.g., rare events or minor side effects). Narrative statements about the size of effect can also describe absolute changes in binary outcomes (e.g., risk difference or number needed to treat, proportion of responders) or continuous outcomes (e.g., a very large reduction in a laboratory parameter), recognizing that for a given relative effect, absolute effects will vary across baseline risks.

  3. 3.

    Clinical importance: whether the impact of effect is meaningful or consequential to specific key stakeholders (patients, policy makers, providers, health care systems). Another term that has been used to describe clinical importance is importance of the effect.4 To make a judgment about this construct, one needs to make a judgment about the size of the effect and another judgment about the importance of the outcome. Thus, we differentiate between the clinical importance of the change in the outcome, and the importance of the outcome itself. Clinically important effects for a given outcome may vary across stakeholders and interventions, as well as across population subgroups (e.g., age, disease severity, comorbidities, preferences/values). Clinical importance is sometimes known (e.g., a minimal clinically important difference [MCID] of a scale11 defined as an effect noticeable to the individual). In other instances, it may be more subjective—and either statistically, arbitrarily, or consensus derived (e.g., a standard deviation or 25% relative improvement in symptom scale scores or a 5 percentage point reduction in mortality may be assumed to be important). When information about clinical importance is unknown or poorly validated, investigators may choose to not use this construct when developing narrative statements. The term clinical importance is appropriate in systematic reviews about clinical topics. However, other terms can be used in non-clinical settings, such as public health importance for example in systematic reviews about public health topics.

  4. 4.

    Statistical significance: narrative statements can include the construct of statistical significance, and refer to the results as statistically significant or not significant. Inference about statistical significance has been made from p values or confidence intervals. The interpretation of statistical significance will differ across systematic reviews based on the specific assumptions and inference frameworks.

  5. 5.

    Strength or certainty of evidence: a global judgment about the certainty in estimates across five domains: risk of bias, directness, precision, consistency, and likelihood of publication bias. The strength of evidence is rated in EPC reports as high, moderate, low, and insufficient. Narrative terms that may be used to correspond to these ratings, respectively, would be declarative statements without adverbs or auxiliary verbs “the intervention reduces mortality,” “probably or likely reduces,” “may reduce,” and “it is uncertain whether the intervention reduces…”.

LIMITATIONS

These constructs were identified by clinicians, health systems, and methodologists involved in the AHRQ EPC program. Patients and care givers may have different needs for information in narrative statements that would need to be addressed to inform them about research findings. It is also clear that these constructs are inter-related and greatly overlap. For example, the size of an effect can affect the judgment about clinical importance. Clinical importance and size of effect may affect the determination of the direction of effect (e.g., differentiating between no effect or small effect). Statistical significance is related to precision of the estimate, which in turn is one of the domains that determine the strength of evidence construct. Judgments about the clinical importance and size of effect constructs are needed to determine the precision and consistency domains of the certainty construct. Despite the overlap, it remains important to recognize these constructs explicitly when developing narrative statements. Lastly, information needed to make judgments about clinical importance are occasionally unavailable, not validated, or not defined a priori. Thus, this construct may not be always feasible to implement.

CONCLUSIONS

Narrative statements that accurately summarize the findings of research synthesis can enhance dissemination and implementation of findings. Narrative statements can be one of the tools that start the process of risk communication and subsequent shared decision-making.12 Statements solely based on statistical significance lack depth and contextual information necessary for decision-making, and may also be misleading. We have explicitly identified five overlapping constructs that can be used to describe treatment effects. These include direction of effect, size of effect, clinical importance, statistical significance, and strength or certainty of evidence. These constructs can be applied to narrative statements about findings from primary studies as well as evidence syntheses. We encourage investigators to choose the constructs that best fit their context and audience. These constructs do not replace or contradict the EPOC4 or GRADE5 approaches, rather, the constructs can facilitate and operationalize the implementation of these two approaches.