Types of studies

We will consider three types of study design where effects of exposure to 'Summary of findings' tables of systematic reviews of the effects of healthcare interventions on one or more outcome is measured:

• randomised trials;

• non‐randomised trials;

• cross‐over trials.

We will follow the Cochrane Effective Practice and Organisation of Care (EPOC) Group definitions of these experimental study types (EPOC 2013a). We will include both published and unpublished studies. We anticipate few randomised trials on this topic because 'Summary of findings' tables are a relatively new intervention. Therefore, we have broadened our inclusion criteria to include the above‐mentioned study types to help us determine the potential of 'Summary of findings' tables to communicate key findings of systematic reviews.

Types of data

We will include data from published, unpublished and grey literature comparing standard/static or interactive 'Summary of findings' (i'Summary of findings') tables or both, as described by GRADE (Guyatt 2011; Guyatt 2013a; Guyatt 2013b; Agoritsas 2015) with other types of summaries of systematic reviews.

We will include studies that recruit any participant type that uses 'Summary of findings' tables of systematic reviews including: patients, families/carers, healthcare professionals, policy makers, health systems managers, systematic review authors or other stakeholders.

Types of methods

We will include studies that compare:

• the effects of 'Summary of findings' tables versus full versions of systematic reviews on communicating key findings of systematic reviews of the effects of healthcare interventions;

• the effects of 'Summary of findings' tables plus full review versus full review (no 'Summary of findings' tables);

• the effects of 'Summary of findings' tables versus other summaries of systematic reviews on communicating key findings of systematic reviews of the effects of healthcare interventions;

• the effects of interactive 'Summary of findings' tables versus static 'Summary of findings' tables on communicating key findings of systematic reviews of the effects of healthcare interventions.

Types of outcome measures

Electronic searches

We will run electronic or manual searches of the following online resources:

• Electronic databases: the Cochrane Library, the Campbell Collaboration, PubMed, CINAHL, LILACS, Web of Science, SCOPUS, Embase, PsycINFO, Epistemonikos.

• International trials registers such as the Cochrane Central Register of Controlled Trials (CENTRAL), PROSPERO, ClinicalTrials.gov, WHO International Clinical Trials Registry Platform (ICTRP) portal.

• Grey literature sources such as reports/dissertations/theses databases and databases of conference abstracts e.g. Cochrane Colloquium abstracts, ETHOS, OpenGrey, ISI Web of Knowledge and websites of key organisations e.g. GRADE, Epistemonikos.

Searching other resources

Reference lists

We will search reference lists of included studies and similar systematic reviews to find additional relevant resources.

Correspondence

If deemed appropriate, we will contact individuals or groups known to have experience or knowledge of 'Summary of findings' tables e.g. researchers, review authors, members of the Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence (DECIDE) collaboration, GRADE Working Group, and the Cochrane Applicability and Recommendations Methods Group to identify and locate additional resources or studies which have not yet been published or are not readily accessible.

Selection of studies

Two review authors (Aislinn Conway (AC)) and (Declan Devane (DD)) will independently screen titles and abstracts of all citations identified by searches against inclusion criteria based on types of studies, types of interventions and participants. The citations will be sorted into the following groups; 'include', 'full‐text review' and 'exclude'. Both authors will review full versions of papers where it is unclear whether prespecified eligibility criteria have been met. If, after discussion, there is still disagreement regarding study selection, a third review author (Mike Clarke (MC)) will be provided with a full‐text copy of the article for comment and judgement as to whether to include. Reference management software will be used to import all references from databases and other print and electronic sources into a single place accessible to authors.

Data extraction and management

Two review authors (AC and DD) will independently complete tailored data extraction forms for each of the studies. We will discuss discrepancies and if resolution is not reached, we will consult a third author.

Items extracted will include the following.

• Authors

• Year of Publication

• Language

• Setting

• Country

• Study design

• Participants:

  • Professional or non‐professional group e.g. patients

  • Level of experience using 'Summary of findings' tables

• Intervention:

  • Characteristics of intervention e.g. format, timing, setting

• Comparison:

  • Details of comparison intervention

• Outcomes:

• • User understanding of key findings of systematic reviews measured by the ability to correctly answer factual questions about the review

  • Self‐perceived understanding of key findings of systematic reviews as reported by the user

  • Self‐reported influence on decision‐making

  • Time taken to read summary and extract relevant information

  • Accessibility of the main findings of the review

  • User satisfaction/preferences/attitudes

  • Length of time during which outcomes were measured after initiation of the intervention

  • Whether follow‐up occurred, if so, length of follow‐up and follow‐up points

• Data to assess the risk of bias of included studies e.g. sequence generation, allocation concealment, blinding of study participants and personnel, blinding of outcome assessors, withdrawals or incomplete outcome data, selective reporting or other sources of bias

• Funding sources

Assessment of risk of bias in included studies

Two review authors (AC and DD) will assess the risk of bias for each study independently. We will use the criteria described in the Cochrane ‘Risk of bias’ criteria (Higgins 2011) and in section 6.4 of the Data Collection Checklist (EPOC 2010) for randomised trials and the Cochrane EPOC Review Group guidance on risk of bias criteria (EPOC 2015) and the Cochrane EPOC Review Group guidance (EPOC 2013b) if our review includes more than one study design. Our inclusion of non‐randomised studies brings a greater potential for bias (Higgins 2011). We will contact study authors when information is missing or if clarification is required. Two review authors will apply the 'Risk of bias' criteria to each study independently and differences will be resolved by consulting a third review author (ST).

The following criteria are recommended for randomised trials (RTs), non‐randomised trials (NRTs) and cross‐over studies.

Measures of the effect of the methods

Dichotomous data (correct/incorrect answers on tests of understanding of key findings of systematic reviews) will be determined using a risk ratio (RR) with a 95% confidence interval (CI).

Ordinal scale data outcomes reported in this way will be collapsed into dichotomous outcomes.

Continuous data will be analysed using mean difference (MD) with the 95% CI if the measurement scale is the same. If the scale is different, standardised mean differences (SMD) with 95% CIs will be used.

Unit of analysis issues

Randomised trials will be analysed separately from the other types of study design.

Cluster‐randomised trials included in the systematic review will be identified as such. We will report the baseline comparability of clusters and consider statistical adjustment if it may help to reduce an imbalance.

We will estimate the intracluster correlation coefficient (ICC) as described by Higgins 2011 using information from the study if it is available or, from an external estimate obtained from a similar study. If we do this, we will conduct sensitivity analyses to explain variation in ICC values.

Studies with multiple intervention groups we will include and analyse groups which are relevant to our review. However, all intervention groups will be clearly listed in the ‘Characteristics of included studies’ table. To avoid “double counting” data for studies that could contribute more than one control group, we will combine comparison groups to create a single pair‐wise comparison (Higgins 2011).

Dealing with missing data

We will contact authors when a gap is identified in studies. If we decide that there may be reasons to impute missing data e.g. to explore the impact of missing data in the sensitivity analysis, we will discuss the potentials harms and benefits of this. If the missing data are substantial, analysis with imputed data may be futile.

We will narratively explore the potential impact of missing data in the discussion section of the review.

Assessment of heterogeneity

We have specified that we will include non‐randomised trials in this review which may lead to increased statistical heterogeneity. We will assess heterogeneity by visually inspecting a forest plot of included studies. The location of point estimates, the degree to which confidence intervals overlap and the presence and results of meta‐analysis will be taken into account. Next, we will test for the presence of heterogeneity using the Chi² test. If the P value is low (less than 0.10), the likelihood of heterogeneity will increase.

We will quantify the extent of heterogeneity by calculating an estimation of the I² statistic. We will follow the guidance outlined in Section 9.5.2 of the Higgins 2011:

• 0% to 40%: might not be important;

• 30% to 60%: may represent moderate heterogeneity*;

• 50% to 90%: may represent substantial heterogeneity*;

• 75% to 100%: considerable heterogeneity*.

*The importance of the observed value of I² depends on (i) magnitude and direction of effects and (ii) strength of evidence for heterogeneity (e.g. P value from the Chi² test, or a confidence interval for I²). If our I² value indicates that heterogeneity is a possibility and either the Tau² is greater than zero, or the P value is low (less than 0.10), heterogeneity may be due to a factor other than chance.

If we identify methodological or statistical heterogeneity, we will not pool results into a meta‐analysis. Instead we will carry out a narrative synthesis, grouping trials with similar populations and interventions together to attempt to identify reasons for heterogeneity.

Assessment of reporting biases

If 10 or more studies are included in a meta‐analysis, we will create a funnel plot to investigate whether bias may exist unless all studies are of a similar size. We will use the funnel plot test proposed by Egger 1997. If we notice asymmetry we cannot conclude that reporting biases exist however, we will consider the sample sizes and presence and possible influence of outliers. We will discuss potential explanations such as publication bias or poor methodological quality of included studies and subsequently perform a sensitivity analysis.

Data synthesis

We will use Review Manager software (RevMan 2014) to conduct our statistical analysis and undertake meta‐analysis if it is deemed appropriate. Considering the differences in the participant groups, the comparisons and the outcomes in this review, we will use a random‐effects model. The pooled estimate of the effects will estimate the mean effects across the groups, comparisons and methods of outcome evaluation. Both within‐study and between study variability will be addressed.

If we do not deem it appropriate to conduct meta‐analyses we will present a systematic, narrative summary of the results.

'Summary of findings' table

Two review authors (AC, DD) will assess the quality of the evidence. Based on the methods described in Section 8.5 of chapter 12 of the Cochrane Handbook for Systematic Reviews of Interventions (Higgins 2011) and by GRADE (Guyatt 2013a; Guyatt 2013b), we will create 'Summary of findings' tables for the main comparisons of the review: 'Summary of findings' tables versus full versions of systematic reviews; 'Summary of findings' tables plus full review versus full review (no 'Summary of findings' tables); 'Summary of findings' tables versus other summaries of systematic reviews; and interactive versus static 'Summary of findings' tables.

We will present the following primary and secondary outcomes for each comparison: user understanding of key findings of systematic reviews, self‐perceived understanding of key findings of systematic and self‐reported influence on decision‐making, time taken to read summary and extract relevant information, accessibility of the main findings of the review, user satisfaction/preferences/attitudes and other outcome(s) of main interest, as outlined in the section on Types of outcome measures. We will describe the study settings and number of participants and studies addressing each outcome. For each assumed risk cited in the table(s), we will provide a source and rationale, and the GRADE system will be used to assess the quality of the evidence using GRADEpro software or the GRADEpro GDT app. If meta‐analysis is not appropriate or the units of analysis cannot be compared, we will present results in a narrative 'Summary of findings' table format (using Chan 2011 for guidance). If we do this, the imprecision of the evidence will be an issue of concern due to the lack of a quantitative effect measure.

Subgroup analysis and investigation of heterogeneity

If visual inspection of forest plots, Chi² test, I² statistic and Tau² indicate that statistical heterogeneity could be present, a subgroup analysis will be carried out.

A subgroup analysis will be deemed appropriate if included studies satisfy criteria to assess credibility of subgroup analyses (Oxman 1992; Sun 2010).

The following are our a priori subgroups:

• different participant groups e.g. patients, policy makers or healthcare professionals;

• intervention characteristics e.g. different formats of 'Summary of findings' tables, different summarisation products;

• type of study.

Sensitivity analysis

We will use the GRADE approach (Guyatt 2008) to assess the level of quality of the evidence and thereby, interpret the results. This involves the GRADE classification terms: high, moderate, low or very low. GRADE is characterised by eight criteria for authors to consider (Schünemann 2013).

• Risk of bias (potential to reduce level of quality of evidence by one or two levels)

• Inconsistency of results (potential to reduce level of quality of evidence by one or two levels)

• Indirectness of evidence (potential to reduce level of quality of evidence by one or two levels)

• Imprecision of results (potential to reduce level of quality of evidence by one or two levels)

• Risk of publication bias (potential to reduce level of quality of evidence by one or two levels)

• Magnitude of effect (potential to increase level of quality of evidence by one or two levels)

• Dose response gradient (potential to increase level of quality of evidence by one level)

• Influence of residual plausible confounding (potential to increase level of quality of evidence by one level)

We will downgrade randomised trials by one, two or three levels according to the severity of the study limitations (the first five factors listed above). We will upgrade non‐randomised trials if their results show large effects and bias is not evident, or we will downgrade them if they demonstrate limitations as listed above.

We will use The GRADEpro Guideline Development Tool to create a 'Summary of findings' table incorporating our results.