A pilot study assessing the similarity between core outcome sets and outcomes included in health technology assessments

Introduction

Clinical trials are performed to evaluate the effects of treatment interventions, with the gold standard being randomised controlled trials.¹ Fundamentally this includes a test and a control treatment, with random assignment of treatment groups and at least one outcome measure.² Researchers decide on outcomes which best answer their research question, and consequently differences arise between trials in the same field as the chosen outcome measures are often of particular relevance to each study. The lack of uniformity amongst trial designs is abundant, demonstrated by a review of 8942 oncology trials which revealed that over 25,000 outcomes occurred only once or twice³; furthermore the top cited and most accessed Cochrane reviews in 2009 described problems due to inconsistences in the reported outcomes in included studies.⁴ Discrepancies are also observed when examining how outcomes are measured; for example, a survey of 10,000 trials investigating schizophrenia discovered 2194 different measurement scales were employed.⁵ This produces a substantial challenge in data analysis, limiting the ability to compare studies, synthesise the available evidence and perform meta-analysis, ultimately leading to avoidable research waste.⁶ There are also questions concerning whether trial outcomes are always relevant to patients or clinicians, meaning statistically significant results may have limited clinical bearing, hence not translating into improved clinical care.⁷

In January 2010 the Core Outcome Measures in Effectiveness Trials (COMET) Initiative was launched with the aim of bringing together people interested in the development and application of core outcome sets (COS). COS address the issue of inconsistency and outcome reporting bias, while reducing the difficulties facing systematic reviewers due to heterogeneity in outcome measures.⁸ COS are an agreed standardised collection of outcomes which should be reported as a minimum for all trials in the corresponding clinical area. The Core Outcome Set-STAndard for Development (COS-STAD) provides criteria against which to assess the quality of COS.⁹ Researchers are not restricted to the COS, but there is an expectation it will always be reported.⁸ The importance of COS has become increasingly acknowledged over time, with COMET endorsed by trial funders such as the National Institute for Health Research (NIHR), the Cochrane Collaboration and National Institute for Health and Care Excellence (NICE) . There is currently ongoing work assessing the representation of COS in US Food and Drug Administration (FDA) and European Medicines Agency (EMA) regulatory guidance.¹⁰

Health technology assessments (HTAs) are multidisciplinary processes which use explicit methods to determine the value of a health technology at different points in its lifecycle.¹¹ They are produced to inform decision makers and promote an equitable, efficient and high-quality health system.¹¹ The technologies are often interventions designed to prevent, diagnose or treat medical conditions. The International Network of Agencies for Health Technology Assessments (INAHTA) is a network of 51 different HTA agencies which support healthcare decision making. This network produces the international HTA database, which provides a free, single point of access to information about ongoing and published HTAs.

Recent work has assessed the International Consortium for Health Outcomes Measurement (ICHOM) outcome sets against oncology HTAs, finding that HTAs tend to focus on generic measures which allow comparison across disease areas. However, it should be noted that these outcome sets are developed for routine care, unlike COS which are produced specifically for research so are more relevant for HTAs.¹² Additionally, a review of technology appraisal oncology scopes found that in the majority of cases there was complete overlap with those outcomes in COS; in a small number of exceptions the COS included two additional outcomes to those specified in the scope.¹³ Other work has investigated a trial funder and member of INAHTA, the National Institute for Health Research Health Technology Assessment (NIHR HTA), examining their recommendation for applicants to search for a COS to include in their trial. By examining research applications and then surveying applicants it was found that 38% (36/95) searched for a COS through either the COMET database or another method, e.g. review of the literature; where a published COS existed it was included in 29% (7/24) of cases.¹⁴ There is reason to believe that uptake may have increased, as a limitation was that no COS existed for 68 of the 95 studies assessed at the time of submission.¹⁴ However, this study assessed the uptake of COS in trials rather than HTAs, and to our knowledge, ours is the first study to examine the similarity between COS for research and HTA outcomes. There is potential for COS to further improve the quality of HTAs, providing more uniform and patient-centred evidence to inform clinical guidelines and policy makers, such as NICE.

The purpose of this study was to assess the similarity between COS and outcomes included in HTAs.

Methods

The COMET database, a database of studies relevant to the development of COS, was searched to select individual COS for this review. As a surrogate measure to ensure quality, only COS published between 2015 and 2019 where patients were included were selected, maximising the number of COS-STAD standards met.⁹ The methodology of producing COS has improved with time, hence this 5-year period was chosen to include the most recent COS, which had a greater likelihood of including outcomes most relevant to all appropriate stakeholders. Additionally, COS developed for common diseases and interventions were selected to ensure sufficient HTAs were available for assessment. COS outcomes were extracted by one author (PC) from a pre-existing COMET database of outcomes from all COS for research published up to 2019.

The INAHTA database was searched in February 2021 for relevant HTAs, using the disease name as the search term with the English language filter applied. Where an intervention was specified by the COS this was included as a search term, e.g. obesity AND surgery. Quotation marks were used to refine searches that returned a large number of HTAs or when the results lacked specificity to the desired condition, e.g. searching type 1 diabetes produced HTAs relating to type 2 and gestational diabetes. There were no restrictions by year or country. MeSH terms were employed when there were variations in how search terms were spelt, for example postpartum haemorrhage versus postpartum hemorrhage.

HTAs were accessed using the hyperlinks provided in the INAHTA database. Where the hyperlink did not direct to the HTA but to the publisher’s website, this was searched using the title of the HTA. If a hyperlink did not work, the abstract in the INAHTA record was screened for outcomes if available. HTAs which were irrelevant to the COS, inaccessible (e.g. due to a non-functional hyperlink or no hyperlink was present) or which summarised clinical evidence with no outcome measures stated were excluded. Outcome measures were then extracted from the HTA alongside study identifier, year and data source. Data extraction involved creating a table comprising the core outcomes as headings with each HTA listed below. For each HTA, outcomes were matched to the appropriate headings and then colour coded green, yellow or red to indicate specific, partial or no match respectively. For partial matches, bold and italic text indicated an outcome more general or specific than the COS respectively.

Outcomes were categorised as either a specific match, a partial match or no match compared to the COS, following an approach used previously.^15,16 We considered a match between an HTA outcome and a core outcome to exist if they were either specifically or partially related. We defined a specific match as one where both outcomes corresponded to each other exactly, while a partial match was defined as one where the outcomes correspond to each other non-specifically. Taking the example of swollen joint count as a core outcome, if the HTA included swollen joint count as an outcome it would be considered a specific match, whilst disease activity would be considered a partial match. Partial matches were further categorised according to whether the HTA outcome was either more general or more specific than the COS outcome. The outcome matching process was quality checked by a seconder reviewer (SD). The data extracted from each HTA included the HTA identifier, publisher, publication year and outcome measures. Outcomes were ascertained from stated outcomes of interest or from the PICO (population, intervention, comparator, outcomes) statement. Where outcomes were not explicitly stated, the HTA was reviewed to elicit them and on occasion outcomes had to be ascertained from a series of key research questions. Outcomes identified in the scope of an HTA which could not be reported due to a lack of available data were included in the analysis. Considering the research question “what is the expected beneficial effect of atezolizumab on mortality?” as an example, this was deemed a specific match for overall survival and a partial match for cancer-specific survival.

Within the UK, NICE HTAs are of particular importance. It was noted that no NICE HTAs had been registered on the INAHTA database beyond 2011, so in April 2021, the NICE website was searched for HTAs corresponding to each of the COS from the original cohort. Non-oncology conditions were selected given previous work has examined uptake across oncology HTAs.^12,13 and where no appropriate NICE HTAs were identified, alternative COS were selected, using the same criteria of publication between 2015 and 2019 with patient involvement. The most recently published reports which matched a COS were included, with outcomes compared to core outcomes from the corresponding COS.

The results are presented through tables, graphs and using descriptive statistics. The analysis was performed using Excel 2019. Comparison between the INAHTA and NICE results was not examined given they cover different time periods and utilised different COS.

Results

Ten COS were selected to assess against the INAHTA database. The search terms for each are displayed in Table 1. The search relating to the relapsing remitting multiple sclerosis and clinically isolated syndrome COS was performed as two individual searches to ensure all relevant HTAs were identified. The database search returned a cumulative total of 1057 HTAs, ranging from three for clinically isolated syndrome to 451 for breast cancer. A selection of 10% (45) of the breast cancer results were sampled, with the most recently published HTAs chosen as they were most likely to be accessible, detailed in Figure 1. This selection of HTAs comprised the most recently published HTAs until the 10% sample was reached, encompassing HTAs published between 2015 and 2020. This produced a possible 651 HTAs to be included in the report. Eight of the included COS applied to any intervention, with metabolic and bariatric surgery specified for obesity and treatment specified for postpartum haemorrhage. For each of the ten COS, the majority of the HTAs were not assessed as the hyperlinks listed in the INAHTA database were non-functional, detailed in Table 2. There were several HTAs irrelevant to the COS which were therefore excluded. Other reasons for exclusion included diagnostic accuracy study, HTA where no hyperlink was available, HTA not in English, HTA recommendations, HTA inaccessible on the source website, duplicates and where an updated version of the HTA had already been assessed. Systematic reviews provided over 90% of studies included, the remainder being trials. To minimise similarity to COS being underestimated by reviews, through outcome reporting being limited to what data exists in the literature, outcomes stated in the methods were included in the analysis, independent of the findings.

Figure 1. HTA identification and selection flowchart.

COS, core outcome set; HTA, health technology assessment.

Overall, 119 HTAs were assessed against core outcomes with a maximum of 1318 matches possible, shown in Table 3. In total, there were 562 (43%) core outcome matches, 413 (31%) specific and 149 (11%) partial, with the percentage values rounded to whole numbers. Rheumatoid arthritis had the greatest percentage of specific matches (52%), postpartum haemorrhage returned the greatest percentage of partial matches (50%) while epilepsy and psoriasis both had the greatest percentage of no matches (67%). Only rheumatoid arthritis, type 1 diabetes, acne and postpartum haemorrhage had more total core outcome matches compared to no matches. The number of HTAs assessed ranged from one for acne to 22 for rheumatoid arthritis. The COS for rheumatoid arthritis had the greatest median value of matches with 69% while epilepsy had the fewest with a median of 25%, detailed in Table 4.

To consider how the scope of the HTA compared to that of the COS, each HTA was assessed with regards to the investigated population and intervention. Table 5 shows the majority of the studies clustered around exact scope matches and the COS being broader, both in terms of the population and intervention.

The distribution of HTAs by year ranged from 1998 to 2020, with 69% of included HTAs published in 2012 and beyond. The most HTAs published in a single year was 13 in 2012. A mean of nine HTAs were published per year in the following years between 2013 and 2020. Figure 2 shows from 2009 onward there has been some fluctuation in the degree of match, though ultimately the trend is stationary, remaining close to the mean of 43%.

Figure 2. Percentage of outcome matches by year.

A drawback of these results is that many HTAs were produced prior to publication of the COS. To establish a clearer view on the impact of COS, a subgroup analysis consisting of HTAs developed following publication of the COS was undertaken. This comprised 25 HTAs across 8 of the selected COS. From a possible 373 there were 163 outcome matches, encompassing 109 specific matches and 54 partial matches. Accordingly 44% of HTA outcomes were covered by the COS, ranging from 52% in 2016 to 41% in 2019, with a trend of marginal decrease illustrated in Figure 3. This may reflect the fact that, although the clinical trials community are becoming more aware of COS, they are yet to be adopted by the HTA community.

Figure 3. Percentage of outcome matches by year for HTAs developed after COS publication.

To assess the similarity between COS and recently published NICE HTA outcomes, the NICE website was searched for HTAs corresponding to each of the COS from the original cohort, except breast cancer. NICE HTAs were only identified for four COS, so six alternative COS were selected to include in the analysis. Ten non-oncology HTAs published between 2019 and 2021 were identified from the NICE website and compared against a relevant COS, shown in Table 6. HTA matches ranged from 44% to 100%, with a total of 78% (73/94) outcome matches with NICE HTAs, including 57 (61%) specific matches and 16 (17%) partial matches.

Discussion

This review found that HTAs in the INAHTA database included 43% of core outcomes from COS, with specific matches accounting for 31% and partial matches for 11%.

Conclusion

This novel piece of research includes, to our knowledge, the first comparison of non-oncology HTAs and COS for research. This pilot study highlights that further work is needed to promote awareness and implementation of COS within HTAs, with less than half of core outcomes currently measured. The degree of matching between COS and NICE HTA outcomes, albeit from a small sample, suggests this is both practical and achievable with the additional benefit of increasing patient representation, a particular challenge for HTA agencies. Incorporation of COS will allow HTA findings to better reflect the opinions and preferences of all relevant stakeholders.

Data availability statement