A call to action to harmonize patient-reported outcomes evidence requirements across key European HTA bodies in oncology

Historically, key end points in oncology trials were restricted to survival and tumor reduction [1]. However, in the era of patient-focused drug development, developers of new drugs and biologics for the treatment of cancer are increasingly collecting information directly from patients about outcomes that are important to them in defining treatment success using patient-reported outcomes (PROs) [1]. PROs are reports about the status of a patient's health condition or treatment that come directly from the patient [2]. PROs can measure a variety of concepts, including health-related quality of life (HRQoL), functioning (e.g., physical, psychological and social), symptoms (including disease and treatment related), patient satisfaction, treatment adherence and utility [2,3]. It is a unique way of capturing how the patient feels or functions, which may not be fully captured through traditional end points such as overall survival (OS), progression-free survival (PFS), biomarker measures or adverse events (AEs) [2]. The European Medicines Agency (EMA) [2], US FDA [4], International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) [5] and the UK's Medicines and Healthcare Products Regulatory Agency (MHRA) [6] all consider PRO data derived robustly using fit-for-purpose PRO instruments as key components of decision-making during the benefit–risk appraisal of new drugs and biologic products in oncology.

PROs provide key perspectives from patients to inform the benefit–risk profile of new products, especially in oncology trials in which benefits are often based on surrogate end points when OS data are immature or in which treatment benefit is associated with additional safety risks [7–9]. Here, PROs are used as secondary or exploratory end points and, in some cases, in the statistical testing hierarchy to measure the impact of drug from the patient's perspective [10].

PROs have been reported not to play such an important role in health technology assessments (HTAs), however [11]. Most HTA bodies assess new drugs using comparative clinical benefit assessment and health economic assessment [12–14], but the opportunities for PRO data within these assessments is not entirely clear or consistent across HTA bodies. Given the increase in PRO data generation by developers of new drugs and biologics for the treatment of cancer, it is important that these developers have clear and consistent guidance as well as precedence from HTA bodies to inform their PRO strategy, implementation, analysis and submissions. The authors evaluated method guidance published by the HTA bodies in the three largest European oncology markets with well-established HTA processes – Germany's Gemeinsamer Bundesausschuss (G-BA) and Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (IQWiG), the UK's National Institute for Health and Care Excellence (NICE) and France's Haute Autorité de Santé (HAS) – and critically appraised HTA reports of selected case studies in both solid and hematological tumors to identify the similarities and differences in how PROs were assessed by the HTA bodies.

Methods

Method guidance documents and HTA oncology case studies from G-BA/IQWiG, NICE and HAS were analyzed to evaluate published information relating to several areas of interest, including trial design considerations such as blinding and multiplicity adjustment requirements for PROs, PRO data collection and reporting requirements such as preferred instruments, missing data acceptance threshold, post-progression data collection and PRO data analysis such as clinically meaningful change threshold, preference for time-to-event end point definitions and missing data sensitivity analysis.

As a first step, HTA method guidance, position papers and technical support documents were identified from the HTA bodies' websites and searched for any information relating to PROs, using keywords including ‘patient-reported-outcomes,’ ‘PRO,’ ‘PROs,’ ‘quality of life,’ ‘HRQoL,’ ‘EQ-5D,’ and ‘patient experience.’ The HTA method guidance, position papers and technical support documents with details on PROs were then reviewed to evaluate areas of interest described above. Table 1 shows a list of method guidance that contain information related to PROs.

The second step involved analysis of published HTA reports of selected case studies by the four HTA bodies between October 2015 and November 2020. This analysis was conducted between December 2020 and February 2021 using IQVIA's proprietary database HTA Accelerator [26], which was used in multiple peer-reviewed publications [27–29]. HTA Accelerator contains more than 36,000 HTA report publications covering 40 countries and 100 HTA bodies published since 2011. Data in HTA Accelerator are collated from publicly available HTA reports and structured into a framework of 250 available data elements that also includes information related to PROs. Details related to evaluation of the PRO evidence in these HTA reports on the topics of the role of PRO data in HTA as well as trial design considerations and PRO data collection, reporting and data analysis recommendations as noted earlier were captured. On the basis of these available details, 20 case studies covering more common indications of interest to the study sponsor in both solid and hematological cancer were selected, including lung cancer, ovarian cancer, cervical cancer, breast cancer, prostate cancer, melanoma, multiple myeloma and non-Hodgkin lymphoma. After excluding non-submissions by companies and ongoing assessments by HTA bodies, a total of 70 published assessments across the four HTA bodies for the 20 case studies were retrieved (Appendix Table 1 & 2).

Results

HTA method guidance

HTA bodies in Europe follow different assessment methodologies, mostly based on clinical comparative effectiveness and cost–effectiveness analysis [12–14]. The details on trial design, PRO data collection, reporting and analysis in the method guidance for the 4 HTA bodies in scope are summarized in Table 2.

Role of PRO data in HTA

The impact of PROs on HTA decision-making is clearest in Germany. In assessments by the IQWiG and G-BA, PRO data alone can result in added benefit based on patient-relevant end points such as morbidity and HRQoL [30,31]. This was observed in seven of the 20 case studies reviewed in this study. For example, in the assessment of crizotinib for advanced anaplastic lymphoma kinase (ALK)-positive non-small-cell lung cancer (NSCLC), a hint of minor added benefit in the final G-BA assessment was supported by significant improvement in symptom scales of European Organization for Research and Treatment of Cancer 30-item QoL Questionnaire – Core 30 (EORTC QLQ-C30) and European Organization for Research and Treatment of Cancer 13-item module, the QoL Questionnaire – Lung Cancer 13 (EORTC QLQ-LC13) as well as HRQoL scales of EORTC QLQ-C30. This is despite being an open-label study and OS data not being statistically significant versus cisplatin in combination with pemetrexed or carboplatin in combination with pemetrexed [32]. Details of other case studies are included in Supplementary Appendix Table 3.

In the UK, NICE's decision-making is driven by cost–effectiveness analysis (CEA) in which the EQ-5D: EuroQol – 5 Dimension (EQ-5D) is a key input to determine health state utilities [22]. For example, in the assessment of pembrolizumab for advanced/unresectable metastatic melanoma, although both EORTC QLQ-C30 and EQ-5D data were collected by the manufacturer in the KEYNOTE-006 trial, only EQ-5D data were used by NICE in its appraisal to derive utility values. In comparison, in the G-BA assessment, minor improvements in fatigue, nausea and vomiting scales and the social functioning scale as measured by the EORTC QLQ-C30 supported a considerable added benefit rating in some subgroups. In the NICE report, details on the impact of PRO data other than the EQ-5D to inform the CEA are limited [33,34].

HAS Transparency Committee guidance states that besides efficacy and safety data, evidence demonstrating an improvement in HRQoL can lead to a higher clinical added value rating [19]. Its importance is also stated in the early access HAS doctrine, in which HRQoL is included as a part of the efficacy demonstration [20]. However, in oncology HTAs, PRO data are often not considered by HAS due to several methodological issues [11]. In the case studies analyzed, PRO data were considered only in two cases: abiraterone in metastatic castration-sensitive prostate cancer (mCSPC) and castration-resistant prostate cancer (mCRPC) [35,36]. In the mCSPC assessment, median time until deterioration of quality-of-life measured using FACT-P was longer in the abiraterone group (12.9 months) versus placebo (8.3 months). However, HAS commented that the quality-of-life data measured via FACT-P and other instruments such as BPI-SF and BFI should be interpreted with caution due to the exploratory nature of the analysis and the large number of missing data, for which approximately 50–70% of the questionnaires were completed during treatment, depending on the group from cycle 17 onward [35]. In the mCRPC assessment, the median time until deterioration of quality of life measured using FACT-P was longer in the abiraterone group (12.7 months) versus placebo (8.3 months) [36]. Of the other 17 case studies analyzed, PRO data were rejected in 13 cases due to open-label design, missing data or exploratory nature of HRQoL analysis with no method of controlling alpha risk inflation related to multiplicity. PRO data were not mentioned in four cases [26].

Specific requirements for HTA body to accept PRO data

Trial design

Open-label study

Although IQWiG guidance mentions that PRO data from open-label studies are of limited validity, in practice both G-BA and IQWiG accept PRO data from open-label oncology studies to inform benefit ratings. This was observed in all 12 case studies analyzed in this study in which PRO data submitted were from open-label studies. However, the extent of benefit is downgraded to a hint only [15,26]. Similarly, NICE accepts PRO data from open-label studies as observed in all 10 case studies analyzed [26,37]. On the contrary, HAS does not accept PRO data from open-label studies, as double-blinded condition is one of the criteria for quality-of-life data to be considered in clinical added benefit assessment [19]. This is further confirmed by case studies assessed in which PRO data of all 11 case studies from open-label trials were rejected by HAS [26].

Multiplicity adjustment

The need to prospectively plan for multiplicity adjustment for PRO data was not directly mentioned in guidance for G-BA, IQWIG and NICE [15,17,18,22,25]. However, the HAS Transparency Committee guidance mentioned management of multiplicity of analyses as a key factor for acceptance of HRQoL data [19]. In seven case studies researched, PROs were rejected due to a lack of method to control for type I error due to multiplicity. For example, this was one of the key reasons for rejection of PRO data in the assessment of pomalidomide plus bortezomib plus dexamethasone for adult patients with multiple myeloma [38].

PRO data collection & reporting

Preferred instrument

G-BA and IQWiG accept both generic and disease-specific PRO instruments for benefit assessment if they are validated or established. Cost–effectiveness analysis is not a part of G-BA and IQWiG assessments and hence only the visual analog scales of the EQ-5D instrument are considered as a measure of patient-reported morbidity [15,30]. HAS requires the HRQoL benefit to be demonstrated with the use of validated scales appropriate or specific to the objective and prefers EQ-5D or Health Utility Index for cost–utility analysis [19,21,39].

NICE assesses PRO data mostly as part of its economic evaluation to derive utilities and EQ-5D is the preferred instrument. The EQ-5D-3L value set is considered the reference case and all EQ-5D-5L data should be mapped onto the EQ-5D-3L value set using mapping function developed by Van Hout et al. [23,40]. In the absence of EQ-5D data collection in trials, EQ-5D data can be sourced from the literature, if justification for choosing a particular data set is clearly explained. Alternatively, EQ-5D data can be estimated by mapping other HRQoL measures or health-related benefits observed in the relevant clinical trials to EQ-5D [22,23]. For example, in the assessment of carfilzomib plus lenalidomide plus dexamethasone for previously treated patients with multiple myeloma, NICE preferred to use utility values mapped from EORTC QLQ-C30 trial data using a mapping algorithm from Proskorovsky et al. [41,42]. NICE has published a new manual providing a hierarchy of preferred HRQoL methods when EQ-5D is not available or not considered appropriate [24,25].

Missing data threshold

G-BA and IQWiG have specified a threshold for missing data for the PRO data to be accepted [30]. IQWiG stated in the 2020 method guidance that results will generally not be considered in the benefit assessment if the proportion of study participants (intention to treat) not included in the analysis is >30%. For example, in the assessment of nivolumab for unresectable or metastatic melanoma, PRO data collected using EORTC QLQ-C30 and EQ-5D were not considered by G-BA and IQWiG, as the proportion of evaluable patients in the study arms were too low and causes of missing values or sensitivity analysis were not provided [15,43,44]. In addition, results will not be considered in the benefit assessment if the difference in the proportion of study participants missing is >15% between the study arms. These thresholds are considered as guidance, and exceptions are made if data are demonstrated to be missing at random. G-BA and IQWiG also evaluate missing data due to loss in follow-up: the number of trial participants, time point of losses and reasons for the loss should be evaluated and adequate replacement strategy or statistical analysis methods need to be implemented [15]. In practice, mixed-effects models with repeated measures (MMRM) are considered to be the favorable method compared with last observation carried forward (LOCF) to address uncertainties resulting from missing data [30].

For NICE, there is no missing data threshold provided in its guidance, and the completion rate is generally not critiqued in its assessments [22,26]. Similarly, there is no missing data threshold provided in the HAS guidance; nonetheless, it is an important attribute for PRO data acceptance for HAS [19]. In the assessment of abiraterone for mCSPC, HAS critiqued that PRO data results should be interpreted with caution when missing data are >30%, but this critique did not lead to the data being rejected [35].

Post-progression data collection

IQWiG suggests post-progression PRO data be collected until dropout or death in oncology clinical trials [30]. Although post-progression PRO data were collected in 11 analyzed case studies, collection until death or final survival analysis was conducted only in three cases: in nivolumab in metastatic melanoma [43,44], palbociclib plus aromatase inhibitor or plus fulvestrant trial in metastatic breast cancer [45,46] and ixazomib plus lenalidomide plus dexamethasone trial in relapsed or refractory multiple myeloma [47,48]. Nonetheless, the data were deemed unusable in these cases due to high proportion of missing data, over the 30% threshold.

NICE considers post-progression EQ-5D data as input to utility values in economic modeling. Although post-progression data were collected in eight analyzed case studies, EQ-5D data from clinical trials were only used in three cases to derive utility values. These include nivolumab in metastatic melanoma [49], olaparib in ovarian cancer [50] and ixazomib plus lenalidomide plus dexamethasone in relapsed or refractory multiple myeloma [51]. Nonetheless, NICE prefers to use post-progression data from trials instead of literature as commented in the obinutuzumab plus bendamustine assessment in follicular lymphoma [37]. For HAS, no guidance regarding post-progression data collection is available. This was not mentioned in any of the case studies either because PRO data from most case studies were rejected [19,26].

PRO data analysis

Clinically meaningful change threshold

IQWiG introduced new guidance in 2020 as well as an update in 2022 with regard to within-patient clinically meaningful change in PROs [15,16]. Outcomes are considered clinically meaningful if the responder definition is ≥15% of the scale range (e.g., 15 points on a 0- to 100-point visual analog scale). When response criteria are <15% of the scale range or no response criteria are prespecified, standardized mean difference (Hedges' g) is used. Here, an irrelevance threshold of 0.2 is used for the results to be considered clinically meaningful.

Because case studies analyzed for this study were pre-November 2020, published literature was the common evidence source provided for clinically meaningful change thresholds for G-BA/IQWiG assessments [52,53]. G-BA tends to accept Pickard et al. as a source for minimally important difference (MID) for EQ-5D, whereas IQWiG typically rejects it [53]. For example, in the assessment of olaparib for breast cancer gene 1/2 (BRCA1/2)+ ovarian, fallopian tube and primary peritoneal cancer, IQWiG rejected the time to deterioration of ≥7–10 points threshold for EQ-5D because the Pickard study was not considered suitable to prove the validity of the MID. Hence, IQWiG assessed a mean difference end point in this case. However, G-BA accepted the responder analyses, citing that responder analyses based on MID have general advantages for a clinical evaluation of effects compared with an analysis of mean differences and the validation study in question has already been used in previous assessments [52–55].

For NICE, predefined clinically meaningful change thresholds for EQ-5D are irrelevant for health economic modeling. However, in case studies in which time to deterioration is used to demonstrate a drug's clinical benefit, thresholds from published literature (e.g., Osoba et al.) were accepted by NICE [52]. For example, in the assessment of pembrolizumab for metastatic NSCLC, the sponsor submitted EORTC QLQ-C30 data with a mean difference of ≥10 points being considered clinically meaningful quoting the Osoba et al. and King et al. studies [52,56]. It was also mentioned that clinically meaningful mean difference as low as 4 points have been reported for EORTC QLQ-C30 in NSCLC trials as quoted in the Maringwa et al. study [57]. However, the evidence review group critiqued that it is unclear whether the result of Maringwa analysis is clinically important [58].

For HAS, no predefined clinically meaningful change threshold is provided in its guidance or case studies reviewed in this study [19,26].

Preference for time-to-event (TTE) endpoint definitions

Oncology studies generally have different follow-up times between different arms. Hence, G-BA and IQWiG prefer time-to-event analyses such as time to deterioration or time to improvement for evaluation of PRO data [30].

Some of the accepted definitions are demonstrated in the case study of daratumumab plus bortezomib plus thalidomide plus dexamethasone for multiple myeloma. Here, time to deterioration measured with the EORTC QLQ-C30 and EQ-5D visual analog scale with data collected until day 100 after autologous stem cell therapy was accepted by G-BA. End-point definitions used were ‘Time to deterioration by ≥10 points’ for EORTC QLQ-C30 and ‘Time to deterioration by ≥7 and ≥10 points’ for EQ-5D visual analog scale [59,60].

For NICE appraisals, changes in baseline scores are most common, but time to deterioration is used in some assessments. For example, for the assessment of crizotinib in ALK+ advanced NSCLC, time to deterioration was evaluated as a composite end point for the symptoms of cough, dyspnea and pain in chest using the EORTC QLQ-LC13 module. It was defined as the time from randomization to the earliest date that the patient's scale scores showed a ≥10-point increase after baseline (indicating a worsening of symptoms) in any of the three symptoms. Although the evidence review group mentioned that statistically significant benefits were observed with EORTC QLQ-LC13, only EQ-5D data were used, given it is a preferred instrument to derive utility values [61].

There is no explicit preference by HAS for specific PRO end-point definition, with no comments on end-point definitions assessed in the abiraterone assessments [26,35,36].

Missing data sensitivity analysis

In the G-BA/IQWiG assessments, sensitivity analysis is used to explore robustness of missing-at-random (MAR) assumption [30]. For example, in case of carfilzomib plus lenalidomide plus dexamethasone in multiple myeloma, the scale ‘overall health/quality of life’ was compared between the treatment groups using a likelihood-based MMRM under the MAR assumption during primary analysis. A sensitivity analysis, including a pattern mixture model (missing not at random - MNAR) with an auxiliary variable to consider missing data and a pattern mixture model that considered the time of the last available evaluation, was used to check the robustness [62,63].

NICE specifies use of sensitivity analysis to demonstrate the use of alternative utility values if EQ-5D data are available from both trials and literature. For example, in the original assessment of obinutuzumab plus bendamustine in follicular lymphoma, the sponsor provided sensitivity analysis in which EQ-5D utility scores from the GADOLIN trial as well as literature from Wild et al. were used [26,64]. Sensitivity analysis was conducted to demonstrate the use of literature over trial data as the GADOLIN trial did not capture EQ-5D data from trial participants with advance stages of disease progression [37]. For HAS, there is no mention of missing data sensitivity analysis either in guidance or analyzed case studies [26].

The summary of the PRO evidence requirements for HTA observed from the 20 case studies is shown in Table 3.

Conclusions

A major reason for increased use of PROs in oncology clinical trials can be attributed to the recognition of their importance by regulators including the FDA and EMA and the vision of incorporating the patient voice in reimbursement decisions by HTA bodies, especially IQWiG/G-BA and NICE [1,7]. However, as highlighted by this study, HTA bodies have provided varying levels of detail regarding PRO evidence requirements in their current method guidance. In particular, there is a lack of detail from NICE and HAS on how the HTA bodies would prefer PRO evidence to be analyzed and reported for HTA. Furthermore, there is a lack of consistency across HTA bodies on how PRO evidence is reviewed and considered in oncology HTAs. For instance, PRO evidence alone can lead to added benefit ratings in G-BA and IQWiG assessments, but the same data would be rejected by HAS due to methodological issues. For NICE, the most common impact for PRO data is the use of EQ-5D data in cost–effectiveness analysis.

Discrepancies highlighted in the guidance were also evident in the oncology case studies analyzed. For example, HAS rejected the PRO data in 17 of 19 selected oncology case studies citing methodological considerations such as rejection of data from open-label studies and lack of method for controlling type I error linked to multiplicity. In contrast, PRO data led to added benefit ratings in 7 of 20 selected oncology case studies assessed by G-BA and IQWiG. Oncology case studies assessed by NICE reflected what was provided in the guidance with utilities derived primarily from EQ-5D or literature values. Requirements and preference for missing data thresholds, post-progression data collection and time-to-event end points were clearly highlighted in the G-BA/IQWiG guidance and the same was observed in the case studies.

Some of the key discrepancies across HTA bodies require further exploration. For instance, HAS does not accept HRQoL data from open-label studies. However, studies have found no evidence of significant bias for PROs due the absence of blinding in randomized-controlled trials [65]. The EMA recognizes that blinded studies are not always possible, and PRO data from unblinded trials can be considered for benefit–risk assessment [2]. In another example, IQWiG's requirement of a responder definition of 15% of the range of the respective scale is not aligned with both EMA and FDA guidance [2,4]. Although IQWiG has defined this threshold based on systematic reviews of MIDs, this universal responder definition applied to all PROs has raised concerns among some questionnaire developers and sponsors, as conceptually, the responder definition cannot be the same for symptoms and for more indirect concepts such as quality of life.

Such discrepancies across European HTA bodies make it challenging for sponsors to meet the varying requirements of all HTA bodies and may reduce incentives for the sponsors to generate and provide comprehensive PRO evidence for oncology HTA submissions, despite regulator's proposed actions for patient-focused drug development. Sponsors may also have to include multiple PRO instruments in the same trial to satisfy the requirements by different regulatory and HTA bodies, which may lead to an unnecessary increase in patient burden during the clinical trial. HTA bodies can play a key role in harmonizing PRO evidentiary requirements in collaboration with regulators and sponsors. Further research is required to highlight whether similar discrepancies exist in other therapeutic areas.

Future perspective

Currently, there are therapeutic area-agnostic collaborative efforts between the EMA and FDA, which issued a joint reflection paper on Patient-Focused Drug Development that sets out a list of proposed actions of new ICH guideline for the industry [66]. To encourage the collection of PROs in clinical trials, European HTA bodies have an opportunity to issue uniformed PRO evidence requirements at the European level given the recent acceptance by the EU council for joint HTA clinical assessments (JCA) and the ongoing Setting International Standards of Patient-Reported Outcomes and Quality of Life Endpoints in Cancer Clinical Trials – Innovative Medicines Initiative (SISAQOL-IMI) project, which aims to develop standardized methods to analyze and report findings of PRO evidence from oncology trials [67–71].

Limitations of our study

The study was focused on only four key HTA bodies with three key European HTA processes. To manage the scope of the study, the results were based on the 20 solid tumor or hematological oncology PRO case studies of HTA reports published between October 2015 and November 2020; these case studies may not be representative of all solid tumor and hematological oncology HTA evaluations of PROs. The analysis in the current study was also limited to publicly available information in the method guidance published by the HTA bodies and commentary by the agencies in the HTA reports of the case studies. A more comprehensive review of differences in guidance and PRO data use among regulators, HTA bodies and payers across the US and Europe needs to be undertaken to streamline the PRO evidence requirements.