FormalPara Key Points

Although quantitative benefit–risk models (qBRm) represent useful tools to support decision makers in assessing the benefits and risks throughout the lifecycle of a medical product, few initiatives have been launched to harmonise qBRm approaches applied to vaccines.

The aim of the present review is to provide a complete list of available studies about qBRm applied to vaccines and to describe their characteristics according to the modelling context and the methodological approaches used.

Discrepancies across studies in terms of quality of reporting and methodological approaches used were observed in our review. Thus, we advocate for the development of an operational checklist for improving the reporting in scientific articles.

1 Introduction

Vaccination is one of the most successful public health achievements for disease prevention [1]. Compared with most other medicines, vaccines have some unique features; (i) they can be administered to large populations of healthy subjects, including children [2], and (ii) they can be introduced by health authorities as mandatory [3]. These specificities require vaccines to have a highly favourable benefit–risk profile [4]. In practice, this means that the benefits of vaccination need to be compared with its risks, in order to ensure informed and adequate public health decision making. Benefit–risk assessment (BRA) is the basis of regulatory decisions in both pre- and post-marketing review processes [5]. Although qualitative judgments have been used in BRA of health interventions, quantitative benefit–risk models (qBRm) may provide better consistency, transparency and predictability of decision making [6,7,8].

qBRm integrate evidence from multiple sources to quantify and put into perspective the benefits and risks of a health intervention. They can be simulation models (applying simulation techniques of various complexity degrees to estimate final outcomes) or non-simulation models (using a simple calculation to obtain final outcomes) [8,9,10]. They have been increasingly considered by decision makers (regulatory authorities, pharmaceutical companies, payers, guideline developers, etc.) to support the BRA of drugs and vaccines throughout their lifecycle [11,12,13]. However, although the advantages of using qBRm to weight benefits against risks are recognised by some regulatory authorities, formal guidelines are currently not available [14,15,16].

During the last decade, several initiatives have been undertaken to harmonise qBRm approaches applied to drugs through the identification, appraisal and classification of qBRm methods and the development of recommendations or frameworks to perform qBRm; for example, the Benefit–Risk Methodology Project driven by the European Medicines Agency (EMA) [9, 17], the Pharmacoepidemiological Research on Outcomes of Therapeutics (PROTECT) consortium in Europe [8, 10, 18,19,20,21], the Unified Methodologies for Benefit–Risk Assessment (UMBRA) [22] and the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Risk–Benefit Management Working Group [23].

Vaccine specificities may require specific modelling approaches, like those considering dynamic transmission and the occurrence of both individual and indirect protection through herd immunity. For this reason, similar initiatives aiming at streamlining qBRm approaches applied to vaccines have been launched, that is, the Accelerated Development of Vaccine benefit–risk Collaboration in Europe (ADVANCE) [24] and the Vaccine Monitoring Collaboration for Europe (VAC4EU) [25].

However, to date, a comprehensive review of the literature about qBRm applied to vaccines, comprising details on the modelling context and methodological approaches used, is lacking. To this end, we developed this first paper (Part I) aiming at systematically reviewing publications about qBRm applied to vaccines, and to describe their characteristics. Based on this mapping exercise and in order to ease the interpretation of future qBRm, we developed a second paper (Part II) that proposes standards in reporting qBRm applied to vaccines [26].

2 Materials and Methods

The definitions of technical terminologies (quantitative and qualitative BRA, qBRm) used in this article, and adapted from the glossary developed by PROTECT [8, 27], are provided in the Appendix Table 1 (see Electronic Supplementary Material [ESM]).

2.1 Literature Search and Study Selection

Medline, Scopus and the Institute for Scientific Information (ISI) Web of Knowledge databases were searched to identify articles published from database inceptions up to 31 December 2019. The search strategy combined three key concepts: (i) benefit–risk, (ii) modelling and (iii) vaccines. The search was conducted on 13 March 2020 and was limited to articles in English (see Appendix Table 2, ESM).

All citations were downloaded and imported in the reference management software EndNote version X7 (Thomson-Reuters Corp, New York, NY, USA). Duplicate citations were identified and excluded using the reference management software and manual title examination. Two reviewers (HA and NP) independently screened all titles and abstracts using predefined exclusion criteria (see Appendix Table 3, ESM). Subsequently, all publications retained after screening were assessed for eligibility by examining their full text. Disagreements between the two reviewers were resolved through discussion. A secondary manual search was performed through snowballing the reference lists of the eligible articles to identify additional articles. A grey literature search was performed through searching major public health organisation websites and using targeted search terms in Google.

2.2 Data Extraction

The following data were extracted by reviewers: (i) the general information including study publication date and the funding source; (ii) the modelling context including vaccine indication, the geographical location and the income level of the countries considered in the analyses according to the World Bank classification, the targeted population, the alternatives used for comparison with standard vaccination and the perspectives from which the intervention’s benefit and risk outcomes are evaluated; and (iii) the methodological approaches, that is, the modelling techniques, the model attributes, the benefit–risk measures, the sensitivity/scenario analyses and whether a standardised framework, a tabular representation summarising input parameters, discount rates and utility or preference information, were used or not. See Appendix Table 4 (ESM) for more details and definitions.

The data extraction template was developed based on similar initiatives about qBRm applied to drugs, particularly on the modelling context [9, 18, 28, 29] and the methodological approaches [9, 10, 17, 21, 23], and was adapted to consider vaccine specificities (i.e. dynamic modelling technique, herd immunity and waning effect attributes) [24, 30, 31].

Two reviewers (HA and KB) independently extracted data from the selected articles. Disagreements were resolved through discussion. The authors of the papers selected for data extraction were not contacted to provide additional information.

2.3 Data Extraction

Tables 1 and 2 describe the general information, the modelling context and the methodological approaches of each study. Table 3 summarises the distribution of methodological approaches depending on the use of a simulation model or non-simulation model.

Table 1 Characteristics of the general information and the modelling context used in quantitative benefit–risk models applied to vaccines
Full size table
Table 2 Characteristics of the methodological approaches used in quantitative benefit–risk models applied to vaccines
Full size table
Table 3 Distribution of the methodological approaches stratified by use of a simulation or non-simulation model
Full size table

Distribution of the number of publications according to the source of funding and the modelling techniques used over time is described in Fig. 2 and Appendix Fig. 1 (see ESM).

3 Results

3.1 Selection of Studies

The literature search retrieved 4917 potentially relevant publications, of which 1745 were duplicates and hence excluded. Of the remaining 3172 publications screened, 3081 were excluded due to lack of relevance, as determined by evaluation of their title and abstract. The full texts of the remaining 91 publications were extensively evaluated and discussed, leading to the exclusion of 52 articles. Reasons for exclusion were repetitive data from original studies (2), non-relevant publication type (14), non-qBRm studies (35) and no access to the full-text article (1). Nine additional publications were included through snowballing the reference lists. A total of 48 publications [32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79] were retained for data extraction (Fig. 1). Tables listing the publications and reasons for exclusion and inclusion after assessment of the eligibility of the full texts are provided in the ESM (See Appendix Tables 5 and 6).

Fig. 1
figure 1

PRISMA flow diagram

Full size image

3.2 General Information

A temporal trend analysis of the selected articles according to their publication date showed that the number of publications focusing on vaccine qBRm was limited before the year 2000 and increased from 2008 onwards, with an average of three publications per year (Fig. 2). Ten studies disclosed funding by pharmaceutical companies.

Fig. 2
figure 2

Distribution of the number of publications according to the source of funding over time

Full size image

3.3 Modelling Context of Selected Studies

Most of the vaccines included in the selected publications were targeting viruses (40). Two-third of the studies were performed for three viruses (rotavirus (15), dengue (10) and influenza (6)). By contrast, other indications were only supported by a single publication (e.g. Human Papillomavirus or Hepatitis B virus) (Table 1).

Most countries considered in the analyses were high-income countries (30) and the majority were located in the United States and in Europe (Table 1).

In most studies (45), the targeted population analysed in the selected studies was clearly specified and most referred to paediatric populations (38). Others focused on the general population without any further specification (3) or were conducted in several populations (e.g. paediatric and adult populations or general and high-risk populations) (4).

Most studies compared standard vaccination versus absence of vaccination (30), and only a few compared different vaccines of the same class (3). In some cases (15), the vaccine was compared with several alternatives (e.g. standard vaccination versus absence of vaccination and standard vaccination versus vaccination with extended indication).

A large proportion of studies described the perspectives at a population level only (40), compared with individual (3) or both levels (5).

3.4 Methodological Approaches Used in Selected Studies

Few studies (5) used a structured approach or tool that facilitates the development of the qBRm (Tables 2 and 3). Among them, the multi-criteria decision analysis (MCDA) was the main standardised framework used (3) [34, 60, 61].

Studies included in the review used non-simulation (16) or simulation models (32) to estimate final outcomes. Among the simulation models, the most common types were the dynamic model (12), followed by the decision tree (5), Markov model (4) and microsimulation (1). Of note, one-third (10) of the simulation models did not specify the modelling technique used. We did not observe a temporal trend of modelling techniques (non-simulation versus simulation) (See Appendix Fig. 1 in the ESM).

Most of the qBRm included were static (36), deterministic (31), did not consider herd immunity and waning effect (35), assumed a closed model (i.e. the model does not allow new individual entrances over time) (47) and stayed at an aggregate level (i.e. no tracking of individuals’ behaviour) (44) (Tables 2 and 3).

Almost two-thirds (29) of the studies used a tabular representation to list and summarise all input parameters used in the qBRm.

Few studies (5) used discount rates to reduce the value of benefit and risk outcomes over time.

Health state preference values (also called utilities) are used to represent the strength of individuals’ preferences for different health states. In some studies (8), preference elicitation techniques were performed to weight benefit and risk outcomes. In these studies, participants were interviewed to obtain preference scores by different methods: standard gamble (1), time trade-off (1) and the Index of Well-Being (1); the other studies (5) referred to an expert panel for eliciting or weighing data.

Regarding the benefit–risk measures, one-third of the studies (16) used measures to estimate benefits and risks separately (single indices), followed by composite measures to integrate benefits and risks (trade-off indices) (13) and the remaining ones (19) used both (Tables 2 and 3). The most commonly used single indices were the impact numbers (34) and the number needed to vaccinate/to harm (NNV/NNH) (8). Impact numbers measure the number of events prevented by and/or cases attributable to vaccination within a given population. NNV and NNH provide the number of patients who need to be vaccinated for preventing or inducing one case of the event of interest. The most commonly used trade-off indices were the benefit–risk difference (or net health benefit [NHB]) (18) and the benefit–risk ratio (BRR) (13). NHB estimates the difference between the total various benefits and the total various risks of a specified vaccination. BRR divides a single benefit by a single risk. BRR and NHB expressed all outcomes in the same metric.

Almost all included studies (47) used a visual representation of their quantitative benefit–risk results, notably tables (34), line graphics (23), bar charts (8), scatter plots (6), box plots (5) and area graphs (2).

Most studies (44) conducted sensitivity and/or scenario analyses including one-way deterministic sensitivity analyses (13), multiple-way deterministic sensitivity analyses (4), probabilistic sensitivity analyses (20) and scenario analyses (37) (Tables 2 and 3). Most of the studies using non-simulation models only reported scenario analyses (12). By contrast, most studies with simulation models reported sensitivity (deterministic and/or probabilistic) analyses (25).

4 Discussion

To our knowledge, the present study is the first that systematically reviewed published information about qBRm applied to vaccines. This study was not intended to appraise the qBRm quality but aimed to build a comprehensive repository of the available publications about qBRm applied to vaccines. The present work is the first of two companion papers. Based on this mapping exercise (i.e. Part I), we developed a second paper that proposes standards of reporting for qBRm applied to vaccines (Part II) [26].

4.1 General Information

Since the late 1990s, the development of qBRm has been progressively encouraged by regulatory authorities [14,15,16, 80, 81]. Although the first qBRm applied to vaccines identified in our systematic literature review was published in 1976 [75], most studies were published over the last decade. The number of those developed by pharmaceutical companies has increased over time. This trend may be explained by the recent launch of international consortiums [18, 24, 25] facilitating collaboration between public and private partners to develop tools for assessing benefit–risk, particularly for vaccines.

4.2 Modelling Context

Among the 48 original publications included, the majority targeted the rotavirus, dengue and influenza vaccines. The focus on rotavirus vaccines might be explained by their association with a transient increased risk of intussusception following administration [82]. The high number of publications about the recent dengue vaccine is likely to be caused by the increased risk of ‘secondary-like’ infection in populations [83]; in this context, qBRm were performed to simulate optimal vaccine strategies according to different transmission intensity settings and serological status impacts. For influenza vaccines, the variable nature of the influenza viruses requiring the undertaking of a new vaccine version each fall is a likely explanation [84]. By contrast, qBRm were not performed for some vaccines, notably older ones, such as those for diphtheria or poliomyelitis, probably because the recent adoption of qBRm and the benefit–risk balance of older vaccines is no longer questioned.

Most qBRm targeted high-income countries. Their results may thereby not be representative for countries with lower income. Estimation of benefits and risks of vaccination may indeed vary according to the epidemiology of the diseases and the healthcare system. Those geographical and socio-economical specificities should be considered when interpreting qBRm results. Furthermore, the development of qBRm applied to vaccines in different settings should be encouraged for improving and adapting the decision-making process.

4.3 Methodological Approaches

The present work pointed out the high level of variability in the quality of reporting across studies. First, it was not straightforward to identify input parameters used for each qBRm. One-third of the studies did not use a tabular representation to list and define all input parameters. Second, information on modelling techniques and benefit–risk measures were not sufficiently outlined, which added complexity in extracting the data summarised in Tables 2 and 3. Incomplete and inadequate qBRm reporting may affect their result interpretation, their reproducibility, and potentially, any related decision-making process (e.g. the approval and/or recommendation of a new vaccine or its indications). Developing reporting guidance is therefore the key to helping ensure transparency and reproducibility of qBRm applied to vaccines, but is currently lacking.

One of the main findings of this review was the heterogeneity observed across studies in terms of their methodology. First, two-thirds of the studies used simulation models to estimate final outcomes. Among the simulation models, the number of static and dynamic models was similar. Dynamic models are particularly useful when targeting infectious diseases because they allow taking into account the evolution of certain factors that modulate the risk of infection over time, such as immunity [85]. Of note, except for two publications, all dynamic transmission models were performed for the dengue vaccine. The low number of transmission dynamic models used for other vaccines may be related to their high level of complexity, requiring more advanced programming skills [31]. Second, a few studies incorporated health state preference values, offering the advantage that it represents the strength of individuals’ preferences for different health states, thereby moving beyond the narrow biomedical model for evaluative research [86]. Various techniques were used to obtain health state preference value. Third, benefit–risk measures considered also differed from one study to another. Single, trade-off and both indices were each used in one-third of the selected studies. Single indices are valued for their intuitive interpretation by clinicians and decision makers. Conversely, by using a commensurable score, trade-off indices confer the advantage of providing a straightforward comparison of benefit and risk outcomes, thereby facilitating the BRA. Nevertheless, trade-off indices may be less intuitive and can only be performed when both benefit and risk outcomes use the same metric. Fourth, almost all studies performed uncertainty analyses by varying one or more key parameter values, using various techniques such as scenario (18), sensitivity (7) and both (18). The questions on benefit–risk decision are heterogeneous, due to differences like targeted population, benefit and risk outcomes and importance of indirect effects. Thus, there is no one-size-fits-all approach, as proved by the high level of methodological heterogeneity among the selected publications. This observation highlights the richness of methodological approaches available to perform qBRm applied to vaccines; each of them with their own strengths and weaknesses. The choice of methodological approaches depends on several considerations like the analyst’s technical skills, the required model complexity, the question at hand and the nature of the decision matter, the natural history and features of the particular infectious disease and the data available to parameterise and calibrate the model [87, 88]. Consequently, it is crucial to develop methodological guidance to help researchers use the best methodological approach suited to specific situations. Furthermore, the variety of methodological approaches available reinforces the need for specific standards in reporting.

The present review has some limitations. Even though we aimed to be comprehensive, we cannot guarantee that all relevant studies were identified, mainly because there are no specific keywords for qBRm. Furthermore, grey literature is not identifiable through conventional methods of bibliographic search, and consequently, some relevant websites or other source(s) of information might be missing. However, various search strategies were tested before selecting the one that allowed us to retrieve the largest number of relevant publications, while keeping the number of articles to be reviewed within acceptable limits.

5 Conclusion

This study is the first that systematically reviewed published information about qBRm applied to vaccines. Our findings showed (i) large differences in terms of methodology used to perform qBRm applied to vaccines and (ii) a noticeable heterogeneity across studies in terms of quality of reporting. Since the number of published qBRm studies is increasing over time and given that paucity of formal guidance for qBRm applied to vaccines may affect the confidence in these models’ results, the development of specific standards of reporting should help to ensure the transparency and reproducibility. Part II of this series of companion papers proposes an operational checklist aimed to improve the reporting of qBRm applied to vaccines in scientific articles.