Results from evaluations of models and cost-effectiveness tools to support introduction decisions for new vaccines need critical appraisal

Abstract

The World Health Organization (WHO) recommends that the cost-effectiveness (CE) of introducing new vaccines be considered before such a programme is implemented. However, in low- and middle-income countries (LMICs), it is often challenging to perform and interpret the results of model-based economic appraisals of vaccines that benefit from locally relevant data. As a result, WHO embarked on a series of consultations to assess economic analytical tools to support vaccine introduction decisions for pneumococcal, rotavirus and human papillomavirus vaccines. The objectives of these assessments are to provide decision makers with a menu of existing CE tools for vaccines and their characteristics rather than to endorse the use of a single tool. The outcome will provide policy makers in LMICs with information about the feasibility of applying these models to inform their own decision making. We argue that if models and CE analyses are used to inform decisions, they ought to be critically appraised beforehand, including a transparent evaluation of their structure, assumptions and data sources (in isolation or in comparison to similar tools), so that decision makers can use them while being fully aware of their robustness and limitations.

Background

Over the next decade, several new vaccines will be considered for universal use against infectious diseases of major public health importance. Public health decision makers will need to make choices between vaccines and among various health interventions. New vaccines are more expensive than the traditional dollars-and-cents vaccine costs, while countries increasingly have to bear the full cost of vaccination programmes. Because not all evidence is available to explore the impact of interventions under different scenarios, cost-effectiveness analyses (CEAs) of vaccination programmes tend to be based on mathematical modelling, which should yield a framework and help synthesize information from many sources and also help extrapolate data beyond available data (for example, clinical trials). Given the complexity and the importance of these tools, and to accurately inform vaccine policy makers at the country level, a critical appraisal of these tools' robustness and limitations as well as of the generalizability of model estimates to local contexts needs to be conducted and documented before results are accepted.

A recent systematic literature review showed that cost-effectiveness (CE) has become an increasingly important factor for stakeholders who need to make decisions about adding a new vaccine into national immunization programmes versus alternative uses of resources [1]. However, the same study also pointed out that the majority of these studies were done in high-income countries, while in low- and middle-income countries (LMICs), where fewer studies are performed, economic evaluations have a more limited role in the decision-making process.

The need for increased guidance on CE was clearly identified in a stakeholder review conducted on the World Health Organization's (WHO's) normative role in immunization [2]. Country-level decision makers, particularly those from LMICs, discussed the lack of guidance on the criteria used to assess the model structure, assumptions, robustness and limitations of economic decision-making tools [3]. As a response to these needs, WHO embarked on a series of consultations to assess economic analytical tools to support vaccine introduction decisions for pneumococcal, rotavirus and human papillomavirus vaccines [4–6] Considering that a malaria vaccine is undergoing pivotal phase 3 evaluation, WHO is evaluating impact models that can inform decisions about the use of combinations of antimalarial interventions. The objectives of these assessments are to provide decision makers with a menu of existing CE tools for vaccines and a transparent overview of their characteristics rather than to recommend a single model. The goals of these consultations are to provide policy makers in LMICs with information about the feasibility of applying these models to aid their own decision making, as well as with guidance for interpreting the results from these and similar studies. Country-level decision making for the introduction decisions regarding new vaccines is a complex process that needs to take into account several factors, such as local information on patterns of disease transmission, effective coverage and costs per dose, as these parameters are often the ones that drive the outcomes of the CEA. In addition, the characteristics of the healthcare system and issues pertinent to financing, affordability and sustainability should be important elements in the decision-making process. Moreover, decision makers often experience substantial pressure from many stakeholders and "vaccine advocates" who contribute to the interpretation of the available evidence. The conclusions of the WHO consultations to assess the vaccine economic analytical tools mentioned above led us to argue that relying on CEA results from only a single modelling exercise is not desirable [4–6]. We also reiterate that conducting sensitivity analyses is critical to comprehending the modelling and CEA outcomes. After a critical review of the outcomes of these consultations, the WHO Quantitative Immunization and Vaccine-Related Research Committee endorsed these statements [7].

Discussion

From a public health perspective, limited financial resources should be distributed in a fair and effective manner aimed at achieving the best possible outcomes. Decisions on implementing new vaccines in national immunization programmes should be transparent, unbiased, comprehensive and based on prespecified and evidence-based criteria [8].

Decision makers in LMICs reportedly find the methods used and the interpretation of results from CE studies complex and therefore challenging to use in the decision-making process. In many LMICs, there are not fully functional immunization technical advisory groups to advise national authorities on decisions to introduce new vaccines or to adjust existing immunization strategies [9]. Where they are in place, these authorities are confronted with difficulties in interpreting or adapting results as part of a context-specific analysis. In many instances, countries need to carry out their own economic analyses, as results from existing studies are not fully generalizable to their own contexts, owing to large differences in local factors such as disease epidemiology; vaccine-specific issues (for example, distribution and replacement of serotypes); the magnitude of estimated herd immunity effects, given local immunization strategies and effective coverage; as well as local health care systems, budgets and vaccination policies [10, 11]. Consequently, policy makers often support recommendations to introduce vaccines in the absence of a transparent, formally structured and independent review process. Frequently this is observed in situations in which there is considerable external pressure from the general public and those with vested interests [12–14]. Models become more complex if one attempts to evaluate the community-wide and combined effects of interventions. For instance, pneumococcal and Haemophilus influenzae type b immunizations need to be assessed in comparison to or in combination with other possible interventions, such as community-based treatment, promotion of exclusive breastfeeding and zinc supplementation [15]. Antimalarial interventions should be assessed in the context of the options available to national malaria control programmes [16, 17]. Current dose prices for rotavirus and pneumococcal vaccinations provided by the Global Alliance for Vaccines and Immunisation in 2010 were around US$4.00 to US$7.00 compared to the dollars and cents required for traditional vaccines such as diphtheria, tetanus and pertussis or measles. As this may pose serious financial challenges to the budgets of national immunization programmes in LMICs, policy makers in those countries welcome donor funds to take up vaccines in their national vaccine programmes, often without considering long-term sustainability issues if donor support fades out.

The series of consultations that WHO conducted in 2009 and 2010 to compare CE tools and models using standardized data sets from different WHO regions and predefined vaccine delivery scenarios are useful for several reasons [4–6]. Actual tool comparison promotes transparency. It allows for model structure validation, provides analysts and decision makers with a more in-depth view of disease dynamics and offers access to the assumptions and sources of data used within the various input parameters. This will allow decision makers to draw on the estimates from different models and CEAs. It also will guide future research on appropriate model structures and will encourage data collection using the most important model parameters driving CE results. Comparing uncertainty boundaries around CE results using different modelling tools will identify evidence gaps and hence help guide future data collection efforts (for example, from disease surveillance networks). Furthermore, a tool comparison exercise using standardized data sets and scenarios will facilitate the comparison of robustness and clarity of model structures and assumptions. Better understanding and transparency of existing decision-making tools and models will focus efforts and limited resources, in particular in LMICs, on the generation of better-quality data rather than on additional, often more complex CE models that suffer from the same data paucity.

Summary

Economic decision-making tools are an aid that make it possible to explore the potential impact of various interventions and policy options. They do not necessarily provide a final answer to policy questions, but offer decision makers a structured and logical framework within which to assess the impact of one or more scenarios. Given the inherent complexity of this type of analysis, tools need to be transparent in terms of model structure, parameter values, sources of data and validation process so that the robustness of their results can be assessed. Using uncertainty analysis throughout the process of reviewing one or more tools helps to identify those parameters that will have a greater impact on the results and therefore on the data collection and synthesis priorities which may guide disease surveillance networks to promote an integrated and comprehensive approach to data gathering and analysis in the future. In order for any CEA to be reliable at the country level, access to locally relevant, recent and high-quality data is crucial.

Mathematical models and CEAs should be used as tools to evaluate the impact of interventions under various likely scenarios. Their sole use as advocacy tools in support of a given choice is not adequate. The WHO recommends that CEAs of introducing a new vaccine (and the various strategic options) into a national immunization programme be considered before any specific strategy is implemented. Toward this end, the WHO has released a vaccine- and immunization-specific guide on CEA [18, 19]. It is recommended that models and CEAs of vaccines that use locally relevant data and assumptions are needed to better inform vaccine introduction decisions at the local level, particularly in LMICs. Capacity strengthening in this area at the local level is a critical step in promoting the rational use of these evaluations and tools [12, 20].

Authors' information

The authors are staff members of the WHO Initiative for Vaccine Research, whose mission is to guide, provide vision for, enable, support and facilitate the development, clinical evaluation and worldwide access to safe, effective and affordable vaccines against infectious diseases of public health importance, especially in developing countries.