A central need, when estimating how COVID vaccination affects mortality, is to estimate the counterfactual: What would COVID-19 mortality have been for the vaccinated, if they had not been vaccinated? We use Non-Covid-NMR as a proxy for background mortality risk, and find important differences in background mortality risk between vaccinated and unvaccinated and between two-dose and three-dose recipients. By this measure, two-dose recipients are healthier (have lower non-COVID-NMR) than the unvaccinated, and three-dose vaccinees are healthier than two-dose vaccinees. These selection effects, unless controlled for (through our CEMP measure or in another way) can produce large biases in VE estimates.
Non-Covid-NMR does an excellent job of predicting COVID-19 mortality during the pre-vaccine period, out-of-sample in Indiana (Fig. 1). It performs similarly well in sample, both for Wisconsin as a whole, and for Milwaukee County (Appendix Figure App-1). This suggests that using CEMP as the outcome when measuring RMR provides a good estimate of the protective effects of vaccination relative to the counterfactual. The similarity between unadjusted RMR estimates (Table 2) and multivariate estimates (Table 3) provides further evidence that the CEMP denominator does a good job of controlling for underlying health and mortality risk.
The data on Non-Covid-NMR ratios in Table 1 can be used to assess the extent of selection bias: Assume counterfactually that vaccination was useless against Covid mortality. What RMRs would one estimate, controlling only for age group? Given the high correlation between Non-Covid-NMR and COVID-19 mortality for the unvaccinated, the RMR ratios of vaccinated to unvaccinated in Table 1 provide approximate answers to this question. For example, for ages 40–79 during the Omicron period, estimated three-dose RMR would be 35%, versus true RMR (no vaccine effect) was 100%.
Advantages of the CEMP Measure
CEMP, as a measure of COVID-19 mortality, has attractive features relative to other measures. It relies only on death certificates, which are available for all decedents, but addresses selection effects, by using non-COVID NMR to proxy for population health, which is otherwise difficult to observe. An alternative approach, controlling for comorbidities captured in electronic health records, faces important limitations: comorbidity data may not be fully reported, and, in the U.S., population-level data on comorbidities is not available. Even studies that control for comorbidities often examine only people who seek medical care for COVID-19 infection.[8, 20] This will miss the association among underlying health, who becomes infected, and infection severity.
Using data only on decedents also avoids the challenges in estimating the population at risk. Population statistics may undercount some groups because of non-participation in the Census or the American Community Survey, or inaccurate data. While race/ethnicity can be inaccurately captured in death-certificate data, it is unlikely that inaccuracies will differ systematically between those who die of COVID-19 versus other natural causes.
Overview of Results: Substantial RMRs, Large Value for Boosters
Our analysis provides a number of insights for vaccine effectiveness against mortality and selection effects in who gets vaccinated. First, our two-dose RMRs versus unvaccinated are substantially higher (VE is lower) than in other studies. The higher RMRs reflect our use of CEMP to address selection bias, as well as continued vaccine waning in the Omicron period.
The studies covered by the available systematic reviews report lower two-dose RMRs, from 6–17%, as compared to this study.1,2 We found only one study that reports similar two-dose RMRs for the pre-Omicron, pre-booster period. This study finds an 18% RMR for fully vaccinated U.S. veterans (two mRNA or one J&J) for ages < 65 and 28% for ages 65 + .14 Likely not coincidentally, this study uses the rich VA data to control for an extensive set of comorbidities; it also finds that vaccinees have lower all-cause mortality. We found only one other U.S. study that reports RMR from linked, population-wide mortality and vaccination data. A study of Puerto Rico through mid-October 2021 (thus pre-Omicron and pre-booster), reports two-dose RMR after 144 days (longest period considered) of 14% for Pfizer and 7% for Moderna, versus 3% and 1% soon after vaccination.8 This study does not control for selection effects. We did not find studies of VE against mortality covering the Omicron period.
Second, we find substantial waning of two-dose protection against mortality, with two-dose RMR versus unvaccinated for ages 60 + increasing from 11% in 2Q-2021 to 34% in 1H-2022. This contrasts with prior studies, which typically report limited waning against severe disease and death.1,2 Note that we cannot separate the effects of waning over time from differences in protection against different virus variants.
Third, we find that two-dose RMR increases with age, but boosters provide substantial additional protection. This makes boosters especially important for the elderly, particularly those age 80+. For ages 60 + three-dose RMR is 10% in 4Q-2021 and 11% in 1H-2022, versus 28% and 34% for two-doses. At the same time, three-dose RMRs are higher than reported in prior booster studies,10 again showing the importance of controlling for selection effects. Nonetheless, three-versus-two-dose differences in RMRs for ages 60 + are large, at 18% in 4Q-2021 and 23% in 1H-2022. The reduction in RMR is even higher for ages 80+, at 32% for 1H-2022. In effect, the higher two-dose RMRs that we find leave more room for boosters to reduce mortality, even though we also find higher three-dose RMRs than prior research. Our evidence supports public health messaging and policy that encourages boosters for the elderly.
Fourth, we find stronger relative two-dose protection for ages 18–59 in 2021, compared to older persons in the pre-Omicron period, but not in the Omicron period. Absolute COVID mortality risk after two doses is smaller for younger persons, but boosters are highly effective in reducing that risk: We find zero deaths among younger three-dose recipients. Our results for the Omicron period contrast to the perception among many younger persons that two doses provide sufficient protection.
Fifth, we find that a single dose provides only moderate protection, with RMR versus the unvaccinated around 50%. However, this protection appears to be long-lived. Limited waning has been reported before for the single-dose J&J vaccine.7,21 We find similar results for one-dose mRNA recipients.
An Opportunity for Targeted Booster Messaging
Evidence of vaccine waning first appeared in mid-2021, initially from Israel. Based on this evidence, Israel launched a booster campaign in late July 2021, which reached the whole population by the end of August.22 Other countries soon followed, relying in part in Israeli evidence that boosters added important value. In the U.S., however, FDA scientists publicly questioned the need for boosters.23 An advisory committee to the Food and Drug Administration (FDA) in September 2021 approved only a limited rollout to the elderly and persons at risk due to occupational exposure;24 similarly, an advisory committee to the CDC endorsed boosters only for the elderly.25
Two months later, the FDA and CDC approved boosters for all adults; although a CDC recommendation came only at the end of November, 2021.26 Public health messaging remained muddled, with the value of boosters “lost in the sea of changing recommendations and guidance,”27 Even today, U.S. booster percentages lag many other countries,28,29 and public knowledge of booster recommendations is limited.30 Our study provides strong evidence on booster value for ages 60+, which account for the vast majority of COVID-19 deaths, and nearly all vaccinee deaths.
Toward Enhanced Public Reporting of COVID-19 Mortality
Although many public sources report data on COVID-19 deaths. None reports a comparison to other natural deaths. Reporting both COVID-19 and non-COVID natural deaths, and ideally CEMP (the ratio of the two) would provide valuable information on the risk of death from COVID-19 versus other natural causes. Reporting CEMP would show that the unvaccinated face substantial COVID-19 mortality risk, even at younger ages. This might make more salient the large reductions in mortality risk from vaccination, and the value of a booster dose. Reporting both COVID-19 mortality and other natural mortality could also focus attention to selection effects, and their importance when estimating vaccine effectiveness.
Limitations
This study has important limitations. We study only mortality. COVID-19 mortality is uncommon for younger persons, which limits statistical power. Our data is only for Milwaukee County, which is racially, ethnically, and economically diverse, but its COVID-19 experience may not be representative of other areas.
We do not observe, and thus cannot control for, prior COVID-19 infection. Especially in the Omicron era, many people, both vaccinated and unvaccinated, have already been infected. For them, VE can be understood as measuring the extra protection from hybrid immunity (from prior infection plus vaccination) versus natural immunity alone (from prior infection).
We do not observe individual health characteristics, except through the limited lens of death certificates. There could be vaccinated-vs-unvaccinated differences that affect COVID-19 mortality, not reflected in Non-Covid-NMR.
The CEMP measure has several inherent limitations. Though likely a reasonable proxy for overall health, it does not consider behavioral or other differences between more and less vaccinated groups. Behavioral differences are likely to exist. For example. the unvaccinated or less-vaccinated, may believe COVID is less severe than the maximally vaccinated, and therefore take fewer precautions. Conversely, vaccinated persons may accept greater risks of becoming infected, because they believe they are protected against serious illness. Neither ours nor other VE studies can control for behavioral differences.
The CEMP measure implicitly assumes that COVID-19 infection does not meaningfully affect non-COVID mortality. Yet COVID-19 infection is known to predict higher post-infection mortality from other causes, at least in the near term.31 This will cause downward bias in CEMP values. If this bias is similar for the vaccinated and unvaccinated, RMR estimates should still be unbiased. The downward bias in CEMP could be larger for the unvaccinated, who will on average face more severe COVID-19. If so, then RMR estimates based on CEMP will be somewhat below those we would estimate if we could attribute to COVID-19 these extra natural deaths. However, any bias is too small to be visible in Fig. 2.
COVID-19 deaths could be underreported, but we coded COVID-19 deaths based on reading death certificates; this produced significantly larger counts than ICD-10 codes from the NCHS. Any remaining undercount appears small (Fig. 2).
This study assesses vaccine effectiveness only against mortality, not against other important adverse outcomes including hospitalization and long-term symptoms. A full understanding of vaccine effectiveness must include a broader range of outcomes.