Skip to main content
Download PDF

COVID-19 vaccination effectiveness in the population of Friuli Venezia Giulia, North-East Italy. Control of bias associated with divergent compliance to policies in a test-negative case-control study

BMC Public Health volume 23, Article number: 2476 (2023) Cite this article

Abstract

Background

Vaccine effectiveness (VE) studies consolidate knowledge of real-world effectiveness in different contexts. However, methodological issues may undermine their conclusions: to assess the VE against COVID-19 within the Italian population, a specific threat to validity is related to the consequences of divergent compliance to the Green Pass policy.

Methods

To address this challenge we conducted a test negative case-control (TNCC) study and multiple sensitivity analysis among residents aged ≥ 12 in Friuli Venezia Giulia Region (FVG), North-east Italy, from February 1, 2021 to March 31, 2022. Information regarding 211,437 cases of COVID-19 infection and 845,748 matched controls was obtained from the regional computerized health database. The investigation considered: COVID-19 infection, hospitalization, and death. Multiple conditional logistic regressions adjusted for covariates were performed and VE was estimated as (1-OR COVID-19vaccinated vs. unvaccinated)x100. Mediation analyses were carried out to offset potential collider variables, particularly, the number of swabs performed after the introduction of pandemic restrictions.

Results

Full-cycle VE against infection decreased from 96% (95% CI: 96, 97) in the Alpha period to 43% (95% CI: 42, 45) in the Omicron period. Booster dose raised the protection in Omicron period to 67% (95% CI: 66, 67). Against the evasive Omicron variant, the protection of the booster dose was 87% (95% CI: 83, 90) for hospitalization and 90% (95% CI: 82, 95) for death. The number of swabs performed was included as a covariate in the adjustments, and the mediation analysis confirmed that it was a strong mediator between vaccination and COVID-19-related outcomes.

Conclusions

The study suggests that, under similar TNCC settings, mediation analysis and adjustment for number of diagnostic tests should be included, as an effective approach to the challenge of differential testing behavior that may determine substantial selection bias. This correction allowed us to align with results from other studies that show how full-cycle VE against infection was initially high but decreased over time by variant circulation, counterbalanced by booster dose that raised protection across variants and outcome severity.

Peer Review reports

Background

After randomized clinical trials (RCT) had demonstrated a high level of efficacy of COVID-19 vaccines in protecting against symptomatic infection, the European Medicines Agency (EMA) and the Italian national authority authorized their use and vaccines became available in Italy for priority groups on December 27, 2020 [1].

Since then, the question of vaccine effectiveness (VE) in real-world settings has been widely addressed [2]. Relevant VE topics include how to monitor the VE in the context of clinical factors not completely assessed by clinical trials, to investigate different outcomes, to determine the duration of the protection, and the VE against new Variants Of Concern (VOCs). The constant assessment of the real-world performance of these vaccines is required to justify health policies that must be adopted in different settings. However, observational VE studies, unlike RCT, have to overcome major methodological challenges, particularly in avoiding confounding and selection bias [3]. Therefore, several strategies of bias minimization have to be taken into consideration, as summarized in World Health Organization (WHO) guidance publications [4, 5].

In the pursuit of assessing COVID-19 VE in Italy, a specific threat to validity is related to the consequences of divergent population compliance to policies that have been adopted over the pandemic. In particular, the EU Regulation 2021/953 [6] granted the “Green-Pass” digital certificate to COVID-19 vaccinated, recovered or recently tested negative subjects (within 48 h), thus influencing citizens’ choices on numerous activities. Since the implementation of this European Regulation allowed each member state to introduce its own rules, its consequences varied by country [7]. In Italy, between October 15, 2021 and March 31, 2022 access to public spaces, means of transportation and workplaces [8,9,10] were subject to Green-Pass possession. Therefore, in that period, people’s daily life was regulated by clinical status, COVID-19 infection, contact with infected subjects, vaccination and proof of COVID-19 negative status.

To estimate VE against COVID-19 infection, hospitalization and death, we conducted a case-control study in the population > 12 years old of Region Friuli Venezia Giulia (FVG), North-east Italy. To mitigate potential confounding and biases, especially health seeking behavior bias and collider bias [11] that might have been originated by an heterogeneous adherence to Green Pass regulation, we chose a test-negative, case-control (TNCC) study [4, 12] reinforced by a series of sensitivity analyses.

Methods

Study population

As of January 1, 2022, the population of FVG Region was 1 194 647, accounting for 2.0% of the Italian population [13]. Regions in Italy are administrative and political entities that have authority on public health: during the pandemic they were in charge of organizing and implementing vaccination and testing strategies at the local level, and also of collecting and reporting health surveillance data [14]. The study population included subjects: (a) who had received at least one valid result for an antigenic or molecular SARS-CoV-2 virus swab at a public or private facility authorized by the Central Directorate of Health, Social Policies and Disability of FVG between February 1, 2021 and March 31, 2022, (b) who were residents in FVG at the date of swab collection and (c) who had been residents in FVG also from January 1, 2019 to February 1, 2021. The latter criterion was added to assess subjects’ comorbidities in the 2 years preceding the beginning of the study. Subjects with previous COVID-19 positivity were excluded.

Source of COVID-19 data

The anonymized regional health database allows individual computerized linkage of comorbidity, diagnostic, treatment and outcomes information using a unique key. Specifically, we linked demographic data with swabs collected and COVID-19 vaccine doses, COVID-19-related hospitalizations and deaths, pre-COVID medications and hospital admissions. Certain swabs were excluded from the analysis: i.e., (i) swabs of children under age 12 years; (ii) negative swabs of subjects who turned out to be positive in the following 15 days; (iii) swabs of subjects whose vaccination status against COVID-19 was undefined; (iv) salivary molecular swabs collected before May 14, 2021 and antigenic swabs with positive results carried out in pharmacies before December 31, 2021, in accordance with the surveillance policy of regional and national authorities [15, 16]; (v) all the swabs following the first positive one.

Study design

In this study population, we conducted a TNCC to estimate the COVID-19 VE against COVID-19 infection, hospitalization and death. For each case, four controls were randomly selected without replacement. Matching was based only on the date of swab collection (index date) while age, sex, province of residence and comorbidities score were included in models as covariates to avoid overmatching and control for confounding [17,18,19].

To estimate COVID-19 VE against infection (TNCC-INF), the cases were subjects with a first positive swab. The controls were randomly sampled among negative subjects matched by date of swab collection of the case.

Concerning COVID-19-related hospitalizations (TNCC-HOSP), the cases were swab positive subjects who were hospitalized in a COVID-19 ward, with a date of hospital admission within 30 days preceding the swab or 90 days following the swab date and the controls were sampled among swab negative, not hospitalized subjects.

The rationale behind the choice of these time cut-offs is explained as follow: according to the Italian Ministry of Health, any positive swab taken more than 90 days after the previous one defines a case of “re-infection”, and is considered as a different infectious event [20]. Furthermore, in order to include all hospitalizations of symptomatic subjects to COVID-19 wards with an admission date prior to that of the positive swab, we considered a period of 30 days as adequate.

To evaluate VE against COVID-19-related death (TNCC-DEATH), the cases were defined as subjects who died within 45 days following testing positive, while the controls were sampled among swab negative subjects at the date of death of the matched case.

Data from the Italian National Institute of Statistics show that 89% of COVID-19 related death occur within 30 days from the diagnosis, so the 45-days cut-off was chosen as adequate to include the vast majority of COVID-19 related deaths [21].

Our study covers periods characterized by different prevalent SARS-CoV-2 variants. They included: Alpha (Pango lineage B.1.1.7), Delta (B.1.617.2) and Omicron (B.1.1.529). VOC’s prevalence was estimated by the Italian National Institute of Health (ISS) which ran periodic regional surveys randomly sequencing a sample of daily swabs [22].

As mentioned above, the Italian application of the Green Pass legislation also determined a breaking time in the Italian COVID-19 epidemics, introducing the Green-Pass requirement for work and other daily-life activities [8,9,10]. In particular, the introduction of restrictions in the workplace forced many unvaccinated healthy workers to undergo several swabs every week in order to work, thus affecting the selection in our study. For these reasons our study hypothesis was tested separately in the following four sub-periods: (i) from February 1, 2021 to May 17, 2021 with prevalence of variant Alpha (PERIOD 1); (ii) from May 18, 2021 to October 14, 2021 with the transition and prevalence of variant Delta and before the introduction of Green-Pass requirement in workplaces (PERIOD 2); (iii) from October 15, 2021 to December 19, 2021 with prevalence of variant Delta and after the introduction of Green-Pass requirement in workplaces (PERIOD 3); (iv) from December 20, 2021 to March 31, 2022 with the transition and prevalence of variant Omicron (PERIOD 4).

Intervention

The national vaccination campaign began on December 27, 2020; by February 1, 2021 approximately 1,5% of the regional population > 12 years old had received its second jab [23] (Figure S1). The following COVID-19 vaccines were used in FVG, during the study period, by date: Comirnaty (BNT162b2), from February 1, 2021 to March 31, 2022, full cycle and booster dose; Vaxzevria (ChAdOx1-S), from February 17, 2021 to October 21, 2021, full cycle; Janssen (Ad26.COV2.S), from April 29, 2021 to March 8, 2022, full cycle; Spikevax (mRNA-1273), from February 13, 2021 to March 31, 2022, full cycle and booster dose; Nuvaxovid (NVX-CoV2373), from March 3, 2022 to March 31, 2022, full cycle. Subjects’ vaccination status against COVID-19, assessed on the date of swab’s collection, was defined as follows: (i) None: a person who had not taken any dose or who had taken the first dose and less than 15 days had passed since the first dose; (ii) Partly vaccinated: a person who had received the first dose and at least 15 days had passed since the first dose or a person who had received the second dose but less than 14 days had passed since the second dose; (iii) Full cycle: a person who received the first dose of Janssen or the second dose of another vaccine and at least 15 days had passed since the first Janssen dose or the second dose of another vaccine; (iv) Booster dose: a person who had received the booster dose and at least 15 days had passed since the booster dose.

Covariates

Gender, age, province of residence in FVG at swab’s collection, comorbidities score of each participant and number of swabs performed in the index date were considered as covariates.

Comorbidity was assessed by the Multisource Comorbidity Score (MCS), combining data from administrative health sources of FVG. The MCS is a risk adjustment tool based on hospital discharge diagnoses and drug prescriptions that measures the one-year risk of death. The higher the score, the higher the one-year risk of death [24].

The number of swabs performed in the month of the index date was also considered as a covariate in PERIOD 3 and in PERIOD 4 because testing might have become a confounder or a mediator of the relationship between vaccination status and the study outcomes after the aforementioned legislation was introduced on October 15, 2021.

Statistical analysis

We calculated frequency and percentage distribution of the main characteristics of the residents in FVG who were tested at least once in the study period and of cases and controls stratified by sub-period.

To estimate the association between COVID-19 vaccination and infection-related outcomes, simple and multiple conditional logistic regressions adjusted for covariates were performed. Odds Ratios (OR) and 95% confidence interval (95% CI) were estimated. COVID-19 VE was estimated as (1-ORin COVID−19 vaccinated vs. unvaccinated)x100. Since our choice privileged a design that estimated real-world VE in the actual mixed and waned population, we did not consider nor adjusted for the waning of vaccine protection.

In PERIOD 3 and PERIOD 4, we conducted stratified analysis to assess whether the estimated OR was modified by the number of swabs performed in the month of the index date. Furthermore, to better understand the mediation role of the latter variable (dichotomized in “2 swabs or more” versus “1 swab”), we conducted a mediation analysis between vaccination status (dichotomized in “At least 1 dose” versus “None”) and infection, hospitalization and death [25, 26]. In the TNCC-DEATH for both sub-period, 1000 bootstrap samples were used to estimate bias-corrected bootstrap confidence interval. Finally, in the mediation analysis model in the PERIOD 4 the variable age was considered continuous to achieve model convergence.

All the statistical analyses were performed using SAS (version 9.4 SAS Institute Inc., Cary, NC, USA).

Results

Population base

The main characteristics of the 627 982 subjects, aged 12 and over and residing in FVG with at least one swab in the whole study period are described in Table 1. Swabs sampled in this population base show an abrupt increase in the number of negative tests starting October 15, 2021, as a consequence of the introduction of Green Pass-related restrictions in workplaces (Fig. 1). A concurrent increase in the percentage of swabs taken by unvaccinated people is displayed in Figure S2.

Table 1 Characteristics of residents in FVG with at least one COVID-19 swab in the study perioda
Full size table
Fig. 1
figure 1

Frequency distribution of COVID-19 swabs tested in FVG in the study period by swab’s result

The VOC’s prevalence in the region FVG was estimated by the National Institute of Health (ISS) during periodic surveys [22]

Full size image

For each COVID-19-related outcome (infection, hospitalization and death), results are reported as follows:

  1. (i)

    a table with the results of the multiple conditional logistic regression analysis for each of the four sub-periods considered in the study (for PERIOD 3 and 4 the multiple conditional logistic regressions were also adjusted for the number of swabs performed);

  2. (ii)

    a figure with the results of the mediation analysis performed in the PERIOD 3 and 4;

  3. (iii)

    the frequency’s and percentage distributions of population’s characteristics and further sensitive analysis included in the supplementary materials.

Infection

In TNCC-INF, we identified 211 437 cases and 845 748 controls. The frequency and percentage distributions of the main characteristics of cases and controls are reported in Table S1. The corresponding results of multiple conditional logistic regression are displayed in Table 2: the full cycle VE against infection decreased from 96% (95% CI: 96, 97) in PERIOD 1 to 43% (95% CI: 42, 45) in PERIOD 4. Booster dose was able to rise the protection considerably. The results of mediation analysis (Fig. 2) in PERIOD 3, show that the total effect (TE), estimated by the OR, of vaccination status on COVID-19 infection was 2.78 (95% CI: 2.64, 2.93), but when the TE was decomposed into natural indirect effect (NIE) (OR: 6.97; 95% CI: 6.75, 7.19), and natural direct effect (NDE) (OR: 0.40; 95% CI: 0.38, 0.42), the effect was mainly mediated by the number of swabs performed in the month of the index date. In PERIOD 4 similar results were obtained: TE = 1.88 (95% CI: 1.85, 1.92), NIE = 3.60 (95% CI: 3.56, 3.64), NDE = 0.52 (95% CI: 0.52, 0.53). These results are consistent with those of the multiple conditional logistic regression stratified by the number of swabs performed in the month of the index date, shown in Table S2 and Table S3: the association described as the ORs between COVID-19 infection and vaccination status is confirmed in the different strata in both sub-periods.

Table 2 Multiple conditional logistic regression results by sub-periods. Outcome: COVID-19 infection
Full size table
Fig. 2
figure 2

Mediation analysis of the “number of swabs performed” on the association between COVID-19 infection and vaccination status

Path Diagram: a. Effect of vaccination status on the COVID-19 infection; b. Effect of vaccination status on the COVID-19 infection mediated by the number of swabs performed in the month of the index date

The numbers represent Odds Ratios and their 95% confidence interval (95% CI) in the total, indirect or direct effect estimates in the PERIOD 3 and 4. TE = Total Effect; NIE = Natural Indirect Effect; NDE = Natural Direct Effect

Full size image

Hospitalization

In the TNCC-HOSP, we identified 7867 cases and 31 468 controls. Frequency and percentage distributions of the main characteristics of cases and controls are presented in Table S4. The results of multiple conditional logistic regression, stratified by sub-periods, are presented in Table 3: compared to unvaccinated subjects, full cycle VE against COVID-19-related hospitalization was 97% (95% CI: 95, 98) and 84% (95% CI: 77, 89) in PERIOD 1 and 2, respectively. During PERIOD 3 and 4, the VE against hospitalization increased with increasing doses of vaccine. Particularly, VE of booster dose was 98% (95% CI: 95, 99) in PERIOD 3 and 87% (95% CI: 83, 90) in PERIOD 4.

The results of mediation analysis (Fig. 3) show that in PERIOD 3 the TE of vaccination status on hospitalization was 0.82 (95% CI: 0.60, 1.05), but when the TE was decomposed into NIE (OR: 7.09; 95% CI: 6.16, 8.02) and NDE (OR: 0.12; 95% CI: 0.09, 0.15), the effect was mainly mediated by the number of swabs performed in the month of the index date. In PERIOD 4 similar results were obtained: TE = 0.91 (95% CI: 0.76, 1.06), NIE = 3.39 (95% CI: 3.11, 3.66), NDE = 0.27 (95% CI: 0.23, 0.31). Consistently, multiple conditional logistic regression stratified by the number of swabs performed in the month of the index date (Table S5 and Table S6), suggests a strong association between COVID-19 hospitalization and vaccination for any dose of vaccine and the number of swabs performed. However, in some cases, the ORs were imprecise or could not be estimated due to a small number of observations in the different strata.

Table 3 Multiple conditional logistic regression results by sub-periods. Outcome: COVID-19-related hospitalization
Full size table
Fig. 3
figure 3

Mediation analysis of the “number of swabs performed” on the association between COVID-19-related hospitalization and vaccination status

Path Diagram: a. Effect of vaccination status on the COVID-19-related hospitalization; b. Effect of vaccination status on the COVID-19-related hospitalization mediated by the number of swabs performed in the month of the index date

The numbers represent Odds Ratios and their 95% confidence interval (95% CI) in the total, indirect or direct effect estimates in the PERIOD 3 and 4. TE = Total Effect; NIE = Natural Indirect Effect; NDE = Natural Direct Effect

Full size image

Death

Analysis of the TNCC-DEATH study included 2040 cases and 8160 controls. The frequency and percentage distributions of the main characteristics of cases and controls are reported in Table S7. The corresponding results of multiple conditional logistic regression are presented in Table 4. During the four sub-periods, VE against death increased with increasing vaccine doses performed, although estimates, in some cases, were imprecise due to small number of observations in some levels of vaccination status. Anyway, the VE against death was steadily high, full cycle VE varying from 98% (95% CI: 94; 99) in PERIOD 1 to 63% (95% CI: 31, 80) in PERIOD 4. Booster dose in PERIOD 4 was able to rise the protection to 90% (95% CI: 82, 95). Figure 4 shows the results of mediation analysis conducted in PERIOD 3 and 4. In PERIOD 3, the TE of vaccination status on COVID-19-related death was 2.50 (95% CI: 0.88, 4.80), but when the TE was decomposed into NIE (OR: 6.39; 95% CI: 3.85, 9.24) and NDE (OR: 0.33; 95% CI: 0.17, 0.69), the effect was mainly mediated by the number of swabs performed in the month of the index date. Also during PERIOD 4, the effect of vaccination status on the COVID-19-related death is mediated by of number of swabs performed in the month of the index date: TE = 0.70 (95% CI: 0.49, 0.99), NIE = 3.13 (95% CI: 2.68, 3.70), NDE = 0.22 (95% CI: 0.16, 0.32). The multiple conditional logistic regression stratified by the number of swabs gives consistent results in the strata with 1 swab for both sub-period (Table S8 and Table S9). However, for the remaining strata the ORs are not presented due to the low number of observations.

Table 4 Multiple conditional logistic regression results by sub-periods. Outcome: COVID-19-related death
Full size table
Fig. 4
figure 4

Mediation analysis of the “number of swabs performed” on the association between COVID-19-related death and vaccination status

Path Diagram: a. Effect of vaccination status on the COVID-19-related death; b. Effect of vaccination status on the COVID-19-related death mediated by the number of swabs performed in the month of the index date

The numbers represent Odds Ratios and their 95% confidence interval (95% CI) in the total, indirect or direct effect estimates in the PERIOD 3 and 4. TE = Total Effect; NIE = Natural Indirect Effect; NDE = Natural Direct Effect. One thousand bootstrap samples were used to estimate bias-corrected bootstrap confidence interval. In the model of PERIOD 4 the variable age was considered continuous to achieve model convergence

Full size image

Discussion

At the design stage of our study, we were concerned about the potential presence of selection bias as this is a major issue in all observational studies conducted in the field of VE. To mitigate this issue, we chose a Test-Negative design that confirmed, via testing, that controls were not infected at the time of case diagnosis. In addition, the TNCC approach controls for some health seeking behavior [4]. However, it has been demonstrated that in the Test-Negative design a source of selection bias could be represented by a collider variable related to the propensity to be included in the sample [11].

This was the situation in Italy between October 15, 2021 and March 31, 2022. During the period of the Green-pass requirement for workers, we observed that unvaccinated people had to undergo several tests every week in order to work. This was unrelated to other indications and to the health seeking behavior, and thus introduced a substantial difference in the testing process between vaccinated and unvaccinated, consequently affecting inclusion in the study. In fact, to access their workplace healthy unvaccinated subjects conducted many tests that resulted as negative, and therefore could be selected as controls.

Since such a bias was not intrinsically controlled by the study design, we adopted the abovementioned strategy to conduct for PERIOD 3 and PERIOD 4 adjustment and stratified analysis by the number of swabs performed in the month of the index date: the resulting estimates of VE are in accordance with most available literature on the topic. Also, the mediation analysis that was carried out confirmed the strong mediator role of the number of swabs performed in PERIOD 3 and 4 and the importance of such adjustments.

In the following subsections, separately for each outcome taken into consideration, we discuss the results obtained, provide a comparison of the estimated VE with available evidence, and argue the limitations of the study.

Infection

In the Alpha period, our estimates of protection against infection align well with another Italian study which gave similar results [27], and with other TNCC studies [28]. In particular, one meta-analysis of TNCC studies [29] gave very similar results for the Moderna and Pfizer vaccine, which cover 88% of vaccinations in the FVG region [23]. A comparison with another meta-analysis [30], is available in Table S10. During the Delta phase, our results of PERIOD 2 and PERIOD 3 are quite different. The available evidence seems to align better with PERIOD 3, after the restrictions for unvaccinated were introduced [29,30,31,32]. This also suggests that the adjustment for the number of tests undergone by participants reduced the bias in PERIOD 3 and gave result that are sufficiently externally valid. During PERIOD 4, our results are rather consistent with the evidence from the Italian National Institute of Health (ISS) [33] and other TNCC studies [34, 35].

Stratified results (Table S2 and Table S3) confirmed a strong protective effect of vaccination for any dose of vaccine and number of swabs.

The corresponding mediation analysis was able to demonstrate further a strong protective effect of vaccination with the concurrent indirect effect of testing. In fact, the variable related to the testing behavior is associated with higher odds of infection, as shown in the NIE (ORs: 6.97 and 3.60), and had not been taken into account, this effect would have biased considerably the analysis: therefore, this justifies the adjustment for the number of swabs performed in PERIOD 3 and PERIOD 4. When interpreting this result, we have to bear in mind that higher odds of infection associated with the number of tests performed is not only related to the selection bias, but might reflect also cumulative risk-taking behaviours, as people who choose not to get vaccinated (and will be likely be tested more often due to requirements in place) might also choose not to engage in non-pharmaceutical interventions (NPIs).

The protective effect of the vaccine is made visible in the NDE of the mediation analysis (ORs: 0.40 and 0.52), which represents the effect of the independent variable (dichotomized vaccination status) on the dependent variable (COVID-19 outcome) that is not explained by the mediator, or, in other words, if the mediator was not present.

Potential limitations of the results presented include the effect of previous infection not reported in clinical records: some subject might have developed an immunity which was not documented, and this is especially true in the earlier pandemic stages. If unvaccinated subjects were more prone to develop unrecorded infections, this could underestimate the VE. Differential exposure to risk could also result in spuriously altered VE estimates.

Furthermore, multiple sources of differential health seeking behavior could have been present, for example individuals with vaccination side effects might be more likely to seek medical care and testing, even if the design of the study attempted to tackle this aspect in an unbiased way.

Hospitalization

The main methodological challenges and approach of the TNCC-INF also apply to severe outcomes. As shown in Table S10, regarding hospitalization, our results are consistent with the meta-analysis by Shao et al., since for all VOCs the 95% CIs largely overlap [30]; in PERIOD 1 our results align well also with local evidence [27]. In the two Delta periods, the difference between full cycle estimates is small and both are consistent with local evidence from Italy and with other European studies [31, 32, 36], and this is also for true for estimates in PERIOD 4 [37, 38].

Results for VE of partly vaccinated were limited by the small number of corresponding strata, as seen in Table S4. Also the stratified analysis available in Tables S5 and S6 are affected by the small numbers, and achieve statistical significance mainly in the strata relative to 1 swab performed in the month of the index date.

In the mediation analyses, both TE of PERIOD 3 and PERIOD 4 suggest an overall protective effect, as a combination of both the direct and indirect effects, even if they are not statistically significant. The protective effect is much more evident after the mediator effect is controlled for in the NDE (ORs: 0.12 and 0.27). This confirms that testing behavior plays a crucial role in the overall relationship between vaccination status and the outcome, and needs to be considered when assessing the protective effect of vaccination also regarding hospitalization.

Together with the low numbers of some strata, additional limitations are attributable to factors similar to those discussed in the previous paragraph regarding infections.

Death

Overall, COVID-19 vaccines’ VE was high against death, although the interpretation of results should be cautious due to small numbers in some strata, especially for partial and booster dose VE. Also the multiple conditional logistic regression stratified by number of swabs was largely affected by sparse data.

Keeping this limitation in mind, 95%CI of our estimates largely overlap with the meta-analysis by Shao et al. [30] in every period taken into consideration (Table S10). The TNCC study by Castillo et al. also has similar results for Delta and Omicron [37].

In the mediation analysis, the TE in PERIOD 4 shows, overall, that vaccine status is associated with lower odds of death. The testing behaviour-related variable nonetheless acts as a strong mediator between vaccine status and death, and after it is controlled for, as shown in the NDE, a higher protective effect of vaccine status against death is revealed in both PERIOD 3 and PERIOD 4 (ORs: 0.33 and 0.22).

The definition of cases that we applied to COVID-19-related deaths employed a broad time-cut-off, including subjects who died within 45 days following testing positive. Therefore, it is possible that other clinical factors could have had time to develop and contribute to the patient’s death as well as COVID-19.

Conclusions

In conclusion, the study suggests that due to the complexity of demonstrating VE in this real-world setting for the potential biases inherent in observational investigations, even under a TNCC design, when differential testing behavior is present determining substantial selection bias, mediation analysis and adjustment for number of diagnostic testing should be included. This correction allowed us to align with results from other studies that show how full-cycle VE against infection was initially high but decreased consistently over time by variant circulation, counterbalanced by booster dose which was able to increase protection in every period taken into consideration. A highly effective protection given by COVID-19 vaccines was also demonstrated against hospitalization and death.

Data Availability

The data underlying this article will be shared on reasonable request to the corresponding author.

Abbreviations

CI:

Confidence Interval;

COVID-19:

Coronavirus disease 2019

FVG:

Friuli Venezia-Giulia;

MCS:

Multisource Comorbidity Score;

NDE:

Natural Direct Effect;

NIE:

Natural Indirect Effect;

NPIs:

Non-Pharmaceutical Interventions;

OR:

Odds Ratio;

RCT:

Randomized Clinical Trial;

TE:

Total Effect;

TNCC:

Test-Negative Case Control;

VE:

Vaccine effectiveness;

VOC:

Variant Of Concern;

WHO:

World Health Organization;

References

  1. European Medicines Agency. COVID-19 Medicines. Available from: https://www.ema.europa.eu/en/human-regulatory/overview/public-health-threats/coronavirus-disease-covid-19/covid-19-medicines; Accessed: May 2023.

  2. International Vaccine Access Center. Available from: https://view-hub.org/covid-19/effectiveness-studies; Accessed: May 2023.

  3. Lipstich M, Jha A, Simonsen L. Observational studies and the difficult quest for causality: lessons from vaccine effectiveness and impact studies. Int J Epidemiol. 2016;45:2060–74. https://doi.org/10.1093/ije/dyw124

    Article  Google Scholar 

  4. World Health Organization. Evaluation of COVID-19 vaccine effectiveness - Interim Guidance 17 March 2021. Available from: https://www.who.int/publications/i/item/WHO-2019-nCoV-vaccine_effectiveness-measurement-2021.1; Accessed: May 2023.

  5. Patel MK, Bergeri I, Bresee JS et al. Evaluation of post-introduction COVID-19 vaccine effectiveness: Summary of interim guidance of the World Health Organization, Vaccine. 2021 July; 39: 4013–24. https://doi.org/10.1016/j.vaccine.2021.05.099

  6. Eur-Lex. Regulation (EU) 2021/953. Available from: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32021R0953; Accessed: May 2023.

  7. Montanari Vergallo G, Del Rio A, Negro F, Zaami S. COVID-19 vaccine mandates: what are the current European public perspectives? European review for medical and pharmacological sciences. Eur Rev Med Pharmacol Sci. 2022;26:643–52. https://doi.org/10.26355/eurrev_202201_27891

    Article  CAS  PubMed  Google Scholar 

  8. The Official Gazette of the Italian Republic (Gazzetta Ufficiale Della Repubblica Italiana). Law Decree 6/8/2021, n. 111. Available from: https://www.gazzettaufficiale.it/eli/id/2021/08/06/21G00125/sg; Accessed: May 2023.

  9. The Official Gazette of the Italian Republic (Gazzetta Ufficiale Della Repubblica Italiana). Law Decree 23/7/2021, n. 105. Available from: https://www.gazzettaufficiale.it/eli/id/2021/07/23/21G00117/sg; Accessed: May 2023.

  10. The Official Gazette of the Italian Republic (Gazzetta Ufficiale Della Repubblica Italiana). Law Decree 21/9/2021, n. 127. Available from: https://www.gazzettaufficiale.it/eli/id/2021/09/21/21G00139/sg; Accessed: May 2023.

  11. Griffith G, Morris T, Tudball M, et al. Collider bias undermines our understanding of COVID-19 Disease risk and severity. Nat Commun. 2020;11:5749. https://doi.org/10.1038/s41467-020-19478-2

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Sullivan S, Tchetgen E, Cowling B, et al. Theoretical basis of the test-negative Study Design for Assessment of Influenza Vaccine Effectiveness. Am J Epidemiol. 2016;184:345–53. https://doi.org/10.1093/aje/kww064

    Article  PubMed  PubMed Central  Google Scholar 

  13. Italian Institute of Statistics (ISTAT). Permanent Censuses. Available from: http://dati-censimentipermanenti.istat.it/?lang=it; Accessed: May 2023.

  14. Riccardo F, Ajelli M, Andrianou XD, et al. Epidemiological characteristics of COVID-19 cases and estimates of the reproductive numbers 1 month into the epidemic, Italy, 28 January to 31 March 2020. Euro Surveill. 2020;25:2000790. https://doi.org/10.2807/1560-7917.es.2020.25.49.2000790

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Italian Ministry of Health (Ministero della Salute). 30/12/2021 Circular from the Ministry of Health. Available from: https://www.salute.gov.it/portale/nuovocoronavirus/archivioNormativaNuovoCoronavirus.jsp; Accessed: May 2023.

  16. Italian Ministry of Health (Ministero della Salute). 14/05/2021 Circular from the Ministry of Health. Available from: https://www.salute.gov.it/portale/nuovocoronavirus/archivioNormativaNuovoCoronavirus.jsp; Accessed: May 2023.

  17. Spuri M, Cataldo C, Del Manso M, et al. Epidemiological characteristics of COVID-19 cases in Italy: an analysis from a sex/gender perspective. Ital J Gender-Specific Med. 2022;8:3–9. https://doi.org/10.1723/3769.37561

    Article  Google Scholar 

  18. Rothman K, Greenland S, Lash T. Design strategies to Improve Study Accuracy. Modern epidemiology. 3rd ed. Philadelphia: Lippincott Williams & Wilkins; 2008. pp. 168–83.

    Google Scholar 

  19. Pearce N. Analysis of matched case-control studies. BMJ. 2016; 352: i969. https://doi.org/10.1136/bmj.i969

  20. Italian Ministry of Health (Ministero della Salute). Ministerial Note n. 0037911 dated 20/08/2021. “Flusso dati aggregati Ministero della Salute/Protezione Civile: aggiornamento sulla possibilità di inserimento delle reinfezioni da SARS-CoV-2″. Available from: https://www.seremi.it/content/flusso-dati-aggregati-ministero-della-saluteprotezione-civile-aggiornamento-sulla; Accessed: October 2023.

  21. Italian Institute of Statistics (ISTAT). “Impatto dell’epidemia covid-19 sulla mortalità totale della popolazione residente. Anni 2020–2021 e gennaio 2022” Available from: https://www.istat.it/it/files//2022/03/Report_ISS_ISTAT_2022_tab3.pdf; Accessed: October 2023.

  22. Italian National Institute of Health (Istituto Superiore di Sanità). Monitoring of the variants of the SARS-CoV-2 virus of public interest in Italy. Available from: https://www.epicentro.iss.it/coronavirus/sars-cov-2-monitoraggio-varianti-rapporti-periodici; Accessed: May 2023.

  23. Extraordinary Commissioner for the Covid-19 emergency. Open data on the delivery and administration of COVID-19 vaccines in Italy. Available from: https://github.com/italia/covid19-opendata-vaccini; Accessed: May 2023.

  24. Corrao G, Rea F, Di Martino M, et al. Developing and validating a novel multisource comorbidity score from administrative data: a large population-based cohort study from Italy. BMJ Open. 2017;7:e019503. https://doi.org/10.1136/bmjopen-2017-019503

    Article  PubMed  PubMed Central  Google Scholar 

  25. Rijnhart JJM, Lamp SJ, Valente MJ, et al. Mediation analysis methods used in observational research: a scoping review and recommendations. BMC Med Res Methodol. 2021;21:226. https://doi.org/10.1186/s12874-021-01426-3

    Article  PubMed  PubMed Central  Google Scholar 

  26. VanderWeele T, Tchetgen Tchetgen E. Mediation Analysis With Matched Case-Control Study Designs. Am J Epidemiol. 2016; 183: 869–870. https://doi.org/10.1093/aje/kww038

  27. Flacco M, Soldato G, Acuti Martellucci C et al. Interim Estimates of COVID-19 Vaccine Effectiveness in a Mass Vaccination Setting: Data from an Italian Province. Vaccines. 2021 June; 9: 628. https://doi.org/10.3390/vaccines9060628

  28. Sheikh A, McMenamin J, Taylor B, Robertson C. SARS-CoV-2 Delta VOC in Scotland: demographics, risk of hospital admission, and vaccine effectiveness. Lancet. 2021 June; 397: 2461–2462. https://doi.org/10.1016/S0140-6736(21)01358-1

  29. Chang S, Liu H, Wu J et al. Effectiveness of BNT162b2 and mRNA-1273 Vaccines against COVID-19 Infection: A Meta-Analysis of Test-Negative Design Studies. Vaccines. 2022 March; 10: 469. https://doi.org/10.3390/vaccines10030469

  30. Shao W, Chen X, Zheng C et al. Effectiveness of COVID-19 vaccines against SARS-CoV-2 variants of concern in real-world: a literature review and meta-analysis. Emerg Microbes Infect. 2022; 11: 2383–2392. https://doi.org/10.1080/22221751.2022.2122582

  31. Italian National Institute of Health (Istituto Superiore di Sanità). Integrated surveillance bulletin dated 21/12/2021. Available from: https://www.epicentro.iss.it/coronavirus/2021; Accessed: May 2023.

  32. Fabiani M, Puopolo M, Morciano C, et al. Effectiveness of mRNA vaccines and waning of protection against SARS-CoV-2 Infection and severe covid-19 during predominant circulation of the delta variant in Italy: retrospective cohort study. BMJ. 2022 February;376:e069052. https://doi.org/10.1136/bmj-2021-069052

  33. Italian National Institute of Health (Istituto Superiore di Sanità). Integrated surveillance bulletin dated 30/3/2022. Available from: https://www.epicentro.iss.it/coronavirus/2022; Accessed: May 2023.

  34. Willet B, Grove J, MacLean O et al. The hyper-transmissible SARS-CoV-2 Omicron variant exhibits significant antigenic change, vaccine escape and a switch in cell entry mechanism. Nat Microb. 2022 September; 7: 1161–1179. https://doi.org/10.1101/2022.01.03.21268111

  35. Tseng H, Ackerson B, Luo Y et al. Effectiveness of mRNA-1273 against SARS-CoV-2 Omicron and Delta variants. Nat Med. 2022 February; 28: 1063–1071. https://doi.org/10.1038/s41591-022-01753-y

  36. Martellucci C, Flacco M, Soldato G et al. Effectiveness of COVID-19 Vaccines in the General Population of an Italian Region before and during the Omicron Wave. Vaccines. 2022 April; 10: 662. https://doi.org/10.3390/vaccines10050662

  37. Castillo M, Khaoua H, Courtejoie N. Vaccine-induced and naturally-acquired protection against Omicron and Delta symptomatic infection and severe COVID-19 outcomes, France, December 2021 to January 2022. Euro surveill. 2022 April; 27: 2200250. https://doi.org/10.2807/1560-7917.ES.2022.27.16.2200250

  38. Vitek M, Klavs I, Ucakar V. al. Vaccine effectiveness against severe acute respiratory infections (SARI) COVID-19 hospitalisations estimated from real-world surveillance data, Slovenia, October 2021. Euro surveill. 2022 January; 27: 2101110. https://doi.org/10.2807/1560-7917.ES.2022.27.1.2101110

Download references

Acknowledgements

We thank Carlo Bolzonello for supporting F.D.B. training.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Author notes

  1. Rosolen Valentina and Turoldo Federico are joint first authors.

Authors and Affiliations

  1. Central Directorate for Health, Social Policies and Disability, Friuli Venezia Giulia Region, Via Cassa Di Risparmio 10, Trieste, 34121, Italy

    Valentina Rosolen, Gianna Zamaro, Luigi Castriotta & Fabio Barbone

  2. Department of Medicine, Surgery and Health Sciences, University of Trieste, Strada di Fiume 447, Trieste, 34149, Italy

    Federico Turoldo & Fabio Barbone

  3. Prevention Technical Platform, “AS FO” Western Friuli Health Authority, Via della Vecchia Ceramica 1, Pordenone, 33170, Italy

    Flavio Del Bianco

  4. Department of Infectious Diseases, National Institute of Health (ISS), Viale Regina Elena 299, Rome, 00161, Italy

    Patrizio Pezzotti

  5. Institute of Hygiene and Evaluative Epidemiology, Friuli Centrale University Health Authority, Via Colugna 50, Udine, 33100, Italy

    Luigi Castriotta

Authors
  1. Valentina Rosolen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Federico Turoldo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gianna Zamaro
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Flavio Del Bianco
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Patrizio Pezzotti
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Luigi Castriotta
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Fabio Barbone
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Concept: F.B., V.R., F.T., L.C., G.Z., P.P. Literature review: F.T., V.R., F.B. Formal analysis: V.R., F.D.B., P.P. Writing original draft: V.R., F.T., L.C., P.P., F.B. Writing review and editing: V.R., F.T., F.B., G.Z., P.P., L.C., F.D.B.

Corresponding author

Correspondence to Fabio Barbone.

Ethics declarations

Ethical approval

This study did not require ad hoc ethical approval because all data derive from pre-approved National and Regional COVID-19 testing and vaccination monitoring system which protected any individual privacy.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1: Figure S1

. Trend of vaccination coverage in the population of FVG. Figure S2. Percentage distribution of COVID-19 swabs taken in Friuli Venezia Giulia Region (1/2/2021-31/3/2022), by week, sub-period, and COVID-19 vaccination status. Table S1. Frequency and percentage distribution of cases and controls characteristics by sub-period. Outcome: COVID-19 Infection. Table S2. Frequency and percentage distribution of vaccination status in cases and controls, and multiple conditional logistic regression results stratified by number of swabs performed in the month of the index date. Outcome: COVID-19 Infection, PERIOD 3 (15/10/2021-19/12/2021). Table S3. Frequency and percentage distribution of vaccination status in cases and controls, and multiple conditional logistic regression results stratified by number of swabs performed in the month of the index date. Outcome: COVID-19 infection, PERIOD 4 (20/12/2021-31/3/2022). Table S4. Frequency and percentage distribution of cases and controls characteristics by sub-periods. Outcome: COVID-19-related hospitalization. Table S5. Frequency and percentage distribution of vaccination status in cases and controls, and multiple conditional logistic regression results stratified by number of swabs performed in the month of the index date. Outcome: COVID-19-related hospitalization, PERIOD 3 (15/10/2021-19/12/2021). Table S6. Frequency and percentage distribution of vaccination status in cases and controls, and multiple conditional logistic regression results stratified by number of swabs performed in the month of the index date. Outcome: COVID-19-related hospitalization, PERIOD 4 (20/12/2021-31/3/2022). Table S7. Frequency and percentage distribution of cases and controls characteristics by sub-periods. Outcome: COVID-19-related death. Table S8. Frequency and percentage distribution of vaccination status in cases and controls, and multiple conditional logistic regression results stratified by number of swabs performed in the month of the index date. Outcome: COVID-19-related death, PERIOD 3 (15/10/2021-19/12/2021). Table S9. Frequency and percentage distribution of vaccination status in cases and controls, and multiple conditional logistic regression results stratified by number of swabs performed in the month of the index date. Outcome: COVID-19-related death, PERIOD 4 (20/12/2021-31/3/2022). Table S10. Comparison of our results with the meta-analysis from Shao et al. [30].

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rosolen, V., Turoldo, F., Zamaro, G. et al. COVID-19 vaccination effectiveness in the population of Friuli Venezia Giulia, North-East Italy. Control of bias associated with divergent compliance to policies in a test-negative case-control study. BMC Public Health 23, 2476 (2023). https://doi.org/10.1186/s12889-023-17244-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12889-023-17244-9

Keywords

BMC Public Health

ISSN: 1471-2458

Contact us