Introduction

Influenza is an infectious disease associated with high morbidity and mortality globally1,2, and vaccination is the most effective means of eradication3. In Korea, influenza vaccines have been provided as part of the National Immunization Program (NIP), and eligible groups have continuously expanded since their introduction in 1997. Accordingly, the number of adverse events (AEs) following immunization for influenza has been steadily increasing according to the Korea Adverse Event Reporting System (KAERS)4. Safety profiles are a collection of adverse events5, including the information identified from clinical trials or a post-marketing phase and unidentified to date. Especially, unidentified information or safety signal represents “reported information on a possible causal relationship between an AE and a vaccine or drug, of which the relationship is unknown or incompletely evaluated previously”6. In particular, concerns regarding serious AEs of influenza vaccination have been continuously highlighted, including Guillain–Barré syndrome (GBS), febrile seizure, anaphylaxis, narcolepsy, and Bell’s palsy7,8,9,10,11,12. However, only a few studies on AEs associated with influenza vaccines have been conducted in Korea7,8. Therefore, it is necessary to re-evaluate the safety information of influenza vaccines using the latest and widely collected Korean databases.

Influenza vaccines can be classified into two types in terms of the type of production method or virus immunization: (a) egg-based vaccines and cell-based vaccines for the former, and (b) trivalent influenza vaccines (TIVs) and quadrivalent influenza vaccines (QIVs) for the latter. Several studies have reported that the safety of influenza vaccines may vary according to subtypes13,14,15, but most of the evidence is dependent on the results of randomized clinical trials (RCTs) with a limited population and study period. Therefore, an additional study with a large-scale database is needed to establish safety profiles by subtype.

There are two types of methods to detect safety signals (quantitative and qualitative method), and a variety of quantitative data mining methods are available for post-marketing safety monitoring. Typically, disproportionality-based methods are widely used to detect safety signals based on an imbalance between the reporting rate of specific AEs for a specific drug and that of specific AEs for other drugs. The proportional reporting ratio (PRR), reporting odds ratio (ROR), and information component (IC) are the most common measurements in disproportionality-based methods16. Recently, tree-based scan statistics (TSS) have been applied in the signal detection of vaccines17,18, which were proposed to analyze a database with a hierarchical tree structure with adjustment for multiple comparisons19. Although various data mining methods have been developed and widely used for detecting signals, inconsistent results have been reported using different methods17,18, implying that the use of multiple data mining methods should be considered simultaneously.

Given the limited safety profiles on influenza vaccine and inconsistent results by different data mining methods, we conducted a nationwide, post-marketing surveillance study to detect adverse events following influenza vaccination using three different data mining methods. Firstly, we identified unexpected AEs which are potentially related to influenza vaccination, particularly focusing on five serious AEs of special interest (GBS, febrile seizure, anaphylaxis, narcolepsy, and Bell’s palsy). Secondly, we aimed to provide comparative safety profiles by the subtype of influenza vaccine (type of production method: egg-based vs. cell-based; type of virus immunization: QIVs vs. TIVs). Lastly, we compared the results of signal detection from each data mining methods to evaluate their performance on predicting safety signals.

Results

General characteristics

The basic characteristics of influenza vaccine-related AEs compared with those of other vaccines are provided in Table 1. More AEs were reported in females than in males for both influenza vaccines and all other vaccines, accounting for 68.77% and 57.30%, respectively. The proportion of children below 24 months in the influenza vaccine group was higher than that of other vaccines, while that of individuals 19 years or older in the influenza vaccine group was lower than that of other vaccines. When stratified by route of administration, most of the characteristics between the influenza vaccine and other vaccines were similar, except for the origin and year of the report: most came from a pharmaceutical company for all other vaccines (74.71%), while pharmaceutical companies with regional pharmacovigilance centers (RPVC) for influenza vaccines (52.84% and 42.57%, respectively) and the proportion of serious influenza vaccine-related AEs (5.18%) were less than that of other vaccines (8.06%).

Table 1 Characteristics of adverse event reports following influenza vaccination and all other vaccines between 2005 and 2019.
Full size table

Signal detection

Compared to all other vaccines, we identified 36 AEs as safety signals of influenza vaccines detected by any data mining method at least once (see Supplementary Material S2 online). Among them, seven AEs (injection site inflammation, muscle weakness, quadriplegia, paraplegia, hyperventilation, cachexia, and narcolepsy) were not included in the labeling information and were detected by all data mining methods, except for paraplegia (not in TSS). For serious AEs of special interest, anaphylaxis, neuritis (a preferred term [PT] of GBS), and narcolepsy were detected as signals compared to all other vaccines by all data mining methods, and paralysis (a PT of Bell’s palsy) was detected by IC and TSS (Table 2). Additionally, each method detected a similar number of safety signals (PRR, 29; IC, 30; TSS, 27).

Table 2 Unlabeled and adverse events of special interest among signal detection results of influenza vaccination using disproportionality methods and tree-based scan statistics from 2005 to 2019.
Full size table

Different findings have been reported for influenza vaccine subtypes using various methods. When using TIV as a comparator, 13 AEs were detected as safety signals by any data mining method at least once for QIV, and among them (see Supplementary Material S3 online), two safety signals (injection site bruising and cachexia) were not included in the labeling (Table 3). Moreover, each method generated a similar number of safety signals: 13 signals for PRR and TSS and 12 signals for IC. In contrast, we detected 59 AEs as safety signals at least once for TIV when compared to QIV; 26 safety signals were not included in labeling, but only 5 of these were overlapped by the three methods.

Table 3 Unlabeled signals among signal detection results of quadrivalent and trivalent influenza vaccines using disproportionality methods and tree-based scan statistics from 2005 to 2019.
Full size table

Compared to the egg-based vaccine, we identified 13 AEs as safety signals of cell-based vaccines using at least one data mining method (see Supplementary Material S4 online). Among them, four safety signals (injection site bruising, pneumonitis, encephalopathy, and tachycardia) were not included in the labeling, but only injection site bruising was detected by all three methods (Table 4). PRR, IC, and TSS detected 9, 13, and 6 safety signals, respectively. When using cell-based vaccines as a comparator, 30 AEs were detected as safety signals by any data mining method at least once for egg-based vaccines, 4 of which (leg pain, tremor, injection site mass, and cachexia) were not included in the labeling. PRR, IC, and TSS detected 20, 2, and 23 safety signals, respectively.

Table 4 Unlabeled signals among signal detection results of cell-based and egg-based influenza vaccines using disproportionality methods and tree-based scan statistics from 2005 to 2019.
Full size table

Performance evaluation

We measured the performance indicators of each data mining method based on the confusion matrix, where all detected signals were categorized into true positive (TP), false positive (FP), true negative (TN), or false negative (FN) based on the reference standard. Overall, TSS demonstrated well balance. TSS showed the highest sensitivity (77.8%), positive predictive value (PPV, 77.8%), accuracy (72.1%), and area under the curve (AUC, 70.1%) and the lowest specificity (62.5%). In contrast, IC showed the highest specificity (94.7%) and negative predictive value (NPV, 64.3%). In addition, except for specificity, all PRR measurements were the lowest (Fig. 1).

Figure 1
figure 1

Bar chart of performance measurements for signal detection algorithms about influenza vaccine. Abbreviations: IC, information component; PRR, proportional reporting ratio; TSS, tree-based scan statistic; PPV, positive predictive value; NPV, negative predictive value; AUC, area under the curve.

Full size image

Discussion

We conducted a safety surveillance study on influenza vaccines and their subtypes using a nationwide spontaneous reporting database with three different data mining methods. The total number of signals detected by disproportionality-based analysis and TSS for the influenza vaccine compared to other vaccines were similar, and few unexpected AEs for the overall influenza vaccine were observed. Additional signal detection according to subtype was also conducted, where we noted that a wider range of AEs were detected as signals for TIVs and egg-based vaccines. Regarding the evaluation for signal detection, TSS showed better performance than the other data mining methods, with the optimal balance among all performance measurements.

From the results of the signal detection for the overall influenza vaccine, we identified that most signals were known AEs, such as application site disorders, body as a whole-general disorder, and musculoskeletal system disorders. All five serious AEs of special interest for influenza vaccine were detected as signals, except for febrile seizure. Similar findings were reported in two recent studies, both of which used two disproportionality-based methods (PRR and IC) for safety signal detection of influenza vaccines using the U.S. Vaccine Adverse Event Reporting System (VAERS), where GBS and paralysis were detected as signals by all three methods, respectively20,21. Among the seven detected signals not listed in the labeling in South Korea, injection-site inflammation and muscle weakness are reflected in the U.S. Food and Drug Administration (US-FDA) labeling. Accordingly, injection site inflammation and muscle weakness should be updated after signal evaluation. Of the remaining five unlabeled signals in both the U.S. and South Korea, narcolepsy seems to require urgent assessment. As mentioned earlier, narcolepsy cases grew rapidly following the 2009–2010 H1N1 vaccination in Finland and five other European countries (see Supplementary Material S5 online). Subsequent studies have shown that the AS03-adjuvanted pandemic vaccine Pandemrix (GlaxoSmithKline Biologicals, Wavre, Belgium), which was widely used in these countries during that period, contains a protein structurally similar to the orexin receptor, which is likely to induce an autoimmune response that destroys orexin cells in the brain and causes narcolepsy22. The H1N1 vaccine adjuvanted with AS03 was not used in Korea, but we identified narcolepsy as a safety signal for the influenza vaccine compared with other vaccines in this study, although there were only 11 reported cases. Therefore, signal validation through an in-depth evaluation of these cases is required.

Furthermore, we noted that the distribution of AEs differed between QIVs and TIVs. Local reactions at the injection site were more frequently reported in QIVs than in TIVs in our studies, which is consistent with previous studies. A meta-analysis of 5 RCTs reported that the incidence of injection site pain was higher with QIVs than TIVs with a pooled risk ratio of 1.18 (95% confidence interval: 1.03–1.35)23. In addition, a wider range of AEs were detected as signals for TIVs in our study, which showed inconsistent results with a previous RCT24 that demonstrated that QIVs and TIVs showed similar profiles. This may be attributed to the different distribution of age groups between QIVs and TIVs, with the proportion of the older group (65 years and above) being 0.47% and 10.1%, respectively. The aged population is vulnerable to adverse drug reactions due to age-related physiological changes25.

Moreover, we detected a wider range of AEs as signals with egg-based vaccines compared with cell-based vaccines, suggesting that safety profiles among these two subtypes may differ, which is inconsistent with the results of previous studies. Two RCTs demonstrated that cell-based and egg-based TIVs had comparable safety profiles in children and adolescents < 18 years of age26,27. We believe that this situation may be due to the following reasons. First, compared with cell-based vaccines, an inherent drawback of egg-based vaccines is that impurities or exogenous contamination may cause AEs28. In addition, Korea approved the first cell-based influenza vaccine in 2014, and egg-based influenza vaccines still dominate the majority of the market share29. In our database, cell-based vaccines only accounted for 15% of all cases; therefore, it is possible that some AEs of cell-based vaccines have not yet been revealed. Therefore, it is necessary to collect additional information.

In this study, TSS had the highest accuracy (72.1%), AUC (70.1%), PPV (77.8%), and sensitivity (77.8%). TSS allows the simultaneous evaluation of a large number of individual AE terms and related AE groups, thereby resolving multiple testing problems of disproportionality-based methods30. Two previous studies of the BCG vaccine and pneumococcal vaccine using the KAERS database also showed balanced performance in TSS17,18. However, we found that the sensitivity of TSS in this study was much higher than that in the BCG and pneumococcal vaccine studies (35.3% and 31.7%, respectively). There are two possible explanations for this observation. First, safety profiles are relatively well-established in the influenza vaccine, which targets the entire population as more extensive AEs have been reported and studied, compared to BCG and pneumococcal vaccines, both of which have limited recipients. Furthermore, the ability to detect signals through data mining techniques is largely influenced by the size of the dataset, which differs significantly between influenza and other vaccines (current study, 17,378 AE reports; BCG, 833 AE reports; pneumococcal vaccine, 1135 AE reports).

This study had several limitations. First, quantitative signal detection using passive surveillance systems, such as the KAERS database, has the inherent limitations of missing information, inconsistent quality of individual case safety reports, duplicated reporting, and under-reporting due to lack of awareness31. Therefore, our results do not reflect a causal relationship between vaccines and AEs, and further signal validation and assessment are needed. Second, in this study, the reference standard was established based on Ministry of Food and Drug Safety of South Korea (MFDS)-approved labeling for performance evaluation; thus, the results could be largely affected by the reference standard32. However, we believe that the labeling included all related AEs since influenza vaccines have been used for a long time. Third, our findings should not be overstated that TSS is the best-balanced method to detect safety signals in other vaccines or even in drugs as individual vaccine or drugs has different medical or clinical context. Therefore, further studies should be conducted to confirm our finding on the performance TSS across different vaccines and drugs. Lastly, although TSS showed the well-balanced performance compared to other data mining methods, TSS did not show the best values in all performance measurements, particularly in specificity. This result could be explained by the inverse relationship between sensitivity and specificity when considering the highest sensitivity in TSS, which were about threefold higher compared with the other methods (TSS: 77.8%, PRR: 21.1%, IC: 25.5%). This result suggests that the utilization of multiple data mining methods is reasonable choice to detect safety signals conservatively. In addition, the low values of accuracy and AUC were observed across all methods (TSS: 72.1%, 70.1%; PRR: 63.2%, 57.1%; IC: 66.1%, 60.1%). These results might be influenced by the reference standard that was used for constructing a confusion matrix for performance measurement. However, as the bias from reference standard could affect the performance of all methods, we believe that this bias is unlikely to distort the results of performance. Another possibility is that these methods are not precise or advanced to detect safety signals appropriately. Recently, machine learning algorithms have been utilized in the pharmacovigilance field with relatively high performance, compared with the traditional data mining methods33. However, we did not utilize this method in this study. Therefore, further research involving a variety of new methods such as machine learning is needed.

Conclusions

In this post-marketing surveillance study, we identify 7 new safety signals for influenza vaccine including narcolepsy as a serious AE, and observed TIVs and egg-based influenza vaccine showed a wide range of new safety signals when compared to QIVs and cell-based influenza vaccine, respectively. Further studies are needed to confirm the potential relationship between influenza vaccine and new safety signals, and meanwhile, healthcare providers should evaluate whether vaccinees are susceptible to experiencing these safety signals by an in-depth review of comorbidities or AE-related history to prevent an avoidable risk. In addition, although TSS showed balanced performance in this study, there is still discrepancy in the identified safety signals among different methods, and some measurements were lower than the others. This result suggests that the complementary use of multiple data mining methods should be utilized to detect new safety signals conservatively.

Methods

Data source

We used the Korea Institute of Drug Safety and Risk Management-Korea Adverse Event Reporting System Database (KIDS-KAERS database)34 from 2005 to 2019. This database includes all spontaneous AE reports in South Korea and detailed information on demographics, vaccination, AEs, causality assessment, and clinical information. Information on the vaccine product was also provided. All drugs and AEs are coded according to the Anatomical Therapeutic Chemical Classification (ATC) system and the World Health Organization-Adverse Reaction Terminology (WHO-ART) dictionary35,36, respectively. WHO-ART is established as a tree structure, including system organ class (SOC), high-level terms (HLT), PTs, and included terms (IT) from top to bottom, and the signals were determined at the PT levels in this study.

The need for informed consent was waived, as this study was conducted using a spontaneous reporting database which does not include personal information. This study was approved by the Institutional Review Board of Chung-Ang University in South Korea (1041078-202008-ZZ-209-01). The Institutional Review Board of Chung-Ang University has waived informed consent for this study. All methods were carried out in accordance with the relevant guidelines and regulations.

Study vaccine

All vaccines available in Korea were included. The study vaccine was an influenza vaccine, and all other vaccines were used as comparators for signal detection. Nineteen influenza vaccine products have been approved and used in Korea, with 2 cell-based vaccines, 17 egg-based vaccines, 8 TIVs, and 11 QIVs. A full list of influenza vaccine products used in our study is given in Supplementary Material S1 online.

Selection of AE reports

From all AEs following vaccination collected in our database from 2005 to 2019, we excluded the following: (1) those that were reported as a concomitant vaccine, not a suspected vaccine; (2) those that were not final reports; (3) those that were invalid reports (missing or unspecified drugs or AE codes), and (4) those with report errors (logical error).

From a total of 42,211 reports in our database, we identified 38,221 reports after vaccination. Among these, 17,378 (45.5%) were reported for the influenza vaccine. We identified 17,280 reports, of which 7639 (44.2%) were reported for QIV, 9641 (55.8%) for TIV, 2744 (15.9%) for cell-based influenza vaccine, and 14,536 (84.1%) for egg-based influenza vaccine. A flowchart of the study is presented in Fig. 2.

Figure 2
figure 2

Flowchart of selection for spontaneous reports of adverse event. Abbreviations: KIDS-KAERS, Korea Institute of Drug Safety and Risk Management-Korea Adverse Event Reporting System Database; QIV, quadrivalent influenza vaccine; TIV, trivalent influenza vaccine. AE, adverse event.

Full size image

Selection of five AEs of special interest

Based on literature REVIEWS, we selected the following five possible serious and important AEs observed after influenza vaccination: GBS, febrile conversion, anaphylaxis, narcolepsy, and Bell’s palsy. Detailed information is provided in Supplementary Material S5 online.

Statistical analyses

Descriptive analyses

We calculated the frequency and proportion of baseline demographic characteristics (e.g., sex and age group) and reports (e.g., report type, source, reporter, seriousness, and causality assessment). Chi-squared tests were performed to compare the distribution of baseline characteristics between the influenza vaccine and all other vaccines or among individual subtypes of influenza vaccine (TIVs vs. QIVs and egg-based vs. cell-based vaccines).

Algorithms for signal detection

To detect the signals, three signal detection algorithms were used in this study, including PRR, IC, and TSS.

Disproportionality-based analysis

In this study, we used PRR and IC to generate signal scores for all AE pairs for the influenza vaccine versus all other vaccines, QIVs versus TIVs, and cell-based versus egg-based influenza vaccine. The Medicines and Healthcare Products Regulatory Agency (MHRA) in the United Kingdom and the WHO use PRR and IC, respectively37. The thresholds of signals for each algorithm were defined as follows: PRR ≥ 2, chi-squared test ≥ 4, number of reported AEs ≥ 3, IC with a lower limit of 95% confidence interval ≥ 0.

Tree-based scan statistic (TSS)

TSS is based on log-likelihood ratio (LLR) statistics and is suitable for analyzing a hierarchical structure variable. In this study, a diagnosis tree defined by the WHO-ART hierarchical structure was constructed, and the signals were determined at the PT level to compare them with the signals detected by the other algorithms. The unconditional Bernoulli model was selected to compare the number of AE reports between the two drugs. The expected and observed counts at each leaf were generated under the null hypothesis of an independent relationship between the vaccine and AE. AEs were defined as signals when the p-value of LLR was lower than 0.05, and p-values were generated using a Monte Carlo simulation. The analysis was performed using TreeScan Software38.

Performance evaluation of signal detection algorithms

The performance of disproportionality analysis (PRR, IC) and TSS was evaluated based on performance indicators such as sensitivity, specificity, PPV, NPV, accuracy, and AUC to identify which algorithm was more appropriate for detecting safety signals for influenza vaccines. To evaluate the performance indicators, a reference standard was established based on the MFDS-approved package inserts downloaded on September 13, 2020, from the NeDRUG website of the MFDS39.