https://orcid.org/0000-0002-2685-1546
Figures
Abstract
Vaccines are highly effective in minimizing serious cases of COVID-19 and pivotal to managing the COVID-19 pandemic. Despite widespread availability, vaccination rates fall short of levels required to bring about widespread immunity, with low rates attributed to vaccine hesitancy. It is therefore important to identify the beliefs and concerns associated with vaccine intentions and uptake. The present study aimed to develop and validate, using the AMEE Guide, the Vaccination Concerns in COVID-19 Scale (VaCCS), a comprehensive measure of beliefs and concerns with respect to COVID-19 vaccines. In the scale development phase, samples of Australian (N = 53) and USA (N = 48) residents completed an initial open-response survey to elicit beliefs and concerns about COVID-19 vaccines. A concurrent rapid literature review was conducted to identify content from existing scales on vaccination beliefs. An initial pool of items was developed informed by the survey responses and rapid review. The readability and face validity of the item pool was assessed by behavioral science experts (N = 5) and non-experts (N = 10). In the scale validation phase, samples of Australian (N = 522) and USA (N = 499) residents completed scaled versions of the final item pool and measures of socio-political, health beliefs and outcomes, and trait measures. Exploratory factor analysis yielded a scale comprising 35 items with 8 subscales, and subsequent confirmatory factor analyses indicated acceptable fit of the scale structure with the data in each sample and factorial invariance across samples. Concurrent and predictive validity tests indicated a theoretically and conceptually predictable pattern of relations between the VaCCS subscales with the socio-political, health beliefs and outcomes, and trait measures, and key subscales predicted intentions to receive the COVID-19 vaccine. The VaCCS provides a novel measure to assess beliefs and concerns toward COVID-19 vaccination that researchers and practitioners can use in its entirety or select specific sub-scales to use according to their needs.
Citation: Hamilton K, Hagger MS (2022) The Vaccination Concerns in COVID-19 Scale (VaCCS): Development and validation. PLoS ONE 17(3): e0264784. https://doi.org/10.1371/journal.pone.0264784
Editor: Camelia Delcea, Bucharest University of Economic Studies, ROMANIA
Received: November 30, 2021; Accepted: February 16, 2022; Published: March 14, 2022
Copyright: © 2022 Hamilton, Hagger. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data files and analysis output related to the research are available online: https://osf.io/k96bn/.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
COVID-19 infections have had substantive global social, economic, and health impacts, and contributed considerably to excess deaths worldwide [1–3]. In response, governments have imposed restrictions on movement and gatherings, and introduced other preventive measures (e.g., mandatory face mask wearing, imposing physical distancing rules, encouraging hand hygiene practices) to minimize infection transmission [4, 5]. The rapid development of highly efficacious COVID-19 vaccines and mass mobilization to distribute them has assisted in reducing infection rates, cases needing hospitalization, and bringing the pandemic under control [6, 7]. However, success of COVID-19 vaccination programs is highly dependent on high uptake rates among the population to build widescale immunity. Modeling data suggests that rates of vaccination of above 80% are required for exponential reductions in infection rates and, particularly, numbers of serious cases, hospitalizations, and deaths due to COVID-19 [8, 9]. Based on these data, some governments have begun to ease coronavirus restrictions and reduce requirements to engage in preventive measures contingent on high rates of vaccination uptake [10–12].
While there has been considerable uptake of the COVID-19 vaccine in countries where it has been offered to all adults such as Australia, Singapore, UK, and many European Union countries, the world population rate of fully vaccinated individuals is less than 40%. In addition, countries such as USA, Russia, and South Africa are witnessing a slowing of vaccine uptake [13], and many programs are running out of individuals willing to be vaccinated as a consequence [14, 15]. One contributing factor may be vaccine hesitancy, which represents a psychological state of indecision with respect to getting vaccinated [16]. Research and media reports have indicated considerable hesitancy with respect to COVID-19 vaccines in many populations and it has been identified as a salient contributor to reduced rates of vaccine uptake [17–20]. The issue is compounded by increased attention being given to ‘antivax’ groups and conspiracy theorists in the popular media who propagate misinformation and misperceptions about the efficacy, safety, and side effects of the COVID-19 vaccines [21–23]. Vaccine hesitancy in general, and specific concerns and false beliefs with respect to the COVID-19 vaccines, have considerable potential to stymie vaccination program effectiveness.
In addition to vaccine hesitancy in general, other beliefs (e.g., social and moral norms) [24, 25] and concerns (e.g., trust in government) [26] specific to COVID-19 may also contribute to intentions to receive, and actual uptake of, COVID-19 vaccines. Furthermore, vaccine hesitancy itself is likely to have multiple belief-based determinants [27]. This has compelled research examining these beliefs and the extent to which they account for unique variance in COVID-19 vaccination intentions. Knowledge of such beliefs, and their links to vaccination intentions, is valuable because it will inform the development of messaging included in interventions developed by organizations tasked with maximizing vaccine coverage of COVID-19 vaccine programs. This is especially important given the emergence of new strains of the SARS-CoV-2 virus that causes COVID-19, and the potential of the need for people to receive vaccine booster shots in the face of waning immunity, both of which will necessitate ongoing vaccination programs and accompanying advocacy [28–30].
One limitation of research examining relations between vaccine hesitancy, other vaccine-related beliefs and concerns, and vaccine intentions in the context of the COVID-19 pandemic is the relative dearth of measures with good psychometric properties and adequate construct and concurrent validity. While some measures of vaccine hesitancy and intentions, and other vaccine-related beliefs more broadly, have been developed in the context of COVID-19 or adapted from other measures [29], there is, to date, no validated measure that captures a broad range of beliefs and concerns that would be expected to be associated with COVID-19 vaccination intentions and uptake. Current measures tend to focus on one or a narrow range of factors such as confidence, trust, or vaccine literacy, and neglect the fact that vaccine hesitancy is likely to be a function of multiple beliefs and intrapersonal factors [27, 31].
With respect to vaccine hesitancy, conceptual work and theory suggests that it has multiple determinants such as socio-political beliefs, particularly generalized distrust in authority and medical science [32, 33]; generalized traits which include personality and individual difference constructs that reflect generalized tendencies that impact behavior and decision-making [30, 34–36]; and health-related beliefs and side effect concerns, particularly in terms of the rapid development of COVID-19 vaccines and their authorization for emergency use [37, 38]. Effects of these determinants on COVID-19 vaccine hesitancy and vaccine intentions, are likely to be heightened or exacerbated by contextual factors including the rapid development of the vaccine and the high profile of vaccine scepticism in media outlets particularly those with populist, right-leaning political perspectives. Specifically, the expedited process of vaccine development, a process more rapid than any vaccine in history [39–41], is likely to contribute to concerns among the general public with respect to safety. Concerns over a perceived lack of rigor or extensiveness of trials to test long-term effectiveness, or even a perception that developers ‘cut corners’, may contribute to vaccine concerns. This is likely to be exacerbated by misinformation regarding vaccine development and as well as high profile publicised cases of side effects or associations with nosocomial conditions [42, 43].
Alongside this, the rise of a populist political agenda that is generally sceptical of science and perceives vaccination as governmental interference and overreach, is also likely to heighten concerns relating to COVID-19 vaccine safety and rigor in development [41, 44]. In addition, high-profile vaccine-sceptic personalities and influencers in right-wing media outlets and social media platforms also model vaccine hesitancy and convey an aura of credibility to misinformation on the COVID-19 vaccine to a wide spectrum of the general public [22, 45]. These contextual factors are likely to magnify concerns among individuals pre-disposed to be sceptical and to whom right-wing populist beliefs have the most appeal, particularly those with traits such as right-wing authoritarianism and social dominance orientation.
Taken together, these factors specific to COVID-19 make it a special case with respect to beliefs and concerns about the vaccine, potentially contributing to the slowing rates of vaccination uptake observed in this context. Given the multifactorial nature of concerns surrounding the COVID-19 vaccines, there is a need for a comprehensive measure, which also demonstrates good psychometric properties with sets of constructs representing these specific issues (i.e., socio-political beliefs, personality and individual difference constructs, and health-related beliefs and outcomes), that can inform future intervention design and evaluation for COVID-19 vaccination uptake.
The present study
To date, there is no evidence-based measure that captures sets of beliefs and concerns that are expected to relate to COVID-19 vaccination intentions. This study addresses this evidence gap by developing and validating a comprehensive measure that captures these concerns, the Vaccination Concerns in COVID-19 Scale (VaCCS). The VaCCS will provide researchers and practitioners with a novel measure to assess beliefs and concerns toward COVID-19 vaccination and assist in identifying the determinants of vaccination intention and uptake, informing the development of messaging and interventions that may promote vaccination. Importantly, our approach to validation is intended to produce a scale that is flexible to use such that researchers and practitioners can use the scale in its entirety or select specific sub-scales to use according to their needs.
We followed a systematic, multi-step process to develop and validate the VaCCS [46], which was expected to yield a final scale comprising of sets of items representing multiple COVID-19 vaccine concerns and belief dimensions. We expect concurrent and predictive validity tests to indicate a conceptually predictable pattern of relations between the VaCCS subscales with the socio-political, trait measures, and health beliefs and outcomes. Specifically, we anticipate that the scale will capture beliefs and concerns including, but not limited to: concerns with respect to side effects and concerns over safety attributed to a perceived lack of testing and the rapid development of COVD-19 vaccines, conspiracy beliefs relating to this and other vaccines and about how COVID-19 emerged, generalized fear of vaccines, distrust in the government and particularly the pharmaceutical companies that have developed the vaccines, uncertainty over the outcomes of the COVID-19 pandemic, and general knowledge and understanding of how vaccines work. We therefore expect individuals with higher concerns in these areas to be more likely to hold political views and express general distrust in government, to have higher concerns about vaccines in general and concerns over medicines, lower risk perceptions with respect to COVID-19, and also greater trait-levels of neuroticism and lower conscientiousness.
Method
Participants and recruitment
We followed a systematic, multi-step design process using the AMEE Guide [46] to develop the VaCCS. The AMEE Guide presents a seven-step survey scale design process broadly consisting of a development phase (steps 1–6) and a validation phase (step 7) (see [46] for details). In the scale development phase, samples of Australian (N = 53; MAge = 44.45, SDAge = 19.57, 36% female) and USA (N = 48; MAge = 36.95, SDAge = 12.64, 58% female) residents were recruited via a research panel company to complete an online open response survey. To be eligible for inclusion, participants were required to be aged 18 years or older and have not received a COVID-19 vaccine. Participants received modest compensation for their participation based on expected completion time consistent with the panel company’s published rates. In the final phase of scale development, to assess readability and face validity of scale items, a convenience sample of 10 laypersons (MAge = 39.00, SDAge = 18.86, 50% female) and 5 experts (4 females; comprising clinical and health psychologists, behavior change scientists, and a vaccination researcher) were recruited.
For the subsequent scale validation phase, samples of Australian (N = 522, 60.7% female) and USA (N = 499, 70.7% female) residents were recruited via an online research panel company. To be eligible for inclusion, participants were required to be aged 18 years or older and not having received a COVID-19 vaccine. Participants were not stratified by specific demographic variables as our primary goal was to recruit participants who had not yet been vaccinated so we opted to be more inclusive in the recruitment phase. Data were collected between May 14 and May 28, 2021. At the time of data collection, only Australian residents aged 50 years and over, Aboriginal and Torres Strait Islander peoples aged 18 years and older, those with underlying medical conditions, and those whose employment placed them at high risk of contracting or spreading COVID-19 were eligible to get the COVID-19 vaccine [47]. By contrast, USA residents aged 18 years and older were eligible. We also collected data from an additional sample of vaccinated USA residents (N = 479, 56.8% female) between June 3 and June 7, 2021, which we used to replicate the validation procedures. Participants in this sample were required to have received both doses of an FDA-approved two-dose vaccine (i.e., Pfizer, Moderna) or the one-dose vaccine (i.e., Johnson and Johnson). Participants received modest compensation for their participation based on expected completion time consistent with the panel company’s published rates. Sample characteristics are presented in Table 1.
Design and procedure
Approval for study procedures was granted prior to data collection from the Griffith University Human Research Ethics Committee (#2021/108). A systematic, multi-step design process using the AMEE Guide, which broadly consists of a development phase (steps 1–6) and a validation phase (step 7), was adopted to develop an initial set of items for the VaCCS [46]. In the first few steps of the developmental phase (steps 1–4; [46]), participants from the initial Australia and the USA samples completed the open response survey comprising five structured questions [48] and reported their demographic information. This step was conducted to learn how the population of interest conceptualizes and describes a range of beliefs and concerns with respect to COVID-19 vaccines [46]. Responses were subjected to qualitative content analysis using NVivo 10 qualitative analysis software, which resulted in the extraction of 10 themes: behavior, protection, barriers, emotions, normality, safety, trust, development, efficacy, and social influences (see S1 File).
Next, and concurrent with the open response survey, a rapid review of the literature was conducted followed by a synthesis of the open response survey and literature review data to develop a comprehensive set of survey items for further evaluation. Specifically, these steps were conducted to ensure that the beliefs and concerns raised with respect to COVID-19 vaccines aligned with relevant prior research and theory, to identify existing survey scales or items that might be used or adapted, and to ensure survey items were written in accordance with current best practice and the language used was appropriate to the population of interest [46]. The search was conducted in February and March 2021 using OVID, Medline, Web of Science, and Embase, and also encompassed meta-analyses and systematic reviews investigating the measurement of vaccine beliefs (see S2 File for the search syntax). The search resulted in the identification of 448 records after removal of duplicates. After abstract and full text screening, 10 systematic reviews and meta-analyses were included, from which 63 measures of vaccine beliefs were extracted. We also included the Oxford Coronavirus Explanation, Attitudes, and Narratives survey (making it 64 measures), which was published subsequent to the included reviews [29]. A flowchart of the screening and inclusion process is presented in S3 File and a list of the extracted scales and their sources is presented in S4 File. A pool of 561 items was extracted from the 64 measures identified in the review process, with a total of 480 items after duplicate items were removed. Items were then subjected to a thematic analysis using NVivo 10 qualitative analysis software and subsequently sorted into one of 11 themes and entered into an excel spreadsheet. The themes were largely similar to the themes extracted from the open response survey: behavior, efficacy-protection, barriers, emotions, normality, safety, trust, development, importance, knowledge, and alternative medicine. The items were then examined by the lead investigators for relevance and appropriateness, resulting in the identification of 159 items considered eligible for inclusion in the VaCCS. These items were then further reviewed for similarity of content and expression and cross-referenced with the qualitative open response survey data. This process resulted in the retention of 63 items for further evaluation. Items were adjusted where necessary to make reference to COVID-19 vaccination (see Table 2).
In the final steps of the development phase (steps 5–6; [46]), to ensure the readability and face validity of the scale, a convenience sample of behavioral science experts (N = 5) and lay-persons (N = 10) and were recruited in May 2021. After providing demographic information and informed consent, participants were asked to complete the 63 potential items, and then rate them for readability (“The items are easy to understand for the average person”), relevance (“The measure is relevant to assessing people’s COVID-19 vaccine beliefs and concerns”), and suitability (“The measure is suitable for assessing individuals’ COVID-19 vaccine beliefs and concerns”) on a 5-point Likert type scale (1 = strongly disagree to 5 = strongly agree) developed by the authors. Participants were also given the opportunity to provide further comment in an open response format, which resulted in a small number of edits to item wording to improve clarity. Results indicated that both expert and lay-person groups found the scale easy to understand (M = 4.80, SD = 0.42; M = 4.25, SD = 0.50, respectively), relevant (M = 5.00, SD = 0.00; M = 4.75, SD = 0.50, respectively), and suitable (M = 4.20, SD = 1.14; M = 4.75, SD = 0.50, respectively).
In the scale validation phase (step 7; [46]), participants completed an anonymized online survey comprising study measures alongside measures of socio-demographic variables and characteristics (age, gender, employment status, education level, race, income). This last step in the VaCCS design process was to check for adequate item variance, reliability, and convergent and discriminant validity with respect to other measures [46]. Study measures comprised the candidate VaCCS items identified in the development phase alongside measures of constructs and variables used to test the concurrent and predictive validity of the VaCCS. Specifically, participants completed sets of measures tapping socio-political, personality and individual difference constructs, and health beliefs and outcomes. Where a ‘target’ behavior was mentioned in the study measures (e.g., intention), items in the unvaccinated Australian and USA samples made reference to getting the COVID-19 vaccine, while measures in the USA vaccinated sample referred to getting a booster vaccine in future to control for emerging variants of the virus.
Measures
In the scale validation phase, we assessed concurrent and predictive validity of the VaCCS by including measures of constructs and variables expected to exhibit a characteristic pattern of relations with the VaCCS subscales. The concurrent measures were in three broad categories: socio-political beliefs, personality and individual difference constructs, and health beliefs and outcomes. Unless otherwise stated, final composite scale scores for each participant were computed by averaging scores on each item.
VaCCS.
Participants completed scaled versions of the 63 candidate items for the VaCCS identified in the scale development phase. Participants provided their responses on 7-point scales (1 = completely disagree to 7 = completely agree). The full list of candidate items is presented in Table 2.
Socio-political beliefs.
Vaccine hesitancy. Two measures of vaccine hesitancy were adopted in the present study. The Oxford COVID-19 vaccine hesitancy scale [29] was adopted to measure vaccine hesitancy on seven items (e.g., “When the COVID-19 vaccine is available to me…”) with responses provided on seven point scales (e.g., 1 = I will refuse to get it to 7 = I will get it straight away). Lower scores were associated with higher vaccine hesitancy. A single item measure of vaccine hesitancy was also included (“Overall, how hesitant are you about getting the COVID-19 vaccine?”), with responses provided on a 7-point scale (1 = not at all to 7 = very much). Higher scores indicated a higher level of vaccine hesitancy.
Trust in government. Participants’ trust in governmental organizations’ handling of the COVID-19 vaccine program was measured using a scale developed by Grimmelikhuijsen and Knies [49]. The scale consists of nine items (e.g., “With regard to the COVID-19 vaccination program, do you agree that the Government in office in your country acts in the best interest of its citizens?”) with responses provided on 7-point scales (1 = totally disagree to 7 = totally agree). Higher scores on the scale represented greater trust.
Political orientation. Participants’ political orientation was measured using two items [50]. The first prompted participants to report where their political ideology sits on a scale ranging from “far left” to “far right”. Participants were provided with the following guide: “In political matters, people talk of “the left” and “the right”. Far left can be characterized by an emphasis on equality, progress, freedom, and reform with the economy driven by a cooperative collective agency. The far right can be characterized by an emphasis on order, tradition, nationalism, and authority with the economy driven by market forces. A centrist view would be considered a balance between social equality and social hierarchy with the economy balanced between regulation and free market.” Responses were provided on a sliding scale ranging from 0 to 100, with lower scores representing “far left” views and higher scores representing “far right” views. A second item prompted participants to respond to the following item: “How would you describe your political orientation?” Responses were provided on a sliding scale ranging from 0 to 100, with lower scores representing “strongly progressive” and higher scores representing “strongly conservative” views.
Free will beliefs. Free will beliefs were measured using the free will subscale of the Free Will Inventory [51]. The scale comprises five items (e.g., “People always have free will”) with responses provided on 5-point scales (1 = strongly disagree to 5 = strongly agree). Higher responses reflected greater free will beliefs.
Populist attitudes. Participants reported their populist views on six items (e.g., “Politicians in the Government in office in your country need to follow the will of the people”) [52] with responses provided on 7-point scales (1 = strongly disagree to 7 = strongly agree). Higher scores represented greater trust.
Conspiracy beliefs. Three sets of conspiracy beliefs were measured. Participants’ beliefs in conspiracy theories specific to the COVID-19 vaccine were measured using the COVID-19 conspiracy beliefs scale [53]. General beliefs in vaccine conspiracy theories were measured on the vaccine conspiracy beliefs scale [54]. The scales comprise six (e.g., “COVID-19 is intentionally presented as dangerous in order to mislead the public”) and seven items (e.g., “Vaccine safety data is often fabricated”), respectively with responses provided on 7-point scales (1 = strongly disagree to 7 = strongly agree). General conspiracy beliefs were assessed using the Conspiracy Mentality Questionnaire (CMQ) designed to assess individuals’ tendency to engage in general conspiracy ideation [55]. The scale comprises five items (e.g., “I think that many very important things happen in the world, which the public is never informed about”) with responses provided on 11-point scales (1 = certainly not to 11 = certainly). Higher scores on each scale represent greater beliefs in conspiracy theories.
Vaccine denial. Participants were prompted to report whether they had previously declined a prescribed vaccination (“Have you ever refused or elected to forego a doctor-recommended vaccine?”), with responses provided on a binary scale (1 = no and 2 = yes).
Personality and individual difference constructs.
Personality. Personality dimensions were measured using the Ten-Item Personality Inventory (TIPI) [56]. Participants were instructed to read a set of ten bi-polar adjectives representing the extroversion (e.g., “extraverted, enthusiastic” and “reserved, quiet”), agreeableness (“critical, quarrelsome” and “sympathetic, warm”), conscientiousness (“dependable, self-disciplined” and “disorganized, careless”), neuroticism (“anxious, easily upset” and “calm, emotionally stable”), and openness to experience (“open to new experiences, complex” and “conventional, uncreative”) traits from the five-factor model of personality. They were then prompted to rate the extent to which each pair described them using 7-point scales (1 = strongly disagree to 7 = strongly agree). Higher scale scores represent greater endorsement of the trait.
Intolerance of uncertainty. Participants’ general trait-level uncertainty was measured using the Intolerance of Uncertainty Scale—Short Form (IUS-12) [57]. The scale comprises 12 items (e.g., “Unforeseen events upset me greatly”) and responses were provided on 6-point scales (1 = not at all characteristic of me to 6 = entirely characteristic of me). Higher scales scores reflect greater uncertainty.
Health beliefs and outcomes.
Social cognition constructs. Multi-item measures of intentions to receive the COVID-19 vaccine (e.g., “I intend to get the COVID-19 vaccine when it is offered to me?”) and risk perceptions (e.g., “It would be risky for me to get the COVID-19 vaccine when it is offered to me?”) with respect to getting the COVID-19 vaccine, were developed according to published guidelines [58]. Each measure made reference to getting the COVID-19 vaccine as the target behavior for the Australia and USA unvaccinated samples, and getting the COVID-19 ‘booster’ vaccine in the vaccinated USA sample. Responses were provided on 7-point scales (1 = strongly disagree to 7 = strongly agree). Higher scores represented greater intentions and risk perceptions.
Beliefs about medicines. Participants beliefs and concerns about medicines were measured on the beliefs about medicines questionnaire [59]. The questionnaire comprises eight items (e.g., “Doctors use too many medicines”) and responses are provided on 5-point scales (1 = strongly disagree to 5 = strongly agree). Higher scores represent more concerns or negative beliefs about medicines.
Vaccine confidence. Confidence in vaccinations was assessed using the vaccination confidence subscale of the “5C” Psychological Antecedents of Vaccination Scale [60]. The scale consists of five items (e.g., “Vaccination is unnecessary because vaccine preventable diseases are not common anymore”) with responses provided on 7-point scales (1 = strongly disagree to 7 = strongly agree). Higher scores represent lower confidence in vaccines.
Vaccine knowledge. Participants’ vaccine knowledge was assessed using a single-item vaccination knowledge scale [61]. The scale comprises nine items (e.g., Vaccines are superfluous, as diseases can be treated (e.g., with antibiotics)?”), with participants providing responses on a binary scale (e.g., 1 = yes or 0 = no). Participants also had the option of checking a “I don’t know” option. The final scale score was computed by summing participants’ “yes” responses.
Close contact with vulnerable people. Participants reported the likelihood they would come into close contact with people they considered particularly vulnerable to COVID-19 in their home, in their social groups, and in their workplace on three items (e.g., “Do you have people living in your household who are considered vulnerable to COVID-19?”). Participants provided their responses on a binary scale (e.g., 1 = yes or 0 = no). An overall vulnerability index was produced by summing the “yes” responses.
Self-rating of health. Self-rated health was measured with a single item: “In general, how would you rate your health?” [62]. Responses were provided on a 5-point scale (1 = very poor to 5 = very good). Higher scores reflect better health.
Influenza vaccine history. Participants reported whether or not they had received the influenza vaccine in the current year (“Did you get the seasonal influenza vaccination in the prior year?”), with responses provided on a binary scale (1 = no and 2 = yes), and whether or not they regularly received the influenza vaccine (“When the seasonal influenza vaccine is available each year I…”), with responses provided on a 7-point scale (1 = never get it and 7 = always get it).
Socio-demographic variables.
Participants self-reported their age in years, sex (female, male, non-binary, not specified/prefer not to say), employment status (currently unemployed/full time caregiver, currently full-time employed, part-time employed, on leave without pay/furloughed, retired), annual household income stratified by eleven income levels based on Australia and USA national averages, highest level of formal education (completed junior/lower/primary school, completed senior/high/secondary school, post-school vocational qualification/diploma, undergraduate university degree, postgraduate university degree), and race (Black, Caucasian/White, Asian, Middle-eastern, other, prefer not to say). Dichotomous employment (unemployed vs. employed), highest education level (completed school education only vs. completed post-school education), and race (Caucasian/White vs. non-White) variables were computed for use in subsequent analyses.
Data analysis
Construct validity.
Exploratory factor analysis and item selection. In the first instance, we subjected data on the initial pool of 63 items for the VaCCS to an exploratory factor analysis with the goal of identifying subscales that represent concerns and social cognition beliefs with respect to COVID-19 vaccine. The matrix of correlations among the item pool in the Australia and USA samples was subjected to a principal components analysis. In the analysis, factors with eigenvalues greater than 1.00 were extracted and subjected to oblimin rotation with Kaiser normalization. The standardized structure matrix of item factor loadings for each extracted factor was used as a basis for item selection for the final scale. Items were selected based on the size of their loading on their respective factor and their representativeness in terms of capturing the essence of the factor without duplication. Extracted factors were labeled based on the collective content of the items comprising each extracted factor. The exploratory factor analysis was conducted using the psych [63] and GPArotation [64] packages in R.
Confirmatory factor analysis and factorial invariance. The resulting scale was subjected to confirmatory factor analysis in the Australia and USA samples separately, followed by a multi-group analysis to test for invariance of the proposed factor structure across the samples. In each of the single-sample confirmatory factor analytic models, items pertaining to each factor selected based on the exploratory factor analysis and item review in the previous step were set as indicators of its respective factor with each factor set to correlate with each other. Fit of the models with the data in each sample was assessed using multiple goodness-of-fit criteria: the comparative fit index (CFI) and the Tucker-Lewis index (TLI), the root mean square error of approximation (RMSEA) and its 90% confidence intervals, and the standardized root mean square residual (SRMSR). For a well-fitting model, the CFI and TLI should approach or exceed .900 for a well-fitting model, although values approaching or exceeding .950 are preferable, the RMSEA should be close to .008 with narrow confidence intervals, and the SRMSR should be close to .006 [65, 66]. We also examined solution estimates for the model in each sample including the standardized factor loadings with their 95% confidence intervals and the R-squared value for each item on its hypothesized factor, and the average variance extracted by the set of items for each factor. Factor loadings and R-squared values should be close to .700 and .500, respectively, and average variance extracted should exceed .500 for good structural integrity of the proposed latent factors [67, 68].
Pending adequate fit of the proposed confirmatory factor analysis model in each sample, we tested the invariance of the proposed factor structure across the Australia and USA samples. We followed a recommended invariance routine beginning with a configural model, in which the tenability of the proposed model across the two samples is tested, followed by progressively restrictive models in which the factor loadings (“weak invariance”) and item intercepts (“strong invariance”) are constrained to be equal across the samples. As the goodness-of-fit chi-square is recognized to be over-sensitive as a means to evaluate changes in model fit in the invariance routine, we followed Cheung and Rensvold’s [69] recommendations that invariance is supported if the incremental fit indexes (CFI, TLI) of the constrained models differ from the configural model by less than .01.
Assuming adequacy of the model in the initial samples, we subsequently aimed to replicate these findings by fitting the confirmatory factor analytic model to data on the VaCCS items from a third sample of vaccinated participants from the USA. Fit of the model was evaluated using the same criteria for overall model fit and solution estimates in the unvaccinated samples. All confirmatory factor analytic models were estimated using the lavaan [70] package in R using a maximum likelihood estimation method and full-information maximum likelihood (FIML) imputation of missing data.
Scale scores and reliability coefficients.
Item scores for the VaCCS subscales produced in the factor analyses were averaged to provide scale scores. We similarly computed scale scores for the sets of socio-political, health beliefs and outcomes, and personality and individual difference constructs. The only exceptions were the vulnerable people and COVID-19 knowledge measures; both measures adopted binary scales, so overall scale scores were computed using the sum rather than the average of the item scores. Omega (ω) reliability coefficients were computed to evaluate the internal consistency of each scale with estimates equal to or in excess of .800 considered indicative of good internal consistency [71]. Where scales comprised fewer than three items, we computed the inter-item correlation in lieu of a reliability coefficient.
Concurrent and predictive validity.
We examined correlations between the VaCCS subscales and the sets of socio-political, health beliefs and outcomes, and personality and individual difference constructs measured concurrently with the VaCCS items. Specifically, we correlated each composite scale from the VaCCS with composite scales from the scales from each set and in each sample separately. To minimize the false positivity rate (Type I error rate) due to multiple tests, we applied a Holm-Bonferroni correction method to adjust statistical significance levels. We used linear multiple regression to examine the unique contribution of each VaCCS subscale in predicting intentions to get the COVID-19 vaccine in the Australia and USA unvaccinated samples, and the COVID-19 ‘booster’ vaccine in the USA vaccinated sample. Specifically, we simultaneously regressed measures of intentions on the VaCCS subscales, risk perceptions, vaccine hesitancy, and socio-demographic variables (age, gender, education, employment status, and race) in each sample. In order to avoid including multiple contrast codes for the categorical socio-demographic variables with multiple categories in the regression model, we dichotomized variables with more than one category. Specifically, for the education variable participants were divided into those with primary education as the highest level of education (0) and those with secondary education or higher (1). Employment was divided into employed (1) and unemployed (0). Ethnicity was divided into non-white (1) and white (2) participants. We did not include income because of the large amount of missing data. The correlation and regression analyses were conducted using the psych and lavaan packages, respectively, in R.
Results
Participants
Sample characteristics for each sample are presented in Table 1. Participants in the Australia and unvaccinated USA samples were predominantly white (78.7% and 86.8%, respectively), educated (65.3% and 59%, respectively, reported completing a post-school qualification), predominantly employed in at least part-time work (57.9% and 40.9%, respectively), and reported middle-to-high income (28.7% and 27.1%, respectively). Participants in the vaccinated USA sample were majority white (87.9%), employed in at least part-time work (56.0%), educated (73.7% reported completing a post-school qualification), and reported high-to-middle income (66.0%). Very few participants reported having a previous diagnosis for COVID-19 in the Australia sample (1.2%), and self-reported previous COVID-19 positivity rates were also low in the vaccinated (9.4%) and unvaccinated (7.9%) USA samples. Only one case (0.2%) in each of the Australia and unvaccinated USA samples reported a current COVID-19 diagnosis, compared to 3.8% in the vaccinated USA sample. The majority of the vaccinated USA sample reported receiving a two-dose vaccine regimen (Pfizer, 48.6%; Moderna, 36.3%) with the remainder receiving the Jansen/Johnson & Johnson vaccine (12.9%), or did not know the type of vaccine they received (1.1%) or preferred not to say (1.1%).
Examining differences in sample demographic characteristics across the three samples revealed no significant differences in age, F(2,1497) = 1.313, p = .269. However, we found statistically significant differences in the distribution of participants across categories of gender (χ2 (2) = 21.285, p < .001), employment status (χ2 (8) = 98.585, p < .001), race (χ2 (10) = 105.950, p < .001), income (χ2 (2) = 10.353, p = .006), education level (χ2 (8) = 105.11, p < .001), previous COVID-19 diagnosis (χ2 (2) = 35.221, p < .001), and current COVID-19 diagnosis (χ2 (8) = 31.415, p < .001). Follow-up tests revealed an equal distribution of gender across the USA unvaccinated and vaccinated samples, but the Australian sample had a larger proportion of men than the two USA samples. In addition, proportions of unemployed individuals, retired individuals, and individuals in part-time and full-time work differed across all samples. The demographic profile of the Australia and USA vaccinated sample did not differ, but these samples had significantly larger numbers of black participants and fewer Asian participants than the USA sample. There were no differences in the distribution of individuals in high and low income categories across the Australia and USA unvaccinated samples, but there were significantly more participants in the high income category in the USA vaccinated sample relative to these samples. While there were only marginal differences in proportions of participants across education categories between the Australia and USA unvaccinated samples, the USA vaccinated sample had greater proportions of participants in the higher education categories. Finally, the Australia and the USA vaccinated sample reported a greater number of past COVID-19 diagnoses relative to the USA unvaccinated sample, while the USA vaccinated sample reported a higher proportion of current COVID-19 diagnoses than the Australia and USA unvaccinated sample. These data indicate systematic differences in the demographic characteristics of the three samples, and highlight the imperative of including demographic characteristics as covariates in subsequent cross-national comparisons.
Exploratory factor analysis and item selection
Nine factors explaining 49.1% of the variance in correlations among the initial pool of 63 VaCCS items were extracted in the exploratory factor analysis. The rotated structure matrix with standardized factor loadings of the VaCCS items is presented in Table 2. The content of each factor is described next in the order of variance explained rather than position extracted. The first extracted factor explained 11.2% of the variance and comprised items relating to the efficacy of the COVID-19 vaccine to protect the self and others from infection (e.g., “If I get the COVID-19 vaccine it will help to protect my family and friends against the coronavirus”) and to assist in returning society to normal (e.g., “Getting the COVID-19 vaccine will help to get things back to normal”). This factor was labelled “Beliefs in efficacy and prevention”. The eighth extracted factor explained 8.8% of the variance and consisted of items reflecting trust in the government and other organizations with respect to the COVID-19 vaccine (e.g., “I trust the Government to give me reliable information on the benefits and risks of the COVID-19 vaccine”). This factor was labelled “Trust in authorities”. The second extracted factor explained 7.5% of the variance and comprised items reflecting concerns about the safety and side effects of the vaccine (e.g., “I am concerned about the side effects of the COVID-19 vaccine”). This factor was labelled “Worry about safety and side effects”. The third extracted factor accounted for 4.5% of the variance and consisted of items reflecting concerns that the vaccine may cause the virus in some people (e.g., “The COVID-19 vaccine can cause the coronavirus in some people”). The factor was labelled “Beliefs that the vaccine causes COVID-19”. The fifth extracted factor explained 4.4% of the variance and comprised items reflecting scepticism and lack of trust in the vaccine (e.g., “The COVID-19 vaccine safety data is often made up”). This factor was labelled “Scepticism and mistrust in the vaccine”. The seventh extracted factor explained 3.8% of the variance and consisted of items reflecting fear and anxiety over getting the vaccine (e.g., “I am afraid of getting the COVID-19 vaccine”). This factor was labelled “Fear of the vaccine”. The ninth extracted factor accounted for 3.0% of the variance and comprised items reflecting uncertainty about the vaccine development and hesitancy in getting vaccinated until it had been further evaluated (e.g., “The COVID-19 vaccine is too new so I should wait before deciding to get it”). This factor was labelled “Uncertainty and hesitation in getting vaccinated”. The fourth extracted factor also explained 3.0% of the variance and comprised items relating to adequacy and understanding of information with respect to the COVID-19 vaccine (“Information about the COVID-19 vaccine is easy to understand”). This factor was labelled “Vaccine literacy”). The sixth extracted factor comprised items relating to rights, uptake of the vaccine, and refusal to take the vaccine. Items pertaining to this factor exhibited much lower principal loadings and cross loadings of similar magnitude, so it was dropped in the final reckoning for the VaCCS.
A final item selection procedure for each included subscale was subsequently applied by each lead researcher independently based on three criteria: representativeness of the items, statistical evidence from the exploratory factor analysis (i.e., factor loadings), and the need for parsimony. The procedure resulted in close agreement across the researchers in item selection (average 87.79% agreement, AC1/AC2 agreement statistic = 0.781) [72]. Discrepancies were discussed and the selection criteria and procedure updated accordingly. Subsequent re-application of the procedure to the candidate items by each researcher resulted in perfect agreement for each subscale (100%). The procedure yielded a final scale comprising of 35 items and eight subscales.
Confirmatory factor analyses and invariance analyses
Next, we fit the 8-factor, 35-item VaCCS model to data from the Australia and USA samples separately using confirmatory factor analysis. Pending adequate fit of the model with the data in each sample, we subsequently tested the invariance of the model factor loadings and intercepts across the two samples using multi-group confirmatory factor analysis. We also replicated the confirmatory factor analytic model in the additional USA sample of vaccinated participants.
Goodness-of-fit statistics for the initial confirmatory factor analysis model in each sample are presented in Table 3. Overall, the model exhibited fit statistics that approached acceptable cut-off values for the multiple adopted indices of good fit. However, modification indices (MI) indicated that redundancy across some items from within the same subscale was primarily responsible for the misspecification, and identified pairs of within-factor indicator error covariances that would resolve the misspecification if freely estimated. The error variances associated with the largest misspecification in the model were identical across the two samples. Examination of the content of these items indicated that they tapped similar conceptual content in the measured factor. We therefore set error covariances free between items with the five largest expected parameter change (EPC) statistics identified by the MIs (EPC > 50.000) in each sample. The practice of including error covariances is only advocated when justified by the content of the items involved in the redundancy [73]. In the case of the inclusion of the five error covariances for these models, there was justification given the closeness in the item content. We expected the inclusion of these error covariances would have little consequence on model integrity and we did not want reject a tenable model on trivial grounds. The final model including these error covariances exhibited acceptable goodness-of-fit statistics in both samples (Table 3). Solution estimates of the final model in each sample are presented in Table 4. Factor loadings and R-squared values for each item on its respective factor approached or exceeded the .70 and .50 expected values, respectively, with narrow confidence intervals for the factor loadings.
Given the acceptable fit of the single-sample models, we conducted the invariance analysis of the proposed model according to our a priori invariance routine. The configural model exhibited adequate goodness-of-fit with the data, indicating that the proposed model was tenable across the two samples (Table 3). Estimating models that constrained the factor loadings (weak invariance) and intercepts (strong invariance) revealed minimal differences in model fit across each model in the invariance routine according to Cheung and Rensvold’s criterion for change in the CFI and TLI. We subsequently concluded that there was minimal variance in parameters of the proposed VaCCS model across the two samples.
Finally, we replicated the proposed confirmatory factor analytic model of the VaCCS in the USA vaccinated sample. Consistent with the previous analyses, the model exhibited acceptable goodness-of-fit statistics (Table 3) and solution estimates (Table 4) in this additional sample.
Descriptive statistics and reliability estimates
Descriptive statistics and reliability estimates for the VaCCS subscales and study measures are presented in Table 5, and factor and manifest variable correlations among the subscales presented in Table 6. Reliability estimates indicated acceptable internal consistency for all VaCCS subscales. Reliability estimates for the variables included to test concurrent and criterion validity of the VaCCS were also acceptable, with few exceptions. The most notable exceptions were the two-item scales tapping the five-factor personality constructs, which all exhibited statistically significant inter-item correlations, but each was relatively modest in size, a trend noted elsewhere [56].
Concurrent validity
Correlations between VaCCS subscales and sets of socio-political variables, personality and individual difference constructs, and health beliefs and outcomes are presented in Table 7, Table 8 and Table 9, respectively. Associations between VaCCS subscales and concurrent validity measures formed characteristic patterns of associations consistent with expectations.
Socio-political beliefs.
Focusing on correlations with socio-political beliefs, the VaCCS efficacy, trust, and literacy subscales were statistically significantly and positively correlated with the Oxford vaccine hesitancy scale and government trust in all three samples with medium-to-large effect sizes. These scales were also negatively correlated with COVID-19, vaccine, and general conspiracy beliefs, and vaccine hesitancy with medium-to-large effect sizes, although effects for the literacy subscale were smaller. In keeping with this pattern, the VaCCS worry, cause, scepticism, fear, and uncertainty subscales were negatively correlated with the Oxford scale and government trust, and positively related to all conspiracy beliefs variables and vaccine hesitancy, again with medium-to-large effect sizes and in all samples. The efficacy, trust, and literacy subscales were also positively correlated with free will, but only in the Australia and vaccinated USA samples. In addition, the efficacy and trust subscales were negatively correlated with vaccine denial in all samples, and negatively correlated with political trust except in the vaccinated USA sample, all with small effect sizes. The worry, cause, scepticism, fear, and uncertainty subscales were positively related to political trust and vaccine hesitancy in all samples with small effect sizes.
Personality and individual difference constructs.
No clear pattern of statistically significant correlations emerged between the VaCCS subscales and the personality and individual difference constructs across the samples. Across samples, only the fear subscale was positively associated with intolerance of uncertainty in all three samples with small effect sizes. The only other patterns of correlations were confined to specific samples. The VaCCS worry, cause, scepticism, fear, and uncertainty subscales were negatively correlated with conscientiousness and agreeableness, and positively related to neuroticism, in the USA (vaccinated) sample with small-to-medium effect sizes.
Health beliefs and outcomes.
Turning to correlations with health beliefs and outcomes, the VaCCS efficacy, trust, and literacy subscales were statistically significantly and positively correlated with intentions to get the vaccine, and negatively related to risk perceptions, beliefs about medicines, and vaccine confidence with medium-to-large sized effects in all samples. Analogously, the VaCCS worry, cause, scepticism, fear, and uncertainty subscales were negatively correlated with intentions, and positively correlated with risk perceptions, beliefs about medicines, and vaccine confidence with medium-to-large sized effects and in all samples. The VaCCS efficacy and trust subscales were positively correlated, and the VaCCS worry, cause, scepticism, fear, and uncertainty subscales negatively correlated, with vaccine knowledge except in the vaccinated USA sample. A similar pattern of correlations was exhibited with receiving the influenza vaccine and influenza vaccine regularity, but only in the Australia sample. The VaCCS efficacy and trust subscales were positively correlated with receiving an influenza vaccine and influenza vaccine regularity in all three samples.
Predictive validity
We also examined the unique effects of the VaCCS subscales on intentions to get the COVID-19 vaccine, or, in the case of the USA vaccinated sample, intentions to get a COVID-19 ‘booster’ vaccine, alongside socio-demographic covariates, risk perceptions, and vaccine hesitancy. Results of the linear regression of intentions on the VaCCS subscales, risk perceptions, vaccine hesitancy, and socio-demographic variables in each sample are presented in Table 10. Across the samples, the VaCCS efficacy and trust subscales were statistically significant positive predictors of intentions with small-to-medium effect sizes, alongside a significant negative effect of vaccine hesitancy with a small effect size. The VaCCS uncertainty subscale was also a significant predictor across the samples, with small effect sizes, although the effect was negative in the unvaccinated samples, but positive in the vaccinated sample. This positive, statistically significant effect was unexpected given the large, statistically significant and negative correlation between intention and the uncertainty subscale (r = -.514, p < .001; Table 9). This is likely to be a suppressor effect caused by the substantive correlation between the uncertainty subscale and the other VaCCS subscales in this sample, particularly the scepticism and uncertainty subscales (Table 6). Further, risk perception was also a significant negative predictor of intentions in the unvaccinated samples, with a small effect size, but not in the vaccinated sample. In addition, the VaCCS fear and literacy subscales were significant predictors of intentions in the latter sample. Overall, these constructs accounted for between 62.1% and 76.2% of the variance in vaccine intentions across the samples.
Discussion
The present study reports on the development and initial validation of the Vaccine Concerns in COVID-19 Scale (VaCCS), a psychometric instrument designed to measure individuals’ beliefs and concerns with respect to COVID-19 vaccines. The measure was developed in a multi-stage procedure from first principles comprising a developmental stage, in which an initial item pool was provided via an evidence synthesis of previous vaccine scales and an open-ended survey, and a validation stage in which a final pool of items was selected via rigorous construct and concurrent validity assessment in samples of Australian and the USA residents. The procedure produced a final 35-item scale with eight subscales. The scale exhibited strong psychometric integrity with a coherent factor structure that was invariant across samples. Subscale scores exhibited a predictable pattern of correlations with salient measures of socio-political beliefs, health beliefs and outcomes, and some selected personality and individual difference constructs. In addition, the VaCCS efficacy and trust subscales were consistent positive predictors of intentions to get the COVID-19 vaccine across samples, alongside vaccine hesitancy and risk perceptions. In addition, the uncertainty VaCCS subscale was negatively associated with COVID-19 vaccine intentions in the unvaccinated samples, but positively associated with intentions in the vaccinated sample.
A comprehensive measure of COVID-19 vaccine beliefs and concerns
Our development process yielded an instrument that captures the wide-ranging sets of beliefs likely to be of relevance to individuals’ decisions to get the COVID-19 vaccine. Concerns about the vaccine featured prominently in the scale and were captured by a number of its subscales. For example, the worry and uncertainty subscales reflect safety concerns regarding the rapid development of the vaccines and a lack of long-term studies of their effects, and concerns that the vaccine may lead to adverse side effects. These concerns have also been identified in research examining safety concerns in the context of COVID-19 [74, 75]. In addition, the fear subscale captures anxiety and fear over getting a vaccine, which may also be linked with safety concerns, or stem from a generalized fear of medicines or medical procedures, or with the process of vaccine administration such as a fear of injections or syringes [76]. Further concerns are captured by the scepticism and trust subscales. The trust subscale taps into individuals’ general mistrust of the government, scientists, and pharmaceutical corporations responsible for developing and administering the vaccines, and likely captures generalized “anti-vax” beliefs. Similarly, vaccines have been equated as an instrument of governmental control, particularly with recent mandates that require vaccinations among essential workers [77–79], beliefs captured by the scepticism subscale. There have also been concerns over the efficacy of the vaccines, particularly in light of highly-publicized, albeit rare, cases where vaccinated individuals have been hospitalized with severe COVID-19, and reports of high infection rates among the vaccinated as new, highly contagious variants of the virus spread. The efficacy subscale, in particular, captures these beliefs, particularly concerns that vaccines may not be sufficiently effective or may do more harm to health than good. There is also evidence that individuals may harbor beliefs focused on the vaccine itself and that it may infect individuals with COVID-19 [80, 81]–the cause subscale taps into these beliefs. Such beliefs likely stem from a lack of understanding of the vaccine and how it works, and a perceived lack of clear information on the vaccine and its effects, both of which may be related to low levels of health literacy. Such beliefs identified in the literacy subscale. Taken together, the VaCCS represents a comprehensive measure that captures a range of beliefs individuals may hold with respect to getting a COVID-19 vaccine, and are likely to be implicated in their future decisions to get the vaccine.
Concurrent validity
Given the breadth of beliefs captured by the VaCCS, examination of relations between the subscales and measures of conceptually-related beliefs and constructs was important to provide evidence for their concurrent validity. To this end, we administered measures of beliefs and perceptions expected to reflect the potential origins or consequences of beliefs and concerns about vaccines alongside the new measure.
Socio-political beliefs.
Taking into consideration the social and political context in which the COVID-19 pandemic has occurred and developed, a prominent set of beliefs with which subscales of the VaCCS subscales were expected to be related was socio-political beliefs [37, 38]. Confirming our expectations, we observed statistically-significant correlations between key VaCCS subscales and beliefs that represent lack of trust in the government, beliefs in governmental and broader conspiracy theories, and vaccine hesitancy. Individuals who scored higher on the efficacy and trust subscales were more likely to report high general trust in government and other organizations, and less likely to express vaccine hesitancy or denial, or to endorse conspiracy beliefs about the vaccine. Individuals endorsing the efficacy and trust subscales were also more likely to identify as liberal or politically left-wing. In contrast, those who scored lower on the efficacy and trust subscales, and reported higher levels on the scepticism, uncertainty, worry, and fear subscales, were more likely to report low trust in government and vaccine producers, be vaccine hesitant or be in denial about the pandemic, endorse conspiracy theories, and identifying as more conservative or right-wing political ideology. These patterns of associations were highly consistent across the three samples, suggesting that they were not a symptom of the localized political climate. Taken together, this pattern of associations corroborates research linking concerns, mistrust, and scepticism with reduced likelihood of endorsing the vaccine among those expressing conservative views [37, 38, 79]. The findings are also consistent with high-profile reporters’, politicians’ and other ‘celebrity’ figures’ expressions of mistrust of the vaccines, and promulgation of conspiracy theories without credible evidence, via the populist press and social media [82, 83]. These effects provide evidence supporting the concurrent validity of the VaCCS subscales with respect to the political landscape and dominant views that pervade in during the roll-out of the vaccines.
Interestingly, we observed few relations between the VaCCS subscales and free will beliefs. While we expected that individuals’ concerns that the vaccine represents government interference and overreach, the VaCCS trust and uncertainty subscales were generally not correlated with free will beliefs. One possibility is that free will is associated with attitudes towards vaccine mandates, such as proof of vaccine requirements for travel, work, and entry into bars and restaurants, rather than to the vaccine itself, although these beliefs would be expected to correlate. Alternatively, the lack of associations might be because free will beliefs are highly generalized and focus on global beliefs about capacity and agency, rather than beliefs in specific governmental and organization impingements that might affect specific decisions such as getting the COVID-19 vaccine. The same may apply to relations between the specific beliefs captured by the VaCCS subscales and more generalized views captured in trait-level constructs, which we discuss next.
Personality and individual difference constructs.
We found relatively few statistically significant correlations between the VaCCS subscales and the personality and individual difference constructs. Subscales representing concerns about the vaccine, particularly the cause, scepticism, fear, and uncertainty tended to be negatively related to the agreeableness and conscientiousness personality constructs, and negatively related to neuroticism, in the vaccinated sample but not in the other samples. This suggests that individuals with tendencies toward harmonious social relations and work ethic and organization were less likely to harbor concerns about the vaccine, and were more certain of its credibility. However, the most consistent finding was the positive association between the fear subscale and intolerance for uncertainty across samples. Uncertainty over the effects of the vaccine is mirrored by the level of fear individuals’ express over getting the vaccine. This finding is consistent with theory [84] and research [85, 86] demonstrating that uncertainty is likely to generate feelings of fear in health contexts, and perhaps fear of COVID itself although this needs empirical corroboration, and provides further evidence for the concurrent validity of this subscale. The absence of clear trends in correlations between VaCCS subscales and traits may be a function of the generalized focus of the traits compared to the VaCCS subscales which refer to a specific behavioral context. Traits are expected to be associated with beliefs and behavioral tendencies across a wide range of behaviors and contexts, albeit with relatively small effect sizes, consistent with the trends observed here. Overall, the largely trivial correlations observed with the personality and individual difference constructs do not add substantively to evidence in support of the concurrent validity of the VaCCS.
Health beliefs and outcomes.
The VaCCS subscales were also expected to be consistently related to health beliefs and outcomes, particularly generalized beliefs about medicines, knowledge about vaccines, beliefs about COVID-19 risks, the vulnerability of others around them to infection, and individuals’ past vaccination behavior including the tendency to get an influenza vaccination. Specifically, the efficacy and trust VaCCS subscales were positively related to vaccine knowledge and past history of getting an influenza vaccine, and negatively related to vaccine risk perceptions, concerns about medicines, and lack of confidence in vaccines. A similar characteristic pattern was observed for the VaCCS subscales relating to worry and fear about the vaccine, and scepticism and uncertainty of the government and vaccine producers, with those scoring highly on these subscales more likely to report lower knowledge about the vaccine, view the vaccine as risky, express concerns about medicines, and be less likely to have received an influenza vaccine or regularly get a flu shot. This pattern of effects may be attributable to low generalized health literacy and knowledge. The literacy subscale was consistently and positively related to knowledge about the vaccine, and negatively related to VaCCS subscales relating to concerns and mistrust. This was also corroborated by the positive association between the literacy subscale and vaccine hesitancy, lack of trust in the government, and endorsement of conspiracy beliefs, suggesting that individuals who have generally low ability to take on and interpret information about COVID-19 vaccines may be more vulnerable to misinformation. These findings are consistent with research indicating that individuals’ beliefs about the health concerns and risks with respect to the COVID-19 vaccine are consistent with their generalized health concerns and vaccine behavior [87–89], and, most importantly their motivation to get vaccinated [90, 91]. These beliefs may also be associated with perceptions of risk and fear of COVID-19. However, these relationships were not tested in the current analysis, and we look to future studies to provide empirical verification. Taken together, this pattern of effects provides further support for the VaCCS as an instrument that yields a profile of beliefs with respect to the COVID-19 vaccine that are consistently with related to the health-related beliefs that likely impact future behavior and to motives to get vaccinated.
Prediction of COVID-19 vaccine intention
The VaCCS subscales were also expected to be implicated in individuals’ motivation to get the COVID-19 vaccine in future, or, in the case of the vaccinated sample, the booster vaccine. This was corroborated by positive associations between the efficacy and trust subscales with vaccine intentions, and negative associations with subscales reflecting concerns, scepticism, and mistrust. Also important, however, was our regression analysis, which provided evidence on the VaCCS subscales that contributed most to explaining variance in intentions. Our analysis also examined effects of subscales on intentions concurrent with other beliefs likely to be implicated in vaccine decision-making, namely, risk perceptions and vaccine hesitancy, and controlled for effect of salient demographic covariates. Focusing on the prediction of vaccine intentions in the unvaccinated samples, the efficacy and trust subscales, along with risk perceptions, were the most prominent predictors, while the uncertainty and worry subscales, along with vaccine hesitancy, had much smaller effects. In contrast, effects of the cause, scepticism, fear, and literacy subscales were trivial and not statistically significant. These findings corroborate previous research demonstrating the prominence of risk perceptions, and beliefs in the efficacy of the vaccine and trust in government and vaccine producers, as driving individuals’ intentions to get vaccinated [32, 33, 92, 93], and provides important predictive validity for the VaCCS subscales. The pattern of prediction was similar for the prediction of vaccine booster intentions among the vaccinated sample, with efficacy and trust, along with vaccine hesitancy prominent statistically significant predictors, with smaller the effects of the fear, uncertainty, and literacy subscale. This slightly different pattern of effects for booster intentions indicates the continued importance of endorsing vaccine efficacy and trust, and low vaccine hesitancy, in determining vaccine-related decision making going forward. Findings may also have resonance when it comes to informing messaging and interventions aimed at promoting vaccination uptake. Interventionists may consider developing communications that emphasize the efficacy and trustworthiness of the vaccine, particularly its effectiveness in preventing serious COVID-19 infections that require hospitalization and in reducing the spread of infections, and highlight the robustness of data supporting its safety and the credibility of the pharmaceutical companies that have developed the vaccine.
Strengths, limitations, and avenues for future research
We aimed to develop a comprehensive psychometric instrument to capture individuals’ beliefs and concerns with respect to the COVID-19 vaccine. The scale was developed and validated from first principles in a multi-stage approach with numerous design and methodological strengths, including: (a) identification of candidate beliefs from a review of prior vaccine research and the views and perspectives of individuals eligible to receive the vaccine, the target audience; (b) formation of candidate items through content analysis of candidate beliefs from both sources; (c) rigorous factor analytic validation in multiple samples of residents from two countries during a time when COVID-19 vaccines were being rolled out; and (d) concurrent and predictive validity tests through associations with key socio-political beliefs, health beliefs and outcomes, and personality and individual difference constructs and the prediction of vaccine intentions. While current findings provide good evidence in support of the structural integrity, internal consistency, and validity of the VaCCS, findings should be interpreted in light of a number of limitations. These include the exclusive reliance on correlational data and self-report measures, the lack of a behavioral measure, the non-representativeness of the samples used in the validation studies, and the possibility that the scale does not encompass all beliefs and concerns with respect to getting a COVID-19 vaccine. Next, we outline each of these limitations, and provide some suggestions for future research.
Data used in the validation process of the VaCCS were correlational, and the measures adopted were exclusively self-report. Correlational data are not informative on causal effects and are not able to identify possible ‘third variable’ explanations for correlations. For example, we cannot, on the basis of the current data, conclude that the sets of beliefs captured by the VaCCS are causally related to outcomes such as vaccine intentions or hesitancy. In addition, the current data did not enable us to ascertain the sensitivity and specificity of the VaCCS subscales in measuring belief change, such as belief change that may be brought about by the advent of new information or the introduction of persuasive communications about the COVID-19 vaccines. A further limitation is the reliance on self-report measures, particularly for the measures used in concurrent validity tests, which may introduce error variance to the correlations due to recall and reporting biases. A related limitation is the lack of a behavioral measure of vaccine uptake, which would provide further evidence in support for the predictive validity of the scale. Resolution lies in experimental or intervention research that tests the effects of strategies to alter or manipulate vaccine beliefs and concerns on change in the VaCCS subscales. It would also be important to examine change in VaCCS subscales prior to and after vaccination, or the association between the subscales and subsequent vaccination status over time. Such research will test the value of the VaCCS in capturing change in beliefs and concerns, and its association with vaccine behavior.
A further limitation is that the samples used in the validation phase of the current research were not representative of the Australian or USA populations from which they were drawn. While participants from the current samples generally reflected the population in terms of age and gender distribution, participants were predominantly white, from higher income and education backgrounds, and more likely to be employed. Given research suggesting that COVID-19 infection and vaccination rates differ greatly across different demographic groups, with those from minority race and ethnic groups and those on low incomes or from underserved communities more likely to be infected and have more serious consequences [94–96], and less likely to be vaccinated [37], research replicating current findings in more representative samples, or making explicit comparisons across demographic groups is needed to provide further support for the validity of the VaCCS. It is also important for research to replicate current findings in non-English speaking and under-resourced countries to ensure generalizability of the VaCCS more broadly.
Finally, the VaCCS may not fully encompass all beliefs with respect to COVID-19 vaccinations. The current scale was designed to be comprehensive insofar as it captured the COVID-19 vaccine beliefs and concerns held by the majority of participants in the current samples, and those that have been reflected in previous research on vaccines [97]. However, this does not rule out the possibility that beliefs and concerns salient to specific populations, or idiosyncratic beliefs and concerns relevant to smaller segments of the population, such as religious groups, exist that were not identified in the development stages, or failed to emerge in our factor analyses. For example, beliefs about healthcare access and treatment costs did not emerge from the current analysis. This may be because participants were sourced from countries where vaccine access is assumed to be provided free of charge, or that participants were largely from high-income groups where healthcare access is not a primary consideration. We look to future formative research to examine whether such beliefs may be salient considerations with respect to the COVID-19 vaccine in other national groups, or in groups from underserved communities.
Conclusion
Given the essential role that vaccines play in reducing serious COVID-19 cases and in reducing infection spread, the goal of the current study was to provide researchers, professionals, and policymakers working in healthcare contexts and COVID-19 vaccine vaccination programs with a valid and reliable measure that captures the beliefs likely to be implicated in individuals’ decisions to get vaccinated. Our scale was developed from first principles in a rigorous multi-stage process, and the emergent 36-item, 8-subscale measure demonstrated good psychometric integrity, and concurrent and predictive validity. The current research paves the way for the research community to now apply the scale as a means to assess the strength of beliefs and concerns with respect to the COVID-19 vaccine, and used it as a means to assess candidate correlates of vaccine intentions and behavior. Such research will further contribute to the evidence base for the validity of the scale, and establish evidence for its broader application in diverse populations and contexts.
Supporting information
S1 File. Coding of participant responses and theme extraction in Phase 1 of the study.
https://doi.org/10.1371/journal.pone.0264784.s001
(DOCX)
S4 File. Extracted scale items and their sources.
https://doi.org/10.1371/journal.pone.0264784.s004
(DOCX)
Acknowledgments
We thank Daniel Phipps for his assistance with literature search and data collection and extraction.
References
- 1. Chakraborty I, Maity P: COVID-19 outbreak: Migration, effects on society, global environment and prevention. Sci Tot Environ. 2020; 728:138882. https://doi.org/10.1016/j.scitotenv.2020.138882
- 2. Mofijur M, Fattah IMR, Alam MA, et al.: Impact of COVID-19 on the social, economic, environmental and energy domains: Lessons learnt from a global pandemic. Sust Prod Cons. 2021; 26:343–359. pmid:33072833
- 3. Woolf SH, Chapman DA, Lee JH: COVID-19 as the leading cause of death in the United States. JAMA. 2020. https://doi.org/10.1001/jama.2020.24865
- 4. Hales AH, Wesselmann ED, Hilgard J: Improving psychological science through transparency and openness: An overview. Persp Behav Sci. 2019; 42:13–31. pmid:31976419
- 5.
Mousazadeh M, Naghdali Z, Goldar ZM, et al.: A review of deciphering the successes and learning from the failures in preventive and health policies to stop the COVID-19 pandemic. In Hadi Dehghani M., Karri R. R. and Roy S. (eds), Environmental and Health Management of Novel Coronavirus Disease (COVID-19): Academic Press, 2021, 269–303. https://doi.org/10.1016/B978-0-323-85780-2.00003-2
- 6. Corey L, Mascola JR, Fauci AS, Collins FS: A strategic approach to COVID-19 vaccine R&D. Science. 2020; 368:948–950. pmid:32393526
- 7. Uttarilli A, Amalakanti S, Kommoju P-R, et al.: Super-rapid race for saving lives by developing COVID-19 vaccines. J Integr Bioinform. 2021; 18:27–43. pmid:33761582
- 8.
Doherty Institute: Doherty modelling report. 2021. https://www.doherty.edu.au/uploads/content_doc/DohertyModelling_NationalPlan_and_Addendum_20210810.pdf
- 9. Randolph HE, Barreiro LB: Herd immunity: Understanding COVID-19. Immun. 2020; 52:737–741. pmid:32433946
- 10.
Evans J: PM says 80 per cent of Australians must be vaccinated before lockdowns end. Australian Broadcasting Corporation. 2021. https://www.abc.net.au/news/2021-07-30/national-cabinet-update-covid19-support-vaccine-targets/100338334.
- 11. Han E, Tan MMJ, Turk E, et al.: Lessons learnt from easing COVID-19 restrictions: an analysis of countries and regions in Asia Pacific and Europe. Lancet. 2020; 396:1525–1534. pmid:32979936
- 12. Chow D: Denmark lifts all Covid restrictions, crediting high vaccine uptake. NBC News. 2021. https://www.nbcnews.com/science/science-news/denmark-lifts-covid-restrictions-crediting-high-vaccine-uptake-rcna1967.
- 13. Mathieu E, Ritchie H, Ortiz-Ospina E, et al.: A global database of COVID-19 vaccinations. Nat Hum Beh. 2021; 5:947–953. https://doi.org/10.1038/s41562-021-01122-8
- 14.
Ellyatt H: COVID vaccination rates have slumped in some parts of the world, and experts are worried. CNBC. 2021. https://www.cnbc.com/2021/09/06/covid-vaccinations-have-slumped-in-parts-of-the-us-and-europe.html.
- 15. Horton J: Covid: Where and why is the US vaccine rollout slowing down? BBC News. 2021. https://www.bbc.com/news/57889961.
- 16. Larson HJ: Negotiating vaccine acceptance in an era of reluctance. Hum Vaccin Immunther. 2013; 9:1779–1781. https://doi.org/10.4161/hv.25932
- 17. Sallam M: COVID-19 vaccine hesitancy worldwide: A concise systematic review of vaccine acceptance rates. Vaccines. 2021; 9:160. pmid:33669441
- 18. Soares P, Rocha JV, Moniz M, et al.: Factors associated with COVID-19 vaccine hesitancy. Vaccines. 2021; 9:300. pmid:33810131
- 19. Lane S, MacDonald NE, Marti M, Dumolard L: Vaccine hesitancy around the globe: Analysis of three years of WHO/UNICEF Joint Reporting Form data-2015–2017. Vaccine. 2018; 36:3861–3867. pmid:29605516
- 20. Lazarus JV, Ratzan SC, Palayew A, et al.: A global survey of potential acceptance of a COVID-19 vaccine. Nat Med. 2021; 27:354–354. pmid:33432176
- 21. Puri N, Coomes EA, Haghbayan H, Gunaratne K: Social media and vaccine hesitancy: New updates for the era of COVID-19 and globalized infectious diseases. Hum Vaccin Immunther. 2020; 16:2586–2593. pmid:32693678
- 22. Evanega S, Lynas M, Adams J, Smolenyak K: Coronavirus misinformation: Quantifying sources and themes in the COVID-19 ‘infodemic’. JMIR Preprints. 2020:19/10/2020:25143.
- 23. Sear RF, Velásquez N, Leahy R, et al.: Quantifying COVID-19 content in the online health opinion war using machine learning. IEEE Access. 2020; 8:91886–91893. pmid:34192099
- 24. Hagger MS, Smith SR, Keech JJ, Moyers SA, Hamilton K: Predicting social distancing intention and behavior during the COVID-19 pandemic: An integrated social cognition model. Ann Behav Med. 2020; 54:713–727. pmid:32914831
- 25. Hagger MS, Smith SR, Keech JJ, Moyers SA, Hamilton K: Predicting physical distancing over time during COVID-19: Testing an integrated model. Psychol Health. 2021. pmid:34435916
- 26. Clark C, Davila A, Regis M, Kraus S: Predictors of COVID-19 voluntary compliance behaviors: An international investigation. Glob Transit. 2020; 2:76–82. pmid:32835202
- 27. MacDonald NE: Vaccine hesitancy: Definition, scope and determinants. Vaccine. 2015; 33:4161–4164. pmid:25896383
- 28. Chou W-YS, Budenz A: Considering emotion in COVID-19 vaccine communication: addressing vaccine hesitancy and fostering vaccine confidence. Health Comm. 2020; 35:1718–1722. pmid:33124475
- 29. Freeman D, Loe BS, Chadwick A, et al.: COVID-19 vaccine hesitancy in the UK: The Oxford coronavirus explanations, attitudes, and narratives survey (Oceans) II. Psychol Med. 2020. https://doi.org/10.1017/S0033291720005188
- 30. Hudson A, Montelpare WJ: Predictors of vaccine hesitancy: Implications for COVID-19 public health messaging. Int J Environ Res Pub Health. 2021; 18:8054. pmid:34360345
- 31. Dubé E, Laberge C, Guay M, et al.: Vaccine hesitancy. Hum Vaccin Immunther. 2013; 9:1763–1773. pmid:23584253
- 32. Larson HJ, Clarke RM, Jarrett C, et al.: Measuring trust in vaccination: A systematic review. Hum Vaccin Immunther. 2018; 14:1599–1609. pmid:29617183
- 33. Smith LE, Amlôt R, Weinman J, Yiend J, Rubin GJ: A systematic review of factors affecting vaccine uptake in young children. Vaccine. 2017; 35:6059–6069. pmid:28974409
- 34. Hughes S, Machan L: It’s a conspiracy: Covid-19 conspiracies link to psychopathy, Machiavellianism and collective narcissism. Pers Ind Diff. 2021; 171:110559. pmid:33867616
- 35. Johnson MO: Personality correlates of HIV vaccine trial participation. Pers Ind Diff. 2000; 29:459–467. https://doi.org/10.1016/S0191-8869(99)00206-8
- 36. Lee CHJ, Duck IM, Sibley CG: Personality and demographic correlates of New Zealanders’ confidence in the safety of childhood vaccinations. Vaccine. 2017; 35:6089–6095. pmid:28965999
- 37. Kreps S, Prasad S, Brownstein JS, et al.: Factors associated with US adults’ likelihood of accepting COVID-19 vaccination. JAMA Net Open. 2020; 3:e2025594–e2025594. pmid:33079199
- 38. Guidry JPD, Laestadius LI, Vraga EK, et al.: Willingness to get the COVID-19 vaccine with and without emergency use authorization. Am J Infect Con. 2021; 49:137–142. pmid:33227323
- 39. Dror AA, Eisenbach N, Taiber S, et al.: Vaccine hesitancy: The next challenge in the fight against COVID-19. Eur J Epidemiol. 2020; 35:775–779. pmid:32785815
- 40. Kashte S, Gulbake A, El-Amin Iii SF, Gupta A: COVID-19 vaccines: rapid development, implications, challenges and future prospects. Hum Cell. 2021; 34:711–733. pmid:33677814
- 41. Lin C, Tu P, Beitsch LM: Confidence and receptivity for COVID-19 vaccines: A rapid systematic review. Vaccines. 2021; 9:16. https://doi.org/10.3390/vaccines9010016
- 42. Islam MS, Kamal A-HM, Kabir A, et al.: COVID-19 vaccine rumors and conspiracy theories: The need for cognitive inoculation against misinformation to improve vaccine adherence. PLoS ONE. 2021; 16:e0251605. pmid:33979412
- 43. Lockyer B, Islam S, Rahman A, et al.: Understanding COVID-19 misinformation and vaccine hesitancy in context: Findings from a qualitative study involving citizens in Bradford, UK. Health Exp. 2021; 24:1158–1167. pmid:33942948
- 44. Murphy J, Vallières F, Bentall RP, et al.: Psychological characteristics associated with COVID-19 vaccine hesitancy and resistance in Ireland and the United Kingdom. Nat Comm. 2021; 12:29. pmid:33397962
- 45. Brennen JS, Simon F, Howard PN, Nielsen RK: Types, sources, and claims of COVID-19 misinformation. Retrieved September 29, 2021 from https://reutersinstitute.politics.ox.ac.uk/types-sources-and-claims-covid-19-misinformation
- 46. Artino AR, La Rochelle JS, Dezee KJ, Gehlbach H: Developing questionnaires for educational research: AMEE Guide No. 87. Med Teach. 2014; 36:463–474. pmid:24661014
- 47.
Australian Government: Australia’s COVID-19 vaccine national roll-out strategy. Retrieved November 2021, 2021 from https://www.health.gov.au/sites/default/files/documents/2021/01/covid-19-vaccination-australia-s-covid-19-vaccine-national-roll-out-strategy.pdf
- 48. Martin LR, Petrie KJ: Understanding the dimensions of anti-vaccination attitudes: the vaccination attitudes examination (VAX) scale. Ann Behav Med. 2017; 51:652–660. pmid:28255934
- 49. Grimmelikhuijsen S, Knies E: Validating a scale for citizen trust in government organizations. Int Rev Admin Sci. 2017; 83:583–601. https://doi.org/10.1177/0020852315585950
- 50. Kroh M: Measuring left–right political orientation: The choice of response format. Pub Opin Quar. 2007; 71:204–220. https://doi.org/10.1093/poq/nfm009
- 51. Nadelhoffer T, Shepard J, Nahmias E, Sripada C, Ross LT: The free will inventory: Measuring beliefs about agency and responsibility. Consc Cog. 2014; 25:27–41. pmid:24561311
- 52. Akkerman A, Mudde C, Zaslove A: How populist are the people? Measuring populist attitudes in voters. Comp Pol Stud. 2014; 47:1324–1353. https://doi.org/10.1177/0010414013512600
- 53. Imhoff R, Lamberty P: A Bioweapon or a Hoax? The Link Between Distinct Conspiracy Beliefs About the Coronavirus Disease (COVID-19) Outbreak and Pandemic Behavior. Soc Psychol Pers Sci. 2020; 11:1110–1118. https://doi.org/10.1177/1948550620934692
- 54. Shapiro GK, Holding A, Perez S, Amsel R, Rosberger Z: Validation of the vaccine conspiracy beliefs scale. Papillomavirus Res. 2016; 2:167–172. pmid:29074176
- 55. Bruder M, Haffke P, Neave N, Nouripanah N, Imhoff R: Measuring individual differences in generic beliefs in conspiracy theories across cultures: Conspiracy mentality questionnaire. Front Psychol. 2013; 4:225. pmid:23641227
- 56. Gosling SD, Rentfrow PJ, Swann WB Jr: A very brief measure of the Big-Five personality domains. J Res Pers. 2003; 37:504–528. https://doi.org/10.1016/S0092-6566(03)00046-1
- 57. Carleton RN, Norton MAPJ, Asmundson GJG: Fearing the unknown: A short version of the Intolerance of Uncertainty Scale. J Anx Dis. 2007; 21:105–117. https://doi.org/10.1016/j.janxdis.2006.03.014
- 58.
Ajzen I: Constructing a TPB questionnaire: Conceptual and methodological considerations. Retrieved September 1, 2002, from http://people.umass.edu/~aizen/pdf/tpb.measurement.pdf
- 59. Horne R, Weinman J, Hankins M: The beliefs about medicines questionnaire: The development and evaluation of a new method for assessing the cognitive representation of medication. Psychol Health. 1999; 14:1–24. https://doi.org/10.1080/08870449908407311
- 60. Betsch C, Schmid P, Heinemeier D, et al.: Beyond confidence: Development of a measure assessing the 5C psychological antecedents of vaccination. PLoS ONE. 2018; 13:e0208601. pmid:30532274
- 61. Zingg A, Siegrist M: Measuring people’s knowledge about vaccination: Developing a one-dimensional scale. Vaccine. 2012; 30:3771–3777. pmid:22445808
- 62. Subramanianm SV, Huijts T, Avendano M: Self-reported health assessments in the 2002 World Health Survey: how do they correlate with education? Bull World Health Org. 2010; 88:131–138. pmid:20428370
- 63. Revelle W: psych: Procedures for psychological, psychometric, and personality research. Retrieved May 6, 2018 from https://cran.r-project.org/web/packages/psych/index.html
- 64.
Bernaards C, Jennrich R: GPArotation. Retrieved September 29, 2021 from https://cran.r-project.org/web/packages/GPArotation/GPArotation.pdf
- 65. Marsh HW, Hau KT, Wen Z: In search of golden rules: Comment on hypothesis testing approaches to setting cutoff values for fit indexes and dangers in overgeneralising Hu & Bentler’s (1999) findings. Struc Eq Mod. 2004; 11:320–341. https://doi.org/10.1207/s15328007sem1103_2
- 66. Hu L, Bentler PM: Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Struct Eq Mod. 1999; 6:1–55. https://doi.org/10.1080/10705519909540118
- 67.
Diamantopoulos A, Sigauw JA: Introducing LISREL. Thousand Oaks, CA: Sage, 2000.
- 68. Fornell C, Larcker DF: Structural equation models with unobservable variables and measurement error. J Mark Res. 1981; 18:39–50.
- 69. Cheung GW, Rensvold RB: Evaluating goodness-of-fit indexes for testing measurement invariance. Struc Eq Mod. 2002; 9:233–255. https://doi.org/10.1207/S15328007SEM0902_5
- 70. Rosseel Y: lavaan: An R package for structural equation modeling. J Stat Soft. 2012; 48:1–36. https://doi.org/10.18637/jss.v048.i02
- 71. Nájera Catalán HE: Reliability, population classification and weighting in multidimensional poverty measurement: A Monte Carlo study. Soc Ind Res. 2019; 142:887–910. https://doi.org/10.1007/s11205-018-1950-z
- 72. Gwet KL: Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol. 2008; 61:29–48. pmid:18482474
- 73.
Brown TA: Confirmatory factor analysis for applied research (2nd Ed.). Abingdon, UK: Routledge, 2015.
- 74. Harrison EA, Wu JW: Vaccine confidence in the time of COVID-19. Eur J Epidemiol. 2020; 35:325–330. pmid:32318915
- 75. Karlsson LC, Soveri A, Lewandowsky S, et al.: Fearing the disease or the vaccine: The case of COVID-19. Pers Ind Diff. 2021; 172:110590. https://doi.org/10.1016/j.paid.2020.110590
- 76. Freeman D, Lambe S, Yu L-M, et al.: Injection fears and COVID-19 vaccine hesitancy. Psychol Med. 2021. pmid:34112276
- 77. Debus M, Tosun J: Political ideology and vaccination willingness: Implications for policy design. Pol Sci. 2021; 54:477–491. pmid:34149102
- 78. Lasco G: Medical populism and the COVID-19 pandemic. Glob Pub Health. 2020; 15:1417–1429. pmid:32780635
- 79. Eberl J-M, Huber RA, Greussing E: From populism to the “plandemic”: Why populists believe in COVID-19 conspiracies. J Elec Pub Op Part. 2021; 31:272–284. https://doi.org/10.1080/17457289.2021.1924730
- 80. Stein RA, Ometa O, Pachtman Shetty S, et al.: Conspiracy theories in the era of COVID-19: A tale of two pandemics. Int J Clin Prac. 2021; 75:e13778. pmid:33480171
- 81. Geana MV, Anderson S, Ramaswamy M: COVID-19 vaccine hesitancy among women leaving jails: A qualitative study. Pub Health Nurs. 2021; 38:892–896. pmid:33973268
- 82. Scherr S, Leiner D: The populist hotbed: How political attitudes, resentment, and justice beliefs predict both exposure to and avoidance of specific populist news features in the United States. PLoS ONE. 2021; 16:e0258220. pmid:34618837
- 83. Stecula DA, Pickup M: How populism and conservative media fuel conspiracy beliefs about COVID-19 and what it means for COVID-19 behaviors. Res Polit. 2021; 8:2053168021993979. https://doi.org/10.1177/2053168021993979
- 84. Rogers RW: A protection motivation theory of fear appeals and attitude change. J Psychol. 1975; 91:93–114. pmid:28136248
- 85. Floyd DL, Prentice-Dunn S, Rogers RW: A meta-analysis of research on protection motivation theory. J Appl Soc Psychol. 2000; 30:407–429. https://doi.org/10.1111/j.1559-1816.2000.tb02323.x
- 86. Milne S, Sheeran P, Orbell S: Prediction and intervention in health-related behavior: A meta-analytic review of protection motivation theory. J Appl Soc Psychol. 2000; 30:106–143. https://doi.org/10.1111/j.1559-1816.2000.tb02308.x
- 87. Guillon M, Kergall P: Factors associated with COVID-19 vaccination intentions and attitudes in France. Pub Health. 2021; 198:200–207. pmid:34481275
- 88. Malik AA, McFadden SM, Elharake J, Omer SB: Determinants of COVID-19 vaccine acceptance in the US. E Clin Med. 2020; 26. pmid:32838242
- 89. Sherman SM, Smith LE, Sim J, et al.: COVID-19 vaccination intention in the UK: results from the COVID-19 vaccination acceptability study (CoVAccS), a nationally representative cross-sectional survey. Hum Vaccin Immunther. 2021; 17:1612–1621. https://doi.org/10.1080/21645515.2020.1846397
- 90. Caserotti M, Girardi P, Rubaltelli E, et al.: Associations of COVID-19 risk perception with vaccine hesitancy over time for Italian residents. Soc Sci Med. 2021; 272:113688. pmid:33485215
- 91. Ruiz JB, Bell RA: Predictors of intention to vaccinate against COVID-19: Results of a nationwide survey. Vaccine. 2021; 39:1080–1086. pmid:33461833
- 92. Jennings W, Stoker G, Bunting H, et al.: Lack of trust, conspiracy beliefs, and social media use predict COVID-19 vaccine hesitancy. Vaccines. 2021; 9:593. pmid:34204971
- 93. Schernhammer E, Weitzer J, Laubichler MD, et al.: Correlates of COVID-19 vaccine hesitancy in Austria: Trust and the government. J Pub Health. 2021. pmid:33948665
- 94. Gu T, Mack JA, Salvatore M, et al.: Characteristics associated with racial/ethnic disparities in COVID-19 outcomes in an academic health care system. JAMA Net Open. 2020; 3:e2025197–e2025197. https://doi.org/10.1001/jamanetworkopen.2020.25197
- 95. Muñoz-Price LS, Nattinger AB, Rivera F, et al.: Racial disparities in incidence and outcomes among patients with COVID-19. JAMA Net Open. 2020; 3:e2021892–e2021892. pmid:32975575
- 96. Little C, Alsen M, Barlow J, et al.: The impact of socioeconomic status on the clinical outcomes of COVID-19; a retrospective cohort study. J Comm Health. 2021; 46:794–802. https://doi.org/10.1007/s10900-020-00944-3
- 97. Larson HJ, Jarrett C, Eckersberger E, Smith DMD, Paterson P: Understanding vaccine hesitancy around vaccines and vaccination from a global perspective: A systematic review of published literature, 2007–2012. Vaccine. 2014; 32:2150–2159. pmid:24598724