Types of studies

We will include data from randomised controlled trials (RCTs), including studies reported as full text, those published as abstract only, and unpublished data, with no language restrictions. We will also collect data on adverse events from open‐label extension studies of RCTs.

For both efficacy and safety outcomes, we will also include prospective observational studies, including published data from registries and insurance databases, provided that they are representative of the underlying population.

For safety outcomes, we will also include published data from vaccine safety reporting systems (e.g. the US Centers for Disease Control and Prevention (CDC) and Food and Drug Administration (FDA) Vaccine Adverse Event Reporting System (vaers.hhs.gov)).

Types of participants

We will include adults aged 18 years or older, treated with any immunomodulatory medication for an autoimmune inflammatory rheumatic diseases (AIRDs), such as rheumatoid arthritis, psoriatic arthritis, spondyloarthritis, connective tissue diseases, and vasculitis (for full list see Appendix 1).

A list of immunomodulatory medications used in the treatment of AIRDs is included in Appendix 2. As this is a living review, some medications have been included that may not be available or registered for AIRDs in all countries, and further medications may be added to this list as they become available.

If RCTs include a mixed population of participants, we will exclude it, unless data for adults with autoimmune or inflammatory rheumatic disease are presented separately, or the trialists can provide separate data for such participants. 

Types of interventions

We will include studies comparing any vaccine directed against the SARS‐CoV‐2 virus, or intended to provide immune protection against COVID‐19, with placebo injection or no intervention. We will also include studies comparing COVID‐19 vaccines that differ in mechanism, dose, or dosing regimen interval, timing or temporal relationship to immunomodulatory medications. If no RCTs or controlled observational studies are available, we will also include single‐arm observational studies of COVID‐19 vaccines in people with AIRDs on immunomodulatory therapies.

Types of outcome measures

Electronic searches

We will search the following resources.

• Cochrane COVID‐19 Study Register (covid-19.cochrane.org/).

The Cochrane COVID‐19 Study Register is a specialised register built within the Cochrane Register of Studies (CRS), and is maintained by Cochrane Information Specialists. The register contains study reports from several sources:

1. daily searches of PubMed;

2. daily searches of ClinicalTrials.gov;

3. weekly searches of Embase.com;

4. weekly searches of medRxiv;

5. weekly searches of the WHO International Clinical Trials Registry Platform (ICTRP);

6. monthly searches of the Cochrane Central Register of Controlled Trials (CENTRAL).

Complete data sources and search methods for the register are available at community.cochrane.org/about-covid-19-study-register.

• Epistemonikos COVID‐19 L·OVE platform (app.iloveevidence.com/loves/5e6fdb9669c00e4ac072701d).

The Epistemonikos COVID‐19 L·OVE platform contains primary and secondary study references from the Epistemonikos database, as well as:

1. daily searches of PubMed/MEDLINE;

2. daily searches ClinicalTrials.gov;

3. daily searches of the WHO International Clinical Trials Registry Platform (ICTRP);

4. daily searches of preprint archives, including medRxiv, bioRxiv, SSRN;

5. weekly searches of Embase.

Complete data sources and search method for the platform are available at: app.iloveevidence.com/loves/5e6fdb9669c00e4ac072701d?population=5e7fce7e3d05156b5f5e032a&section=methods&classification=all

We will not limit our search results by language, date, or study design. See Appendix 4 for the search strategies.

For assessments of adverse effects, we will search the websites of the following regulatory agencies:

• US Food and Drug Administration (FDA‐MedWatch (www.fda.gov/Safety/MedWatch/default.htm));

• European Medicines Evaluation Agency (EMEA (www.emea.europa.eu));

• Australian Adverse Drug Reactions Bulletin (www.tga.gov.au/adr/aadrb.htm);

• UK Medicines and Healthcare products Regulatory (MHRA) pharmacovigilance and drug safety updates (www.mhra.gov.uk).

Searching other resources

We will perform supplemental searching by checking reference lists of all primary studies and relevant reviews retrieved by our search. We will search relevant manufacturers' websites for trial information. If we cannot find information about a particular study, we will contact the study authors for information about publication.

To identify planned vaccine surveillance data, we will look for vaccine surveillance programs, and look for the data provided for our population of interest:

• US Centers for Disease Control and Prevention (CDC) Immunization Safety Office (www.cdc.gov/vaccinesafety/ensuringsafety/monitoring/index.html);

• Australian database for reporting of Adverse Events Following Immunisation (AEFI) and associated clinical visits (SAFEVAC (www.safevac.org.au/Home/Index));

• Canadian Adverse Events Following Immunization Surveillance System (CAEFISS (www.canada.ca/en/public-health/services/immunization/canadian-adverse-events-following-immunization-surveillance-system-caefiss.html)).

We will run directed searches for vaccine surveillance programs using Google, and evaluate the first 100 results. We will extract contact information for the data providers in these included studies, and contact them for additional data as required.

We will contact the different programmes for COVID‐19 vaccine data on our population of interest.

We will search for errata or retractions from included studies that are published in full text on PubMed and Retraction Watch Database, and report the date we did this in the review (retractiondatabase.org).

Living systematic review approach

We will run monthly searches in the Cochrane COVID‐19 Study Register and the Epistemonikos COVID‐19 L·OVE platform, from the date of the baseline review search. We will export all results to EndNote, and compare them against previous results. We will also run monthly searches in regulatory agency and vaccine surveillance programme websites, from the date of the baseline search. Prior to the publication of new updates, we will check reference lists for primary studies and relevant reviews as a form of supplementary searching. We will also search relevant manufacturer's websites, check for retractions, and conduct a Google search prior to the publication of new updates. We will contact the corresponding authors of ongoing studies as they are identified, to ask them to advise us when results are available, or if they are willing to share early or unpublished data.

We will either incorporate any studies identified between the baseline searches and publication of the review in the review, or place them in 'Studies awaiting classification'. We will add newly identified studies to the 'What’s new' section, and include them in the update.

We will review search methods (sources, search terms, and frequency) after each update is published, and validate the database strategies against included studies. We will revise search strategies to reflect any changes in text words, to account for software updates to our sources, to add new sources, as needed, or to account for any changes in the eligibility criteria of the review.

Selection of studies

We will download all titles and abstracts retrieved by electronic searching to a reference management database (EndNote), and remove duplicates. We will import the remaining records into Covidence (Covidence).

At least two review authors will independently screen titles and abstracts for potentially relevant studies. We will retrieve the full text of all potentially eligible studies, and at least two review authors will independently screen the full‐text reports for eligibility. We will also identify and record reasons for exclusion of the ineligible studies. Studies reported in abstracts only, unpublished data, and trial reports available as preprints only will be described in the 'Characteristics of studies awaiting classification' section of the full review but outcome data will not be included in the data synthesis.

We will resolve any disagreement through discussion, or if required, we will consult a third person. We will identify and exclude duplicates, and collate multiple reports of the same study, so that each study, rather than each report, is the unit of interest in the review. We will record the selection process in sufficient detail to complete a PRISMA flow diagram and 'Characteristics of excluded studies' table (Page 2021b).

Data extraction and management

We will use a standard data collection form to extract study characteristics and outcome data. We will pilot the form on at least one study from the review. Two review authors will independently extract study characteristics from the included studies. A third review author will spot‐check the study characteristics for accuracy against the trial report. 

We will extract the following study characteristics.

1. Methods: study design, total duration of study, details of any run‐in period, number of study centres and location, study setting, withdrawals, and date of study.

2. Participants: N, mean age, age range, sex, rheumatic disease, disease duration, severity of condition, diagnostic criteria, current immunomodulatory medications (including dose), previous medication including previous disease‐modifying anti‐rheumatic drug (DMARD) use, other medications at baseline, comorbidities, including respiratory disease, inclusion criteria, and exclusion criteria.

3. Interventions: vaccine, dose, schedule, route of administration, vaccine technology, adjuvant, modification of background immunomodulatory drug regimen (e.g. withholding or deferral of immunomodulatory drug prior to vaccination).

4. Outcomes: major and minor outcomes specified and collected, and time points reported.

5. Characteristics of the design of the trial, as outlined in the 'Assessment of risk of bias in included studies' section.

6. Notes: funding for trial, and notable declarations of interest of trial authors, details of any correspondence with study authors.

At least two review authors will independently extract outcome data from the included studies. We will extract the number of events and number of participants per treatment group for dichotomous outcomes, and means and standard deviations and number of participants per treatment group for continuous outcomes. In the ‘Characteristics of included studies’ table, we will note if outcome data were not reported in a usable way, and when data were transformed or estimated from a graph. We will resolve disagreements by consensus, or by involving a third person. One review author will transfer data into a Review Manager 5 file (Review Manager 2020). We will double‐check that data are entered correctly by comparing the data presented in the systematic review with the study reports.

Two review authors will use freely available software (e.g. DigitizeIt) to extract data from graphs or figures, if means and measures of variance are not reported in the text of the included studies.

Assessment of risk of bias in included studies

If RCTs are available, we will assess risk of bias for each study using the Cochrane RoB 1 tool, as outlined in Higgins 2017. At least two review authors will independently assess the risk of bias. We will resolve any disagreements by discussion, or by involving another review author.

We will assess the risk of bias according to the following domains (Higgins 2017):

1. random sequence generation;

2. allocation concealment;

3. blinding of participants and personnel;

4. blinding of outcome assessment;

5. incomplete outcome data;

6. selective outcome reporting;

7. other bias.

We will classify each potential source of bias as high, low, or unclear risk, and provide a quote from the study report with a justification for our judgment in the risk of bias table. We will summarise the risk of bias judgements across different studies for each of the domains listed. We will consider blinding separately for different key outcomes where necessary (e.g. for unblinded outcome assessment, risk of bias for all‐cause mortality may be different than for a self‐reported pain scale). We will consider the impact of missing data by key outcomes.

Where information on risk of bias relates to unpublished data or correspondence with a trialist, we will note this in the risk of bias table. When considering treatment effects, we will take into account the risk of bias for the studies that contribute to that outcome. We will present the figures generated by the RoB 1 to provide summary assessments of the risk of bias.

If controlled non‐randomised studies of interventions (NRSI) are included, we will assess risk of bias based on the Risk Of Bias In Non‐randomized Studies of Interventions (ROBINS‐I) tool (Sterne 2021). Potential confounding domains for studies of this type include age, sex, disease activity, and comorbidities. The co‐interventions of interest are the disease‐modifying anti‐rheumatic drugs or other immunomodulatory medications (as listed in Appendix 2).

It is likely that we will include single‐arm observational studies of COVID‐19 vaccination in this review. For studies of this type, we will adapt the risk of bias assessment criteria for observational studies tool provided by Cochrane Childhood Cancer (Mulder 2019), and described in Nussbaumer‐Streit 2020 (Table 1). Two review authors will independently assess risk of bias of the included single‐arm observational studies, with regard to the selection of the study group, the follow‐up and outcome assessments, and the methods used for risk estimation. To evaluate internal validity, we will assess the risk of selection bias, attrition bias, detection bias, and confounding that is present in the included studies. Assessment will include the following items: representativeness of the study group, completeness of the follow‐up, blinding of the outcome assessors, and adjustment for important confounding factors. To evaluate external validity, we will assess the risk of reporting bias, which includes the following items: definition of the study group, reporting the length of follow‐up, objectiveness of the outcome definition, and definition of the analyses. We will rate the risk of bias as low, moderate, serious, or critical. We will resolve any disagreements by discussion, or by involving another review author.

Measures of treatment effect

We will analyse dichotomous data as risk ratios (RRs) or Peto ORs when the outcome is a rare event (approximately less than 10%), and use 95% CIs. We will analyse continuous data as mean differences (MDs) or standardised mean differences (SMDs), depending on whether the same scale is used to measure an outcome, and 95% CIs. We will enter data presented as a scale with a consistent direction of effect across studies.

When different scales are used to measure the same conceptual outcome (e.g. disability), we will calculate SMDs, with corresponding 95% CIs. We will back‐translate SMDs to a typical scale (e.g. 0 to 10 for pain) by multiplying the SMD by a typical among‐person standard deviation (e.g. the standard deviation of the control group at baseline from the most representative trial) (Higgins 2021a).  We will analyse rate data using Poisson methods.

For dichotomous outcomes, we will calculate the number needed to treat for an additional beneficial outcome (NNTB), or the number needed to treat for an additional harmful outcome (NNTH) from the control group event rate and the relative risk, using the Visual Rx NNT calculator (Cates 2016). We will calculate the NNTB or NNTH for continuous measures using the Wells calculator (available at the CMSG Editorial office: musculoskeletal.cochrane.org/).

For dichotomous outcomes, we will calculate the absolute percent change from the difference in the risks between the intervention and control groups, using GRADEpro GDT, and expressed as a percentage (GRADEpro GDT). We will calculate the relative percent change as the risk ratio ‐ 1, and express it as a percentage.

For continuous outcomes, we will calculate the absolute percent change by dividing the mean difference by the scale of the measure, and express it as a percentage. We will calculate the relative difference as the absolute benefit (mean difference) divided by the baseline mean of the control group, and express it as a percentage.

In the 'Effects of interventions' section, and the 'What happens' column of the summary of findings table, we will provide the absolute percent change, the relative per cent change from baseline, and the NNTB or NNTH.

Unit of analysis issues

For trials with more than two arms, we will describe all study groups in the ’Characteristics of included studies’ table, but we will include in the analysis only intervention groups that meet our review criteria. When the variance of the difference between intervention and comparator is not reported, we will calculate this from the variances of all trial arms. When a study includes multiple relevant treatment arms, we will combine groups to perform a single pairwise comparison (Higgins 2021b). If this prevents identification of potential heterogeneity, we will analyse each group separately against a common control group. However, to ensure that a common control group is not included multiple times in a meta‐analysis that includes several interventions from the same trial, we will proportionately reduce control group data. For example, in a study with two interventions and a single control group, we will halve the numbers of participants and events in the control group. When studies report only differences between treatment groups, as opposed to mean effects for each group, we will analyse data using the generic inverse‐variance function.

Dealing with missing data

We will contact investigators or study sponsors to verify key study characteristics, and obtain missing numerical outcome data when possible (e.g. when a study is identified as abstract only, or when data are not available for all participants). Where this is not possible, and the missing data are thought to introduce serious bias, we will explore the impact of including such studies in the overall assessment of results by completing a sensitivity analysis. We will clearly describe any assumptions and imputations used to handle missing data, and explore the effect of imputation by completing sensitivity analyses.

For dichotomous outcomes (e.g. number of COVID‐19 cases), the event rate will be calculated using the number of participants randomised in the group as the denominator.

For continuous outcomes (e.g. anti‐SARS‐CoV‐2 antibody titre), we will calculate the MD or SMD based on the number of participants analysed at that time point. If the number of participants analysed is not presented for each time point, we will use the number of randomised participants in each group at baseline.

Where possible, we will compute missing standard deviations from other statistics, such as standard errors, CIs or P values. If we are unable to calculate standard deviations, we will impute them (e.g. from other studies in the meta‐analysis) according to the methods recommended in the Cochrane Handbook for Systematic Reviews of Interventions (Deeks 2021).

Assessment of heterogeneity

We will assess clinical and methodological diversity in terms of participants, interventions, outcomes, and study characteristics for the included studies to determine whether a meta‐analysis is appropriate. We will do this by observing these data from the data extraction tables. We will assess statistical heterogeneity by visually inspecting the forest plot for obvious differences in results between the studies, and using I² and Chi² statistics.

As recommended in the Cochrane Handbook for Systematic Reviews of Interventions, an I² value of 0% to 40% might not be important; 30% to 60% may represent moderate heterogeneity; 50% to 90% may represent substantial heterogeneity; and 75% to 100% represents considerable heterogeneity (Deeks 2021). We will keep in mind that the importance of the I² depends on: (i) the magnitude and direction of effects; and (ii) the strength of evidence for heterogeneity.

We will interpret a Chi² test P value ≤ 0.10 as evidence of statistical heterogeneity.

If we identify substantial heterogeneity, we will report it, and investigate possible causes by following the recommendations in Deeks 2021. 

Assessment of reporting biases

We will create and examine a funnel plot to explore possible small study biases. In interpreting funnel plots, we will examine the different possible reasons for funnel plot asymmetry, as outlined in the Cochrane Handbook for Systematic Reviews of Interventions, and relate this to the results of the review; if we are able to pool more than 10 trials, we will undertake formal statistical tests to investigate funnel plot asymmetry, and will follow the recommendations in Page 2021a.

To assess outcome reporting bias, we will check trial protocols against published reports. For studies published after 1 July 2005, we will screen the WHO ICTRP Search Portal for the a priori trial protocol (apps.who.int/trialsearch/). We will evaluate whether selective reporting of outcomes is present.

Data synthesis

We will undertake meta‐analyses only when this is meaningful, i.e. if the treatments, participants, and the underlying clinical question are similar enough for pooling to make sense. We do not plan to mix studies with different design features in the same meta‐analysis (Reeves 2021). We plan to pool outcomes from randomised trials with similar characteristics (participants, interventions, and common comparators, outcome measures, and timing of outcome measurement) to provide estimates of benefit and harm. We plan to synthesise effect estimates using a random‐effects meta‐analysis model based on the assumption that clinical diversity is likely to exist, and that different studies are estimating different intervention effects.

When we cannot pool data, we plan to present effect estimates and 95% CIs of each trial in tables, and summarise the results in text. Our primary analysis will be stratified by fixed dose reduction or disease activity‐guided dose reduction versus discontinuation.

Subgroup analysis and investigation of heterogeneity

We plan to carry out the following subgroup analyses, to assess if the benefits or risks of COVID‐19 vaccination are modified by the underlying rheumatic disease or the type of immunomodulatory therapy.

1. Participants with inflammatory arthritis (including rheumatoid arthritis, psoriatic arthritis, and axial spondyloarthritis) compared to those with autoimmune connective tissue diseases (including systemic lupus erythematosus, systemic sclerosis, inflammatory myopathy, Sjögren’s syndrome, systemic vasculitis, and undifferentiated connective tissue disease).

2. Participants treated with rituximab compared to other immunomodulatory medications.

3. Participants treated with glucocorticoids (equivalent to prednisolone dose of > 10 mg/day), compared to low‐dose or no glucocorticoid use.

4. Participants treated with methotrexate compared to other immunomodulatory medications.

We will analyse the subgroups using the following outcomes:

1. COVID‐19 infection;

2. severe COVID‐19 disease; 

3. severe adverse events.

We will use the formal test for subgroup interactions in Review Manager 2020, and use caution in the interpretation of subgroup analyses, as indicated in the Cochrane Handbook for Systematic Reviews of Interventions (Deeks 2021). We will compare the magnitude of the effects between the subgroups by assessing the overlap of the CIs of the summary estimate. Non‐overlap of the CIs indicates statistical significance.

Sensitivity analysis

If RCTs are identified, we will investigate the robustness of the treatment effect to potential selection bias and potential detection bias by completing the following sensitivity analyses.

• Selection bias: remove the trials that reported inadequate or unclear allocation concealment from the meta‐analysis, to see if this changes the overall treatment effect.

• Detection bias: remove the trials that reported inadequate or unclear participant blinding from the meta‐analysis, to see if this changes the overall treatment effect.

If only observational data are available, we will complete a sensitivity analysis by including only prospectively‐collected data (i.e. prospective cohort studies).

Summary of findings and assessment of the certainty of the evidence

We will create a summary of findings (SoF) table for the primary comparison using the following outcomes at their primary time points:

1. COVID‐19 infection;

2. severe COVID‐19 disease, including death, hospitalisation due to COVID‐19, mechanical ventilation, days in intensive care unit, or as defined by study authors;

3. disease‐specific adverse events (including worsening or flare‐up of the underlying AIRD);

4. serious adverse events;

5. adverse events.

Two review authors will independently assess the certainty of the evidence. We will use the five GRADE considerations (study limitations, consistency of effect, imprecision, indirectness, and publication bias) to assess the certainty of a body of evidence as it relates to the studies that contribute data to the meta‐analyses for the outcomes, and report the certainty of evidence as high, moderate, low, or very low. We will use methods and recommendations described in the Cochrane Handbook forSystematic Reviews of Interventions (Schünemann 2021a). We will use the current version of GRADEpro GDT software to prepare the SoF tables (GRADEpro GDT). We will justify all decisions to downgrade or upgrade the certainty of evidence for each outcome using footnotes, and we will make comments to aid the reader’s understanding of the review when necessary. We will provide the NNTB or NNTH, absolute and relative per cent change in the 'What happens’ column of the SoF table as described in the ‘Measures of treatment effect’ section above, with the exception of the absolute difference for dichotomous outcomes which is displayed by default in the GRADEPro GDT view.

For NRSI assessed with ROBINS‐I, we will perform GRADE ratings by starting at high certainty and downgrade as appropriate; for single‐arm studies, or other observational studies where the use of ROBINS‐I would not be appropriate, we will start at low certainty as per the guidance from the Cochrane Handbook for Systematic Reviews of Interventions (Schünemann 2021a).