https://orcid.org/0000-0002-5213-4364
Figures
Abstract
The time-varying reproduction number (Rt) is an important measure of transmissibility during outbreaks. Estimating whether and how rapidly an outbreak is growing (Rt > 1) or declining (Rt < 1) can inform the design, monitoring and adjustment of control measures in real-time. We use a popular R package for Rt estimation, EpiEstim, as a case study to evaluate the contexts in which Rt estimation methods have been used and identify unmet needs which would enable broader applicability of these methods in real-time. A scoping review, complemented by a small EpiEstim user survey, highlight issues with the current approaches, including the quality of input incidence data, the inability to account for geographical factors, and other methodological issues. We summarise the methods and software developed to tackle the problems identified, but conclude that significant gaps remain which should be addressed to enable easier, more robust and applicable estimation of Rt during epidemics.
Author summary
Many software tools have been developed to support real-time outbreak response, allowing epidemiologists and other health professionals to characterise new and persistent pathogens and their threat to populations. Some of these tools allow the user to estimate how rapidly a pathogen is spreading through a population in real-time. This knowledge is crucial for a timely and effective response, helping to inform the optimal choice of control strategies. However, these tools are not perfect, and may need to be adapted to make them applicable to more pathogens and a variety of contexts. Our study summarises the different tools available in the literature and identifies the remaining issues that are yet to be addressed. We combine these findings with feedback from users of a popular tool, EpiEstim, to better understand the key priorities for development going forward.
Citation: Nash RK, Nouvellet P, Cori A (2022) Real-time estimation of the epidemic reproduction number: Scoping review of the applications and challenges. PLOS Digit Health 1(6): e0000052. https://doi.org/10.1371/journal.pdig.0000052
Editor: Michele Tizzoni, ISI Foundation: Fondazione ISI - Istituto per l’lnterscambio Scientifico, ITALY
Received: December 16, 2021; Accepted: April 27, 2022; Published: June 27, 2022
Copyright: © 2022 Nash et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The authors confirm that the data supporting the findings of this study are available within the article and its supplementary materials.
Funding: RKN acknowledges funding from the Medical Research Council (MRC) Doctoral Training Partnership (grant reference MR/N014103/1). AC acknowledges funding by the National Institute for Health Research (NIHR) Health Protection Research Unit in Modelling and Health Economics, a partnership between Public Health England, Imperial College London and LSHTM (grant code NIHR200908); and acknowledges funding from the MRC Centre for Global Infectious Disease Analysis (reference MR/R015600/1), jointly funded by the UK Medical Research Council (MRC) and the UK Foreign, Commonwealth & Development Office (FCDO), under the MRC/FCDO Concordat agreement and is also part of the EDCTP2 programme supported by the European Union; and acknowledges funding by Community Jameel. The views expressed are those of the author(s) and not necessarily those of the NIHR, Public Health England or the Department of Health and Social Care. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: I have read the journal’s policy and the authors of this manuscript have the following competing interests: AC has received payment from Pfizer for lecturing on a course for mathematical modelling of infectious disease transmission and vaccination.
Introduction
Transmissibility quantifies how easily a pathogen can spread through a population and can depend on numerous factors, such as the pathogenicity of the infectious agent but also the current level of immunity, demographics, and connectivity in the population. Transmissibility is typically measured by the time-varying (or effective) reproduction number (Rt), which represents the average number of secondary infections generated by a case at time t of an outbreak.[1–4] Rt is an important element of outbreak analysis as it indicates whether case numbers are rising (Rt > 1) or falling (Rt < 1) and by how much. This can guide intervention planning and help to determine whether current control measures are effective, and if not, to what extent they need to be intensified. Rt estimates can also feed into real-time incidence forecasts, which can assist logistics and resource planning, for instance, by identifying whether hospital bed capacity is likely to be exceeded or if resources need to be allocated to specific areas.[5–8]
Many methods have been developed to estimate Rt in real-time; one of the most commonly used is through the renewal equation (Eq 1), which relies on a branching process model. The model assumes that the incidence of new cases on day t (It) can be represented by a Poisson process:
(1)
where Rt is the time-varying reproduction number (i.e. the average number of cases caused by a primary case infected at time t, assuming that conditions remain the same after time t), and the past incidence (It-s) is weighted by ωs, the probability mass function of the generation time (the time between infection in a case and their infector).[3, 9] In practice, as infection itself is difficult to observe, the incidence of symptomatic cases can be used instead and ωs can be approximated by the serial interval (SI, the time between symptom onset in a case and their infector. See Britton and Scalia Tomba for the implications of this approximation).[10] Cori et al. developed a method to estimate Rt from the renewal equation that is suitable for real-time application.[11] The method is implemented in the R package EpiEstim, where estimation is performed over user-defined time windows within which Rt is assumed constant. Longer windows typically lead to smoother estimates, but may artificially hide some of the temporal variability.[12]
Rt estimates can then be used to project forward in time and forecast future epidemic trajectories (using Eq 1, see Fig 1), as implemented in the R package projections.[13]
In the renewal equation, the incidence at time t (It) is expressed as a function of the serial interval distribution (ωs), the time-varying reproduction number (Rt) and the past incidence (It-s).
The COVID-19 pandemic has generated a surge in methodological and tool developments to estimate Rt, as well as numerous applications of these methods. We aimed to describe the landscape of methods and real-world applications of Rt estimation, and to identify potential gaps that could be addressed to make those methods more easily applicable and more useful in practice in the future.
EpiEstim has been widely used for Rt estimation during the COVID-19 pandemic and has been shown to perform better than many other methods in terms of estimation accuracy.[12] Therefore, we use EpiEstim as a starting point to identify ways in which renewal equation methods have been used and modified.[12] We reviewed research articles that either used EpiEstim (the method or the software) or described a modified approach to address unmet needs. Alongside this scoping literature review, a questionnaire was distributed to known users of EpiEstim and advertised on social media to gather information on issues that were unlikely to be described in publications, e.g. computational speed or usability. By collating the findings, we aim to reveal some of the key challenges when estimating Rt in real-time and those that are yet to be addressed.
Methods
Scoping review: Search strategy and selection criteria
Google scholar was used to identify all articles or reports up to 10th December 2020 that cited one of the two papers describing EpiEstim (Cori et al. 2013 or Thompson et al. 2019).[3, 14] After full text screening, two databases were compiled using the inclusion and exclusion criteria outlined in Table 1, collating papers or reports that used: i) an unmodified version of the EpiEstim method or software and ii) a modified version of the EpiEstim method or software.
Evaluation of R package or tool usability
Each R package or tool identified in the scoping review was appraised in terms of its ease of installation, available documentation (e.g. vignette or tutorials) and speed of estimation of the reproduction number (see Table B and section 3 in S1 Text for more detail).
Questionnaire
We designed an online questionnaire including 16 questions aiming to reveal the key challenges users encountered when using EpiEstim to estimate Rt. It was shared via Twitter and distributed to a list of 41 known EpiEstim users.[15] The questionnaire was live between 11th and 31st January 2021.
Results
281 papers were included, which cited the original Cori et al. 2013 paper or the paper by Thompson et al. 2019 describing the latest version of EpiEstim (Fig 2).[3, 14] Papers investigated a variety of diseases; the most common was COVID-19, followed by Ebola and influenza (Figure A in S1 Text). The majority (n = 242) had used an unmodified version of EpiEstim (Table A in S1 Text), and 54 papers described at least one modification.[7, 16–67] Fifteen papers fell into both categories i.e., used the original EpiEstim method as a baseline to compare a new approach to.[16–30]
Note: fifteen papers are counted as both unmodified and modified (i.e., EpiEstim was used as a baseline to compare a modified approach to).
We classified the 106 issues addressed in the 54 papers proposing modifications into 13 categories (Fig 3). The most common included delays in the reporting of cases or missing data (n = 20), weekly administrative noise (n = 11), choice of prior for Rt (n = 15), and modelling different regions at the same time (n = 13). Each modification type was further categorised into four themes: 1) incidence data, 2) other input data or method modification 3) extension to account for geographical or spatial factors, and 4) practical or logistical issues.
Papers which made multiple modifications are counted more than once. Asterisks are present next to the modification category if an R package or opensource tool has been identified that addresses an issue within that category. See text for a description of all modifications and Table 2 for a summary of the R packages identified.
The ticks (✓) indicate whether an R package includes that modification type and rows that are grey show the modification types that have not been incorporated into a known R package or tool. The final three rows summarise additional exploration by the authors to assess how easily each package/tool can be installed and used. For full details of how each classification (very good = ✓✓, good = ✓, poor = ✗) was defined, see Table B in S1 Text and the associated text (section 3).
Modifications involving incidence data
The accuracy of Rt estimates directly depend on the quality of the input data, particularly the incidence data, which can be influenced by several factors. Incidence by date of infection is best suited to measure real-time transmission, but it is rarely available. A common workaround is to use incidence by date of symptom onset, resulting in “delayed” Rt estimates, representing transmission one incubation period before time t.[3] There are often additional delays between symptom onset and subsequent reporting of cases, leading to right censoring when observing cases in real time.[3] Recent incidence observed in real-time is therefore often underestimated, which can bias Rt estimates if not accounted for. This is further complicated when reporting delays vary over time, e.g. at the start of an outbreak when robust reporting systems are not yet in place. Beyond issues related to delays in reporting, some cases may not be reported at all; however this should not bias Rt estimates, unless reporting varies over short time scales e.g. if testing capacity fluctuates.[3]
Thirty percent of modifications (n = 32) addressed issues with incidence data. The most common was to account for reporting delays or missing data (i.e. underreporting or cases with missing date information) (n = 20).
The R packages bayEStim, EpiNow, and EpiNow2 (which is also used to generate Rt estimates for routine forecasting on the website EpiForecasts [52, 54]) all allow back-calculation of the incidence of symptom onset or infection from the incidence of reported cases using a reporting delay or incubation period distribution.[18, 25–27, 31, 47, 52, 54, 55, 68, 69] Such delays can be informed by individual case timelines or by characterising the delay after which reported incidence stabilises. Huisman et al. deconvolve over the observed incidence time series with a delay distribution that is specific to the type of case detection (e.g. positive test, hospital admission, death).[20] Li et al. use region and time dependent delay distributions.[23]
Some papers adjusted their approach to explicitly account for underreporting.[20, 23, 30, 41, 42] The R package Epidemia does so through a time-varying reporting rate.[37, 46, 70] A commonly adopted approach to circumvent time-varying case reporting is to use the incidence of deaths, as reporting of deaths is usually less prone to time fluctuations than reporting of cases. Nouvellet et al. and Djafaara et al. used COVID-19 death data to estimate Rt directly using the renewal equation.[17, 59] Others have estimated Rt from the incidence of infection or symptom onset, back-calculated from the observed incidence of death.[22, 44] Epidemia and EpiNow2 use a similar approach, and estimates can be based on incidence by date of infection, hospital admission or death.[18, 25–27, 31, 37, 46, 47, 52, 54]
The second most common modification was adjusting for weekly administrative noise (n = 11), where reporting can vary depending on the day of the week e.g. due to fewer tests being performed or longer processing times on the weekends.[71] As the default, EpiEstim estimates Rt over weekly time windows to smooth out these temporal trends. Modifications (including in EpiNow2) have been made to account for weekly noise in the incidence data by explicitly including a day of the week effect.[18, 25–27, 31, 34, 35, 47, 52, 54, 56]
One paper adjusted the temporal resolution of incidence data. Currently, EpiEstim requires the incidence data and the SI distribution to be on the same time scale. For acute infections, where the SI is short, this means weekly incidence is not supported and daily incidence is required as an input. Yamauchi et al. use EpiEstim on daily incidence inferred from weekly observations using smoothing splines.[66]
Other input data or method modification
As the default, EpiEstim uses a gamma distributed prior for Rt, with a mean and standard deviation of 5. Such a high mean value avoids estimating a subcritical reproduction number (Rt < 1) unless the data supports it, but can lead to overestimation of Rt in periods with low incidence. Priors for Rt are fixed across all time windows, but the length of the time window, which can be varied, determines the smoothness of the resulting estimates. Many methods have addressed the problem of temporal smoothness using alternative, often autoregressive, prior specifications.[18, 24–28, 30, 31, 37, 46, 47, 52, 54, 63, 64] Turbé et al. updated the gamma distributed prior for each time window with the previous posterior, so that estimates benefit from the information that becomes available over time.[28] Other methods, including those implemented in Epidemia, EpiNow2, and another opensource tool, EpiFilter, can control the temporal variation in Rt using random walks, such as gaussian processes.[18, 24–27, 30, 31, 37, 46, 47, 52, 54, 63, 64, 72] In Epidemia, in addition to being smoothed using a random walk, Rt can also be constrained by covariates such as mobility, and these covariates themselves are given priors.[37, 46]
Overall, fifteen papers described modifications related to the temporal smoothing of successive Rt estimates, which in EpiEstim involves a subjective time window choice over which each Rt is estimated. Choosing a time window length is a trade-off between identifying important temporal trends and smoothing out random noise in the data. Pathogens with short SIs and/or in the context of fast-changing transmission (e.g. due to control measures) will require shorter time windows to capture the former, while in low incidence settings, longer time windows may be needed to encompass more data. Four papers use metrics (either the Accumulated Prediction Error, implemented in the R package called APEestim, or the Fisher Information Approximation) to identify the optimal time window.[19, 24, 60, 61, 73] Instead of relying on a time window, methods which can control temporal variation using random walks can incorporate parameters which control the autocorrelation of successive Rt estimates.[18, 24–27, 30, 31, 47, 52, 54, 63, 64]
Six papers addressed issues with Rt estimation when incidence is low, where EpiEstim would typically return the prior.[21, 24, 43, 49, 50, 63] The R package earlyR is the frequentist equivalent of EpiEstim, and therefore returns estimates that are only data-driven, but it is not set up to estimate time-varying reproduction numbers.[21, 43, 49, 50, 74] EpiFilter can use data before and beyond time t to estimate Rt, which means that in addition to real-time estimation it can facilitate more robust retrospective estimation, particularly in low incidence settings.[24, 63]
In EpiEstim, the SI distribution can be specified as parametric (where only the mean and standard deviation are required), non-parametric (where the whole discrete distribution is provided), or “uncertain”, where a range of potential SIs are considered. In 2019, Thompson et al. extended the framework so that the SI distribution can be directly estimated within EpiEstim from data on known pairs of primary and secondary cases.[14] Five papers propose alternative approaches to derive the SI that they subsequently use in EpiEstim or another R package.[33, 43, 45, 51, 67] Al-Wahaibi et al. used an alternative R package, Epicontacts, to estimate the SI distribution directly from contact tracing data.[33, 75] Donnat and Holmes effectively assume a linearly decreasing probability mass function for the SI with parameters optimised based on past incidence data.[43] Knight et al. explored a method to permit distributions that allow negative SI values, which occur when transmission can take place before symptom onset as e.g. for COVID-19.[51] For applicability to vector-borne disease, some approaches modelled the SI as a distribution for both the human and vector.[45, 67]
Less common methodological modifications included aiming to improve the quantification of uncertainty using a Bayesian filtering method [39] or jointly estimating the reproduction number and the early incidence to account for highly variable reporting over time.[4, 7]
Modifications involving geographical factors
The simple renewal equation detailed in Eq 1 describes transmission in a single closed location. Thirteen papers developed or used methods that allow Rt to be estimated simultaneously for different locations. EpiNow2 simply adds the functionality to supply incidence for multiple locations at once and runs the same model to estimate Rt for each region in turn, producing a joint output facilitating comparisons between locations.[18, 25–27, 31, 47, 52, 54] Abry et al.’s approach enables simultaneous estimation of Rt from incidence time series in different locations, either independently or jointly using spatial smoothing.[32] Some approaches model transmission in different regions using linear predictors (which can include environmental, demographic, or intervention related factors), as in Epidemia, where parameters can be fixed or varying by location.[37, 46, 56, 64]
Six papers further modified the approach to explicitly account for connectivity.[36, 40, 53, 58, 59, 66] To account for connectivity between individuals or locations, Ng et al. weighted the probability of transmission between two cases depending on the distance between them, [58] while other studies inferred connectivity between locations using a gravity model approach.[36, 40] Chirombo et al. developed an estimator for a spatiotemporal R that accounts for susceptibility and infectiousness levels in each location and connectivity between them.[40] Backer et al. identified areas with a key role in disease spread by combining estimates of location-specific Rt with estimated movements out of each location to other areas.[36]
Transmissibility may also be constrained by explicitly using mobility data.[53, 59, 66] Yamauchi et al. used a similar approach to Backer et al. but used census data to inform movement between jurisdictions.[36, 66] In a less specific way, several studies examined the extent to which reduction in the level of mobility due to social distancing led to reductions in Rt.[53, 59]
A simpler approach to estimate Rt for a location whilst accounting for cases arising elsewhere is to distinguish between imported and locally acquired cases, as proposed in the EpiEstim extension by Thompson et al.[14] Five other papers described similar modifications to account for imported cases or for an exogenous force of infection.[38, 48, 57, 63, 65] Bourhy et al. extended the renewal equation to incorporate the rate of introduction of rabies into a city.[38] Two papers estimated Rt separately for locally acquired versus overseas acquired cases in order to model the effects of different interventions targeted at each group.[48, 65]
Modifications with practical or logistical applications
Rt estimation is often used as the basis for practical decision making in response to changing transmissibility in real-time. For example, the implementation of interventions, such as social distancing measures, estimating the adequacy of current control measures, or deciding when to declare that an outbreak has ended.[16, 76–78] Despite the clear link between Rt estimation and policymaking, very few modifications (let alone R packages) have been made to directly answer important practical or logistical questions.
Modifications were made to estimate the impact of potential drivers of transmission on Rt estimates (n = 3). Wang et al. modelled the effects of interventions by incorporating additional parameters to explicitly account for the impact of interventions on the infectious period or the overall level of infectiousness of infected cases.[29] Epidemia accounts for extrinsic factors, such as interventions and changes in mobility, by creating a linear predictor which includes the factors influencing changes in transmission.[37, 46]
Three papers sought to develop probabilistic methods to predict local disease elimination or the risk of epidemic resurgence, one of which is implemented in EpiFilter.[16, 62, 63]
A summary of the R packages and software tool that incorporate some of the modifications identified through the literature review is available in Table 2. We also performed additional exploration to assess their usability and the speed of estimation of the reproduction number (when relevant examples were provided). This is a non-comprehensive assessment based on the R packages/tools available at the time of writing (see Table B and section 3 in S1 Text and the documentation of each package for more details). Two of the seven were very easy to install, and the majority (n = 5) provided documentation and vignettes. Of the R packages/tool that estimate the reproduction number, three were able to perform the estimation (using provided examples) in under a minute, however, only earlyR scored full marks in the usability category.
Questionnaire
The questionnaire had 17 responses from 11 countries (Figure B in S1 Text). The majority of respondents were academic researchers, public health consultants or health professionals (Figure C in S1 Text). The small sample size makes it difficult to statistically analyse the responses, but overall, the questionnaire respondents echoed the broad themes identified during the scoping review (Fig 3, Table C in S1 Text). In addition, three other themes were identified: usability, speed and compatibility (Table C in S1 Text).
The most common issue affecting 53% of questionnaire respondents was usability, e.g. difficulties in using the package (at least initially–see p5 in S1 Text). Respondents suggested links to more resources, such as R code examples, to demonstrate how to apply and interpret outputs of EpiEstim in multiple scenarios. Speed issues were reported by 18% of questionnaire respondents, all using the ‘uncertain_si’ feature, where Rt estimates are integrated over a range of serial interval distributions. Compatibility issues were reported by users wanting to use alternative programming languages, such as JAGS or Julia, or wanting to use EpiEstim in a workflow with other R packages and suggested the addition of information on how to do so (see section 4 in S1 Text for further details).
Discussion
To improve methods of Rt estimation, it is crucial to first gain an understanding of the current key challenges preventing, or restricting, practical applications of these methods to quantify pathogen transmissibility in real-time. To get an overview of such issues, we chose EpiEstim, one of the most popular and best-performing methods for estimating Rt in real-time, as a case study.[12] We analysed the literature citing EpiEstim as well as direct feedback from EpiEstim users collected through a questionnaire. We have also summarised the R packages identified that use alternative methodologies.
The most common challenge identified was to deal with factors influencing the quality of incidence data. Addressing frequently encountered issues, such as delays in reporting, underreporting, and administrative noise, appeared to be a priority in the field. Delays in reporting of cases or time-varying reporting rates have been a particular issue throughout the COVID-19 pandemic, linked to a high proportion of asymptomatic cases, fluctuations in testing, and health systems being under strain.[79–81] To counter some of these issues, R packages such as bayEStim and EpiNow2 explicitly account for delay distributions enabling “nowcasting”, which aims to eliminate any time-lag when estimating the impact of control measures or factors such as changes in behaviour.[68, 69]
When incidence by date of infection or symptom onset was unavailable or unreliable, some approaches used incidence by date of death. For instance, COVID-19 case data was initially very sporadic, whilst death data was more reliably reported. The R packages Epidemia and EpiNow2 allow the use of death data to maximise the use of available information in real-time.[69, 70] In many epidemics, including COVID-19, the reliability of case data improves over time and can provide more up-to-date insights into changing transmission dynamics. The ability to use either dataset means they can complement each other during different phases of the outbreak. However, the suitability of the type of data used will depend on the pathogen under investigation and the local context.
An issue that, to our knowledge, has yet to be addressed by any R package is the ability to estimate Rt using weekly-aggregated incidence data, which was highlighted in questionnaire responses and is a frequent query in correspondence with users of EpiEstim. This is a key ongoing issue, particularly for diseases such as influenza and Zika, where incidence is typically reported on a weekly basis, [45, 82] but also for COVID-19, with several US states having recently moved from reporting daily to weekly cases only.[83] While it is possible to supply EpiEstim and EpiNow2 with weekly-aggregated data if the SI is provided on the same time scale, aggregating a short SI to a weekly distribution may affect the quality of the Rt estimates and is not possible when the SI is shorter than a week.[45]
The scoping review identified additional methodological or data-related issues. These led to the development of methods that are more appropriate for use during periods of low incidence, [21, 24, 43, 49, 50, 63] or offer alternatives to allow less subjective and more flexible inputs into EpiEstim, e.g. the prior for Rt, the SI distribution, or an alternative way of temporally smoothing Rt estimates.[18, 19, 21, 24–28, 30, 31, 33, 37, 43, 45–47, 49–52, 54, 60, 61, 63, 64, 67]
Modifications concerning geographical factors were also common in the literature, although only mentioned by one questionnaire respondent. Beyond accounting for imported cases [14, 63] and estimating Rt across different regions at the same time, [18, 25, 27, 31, 37, 46, 47, 52, 54] we found no R packages that explicitly addressed spatial interactions. This means that, aside from bespoke studies, [36, 40, 53, 56, 58, 59, 64, 66] important aspects of the spatial structure in transmission may have been overlooked. For instance, to our knowledge, there is no ready-to-use tool to analyse how easing restrictions in an area may influence transmission in neighbouring locations.
Another theme highlighted in both the scoping review and questionnaire responses was practical or logistical issues, such as evaluating or accounting for the impact of interventions, assessing when elimination has been reached, or extending the framework for logistical planning in hospitals. Transmissibility estimates ultimately inform intervention design, e.g. to select interventions that appear most effective and identify locations which should be targeted for implementation. The ability to estimate the impact of interventions on Rt across different countries is a useful feature of Epidemia, which uses a logistic regression framework to infer the impact of non-pharmaceutical interventions (e.g. social distancing) and reductions in mobility on transmission.[70] Additionally, an important practical consideration is when one can confidently estimate that elimination has been reached, which has been addressed by EpiFilter.[24, 63, 72] More R packages that directly address a variety of recurring logistical issues is a promising avenue for future research in the field.
Despite the small sample size (n = 17) and the imperfect nature of the questionnaire (see section 4 of S1 Text), it reinforced many of the findings from the scoping review and revealed additional challenges with computational speed, usability, and compatibility. We evaluated each of the identified R packages and tool based on these criteria and found that, of the packages that estimate the reproduction number, only earlyR scored full marks in the usability category (Table 2, Table B in S1 Text). These themes emphasise that package developers (of both new and existing methods) should ensure that increased functionality does not come at the expense of speed, and that additional resources are available to aid understanding of the package and how it can be used in conjunction with others in an outbreak analysis workflow. In response, we created a new EpiEstim vignette, which intends to provide a greater range of examples of using EpiEstim in practice.[84] A suggestion to improve usability by providing a library of pre-built configurations for different pathogens has already begun to be addressed to some extent by the R package EpiNow2.[69]
This is a fast-moving field and novel extensions or applications of EpiEstim-like methods are continuously emerging. It is therefore likely that we have missed valuable modifications not covered by articles available at the time of our literature search. This includes an extension to the method by Johnson et al. to account for superspreading, and work by the authors of this study, for example, the development of multi-variant EpiEstim (MV-EpiEstim) to estimate the transmission advantage of new pathogen variants or strains in real-time.[85, 86] It is also possible that through using EpiEstim as a starting point for our literature search, we may have missed relevant papers that did not cite the method. However, given that EpiEstim has been recognised as the most reliable currently existing approach for real-time Rt estimation, [12] we believe it is unlikely that we would have missed important contributions that would alter our conclusions. To our knowledge, there are still numerous issues regarding real-time estimation of the reproduction number that remain unaddressed by R packages or opensource software. For instance, we found no readily available tools that consider non-spatial population heterogeneities or the ability to include time-varying generation times.[87, 88]
The intention of this review was to provide a broad overview of the current landscape of renewal-equation based Rt estimation methods and tools. Therefore, beyond providing an appraisal of the usability of each identified R package/tool in terms of ease of installation, available documentation, and speed, we have not tested or compared their performance. Future work should focus on a more systematic and critical evaluation of the estimation accuracy of R packages intending to tackle the same issues. Moreover, it would be interesting to characterise the added value of additional features in relation to their cost, such as longer computational time.
Conclusion
The quality of incidence data continues to pose significant challenges to real-time estimation of Rt. Numerous methods and R packages have been developed to address some of these issues, but significant gaps remain, such as the inability to directly use temporally aggregated incidence data. Despite the importance of spatial factors in transmission, no R package has been identified to account for movement or interactions between locations when estimating Rt. It is also clear that extensions to these methods could allow for rapid translation of Rt estimates into logistically relevant outputs, such as predicting how the demand for hospital resources may change in real-time. Addressing these recurring issues and extending the methodology to directly answer important practical and logistical questions are key priorities for widening the applicability of Rt estimation methods during epidemics. However, package developers should keep in mind that speed, ease of use, and access to sufficient resources, will be key to the uptake of these new or improved tools.
Supporting information
S1 Text.
Figure A. The disease or pathogen under investigation in A) the papers that used an unmodified version of the EpiEstim package or the method and B) the papers that used a modified version of the approach or package. The category “multiple” refers to papers where more than one disease or pathogen were investigated. Note the diseases are different in both panels. Table A. Summary table of the papers identified that used an unmodified version of EpiEstim (n = 242). Table B. Usability of each R package or tool* identified in the scoping review. This table shows a full breakdown of how the classifications (very good = ✓✓, good = ✓, poor = ✗) were determined for the “additional exploration” section of Table 2 within the main text. For the ‘ease of installation’ and ‘documentation and tutorials’ sections, each criterion was allocated a score, shown in squared brackets, and the overall classification was determined by the sum of the scores. For the ‘speed’ section, each author used the system.time() function in R to determine the run time of the main function of the package available in the provided examples. ** The classification (<10s = ✓✓, >10s – 5min = ✓, >5min = ✗) was decided based on the time category agreed on by at least 2 out of the 3 computers. Figure B. Map showing the country in which each questionnaire respondent is based. The majority of responses were from the USA (n = 4), followed by Canada (n = 2), France (n = 2) and Indonesia (n = 2). There was one response from each of Austria, Bermuda (circled), Germany, India, Peru, Uruguay, and the UK. Figure C. A) The profession of each questionnaire respondent and B) the purpose of their analysis. Respondents could select more than one answer for both questions. Figure D. A) Disease(s) investigated by each respondent. B) Categories of input data. Respondents could select more than one answer for both questions. Figure E. Broad reason for the use of EpiEstim. Respondents could select more than one answer for this question. Figure F. Questionnaire responses to A) how well the package met the needs of each respondent on a scale from 1 to 5 (1: “badly”, 5: “very well”), and B) “Which features do you think could be improved?”. Table C. Summary of the issues and suggestions reported in questionnaire feedback categorised by broad theme. n is the number of respondents and % is the percentage of the 17 respondents who reported or made a suggestion regarding the issue.
https://doi.org/10.1371/journal.pdig.0000052.s001
(PDF)
References
- 1. Fraser C, Donnelly CA, Cauchemez S, Hanage WP, Van Kerkhove MD, Hollingsworth TD, et al. Pandemic Potential of a Strain of Influenza A (H1N1): Early Findings. Science. 2009 Jun 19;324(5934):1557–61. pmid:19433588
- 2. Cauchemez S, Fraser C, Van Kerkhove MD, Donnelly CA, Riley S, Rambaut A, et al. Middle East respiratory syndrome coronavirus: quantification of the extent of the epidemic, surveillance biases, and transmissibility. Lancet Infect Dis. 2014 Jan;14(1):50–6. pmid:24239323
- 3. Cori A, Ferguson NM, Fraser C, Cauchemez S. A New Framework and Software to Estimate Time-Varying Reproduction Numbers During Epidemics. Am J Epidemiol. 2013 Nov 1;178(9):1505–12. pmid:24043437
- 4. Nouvellet P, Cori A, Garske T, Blake IM, Dorigatti I, Hinsley W, et al. A simple approach to measure transmissibility and forecast incidence. Epidemics. 2018 Mar;22:29–35. pmid:28351674
- 5. WHO Ebola Response Team. Ebola Virus Disease in West Africa—The First 9 Months of the Epidemic and Forward Projections. N Engl J Med. 2014 Oct 16;371(16):1481–95. pmid:25244186
- 6. WHO Ebola Response Team. West African Ebola Epidemic after One Year—Slowing but Not Yet under Control. N Engl J Med. 2015 Feb 5;372(6):584–7. pmid:25539446
- 7. The Ebola Outbreak Epidemiology Team, Barry A, Ahuka-Mundeke S, Ahmed YA, Allarangar Y, Anoko J, et al. Outbreak of Ebola virus disease in the Democratic Republic of the Congo, April–May, 2018: an epidemiological study. The Lancet. 2018 Jul 21;392(10143):213–21.
- 8. Finger F, Funk S, White K, Siddiqui MR, Edmunds WJ, Kucharski AJ. Real-time analysis of the diphtheria outbreak in forcibly displaced Myanmar nationals in Bangladesh. BMC Medicine. 2019 Mar 12;17(1):58. pmid:30857521
- 9. Fraser C. Estimating Individual and Household Reproduction Numbers in an Emerging Epidemic. PLoS One [Internet]. 2007 Aug 22 [cited 2021 Jun 4];2(8). Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1950082/
- 10. Britton T, Scalia Tomba G. Estimation in emerging epidemics: biases and remedies. Journal of The Royal Society Interface. 2019 Jan 31;16(150):20180670. pmid:30958162
- 11.
EpiEstim R package v2.2–3: A tool to estimate time varying instantaneous reproduction number during epidemics. (https://cran.r-project.org/web/packages/EpiEstim/index.html)
- 12. Gostic KM, McGough L, Baskerville EB, Abbott S, Joshi K, Tedijanto C, et al. Practical considerations for measuring the effective reproductive number, Rt. PLOS Computational Biology. 2020 Dec 10;16(12):e1008409. pmid:33301457
- 13.
Projections R package: Project future case incidence. (https://cran.rproject.org/web/packages/projections/projections.pdf)
- 14. Thompson RN, Stockwin JE, van Gaalen RD, Polonsky JA, Kamvar ZN, Demarsh PA, et al. Improved inference of time-varying reproduction numbers during infectious disease outbreaks. Epidemics. 2019 Dec 1;29:100356. pmid:31624039
- 15.
MRC Centre for Global Infectious Disease Analysis. [Internet] 11 Jan 2021 [Cited 24 May 2021] Available from: https://twitter.com/MRC_Outbreak/status/1348586478394339328
- 16. Djaafara BA, Imai N, Hamblion E, Impouma B, Donnelly CA, Cori A. A Quantitative Framework for Defining the End of an Infectious Disease Outbreak: Application to Ebola Virus Disease. American Journal of Epidemiology. 2021 Apr 6;190(4):642–51. pmid:33511390
- 17. Djaafara BA, Whittaker C, Watson OJ, Verity R, Brazeau NF, Widyastuti W, et al. Quantifying the dynamics of COVID-19 burden and impact of interventions in Java, Indonesia. medRxiv. 2021 Feb 15;2020.10.02.20198663.
- 18. Esra R, Jamieson L, Fox MP, Letswalo D, Ngcobo N, Mngadi S, et al. Evaluating the impact of non-pharmaceutical interventions for SARS-CoV-2 on a global scale. medRxiv. 2020 Aug 5;2020.07.30.20164939.
- 19. Garcia LP, Traebert J, Boing AC, Santos GFZ, Pedebôs LA, d’Orsi E, et al. The potential spread of Covid-19 and government decision-making: a retrospective analysis in Florianópolis, Brazil. Rev bras epidemiol [Internet]. 2020 Sep 30 [cited 2021 Jun 4];23. Available from: https://www.scielo.br/j/rbepid/a/WJLGyZfwFkfPGVLMVW5y8ch/?lang=en pmid:33027433
- 20. Huisman JS, Scire J, Angst DC, Neher RA, Bonhoeffer S, Stadler T. Estimation and worldwide monitoring of the effective reproductive number of SARS-CoV-2. medRxiv. 2020 Nov 30;2020.11.26.20239368.
- 21. Khosravi A, Chaman R, Rohani-Rasaf M, Zare F, Mehravaran S, Emamian MH. The basic reproduction number and prediction of the epidemic size of the novel coronavirus (COVID-19) in Shahroud, Iran. Epidemiology & Infection [Internet]. 2020 ed [cited 2021 Jun 4];148. Available from: https://www.cambridge.org/core/journals/epidemiology-and-infection/article/basic-reproduction-number-and-prediction-of-the-epidemic-size-of-the-novel-coronavirus-covid19-in-shahroud-iran/B9ED8A0969C0F36DCC72E4ABB45CCC19
- 22. Li C, Romagnani P, Anders H-J. Novel Criteria for When and How to Exit a COVID-19 Pandemic Lockdown. Front Big Data [Internet]. 2020 [cited 2021 Jun 4];3. Available from: https://www.frontiersin.org/articles/10.3389/fdata.2020.00026/full
- 23. Li Y, Undurraga EA, Zubizarreta JR. Effectiveness of Localized Lockdowns in the COVID-19 Pandemic. medRxiv. 2021 Mar 30;2020.08.25.20182071.
- 24. Parag KV. Improved estimation of time-varying reproduction numbers at low case incidence and between epidemic waves. medRxiv. 2021 May 19;2020.09.14.20194589. pmid:34492011
- 25.
Price DJ, Shearer FM, Meehan M, McBryde E, Golding N, McVernon J, et al. Estimating the case detection rate and temporal variation in transmission of COVID-19 in Australia: Technical Report 14th April 2020 [Internet]. Doherty Institute; 2020 Apr p. 11. Available from: https://www.doherty.edu.au/uploads/content_doc/Estimating_changes_in_the_transmission_of_COVID-19_April14-public-release.pdf
- 26. Salas J. Improving the estimation of the COVID-19 effective reproduction number using nowcasting. Stat Methods Med Res. 2021 Sep;30(9):2075–84. pmid:33951962
- 27. Sherratt K, Abbott S, Meakin SR, Hellewell J, Munday JD, Bosse N, et al. Exploring surveillance data biases when estimating the reproduction number: with insights into subpopulation transmission of Covid-19 in England. medRxiv. 2021 Mar 18;2020.10.18.20214585. pmid:34053260
- 28. Turbé H, Bjelogrlic M, Robert A, Gaudet-Blavignac C, Goldman J-P, Lovis C. Adaptive Time-Dependent Priors and Bayesian Inference to Evaluate SARS-CoV-2 Public Health Measures Validated on 31 Countries. Front Public Health [Internet]. 2021 [cited 2021 Jun 4];8. Available from: https://www.frontiersin.org/articles/10.3389/fpubh.2020.583401/full pmid:33553088
- 29. Wang S, Yang X, Li L, Nadler P, Arcucci R, Huang Y, et al. A Bayesian Updating Scheme for Pandemics: Estimating the Infection Dynamics of COVID-19. IEEE Computational Intelligence Magazine. 2020 Nov;15(4):23–33.
- 30. Wilder B, Mina MJ, Tambe M. Tracking disease outbreaks from sparse data with Bayesian inference. arXiv:200905863 [cs, stat] [Internet]. 2020 Sep 12 [cited 2021 Jun 4]; Available from: http://arxiv.org/abs/2009.05863
- 31. Abbott S, Hellewell J, Thompson RN, Sherratt K, Gibbs HP, Bosse NI, et al. Estimating the time-varying reproduction number of SARS-CoV-2 using national and subnational case counts. Wellcome Open Res. 2020 Dec 8;5:112.
- 32. Abry P, Pustelnik N, Roux S, Jensen P, Flandrin P, Gribonval R, et al. Spatial and temporal regularization to estimate COVID-19 reproduction number R(t): Promoting piecewise smoothness via convex optimization. PLOS ONE. 2020 Aug 20;15(8):e0237901. pmid:32817697
- 33. Al-Wahaibi A, Al Manji A, Al Maani A, Al Rawahi B, Al Harthy K, Alyaquobi F, et al. COVID-19 epidemic monitoring after non-pharmaceutical interventions: The use of time-varying reproduction number in a country with a large migrant population. Int J Infect Dis. 2020 Oct;99:466–72. pmid:32829052
- 34. Alvarez L, Colom M, Morel J-D, Morel J-M. Computing the daily reproduction number of COVID-19 by inverting the renewal equation. medRxiv. 2021 Apr 21;2020.08.01.20165142. pmid:34876517
- 35. Alvarez L, Colom M, Morel J-M. Removing weekly administrative noise in the daily count of COVID-19 new cases. Application to the computation of Rt. medRxiv. 2020 Nov 18;2020.11.16.20232405.
- 36. Backer JA, Wallinga J. Spatiotemporal Analysis of the 2014 Ebola Epidemic in West Africa. PLOS Computational Biology. 2016 Dec 8;12(12):e1005210. pmid:27930675
- 37. Bhatt S, Ferguson N, Flaxman S, Gandy A, Mishra S, Scott JA. Semi-Mechanistic Bayesian Modeling of COVID-19 with Renewal Processes. arXiv:201200394 [stat] [Internet]. 2020 Dec 29 [cited 2021 Jun 4]; Available from: http://arxiv.org/abs/2012.00394
- 38. Bourhy H, Nakouné E, Hall M, Nouvellet P, Lepelletier A, Talbi C, et al. Revealing the Micro-scale Signature of Endemic Zoonotic Disease Transmission in an African Urban Setting. PLOS Pathogens. 2016 Apr 8;12(4):e1005525. pmid:27058957
- 39. Capistrán MA, Capella A, Christen JA. Filtering and improved Uncertainty Quantification in the dynamic estimation of effective reproduction numbers. arXiv:201202168 [stat] [Internet]. 2020 Dec 3 [cited 2021 Jun 4]; Available from: http://arxiv.org/abs/2012.02168
- 40.
Chirombo JJ. Modelling spatial processes of infectious diseases [Internet]. Faculty of Health and Medicine, Lancaster University; 2018. Available from: https://core.ac.uk/download/pdf/211246237.pdf#page=55
- 41. Cowling BJ, Ali ST, Ng TWY, Tsang TK, Li JCM, Fong MW, et al. Impact assessment of non-pharmaceutical interventions against coronavirus disease 2019 and influenza in Hong Kong: an observational study. The Lancet Public Health. 2020 May 1;5(5):e279–88. pmid:32311320
- 42. De Simone A, Piangerelli M. A Bayesian approach for monitoring epidemics in presence of undetected cases. Chaos, Solitons & Fractals. 2020 Nov 1;140:110167. pmid:32868967
- 43. Donnat C, Holmes S. Modeling the heterogeneity in COVID-19’s reproductive number and its impact on predictive scenarios. Journal of Applied Statistics. 2021 Jun 22;0(0):1–29.
- 44. Evans CM, Egan JR, Hall I. Pneumonic Plague in Johannesburg, South Africa, 1904—Volume 24, Number 1—January 2018—Emerging Infectious Diseases journal—CDC. Emerging Infectious Diseases [Internet]. 2018 Jan [cited 2021 Jun 4];24(1). Available from: https://wwwnc.cdc.gov/eid/article/24/1/16-1817_article
- 45. Ferguson NM, Cucunubá ZM, Dorigatti I, Nedjati-Gilani GL, Donnelly CA, Basáñez M-G, et al. Countering the Zika epidemic in Latin America. Science. 2016 Jul 22;353(6297):353–4. pmid:27417493
- 46. Flaxman S, Mishra S, Gandy A, Unwin HJT, Mellan TA, Coupland H, et al. Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe. Nature. 2020 Aug;584(7820):257–61. pmid:32512579
- 47.
Golding N, Price DJ, Shearer FM, Moss R, Meehan MT, McBryde E, et al. Estimating temporal variation in transmission of COVID-19 and adherence to social distancing measures in Australia: Technical Report 5 May 2020 [Internet]. Doherty Institute; 2020 May p. 19. Available from: https://www.doherty.edu.au/uploads/content_doc/Estimating_changes_in_the_transmission_of_COVID-19_May5-public-release1.pdf
- 48.
Golding N, Shearer FM, Moss R, Dawson P, Gibbs L, Alisic E, et al. Estimating temporal variation in transmission of COVID-19 and adherence to social distancing measures in Australia: Technical Report 15 May 2020 [Internet]. Doherty Institute; 2020 May p. 46. Available from: https://www.doherty.edu.au/uploads/content_doc/Technical_report_15_Maypdf.pdf
- 49. Kanagarathinam K, Algehyne EA, Sekar K. Analysis of ‘earlyR’ epidemic model and time series model for prediction of COVID-19 registered cases. Mater Today Proc [Internet]. 2020 Oct 14 [cited 2021 Jun 4]; Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7556808/ pmid:33078097
- 50. Kanagarathinam K, Sekar K. Estimation of the reproduction number and early prediction of the COVID-19 outbreak in India using a statistical computing approach. Epidemiol Health [Internet]. 2020 May 9 [cited 2021 Jun 4];42. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7644936/ pmid:32512670
- 51. Knight J, Mishra S. Estimating effective reproduction number using generation time versus serial interval, with application to covid-19 in the Greater Toronto Area, Canada. Infectious Disease Modelling. 2020 Jan 1;5:889–96. pmid:33163739
- 52. Li Y, Campbell H, Kulkarni D, Harpur A, Nundy M, Wang X, et al. The temporal association of introducing and lifting non-pharmaceutical interventions with the time-varying reproduction number (R) of SARS-CoV-2: a modelling study across 131 countries. The Lancet Infectious Diseases. 2021 Feb 1;21(2):193–202. pmid:33729915
- 53. Lim JT, Dickens BSL, Choo ELW, Chew LZX, Koo JRH, Tam C, et al. Revealing regional disparities in the transmission potential of SARS-CoV-2 from interventions in Southeast Asia. Proceedings of the Royal Society B: Biological Sciences. 2020 Aug 26;287(1933):20201173. pmid:32842911
- 54. Liu Y, Morgenstern C, Kelly J, Lowe R, Munday J, Villabona-Arenas CJ, et al. The impact of non-pharmaceutical interventions on SARS-CoV-2 transmission across 130 countries and territories. BMC Medicine. 2021 Feb 5;19(1):40. pmid:33541353
- 55. Lytras T, Sypsa V, Panagiotakos D, Tsiodras S. An improved method to estimate the effective reproduction number of the COVID-19 pandemic: lessons from its application in Greece. medRxiv. 2020 Sep 22;2020.09.19.20198028.
- 56. Martinez-Beneito MA, Mateu J, Botella-Rocamora P. Spatio-temporal small area surveillance of the Covid-19 pandemics. arXiv:201103938 [stat] [Internet]. 2020 Nov 8 [cited 2021 Jun 4]; Available from: http://arxiv.org/abs/2011.03938
- 57. Mishra S, Berah T, Mellan TA, Unwin HJT, Vollmer MA, Parag KV, et al. On the derivation of the renewal equation from an age-dependent branching process: an epidemic modelling perspective. arXiv:200616487 [q-bio, stat] [Internet]. 2020 Jun 29 [cited 2021 Jun 4]; Available from: http://arxiv.org/abs/2006.16487
- 58. Ng T-C, Wen T-H. Spatially Adjusted Time-varying Reproductive Numbers: Understanding the Geographical Expansion of Urban Dengue Outbreaks. Sci Rep. 2019 Dec 16;9(1):19172. pmid:31844099
- 59. Nouvellet P, Bhatia S, Cori A, Ainslie KEC, Baguelin M, Bhatt S, et al. Reduction in mobility and COVID-19 transmission. Nat Commun. 2021 Feb 17;12(1):1090. pmid:33597546
- 60. Parag KV, Donnelly CA. Adaptive Estimation for Epidemic Renewal and Phylogenetic Skyline Models. Systematic Biology. 2020 Nov 1;69(6):1163–79. pmid:32333789
- 61. Parag KV, Cowling BJ, Donnelly CA. Deciphering early-warning signals of the elimination and resurgence potential of SARS-CoV-2 from limited data at multiple scales. medRxiv. 2021 Jan 5;2020.11.23.20236968. pmid:34905965
- 62. Parag KV, Donnelly CA. Using information theory to optimise epidemic models for real-time prediction and estimation. PLOS Computational Biology. 2020 Jul 1;16(7):e1007990. pmid:32609732
- 63. Parag KV, Donnelly CA, Jha R, Thompson RN. An exact method for quantifying the reliability of end-of-epidemic declarations in real time. PLOS Computational Biology. 2020 Nov 30;16(11):e1008478. pmid:33253158
- 64. Rotejanaprasert C, Lawpoolsri S, Pan-ngum W, Maude RJ. Preliminary estimation of temporal and spatiotemporal dynamic measures of COVID-19 transmission in Thailand. PLOS ONE. 2020 Sep 24;15(9):e0239645. pmid:32970773
- 65. Wu P, Tsang TK, Wong JY, Ng TW, Ho F, Gao H, et al. Suppressing COVID-19 transmission in Hong Kong: an observational study of the first four months. Research Square [Pre-print] [Internet]. 2020 Jun 9 [cited 2021 Jun 4]; Available from: https://www.researchsquare.com/article/rs-34047/v1
- 66. Yamauchi T, Takeuchi S, Yamano Y, Kuroda Y, Nakadate T. Estimation of the effective reproduction number of influenza based on weekly reports in Miyazaki Prefecture. Sci Rep. 2019 Feb 22;9(1):2539. pmid:30796315
- 67. Zhao S, Musa SS, Meng J, Qin J, He D. The long-term changing dynamics of dengue infectivity in Guangdong, China, from 2008–2018: a modelling analysis. Transactions of The Royal Society of Tropical Medicine and Hygiene. 2020 Jan 6;114(1):62–71. pmid:31638154
- 68. Lytras T. Estimate epidemic effective reproduction number in a Bayesian framework [R package bayEStim version 0.0.1] [Internet]. Available from: https://github.com/thlytras/bayEStim
- 69. Abbott S, Hellewell J, Sherratt K, Gostic K, Hickson J, et al. EpiNow2: Estimate Real-Time Case Counts and Time-Varying Epidemiological Parameters [Internet]. 2020 [cited 2021 Jun 7]. Available from: https://CRAN.R-project.org/package=EpiNow2
- 70.
Epidemia [R package version 1.0.0] [Internet]. Imperial College London; 2021 [cited 2021 Jun 7]. Available from: https://github.com/ImperialCollegeLondon/epidemia
- 71.
UK Government. Coronavirus in the UK—About the data [Internet]. [cited 2021 Sep 22]. Available from: https://coronavirus.data.gov.uk/details/about-data
- 72. Parag K. kpzoo/EpiFilter [Internet]. 2021 [cited 2021 Jun 7]. Available from: https://github.com/kpzoo/EpiFilter
- 73. Parag K. kpzoo/APEestim [Internet]. 2020 [cited 2021 Jun 7]. Available from: https://github.com/kpzoo/APEestim
- 74. Jombart T, Cori A, Nouvellet P, Skarp J, Kamvar ZN, Taylor T. earlyR: Estimation of Transmissibility in the Early Stages of a Disease Outbreak [Internet]. 2020 [cited 2021 Jun 7]. Available from: https://CRAN.R-project.org/package=earlyR
- 75. Campbell F, Jombart T, Randhawa N, Sudre B, Nagraj VP, Crellen T, et al. epicontacts: Handling, Visualisation and Analysis of Epidemiological Contacts [Internet]. 2021 [cited 2021 Jun 7]. Available from: https://CRAN.R-project.org/package=epicontacts
- 76.
The R value and growth rate [Internet]. GOV.UK. [cited 2022 Feb 21]. Available from: https://www.gov.uk/guidance/the-r-value-and-growth-rate
- 77. Ferguson NM, Cummings DAT, Cauchemez S, Fraser C, Riley S, Meeyai A, et al. Strategies for containing an emerging influenza pandemic in Southeast Asia. Nature. 2005 Sep;437(7056):209–14. pmid:16079797
- 78. Pourbohloul B, Meyers LA, Skowronski DM, Krajden M, Patrick DM, Brunham RC. Modeling Control Strategies of Respiratory Pathogens. Emerg Infect Dis. 2005 Aug;11(8):1249–56. pmid:16102315
- 79. Buitrago-Garcia D, Egli-Gany D, Counotte MJ, Hossmann S, Imeri H, Ipekci AM, et al. Occurrence and transmission potential of asymptomatic and presymptomatic SARS-CoV-2 infections: A living systematic review and meta-analysis. PLoS Med. 2020 Sep;17(9):e1003346. pmid:32960881
- 80. Byambasuren O, Cardona M, Bell K, Clark J, McLaws M-L, Glasziou P. Estimating the extent of asymptomatic COVID-19 and its potential for community transmission: Systematic review and meta-analysis. Official Journal of the Association of Medical Microbiology and Infectious Disease Canada. 2020 Dec 1;5(4):223–34.
- 81.
Coronavirus (COVID-19) Testing—Statistics and Research [Internet]. Our World in Data. [cited 2021 Jun 7]. Available from: https://ourworldindata.org/coronavirus-testing
- 82.
WHO | FluNet [Internet]. WHO. World Health Organization; [cited 2021 Jun 7]. Available from: http://www.who.int/influenza/gisrs_laboratory/flunet/en/
- 83.
COVID-19 Data Repository by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University. [Internet] Johns Hopkins University. [cited 2021 Oct 15] Available from: https://github.com/CSSEGISandData/COVID-19
- 84.
EpiEstim R package vignette. [Internet] [cited 2021 Oct 15] Available from: https://github.com/mrc-ide/EpiEstim/blob/master/vignettes/full_EpiEstim_vignette.Rmd
- 85. Johnson KD, Beiglböck M, Eder M, Grass A, Hermisson J, Pammer G, et al. Disease momentum: Estimating the reproduction number in the presence of superspreading. Infectious Disease Modelling. 2021 Jan 1;6:706–28. pmid:33824936
- 86. Bhatia S, Wardle J, Nash RK, Nouvellet P, Cori A. A generic method and software to estimate the transmission advantage of pathogen variants in real-time: SARS-CoV-2 as a case-study [Internet]. medRxiv; 2021 [cited 2022 Feb 21]. p. 2021.11.26.21266899. Available from: https://www.medrxiv.org/content/10.1101/2021.11.26.21266899v1
- 87. Green W, Ferguson N, Cori A. (In press) Inferring the reproduction number using the renewal equation in heterogenous epidemics. The Royal Society.
- 88. Griffin J, Casey M, Collins Á, Hunt K, McEvoy D, Byrne A, et al. Rapid review of available evidence on the serial interval and generation time of COVID-19. BMJ Open. 2020 Nov 1;10(11):e040263. pmid:33234640