https://orcid.org/0000-0002-4066-873X
Figures
Abstract
We aimed to review the validity of existing prediction models for cardiovascular diseases (CVDs) in Asia. In this systematic review and meta-analysis, we included studies that validated prediction models for CVD risk in the general population in Asia. Various databases, including PubMed, Web of Science conference proceedings citation index, Scopus, Global Index Medicus of the World Health Organization (WHO), and Open Access Thesis and Dissertations (OATD), were searched up to November 2022. Additional studies were identified through reference lists and related reviews. The risk of bias was assessed using the PROBAST prediction model risk of bias assessment tool. Meta-analyses were performed using the random effects model, focusing on the C-statistic as a discrimination index and the observed-to-expected ratio (OE) as a calibration index. Out of 1315 initial records, 16 studies were included, with 21 external validations of six models in Asia. The validated models consisted of Framingham models, pooled cohort equations (PCEs), SCORE, Globorisk, and WHO models, combined with the results of the first four models. The pooled C-statistic for men ranged from 0.72 (95% CI 0.70 to 0.75; PCEs) to 0.76 (95% CI 0.74 to 0.78; Framingham general CVD). In women, it varied from 0.74 (95% CI 0.22 to 0.97; SCORE) to 0.79 (95% CI 0.74 to 0.83; Framingham general CVD). The pooled OE ratio for men ranged from 0.21 (95% CI 0.018 to 2.49; Framingham CHD) to 1.11 (95%CI 0.65 to 1.89; PCEs). In women, it varied from 0.28 (95%CI 0.33 to 2.33; Framingham CHD) to 1.81 (95% CI 0.90 to 3.64; PCEs). The Framingham, PCEs, and SCORE models exhibited acceptable discrimination but poor calibration in predicting the 10-year risk of CVDs in Asia. Recalibration and updates are necessary before implementing these models in the region.
Citation: Nomali M, Khalili D, Yaseri M, Mansournia MA, Ayati A, Navid H, et al. (2023) Validity of the models predicting 10-year risk of cardiovascular diseases in Asia: A systematic review and prediction model meta-analysis. PLoS ONE 18(11): e0292396. https://doi.org/10.1371/journal.pone.0292396
Editor: Hean Teik Ong, HT Ong Heart Clinic, MALAYSIA
Received: July 4, 2023; Accepted: September 19, 2023; Published: November 30, 2023
Copyright: © 2023 Nomali et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Cardiovascular diseases (CVDs) are the leading cause of worldwide disease burden. The number of total CVD cases has nearly doubled from 271 million in 1990 to 523 million in 2019, and the number of CVD deaths increased from 12.1 million in 1990 to 18.6 million in 2019 [1]. In Asia, CVD is the major cause of death [2], in which CVD deaths surged from 5.6 million to 10.8 million between 1990 and 2019 [3].
It is expected that the number of people affected by CVD will significantly rise due to factors such as population growth and aging, particularly in Northern Africa and Western Asia, Central and Southern Asia, Latin America and the Caribbean, and Eastern and Southeastern Asia, where the percentage of elderly individuals is anticipated to double by 2050 [1].
Since Asian countries have become more Westernized, they have been eating more fat, which has led to an increase in serum total cholesterol, and may have contributed to a rise in coronary heart disease in this region [4].
The burden of CVD varies regionally and nationally due to differences in the incidence of risk factors as well as the availability of medical care [5]. Some risk factors cannot be removed entirely to prevent CVDs. Therefore, prevention may be a valuable way to manage CVDs in the general population [6], and using risk prediction models to target the population is ongoing [7]. It is also recommended by the American College of Cardiology/ American Heart Association (ACC/AHA) [8], the European Society of Cardiology (ESC) [9], and HEARTS [10] guidelines.
Various models have been developed in Western countries to estimate the risk of CVDs [11–19]. However, differences in risk profiles and access to healthcare facilities may prevent being applied to Asia without any validation and performance assessment [20,21]. In addition, a systematic review by Damen et al. in 2016 indicated that only 19% of developed models for CVDs have been externally validated by independent researchers [22].
Although previous systematic reviews and meta-analyses evaluated the performance of the externally validated Framingham models and pooled cohort equations (PCEs) for the general population in 2019 [23] and the general Chinese population in 2022 [24], the performance of existing prediction models are not adequately assessed in Asia [21,25–27]. For the first time, we aimed to review the performance of existing prediction models for predicting the 10-year risk of CVDs in Asia through a systematic review and meta-analysis. Our findings demonstrate a complete picture of the CVD model’s performance in Asia, which helps researchers to choose the best model for designing prediction model studies, and clinicians and policymakers to choose the best one to apply in the prevention programs.
Materials and methods
This was a systematic review and prediction model meta-analysis, which was approved by the Research Ethics Committee (REC) of the School of Public Health & Allied Medical Sciences at TUMS on March 12, 2022 (approval ID: IR.TUMS.SPH.REC.1400.353).
Eligibility criteria
In this review, studies that externally validated existing prediction models to estimate the 10-year risk of CVDs for both men and women in the general population in Asia were included. The PICOTS components were defined as follows:
- Participants (P): General population in Asia
- Intervention (I) and comparators (C): independent investigators externally validate CVD prediction models in Asia.
- Outcome (O): Outcome for which the original CVD models were developed (i.e., risk of CVD)
- Timing/prediction horizon (T): 10 years
- Setting (S): Primary healthcare
In addition, we excluded review articles, studies that developed new models instead of validating the existing prediction models, and studies that validated models for vascular diseases or stroke or validated exclusively in patients or an area other than Asia or with inadequate data regarding model performance measures. We also excluded studies that performed external validation as a phase of model development in Western countries. It should be noted that we did not consider any language or time limitations.
Information sources and search strategy
Our data sources were databases, including PubMed, Web of Science conference proceedings citation index, Scopus, Global Index Medicus of the World Health Organization (WHO), and Open Access Thesis and Dissertations (OATD).
We searched in PubMed through the following search strategy: (("Cardiovascular Diseases"[Mesh]) AND ("risk chart" OR "risk score" OR "risk equation" OR "risk algorithm" OR "risk prediction" OR "risk assessment") AND (validation OR calibration) AND (Asia OR "Middle east")) from inception until November 28, 2022. The search strategy for other databases was modified, as shown in S1 Table. We also checked the reference list of included studies and related reviews to identify additional studies.
Selection process
Two authors (MN and DK) independently screened the retrieved records based on title and abstract. After that, we evaluated the full text of the papers and selected those that met the mentioned eligibility criteria. Any disagreement was resolved by discussion with the third person (SN).
Data items and data extraction process
We extracted the following variables from the included models: the first author’s name, publication year, study country, WHO region using www.who.int/countries, prediction model, population (i.e., the data source used for external validation), predictors, follow-up time, study outcome and outcome definition, sample size (i.e., total and in men and women, separately), number of events, statistical modeling method and analysis (i.e., handling missing data), and model performance indices, including C-statistic for discrimination and observed-to-expected (OE) ratio for calibration.
To calculate the observed-to-expected ratio, we extracted the total sample size and the total number of observed (O) and expected (E) events across gender [28]. Otherwise, we used WebPlotDigitizer (version 4.6) to get the probability for the decile of risks [29]. For model discrimination, we obtained the C-statistic and the 95% confidence interval (CI) or standard error (SE) for men and women separately. For articles that validated multiple models in one study, we did separate data extractions for each model.
One reviewer (MN) screened the full text of included studies for data extraction, and relevant data were extracted. Another reviewer (MY) checked the extracted items. A third reviewer (DK) also reached a consensus regarding any disagreements.
Risk of bias assessment
To assess the risk of bias in included studies, we used the prediction model risk of bias assessment tool (PROBAST) [30]. It contained 20 signaling questions across four domains of participants, predictors, outcome, and analysis, which were answered as yes, no, and no information. Finally, the results were summarized into three groups: low, high, or unclear risk of bias. The methodological quality of included studies was assessed by two independent reviewers (MN, MM). We resolved disagreements by discussing and consulting with a third person (SN) to reach a consensus.
Study measures
This study’s measures were C-statistic, which indicates model discrimination, and OE ratio, which indicates model calibration. Discrimination was the ability of a model to separate high-risk individuals from low-risk individuals and was typically reported as a measure of concordance, the c-statistic, which varies between 0.5 and 1.0 [31]. The C-statistic of 0.5–0.7, 0.70–80, 0.80–0.90, and ≥0.90 indicated poor, acceptable, excellent, and outstanding discriminative ability, respectively [32]. Calibration was the agreement between observed and expected outcomes. The OE ratio less than 1 indicated overestimation, and the OE greater than 1 indicated underestimation of the risk [33].
Statistical analysis
The OE ratio and its standard error were calculated using the equations provided in Debray et al. study [26] to estimate the total O: E ratio from other information in a preliminary study.
Meta-analyses of the OE ratio and the C-statistic for CVD models were performed by random effects model with restricted maximum likelihood estimation [34] and 95% confidence intervals (CI) to quantify the uncertainty of the pooled performance estimates.
As previously recommended for the meta-analysis of prediction models [34], we also reported 95% prediction intervals (PI) to depict the between-study heterogeneity. The PI sets limits on the expected performance of future model validation, which will be comparable to the studies included in the meta-analysis.
Because of methodological heterogeneity between various prediction models, we did not provide all study combinations, and we provided the meta-analysis stratified by model and gender. In addition, some studies have reported a 5-year risk of CVDs due to short follow-up, despite using CVD models originally developed to predict a 10-year risk. In this case, we extrapolated observed and expected event probabilities to 10 years using the formula based on the Poisson distribution [23,28,34].
We also did sensitivity analysis by excluding studies that reported 5-year risk evaluating their effect on the overall model performance. In addition, despite the recommendation of previous studies to perform the random effects-model meta-analysis for prediction model studies [28,34], we also quantitatively synthesized the performance measures using the fixed-effect model for the Framingham CHD and SCORE risk models because of being validated twice [35]. We used R version 4.2.2, the metamisc package, and the valmeta function to perform the meta-analyses. For a graphical depiction of the PROBAST risk of bias assessment, we used STATA/IC version 14.2 (Stata Corp LP College Station, TX, USA) and the QUADAS module.
Results
Study selection and characteristics
From 1315 records identified through searching in databases and the reference list for included studies and related reviews [21–27,36], a total of 50 full texts were reviewed to assess whether they met the eligibility criteria, and ultimately, 16 studies were included in the review. Fig 1 illustrates the process of searching, screening, and selecting records.
Five studies [38–42] simultaneously validated two risk models. Therefore, 21 prediction models were validated by 16 included studies. Fifteen studies were published in English, and only one was written in Chinese [43], which were considered in the review regardless of language.
Table 1 shows the characteristics of validated risk models in Asia. According to Table 1, Framingham coronary heart disease (CHD) [44], Framingham general CVD [45], PCEs [46], SCORE [47], Globorisk [19], and WHO [48] models were risk scores externally validated in Asia. Out of 21 validated models, 15 models (71%) were validated in the Western Pacific region of the WHO (i.e., in China, Japan, and Korea), and six models (29%) were validated in the Eastern Mediterranean region (i.e., in Iran). There were external validations of the Framingham general CVD [45], PCEs [46], and SCORE [47] risk models in both the Eastern Mediterranean and Western Pacific regions. In contrast, the Framingham CHD [44] was only validated in the Western Pacific region. The Globorisk [19] and WHO [48] risk models were validated for the Eastern Mediterranean and Western Pacific regions, respectively (Table 1).
Risk of bias assessment
Fig 2 shows a summary of the risk of bias analysis by PROBAST. Two domains of participant selection and outcome had a low risk of bias. In only one study, the predictors were unclear, and others were at low risk of bias for predictors. In terms of the analysis domain, more than 50% of the studies were judged to have a high risk of bias because they did not adequately handle missing data and used a complete case approach instead of multiple imputations. Thus, most studies were scored high risk of bias (Fig 2). Additional detailed information has been provided in S2 Table.
Performance of the validated models
The statistical performances, including discriminative ability using concordance statistics and calibration using a calibration plot, were reported in all validated models. The clinical usefulness of decision curve analysis was reported in only six models (29%) compared to the statistical performance reported in all models.
Framingham CHD [44], Framingham general CVD [45], PCEs [46], and SCORE risk models [47] were validated more than once, and we quantitatively synthesized their performance measures. The performance of the Globorisk [19] and WHO [48] models that were validated once is shown in Table 1.
Discrimination
The forest plot in Fig 3 displays the C-statistic of risk models for both genders. The pooled C-statistic for men ranged from 0.72 (95% CI 0.70 to 0.75 and 95% PI 0.65 to 0.79) for the PCEs to 0.76 (95% CI 0.74 to 0.78 and 95% PI 0.70 to 0.81) for the Framingham general CVD model. In women, it varied between 0.74 (0.95% CI 0.22 to 0.97) for the SCORE and 0.79 (95% CI 0.74 to 0.83 and 95% PI 0.63 to 0.89) for the Framingham general CVD (Fig 3).
Forest plot of C-statistic by risk models among men (left) and women (right).
Calibration
Fig 4 provides a Forest plot of the OE ratio by risk models across gender. The Framingham model for CHD indicated overprediction for both men and women. The Framingham general CVD and SCORE models showed an overprediction among men and an underprediction among women. The PCEs indicated an underprediction in both men and women (Fig 4).
Forest plot of OE ratio by risk models among men (left) and women (right).
According to Fig 4, the pooled OE ratio for men ranged from 0.21 (95% CI 0.018 to 2.49) for the Framingham CHD model to 1.11 (95% CI 0.65 to 1.89, and 95% PI 0.21 to 5.77) for the PCEs. In women, it varied between 0.28 (95% CI 0.33 to 2.33) for the Framingham CHD model and 1.81(95% CI 0.90 to 3.64, and 95% PI 0.21 to 15.74) for the PCEs (Fig 4).
Sensitivity analysis
The discrimination for the Framingham general CVD model and PCEs did not change after excluding studies that provided a 5-year risk of CVDs in both men and women, while their calibration decreased (S1 and S2 Figs).
The C-statistic for the Framingham CHD through the Fixed-effect model was 0.75 (95% CI 0.74 to 0.76) in men and 0.80 (95%CI 0.78 to 0.82) in women. For the SCORE risk model, the C-statistics through the Fixed-effect model were 0.73 (95%CI 0.71 to 0.75) and 0.72 (95% CI 0.70 to 0.73) in men and women, respectively.
The OE ratio for the Framingham CHD through the Fixed-effect model was 0.18 (95% CI 0.17 to 19) in men and 0.25 (95%CI 0.23 to 0.28) in women. For the SCORE risk model, the OE ratio was 0.98 (95% CI 0.90 to 1.07) and 1.09 (95% CI 0.98 to 1.22) in men and women, respectively.
Discussion
To the best of our knowledge, for the first time, we have systematically reviewed the performance of models for predicting the 10-year risk of CVDs in the general population in Asia. The Framingham CHD, Framingham general CVD, PCEs, SCORE and Globorisk models, and WHO risk charts for East Asia were all risk scores that were externally validated in this region. The Framingham general CVD and PCEs were the most validated tools in this study. Additionally, Globorisk and WHO models were validated once. We found an acceptable discriminative ability for all models in men and women. The Framingham CHD model overestimated the risk in both genders. However, the Framingham general CVD and SCORE model overestimated the risk in men and underestimated it in women. The PCEs underestimated the risk in both men and women.
A systematic review and meta-analysis by Damen et al. in 2019 [21] evaluated the performance of the Framingham CHD model and PCEs. In 2022, Zhiting et al. [24] assessed the performance of the Framingham general CVD model and PCEs in the general Chinese population through a systematic review. Both systematic reviews focused on the most validated models. Neither of the two systematic reviews considered the SCORE model, which was noticed in the current review. Therefore, they did not provide a complete picture of the validated model’s performance in Asia, which necessitated this study.
Discrimination and calibration are model performance evaluation’s two most important aspects [59]. In a systematic review, 64% of models had the C-statistic or area under the receiver operating characteristic curve (AUC-ROC) as a measure of discrimination. Calibration plot (26%), observed to the expected ratio (26%), or Hosmer-Lemeshow test (14%) were among the methods used to report calibration [22]. It was shown that reporting discrimination increased from 14% in 1976 to 77% in 2013. In contrast, reporting calibration decreased from 76% in 1976 to 59% in 2013 [22]. In this review, all studies reported a C-statistic to demonstrate the discriminative ability of prediction models, and they used calibration plots to show the agreement between the observed and expected event probabilities. In line with the previous review [22], some studies published before 2015 did not report additional calibration measures, such as the OE ratio. Calibration is often considered the Achilles heel of prediction analysis, as it enables us to understand the model performance in a particular setting, identify where predictions may be inaccurate, and determine if the model needs to be updated [60]. Discrimination and calibration are essential but not sufficient for clinical usefulness. Reporting decision curves or net benefit is necessary to gain further insight into the feasibility of such models for clinical use [61]. In this review, only one-third of the validated models provided this information. As suggested by the TRIPOD guideline [62], it is advisable to present performance measures such as discrimination, calibration, and decision curve or net benefit when using a model to assist clinical decision-making [61].
In this review, the discriminative ability of the models varied between 0.7 and 0.8. In the previous review by Damen et al. in 2019, the discriminative ability of the Framingham CHD model and PCEs ranged between 0.68 and 0.74 [23]. It was also between 0.72 and 0.76 for the Framingham general CVD model and PCEs in the Chinese population [24]. The absence of a consistent metric in models and significant differences in case mix may affect the accuracy of this method. For instance, where the case mix is too homogeneous, we found poor discrimination. As a result, it is recommended to consider case-mix variation when evaluating discrimination measures [63].
A systematic review found that the Framingham CHD model and PCEs overestimated the risk of cardiovascular disease, particularly in higher-risk populations [23]. In a systematic review of the general Chinese population, the Framingham general CVD model and PCEs overestimated the risk in men and underestimated it in women, respectively [24]. In our study, the Framingham CHD model overestimated the risk in line with previous studies. However, the Framingham general CVD model and PCEs behaved differently. SCORE model behaved like the Framingham general CVD model. The difference in model calibration could be attributed to case-mix [23,64] and the healthcare system difference [23]. Using a model based on a particular region is inaccurate when assessing the actual risk in another area [24]. Applying these models as initially developed to make treatment decisions could result in either over- or undertreatment of individuals, leading to unnecessary burden to individuals and the health care system. As a result, it is recommended that none of these models be put to use without recalibrating the baseline risk or hazard according to the local setting [23].
There were only two studies for the Framingham CHD and SCORE risk models, and we combined the performance measures through the random-effects model. Validation studies are different in design, conduct, and case mix; we cannot attribute between study variations to chance only. Therefore, it is necessary to consider the heterogeneity through the meta-analysis rather than ignore it [28], and using random-effects models is generally recommended [34]. On the other hand, despite the technical possibility of applying random-effects meta‐analysis, it is not sensible in the case of two studies. Thus, the Fixed-effect model is recommended unless there are strong arguments against the common effect assumption [35]. Therefore, we also performed a fixed-effect meta-analysis for the aforementioned models. Random-effects models produce wider 95%CI compared to the Fixed-effect model, which indicates lower certainty on the pooled estimates and the need for more validation studies in this region.
Study limitations
Our study had several limitations. First, our source of information was restricted to what the authors provided in the primary studies, and we had to exclude one study with insufficient data, which we could not obtain from the authors. Also, we were unable to perform a meta-analysis for the WHO risk charts and Globorisk model because the full text was not available despite contact with the authors, and one-time validation, respectively. On the other hand, WHO risk charts and Globorisk models have been externally validated in their original papers. However, we did not consider the external validation results as part of model development in this paper. In addition, we had a small number of studies and used a random effects model, in which the point estimate and confidence interval may provide a false sense of certainty. However, few suitable alternatives exist when limited studies are available [65]. Second, for some studies before 2015 that did not report the number of observed and expected events, we had to use the calibration plot to estimate the OE ratio without the authors`proof. Third, due to the limited number of validated studies [28], we did not perform a meta-regression analysis to detect possible reasons for heterogeneity for each model. Fourth, we compared the statistical performance measures, but we were unable to combine the models`clinical usefulness to find the therapeutic threshold for each risk model because there were only a few studies. Fifth, over half of the studies were of low quality because they did not handle missing data well, which might affect the study measures. Sixth, because of the descriptive nature of the study and due to the limited number of included studies for each model (range of 2 to 8 studies), we did not evaluate the publication bias.
Conclusion
In this study, the Framingham CHD and general CVD, PCEs, and SCORE models performed differently to predict the 10-year risk of CVDs in Asia. Although these four models indicated an acceptable level of discrimination in this region, their calibration differed among men and women. The Framingham CHD model and PCEs over-and underestimated the risk, respectively. At the same time, the Framingham general CVD and SCORE models overestimated the risk in men and underestimated it in women. If a local recalibration and updating are not made, it will lead to over- or undertreatment in clinical practice and additional harm and cost. Further research should focus on external validation of the existing prediction models or recalibration and updating these models for local settings to guide CVD prevention in this region and target the population based on risk predictions. In addition, these models’ decision curves and net benefits should be assessed to help clinical decision-making.
Supporting information
S1 Fig. Forest plot of C-statistic by risk models among men (left) and women (right) after excluding studies provided 5-year risk of CVDs.
https://doi.org/10.1371/journal.pone.0292396.s002
(TIF)
S2 Fig. Forest plot of OE ratio by risk models among men (left) and women (right) after excluding studies provided 5-year risk of CVDs.
https://doi.org/10.1371/journal.pone.0292396.s003
(TIF)
S1 Table. Search strategy used to retrieve related documents (Search date: November 28 2022).
https://doi.org/10.1371/journal.pone.0292396.s004
(DOCX)
S2 Table. Summary of risk of bias analysis by PROBAST for included studies.
https://doi.org/10.1371/journal.pone.0292396.s005
(DOCX)
Acknowledgments
This article relates to the dissertation of Mahin Nomali for a Ph.D. degree in Epidemiology (no. 240/1071) at the School of Public Health of Tehran University of Medical Sciences (TUMS). We would like to thank the University of Santiago de Compostela for facilitating a year of access to the databases through the USC student service account.
References
- 1. Roth GA, Mensah GA, Johnson CO, Addolorato G, Ammirati E, Baddour LM, et al., Global Burden of Cardiovascular Diseases and Risk Factors, 1990–2019: Update From the GBD 2019 Study. J Am Coll Cardiol, 2020. 76(25): p. 2982–3021 pmid:33309175
- 2. Zhao D, Epidemiological Features of Cardiovascular Disease in Asia. JACC: Asia, 2021. 1(1): p. 1–13 pmid:36338365
- 3. Zhao D, Liu J, Wang M, Zhang X, and Zhou M, Epidemiology of cardiovascular disease in China: current features and implications. Nat Rev Cardiol, 2019. 16(4): p. 203–212 pmid:30467329
- 4. Ueshima H, Sekikawa A, Miura K, Turin TC, Takashima N, Kita Y, et al., Cardiovascular Disease and Risk Factors in Asia. Circulation, 2008. 118(25): p. 2702–2709 pmid:19106393
- 5. GBD 2017 Risk Factor Collaborators (2018), Global, regional, and national comparative risk assessment of 84 behavioural, environmental and occupational, and metabolic risks or clusters of risks for 195 countries and territories, 1990–2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet, 2018. 392(10159): p. 1923–1994 pmid:30496105
- 6. Ciumărnean L, Milaciu MV, Negrean V, Orășan OH, Vesa SC, Sălăgean O, et al., Cardiovascular Risk Factors and Physical Activity for the Prevention of Cardiovascular Diseases in the Elderly. Int J Environ Res Public Health, 2021. 19(1) pmid:35010467
- 7. Farzadfar F, Cardiovascular disease risk prediction models: challenges and perspectives. Lancet Glob Health, 2019. 7(10): p. e1288–e1289 pmid:31488388
- 8. Arnett DK, Blumenthal RS, Albert MA, Buroker AB, Goldberger ZD, Hahn EJ, et al., 2019 ACC/AHA Guideline on the Primary Prevention of Cardiovascular Disease: A Report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines. Circulation, 2019. 140(11): p. e596–e646 pmid:30879355
- 9. Visseren FLJ, Mach F, Smulders YM, Carballo D, Koskinas KC, Bäck M, et al., 2021 ESC Guidelines on cardiovascular disease prevention in clinical practice. Eur Heart J, 2021. 42(34): p. 3227–3337 pmid:34458905
- 10. World Health Organization, HEARTS technical package for cardiovascular disease management in primary health care: risk based CVD management. Licence: CC BY-NC-SA 3.0 IGO. 2020, World Health Organization: Geneva.
- 11. Ridker PM, Buring JE, Rifai N, and Cook NR, Development and validation of improved algorithms for the assessment of global cardiovascular risk in women: the Reynolds Risk Score. Jama, 2007. 297(6): p. 611–619 pmid:17299196
- 12. Ridker PM, Paynter NP, Rifai N, Gaziano JM, and Cook NR, C-reactive protein and parental history improve global cardiovascular risk prediction: the Reynolds Risk Score for men. Circulation, 2008. 118(22): p. 2243–2251, 2244p following 2251 pmid:18997194
- 13. Anderson KM, Odell PM, Wilson PW, and Kannel WB, Cardiovascular disease risk profiles. Am Heart J, 1991. 121(1 Pt 2): p. 293–298 pmid:1985385
- 14. Goff DC Jr, Lloyd-Jones DM, Bennett G, Coady S, D’Agostino RB, Gibbons R, et al., 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation, 2014. 129(25 Suppl 2): p. S49–73 pmid:24222018
- 15. Woodward M, Brindle P, and Tunstall-Pedoe H, Adding social deprivation and family history to cardiovascular risk assessment: the ASSIGN score from the Scottish Heart Health Extended Cohort (SHHEC). Heart, 2007. 93(2): p. 172–176 pmid:17090561
- 16. Hippisley-Cox J, Coupland C, and Brindle P, Development and validation of QRISK3 risk prediction algorithms to estimate future risk of cardiovascular disease: prospective cohort study. BMJ, 2017. 357: p. j2099 pmid:28536104
- 17. Assmann G, Cullen P, and Schulte H, Simple scoring scheme for calculating the risk of acute coronary events based on the 10-year follow-up of the prospective cardiovascular Münster (PROCAM) study. Circulation, 2002. 105(3): p. 310–315 pmid:11804985
- 18. Ueda P, Woodward M, Lu Y, Hajifathalian K, Al-Wotayan R, Aguilar-Salinas CA, et al., Laboratory-based and office-based risk scores and charts to predict 10-year risk of cardiovascular disease in 182 countries: a pooled analysis of prospective cohorts and health surveys. Lancet Diabetes Endocrinol, 2017. 5(3): p. 196–213 pmid:28126460
- 19. Hajifathalian K, Ueda P, Lu Y, Woodward M, Ahmadvand A, Aguilar-Salinas CA, et al., A novel risk score to predict cardiovascular disease risk in national populations (Globorisk): a pooled analysis of prospective cohorts and health examination surveys. Lancet Diabetes Endocrinol, 2015. 3(5): p. 339–355 pmid:25819778
- 20. Mendis S, Cardiovascular risk assessment and management in developing countries. Vasc Health Risk Manag, 2005. 1(1): p. 15–18 pmid:17319094
- 21. Gopal DP and Usher-Smith JA, Cardiovascular risk models for South Asian populations: a systematic review. Int J Public Health, 2016. 61(5): p. 525–534 pmid:26361963
- 22. Damen JA, Hooft L, Schuit E, Debray TP, Collins GS, Tzoulaki I, et al., Prediction models for cardiovascular disease risk in the general population: systematic review. Bmj, 2016. 353: p. i2416 pmid:27184143
- 23. Damen JA, Pajouheshnia R, Heus P, Moons KGM, Reitsma JB, Scholten R, et al., performance of the Framingham risk models and pooled cohort equations for predicting 10-year risk of cardiovascular disease: a systematic review and meta-analysis. BMC Med, 2019. 17(1): p. 109 pmid:31189462
- 24. Zhiting G, Jiaying T, Haiying H, Yuping Z, Qunfei Y, and Jingfen J, Cardiovascular disease risk prediction models in the Chinese population- a systematic review and meta-analysis. BMC Public Health, 2022. 22(1): p. 1608 pmid:35999550
- 25. Baart SJ, Dam V, Scheres LJJ, Damen J, Spijker R, Schuit E, et al., Cardiovascular risk prediction models for women in the general population: A systematic review. PLoS One, 2019. 14(1): p. e0210329 pmid:30620772
- 26. Rossello X, Dorresteijn JA, Janssen A, Lambrinou E, Scherrenberg M, Bonnefoy-Cudraz E, et al., Risk prediction tools in cardiovascular disease prevention: A report from the ESC Prevention of CVD Programme led by the European Association of Preventive Cardiology (EAPC) in collaboration with the Acute Cardiovascular Care Association (ACCA) and the Association of Cardiovascular Nursing and Allied Professions (ACNAP). Eur J Cardiovasc Nurs, 2019. 18(7): p. 534–544 pmid:31234638
- 27. Zhang Y, Miao H, Chia YC, Buranakitjaroen P, Siddique S, Shin J, et al., Cardiovascular risk assessment tools in Asia. J Clin Hypertens (Greenwich), 2022. 24(4): p. 369–377 pmid:34981620
- 28. Debray TP, Damen JA, Snell KI, Ensor J, Hooft L, Reitsma JB, et al., A guide to systematic review and meta-analysis of prediction model performance. Bmj, 2017. 356: p. i6460 pmid:28057641
- 29. Rohatgi A. WebPlotDigitizer. 4.6 2022 [cited 2023; Available from: https://automeris.io/WebPlotDigitizer.
- 30. Moons KGM, Wolff RF, Riley RD, Whiting PF, Westwood M, Collins GS, et al., PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration. Ann Intern Med, 2019. 170(1): p. W1–w33 pmid:30596876
- 31. Steyerberg EW, Harrell FE Jr, Borsboom GJ, Eijkemans MJ, Vergouwe , and Habbema JD, Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol, 2001. 54(8): p. 774–781 pmid:11470385
- 32.
Hosmer DW Jr, Lemeshow S, and Sturdivant RX, Applied logistic regression. Vol. 398. 2013, New York: Wiley.
- 33. Nattino G, Lemeshow S, Phillips G, Finazzi S, and Bertolini G, Assessing the Calibration of Dichotomous Outcome Models with the Calibration Belt. The Stata Journal, 2017. 17(4): p. 1003–1014
- 34. Debray TP, Damen JA, Riley RD, Snell K, Reitsma JB, Hooft L, et al., A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes. Stat Methods Med Res, 2019. 28(9): p. 2768–2786 pmid:30032705
- 35. Bender R, Friede T, Koch A, Kuss O, Schlattmann P, Schwarzer G, et al., Methods for evidence synthesis in the case of very few studies. Res Synth Methods, 2018. 9(3): p. 382–392 pmid:29504289
- 36. Eichler K, Puhan MA, Steurer J, and Bachmann LM, Prediction of first coronary events with the Framingham score: a systematic review. Am Heart J, 2007. 153(5): p. 722–731, 731.e721-728 pmid:17452145
- 37. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al., The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Rev Esp Cardiol (Engl Ed), 2021. 74(9): p. 790–799 pmid:34446261
- 38. Fahimfar N, Fotouhi A, Mansournia MA, Malekzadeh R, Sarrafzadegan N, Azizi F, et al., Prediction of Cardiovascular Disease Mortality in a Middle Eastern Country: Performance of the Globorisk and Score Functions in Four Population-Based Cohort Studies of Iran. Int J Health Policy Manag, 2022. 11(2): p. 210–217 pmid:32668896
- 39. Jiang Y, Ma R, Guo H, Zhang X, Wang X, Wang K, et al., External validation of three atherosclerotic cardiovascular disease risk equations in rural areas of Xinjiang, China. BMC Public Health, 2020. 20(1): p. 1471 pmid:32993590
- 40. Bae JH, Moon MK, Oh S, Koo BK, Cho NH, and Lee MK, Validation of Risk Prediction Models for Atherosclerotic Cardiovascular Disease in a Prospective Korean Community-Based Cohort. Diabetes Metab J, 2020. 44(3): p. 458–469 pmid:31950769
- 41. Lee CH, Woo YC, Lam JK, Fong CH, Cheung BM, Lam KS, et al., validation of the Pooled Cohort equations in a long-term cohort study of Hong Kong Chinese. J Clin Lipidol, 2015. 9(5): p. 640–646.e642 pmid:26350809
- 42. Li J, Liu F, Yang X, Cao J, Chen S, Chen J, et al., Validating World Health Organization cardiovascular disease risk charts and optimizing risk assessment in China. Lancet Reg Health West Pac, 2021. 8: p. 100096 pmid:34327424
- 43. Tang X, Zhang DD, He L, Cao Y, Wang JW, Li N, et al., [Application of the China-PAR risk prediction model for atherosclerotic cardiovascular disease in a rural northern Chinese population]. Beijing Da Xue Xue Bao Yi Xue Ban, 2017. 49(3): p. 439–445.
- 44. Wilson PW D’Agostino RB, Levy D, Belanger AM, Silbershatz H, and Kannel WB, Prediction of coronary heart disease using risk factor categories. Circulation, 1998. 97(18): p. 1837–1847 pmid:9603539
- 45. D’Agostino RB Sr, Vasan RS, Pencina MJ, l PA, Cobain M, Massaro JM, et al., General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation, 2008. 117(6): p. 743–753 pmid:18212285
- 46. Goff DC Jr, Lloyd-Jones DM, Bennett G, Coady S, D’agostino RB, Gibbons R, et al., 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation, 2014. 129(25_suppl_2): p. S49–S73.
- 47. Conroy RM, Pyörälä K, Fitzgerald AP, Sans S, Menotti A, De Backer G, et al., Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. Eur Heart J, 2003. 24(11): p. 987–1003 pmid:12788299
- 48. World Health Organization cardiovascular disease risk charts: revised models to estimate risk in 21 global regions. Lancet Glob Health, 2019. 7(10): p. e1332–e1345 pmid:31488387
- 49. Liu J, Hong Y, D’Agostino RB Sr, Wu Z, Wang W, Sun J, et al., Predictive value for the Chinese population of the Framingham CHD risk assessment tool compared with the Chinese Multi-Provincial Cohort Study. Jama, 2004. 291(21): p. 2591–2599 pmid:15173150
- 50. Jee SH, Jang Y, Oh DJ, Oh BH, Lee SH, Park SW, et al., A coronary heart disease prediction model: the Korean Heart Study. BMJ Open, 2014. 4(5): p. e005025 pmid:24848088
- 51. Bozorgmanesh M, Hadaegh F, and Azizi F, Predictive accuracy of the ’Framingham’s general CVD algorithm’ in a Middle Eastern population: Tehran Lipid and Glucose Study. Int J Clin Pract, 2011. 65(3): p. 264–273 pmid:21314863
- 52. Khalili D, Hadaegh F, Soori H, Steyerberg EW, Bozorgmanesh M, and Azizi F, Clinical usefulness of the Framingham cardiovascular risk profile beyond its statistical performance: the Tehran Lipid and Glucose Study. Am J Epidemiol, 2012. 176(3): p. 177–186 pmid:22814370
- 53. Sepanlou SG, Malekzadeh R, Poustchi H, Sharafkhah M, Ghodsi S, Malekzadeh F, et al., The clinical performance of an office-based risk scoring system for fatal cardiovascular diseases in North-East of Iran. PLoS One, 2015. 10(5): p. e0126779 pmid:26011607
- 54. Jung KJ, Jang Y, Oh DJ, Oh BH, Lee SH, Park SW, et al., The ACC/AHA 2013 pooled cohort equations compared to a Korean Risk Prediction Model for atherosclerotic cardiovascular disease. Atherosclerosis, 2015. 242(1): p. 367–375 pmid:26255683
- 55. Khalili D, Asgari S, Hadaegh F, Steyerberg EW, Rahimi K, Fahimfar N, et al., A new approach to test validity and clinical usefulness of the 2013 ACC/AHA guideline on statin therapy: A population-based study. Int J Cardiol, 2015. 184: p. 587–594 pmid:25769004
- 56. Tang X, Zhang D, He L, Wu N, Si Y, Cao Y, et al., performance of atherosclerotic cardiovascular risk prediction models in a rural Northern Chinese population: Results from the Fangshan Cohort Study. Am Heart J, 2019. 211: p. 34–44 pmid:30831332
- 57. Liu X, Shen P, Zhang D, Sun Y, Chen Y, Liang J, et al., Evaluation of Atherosclerotic Cardiovascular Risk Prediction Models in China: Results From the CHERRY Study. JACC Asia, 2022. 2(1): p. 33–43 pmid:36340248
- 58. Sawano M, Kohsaka S, Okamura T, Inohara T, Sugiyama D, Watanabe M, et al., validation of the european SCORE risk chart in the healthy middle-aged Japanese. Atherosclerosis, 2016. 252: p. 116–121 pmid:27521900
- 59. Royston P and Altman DG, External validation of a Cox prognostic model: principles and methods. BMC Med Res Methodol, 2013. 13: p. 33 pmid:23496923
- 60. Van Calster B, McLernon DJ, van Smeden M, Wynants L, and Steyerberg EW, Calibration: the Achilles heel of predictive analytics. BMC Med, 2019. 17(1): p. 230 pmid:31842878
- 61. McLernon DJ, Giardiello D, Van Calster B, Wynants L, van Geloven N, van Smeden M, et al., Assessing Performance and Clinical Usefulness in Prediction Models With Survival Outcomes: Practical Guidance for Cox Proportional Hazards Models. Ann Intern Med, 2023. 176(1): p. 105–114 pmid:36571841
- 62. Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, et al., Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med, 2015. 162(1): p. W1–73 pmid:25560730
- 63. Altman DG, Prognostic Models: A Methodological Framework and Review of Models for Breast Cancer. Cancer Investigation, 2009. 27(3): p. 235–243 pmid:19291527
- 64. Vergouwe Y, Moons KG, and Steyerberg EW, External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol, 2010. 172(8): p. 971–980 pmid:20807737
- 65. Borenstein M, Hedges LV, Higgins JP, and Rothstein HR, Introduction to Meta-Analysis. 2009.