Performance evaluation of 10-year ultrasound image-based stroke/cardiovascular (CV) risk calculator by comparing against ten conventional CV risk calculators: A diabetic study
Introduction
Cardiovascular diseases (CVD) including heart and stroke are the major cause of mortality and morbidity around the world [1]. The spread of CVD is almost similar in every region of the world including developed and developing countries [2,3]. Atherosclerosis is the primary cause of cardiovascular disease [4]. Coronary and carotid arteries are the main sources that supply the oxygenated blood to the heart and the brain, the two main organs that are responsible for heart attack and stroke. Deposition of atherosclerotic plaque within the vessel wall obstructs this blood supply causing myocardial infarction (MI) or stroke. According to INTERHEART1 [5] and INTERSTROKE1 [6] studies, 90% of the mortalities due to these CVD are attributed to the conventional cardiovascular risk (CCVR) factors such as age, ethnicity, systolic blood pressure (SBP), low-density lipoproteins cholesterol (LDL-C), total cholesterol (TC), smoking, diabetes mellitus (DM), and family history (FH). These risk factors have an additive effect on atherosclerotic vascular biomarkers causing an elevation in CVD/Stroke risk.
Preventive interventional methods (i.e., use of statins) require the determination of baseline as well as the long-term risk of a patient to prevent the onset of CVD. Risk estimation can be possible using cardiovascular (CV) risk calculators. From here on, we will use CCVRCs for “conventional cardiovascular risk calculators”. At the present stage, there are more than 100 CCVRCs available [7]. Most prominently and clinically used CCVRCs are FRS [8], UKPDS56 [9], UKPDS60 [10], QRISK3 [11], RRS [12], NIPPON [13], SCORE [14], WHO [15], PCRS (Atherosclerosis CVD - ASCVD) [16], and PROCAM [17] (The expansions of the abbreviation is shown in Appendix-Table 1). This was the main motivation for using only these ten CCVRCs in this study to evaluate the performance of AECRS1.010yr and to further study which calculator was most closely related to AECRS1.010yr. In this manuscript, the suffix ‘10yr’ indicates the risk value after ten years. CCVR calculators generally provide a risk estimation based on traditional risk factors. Recent guidelines indicate the use of some primitive CV risk calculators to assess the 10-year risk of CVD and to decide the statin eligibility for the patients [[18], [19], [20]]. Although such CCVRCs can provide the basis for the prescription of statins (i.e., lipid-lowering medications) for treating the risk of CVD, they sometimes do the not explain the CV events due to the morphological variations in the atherosclerotic blood vessels. Furthermore, CCVR factors do not provide any information about the arterial plaque variations or their components. However, since CCVR factors contribute to the risk of stroke/coronary heart disease (CHD), they are also termed as risk factors.
Advancements in imaging techniques especially non-invasive imaging modalities such as carotid ultrasound provide a cost-effective tool to assist physicians in understanding the morphological variations of the blood vessels due to atherosclerotic plaque tissues [19,21]. For example, carotid intima-media thickness (cIMT) and total plaque area (TPA) are the two important image-based phenotypes of carotid atherosclerosis that can provide information about the cardiovascular health of patients [[22], [23], [24], [25], [26], [27]] Note that there were a total of 28 CCVR factors considered in this study, and their abbreviations are all expanded in Appendix-Table 2. Since 2000, seven clinical guidelines have been recommended for using cIMT and/or carotid plaque for assisting the CVD risk estimation [28]. In 2010, Nambi et al. [29] demonstrated a significant improvement in CHD by taking into consideration the image-based phenotypes such as cIMT and carotid plaque. Stein et al. [30] also recommended the use of cIMT for CHD risk prediction. Polak et al. [31] presented a study with 296 participants and reported that CV outcomes can be predicted using the mean and the maximum cIMT. Association between intima-media thickness (IMT) in the common carotid artery (CCA) and coronary SYNTAX score was demonstrated by Ikeda et al. [32] in their study of 500 diabetic patients. The same group showed the link between carotid bulb plaques and coronary SYNTAX score in diabetic patients [33]. Recently, Suri et al. [34] showed the role of cIMT and variations in IMT (IMTV) to predict the Leukoaraiosis disease in adults. Thus, there is an additional benefit by considering the carotid image-based phenotypes in the risk prediction models along with CCVR factors for risk estimation.
Conventional cardiovascular risk calculators do not consider carotid image-based phenotypes in their risk prediction models. Therefore, the extent of atherosclerotic plaque measured using average intima-media thickness (IMTave), maximum IMT (IMTmax), minimum IMT (IMTmin), IMT variability (IMTV), and the morphology or echolucency [35] of the plaque (such as hyper- or hypo-echoic intensities of the grayscale images [36]) remains unexplained by such as CCVRCs. Note that both carotid plaque or wall thickening and CCVR factors are time (or age) dependent [[37], [38], [39]]. Thus, one can compute the 10-year risk of stroke/CVD by knowing the projection rates of the image-based phenotypes due to the conventional risk factors and then integrating them with current image-based phenotypes. These 10-year image-based phenotypes (such as IMTave10yr, IMTmax10yr, IMTmin10yr, and IMTV10yr) lead to the formation of the overall risk or composite risk score (CRS). This is known as an integrated calculator, integrated in the sense that conventional risk factors are integrated or fused with image-based phenotypes. Since our image-based phenotypes, to begin with, were measured automatically using AtheroEdge (AE) system (AtheroPoint, Roseville, CA, USA), we call this composite risk score as AECRS1.010yr.
The main objective of the study is to evaluate AECRS1.010yr (an integrated stroke/CV risk calculator) by comparing against the 10-year CCVRCs using several statistical metrics. The theory of these metrics is to evaluate the metric criteria given the CCVR factors for each patient corresponding to these eleven CV risk calculators (ten CCVRCs and the integrated calculator - AECRS1.010yr). These metrics are either based on how well the CCVR calculators find the high-risk patients in terms of similarity with AECRS1.010yr (so-called, similarity metric represented by a metric type of M1), or how far AECRS1.010yr is from the ten CCVR calculator constellations or cluster (clustering metric represented by a metric type of M2), or how the overall cohort of patients (or population) is doing in terms of its precision when compared against AECRS1.010yr (precision-of-merit metric represented by a metric type of M3), or ability to find the mean statistics for all CCVRCs against AECRS1.010yr (figure-of-merit metric represented by a metric type of M4), or how each patient's risk in CCVRC deviates against AECRS1.010yr (metric-type M5), or how each patient's risk in AECRS1.010yr deviates against CCVRC (metric-type of M6), or adapt kappa statistics to see the disparity between readings (kappa metric-type of M7), or the ability to use logistic regression and rank CCVRCs using odds ratio against AECRS1.010yr (metric-type M8). The overall idea is to understand which CCVRC is ranked closer to the AECRS1.010yr. The list of the eight statistical metrics and expansion of the abbreviations and symbols pertaining to performance metrics are listed in Appendix-Table 3 and Appendix-Table 4, respectively.
There are two hypotheses in this study: (i) The calculator which has a direct impact on atherosclerosis disease should show the highest area-under-the-curve (AUC) and (ii) we believe that the calculator that uses a larger number of CCVR factors is likely to be closer to the AECRS1.010yr risk calculator, since AECRS1.010yr is an integrated calculator. Our current study has several novelties: (i) a carotid ultrasound image-based phenotypes 10-year risk prediction model called AECRS1.010yr has been proposed; (ii) performance of AECRS1.010yr was evaluated against CCVRCs (Fig. 1); and (iii) a novel performance evaluation scheme consisting of eight statistical metrics has been proposed to compare the closeness of AECRS1.010yr against the ten CCVRCs.
The layout of the paper is as follows. Section 2 discusses the study population and image acquisition protocol. The methodology for the measurements of carotid image-based phenotypes has been discussed in section 3. The statistically derived metrics for evaluating the closeness factor between AECRS1.010yr and CCVR calculators have been presented in section 4. Results are presented in section 5. Discussion and benchmarking have been discussed in section 6. The paper concludes in section 7.
Section snippets
Study population
Between July 2009 and December 2010, a Japanese cohort of 202 patients was recruited from Toho University, Japan (Approved by the Institutional Review Board) and written consents were taken from all the participants. A total of 404 B-Mode ultrasound scans were collected from both left and right CCA. The scans were retrospectively analyzed by two operators (novice and experienced) as well as an expert who had 15 years of experience in the field of radiology. This study presents a unique and
Methodology for composite risk score for integrated calculator
Our scheme of evaluation of an integrated calculator AECRS1.010yr is based on several metrics whose fundamental assumption is to examine which of the popular ten CCVRCs is closest to AECRS1.010yr . This requires three important components: (a) computation of AECRS1.010yr; (b) risk computation from the ten CCVRCs (RC1 to RC10, here RC indicates the risk calculator), and (c) design of the eight metrics (M1 to M8) which were used for evaluation of AECRS1.010yr. A brief description of the system is
Metrics for evaluating AECRS1.010yr against ten CCVRCs
The main contribution of this study is to evaluate the AECRS1.010yr by comparing against the ten popular CCVRCs. The design of these metrics is based on the closeness of the ten unique clusters representing the ten different CCVRCs to the integrated AECRS1.010yr. The design of the metrics provides full coverage in every way by leveraging on statistics to define this closeness. This full coverage means that we investigate the closeness or ranking of ten CCVRCs w.r.t AECRS1.010yr by considering
Baseline evaluation
Table 1 indicates the baseline characteristics for the Japanese Population recruited in the current study. The pool of 202 patients had a mean age of 68.97 ± 10.96 years, ranging from 29 years to 88 years. The Japanese cohort in the current study had 50% of patients which were less than 70 years old and about 33% of patients were less than 55 years. Further, 20% of this cohort (less than 70 years old) had high diabetes (≥6.5%). Second, 50% of the cohort (that is less than 70 years old) also had
Summary and hypothesis
A comparative study of 11 CV risk calculators including ten CCVRCs (FRS, UKPDS56, UKPDS60, RRS, QRISK3, PCRS, SCORE, NIPPON, PROCAM, and WHO) and one “image-based phenotype fused with CCVR factors”, so-called “integrated risk calculator” (named as: AECRS1.010yr) was presented to perform a risk stratification of a diabetic Japanese cohort. Eight different statistical metrics (M1-M8) were designed to evaluate the performance of AECRS1.010yr by comparing against the ten CCVRCs. Each performance
Conclusion
We presented an integrated risk calculator called “AtheroEdge Composite Risk Score (AECRS1.010yr)” that was recently developed and computes the 10-year risk of carotid image phenotypes by integrating conventional cardiovascular risk factors (CCVRFs). We study the closeness between AECRS1.010yr against the ten other currently available conventional cardiovascular risk calculators (CCVRCs): QRISK3, Framingham Risk Score (FRS), United Kingdom Prospective Diabetes Study (UKPDS) 56, UKPDS60,
Conflicts of interest
None.
Financial Support/grants
None.
Disclosure
Dr. Jasjit Suri is affiliated to AtheroPoint™, focused in the area of stroke and cardiovascular imaging.
Acknowledgements
None.
Narendra N. Khanna, MD, DM, FACC is Advisor to Apollo Group of Hospitals in India and is working as Senior Consultant in Cardiology & Coordinator of Vascular Services at Indraprastha Apollo Hospital, New Delhi.
References (78)
- et al.
Erectile dysfunction and risk of cardiovascular disease
J. Am. Coll. Cardiol.
(2011) - et al.
Association of psychosocial risk factors with risk of acute myocardial infarction in 11 119 cases and 13 648 controls from 52 countries (the INTERHEART study): case-control study
Lancet
(2004) - et al.
Global and regional effects of potentially modifiable risk factors associated with acute stroke in 32 countries (INTERSTROKE): a case-control study
Lancet
(2016) - et al.
Morphologic TPA (mTPA) and composite risk score for moderate carotid atherosclerotic plaque is strongly associated with HbA1c in diabetes cohort
Comput. Biol. Med.
(2018) - et al.
Use of carotid ultrasound to identify subclinical vascular disease and evaluate cardiovascular disease risk: a consensus statement from the American Society of Echocardiography Carotid Intima-Media Thickness Task Force endorsed by the Society for Vascular Medicine
J. Am. Soc. Echocardiogr.
(2008) - et al.
Carotid intima-media thickness and presence or absence of plaque improves prediction of coronary heart disease risk: the ARIC (Atherosclerosis Risk in Communities) study
J. Am. Coll. Cardiol.
(2010) - et al.
Carotid intima-media thickness, plaques, and cardiovascular disease risk: implications for preventive cardiology guidelines
J. Am. Coll. Cardiol.
(2010) - et al.
Improved correlation between carotid and coronary atherosclerosis SYNTAX score using automated ultrasound carotid bulb plaque IMT measurement
Ultrasound Med. Biol.
(2015) - et al.
Deep learning strategy for accurate carotid intima-media thickness measurement: an ultrasound study on Japanese diabetic cohort
Comput. Biol. Med.
(2018) - et al.
Accurate cloud-based smart IMT measurement, its validation and stroke risk stratification in carotid ultrasound: a web-based point-of-care tool for multicenter clinical trial
Comput. Biol. Med.
(2016)
A state of the art review on intima–media thickness (IMT) measurement and wall segmentation techniques for carotid ultrasound
Comput. Methods Progr. Biomed.
Ultrasonographic measure of carotid plaque burden
JACC (J. Am. Coll. Cardiol.): Cardiovascular Imaging
Comparison between manual and automated analysis for the quantification of carotid wall by using sonography. A validation study with CT
EJR (Eur. J. Radiol.)
The effect of smoking on carotid intima–media thickness progression rate and rate of lumen diameter reduction
Eur. J. Intern. Med.
Comparison of different cardiovascular risk score calculators for cardiovascular risk prediction and guideline recommended statin uses
Indian Heart J.
Comparison of the Framingham Risk Score, SCORE and WHO/ISH cardiovascular risk prediction models in an Asian population
Int. J. Cardiol.
Web-based accurate measurements of carotid lumen diameter and stenosis severity: an ultrasound-based clinical tool for stroke risk assessment during multicenter clinical trials
Comput. Biol. Med.
PCA-based polling strategy in machine learning framework for coronary artery disease risk assessment in intravascular ultrasound: a link between carotid and coronary grayscale plaque morphology
Comput. Methods Progr. Biomed.
A state of the art review on intima-media thickness (IMT) measurement and wall segmentation techniques for carotid ultrasound
Comput. Methods Progr. Biomed.
Ultrasound IMT measurement on a multi-ethnic and multi-institutional database: our review and experience using four fully automated and one semi-automated methods
Comput. Methods Progr. Biomed.
Cardiovascular diseases in India
Circulation
Inflammation and atherosclerosis
Circulation
Comparison of cardiovascular disease risk calculators
Curr. Opin. Lipidol.
General cardiovascular risk profile for use in primary care: the Framingham Heart Study
Circulation
The UKPDS risk engine: a model for the risk of coronary heart disease in Type II diabetes (UKPDS 56)
Clin. Sci.
UKPDS 60: risk of stroke in type 2 diabetes estimated by the UK Prospective Diabetes Study risk engine
Stroke
Development and validation of QRISK3 risk prediction algorithms to estimate future risk of cardiovascular disease: prospective cohort study
BMJ
Development and validation of improved algorithms for the assessment of global cardiovascular risk in women: the Reynolds Risk Score
Jama
Risk assessment chart for death from cardiovascular disease based on a 19-year follow-up study of a Japanese representative population
Circ. J.
Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project
Eur. Heart J.
ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of cardiology/American heart association Task Force on practice guidelines
J. Am. Coll. Cardiol.
Simple scoring scheme for calculating the risk of acute coronary events based on the 10-year follow-up of the prospective cardiovascular Münster (PROCAM) study
Circulation
Associations between cardiovascular risk factors, inflammation, and progression of carotid atherosclerosis among smokers
Nicotine Tob. Res.
Carotid intima-media thickness measurement in cardiovascular screening programmes
Eur. J. Cardiovasc. Prev. Rehabil.
Cited by (42)
Deep-stratification of the cardiovascular risk by ultrasound carotid artery images
2024, Biomedical Signal Processing and ControlDeep learning artificial intelligence framework for multiclass coronary artery disease prediction using combination of conventional risk factors, carotid ultrasound, and intraplaque neovascularization
2022, Computers in Biology and MedicineCitation Excerpt :There were four common observations from these six studies: (i) none of them had used deep learning algorithms for the prediction of CVD risk, (ii) further, only two of them had used multiclass endpoints, (iii) none of them had done scientific validation in their ML or DL-based systems, (iv) two of the above studies used image-based phenotypes as part of covariates. Note that this was also present in previous studies that showed an increment in performance for CVD risk stratification by combining these phenotypes [28,75,91–94]. Deep Learning recurrent neural models like RNN and its more sophisticated counterparts LSTM and GRU are model architectures that are primarily used in tasks involving the prediction and processing of temporal sequences [95].
Understanding the bias in machine learning systems for cardiovascular disease risk assessment: The first of its kind review
2022, Computers in Biology and MedicineCitation Excerpt :There are popular calculators used for CVD risk assessment such as the QRISK3 calculator [10], Framingham risk score (FRS) [11], the systematic coronary risk evaluation score (SCORE) [12], the Reynolds risk score (RRS) [13], and the atherosclerosis cardiovascular disease (ASCVD) [14]. Typically, these calculators use conventional and indirect risk factors (or office-based biomarkers (OBBM)) and laboratory-based biomarkers (LBBM) [15,16]. These risk prediction algorithms follow specific guidelines such as the European Society of Cardiology (ESC) and the American College of Cardiology/American Heart Association (ACC/AHA) [17,18].
A machine learning framework for risk prediction of multi-label cardiovascular events based on focused carotid plaque B-Mode ultrasound: A Canadian study
2022, Computers in Biology and MedicineArtificial intelligence framework for predictive cardiovascular and stroke risk assessment models: A narrative review of integrated approaches using carotid ultrasound
2020, Computers in Biology and MedicineCitation Excerpt :Increased values of these phenotypes reflect an increase in CVD risk profiles, and so, they can potentially improve the CVD risk prediction models. Several recent studies have shown to combine the effects of carotid ultrasound image-based phenotypes (CUSIP) and conventional cardiovascular risk factors (CCVRF) for the prediction of the 10-year CVD/stroke risk [32–35,70–76]. Artificial intelligence (AI) techniques, including machine learning (ML) and deep learning-based algorithms, have gained popularity in the medical community [77–80] and showing signs for its usefulness in the future for tailoring the patient-specific conventional approaches for predictive and personalized medicine.
Narendra N. Khanna, MD, DM, FACC is Advisor to Apollo Group of Hospitals in India and is working as Senior Consultant in Cardiology & Coordinator of Vascular Services at Indraprastha Apollo Hospital, New Delhi.
Ankush D. Jamthikar, M.Tech. has received MTech in Communication System and currently working as PhD research scholar at Visvesvaraya National Institute of Technology, Nagpur, India.
Deep Gupta, PhD is an Assistant Professor in the Electronics & Communication Engineering Department, VNIT, Nagpur (India). He received his Ph.D. in Medical Image Processing form Indian Institute of Technology Roorkee, India.
Andrew Nicolaides, MS, FRCS, PhD (Hon) is currently the Professor Emeritus at Imperial College, London. He is the co-author of more than 500 original papers and editor of 14 books.
Tadashi Araki, MD received the MD degree from Toho University, Japan in 2003. His research topics include coronary intervention, intravascular ultrasound (IVUS) and peripheral intervention.
Luca Saba, MD is with A.O.U. Cagliari, Italy. His research interests are in Multi-Detector-Row Computed Tomography, Magnetic Resonance, Ultrasound, Neuroradiology, and Diagnostic in Vascular Sciences.
Dr. Elisa Cuadrado Godia, Ph. D., is with IMIM – Hospital del Mar, Passeig Marítim 25–29, Barcelona, Spain. She works with Neurovascular Research Group and her current research interests are in the field of carotid artery disease and cardio vascular diseases.
Aditya Sharma, MD is a cardiologist at University of Virginia, where he directs the anticoagulation clinic and the medical optimization clinic, which helps patients with vascular disorders manage their risk factors with medication. His research interests are Peripheral arterial disease, venous thromboembolism, and fibromuscular dysplasia.
Tomaz Omerzum, MD is currently working as at University Medical Centre Maribor, Slovenia. His Research interests are radiology, and cardiovascular medicine.
Harman S. Suri is currently pursuing his BS from Brown University, Providence, RI, USA. He worked in summers of 2015 in the area of telemedicine-based Autism industry at Behavioral Imaging, Boise, Idaho, USA.
Ajay Gupta, MD, MS currently working as associate professor of radiology and neuroscience at Weil Cornell Medical College New York, USA. His research is focused on neuroradiology.
Sophie Mavrogeni, MD, PhD currently working at Cardiology Clinic, Onassis, Athens, GREECE. Her research is focused on Non-ischemic Cardiomyopathy, Dystrophinopathies, myocarditis and rheumatic diseases.
Monika Turk, MD is currently working as Physician at Department of Neurology, University Medical Centre Maribor, Maribor, Slovenia.
John R. Laird, MD, FACC is with St. Helena Hospital, CA, USA. Professor Laird is an internationally renowned interventional cardiologist and his expertise is innovative procedures for carotid artery disease.
Athanasios Protogerou, MD, PhD currently working at the Department of Cardiovascular Prevention & Research Unit Clinic, National and Kapodistrian Univ. of Athens, GREECE. His research is focused on Cardiovascular prevention.
Petros P. Sfikakis, MD, is the Dean of the School of Medicine at National and Kapodistrian University of Athens, Greece, and a Professor of Internal Medicine and Rheumatology. One of his main research interest focus in the cardiovascular outcomes of patients with immune-mediated diseases.
George D. Kitas, MD, PhD, FRCP is director of Research & Development-Academic Affairs, Dudley Group NHS Foundation Trust, Dudley, UK. He is an Honorary Professor of Rheumatology at the Arthritis Research UK Epidemiology Unit.
Vijay Viswanathan MD, PhD, FRCP is currently working as a chairman and director of MV Hospital for Diabetes and M Viswanathan Diabetes Research Centre, Chennai, India. His Research interests are Endocrinology Diabetes and Metabolism, Diabetes and foot research.
Jasjit S. Suri, Ph.D., MBA, Fellow AIMBE is with AtheroPoint™, Roseville, CA, USA. He is an innovator, visionary, scientist and an internationally known world leader in the field of biomedical imaging. Dr. Suri is a recipient of Marquis Life Time Achievement Award.