Performance evaluation of 10-year ultrasound image-based stroke/cardiovascular (CV) risk calculator by comparing against ten conventional CV risk calculators: A diabetic study

Abstract

Motivation

AtheroEdge Composite Risk Score (AECRS1.0_10yr) is an integrated stroke/cardiovascular risk calculator that was recently developed and computes the 10-year risk of carotid image phenotypes by integrating conventional cardiovascular risk factors (CCVRFs). It is therefore important to understand how closely AECRS1.0_10yr is associated with the ten other currently available conventional cardiovascular risk calculators (CCVRCs).

Methods

The Institutional Review Board of Toho University approved the examination of the left/right common carotid arteries of 202 Japanese patients. Step 1 consists of measurement of AECRS1.0_10yr, given current image phenotypes and CCVRFs. Step 2 consists of computing the risk score using ten different CCVRCs given CCVR factors: QRISK3, Framingham Risk Score (FRS), United Kingdom Prospective Diabetes Study (UKPDS) 56, UKPDS60, Reynolds Risk Score (RRS), Pooled cohort Risk Score (PCRS or ASCVD), Systematic Coronary Risk Evaluation (SCORE), Prospective Cardiovascular Munster Study (PROCAM) calculator, NIPPON, and World Health Organization (WHO) risk. Step 3 consists of computing the closeness factor between AECRS1.0_10yr and ten CCVRCs using cumulative ranking index derived using eight different statistically derived metrics.

Results

AECRS1.0_10yr reported the highest area-under-the-curve (0.927;P < 0.001) among all the risk calculators. The top three CCVRCs closest to AECRS1.0_10yr were QRISK3, FRS, and UKPDS60 with cumulative ranking scores of 2.1, 3.0, and 3.8, respectively.

Conclusion

AECRS1.0_10yr produced the largest AUC due to the integration of image-based phenotypes with CCVR factors, and ranked at first place with the highest AUC. Cumulative ranking of ten CCVRCs demonstrated that QRISK3 was the closest calculator to AECRS1.0_10yr, which is also consistent with the industry trend.

Introduction

Cardiovascular diseases (CVD) including heart and stroke are the major cause of mortality and morbidity around the world [1]. The spread of CVD is almost similar in every region of the world including developed and developing countries [2,3]. Atherosclerosis is the primary cause of cardiovascular disease [4]. Coronary and carotid arteries are the main sources that supply the oxygenated blood to the heart and the brain, the two main organs that are responsible for heart attack and stroke. Deposition of atherosclerotic plaque within the vessel wall obstructs this blood supply causing myocardial infarction (MI) or stroke. According to INTERHEART¹ [5] and INTERSTROKE¹ [6] studies, 90% of the mortalities due to these CVD are attributed to the conventional cardiovascular risk (CCVR) factors such as age, ethnicity, systolic blood pressure (SBP), low-density lipoproteins cholesterol (LDL-C), total cholesterol (TC), smoking, diabetes mellitus (DM), and family history (FH). These risk factors have an additive effect on atherosclerotic vascular biomarkers causing an elevation in CVD/Stroke risk.

Preventive interventional methods (i.e., use of statins) require the determination of baseline as well as the long-term risk of a patient to prevent the onset of CVD. Risk estimation can be possible using cardiovascular (CV) risk calculators. From here on, we will use CCVRCs for “conventional cardiovascular risk calculators”. At the present stage, there are more than 100 CCVRCs available [7]. Most prominently and clinically used CCVRCs are FRS [8], UKPDS56 [9], UKPDS60 [10], QRISK3 [11], RRS [12], NIPPON [13], SCORE [14], WHO [15], PCRS (Atherosclerosis CVD - ASCVD) [16], and PROCAM [17] (The expansions of the abbreviation is shown in Appendix-Table 1). This was the main motivation for using only these ten CCVRCs in this study to evaluate the performance of AECRS1.0_10yr and to further study which calculator was most closely related to AECRS1.0_10yr. In this manuscript, the suffix ‘10yr’ indicates the risk value after ten years. CCVR calculators generally provide a risk estimation based on traditional risk factors. Recent guidelines indicate the use of some primitive CV risk calculators to assess the 10-year risk of CVD and to decide the statin eligibility for the patients [[18], [19], [20]]. Although such CCVRCs can provide the basis for the prescription of statins (i.e., lipid-lowering medications) for treating the risk of CVD, they sometimes do the not explain the CV events due to the morphological variations in the atherosclerotic blood vessels. Furthermore, CCVR factors do not provide any information about the arterial plaque variations or their components. However, since CCVR factors contribute to the risk of stroke/coronary heart disease (CHD), they are also termed as risk factors.

Advancements in imaging techniques especially non-invasive imaging modalities such as carotid ultrasound provide a cost-effective tool to assist physicians in understanding the morphological variations of the blood vessels due to atherosclerotic plaque tissues [19,21]. For example, carotid intima-media thickness (cIMT) and total plaque area (TPA) are the two important image-based phenotypes of carotid atherosclerosis that can provide information about the cardiovascular health of patients [[22], [23], [24], [25], [26], [27]] Note that there were a total of 28 CCVR factors considered in this study, and their abbreviations are all expanded in Appendix-Table 2. Since 2000, seven clinical guidelines have been recommended for using cIMT and/or carotid plaque for assisting the CVD risk estimation [28]. In 2010, Nambi et al. [29] demonstrated a significant improvement in CHD by taking into consideration the image-based phenotypes such as cIMT and carotid plaque. Stein et al. [30] also recommended the use of cIMT for CHD risk prediction. Polak et al. [31] presented a study with 296 participants and reported that CV outcomes can be predicted using the mean and the maximum cIMT. Association between intima-media thickness (IMT) in the common carotid artery (CCA) and coronary SYNTAX score was demonstrated by Ikeda et al. [32] in their study of 500 diabetic patients. The same group showed the link between carotid bulb plaques and coronary SYNTAX score in diabetic patients [33]. Recently, Suri et al. [34] showed the role of cIMT and variations in IMT (IMTV) to predict the Leukoaraiosis disease in adults. Thus, there is an additional benefit by considering the carotid image-based phenotypes in the risk prediction models along with CCVR factors for risk estimation.

Conventional cardiovascular risk calculators do not consider carotid image-based phenotypes in their risk prediction models. Therefore, the extent of atherosclerotic plaque measured using average intima-media thickness (IMT_ave), maximum IMT (IMT_max), minimum IMT (IMT_min), IMT variability (IMTV), and the morphology or echolucency [35] of the plaque (such as hyper- or hypo-echoic intensities of the grayscale images [36]) remains unexplained by such as CCVRCs. Note that both carotid plaque or wall thickening and CCVR factors are time (or age) dependent [[37], [38], [39]]. Thus, one can compute the 10-year risk of stroke/CVD by knowing the projection rates of the image-based phenotypes due to the conventional risk factors and then integrating them with current image-based phenotypes. These 10-year image-based phenotypes (such as IMT_ave10yr, IMT_max10yr, IMT_min10yr, and IMTV_10yr) lead to the formation of the overall risk or composite risk score (CRS). This is known as an integrated calculator, integrated in the sense that conventional risk factors are integrated or fused with image-based phenotypes. Since our image-based phenotypes, to begin with, were measured automatically using AtheroEdge (AE) system (AtheroPoint, Roseville, CA, USA), we call this composite risk score as AECRS1.0_10yr.

The main objective of the study is to evaluate AECRS1.0_10yr (an integrated stroke/CV risk calculator) by comparing against the 10-year CCVRCs using several statistical metrics. The theory of these metrics is to evaluate the metric criteria given the CCVR factors for each patient corresponding to these eleven CV risk calculators (ten CCVRCs and the integrated calculator - AECRS1.0_10yr). These metrics are either based on how well the CCVR calculators find the high-risk patients in terms of similarity with AECRS1.0_10yr (so-called, similarity metric represented by a metric type of M1), or how far AECRS1.0_10yr is from the ten CCVR calculator constellations or cluster (clustering metric represented by a metric type of M2), or how the overall cohort of patients (or population) is doing in terms of its precision when compared against AECRS1.0_10yr (precision-of-merit metric represented by a metric type of M3), or ability to find the mean statistics for all CCVRCs against AECRS1.0_10yr (figure-of-merit metric represented by a metric type of M4), or how each patient's risk in CCVRC deviates against AECRS1.0_10yr (metric-type M5), or how each patient's risk in AECRS1.0_10yr deviates against CCVRC (metric-type of M6), or adapt kappa statistics to see the disparity between readings (kappa metric-type of M7), or the ability to use logistic regression and rank CCVRCs using odds ratio against AECRS1.0_10yr (metric-type M8). The overall idea is to understand which CCVRC is ranked closer to the AECRS1.0_10yr. The list of the eight statistical metrics and expansion of the abbreviations and symbols pertaining to performance metrics are listed in Appendix-Table 3 and Appendix-Table 4, respectively.

There are two hypotheses in this study: (i) The calculator which has a direct impact on atherosclerosis disease should show the highest area-under-the-curve (AUC) and (ii) we believe that the calculator that uses a larger number of CCVR factors is likely to be closer to the AECRS1.0_10yr risk calculator, since AECRS1.0_10yr is an integrated calculator. Our current study has several novelties: (i) a carotid ultrasound image-based phenotypes 10-year risk prediction model called AECRS1.0_10yr has been proposed; (ii) performance of AECRS1.0_10yr was evaluated against CCVRCs (Fig. 1); and (iii) a novel performance evaluation scheme consisting of eight statistical metrics has been proposed to compare the closeness of AECRS1.0_10yr against the ten CCVRCs.

The layout of the paper is as follows. Section 2 discusses the study population and image acquisition protocol. The methodology for the measurements of carotid image-based phenotypes has been discussed in section 3. The statistically derived metrics for evaluating the closeness factor between AECRS1.0_10yr and CCVR calculators have been presented in section 4. Results are presented in section 5. Discussion and benchmarking have been discussed in section 6. The paper concludes in section 7.