Fully Automatic Coronary Calcium Score Software Empowered by Artificial Intelligence Technology: Validation Study Using Three CT Cohorts

Since Arthur Agatston and his radiologist colleague, Warren Janowitz, proposed the coronary artery calcium (CAC) score based on electron beam computed tomography (CT) in 1990 [1], the Agatston score has been used to assess the risk of cardiovascular clinical outcomes [2, 3]. More than 30 years after its introduction, CAC scoring still requires manual inputs from skilled professionals, which is associated with prolonged processing time [4, 5]. Before the era of deep learning, researchers proposed feature-based or atlas-based models (as well as combined feature- and atlas-based models) for CAC scoring automation [6, 7, 8, 9, 10]. Some researchers have recently proposed a deep learning-based automatic CAC scoring (CAC_auto) system, with the reliability of calcium measurement mainly being evaluated using per-patient comparison [11, 12, 13, 14] or per-vessel comparison [15]. Detailed error analysis is essential for applying any automatic algorithm to routine imaging workflows, and, for this, per-vessel and per-lesion analyses may be more suitable than per-patient analyses. Additionally, analyses of the causes of errors can help improve the algorithms for CAC_auto.

For the training of artificial intelligence (AI) algorithms based on the end-to-end training strategy [10, 11, 12], manual labels for the location of CAC (e.g., right coronary artery [RCA] or left main artery [LM]) are required. A weakness of this end-to-end training strategy may be that calcium near the coronary artery (e.g., cardiac valves or aortic wall calcium) can be mistaken for CAC. Moreover, the slice-by-slice inference of CAC may take a relatively long time [1]. If an algorithm can identify the coronary tree, aortic root, and cardiac ventricles automatically in a CAC CT image without intravenous contrast enhancement, CAC_auto can be performed easily by including only calcium that overlaps the coronary artery tree region.

We developed a deep learning-based fully automatic calcium scoring system using non-enhanced CT, the atlas-based CAC_auto. This study aimed to validate the atlas-based CAC_auto using previously published cardiac CT cohort data from a single institution with the manually segmented CAC scoring (CAC_hand) system as the reference standard.

The Institutional Review Board of Asan Medical Center (Seoul, Korea) approved this study, and the need for informed consent was waived due to its retrospective design (IRB No. 2018-1155). To minimize the possibility of unintentionally sharing information that can be used to reidentify private information, a subset of the data generated for this study are available at the center for open science websites (https://osf.io/mu5k8/).

Development of CAC_Auto

The concept of the atlas-based CAC_auto, empowered by deep learning technology, is illustrated in Figure 1. If we know the spatial information (e.g., coronary artery, aortic valve, left ventricular myocardium) of a CAC image, CAC_auto can be performed easily. To perform semantic segmentation on a non-enhanced CAC image, we required a ‘labeled dataset’ for the training of a segmentation algorithm. However, the labeling of coronary and non-coronary structures on a non-enhanced CAC image can be challenging because of the low attenuation differences between lesions. Therefore, we first generated a labeled mask on coronary CT angiography (CCTA) (Step 1 in Fig. 1). For the development dataset, 100 CCTA images (n = 100, mean age 64.1 ± 8.2 years; 76.3% male; mean body mass index, 25.3 ± 2.3 kg/m²; three CT vendors and six scanners) were chosen from a cohort used in a previously published investigation of CCTA-based myocardial segmentation [16]. All labeled masks in the development dataset were confirmed by a cardiac radiologist. The second step was the transfer of spatial information from CCTA to CAC images for obtaining the ‘labeled CAC dataset’ (Step 2 in Fig. 1) by image registration of the CCTA and CAC. The registration process, consisting of affine registration followed by B-spline registration, performed using the elastix [17, 18], was applied to align the CAC images to the CCTA images with the labeled masks. This allowed a voxel-by-voxel comparison of the registered images. The final step was to develop a deep learning algorithm for the semantic segmentation of CAC images (Step 3 in Fig. 1). The basic network architecture is based on the three-dimensional (3D) U-Net [19], which was designed to input the 3D patch extracted from volumetric CT images and output the 3D patch from the multi-label semantic images. The transferred coronary artery region mask was dilated twice with 2 × 2 × 1 morphological structuring elements to compensate for misregistration errors and include nearby regions. It interpolated the 3D CT images to have 1 mm resolutions in the x-, y-, and z-directions. The intensity of the voxels was adjusted with window width and level values of 350 and −100 Houndsfield units (HU), respectively. The network input consisted of 3D-patch-based CT images with 64 × 64 × 64 voxels for each, which were randomly sampled for training procedures, similar to the structure used in our previous study [20]. The loss function of the network was the cross-entropy loss function, and the optimizer used was RMSprop. The training was performed with eight mini-batches of the 3D patch, a 10⁻⁵ learning rate, and 105 iterations. The two 3D-patch-based U-Net models were trained separately for coronary artery regions and other surrounding structures, including the myocardium, left and right ventricle, aorta, papillary muscles, and rib. Calcium lesions were detected when the initial candidate was in contact with the coronary artery region, and it did not belong to other structures.

Fig. 1
Development process of the atlas-based coronary calcium scoring software.
Step 1. Manual labeling of coronary and non-coronary regions using CCTA. Step 2. Spatial information transfer from CCTA to CAC images without contrast enhancement using image registration. Step 3. Deep learning for semantic segmentation of CAC images. CAC = coronary artery calcium, CCTA = coronary computed tomography angiography, 3D = three-dimensional

• Click for larger image
• Download as PowerPoint slide

Validation Dataset

Three previously published CCTA cohort datasets were chosen for the CAC_auto validation dataset as follows: 1) asymptomatic population for health check-up (screening group; n = 2653) [21]; 2) symptomatic patients who underwent invasive fractional flow reserve measurements (FFR group; n = 222) [22]; and 3) patients with mitral valve prolapse who underwent preoperative CCTA (valve group, n = 145) [23]. Among the 3020 individuals in these cohorts, 35 were excluded because of the lack of CAC image data, previous coronary stent insertion, or overlapping of cohorts. Finally, 2985 individuals were included in the validation dataset. The baseline and CT imaging characteristics are presented in Table 1. The screening, FFR, and valve groups underwent CT examinations between 2008 and 2009, 2011 and 2015, and 2011 and 2013, respectively. Among the 2985 enrollees, 1761 (59.0%), 249 (8.3%), 582 (19.5%), 277 (9.3%), and 116 (3.9%) belonged to the CAC 0, 1–10, 11–100, 101–400, and > 400 risk categories, respectively. Four types of CT machines from two companies (GE, Siemens) were used in this study, and all the scans were taken in electrocardiography-triggered mode with a slice thickness of 2.5 or 3 mm.

Table 1
Characteristics of Validation Datasets

• Click for larger image
• Click for full table
• Download as Excel file

Automatic Calcium Scoring: CAC_Auto

CAC_auto can be performed via batch processing without user interactions. After all CT data were loaded in a desktop computer (Intel Core 17-8700 3.19 GHz, 32 GB RAM, NVIDIA GeForce GTX 1060 6 GB), a research prototype software (AVIEW CAC, Coreline Soft, Co. Ltd.) automatically determined the CAC score, and there was no need to open the CT images (Fig. 2). The processing time for CAC_auto was recorded, and all the CAC results were downloaded as a single comma-separated value file. The labeled mask for the CAC_auto was saved for per-lesion comparison with the CAC_hand mask.

Fig. 2
CAC_auto result for a patient with severe calcifications in both the aortic and mitral valves.
CAC_auto can differentiate the coronary artery tree mask, aortic root mask (AO), and other cardiac area masks for non-enhanced calcium scoring of computed tomography images (central image with color masks). It is possible to distinguish calcium located in the heart region from that in the coronary artery or other areas, such as the aortic valve and MA. Additionally, the coronary tree mask enables the localization of calcium within blood vessels (i.e., LM, LAD, LCX, and RCA). AO = aorta, CAC_auto = automatic coronary artery calcium scoring, LAD = left anterior descending artery, LCX = left circumflex artery, LM = left main artery, MA = mitral annulus, RCA = right coronary artery

• Click for larger image
• Download as PowerPoint slide

We developed a novel atlas-based CAC_auto system and validated it using three CT angiography cohort datasets and a manually generated reference standard (CAC_hand). The major findings of this study were as follows: 1) CAC_auto demonstrated high reliability for the coronary calcium score (ICC 0.99) and volume (ICC 0.99) measurements, and it yielded a high accuracy for the assignment of risk categories (kappa value 0.94); 2) during the lesion-based analysis, CAC_auto provided a high sensitivity (93.3%) and a low false-positive rate (0.11 false-positive lesions per patient); 3) among the 333 false-positive lesions, the true anatomical false-positive rate, except for artifacts and human errors, was as low as 0.06 lesions per patient. Several of the anatomical errors were caused by the aortic wall and pericardial calcium, and the extracardiac error rate was low.

We followed the atlas-based approach [8, 9] but incorporated a deep-learning-based semantic segmentation model to replace the time-consuming non-rigid registration of the multi-atlas method. The false-positive rate of 0.11 per patient in the present study was significantly lower than what was reported previously for the lesion-based analysis by Kurkure et al. [7] (4.7 false positives per patient), Shahzad et al. [9] (1.0 to 1.5 false positives per patient), and Wolterink et al. [10] (0.2 false positives per patient). Recently, two studies adopting deep learning-based methods reported high reliability for CAC_auto defined by ICC and kappa values [11, 12], which was comparable to that of the present study. However, the investigators did not perform per-lesion analyses. Moreover, the sample sizes of electrocardiogram-synchronized CT scans were relatively small: 87 scans by van Velzen et al. [11] and 511 scans by Martin et al. [12]. Supplementary Table 1 summarizes the previous publications [6, 7, 8, 9, 10, 11, 12, 13, 14, 24] related to CAC_auto. The present study used the largest electrocardiogram-synchronized CT dataset (n = 2985) and showed comparable results to those of state-of-the-art studies, as reflected in the ICC (0.99), per-lesion sensitivity (93.3%), and false-positive rate per CT scan (0.11).

Isgum et al. [6] proposed a feature-based method that extracted the size, shape, spatial, and appearance features from calcium candidate objects and applied classifiers to minimize the false-positive rate. Kurkure et al. [7] also utilized features and extended the classifier model in a two-stage mode. These feature-based approaches do not require the spatial information of coronary arteries. However, the results were not comprehensive and required careful selection of several operational parameters for calcium candidate object detection and classification. In another method, Brunner et al. [8] presented an atlas-based approach using coronary artery region models. This method incorporates image transformation to align cardiac volumes across patients and provides coronary artery zones and sections. However, the performance of this atlas-based method was inferior to that of feature-based methods. To combine the benefits of feature- and atlas-based methods, Shahzad et al. [9] proposed a multi-atlas-based method. Spatial information was obtained from the pairwise registration of the 10 atlas images. However, this multi-atlas-based method had a scalability problem. The method was slower when more atlases were included because of the heavy computation of non-rigid registration. In our previous study, when 100 atlases were used, it took approximately 30 minutes [25]. In the present study, to solve the problem of atlas-based methods using registration, we incorporated deep learning-based semantic segmentation to reduce the execution time.

Some researchers have recently proposed deep-learning-based CAC_auto techniques [6, 11, 12]. Lessmann et al. [13] used a convolutional neural network (CNN) to classify calcium candidate objects. In this approach, two 2.5D CNNs with different receptive fields were consecutively applied. This algorithm can eliminate the feature extraction step in previous approaches and can be trained on large datasets. However, this did not provide spatial information about the coronary artery regions or other surrounding structures. It also requires more computation time to apply voxel-by-voxel classification. Martin et al. [12] presented a multistep deep-learning model. The first step was used to identify and segment the regions, such as the coronary artery, aorta, aortic valve, and mitral valve. The second step classified the voxels as coronary calcium. Zhang et al. [15] reported a multi-step deep learning model, which incorporated finding the calcium candidate region in the axial image and performing a false-positive reduction using shape constraints. The incremental value of this current approach is that it can precisely detect coronary artery regions with a deep learning model based on semantic segmentation in a single step. This method can also provide regional information on the coronary artery and surrounding structures, such as the aorta, ventricular chambers, and myocardium. Therefore, this method can be easily extended to the segmentation of the aortic and mitral valves. On the other hand, the previously used methods share a structural similarity, related to the initial candidate detection and false-positive reduction [12, 13, 15]. In the first part, a convolutional deep learning model was used to determine whether the region of interest was a coronary artery and it included the aorta, aortic valve, and mitral valve. In the latter part, a convolutional deep learning model was applied to classify whether calcium candidates above 130 HU were coronary calcium. The latter part was added because the number of false positives was large only for the first part.

Although CAC_auto exhibited high reliability in this study, there were still some outliers, as well as false-positive and false-negative errors. The limits of agreement in the high CAC (> 400) group were still high (37.7 ± 197.0) despite a high ICC value. Because the ICC is calculated using variance, the ICC value tends to increase when the data range is wide [26]. Further studies to improve the CAC_auto algorithm are warranted. Two approaches for the improved algorithm were considered. The first approach involves image quality improvement using deep learning. The most frequent reason for mismatched lesions in this study was image noise or motion artifacts. In our previous study [27], we effectively reduced noise in low-dose cardiac CT. The second approach is to improve the training dataset in terms of size and quality. Owing to the nature of AI technology, generalization can be increased when various types of datasets are included during training. In this study, 100 cases of hard-labeled (pixel-by-pixel labeling) data were used for training. We are considering not only adding more training data but also applying data augmentation to capture the variability of the heart or aorta shape due to severe cardiovascular disease. Moreover, a reliable but straightforward verification step for CAC_auto, such as multiple color-coded volume rendering of CAC (Fig. 2), would help shorten the time needed for human confirmation in a CAC_auto workflow. Second, although our study included three different cohort datasets representing three different scenarios and four CT machines, it was still fraught with the limitations of single-center retrospective studies. In addition, a relatively small number (n = 116) of patients with high CAC (> 400) would not be sufficient for AI model validation. Therefore, further multicenter investigations covering larger varieties of CT machines, disease spectrum, and CAC imaging protocols are required. Third, the present study did not evaluate whether CAC_auto streamlined the actual CAC imaging workflow. In follow-up studies, it will be necessary to confirm that CAC_auto can reduce the analysis time while maintaining accuracy when compared with manual work. Finally, since our study used an atlas-based method, theoretically, mitral annular calcification and aortic valve calcification could be automatically evaluated. Considering recent research that reported a link between structural heart disease and non-coronary calcification [28], automatic quantification of valvular or paravalvular calcium would also be promising. Further investigation of the automatic quantification of non-coronary calcifications is warranted.

In conclusion, the atlas-based CAC_auto empowered by deep learning provided accurate calcium score measurement as compared with manual method and risk category classification, which could potentially streamline CAC imaging workflows.

The Supplement is available with this article at https://doi.org/10.3348/kjr.2021.0148.

Supplementary Table 1

Comparison of Studies of Methods Designed for Automatic Calcium Scoring in CT

Supplementary Fig. 1

Scatter plots showing the correlation between manual CAC scoring (CAC_hand) and automatic CAC scoring (CAC_auto).