Introduction

Aortic stenosis (AS) is the most common valvular heart disease with increasing prevalence in the elderly population in Europe and North America1,2. For its treatment, transcatheter aortic valve replacement (TAVR) has become a well-established option in various patient groups3,4,5,6. Sufficient pre-procedural anatomical visualisation, quantification of the structures of interest and subsequent adequate valve size selection are all crucial to reduce the incidence of complications such as valve prosthesis dislocation or annulus rupture by an under- or oversized prosthesis7,8. Due to its high spatial resolution, multidetector computed tomography (MDCT) has become the standard pre-procedural imaging method in patients undergoing TAVR9. Previous studies showed high levels of inter-observer agreement in terms of MDCT measurements and valve size selection, but were primarily focussing on readers with existing imaging experience without experience in structural intervention10,11. To date, national guidelines do not define how much imaging experience is required to analyse pre-procedural MDCT scans, or the qualifications necessary to perform the analysis in order to select an appropriate prosthesis size12. Therefore, we carried out a comprehensive assessment of MDCT post-processing analyses of reproducibility and accuracy of valve selection in different healthcare professionals including two structural interventionalists, an imaging specialist, a cardiac surgeon, a general physician and a medical student in a tertiary cardiology hospital in order to define minimum post-processing competence levels and provide guidance on required performance for pre-procedural planning.

Methods

Twenty patients with severe AS as confirmed by echocardiography and who underwent TAVR in 2019 were enrolled; patients with bicuspid AS were excluded. This study was conducted according to the principles of the Helsinki Declaration.

Multidetector computed tomography

A dual-source CT scanner (SOMATOM Force, Siemens Healthcare GmbH, Erlangen, Germany) was used to generate contrast-enhanced MDCT scans in a prospectively ECG-triggered high-pitch spiral acquisition mode. The region of interest extended from the clavicles to the femoral heads. CT angiography was performed with bolus tracking in the descending aorta using a contrast agent bolus of 80 ml (Imeron 350, Bracco Imaging, Konstanz, Germany) followed by a 40 ml saline chaser, both at a flow rate of 4 ml/sec. Scan parameters were as follows: 2 × 192 × 0.6 mm collimation, 250 ms rotation time, pitch of 3.2, automated tube current adaption. A small field of view data set with medium soft convolution kernel (Siemens Bv36), 0.75 mm slice thickness and 0.5 mm slice increment was generated for the assessment of the aortic annulus, root, and valve morphology and dimensions. All data were analysed using dedicated software (3 Mensio, Structural Heart, V9.1., Pie Medical Imaging, Maastricht, Netherlands).

Study design

Five different healthcare professionals of varying levels of experience were involved in this study to assess the impact of experience on post-processing of MDCT scans. All observers focused their analysis on the structures which determine valve selection, whereas the vascular approach was not considered. The parameters obtained in this study are displayed in Table 1. The group of observers included two interventional structural heart cardiologists (reference structural interventionalist and a second one), both with more than 5 years of experience in TAVR, one cardiac imaging specialist (cardiologist by training), one cardiac surgeon with more than 5 years of experience in TAVR and SAVR, one general physician and one medical student. All observers were given a brief training on the software provided by the company. This training was comprised of a presentation on how to use the software including an exemplary demonstration of an evaluation in one patient. In addition, all observers underwent a short briefing to standardize the measurements according to the study protocol. All scans were analysed twice in a standardized approach by each observer (with exception of the structural interventionalist two), with at least 4 weeks between the two runs, to assess intra-observer variability in regard to the measured parameters (aortic annulus area, aortic annulus area derived diameter, aortic annulus diameter average, aortic annulus perimeter, sinus of valsalva diameter). The measurements of the structural interventionalists two were used to assess the inter-observer variability in terms of MDCT measurements as well as valve size selection.

Table 1 Measured parameters using 3 Mensio for structural heart disease.
Full size table

After each run, each observer had to choose the valve size for the Medtronic CoreValve (23 mm, 26 mm, 29 mm, 34 mm) and the Edwards SAPIEN 3™ valve (23 mm, 26 mm, 29 mm) that were used at the Heart Centre Göttingen, at that time. The selection was based on the measurements and a recommendation sheet provided by the manufacturer (Tables 2 and 3). To determine the inter-observer variability, the first run measurements of the observers were compared to the structural interventionalist, who was considered as reference.

Table 2 Size selection sheet for Edwards SAPIEN 3™, image source: “With the kind permission of Edwards Lifesciences”.
Full size table
Table 3 Size selection sheet for Medtronic CoreValve; image source: “With the kind permission of Medtronic GmbH”.
Full size table

Statistics

Statistical analysis was conducted using Microsoft Excel 2016 (Microsoft Corporation, Redmond, Washington USA) and IBM SPSS Statistics version 26 for Windows (International Business Machines Corporation (IBM® Corp.), Armonk, New York, USA). Data are expressed as mean ± standard deviations. Normal distribution was tested using the Shapiro–Wilk test. Non-normally distributed data were compared using Mann-Withney U and Kruskal–Wallis tests as appropriate. For between-group comparisons in normally distributed data, t- or ANOVA testing were carried out as appropriate. P-values provided are two-sided, an alpha level of ≤ 0.05 was considered statistically significant.

Furthermore, intra- and inter-observer variability was assessed using three different methods: intraclass correlation coefficients (ICC), Bland Altman analysis, and coefficients of variation (CoV). Bland Altmann analysis reveals “mean differences”. When the compared measurements revealed exactly the same result, all the differences would be equal to zero. A deviation to zero represents the average deviation of measurement x to measurement y13. The CoV was defined as the standard deviation of the differences divided by the mean14,15. The level of agreement was defined as follows: excellent for ICC > 0.74, good for ICC 0.60–0.74, for ICC 0.40–0.59, and poor for ICC < 0.416.

Ethical approval

This study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the local ethics committee of the University Medical Center Göttingen (10/5/16). Informed consent was obtained from all subjects involved in the study.

Results

Demographics

Patients´ characteristics are displayed in Table 4. A total of 14 (70%) patients were male. Mean age was 78 ± 6 years and ranged from 61 to 90 years. AS was confirmed in all cases by echocardiography with a mean transvalvular peak velocity (V max) of 4.1 ± 0.7 m/s and an average transvalvular mean gradient of 40.3 ± 14.4 mmHg. The estimated mean aortic valve area was 0.7 ± 0.3 cm2.

Table 4 Patients´ characteristics.
Full size table

The most common comorbidity was hypertension (95%), followed by atrial fibrillation and coronary artery disease (70% and 65%, respectively). Diabetes mellitus was present in 4 patients (20%) and chronic obstructive pulmonary disease in 2 patients (10%).

Assessment of aortic anatomy for valve size selection

Mean annulus area measured by the different observers ranged between 502.87 mm2 and 571.32 mm2, which resulted in significant differences between the observers (p < 0.001). The mean aortic annulus area derived diameter was measured between 25.1 ± 2.9 mm and 26.8 ± 3.3 mm and differed significantly between the observers (p < 0.001). The aortic annulus area derived annulus diameter differed numerically only minimally from the measured averaged aortic annulus diameter and ranged between 25.4 ± 2.7 mm and 27.0 ± 3.4 mm. Mean annulus perimeter varied from 80.3 ± 9.1 mm to 91.0 ± 11.9 mm with significant differences between the observers (p < 0.001). The average measured diameter of the sinus of valsalva (SOV) ranged between 33.3 ± 4.5 mm and 33.6 ± 4.9 mm (p = 0.504). Figure 1A–D illustrates the measured values by the 5 observers and also indicates significant inter-observer differences for all aforementioned parameters (online data supplement Figure S1 includes also structural interventionalist two).

Figure 1
figure 1

Inter-observer measurement presentations of (A) Aortic annulus area; (B) Aortic annulus average diameter; (C) Aortic annulus perimeter; (D) Sinus of Valsalva; p-values < 0.05 indicate significant differences.

Full size image

Valve size selection

Overall, for the Edwards SAPIEN 3™ valve the medium-sized valve (26 mm) was chosen most frequently (49%) followed by the largest one (29 mm, 38%) and the smallest one (23 mm, 13%). With regard to the Medtronic CoreValve, the most frequently selected valve size was the largest (34 mm, 52%) followed by the medium (29 mm, 40%) and small (26 mm, 8%) sizes. The smallest Medtronic CoreValve (23 mm) was not selected by any of the operators. There was no difference on an individual observer level. Details are given in the online data supplement (online data supplement Table S1–S2).

Intra- and inter-observer variability of CT measurements

Excellent intra-observer agreement was seen for all observers (Table 5 and online data supplement Figure S2–S6). The structural interventionalist showed the best intra-observer agreement and reproducibility for all measured MDCT parameters, with the exception of SOV. The medical student had the lowest reproducibility in 3 out of 5 categories (aortic annulus area derived diameter, aortic annulus average diameter and aortic annulus perimeter). Numerically small but statistically significant differences were observed between the initial and the repeated analysis runs for the following parameters:

Table 5 Intra-observer agreement and reproducibility.
Full size table

1. Annulus area, aortic annulus area derived diameter, aortic annulus average diameter and aortic annulus perimeter in the measurements by the cardiac surgeon; 2. The aortic annulus perimeter in the measurements of the medical student; 3. The SOV measurements of the general physician (online data supplement Figures S7–S10).

There were no statistically significant differences between the two runs for the structural interventionalist and the imaging specialist.

Compared to the structural interventionalist the inter-observer agreement and reproducibility were excellent for all analysed data including the analyses of the structural interventionalist two, the imaging specialist, the cardiac surgeon and the general physician. In addition, the medical student showed also excellent reproducibility for 4 out of 5 recorded parameters. Only the aortic annulus perimeter agreement did not reach excellent inter-observer reproducibility and was considered “good” compared to the structural interventionalist. The highest agreement with the structural interventionalist was observed for the second structural interventionalist followed by the imaging specialist, the general physician and the cardiac surgeon (Table 6).

Table 6 Inter-observer agreement and reproducibility.
Full size table

Intra- and inter-observer variability for valve size selection

All observers reached an excellent level of intra-observer agreement for valve size selection. When estimating the ICC without taking into account the different valve manufacturers, excellent intra-observer agreements were found for all observers, with the highest intra-observer agreement for the structural interventionalist [ICC 0.991, 95% CI 0.983–0.995], followed by the cardiac surgeon [ICC 0.944, 95% CI 0.893–0.970] the general physician [ICC 965, 95% CI 0.934–0.982], the imaging specialist [ICC 0.909, 95% CI 0.827–0.952] and the medical student [ICC 0.910, 95% CI 0.830–0.952]. When assessing in terms of manufacturers, the ICC for selection of the Edwards SAPIEN 3™ valve ranged between 0.776 and 0.945, whereas ICC for the Medtronic CoreValve varied between 0.816 and 0.989. The structural interventionalist had the best reproducibility for both valve types (Edwards SAPIEN 3™: ICC 0.945, 95% CI 0.826–0.978, Medtronic CoreValve: ICC 0.989, 95% CI 0.973–0.996). A detailed overview of the intra-observer ICC is shown in Table 7.

Table 7 Intra- and inter-observer ICC for valve selection.
Full size table

In terms of the inter-observer agreement excellent agreement was found for the structural interventionalist two [ICC 0.984, 95% CI 0.969–0.991], the cardiac surgeon [ICC 0.947, 95% CI 0.900–0.972], the general physician [ICC 0.944, 95% CI 0.895–0.971] and the imaging specialist [ICC 0.933, 95% CI 0.873–0.964] while the medical student only reached a fair agreement [ICC 0.507, 95% CI 0.067–0.739] compared to the reference standard (structural interventionalist). When assessing according to the different valve manufacturers, the highest level of agreement was found for the structural interventionalist two (Edwards SAPIEN 3™: ICC 0.977, 95% CI 0.941–0.991; Medtronic CoreValve: ICC 0.966, 95% CI 0.915–0.987) followed by the cardiac surgeon with an ICC of 0.871 [95% CI 0.626–0.951) for the Edwards SAPIEN 3™ valve. The lowest agreement was found for the medical student [ICC 0.737, 95% CI 0.092–0.909). In regard to the Medtronic valve size selection, there was excellent agreement at comparable levels for all observers, with the highest for the medical student [ICC 0.897, 95% CI 0.723–0.960] and the lowest for the imaging specialist [ICC 0.867, 95% CI 0.670–0.947). A detailed representation of the ICC is given in Table 7. Furthermore, Fig. 2 depicts the valve size selection agreement of the different observers and the structural interventionalist, whereas Figure S11 of the online data supplement illustrates the annulus area measurements in three selected patients where the measurements resulted in different valve size selections as compared to the structural interventionalist.

Figure 2
figure 2

Valve size selection compared to the interventionalist for each patient, separated by valve type.

Full size image

Discussion

To our knowledge, this is the first study investigating the impact of the experience of different healthcare professionals involved in the field of MDCT measurements and valve size selection prior to TAVR. We included several different healthcare professionals: two structural interventionalists, a cardiac surgeon, an imaging specialist, a general physician and a medical student. The following findings are noteworthy: First, MDCT TAVR planning analyses were feasible and robust, with minimal differences between all observers studied, after they all received a brief introduction to the analysis software. Second, among the different observers (excluding the second structural interventionalist) the imaging specialist had the highest and the medical student the lowest agreement with the reference structural interventionalist. Third, in terms of valve size selection the structural interventionalist had the highest level of consistency between repeated analyses which was superior to all other observers. This suggests that besides an adequate analysis of pre- TAVR MDCT scans, clinical and interventional experience should be a prerequisite for consistent and safe procedural planning. Taken together, high quality measurements of the relevant aortic anatomy can be obtained relatively easily based on state-of-the-art MDCT TAVR planning scans, which can then be utilized for accurate determination of the intervention.

It is known that with increasing experience—defined by the number of TAVR procedures performed—the rate of periprocedural complications (mortality, vascular complications, bleeding) decreases17. An accurate selection of the valve size also contributes to a reduced complication rate. Therefore, accurate pre-procedural imaging is crucial to assure optimal patient outcome. Transoesophageal echocardiography was initially used to estimate the right valve size, but nowadays MDCT has become the standard procedure, which has led to a reduction in paravalvular regurgitation18,19. In line with previous MDCT studies, our study demonstrates excellent intra-observer and inter-observer results for all observers involved, independent of the level of experience, when following a standardized analysis procedure10,11,19,20,21,22,23. Due to the high measurement accuracy achieved with MDCT, it initially seems counterintuitive that there were differences in the selection of the valve sizes between and within the observers in the current study. One possible explanation may be that the structural interventionalists traced the border of the region of interest consistently within the outer contrast region, while the other observers delineated the borders around the contrast region resulting in slightly smaller measurements of the interventionalists (please see Figure S11). In cases with a high AVC load the interventionalists bisected the calcifications considering the blooming artefact, which was not as consistently performed by the other observers. Both may have impacted the measurements slightly and consequently valve size selection. A further possible explanation is, that valve size selection is not based on one single parameter but rather, when using the recommendation sheet, on up to four different parameters. There is a slight overlap where one or the other valve size may be chosen (Tables 2 and 3). Therefore, differences may be explained by cases in which the different parameters do not fit one size only, or in the presence of measurements between 2 valve sizes. Horehledova et al. showed, that only 30 to 60% of annulus diameter, annulus area and annulus perimeter measurements indicated the same prosthesis size24. In such cases, the level of clinical experience observer becomes more relevant. This may be why the best intra-observer agreement regarding valve size selection was observed for the structural interventionalist, who encounters similar scenarios in their clinical routine and thus targets them in a more structured manner. This may also be the explanation why the structural interventionalist always selected the same valve size in cases of borderline levels of the sizing parameters, while the other observers were not as consistent in this aspect. In addition, the level of experience of the observer most likely also plays a role in the measurements: a higher agreement was again seen in our study for the structural interventionalist compared to the other observers. While non-interventionalists probably relied mostly on their measured parameters and selected the valve size accordingly, the structural interventionalists may have also intuitively based their decisions on appearance and anatomy, e.g. calcium distribution. However, to reduce peri-interventional complications during TAVR, aspects additional to the dimensional measurements (Table 1) have to be considered. These include the shape and size of the aortic annulus, the LVOT ascending aorta angle as well as the extent and distribution of aortic valve calcification. A comprehensive assessment of all parameters involved make it possible to reduce the risk of paravalvular regurgitation, which is associated with worse outcome25. Therefore, it is not surprising that we observed differences in valve size selection solely based on MDCT aortic annulus measurements by the observers who were not specialized in structural interventions. The same is true for reducing the risk of a rare but life-threatening aortic annulus rupture. The risk of annulus rupture is associated with high amounts of LVOT calcification and also with prosthetic valve oversizing26. All these aspects are not represented in the valve size selection recommendation sheet; they are solely based on clinical experience. There is a multitude of evidence that dedicated training can further improve operator performance27,28,29. Therefore respective programs should be defined and incorporated in relevant guidelines. Nevertheless the excellent intra-observer reproducibility of the aforementioned parameters for all observers is quite reassuring in that these clinical decisions were based on very robust data that can be accurately and safely assessed and then interpreted by an experienced physician.

The software for structural heart disease used in the current study offers an excellent intra- and inter-observer reproducibility in regard to aortic valve measurements. This is likely due to its simplicity and the usability of the program as well as the good image quality of the MDCT technology, which, in contrast to echocardiography, is independent of the examiner. Taken together, successful pre-TAVR MDCT imaging should always be analysed by the actual implanting physician with subsequent valve size selection to allow for safe planning of the procedure based on individual anatomical and clinical patient data. In this regard, our study suggests that clinical experience is likely more important for adequate decision making as compared to the MDCT measurements which can be very accurately performed with generation of highly reproducible measurements after a brief introduction to current MDCT software programs.

Furthermore, borderline decisions on valve size selection will never fully rely on standardized measurement recommendation sheets but rather on the observer’s experience. In clinical practice steps for appropriate selection of a device in borderline cases should always include: 1. a repeated analysis of the annulus area and annulus perimeter to rule out measurement errors; 2. the amount and distribution of calcification and the height of the coronary arteries should be taken into consideration; 3. in situations with large calcifications in the device landing zone ,the smaller valve should be chosen to reduce the risk of an annulus rupture; 4. if there is only a small amount of calcification, the larger valve size should be considered because of a resulting larger valve area and a lower risk of a paravalvular regurgitation; 5. if possible, in borderline situations a second experienced structural interventionalist should be involved with subsequent selection of the adequate valve size in a team approach.

Limitations

Some limitations of our study need to be addressed. First, our results are derived from a single vendor software package and may not apply to different analyses tools. However, the software package used is well established in clinical routine, and thus our results can be considered clinically meaningful. Second, the selection of the valve size was only based on the measurements of the annulus and the SOV and did not take the vascular access or coronary anatomy into account. However, since this was true for all observers, this bias can be considered a systematic bias. Third, each group of “differently experienced healthcare professionals” (aside from the structural interventionlists) were represented by one person only, all of whom had varying exposure to MDCT measurements before the study was started. Finally, our study included Medtronic CoreValve and the Edwards SAPIEN 3™ valves which were standard devices at the time of data collection.

Conclusions

MDCT planning and accurate valve size selection remains essential for TAVR without complications. Current state-of-the-art MDCT analysis software provides excellent reproducibility for anatomical measurements irrespective of the level of pre-existing experience. However, the highest levels of confidence in terms of valve size selection are achieved by implanting physicians, which can likely be attributed to observer experience, reflecting current clinical practice.