Clinical Validation of a Deep Learning-Based Hybrid (Greulich-Pyle and Modified Tanner-Whitehouse) Method for Bone Age Assessment

Bone age assessment is crucial for evaluating pediatric growth and maturity [1]. Bone age is an important parameter for the assessment of the progress and treatment of various pediatric endocrine diseases. Furthermore, it can be used to predict adult height [2, 3]. Recently, the high prevalence of precocious puberty [4], increased interest in the height of children, and increased usage of growth hormone therapy emphasize the need for assessing bone age [5].

In clinical practice, left hand and wrist radiography-based bone age assessments, such as the Greulich-Pyle (GP) [6] and the Tanner-Whitehouse 3 (TW3) methods [7], are widely used. The GP method is an atlas-based method, which determines bone age by comparing the radiographs of the hand and wrist with the most similar standard radiographs in the GP atlas. It is a simple method that is readily available in clinical practice. However, several studies have suggested possible issues, including inter- and intra-observer variability and the dependency of the accuracy on the experience of the clinician [8, 9, 10]. Furthermore, it is a semi-quantitative method because the GP atlas mainly covers one year. Recently, deep learning-based automatic bone age assessment models have been developed to overcome these issues [11, 12, 13]. According to Stanford University researchers, the mean absolute difference (MAD) between the automatic bone age assessment tools based on the GP atlas and standard bone age, which is determined by three radiologists, is 0.5 years [14].

The TW method is a scoring system that measures the individual bone maturity score and evaluates bone age by summing the scores. After two revisions, the current version (TW3) was proposed, which used the maturity of the radius, ulna, and short bones [7]. It is a quantified method and, therefore, is more accurate and has a higher reproducibility than the previous GP method [8, 9, 10]. However, the TW method also has certain limitations. First, it is a more complex method that takes a longer time than the GP method [8, 15]. Second, it classifies individual bones based on nine maturity grades (A to I); thus, the classification is sometimes ambiguous and has inherent deviation because one particular bone shape can have two different pre-defined labels of the same feature [16, 17].

To overcome the limitations of both methods, we developed a hybrid GP and TW artificial intelligence (AI) bone age assessment software. This software was trained using hand and wrist radiography based on both the GP and modified TW methods, which use seven regions (radius, ulna, distal phalange, middle phalange, and proximal phalange, metacarpal of the third digit, and metacarpal of the first digit) instead of 13 regions for the TW3 method. The holistic hand image analysis based on the GP method in our software can cover all the regions that are not included in the TW3 method, and the holistic hand image analysis is further reinforced by minutely assessing the individual region of interest (ROI) based on the modified TW method, which improves the classification performance by zooming in more relevant regions, as discussed in a previous study[13]. Eventually, the final bone age in this software was obtained from the integrated analysis of both holistic image analysis and ROI analysis in a fully automatic manner. The purpose of this study was to evaluate the accuracy and clinical efficacy of a deep learning-based hybrid GP and modified TW AI model for assessing bone age.

This study was approved by the Institutional Review Board and Ethics Committee of the Korea University Anam Hospital (IRB No. 2019AN0010). Informed consent was waived because the data were collected retrospectively and analyzed anonymously. The study complied with the ethical principles of the Helsinki Declaration of 1964, which was revised by the World Medical Organization in Edinburgh in 2000. This study involved three major steps: model development, external validation, and statistical analysis. Details about the steps and the study population are provided in Figure 1.

Fig. 1
Overview of entire steps of the study including participant selection.
The model was developed using two public datasets. A total of 15611 hand radiographs were used as training, validation, and test sets. A total of 102 hand radiographs were used for external validation. Finally, statistical analysis was performed. GP = Greulich-Pyle, RSNA = Radiological Society of North America

• Click for larger image
• Download as PowerPoint slide

Model Development

The model consists of three steps: ROI detection, region maturity classification, and integrated bone age. The steps are summarized in Figure 2. First, seven regions based on TW3 were automatically detected using the convolutional neural network (CNN) algorithm. While TW3 used 13 ROIs for bone age assessment, the proposed modified method used seven regions (radius, ulna, distal phalange, middle phalange, proximal phalange, metacarpal of the third digit, and metacarpal of the first digit) to improve the labelling efficiency during the training steps. We postulated that the short bones of the first and fifth fingers, except the metacarpal of the first finger, correlate highly with those of the third finger, as reported in a previous study [18]. Second, each ROI and the holistic hand image were automatically classified for maturity using the CNN algorithm. While TW3 uses nine stages, from A to I, to assess the maturity of each ROI, our method used 34 stages with a 6-month gap from 1.5 to 18 years for maturity, which could be more accurate and intuitive. In addition, the holistic hand image is automatically classified and applied to our AI model because the holistic image can provide the maturity features of regions that are not included in the ROIs of TW3. In this ROI and the holistic hand image maturity classification procedure, we used the doctors' ratings as the reference standard for each CNN model training. Finally, the features from each ROI and the entire hand were integrated and classified to provide the final bone age estimate of the input image. Basically, the prediction probability distributions of the maturity stages of the regions were concatenated and inputted to the final integration step, which used a fully connected neural network model. This entire procedure for bone age assessment, composed of three steps, is fully automatic. The model was implemented using an open-source machine learning library (TensorFlow version 0.9.0; Google). Figure 3 shows a sample image of the mediAI-BA, the automatic solution interface of this software.

Fig. 2
Overview of Greulich-Pyle and modified Tanner-Whitehouse hybrid bone age assessment models.
ROI = region of interest

• Click for larger image
• Download as PowerPoint slide

Fig. 3
Result of automatic bone age assessment program (mediAI-BA) including analysis of detailed area of interest.
Automatic hybrid method-derived bone age is observed in the left upper corner ①. The user can choose from ② the seven regions of interest (DP, MP, PP, and MC of the third finger; radius, ulna, and MC of the first finger), and third digit middle phalanx image with its respective maturity degree is shown ③. Heatmap overlay is selected ④ and is shown. DP = distal phalange, MC = metacarpal, MP = middle phalange, PP = proximal phalange

• Click for larger image
• Download as PowerPoint slide

We used two public datasets. The first is from the Radiological Society of North America (RSNA) 2017 challenge, which includes 14236 hand radiographs from Stanford University and Colorado University [19], and the other is from the Digital Hand Atlas [20], which includes 1375 hand radiographs from the University of Southern California. We used 10% of the dataset as the validation set and 200 images as the test set.

Our study verified the accuracy and clinical efficacy of the newly developed GP and modified TW hybrid AI bone age assessment model. Our model had an accuracy similar to that of human experts, with the upper limit of the 95% CI of the MAD between the AI bone age assessment measurement and the reference standard being less than 0.5 years. Our model shortened the reading time by approximately 35% for two additional radiologists. Kim et al. [21] showed that reading times were reduced by 18.0% and 40.0% for each of the two reviewers. Other studies have also reported that AI-assisted bone age assessment can reduce the interpretation time [22]. Furthermore, the accuracy of bone age assessment was significantly improved in the case of one additional radiologist (reviewer 5). Additionally, the ICC of two additional radiologists was somewhat improved during model assistance. This means that this model could help improve the inter-observer reliability, as in previous studies [12, 14, 23].

Recently, AI with deep learning has been applied to musculoskeletal radiology, including image interpretation, such as fracture detection and bone age assessment [24]. Several studies have suggested that CNN bone age assessment is as accurate as that of experts and has clinical efficacy [11, 12, 14, 23]. Tajmir et al. [23] showed that AI assistance improves the performance of radiologists. Our study also showed that the AI tool improved the performance of the radiologist in terms of reducing the interpretation time and improving the inter-observer reliability. Furthermore, our model improved the diagnostic accuracy of bone age assessment for less experienced radiologists.

Contrary to previous studies [22], our model has the following advantages. First, our hybrid model complemented the limitations of GP and TW by focusing on the regions that are highly related to bone maturity changes and by applying finer-grained maturity stages than TW3, which resulted in a reliable and accurate bone age estimate. Second, our model reflects the comprehensive human decision-making process in a clinical setting whereby experts exploit the ROIs of bone rather than strictly use the GP atlas. Finally, the black box nature of CNNs [25, 26], we thought, could be partially resolved by integrating two different methods (GP and TW) within the medical domain during the model development process.

Our model showed the integrated bone age, as well as two different results, based on the detailed ROIs and a holistic image. The current commercial automated bone age assessment system, BoneXpert (Visiana Aps, Holte, Denmark, http://www.boneexpert.com) with its recently launched version 3.0 (September 2019), is accurate [27], and it evaluates the bone age according to both the GP and alternative TW2 methods [28], which is similar to our model. BoneXpert is based on a feature extraction technique that reconstructs the border of 15 bones (including metacarpal, phalangeal bones, distal radius, and ulna) [29], which is different from our model. Several AI bone age assessment models have been developed using all the bones included in the radiograph and sometimes display the sensitive region of the image like a heat map [14, 21]. The mean sensitive regions of the image were determined by the model, not by clinicians, and were different during the serial follow-up.

The MAD and RMSE of our model were 0.39 and 0.50, respectively. This value shows that the accuracy of our hybrid model is similar to that of previous studies [14, 15, 18, 29]. The MAD and RMSE of the GP method-based AI model MAD and RMSE were 0.50 and 0.63 [14]. Another GP method-based model called “HH-boneage” has a MAD of 0.46 and an RMSE of 0.62 [15]. The 9-stage TW method showed the MAD and RMSE for 7 ROIs of 0.59 and 0.76 [18]. BoneXpert version 3.0 showed an RMSE of 0.63 [29]. It was observed that the test data sets, including the number of images, differ among these studies, and the values for comparing the automated assessed bone age with the standard reference were obtained by the different expert groups. Therefore, the results cannot be directly compared with the RMSE and MAD values. However, these values corroborate the accuracy of the proposed model.

However, our study had some limitations. First, this was a single-center retrospective study with only a few participants; in particular, only six participants aged < 5 years and > 15 years were enrolled. In the future, prospective multicenter large-sample studies are needed. Second, our study included a single-ethnicity external validation. Previous studies have shown racial differences in certain age bone growth patterns, which can affect bone age assessment [30, 31]. However, our model was trained on open data, including those on ethnicity; therefore, we believe that it could be used globally. Third, we only compared the accuracy of our model and the GP method. A recent article suggested that the question of whether AI bone age assessment should be compared with other bone age assessment methods, including the TW method or using other imaging modalities such as MRI or ultrasonography instead of the left hand and wrist radiography persists [15]. Therefore, further comparative studies are required to confirm this. Finally, our model could not detect disorders such as congenital syndrome or rickets, a similar limitation to that of other recent AI models. However, the purpose of AI bone age assessment models is to assist the radiologist and not to use it independently in a clinical setting.

In conclusion, this new hybrid GP and modified TW AI bone age assessment model was accurate for bone age assessment. Furthermore, it appeared to improve the clinical efficacy by reducing the interpretation time and improving the inter-observer reliability.