Determining the reliability of diagnosis and treatment using artificial intelligence software with panoramic radiographs

Artificial intelligence (AI) is a scientific field that is focused on replicating human neurological processes, such as problem-solving, object recognition, and decision-making, through machines.1 Machine learning, a subset of AI, draws inferences from the information stored in a database. The more data it analyzes, the more it improves its future performance.2

Neural networks were among the earliest types of AI algorithms developed. The computational power of these networks hinges on the quality and volume of training data, which enable the networks to refine their connections. Networks with a simple structure and only a few layers are referred to as “shallow” learning neural networks, while those with numerous and larger layers are termed “deep” learning neural networks.3 Deep learning is a subset of machine learning. Deep learning constructs known as convolutional neural networks (CNNs) analyze and categorize data, thereby supporting deep learning capabilities, much like neurons in the human brain. They are primarily utilized to process large and intricate images.4, 5

One AI software application, Diagnocat (Diagnocat Inc., San Francisco, CA, USA), employs a deep CNN that incorporates U-Net-like and Mask R-CNN architectural models for segmentation and diagnosis.6 The U-Net model is known for its semantic segmentation capabilities, while the Mask R-CNN model is utilized for instance segmentation. The latter enables the measurement of the classification performance of individual objects using classification metrics. U-Net-like models are considered state-of-the-art architectures for object detection and segmentation tasks.7

Deep learning methods have found applications in a variety of medical fields. These include determining pediatric bone age from direct radiographs,8 detecting breast cancer in mammography,9 identifying skin cancer,10 and detecting brain tumors.11 CNNs can be utilized in dental radiology on images with complex layers, such as cone-beam computed tomography scans, as well as on panoramic and periapical radiographs.12, 13, 14 In the field of dentistry, AI has been employed for a range of tasks. These include tooth numbering, assessing the relationships of third molars to the mandibular canal, planning dental implants, evaluating periapical pathologies, detecting dental caries, evaluating osteoporotic changes, and examining jaw tumors.13, 15, 16, 17, 18, 19, 20, 21, 22, 23

The objective of this study was to evaluate the accuracy and efficacy of an AI program in identifying dental conditions using panoramic radiographs (PRs), as well as to assess the appropriateness of its treatment recommendations.

This study received approval from the ethics committee of the Near East University Faculty of Medicine, under protocol number YDU/2022/99-1473. The research was conducted in line with the principles outlined in the Declaration of Helsinki.

In this retrospective study, 100 PRs (representing 4497 teeth) of patients between 20 and 67 years old, taken between October 2017 and January 2020, were randomly selected from the Near East University Faculty of Dentistry’s database. The patients exhibited a variety of dental conditions, including caries, missing teeth, temporomandibular disorders, periodontal disease, and so on. Clinical examination records of the selected patients were carefully secured to verify the existence of the dental issues identified on the radiographs. To increase the complexity of the program, the study included panoramic images of patients with multiple dental conditions. Patients with fixed prosthetics, implants, caries, periapical lesions, furcation lesions, missing or restored teeth, root canal treatment, and periodontal disease were included. However, images of patients with mixed dentition, radiographs of insufficient quality for diagnostic purposes, and images including artifacts were excluded.

Imaging and evaluation

Regarding PR image generation, a Planmeca CC Proline device (Planmeca, Helsinki, Finland) was used to obtain digital radiographs. The radiographs were acquired in strict adherence to a standardized protocol, which involved stabilizing the natural head position with ear rods, as recommended by the manufacturer. Images were exposed at a peak kilovoltage of 70–84 kVp and a current of 12–14 mA, for 12–16 s each at a magnification of ×1.25.

All PR data were anonymized in Digital Imaging and Communications in Medicine format. Three oral and maxillofacial radiologists, with varying levels of clinical experience (7 years, 12 years, and 15 years), evaluated the images. Prior to the study, research calibration was conducted via videoconference. A list of dental states to be evaluated on the radiographs was also prepared (Table 1). Subsequently, the PRs were uploaded to the deep CNN software (Diagnocat Inc.). The diagnosis and treatment plans for each participant were then reassessed by comparing them with the issues identified by the observers (Fig. 1).

Fig. 1
A representative panoramic image shows a diagnosis automatically generated by Diagnocat.

• Click for larger image

Table 1
List of evaluated dental states

• Click for larger image

Model for pathology detection on panoramic images

Detecting pathologies on PRs can pose a challenge for AI, given that each image presents an entire set of teeth. At this scale, the pathologies are minute and can be difficult to detect. Furthermore, the process of annotating all pathologies is intricate and time-consuming, resulting in a training dataset with numerous missing labels.

The proposed solution was a pipeline involving 2-stage detectors. One of these was used to identify areas of interest, another to prepare pseudo-labels, and the last one to predict pathology. In addition to PRs, an annotated dataset of spot X-rays was utilized to enhance the base dataset.

The total dataset contained a substantial quantity of partially annotated data, which were not immediately usable. This is because based on these data, the model would be unable to differentiate between true negatives and false negatives and could fail to learn to identify poorly labeled pathologies. To overcome this issue, this study suggests using the model’s predictions, trained on a subset of data, as pseudo-labels. These pseudo-labels are then incorporated into the final model during training. In this study, a Mask R-CNN architectural model24 was utilized for labeling and segmentation tasks. This model was equipped with multiple heads, each trained to detect a specific group of labels. Each label group was defined in such a way that the data containing this set of labels were fully annotated. Therefore, during training, each sample was processed only through the heads that predicted the labels annotated on that particular image. The prediction of the model was presented as a set of pathologies, each accompanied by a probability, bounding box, and mask.

The model employed to identify regions of interest was separately trained for tooth detection. The training dataset was composed of 4500 images featuring annotated teeth, inclusive of missing teeth. The objective of these models was to identify teeth by segmenting their masks and defining their numbering. A 2-stage detector, Mask R-CNN, was utilized for this task, with a pre-trained ResNet-10125 serving as the backbone. The predictions generated by the trained model were used to designate the mouth area as the region of interest. The model’s output consisted of bounding boxes and segmentation masks with predicted numbers for each tooth. The coordinates of the mouth area were determined by the minimum and maximum values of the x and y coordinates for all detected teeth, expanded by a selected number of pixels (Fig. 2).

Fig. 2
Image illustrating the predictions made by the trained model, used to define the mouth region as the area of interest within the panoramic radiograph.

• Click for larger image

This model was trained using approximately 5430 partially annotated panoramic images from various manufacturers, featuring diverse pathologies. The dataset was further expanded by an additional 18000 intraoral X-rays. The model was designed to detect 21 different states. Initially, all panoramic images were supplemented with pseudo-labels from previously trained models. Each image was then cropped based on predictions from the ROI detector, and these cropped images were subsequently fed into the model. The chosen architecture for the model was the Cascade R-CNN,26 which differs from the Mask R-CNN in that it iteratively refines box prediction and takes class predictions as the average outcome for each cascade layer. This approach improves the prediction quality and reduces overfitting. To enhance the generalization of the model and improve performance, various augmentations were applied to the input data. These included randomized crop, rotation, brightness, contrast, image downscaling, blur, and noise; optical distortion; grid distortion; and contrast-limited adaptive histogram equalization.27 For X-ray images, this study also utilized mosaic augmentation, which produced images as depicted in Figure 3. Mosaic augmentation is a technique employed in dental radiography to enhance object detection. It involves extracting foreground ROIs from sparse samples and combining them to form a new image. To address scale variation, chips are magnified and suitable regions are chosen using sliding windows, with the zoom factor serving as a guide. This augmentation is applied to all training samples, while general samples do not require rescaling. Mosaic augmentation introduces complex backgrounds, which assists in object detection in a variety of contexts. It significantly enhances accuracy and robustness in the analysis of dental radiographs.28

Fig. 3
Mosaic data augmentation teaches the model to recognize objects in different locations, eliminating the need for a single specific context. This method of data augmentation combines numerous training images into a single image, using random proportions.

• Click for larger image

In the inference process, a panoramic image must initially undergo ROI detection, similar to the data preprocessing stage. Subsequently, it is processed through the cascade model. The predictions of the model are then calibrated and presented as the final output.

A total of 4497 teeth with panoramic radiographs from 100 patients were analyzed by all 3 researchers. As depicted in Table 2, instances where both the software and the researchers reached the same diagnosis are labeled as “correctly diagnosed (true positive).” Situations where the researchers made a correct diagnosis but the AI misdiagnosed the condition are referred to as “misdiagnosed (false negative).” Conversely, cases where the researchers incorrectly diagnosed the condition but the AI correctly identified the issue are termed “over-diagnosed (false positive).” Table 3 presents a compatibility chart between the researchers and the software, detailing the detection of all dental conditions for each dental state.

Table 2
Sensitivity and specificity of assessment using Diagnocat software

• Click for larger image

Table 3
Kappa values for inter-observer assessments and comparisons to the ground truth for each condition evaluated

• Click for larger image

The AI demonstrated almost perfect agreement (above 0.81) with the ground truth in most assessments (Table 2). The sensitivity was very high (exceeding 0.8) for the assessment of healthy teeth, artificial crowns, dental calculus, missing teeth, fillings, lack of interproximal contact, missing teeth, periodontal bone loss, and implants. Substantial sensitivity (above 0.6) was observed for cast posts and posts, short fillings, open margins, furcation lesions, and impacted teeth (Table 2).

The sensitivity was found to be low for the assessment of caries, periapical lesions, pontics, voids in the root canal filling, and overhangs (Table 2).

Statistical analysis revealed high kappa values for all inter-examiner assessments, demonstrating the strong reliability of the ground truth and the reproducibility of the reports (Table 3). The statistical evaluation indicated good reliability between the ground truth and AI software results, with the exception of assessments for caries (kappa: 0.365), periapical lesions, (kappa: 0.540), pontics (kappa: 0.517), overhangs (kappa: 0.604), and furcation lesions (kappa: 0.361) (Table 3).

The rise in AI software usage in medical fields has extended to dentistry, with new programs being developed specifically for this sector. This study examined one such software, Diagnocat, and its capacity to detect various dental conditions in PR. The software demonstrated the highest accuracy in identifying dental states associated with missing teeth, while its sensitivity was lowest for missed canals. The Diagnocat program also correctly identified primary teeth in 2 patients, although these findings were not included in the statistical analysis. Previous research has indicated that deep learning methods supported by CNNs are highly acceptable in areas such as classification, scanning, and segmentation in medical image analysis.29 Various CNN architectures have been employed across different imaging modalities in the literature. Those with a U-Net-like architecture are typically favored for segmentation and volumetric data.30 The literature also suggests that software utilizing a U-Net-like architecture yields impressive results in image segmentation. However, training the U-Net network requires an excessive number of parameters.31 Diagnocat AI software employs a deep CNN with U-Net-like and Mask R-CNN architectural models for diagnosis.6

Various published studies have utilized Diagnocat software to explore different aspects of dental practice.15, 16, 17 One such study investigated the ability of deep CNNs to detect periapical lesions, and the results demonstrated success.15 In another study, Orhan et al.16 examined the effectiveness of AI in diagnosing impacted third molars, assessing their relationship with neighboring anatomical structures, and determining the number of root canals. Using cone-beam computed tomography images as the gold standard, the study indicated that the deep CNN method was successful in detecting impacted third molars and evaluating their relationship with anatomical structures. Kurt Bayrakdar et al.17 evaluated the use of AI in implant planning. In bone height measurements, they found no statistical difference between manual measurements and those made by AI. Consequently, they suggested that AI applications should be developed for implant planning. Ezhov et al.32 conducted a study to test the clinical performance, accuracy, and time efficiency of Diagnocat in diagnosing anatomical landmarks and pathologies. They also evaluated the clinical effectiveness and reliability of these diagnoses. The study concluded that Diagnocat software could potentially enhance the decision-making processes of clinicians.

In their study, Lee et al.18 utilized the GoogLeNet Inception v3 CNN network for preprocessing and transfer learning. The research yielded caries diagnostic accuracy levels of 89.0% for the premolar model, 88.0% for the molar model, and 82.0% for a model incorporating both premolars and molars. In a separate study, Kositbowornchai et al.33 employed a probabilistic neural network design to evaluate vertical root fracture. They reported that the highest sensitivity, specificity, and accuracy rates for detecting vertical root fractures were 98%, 90.5%, and 95.7%, respectively.

The efficacy of AI has been explored in distinct areas, such as the identification of dental caries,18 jaw tumors,20 and root fractures,33 as well as the examination of root canal morphology,34 periodontal diseases,35 and periapical lesions.15 However, no study has yet consolidated these topics into a single comprehensive investigation. Another strength of the present research is the preference for patient radiographs with established clinical examination results. This approach allowed us to compare the accuracy of AI diagnoses with actual clinical findings.

In the literature, various studies have explored the use of CNN architecture in different dental imaging modalities. Lin et al.36 examined the capacity of AI to identify and categorize teeth on dental bitewing radiographs, while Hosntalab et al.37 applied the technology to multi-slice computed tomography images. Chen et al.14 investigated the use of AI in detecting and numbering teeth on periapical radiographs. Arik et al.38 studied the role of AI in defining anatomical landmarks on cephalometric radiographs, reporting a high success rate in landmark detection. The present study found the lowest reliability in the assessment of caries, periapical lesions, pontic voids in the root canal, and overhangs. The diagnosis of caries is a clinical judgment concerning the presence of the disease. The detection of caries lesions involves identifying signs of caries, either clinically or radiographically. The use of a dental probe can enhance visual evaluation by providing tactile feedback. However, no consensus exists on the benefits of probing, as this action can cause the weakened enamel to collapse and the lesion to progress.39, 40 Furthermore, a dental probe can potentially transport bacteria from one surface to another, serving as a source of contamination. Radiographic detection involves identifying a radiolucency, interpreted as a caries lesion, on a dental radiograph.40 Visual evaluation of the approximal surfaces of the posterior teeth can be challenging. However, bitewing radiography can offer valuable information about the caries status of the approximal surfaces of the tooth crowns.39, 40, 41 Nonetheless, in bitewing radiographs, the proximal surfaces can overlap due to angulation errors, and the status of the occlusal surfaces remains unclear.42

AI techniques, such as Bayesian networks, artificial neural networks, or deep learning, can be utilized to develop computer-based systems. These systems can assist physicians in making decisions about their patients’ care. The increasing volume of data resulting from digitization in the radiology workflow provides an opportunity to maximize the use of machine learning techniques. This study evaluated the effectiveness of AI in detecting various dental conditions, including caries.

Lee et al.18 evaluated the efficacy of the CNN algorithm in detecting dental caries in periapical radiographs, finding it to be highly accurate. The study revealed that the AI system was significantly more sensitive than dentist evaluation, while its specificity was not significantly lower.43 A systematic review also evaluated the neural networks used in detecting and diagnosing dental caries. The review examined the conditions of these studies and their variable parameters, but the authors were unable to draw definitive conclusions. The definition of detected caries was not detailed in all studies, and not all specified the type of caries. Each study included in the review utilized a different neural network. These variations complicated the conclusions about the subject and the reliability, or lack thereof, of a neural network in the detection and diagnosis of caries.43, 44

The inability to conduct a histopathological examination to verify caries diagnosis was a notable limitation of the present study. This limitation applies to the assessments made by both observers and AI. In the present study, PR was the preferred imaging method due to its common use in dental radiographic examinations. PR allows for the scanning of a large anatomical region while requiring a relatively low dose of radiation. However, the quality and magnification of dental PR images can vary based on the patient's positioning.45 Therefore, the use of trans-hospital or hybrid datasets from multiple machines and conditions is important for achieving meaningfully high accuracy in the clinical application of deep learning.46 Notably, in this study, PRs were obtained from a single device in a single center, which constitutes a second limitation of this research.

In conclusion, despite the limitations of the study, the synthesized data suggest that AI-based decision support systems can serve as a valuable tool in detecting dental conditions, using PR for clinical dental applications.