A hybrid mask RCNN-based tool to localize dental cavities from real-time mixed photographic images

Literature review

The recent advancements in artificial intelligence and deep learning techniques reduced human efforts and error rates in dental caries localization (Volinski, 2016). The clinical practices employed various machine learning approaches in dental caries diagnosis (Prados-Privado et al., 2020). The dental carious regions’ localization approaches use image processing techniques, classifiers, neural networks, etc., to detect faulty dental regions (Jader et al., 2018; Ezhov et al., 2021). In addition to this, researchers have proposed dental caries localization tools. We will discuss dental caries localization approaches and the tools in the following. We also discuss state-of-the-art dental caries detection and treatment techniques, particularly in a clinical context.

Dental caries localization approaches

The dental caries localization approaches use a variety of image types to diagnose carious regions in clinical settings. The image type includes but is not limited to panoramic images, ultrasound images, bitewing radiographs, colored photographs, X-ray images, etc. (Kumar, Bhadauria & Singh, 2021). The employment of panoramic X-ray images in the localization of dental images is imprudent of imaginary surrounding areas (chin, spine bones, jaws, etc.) (Lee et al., 2018). The localization requires multiple images in a single dental cavity localization (Liang et al., 2020). The existing research also shows less satisfactory results in dental cavities detection via X-ray panoramic dental images due to their complexity (Dibeh, Hilal & Charara, 2018). The panoramic radiographs may cause mistreatment due to misdiagnosis in some instances (Endres et al., 2020). Utilizing ultrasound images to local dental cavities via deep learning approaches requires vast expertise in scanning and ultra-sound operations (Nguyen et al., 2021). Bitewing radiographs show less sensitivity for a variety of services, including proximal and occlusal. Their performance may not be satisfactory in detecting non-cavitated lesions (Chen, Stanley & Att, 2020). Exploiting machine-readable intra-oral dental photographs (colored) taken via high-resolution cameras gives accurate results (Kühnisch et al., 2021). Recent literature is evidence of the employment of colored photographs taken via smartphones in dental cavities detection. The localization sensitivity is between 75% and 100%, particularly in sound teeth and extensive lesions (Bayraktar & Ayan, 2021). Recent research also shows that the X-ray radiographs provide dental carious region detection accuracy even in the case of small datasets (Rad, Rahim & Kolivand, 2018).

The employment of colored and X-ray images is less challenging than other image formats since the mentioned image types are less complex and widely available in clinical settings. The following sub-sections discuss the existing schemes and underlying machine learning models, which are mainly employed to detect carious regions from colored photographs and X-ray radiographs.

Localization via colored photographs

Koutsouri et al. (2013) proposed a non-supervised method to detect carious regions. The authors applied a k-means clustering algorithm to segment caries and extracted features from RGB channels. Human experts annotated the images. Vashishth, Kaushal & Srivastava (2014) proposed an approach to detect dental caries in radiographic and intra-oral images. The authors converted the colored images into black and white and used a Gaussian filter to enhance the image quality. The authors segmented each image into various segments and further classified carious images by exploiting threshold computed via black pixels. The mentioned approach mainly spotted the black regions as dental cavities. Berdouses et al. (2015) developed a model to detect the pit and fissure cavity from colored images. The authors worked on premolars and molars and eventually converted the 100 colored images to grey-scale images. The authors used the k-means algorithm to eliminate the uninterested areas. Moreover, the authors further employed j48, Random Tree, Random Forests, SVM, and Naive Bayes classifiers to detect carious images. Ghaedi et al. (2014) proposed an automated cavity detection and scoring system. The authors worked on optical colored images and used histogram-based contrast enhancement. The entropy, gradient, and matrix elements were extracted and used in random forest to classify the images. Navarro et al. (2019) proposed a model to localize the smooth surface cavity by considering only colored dental images. The histogram equalization was done pre-processing, and the image was segmented further into 10 × 10 blocks. The intensity, gradient, hue, saturation, and entropy features were extracted from each block. The authors used SVM and decision trees as classifiers.

Localization via X-ray images

Oprea et al. (2008) classified only enamel, dentin, and pulp caries using rule-based classification to detect the tooth edge. The carious region was identified by calculating the width of the cavity and adjacent black-white pixels. The authors categorized regions of size more than 2 mm as dentin caries. Datta & Chaki (2015) proposed a model to detect the cavities using a Wiener filter. The authors segmented the images using a cluster-based segmentation technique. The authors extracted the regions as caries containing many black pixels lacking the detection of carious lesion shape and depth. ALbahbah, El-Bakry & Abd-Elgahany (2016) proposed an approach to detect dental caries using a dataset containing 100 panoramic images. The carious regions were segmented based on a threshold, and a gradient histogram was extracted from the images. Finally, an SVM classifier was used to classify the carious and non-carious teeth. Prajapati, Nagaraj & Mitra (2017) proposed a model to classify dental diseases using Convolutional Neural Network (CNN) and transfer learning. The authors used the VGG16 pre-trained model to classify dental diseases, such as dental caries, periapical infection, and periodontitis. Naam et al. (2017) proposed a methodology to identify dental caries using 27 X-ray panoramic images covering the whole mouth area. The tooth regions were manually extracted by cropping the images. They extracted gradient and intensity features from the images and performed SVM classification to classify the carious and non-carious dental images. Jader et al. (2018) proposed a technique to segment 1,224 X-ray images. They used R-CNN to segment the tooth regions, used the classifier named ResNet101.

Most of the existing X-ray based approaches exploited the low-level image features in dental caries localization. However, a research gap exists in deep learning approaches to localize the dental cavities in bounding boxes and solely classify the carious regions. The mentioned approaches usually employ colored photographic or X-ray radiographic images to local dental carious regions. We discuss dental caries localization via state-of-the-art deep learning approaches in the following.

Dental caries localization: deep learning approaches

Lee et al. (2018) investigated the Deep CNN models to detect and diagnose caries via periapical radiographs of molars, premolars, and molars. Cantu et al. (2020) detected caries lesions via a convolutional neural network (U-Net) by employing bitewing dental radiographs. They found via experimentation that deep learning may facilitate the dentist to detect and locate caries more efficiently (Cantu et al., 2020). Zhang et al. (2020) employed a deep learning-based system containing a CNN to detect dental caries from oral photographs. They found convolutional neural network promising over dental photographs captured via ordinary consumer cameras (Zhang et al., 2020). Casalegno et al. (2019) trained the CNN model to automate the detection and localization of early-stage dental lesions/caries from radiographic transillumination (TI) images. Karimian et al. (2018) presented the CNN model trained over the optical coherence tomography (OCT) imaging modality to classify the oral tissues of humans and detect the early dental caries. They also validated the approach over OCT images of enamels, cortical bones, trabecular bones, muscular tissues, and fatty tissues (Karimian et al., 2018).

Dental caries localization: tools

A few software exists for caries detection and localization. For instance, Logicon caries detection software is used to detect proximal caries (Bentsen, Svensson & Wenzel, 2001). Proximal caries exists in the distal and mesial surface of the tooth. The software enhances the dental X-ray radiographs and highlights the carious region. However, the tool is less efficient than human observers (Bentsen, Svensson & Wenzel, 2001; Wenzel, 2002). Curve Dental (https://www.curvedental.com/) is an automated mid-level tool preferable for dental practitioners. The tool is comprehensive and considers everything from imaging to patient education to scheduling and billing. The SOTA (https://www.curvedental.com/). Image Software is specially designed to handle dental imaging. Practitioners use high-quality colored photographic images of all sizes for training purposes. It mainly adjusts image filters to localize carious regions. The software reduces the amount of time needed to assist the dental education training. Tab32 (https://www.tab32.com/) is cloud-based dental software consisting of a modern interface in growing dental offices. Many practitioners can analyze dental imaging, maintain electronic health records and patient communication. Generally, the localization tools enable the manual labeling of dental carious regions. DIAGNOcam (https://www.kavo.com/dental-xray-machines-and-diagnostics/diagnocam-diagnostic-devices) is an integrated camera-oriented tool that detects the early enamel and dentin caries directly from the captured dental photographs (Casalegno et al., 2019).

Motivation

The existing approaches, traditional or advanced, handle a single dental image type, i.e., colored photographs, or X-ray radiographs. To the best of our knowledge, the detection and localization of dental carious regions in rectangle boxes to enable further exploration by dentists via multiple image types is not yet common. Mainly the techniques provide the binary classification of dental images (Sornam & Prabhakaran, 2017). It becomes difficult for dentists to find carious regions in the dental images due to the lack of usable carious region detection and localization tools. The deep learning approaches that extract high-level image features and localize the dental image carious regions are discussed in the literature, primarily focusing X-ray based radiographic images (Milošević et al., 2022; Morid, Borjali & Fiol, 2021). Moreover, a functional tool to detect and localize dental carious regions is unavailable for mixed images, i.e., colored photographs and X-ray radiographs. This research proposed a hybrid approach that works with both colored photographic and X-ray radiographic dental images using the M-RCNN model.

Dataset collection and annotation

To the best of our knowledge, there is no publicly available mixed annotated dataset containing colored photographs and X-ray radiographs of dental caries. Similarly, the raw datasets containing dental carious regions are also inadequately available and fewer in quantity. The existing datasets limit the research in dental cavity localization approaches. To facilitate future cavity localization approaches, we tried to provide a publicly available mixed dataset containing annotated colored photographic and X-ray radiographic images. We formed an archive of 936 manually annotated X-ray radiographs’ datasets collected in a significant proportion from a hospital. A written consent was obtained from all the participants. In addition, a collection of 90 manually annotated colored photographic dataset is created. The dataset is available at Zenodo repository (https://doi.org/10.5281/zenodo.5680365). The dataset can be used publically since permission is granted from the CMH Rawalakot (included in the Zenodo repository). The dataset details are given in Table 1.

The dataset $D{S}_1$ contains annotated colored photographic dental images, accessible from the Digital Imaging Library of the Optics Department Granada University, Spain. This dataset consists of color dental photographs taken with a Canon EOS 7D digital single-lens reflex color camera with a 135 mm standard lens. Moreover, the dataset constitutes different resolution and file format photographs. The aforementioned database is publicly available under a standard license and used by many researchers (Datta & Chaki, 2015; ALbahbah, El-Bakry & Abd-Elgahany, 2016). We employed jpg file format because the tiff file format is larger and increases processing time (Rosado et al., 2020). Moreover, we used Set-7 and Set-8, containing photographs of dried oral caries (Eckhard et al., 2012). Collectively, this dataset consisted of 90 colored photographic images. The dataset $D{S}_2$ contains annotated dental X-ray radiographs collected from various sources. We used publicly available Vahab Archive (https://mynotebook.labarchives.com/share/Vahab/) containing 120 X-ray images (Koutsouri et al., 2013). We also locally collected 816 X-ray images from CMH, Rawalakot, Pakistan. The locally managed X-ray images dataset was merged with the Vahab archive dataset. It collectively increased our X-ray dataset to a total of 936 dental X-ray images.

The dataset $D{S}_3$ also contains mixed annotated dental images. It consists of publicly available colored photographic and X-ray radiographic images, including 90 photographic colored images collected from Digital Imaging Library Granada Spain and 120 dental X-ray images from Vahab Archive. We did not include the locally collected X-ray radiographs to balance the ratio of colored photographs and X-ray radiographs for accurate training and testing of the proposed M-RCNN model. It is noteworthy that a significantly unbalanced ratio of training images can cause the model overfitting issues and may lead to biased results (Ren et al., 2012). The mixed dataset collectively consists of 210 images belonging to different types.

Localization approach

Overview

We proposed a transfer learning approach consisting of six major phases, as shown in Fig. 1.

Firstly, we gathered and annotated the dental colored photographs, X-ray radiographs, and mixed (i.e., colored photographs and X-ray radiographs) dataset (Fig. 1A). Secondly, we performed image enhancement to improve the image quality for improved carious region detection (Fig. 1B). Thirdly, we annotated the carious regions via human dental experts to train our model (Fig. 1C). Fourthly, we passed the annotated images to M-RCNN model (Fig. 1D). The M-RCNN learns the carious tooth regions from the annotated images for autonomous detection. The M-RCNN neural network enables proposal generation, localization, and classification. Fifthly, the trained M-RCNN automatically detects the carious regions and encloses them in rectangles (Fig. 1E). Finally, the accuracy of cavity detection is analyzed (Fig. 1F). We also developed a graphical user interface (GUI) based dental carious region detection tool by employing the M-RCNN. The tool source, configuration, and installation information are publicly accessible from Zenodo Repository (https://doi.org/10.5281/zenodo.5680365). The researchers/practitioners can download the tool from the Zenodo repository and install it along with GUI on their personal computers by following the tool’s installation instructions.

System architecture

A software architecture creates, interprets, analyzes, and designs solutions within a particular knowledge domain (May, 2011). The architecture decomposes domain problems into subsystems, layers, and integrated components. The architectural design envisions the systems in broader perspectives and elaborates the communication among the components distributed in non-overlapping layers, which may pertain to the functionalities associated with them (Richards, 2015; Perry & Wolf, 1992). We also suggested a layered-based architecture to enable the instantiation of the proposed approach. Fig. 2 depicts the architectural design of the proposed approach containing data, intelligence, and application layers.

Data layer

The data layer pertains to the details related to collection, normalization, and annotation of dental image datasets for further training and localization, as shown in Fig. 2A. The data layer annotates the mixed dental images collected from various sources. It performs necessary histogram equalization via sharpening filters to uniformly distribute the color and grey-scale intensities. The data layer further employed the lblimg tool to manually annotate the dental images with the support of human dental experts. The data layer finally stores the annotated images in three separate datasets containing the colored photographs, X-rays radiographs, and mixed dental images.

Intelligence layer

The intelligence layer provides training and prediction of dental carious regions as shown in Fig. 2. The intelligence layer takes annotated dental images from the three distinct datasets, i.e., colored, X-ray, and mixed. It further employs the ResNet feature extractor module to extract the high-level features from the dental images. The M-RCNN training module generates the RPN by employing the high-level features and annotations in the classification, localization, and annotation of dental carious regions. Ultimately, the intelligence layer enables the input of a dental image (colored photograph or X-ray radiograph) and localizes the carious regions.

Presentation layer

The application layer envisions the instantiation of the proposed approach over the real generic applicative scenarios, as shown in Fig. 2C, the presentation layer employs the Model View Controller (MVC) architecture to instantiate dental carious regions localization tool. The view is the front-end interface; the model is the trained M-RCNN over the dataset of mixed dental images (colored and X-ray images); the controller contains the logic to activate the M-RCNN model. The presentation layer provides a search engine that accepts an image and localizes the carious regions. The search engines enable the processing via the Django web framework. The application interface directly links the controller layer to trigger the M-RCNN model, which processes the dental image and localizes the carious region. The application interface displays the localized version of the dental image.

Instantiation

The deep learning models are widely exploited in the detection of dental cavities. The basic CNN architecture considers VGG-16, Faster RCNN, YOLO, Mask RCNN, etc., as pre-trained deep learning models. The employment of the CNN models in dental healthcare and clinical setting is subject to their suitability and specifications (Ezhov et al., 2021; Koutsouri et al., 2013). The researchers commonly employed a convolutional neural network to diagnose diseases in dental images. However, they used VGG-16 as a pre-trained model in oral disease classifications (Prajapati, Nagaraj & Mitra, 2017). The YOLO framework employs a single CNN model and entire image to detect possible objects in images and regions of interest. The YOLO gives satisfactory results in bitewing dental images (Bayraktar & Ayan, 2021). The M-RCNN is an extension of Faster RCNN and is used to predict multiple regions of interest (Prados-Privado et al., 2020). Researchers, such as Zhu, Piao & Kim (2020) used M-RCNN to identify tooth regions in images instead of YOLO and Faster RCNN. Our investigation reveals the distinct implication of M-RCNN model in the detection of dental cavities via multiple image types.

This research proposed a generic approach that can localize carious regions in mixed (colored dental photographic and X-ray radiographic) images using the transfer learning approach with minimum supervision requirements from an expert (Jader et al., 2018). We employed a pre-trained M-RCNN model with the conjunction of preprocessing techniques (ResNet50 for feature extraction and classification purpose along with gradient descent). We exploited the laplacian sharpen filter to boost the carious regions’ pixels in dental images and renowned Scikit-image version 0.15.0 (python package built upon NumPy, SciPy, and matplotlib) for image processing since it contains several algorithms and works with several types of heterogeneous data structures. The preprocessing highlighted the dark regions; this enhanced the lighter pixels of dental images. We firstly considered transfer learning via feature extraction scheme since our dataset is heterogeneous, containing colored dental photographic and X-ray radiographic dental images. The M-RCNN is a pre-trained model for end-to-end learning on our custom dental dataset with ResNet50 for feature extraction. It is essential to note that the approach can also work on heterogeneous localization architectures. Afterward, region-based annotated images are passed to a pre-trained M-RCNN model to locate the carious region.

The M-CRNN model was instantiated using the python programming language on an ordinary Intel Core (TM) i3- 4010U CPU @ 1.70 GHz, having 4.00 GB of RAM and Django web framework (https://www.djangoproject.com/). The Django web framework is considered a high-level python web development framework. We designed and developed a complete toolkit that facilitates a dentist in diagnosing carious teeth regions. Figure 3 depicts the interface details of the dental cavity detection and localization tool. The dental cavity detection tool helps the end-user to self-diagnose the carious tooth regions. Along with that, the tool facilitates in reducing the excessive workload from the dentists. The tool detects carious regions from colored photographic or X-ray radiographic images in three simple steps. In the first step, the user uploads a tooth image (Fig. 3A). Afterward, the user clicks the submit button (Fig. 3B). The uploaded picture is passed to the proposed model, and the detected localized carious region output image is displayed on the user interface for the input X-ray radiograph (Fig. 3D) or colored photographs (Fig. 3E).

Dataset validation

We hired two qualified dentists ( $D_1$ and $D_2$) with more than 3 years of work experience to manually annotate carious regions using the lblimg tool. The dentists used bounding boxes to enclose each carious region. A perfect bounding box is the one that covers each carious region of a dental image. Afterwards, the model is trained disjointly on the annotated dataset generated by $D_1$ and $D_2$. We examined the correctness of the annotated dataset using two different values of Intersection over Union (IoU). This procedure was adopted to check the overlapping ratio of expected and truth rectangles. Average precision measure is used to validate the correctness of the annotated dataset. The dataset information and the results obtained are displayed in Table 2.

Table 2 highlights the correctness and validity of our gathered and annotated datasets. The table depicts the comparison of 816 locally annotated instances of the X-ray radiographs and publicly available instances of 120 X-ray radiographs and 90 colored photographs. On average, a subtle difference of 4% between the scores $D_1$ and $D_2$ is observed. Hence, 96% of agreement exists between the dentists on the correctness of locally annotated and publicly available datasets of annotated images. It shows the high accuracy of the locally gathered and annotated dataset of dental images.

M-RCNN model evaluation

We split our datasets into training and testing sets. The training dataset is used to train the proposed M-RCNN model. The testing dataset is used to evaluate the accuracy of the proposed trained M-RCNN model. We trained and tested the proposed M-RCNN model on the colored photographs, X-ray radiographs, and mixed images dataset seperately. The training and testing dataset split ratio was set respectively to 70:30 for colored photographic dataset and 80:20 for the X-ray radiographic and mixed datasets, the most commonly used dataset splitting standard in machine learning research (Wenzel, 2002). Due to the lack of a large number of the publically available colored dataset, we increased the training dataset size for the colored photographic dataset to produce realistic results. We assessed the performance of the proposed model using precision and recall measures over the colored photographic, X-ray radiographic, and mixed dental image datasets. The experimental parameters and results obtained are explained in subsequent sections.

Empirical results

The empirical evaluation via precision and recall gives satisfactory results. The difference in the precision and recall measures on different types of dental image datasets is marginal. Hence the proposed M-RCNN is equally applicable to the different types of dental images datasets. Figure 4 shows the testing and training dataset results. In the colored photographic dataset, our approach achieved 88.02% and 85.45% precision in training and testing, respectively. Our approach gained 90.32% and 88.05% recall in the training and testing dataset, respectively. In the X-ray radiographic dataset, the precision in training and testing is 98.73% and 95.75%, respectively. The recall in training and testing is 98.75% and 96.24%, respectively. We achieved 85.67% and 81.02% precision in training and testing in the mixed dataset, respectively. We reached 86.5% and 73.78% recall in training and testing, respectively.

The dental carious regions’ localization results of the M-RCNN model are relatively similar. The difference over distinct datasets containing colored photographic, X-ray radiographic, and mixed dental images is marginal. However, results are somewhat compromised over the separate datasets. The results can be fine-tuned by taking the optimized dental images and by applying different hyper-parameters. We examined the correctness of colored photographic, X-ray radiographic, and mixed images, i.e., $D{S}_1$, $D{S}_2$, and $D{S}_3$, respectively, by employing different values of IoU. We adopted the procedure to check the overlapping ratio of correct results over other datasets by exploiting the proposed M-RCNN model. The precision and recall measures are used to validate the correctness of the distinct (colored photographic, X-ray radiographic, and mixed) annotated dataset. Table 4 depicts the dataset information and the corresponding results.

Table 4:

Correctness validation scores in datasets DS1, DS2, and DS2 containing colored photographic, X-ray radiographic, and mixed dental images, respectively by employing M-RCNN model in terms of precision (P) and recall (R) measures.

	DS1: Colored dental dataset		DS2: X-ray dental dataset		DS3: Mixed dataset
	P	R	P	R	P	R
@IoU = 0.6	95.75	96.24	84.45	85.05	81.02	83.78
@IoU = 0.7	89.06	92.09	79.54	80.45	76.02	78.78

DOI: 10.7717/peerj-cs.888/table-4

The tool’s evaluation

We measured the usability of our designed and developed dental cavity detection tool using a standard System Usability Scale (SUS) instrument given in literature (Bangor, Kortum & Miller, 2009; Brooke, 2021). The SUS instrument consists of 10-items to uncover the diverse usability dimensions in the design of interactive tools. The instrument is available at a web source (https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html). According to the literature, a minimum of 12–14 participants are required to reveal satisfactory results (Tullis & Stetson, 2004; Nielsen & Landauer, 1993). Therefore, we invited 17 participants. Among the 17 participants, 3 were dentists, 4 were dentist assistants, and 10 were final-year dental students. The evaluation was conducted in a silent room, having a workstation equipped with necessary hardware and software support.

The participants were asked to perform a complex scenario containing the structured sub-tasks after giving them a brief introduction and demonstration of the dental cavity detection tool. In the scenario, the users were asked to (i) capture a dental photograph image having a cavity via ordinary camera, (ii) upload a dental image over the dental cavity detection tool, (iii) analyze the carious region localized by the dental cavity detection tool, (iv) capture a dental photograph image having no cavity via ordinary camera, (v) upload a dental image over the dental cavity detection tool, and (vi) analyze the suggested results and exit the dental cavity detection tool.

Upon completing the mentioned scenario, the participants were asked to give responses via the SUS instrument. Table 5 highlights the SUS experimental results. According to the SUS analysis, the dentists were the most satisfied with the tool, with the SUS scale score obtained 90.8%. Then, the students were the most satisfied, with a SUS scale score equal to 85.3%. Finally, the dentist assistants also rated the proposed dental cavity detection tool, and the SUS score obtained 84.4%. The usability ratings provided by each participant are shown in Table 5. Overall, the proposed dental cavity detection tool achieved an 86.82% SUS scale score, graded as A+ grade, and falls in the top 95–100 percentile software category in terms of usability (Lewis & Sauro, 2018).