Next Article in Journal
Procrastination Mediates the Relationship between Problematic TikTok Use and Depression among Young Adults
Previous Article in Journal
The Impact of Anti-Amyloid Immunotherapies on Stroke Care
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Diagnostic Accuracy of Cone Beam Computed Tomography and Periapical Radiography for Detecting Apical Root Resorption in Retention Phase of Orthodontic Patients: A Cross-Sectional Study

1
Institute of Orthodontics, Faculty of Medicine, University of Coimbra, 3000-075 Coimbra, Portugal
2
Center for Innovation and Research in Oral Sciences (CIROS), Faculty of Medicine, University of Coimbra, 3000-075 Coimbra, Portugal
3
Forensic Dentistry Laboratory, Faculty of Medicine, University of Coimbra, 3000-075 Coimbra, Portugal
4
Dentistry Department, Faculty of Medicine, University of Coimbra, 3000-075 Coimbra, Portugal
5
Coimbra Institute of Engineering, Polytechnic Institute of Coimbra, 3030-199 Coimbra, Portugal
6
Research Centre in Asset Management and System Engineering (RCM2+), Polytechnic Institute of Coimbra, 3030-199 Coimbra, Portugal
7
Institute of Endodontics, Faculty of Medicine, University of Coimbra, 3000-075 Coimbra, Portugal
8
Coimbra Institute for Clinical and Biomedical Research (iCBR) and Center of Investigation on Environment Genetics and Oncobiology (CIMAGO), Faculty of Medicine, University of Coimbra, 3000-548 Coimbra, Portugal
*
Author to whom correspondence should be addressed.
J. Clin. Med. 2024, 13(5), 1248; https://doi.org/10.3390/jcm13051248
Submission received: 18 January 2024 / Revised: 11 February 2024 / Accepted: 19 February 2024 / Published: 22 February 2024
(This article belongs to the Section Dentistry, Oral Surgery and Oral Medicine)

Abstract

:
Objectives: This clinical study aimed to evaluate and compare the diagnostic accuracy of intraoral periapical radiography (PR) and cone beam computed tomography (CBCT) in detecting external apical root resorption (EARR) in orthodontic patients during the retention phase. Methods: The research involved 41 Caucasian patients who had undergone comprehensive orthodontic treatment, with a total of 328 teeth analyzed. The Kappa values for inter- and intra-examiner agreement were high for both PR and CBCT, indicating a robust level of agreement among examiners. The study used a four-point scale for classifying EARR. Results: This study showed comparable accuracy, sensitivity, and specificity between PR and CBCT when using the most stringent criterion of “Definitely present”. The data suggested that CBCT outperformed PR when using a less stringent criterion (“Definitely present” or “Probably present”), particularly for maxillary incisors. However, overall diagnostic performance, as measured by the area under the ROC curve, showed only a slight advantage for CBCT over PR. Areas under the ROC curve range between 0.85 and 0.90 for PR and between 0.89 and 0.92 for CBCT. According to DeLong’s test, there is no evidence to conclude that the area under the ROC curve is different for PR and CBCT. Conclusions: Both PR and CBCT are accurate diagnostic tools for identifying EARR, with PR being deemed more suitable for routine clinical use due to its cost-effectiveness and lower radiation exposure. The findings emphasize the importance of considering the risk-benefit ratio when deciding on imaging modalities for monitoring EARR in orthodontic patients.

1. Introduction

External apical root resorption (EARR) is a common undesirable iatrogenic outcome of orthodontic treatment [1,2,3]. It is characterized by the permanent loss of apical radicular tissue (cementum and dentin) as a result of the action of clastic cells released during the inflammatory process induced by the orthodontic forces [4,5,6,7]. Root resorption is a complex phenotype with a multifactorial etiology associated with biological and mechanical factors [8,9,10,11]. Although all dentition can be affected by root resorption, the literature supports the idea that maxillary incisors are the most frequently and severely affected teeth, followed by mandibular incisors and first molars [12,13,14].
Much effort has been made to determine a scale to classify EARR severity. This is a hard task due to the complex etiology of root resorption and the difficulty of interpreting diagnostic exams. EARR is frequently classified as mild, moderate, severe, or extreme loss. In severe and extreme cases, root reduction can compromise tooth function and longevity [15,16,17]. Thus, it is extremely important to conduct radiographic monitoring of orthodontic patients to diagnose and control EARR. According to Fuss et al. [18], radiographically, the loss of root structure is located in the apical third of the root, and no signs of radiolucency can be observed in the bone or root. Several imaging exams, such as panoramic radiography, periapical radiography (PR), face profile teleradiography, and cone beam computed tomography (CBCT), have been used to diagnose EARR [19,20,21,22].
Periapical radiography is the most frequently used imaging method in daily practice [17]. The parallax technique allows us to obtain accurate and reproducible radiographic images and is used to define the lesion’s location [2,23,24]. However, being a two-dimensional technique, PR has limitations when evaluating the damage to the dental root. In addition, the overlap of anatomic structures and the distortion projection errors may result in blurred images [4,17,23,25]. Due to these factors, many studies claim that 3D lesions’ inference from PR is not reliable [4,26,27]. Furthermore, Andreasen et al. [28] and Goldberg et al. [29] reported that resorption lesions less than 0.3 mm deep and 0.6 mm diameter cannot be detected by periapical radiography.
CBCT has proven to be a promising alternative method to diagnose EARR [30]. This tool allows high-quality 3D image visualization of maxillofacial structures. Volumetric analysis of the root resorption provides accurate localization and quantification of the resorption lesions [25,27,31]. Compared with conventional computed tomography [2,5], CBCT needs lower doses of radiation and also produces images with no anatomical overlapping and with little distortion, allowing high sensitivity and specificity of images [5,25]. Conventional X-ray techniques, such as PR, require lower doses of radiation than CBCT, provided that the field of view (FOV) and voxel size are in the standard orthodontics clinical range [25]. Further, CBCT radiation doses can be multiplied by a factor of fifteen, depending on the selected resolution, for the same field of view [32].
Several studies have been published to discuss the ideal diagnostic exam (high validity and reliability) for root resorption. These studies used different samples and methods to obtain their results: teeth with external and internal inflammatory resorption (associated with endodontic treatment or not) [4,21]; artificial and natural root resorption in extracted teeth [17]; extracted deciduous teeth [23]. Freitas et al. [33] developed a pioneer in vivo study to assess differences in the frequency of EARR using 58 patients, explained by the imaging method (PR or CBCT). In the previous studies, the presence of EARR was only compared after orthodontic treatment related to long-term concerns. Evaluation PR versus CBCT as diagnostic tools were not yet reported in terms of accuracy, sensitivity, specificity, or other indicators, such as the Area Under the Receiver Operating Characteristic (ROC) Curve (AUC). The ROC curve shows the trade-off between sensitivity and specificity across different threshold settings for a classifier. It is a numerical measure of the ROC curve’s performance. The higher the AUC, the better the classifier is at distinguishing between positive and negative cases.
To the best of our knowledge, other than studies with impacted canines [34], there are no in vivo studies comparing the diagnostic accuracy of two imaging systems, PR and CBCT, for detecting EARR in orthodontic patients in the retention phase. The aim of this research was to evaluate and compare the accuracy of intraoral PR with CBCT in the diagnosis of external apical root resorption of orthodontic patients in the retention phase.

2. Materials and Methods

2.1. Data Collection

This study was a diagnostic accuracy cross-sectional study, and the patients from the Orthodontic Institute of the Faculty of Medicine of the University of Coimbra were invited to participate.
Those who accepted and met the inclusion criteria were selected, resulting in a study that included a total of 41 Caucasian patients.
The inclusion criteria for patient selection were established as follows: patients of Portuguese Caucasian origin; have performed a CBCT examination because of a particular condition of impacted wisdom teeth previously to this study; have received comprehensive orthodontic treatment (straight-wire technique); have a clinical file allowing the collection of the patient’s complete clinical information; patients in a retention phase for at least 6 months; with no genetic craniofacial malformation and no congenitally missing teeth; have no supernumerary or impacted canines or incisors; have no incisors with endodontic treatment and absence of periodontal disease.
The 41 patients included 14 males and 27 females, with an average age of 24.37 years (s.d. ± 5.9). In what concerns to the initial orthodontic diagnosis, all the patients had class I and Class II malocclusion, medium or thick gingival biotype, a reasonable quantity of bone in both incisor regions, and reasonable oral hygiene. In relation to the treatment plan, the average duration of orthodontic treatment was 27 ± 6 months; the patients didn’t need teeth extractions, and the incisor movements were retroinclination, proinclination, root torque, and intrusion. However, the intrusion was only in moderate amounts. All maxillary and mandibular incisors (eight teeth) were evaluated per patient, resulting in a total of 328 teeth being analyzed.
The purpose of this study was explained to all participants, and ethical committee approval was received from the Faculty of Medicine, University of Coimbra (ref. CE-020/2017, issued on 27 March 2017). Written consent forms were also obtained in accordance with the ethical principles of medical research and human rights, as stated in the Helsinki Declaration (2002 version, www.wma.net/e/policy/b3.htm, accessed on 3 January 2020), and details of such approval are included in the text.

2.2. Radiographic Technique

The digital periapical radiographs were taken by an intra-oral X-ray machine (Siemens®, Heliodent EC, 70 Dentotime, Munich, Germany) using a photosensitive phosphor plate (Carestream Dental, Atlanta, GA, USA, CS 7600, size 2—31 mm × 41 mm), with exposure parameters of 60 KV, 7 mA and 0.08 s. The images were obtained using the digital imaging system (Carestream Dental, CS 7600), and all the PR were obtained using the parallelism technique and an intraoral positioner Rinn XCP (Dentsply Sirona™, Charlotte, NC, USA). For each patient, three radiographs were performed (teeth 11 and 12; teeth 21 and 22; teeth 42, 41, 31, and 32), resulting in a total of 123 radiographs (Figure 1).

2.3. CBCT Technique

The DICOM images of CBCT were taken using 3D equipment (i-CAT®, Imaging Sciences International, Hatfield, PA, USA). Tomographic images were obtained with the following parameters: voxel size 0.25 mm, exposure time 14.5 s, and FOV (Field-of-View) 100–160 mm.
The CBCT slice analysis was performed with the OnDemand3DTM App Software, https://www.ondemand3d.com assessed on 5 May 2021 (Cybermed, Inc., Seoul, Republic of Korea). All images were assessed on a portable computer in a dark room with no time limit. A video of the process is available as Supplementary Material (Video S1). The 3D tool was selected, followed by the manual adjustment of the long axis of the anterior teeth in the upper and lower arch. The correct orientation of each tooth axis was performed in the three planes (Figure 2) with cut intervals of 0.125 mm and a thickness of 1 mm. For the dental analysis, the image of the sagittal plane coinciding with the corono-radicular longitudinal axis was chosen for each tooth (Figure 3).

2.4. Radiological Assessment

A committee of two skilled dental doctors with more than 20 years of clinical experience each decided, for each tooth, the presence or absence of EARR. This committee used PR and CBCT simultaneously to reach a consensus classification (presence or absence), the so-called gold-standard diagnostic for each tooth. Due to an inconclusive consensus diagnosis of mild root resorption, 104 teeth from the 41 patients were excluded from the study.
The remaining 224 teeth were evaluated by a second group of examiners, four post-graduate orthodontics students in their third year of training. Before making their judgment about EARR, these examiners were re-trained and reminded of the main characteristics of root resorption. A special training program was carried out with examples of EARR in CBCT and PR in order to solidify their knowledge of the imaging interpretation of the characteristics of the EARR. The examiners were only included in this study after having demonstrated the appropriate skills to evaluate accurately. Direct access to manipulate and view the 3D DICOM images themselves was given.
The selected examiners evaluated EARR using Patel et al. [5] modified scoring system with a four-point scale: 1—“Definitely present”; 2—“Probably present”; 3—“Probably absent”; 4—“Definitely absent” (Table 1). A sequential evaluation approach was established as follows: 1st session—analysis of PR, for a maximum of 5 patients per day; 2nd session—analysis of CBCT scans, for a maximum of 5 patients per day; 3rd session—evaluate/repeat PR and CBCT scans of 3 random patients (24 teeth for each exam). The purpose of the third session was to assess the intra-examiner agreement, which was performed two weeks after the previous sessions.

2.5. Data Analysis

The four-point scale used by examiners allows three different definitions of a positive test for EARR: (1) only if the interpretation was “Definitely present” (i.e., the most stringent criterion); (2) if the interpretation was “Definitely present” or “Probably present”; (3) if the interpretation was “Definitely present” or “Probably present” or “Probably not present” (i.e., the most lenient criterion). For the first two criteria, accuracy, sensitivity, specificity, and predictive values were determined using the gold standard of the consensus committee. These calculations were performed for each examiner and diagnostic test (PR and CBCT), and their means and standard deviations by diagnostic test were reported.
Three pairs of sensitivity and specificity values are needed to describe the overall performance of the diagnostic tests (CBCT and PR). For each examiner and diagnostic test, the area under the ROC curve, defined by those three dots, was used as a measure of overall diagnostic performance, and means and standard deviations by the diagnostic test were reported. Also, for each examiner, CBCT and PR areas under the ROC curve were non-parametrically compared using DeLong’s test.
Results were also reported separately for the following teeth groups: maxillary and mandibular incisors, mandibular incisors only, maxillary incisors only, central incisors, and lateral incisors.
A statistical computation was performed using R software (version 3.4.4) [34]. The R package pROC [35] was used to determine the area under the ROC curve and to perform DeLong’s test. Inter-examiner and intra-examiner agreements were assessed using the R package irr [36] and the metric ICC (Intra-class Correlation Coefficient), based on a mean rating and a 2-way random-effect model with a consistence agreement and absolute agreement, respectively.

3. Results

The Kappa value for inter-examiner agreement in the EARR diagnostic was 0.93 and 0.92 for PR and CBCT, respectively. The mean Kappa value for the intra-examiner agreement was 0.94 for both PR and CBCT.
Table 2 shows the frequency of teeth with EARR according to the consensus committee. EARR prevalence in this sample was between 67% (mandibular and central incisors) and 75% (lateral incisors).
Using the most stringent criterion for EARR presence (“Definitely present”), all the performance results are approximately the same for PR and CBCT. However, the data suggests that PR performance has more variability across examiners than CBCT (Table 3A). Specificity and positive predictive values are very high in all cases as a consequence of using the most stringent criterion. For both PR and CBCT, accuracy is quite low using these criteria, ranging from 0.49 (PR—mandibular incisors only) to 0.67 (CBCT—maxillary incisors only). EARR diagnosis in mandibular incisors turns out to be the hardest task for examiners.
Using the criteria for EARR presence “Definitely present” or “Probably present” (Table 3B), CBCT outperformed PR except for lateral incisors, where CBCT variability across examiners was higher than in other teeth groups. Overall, variability across examiners decreased when compared to the use of the “Definitely present” criterion only, and, as expected, specificity and positive predictive value have decreased. For both PR and CBCT, accuracy is good using these criteria, ranging from 0.77 (PR—maxillary incisors only and mandibular incisors only) to 0.93 (CBCT—maxillary incisors only).
Figure 4 depicts the ROC curve for each examiner for both PR and CBCT. Areas under the ROC curve are approximately the same for PR and CBCT. Thus, it suggests that, overall, using all three different definitions of a positive test for EARR, CBCT performs only slightly better than PR for all examiners.
However, according to DeLong’s test, there is no evidence to conclude that overall diagnostic performance is different for PR and CBCT (Table 4). It is worth noting that the results only suggest otherwise for one examiner and only for mandibular incisors. As the area under the ROC curve ranges between 0 and 1 and for both PR and CBCT values range between 0.83 (mandibular incisors) and 0.94 (maxillary and lateral incisors), it can be concluded that both have good ability to discriminate between teeth with and without EARR.

4. Discussion

During orthodontic treatment, radiological methods are crucial in identifying and monitoring root resorption due to the usual absence of clinical signs associated with this type of lesion before it reaches the latter stages [8,25,37].
Classification was performed using medical images (PR and CBCT) obtained after orthodontic treatment. The complexity of this task was increased by the fact that PR and CBCT images were not available before treatment. Consequently, the committee decided to exclude 104 out of the 328 teeth due to inconclusive consensus diagnoses about the presence of EARR. The excluded teeth were just those with mild EARR, thus out of the scope of clinical relevance.
A four-point scale was used by trained examiners to classify their level of confidence in detecting the presence of EARR, using PR and CBCT in different sessions. The Kappa values for inter-examiner agreement in the diagnosis of EARR were 0.93 and 0.94 for PR and CBCT, respectively, revealing a high level of agreement between examiners in this study. Patel et al. [5] had inter-examiner values of 0.365 and 0.925 for PR and CBCT, respectively, suggesting a poor concordance for PR. Schröder et al. [17] also reported an inter-examiner agreement greater than 0.80 for CBCT. The studies with high inter-examiner values are probably related to better training in PR interpretation. Regarding the intra-examiner agreement, Patel et al. [5] reported 0.625 for PR and 0.966 for CBCT, and Schröder et al. [17] obtained a mean value greater than 0.80 for CBCT. In the present study, the mean intra-examiner agreement was 0.94 for both PR and CBCT, suggesting a high level of agreement at different time points. The excellent concordance level for PR might be related to (1) the periapical radiography parallelism technique being well performed, (2) the use of digital PR instead of analogical PR, and (3) examiners who were well-trained in PR interpretation.
The scale used by the examiners in this study allowed three different definitions of a positive test for EARR. The strictest criterion, “Definitely present”, results in values for accuracy, sensitivity, specificity, positive predictive values, and negative predictive values for PR that differ from CBCT by less than 1%. Creanga et al. [38] and Schröder et al. [17] reported differences in sensitivity and specificity for PR and CBCT that range from 18% for sensitivity to 8% for specificity. Regarding the intermediate criterion for a positive test, “Definitely present” or “Probably present”, the accuracy of CBCT outperformed PR by about 6% for all incisors and about 16% for maxillary incisors. For lateral incisors, the accuracy of PR was better than CBCT by 6%. For both PR and CBCT, accuracy is higher using this criterion (77–93%) than the strictest criterion, “Definitely present”. EARR diagnosis of mandibular incisors turned out to be the hardest location for the examiners.
According to Metz [39], a value of 0.75–0.80 for the area under the ROC curve indicates that the imaging method is acceptable. In our study, the diagnostic accuracy for the detection of EARR was between 0.85 and 0.90 for PR and between 0.89 and 0.92 for CBCT. This suggests that CBCT performs only slightly better than PR. Therefore, based on areas under the ROC curve, we can conclude that both tests are accurate diagnostic tools to identify apical root resorption. The unexpectedly high performance of PR diagnostic could be a result of the irregular margins of root resorption lesions, which can be pathognomonic in this kind of lesion.
According to our study, PR and CBCT seem to be reliable tools to analyze and diagnose EARR. Our results are supported by a systematic review [25] concluding that the CBCT is a valuable tool to examine EARR during or at the end of orthodontic treatment. However, the average EARR measured with CBCT seems to lack clinical relevance. Another systematic review and meta-analysis [1] refer to a different idea, suggesting that CBCT has a higher diagnostic efficacy than PR. A recent study [40] of the diagnostic accuracy of CBCT and 2D imaging methods in the 3D localization and assessment of maxillary impacted canines suggests that the diagnostic accuracy of CBCT outperformed 2D radiography in localizing the position of the impacted canines and the resorption of the adjacent incisors. However, a comparison with the present study is difficult because they are treating a special situation of impacted canines.
In 2022, another recent systematic review [41] compared CBCT and panoramic radiography for the assessment of root resorption on the second molar associated with third molar impaction. This is relevant for our patients who present this condition. They concluded that more EARR is assessed in CBCT compared to panoramic, but there was considerable agreement between these two medical exams in the assessment of EARR, especially in the absence of the pathology rather than its presence.
Other authors [42] performed a comparative analysis of the accuracy of PR and CBCT for diagnosing complex endodontic pathoses, and CBCT had higher diagnostic accuracy in complex endodontic pathoses compared to PR. Nevertheless, CBCT failed to diagnose some pathologies in 33% of teeth, concluding that CBCT should be considered for selective cases where PR has diagnostic ambiguity.
Several authors have provided support for our study outcomes. Earlier investigations [8,26] found no significant disparities in the accuracy of root defect detection between periapical radiography (PR) and cone-beam computed tomography (CBCT). However, contrasting perspectives exist in the literature. Notably, Patel et al. [5] asserted that CBCT exhibits superior accuracy compared to PR in an endodontic clinical study. These findings align with the observations of Yi et al. [1], who demonstrated significantly elevated sensitivity and a larger area under the receiver operating characteristic (ROC) curve for CBCT based on a meta-analysis incorporating only five out of fifteen studies comparing PR to CBCT, all conducted in simulated models. It is crucial to acknowledge the limitations of such simulations, as induced lesions on the root surface, created artificially with mechanical instruments, may not faithfully replicate the true clinical nature of lesions. Given that resorption manifests with irregular cavities rather than perfect hemispherical shapes [2,4,6,17,38,43], the sensitivity and specificity of imaging methods may not accurately reproduce clinical conditions.
Scientific evidence [17] underscores the inherent challenge of observing and identifying in vivo external apical root resorption (EARR) compared to ex vivo scenarios, rendering a direct comparison with our in vivo results challenging. Furthermore, orthodontic research, relying on limited scientific evidence regarding performance metrics for the diagnosis and measurement of EARR, has been used to advocate the use of CBCT in some studies [44,45].
Our study presents some limitations: (1) lack of PR and CBCT images before orthodontic treatment (only used panoramic and cephalometric radiography) as PR and CBCT imaging methods were only used in an orthodontic retention phase, hampering the comparison of root length before and after treatment. However, this is not absolutely essential to the research once our aim was to evaluate and compare the accuracy of PR with CBCT in the diagnosis of EARR and not the development of root resorption due to the orthodontic treatment; (2) difficulty in applying an EARR quantitative measure and confirming the real extension of root resorption lesions (in vivo study); (3) the absence of a sample size calculation; and (4) the evaluation of the eight incisors instead of all the teeth.
The reason why we decided not to evaluate all teeth, and instead only the upper and lower incisors, was due to the high radiation that it would require to perform PR on all the teeth. Nevertheless, our work overcomes some limitations encountered in prior studies. First, in contrast to other in vivo investigations, this study encompasses a notably larger sample size [2,4,6,38,43]. Second, our approach avoids reliance on artificial lesions, enabling a genuine clinical assessment of imaging methods for root resorption. Consequently, our study is more pragmatic and rooted in everyday clinical observations. Lastly, our methodology facilitates performance analysis not only in terms of positive and negative outcomes for external apical root resorption (EARR) but also permits a nuanced clinical evaluation ranging from “Definitely present” to “Probably present.” These enhancements contribute to the robustness and clinical relevance of our findings, providing a more comprehensive understanding of the diagnostic capabilities of the assessed imaging methods in the context of EARR.
In clinical practice, PR is a reasonably priced, simple, and quick image acquisition tool [46]. However, for EARR detection, there are some limitations that require special care regarding the interpretation of the obtained results, namely the 2D images of a 3D structure, the presence of anatomical overlap, or the production of distorted and blurred images [37]. Due to these limitations, the detection and grading of smaller lesions situated in certain root surface sites become more difficult and less accurate [5,23]. Previous studies indicated that the 2D nature of the images of periapical radiography is less accurate in determining the location and severity/size of resorption lesions [15,38]. However, the acquisition of PR images from different angulations allows the gathering of more information. On the other hand, CBCT produces good-quality 3D images and eliminates anatomical overlaps [37], enabling the detection of EARR even when the lesions are smaller than 1 mm.
The present study did not find significant statistical differences in the performance of PR and CBCT for root resorption detection. Based on our findings, on the guidelines recently developed [19,21], on the high cost, on the radioactive burden, and based on ALARA (As Low as Reasonably Achievable) principles, CBCT should not be considered a routine exam, and its risk/benefit ratio must be taken into account.
Given that root resorption is an ongoing process necessitating vigilant monitoring throughout treatment, periapical radiography (PR) emerges as the preferred method for routine clinical use. However, in instances involving intricate cases or where there is a suspicion of a more severe form of external apical root resorption (EARR), cone-beam computed tomography (CBCT) can prove to be a pivotal asset [47]. CBCT facilitates a more comprehensive assessment, thereby enhancing informed decision-making for devising an optimal treatment plan and striving for the most favorable outcome.
In future clinical investigations, it would be worthwhile to undertake a similar study incorporating varying periapical radiographic angulations. Additionally, the inclusion of before-and-after treatment radiographic images could provide valuable insights, further advancing our understanding of the diagnostic capabilities of different imaging modalities in monitoring and assessing root resorption progression over the course of treatment.

5. Conclusions

This study contributes valuable insights by revealing the absence of significant differences in the accuracy of detecting external apical root resorption (EARR) associated with orthodontic treatment between periapical radiography (PR) and cone-beam tomography. In weighing the advantages and disadvantages of these two imaging methods, our findings suggest that periapical radiography (PR) stands out as the most suitable radiographic technique for the precise diagnosis of EARR in orthodontic clinical practice. This recommendation is grounded in the study’s rigorous evaluation of diagnostic accuracy and underscores the pragmatic utility of PR in effectively assessing and monitoring EARR in the orthodontic context.

Supplementary Materials

The following supporting information can be downloaded at https://www.mdpi.com/article/10.3390/jcm13051248/s1, Video S1: Video demonstrating the CBCT assessment.

Author Contributions

Conceptualization, S.A.P., A.C.-R. and N.L.; methodology, S.A.P. and N.L.; software, N.L. and A.C.-R.; validation, S.A.P., N.L. and J.M.S.; formal analysis, S.A.P. and J.M.S.; investigation, S.A.P., L.M. and A.M.; resources, S.A.P.; data curation, J.M.S.; writing—original draft preparation, S.A.P., A.C.-R., L.M., A.M. and N.L.; writing—review and editing, S.A.P. and J.M.S.; visualization, J.M.S.; supervision, S.A.P.; project administration, S.A.P. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki and approved by the Institutional Review Board (or Ethics Committee) from the Faculty of Medicine, University of Coimbra (ref. CE-020/2017, date of approval 27 March 2017).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data presented in this study are available on request from the first and the last authors.

Acknowledgments

We thank Vicente C., Scherzberg J., Queiroga J., and Bento S. from the Dentistry Department, Faculty of Medicine University Coimbra, Coimbra, Portugal, for their valuable assistance in this project.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Yi, J.; Sun, Y.; Li, Y.; Li, C.; Li, X.; Zhao, Z. Cone-beam computed tomography versus periapical radiograph for diagnosing external root resorption: A systematic review and meta-analysis. Angle Orthod. 2017, 87, 328–337. [Google Scholar] [CrossRef]
  2. Ren, H.; Chen, J.; Deng, F.; Zheng, L.; Liu, X.; Dong, Y. Comparison of cone-beam computed tomography and periapical radiography for detecting simulated apical root resorption. Angle Orthod. 2013, 83, 189–195. [Google Scholar] [CrossRef] [PubMed]
  3. Feller, L.; Khammissa, R.A.G.; Thomadakis, G.; Fourie, J.; Lemmer, J. Apical External Root Resorption and Repair in Orthodontic Tooth Movement: Biological Events. Biomed. Res. Int. 2016, 2016, 4864195. [Google Scholar] [CrossRef] [PubMed]
  4. Shokri, A.; Mortazavi, H.; Salemi, F.; Javadian, A.; Bakhtiari, H.; Matlabi, H. Diagnosis of simulated external root resorption using conventional intraoral film radiography, ccd, psp, and cbct: A comparison study. Biomed. J. 2013, 36, 18–22. [Google Scholar] [CrossRef] [PubMed]
  5. Patel, S.; Dawood, A.; Wilson, R.; Horner, K.; Mannocci, F. The detection and management of root resorption lesions using intraoral radiography and cone beam computed tomography—An in vivo investigation. Int. Endod. J. 2009, 42, 831–838. [Google Scholar] [CrossRef] [PubMed]
  6. Durack, C.; Patel, S.; Davies, J.; Wilson, R.; Mannocci, F. Diagnostic accuracy of small volume cone beam computed tomography and intraoral periapical radiography for the detection of simulated external inflammatory root resorption. Int. Endod. J. 2011, 44, 136–147. [Google Scholar] [CrossRef] [PubMed]
  7. Aidos, H.; Diogo, P.; Santos, J.M. Root resorption classifications: A narrative review and a clinical aid proposal for routine assessment. Eur. Endod. J. 2018, 3, 134–145. [Google Scholar] [CrossRef]
  8. Kumar, V.; Gossett, L.; Blattner, A.; Iwasaki, L.R.; Williams, K.; Nickel, J.C. Comparison between cone-beam computed tomography and intraoral digital radiography for assessment of tooth root lesions. Am. J. Orthod. Dentofac. Orthop. 2011, 139, e533–e541. [Google Scholar] [CrossRef]
  9. Mohandesan, H.; Ravanmehr, H.; Valaei, N. A radiographic analysis of external apical root resorption of maxillary incisors during active orthodontic treatment. Eur. J. Orthod. 2007, 29, 134–139. [Google Scholar] [CrossRef]
  10. Pereira, S.; Lavado, N.; Nogueira, L.; Lopez, M.; Abreu, J.; Silva, H. Polymorphisms of genes encoding P2X7R, IL-1B, OPG and RANK in orthodontic-induced apical root resorption. Oral Dis. 2014, 20, 659–667. [Google Scholar] [CrossRef]
  11. Deng, Y.; Sun, Y.; Xu, T. Evaluation of root resorption after comprehensive orthodontic treatment using cone beam computed tomography (CBCT): A meta-analysis. BMC Oral Health 2018, 18, 116. [Google Scholar] [CrossRef]
  12. Weltman, B.; Vig, K.W.L.; Fields, H.W.; Shanker, S.; Kaizar, E.E. Root resorption associated with orthodontic tooth movement: A systematic review. Am. J. Orthod. Dentofac. Orthop. 2010, 137, 462–476. [Google Scholar] [CrossRef] [PubMed]
  13. Årtun, J.; Van ’t Hullenaar, R.; Doppel, D.; Kuijpers-Jagtman, A.M. Identification of orthodontic patients at risk of severe apical root resorption. Am. J. Orthod. Dentofac. Orthop. 2009, 135, 448–455. [Google Scholar] [CrossRef] [PubMed]
  14. Aman, C.; Azevedo, B.; Bednar, E.; Chandiramami, S.; German, D.; Nicholson, E.; Nicholson, K.; Scarfe, W.C. Apical root resorption during orthodontic treatment with clear aligners: A retrospective study using cone-beam computed tomography. Am. J. Orthod. Dentofac. Orthop. 2018, 153, 842–851. [Google Scholar] [CrossRef] [PubMed]
  15. Ponder, S.N.; Benavides, E.; Kapila, S.; Hatch, N.E. Quantification of external root resorption by low- vs high-resolution cone-beam computed tomography and periapical radiography: A volumetric and linear analysis. Am. J. Orthod. Dentofac. Orthop. 2013, 143, 77–91. [Google Scholar] [CrossRef] [PubMed]
  16. Roscoe, M.G.; Meira, J.B.C.; Cattaneo, P.M. Association of orthodontic force system and root resorption: A systematic review. Am. J. Orthod. Dentofac. Orthop. 2015, 147, 610–626. [Google Scholar] [CrossRef] [PubMed]
  17. Deliga Schröder, Â.G.; Westphalen, F.H.; Schröder, J.C.; Fernandes, Â.; Westphalen, V.P.D. Accuracy of Digital Periapical Radiography and Cone-beam Computed Tomography for Diagnosis of Natural and Simulated External Root Resorption. J. Endod. 2018, 44, 1151–1158. [Google Scholar] [CrossRef] [PubMed]
  18. Fuss, Z.; Tsesis, I.; Lin, S. Root resorption—Diagnosis, classification and treatment choices based on stimulation factors. Dent. Traumatol. 2003, 19, 175–182. [Google Scholar] [CrossRef]
  19. Sondeijker, C.F.W.; Lamberts, A.A.; Beckmann, S.H.; Kuitert, R.B.; van Westing, K.; Persoon, S.; Kuijpers-Jagtman, A.M. Development of a clinical practice guideline for orthodontically induced external apical root resorption. Eur. J. Orthod. 2020, 42, 115–124. [Google Scholar] [CrossRef]
  20. NVVO. Wortelresorptie in de Orthodontie. 2018. Available online: https://www.orthodontist.nl/files/wortelresorptie-in-de-orthodontie.pdf (accessed on 20 April 2020).
  21. NVVO. Richtlijn Orthodontische Radiologie. 2018, pp. 1–122. Available online: https://orthodontist.nl/ (accessed on 20 April 2020).
  22. Parrales-Bravo, C.; Friedrichsdorf, S.P.; Costa, C.; Paiva, J.B.; Iglesias-Linares, A. Does endodontics influence radiological detection of external root resorption? An in vitro study. BMC Oral Health 2023, 23, 221. [Google Scholar] [CrossRef]
  23. Alamadi, E.; Alhazmi, H.; Hansen, K.; Lundgren, T.; Naoumova, J. A comparative study of cone beam computed tomography and conventional radiography in diagnosing the extent of root resorptions. Prog. Orthod. 2017, 18, 37. [Google Scholar] [CrossRef] [PubMed]
  24. Whaites, E. Princípios de Radiologia Odontológica, 4th ed.; Elsevier: Rio de Janeiro, Brazil, 2009. [Google Scholar]
  25. Samandara, A.; Papageorgiou, S.N.; Ioannidou-Marathiotou, I.; Kavvadia-Tsatala, S.; Papadopoulos, M.A. Evaluation of orthodontically induced external root resorption following orthodontic treatment using cone beam computed tomography (CBCT): A systematic review and meta-analysis. Eur. J. Orthod. 2019, 41, 67–79. [Google Scholar] [CrossRef] [PubMed]
  26. Lima, T.F.R.; Gamba, T.D.O.; Zaia, A.A.; Soares, A.D.J. Evaluation of cone beam computed tomography and periapical radiography in the diagnosis of root resorption. Aust. Dent. J. 2016, 61, 425–431. [Google Scholar] [CrossRef] [PubMed]
  27. Da Silva Campos, M.J.; Silva, K.S.; Gravina, M.A.; Fraga, M.R.; Vitral, R.W.F. Apical root resorption: The dark side of the root. Am. J. Orthod. Dentofac. Orthop. 2013, 143, 492–498. [Google Scholar] [CrossRef]
  28. Andreasen, F.M.; Sewerin, I.; Mandel, U.; Andreasen, J.O. Radiographic assessment of simulated root resorption cavities. Dent. Traumatol. 1987, 3, 21–27. [Google Scholar] [CrossRef] [PubMed]
  29. Goldberg, F.; De Siivio, A.; Dreyer, C. Radiographie assessment of simulated external root résorption cavities in maxillary incisors. Endod. Dent. Traumatol. 1998, 14, 133–136. [Google Scholar] [CrossRef] [PubMed]
  30. Guo, Y.; He, S.; Gu, T.; Liu, Y.; Chen, S. Genetic and clinical risk factors of root resorption associated with orthodontic treatment. Am. J. Orthod. Dentofac. Orthop. 2016, 150, 283–289. [Google Scholar] [CrossRef]
  31. Kapila, S.D.; Nervina, J.M. CBCT in orthodontics: Assessment of treatment outcomes and indications for its use. Dentomaxillofac. Radiol. 2015, 44, 20140282. [Google Scholar] [CrossRef]
  32. Ludlow, J.B.; Timothy, R.; Walker, C.; Hunter, R.; Benavides, E.; Samuelson, D.B.; Scheske, M.J. Ffective dose of dental CBCT—A meta analysis of published data and additional data for nine CBCT units. Dentomaxillofac. Radiol. 2015, 44, 20140197. [Google Scholar] [CrossRef]
  33. de Freitas, J.C.; Lyra, O.C.P.; de Alencar, A.H.G.; Estrela, C. Long-term evaluation of apical root resorption after orthodontic treatment using periapical radiography and cone beam computed tomography. Dent. Press J. Orthod. 2013, 18, 104–112. [Google Scholar] [CrossRef]
  34. R Core Team. R: A Language and Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2018. [Google Scholar]
  35. Robin, X.; Turck, N.; Hainard, A.; Tiberti, N.; Lisacek, F.; Sanchez, J.-C.; Müller, M. pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinform. 2011, 12, 77. [Google Scholar] [CrossRef]
  36. Gamer Matthias, L.J.; SIFP. irr: Various Coefficients of Interrater Reliability and Agreement, R Package Version 084; R Foundation for Statistical Computing: Vienna, Austria, 2012. [Google Scholar]
  37. Dudic, A.; Giannopoulou, C.; Leuzinger, M.; Kiliaridis, S. Detection of apical root resorption after orthodontic treatment by using panoramic radiography and cone-beam computed tomography of super-high resolution. Am. J. Orthod. Dentofac. Orthop. 2009, 135, 434–437. [Google Scholar] [CrossRef] [PubMed]
  38. Creanga, A.G.; Geha, H.; Sankar, V.; Teixeira, F.B.; McMahan, C.A.; Noujeim, M. Accuracy of digital periapical radiography and cone-beam computed tomography in detecting external root resorption. Imaging Sci. Dent. 2015, 45, 153–158. [Google Scholar] [CrossRef] [PubMed]
  39. Metz, C. Some practical issues of experimental design and data analysis in radiological ROC studies. Investig. Radiol. 1989, 24, 234–245. [Google Scholar] [CrossRef]
  40. Alfailany, D.T.; Shaweesh, A.I.; Hajeer, M.Y.; Brad, B.; Alhaffar, J.B. The diagnostic accuracy of cone-beam computed tomography and two-dimensional imaging methods in the 3D localization and assessment of maxillary impacted canines compared to the gold standard in-vivo readings: A cross-sectional study. Int. Orthod. 2023, 21, 100780. [Google Scholar] [CrossRef] [PubMed]
  41. Moreira-Souza, L.; Butini Oliveira, L.; Gaêta-Araujo, H.; Almeida-Marques, M.; Asprino, L.; Oenning, A.C. Comparison of CBCT and panoramic radiography for the assessment of bone loss and root resorption on the second molar associated with third molar impaction: A systematic review. Dentomaxillofac. Radiol. 2022, 51, 20210217. [Google Scholar] [CrossRef] [PubMed]
  42. Keerthana, G.; Singh, N.; Yadav, R.; Duhan, J.; Tewari, S.; Gupta, A.; Sangwan, P.; Mittal, S. Comparative analysis of the accuracy of periapical radiography and cone-beam computed tomography for diagnosing complex endodontic pathoses using a gold standard reference—A prospective clinical study. Int. Endod. J. 2021, 54, 1448–1461. [Google Scholar] [CrossRef]
  43. Kamburoǧlu, K.; Kurşun, Ş.; Yüksel, S.; Öztaş, B. Observer ability to detect ex vivo simulated internal or external cervical root resorption. J. Endod. 2011, 37, 168–175. [Google Scholar] [CrossRef]
  44. Li, Y.; Deng, S.; Mei, L.; Li, Z.; Zhang, X.; Yang, C.; Li, Y. Prevalence and severity of apical root resorption during orthodontic treatment with clear aligners and fixed appliances: A cone beam computed tomography study. Prog. Orthod. 2020, 21, 1. [Google Scholar] [CrossRef]
  45. Aras, I.; Unal, I.; Huniler, G.; Aras, A. Root resorption due to orthodontic treatment using self-ligating and conventional brackets: A cone-beam computed tomography study. J. Orofac. Orthop. 2018, 79, 181–190. [Google Scholar] [CrossRef]
  46. Bernardes, R.A.; de Paulo, R.S.; Pereira, L.O.; Duarte, M.A.H.; Ordinola-Zapata, R.; de Azevedo, J.R. Comparative study of cone beam computed tomography and intraoral periapical radiographs in diagnosis of lingual-simulated external root resorptions. Dent. Traumatol. 2012, 28, 268–272. [Google Scholar] [CrossRef]
  47. Ahangari, Z.; Nasser, M.; Mahdian, M.; Fedorowicz, Z.; Marchesan, M.A. Interventions for the management of external root resorption. Cochrane Database Syst. Rev. 2015, 2015, CD008003. [Google Scholar] [CrossRef]
Figure 1. Sample of periapical radiography of the lower incisors assessed in this study.
Figure 1. Sample of periapical radiography of the lower incisors assessed in this study.
Jcm 13 01248 g001
Figure 2. Example of 3-dimensional orientation for CBCT images. The correct orientation of each tooth axis was performed in the three planes: (a) coronal slice, (b) sagittal slice, (c) axial slice, and (d) 3D reconstruction.
Figure 2. Example of 3-dimensional orientation for CBCT images. The correct orientation of each tooth axis was performed in the three planes: (a) coronal slice, (b) sagittal slice, (c) axial slice, and (d) 3D reconstruction.
Jcm 13 01248 g002
Figure 3. Example of CBCT Sagittal slices used to evaluate the upper and lower incisors of a patient, representing: (a) tooth 12; (b) tooth 11; (c) tooth 21; (d) tooth 22; (e) tooth 42; (f) tooth 41; (g) tooth 31; (h) tooth 32.
Figure 3. Example of CBCT Sagittal slices used to evaluate the upper and lower incisors of a patient, representing: (a) tooth 12; (b) tooth 11; (c) tooth 21; (d) tooth 22; (e) tooth 42; (f) tooth 41; (g) tooth 31; (h) tooth 32.
Jcm 13 01248 g003
Figure 4. ROC curves by examiner: (a) using PR; (b) using CBCT. AUROC1, AUROC2, AUROC3, and AUROC4—non-parametric area under the ROC curve for examiner 1, 2, 3, 4, respectively.
Figure 4. ROC curves by examiner: (a) using PR; (b) using CBCT. AUROC1, AUROC2, AUROC3, and AUROC4—non-parametric area under the ROC curve for examiner 1, 2, 3, 4, respectively.
Jcm 13 01248 g004
Table 1. The questionnaire used to assess each tooth by the committee of two highly skilled dental doctors (Committee) and by the group of four post-graduate orthodontists (Examiners).
Table 1. The questionnaire used to assess each tooth by the committee of two highly skilled dental doctors (Committee) and by the group of four post-graduate orthodontists (Examiners).
PresentAbsent
Committee
EARRDefinitely PresentProbably PresentProbably AbsentDefinitely Absent
Examiners
Table 2. Absolute frequency (relative frequency) of teeth with EARR according to the consensus committee.
Table 2. Absolute frequency (relative frequency) of teeth with EARR according to the consensus committee.
Teeth GroupsEARR Present
n (%)
EARR not Present
n (%)
n
maxillary and mandibular incisors159 (71%)65 (29%)224
maxillary incisors only89 (74%)31 (26%)120
mandibular incisors only70 (67%)34 (33%)104
central incisors72 (67%)36 (33%)108
lateral incisors87 (75%)29 (25%)116
Table 3. (A). Mean (standard deviation) of accuracy, sensitivity, specificity, PPV and NPV for EARR diagnostic using PR and CBCT at the most stringent criterion “Definitely present”. (B). Mean (standard deviation) of accuracy, sensitivity, specificity, PPV and NPV for EARR diagnostic using PR and CBCT with the criteria “Definitely present” or “Probably present”.
Table 3. (A). Mean (standard deviation) of accuracy, sensitivity, specificity, PPV and NPV for EARR diagnostic using PR and CBCT at the most stringent criterion “Definitely present”. (B). Mean (standard deviation) of accuracy, sensitivity, specificity, PPV and NPV for EARR diagnostic using PR and CBCT with the criteria “Definitely present” or “Probably present”.
(A)
Teeth groups AccuracySensitivitySpecificityPPVNPV
maxillary and mandibular incisorsPR0.60 (0.13)0.43 (0.19)0.99 (0.02)0.99 (0.09)0.43 (0.08)
CBCT0.59 (0.04)0.43 (0.05)0.99 (0.01)0.99 (0.01)0.42 (0.02)
maxillary incisors onlyPR0.69 (0.14)0.59 (0.20)0.98 (0.03)0.99 (0.01)0.48 (0.12)
CBCT0.67 (0.06)0.56 (0.07)0.98 (0.03)0.99 (0.02)0.44 (0.05)
mandibular incisors onlyPR0.49 (0.13)0.24 (0.19)1 (0.00)1 (0.00)0.40 (0.07)
CBCT0.50 (0.05)0.26 (0.08)0.99 (0.01)0.99 (0.03)0.49 (0.03)
central incisorsPR0.63 (0.14)0.45 (0.22)0.99 (0.03)0.99 (0.02)0.49 (0.1)
CBCT0.64 (0.03)0.47 (0.04)0.99 (0.03)0.99 (0.03)0.48 (0.02)
lateral incisorsPR0.56 (0.13)0.42 (0.17)1 (0.00)1 (0.00)0.37 (0.07)
CBCT0.55 (0.05)0.40 (0.07)0.99 (0.02)0.99 (0.01)0.36 (0.03)
(B)
Teeth groups AccuracySensitivitySpecificityPPVNPV
maxillary and mandibular incisorsPR0.82 (0.03)0.83 (0.08)0.79 (0.11)0.91 (0.04)0.67 (0.08)
CBCT0.88 (0.03)0.89 (0.05)0.84 (0.04)0.93 (0.01)0.78 (0.08)
maxillary incisors onlyPR0.77 (0.17)0.79 (0.29)0.73 (0.23)0.91 (0.07)0.69 (0.25)
CBCT0.93 (0.01)0.96 (0.03)0.85 (0.09)0.95 (0.03)0.87 (0.06)
mandibular incisors onlyPR0.77 (0.05)0.73 (0.12)0.87 (0.10)0.93 (0.05)0.62 (0.08)
CBCT0.83 (0.07)0.82 (0.10)0.84 (0.09)0.91 (0.04)0.71 (0.11)
central incisorsPR0.79 (0.02)0.84 (0.07)0.71 (0.13)0.86 (0.05)0.70 (0.07)
CBCT0.88 (0.02)0.92 (0.03)0.82 (0.03)0.91 (0.02)0.84 (0.05)
lateral incisorsPR0.84 (0.04)0.83 (0.08)0.90 (0.10)0.96 (0.03)0.65 (0.10)
CBCT0.78 (0.17)0.74 (0.26)0.90 (0.10)0.96 (0.03)0.60 (0.20)
PPV—positive predictive value NPV—negative predictive value.
Table 4. Mean (standard deviation) of the area under the ROC curve for PR and CBCT.
Table 4. Mean (standard deviation) of the area under the ROC curve for PR and CBCT.
Teeth GroupsPRCBCTp-Value a
maxillary and mandibular incisors0.87 (0.02)0.90 (0.01)0.294, 0.228, 0.199, 0.436
maxillary incisors only0.91 (0.01)0.94 (0.03)0.072, 0.523, 0.131, 0.361
mandibular incisors only0.83 (0.02)0.86 (0.05)0.747, 0.012, 0.671, 0.803
central incisors0.85 (0.02)0.91 (0.02)0.074, 0.281, 0.151, 0.189
lateral incisors0.91 (0.02)0.91 (0.02)0.745, 0.430, 0.729, 0.476
a DeLong’s test for the area under the ROC curve comparison. p-value for each examiner.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Pereira, S.A.; Corte-Real, A.; Melo, A.; Magalhães, L.; Lavado, N.; Santos, J.M. Diagnostic Accuracy of Cone Beam Computed Tomography and Periapical Radiography for Detecting Apical Root Resorption in Retention Phase of Orthodontic Patients: A Cross-Sectional Study. J. Clin. Med. 2024, 13, 1248. https://doi.org/10.3390/jcm13051248

AMA Style

Pereira SA, Corte-Real A, Melo A, Magalhães L, Lavado N, Santos JM. Diagnostic Accuracy of Cone Beam Computed Tomography and Periapical Radiography for Detecting Apical Root Resorption in Retention Phase of Orthodontic Patients: A Cross-Sectional Study. Journal of Clinical Medicine. 2024; 13(5):1248. https://doi.org/10.3390/jcm13051248

Chicago/Turabian Style

Pereira, Sónia A., Ana Corte-Real, Ana Melo, Linda Magalhães, Nuno Lavado, and João Miguel Santos. 2024. "Diagnostic Accuracy of Cone Beam Computed Tomography and Periapical Radiography for Detecting Apical Root Resorption in Retention Phase of Orthodontic Patients: A Cross-Sectional Study" Journal of Clinical Medicine 13, no. 5: 1248. https://doi.org/10.3390/jcm13051248

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop