Commentary

The objective of the systematic review by Antoszewska-Smith and colleagues1 was to compare the effectiveness of orthodontic mini-implants (OMIs) as anchorage devices with conventional orthodontic anchorage methods in patients in need of space closure of extracted maxillary first premolars without losing molar anchorage. The difference in anchorage loss, ie mesial movement of the maxillary first molars, between these techniques was the primary outcome measure. For this commentary we assessed the quality of the systematic review. We used the AMSTAR and ROBIS tools to score respectively the methodological validity and the risk of bias in the systematic review.2,3,4,5 These instruments were applied independently to this review by the two authors (RMR and LD) of this commentary.

The outcomes of our appraisals with the AMSTAR and ROBIS tools were summarised in Tables 1 and 2 and were assigned in complete agreement between both reviewers.

Table 1 AMSTAR scores for the systematic review by Antoszewska-Smith et al.1
Full size table
Table 2 Tabular presentation for ROBIS results for the systematic review by Antoszewska-Smith et al.1
Full size table

Limitations of the systematic review

Table 3 presents the rationale for the AMSTAR and ROBIS scores and also lists additional limitations of the review. These issues are explained here.

Table 3 Limitations of the systematic review by Antoszweska-Smith et al.1
Full size table

Prior to starting a systematic review investigators should (1) prioritise their research questions with pertinent stakeholders to assess whether their planned questions are necessary; 6 (2) assess whether the review has already been done previously. If so, authors should assess whether a new review is indicated, for example in the context of the limitations of the earlier review. The main text and the references of the review by Antoszewska-Smith and co-authors1 showed that they did not undertake such assessments. For example, they did not consider a recent Cochrane review by Jambi and co-workers7 that asked similar research questions. We consulted the PROSPERO8 register and various online protocol repositories,9,10 but were unable to identify a protocol of the systematic review by Antoszewska-Smith and co-authors.1 Not registering or publishing of protocols can introduce various biases such as selective reporting and publication of outcomes.11 Pilot testing of any of the research methods was also not reported in our appraised review.

Carefully defined eligibility criteria permit the reproducibility of a review and reduce sources of medical uncertainties such as the variability in participants, interventions, comparators, outcomes etc. Numerous eligibility criteria in this review were either not defined or were incomplete. For example: (1) a definition of eligible controlled clinical trials was missing; (2) eligible characteristics for participants such as age, sex, and other demographic items were not given. Restrictions for the type of setting were also not reported; (3) reporting on both the interventions and comparators was suboptimal. For example, the characteristics of eligible interventions with OMIs and conventional anchorage methods were not defined; (4) An eligible duration of treatment was not reported; (5) A clear time point for measuring outcomes was also not defined. It was not reported whether closure of extraction spaces with or without paralleling the roots was considered as the endpoint of treatment; (6) Eligible methods for measuring outcomes, eg model or cephalometric analyses were also not reported; (7) Language bias was a problem, because only articles reported in the English language were eligible

The authors screened a wide spectrum of electronic databases and also manually searched various pertinent journals and the references of included studies. However, they did not consult: (1) the Grey literature; (2) researchers and sponsors to obtain information on unknown or ongoing studies; (3) references of review articles on OMIs. The search strategy was probably sufficient, because it covered the same keywords as other systematic reviews on OMIs.7,12 However, we do not know whether this search strategy was pilot tested and whether an information specialist with expertise in searching the biomedical literature was consulted to validate the search strategy. A list of excluded studies with the rationale for exclusion was also not reported. Not including such a list makes the reproducibility of the review impossible and also introduces study selection bias.13

For the assessment of the quality of the eligible controlled clinical trials studies, the reviewers applied the Newcastle-Ottawa scale. This instrument lacks comprehensive manuals with instructions for users, which has resulted in low reliability between reviewers.14,15,16 To deal with these shortcomings, the Cochrane Bias Methods Group and the Cochrane Non-Randomised Studies Group developed a new tool for assessing risk of bias in non-randomised studies of interventions (ROBINS-I tool).17This instrument, previously known as the ACROBAT-NRSI tool18 should have been used for the bias assessment in the eligible controlled clinical trials. The authors assigned the quality of evidence as moderate for both the included randomised controlled trials and the controlled clinical trials, but this score was based on test methods developed by the authors themselves. Instead, they should have implemented a validated instrument such as the GRADE approach for assessing the quality of evidence.19

It was unclear whether conducting meta-analyses was indicated, because definitions of eligible outcomes were underreported in the eligibility criteria and in the review as a whole. It was therefore unclear whether all studies included in the meta-analyses had measured the same outcomes and with the same test instruments. It was also unclear whether outcomes on failed OMIs were included in the intervention groups. This is important, because excluding these outcomes can seriously upgrade the effectiveness of these devices. The same issue applies to excluding outcomes on poor headgear collaborators from the comparator groups. The authors also included a non-eligible study20 in two of the meta-analyses for a secondary outcome. The inclusion of this study is particularly problematic, because it skewed the meta-analysis on tipping of molars in favour of OMIs.

The authors of the appraised review did not assess any outcome on adverse effects of interventions. When considering the implementation of a new health technology, clinicians want to know both the benefits and the adverse outcomes of the intervention of interest. It is therefore mandatory in Cochrane reviews to assess the findings of at least one adverse effect as a primary outcome.

Competing interests can influence how research studies are designed, conducted and reported, which could divert outcomes away from the truth. In addition, it has been estimated that around 50% of the studies that involve researchers with conflicts of interests do not declare them.21 Full transparency on the role of a sponsor or funder during any part of the review process is therefore key.22 A statement on potential conflicts of interest was not included in the appraised review by Antoszewska-Smith and co-workers.1

Conclusions

Prior to applying the findings of a systematic review on a specific intervention to a patient, clinicians need to exclude numerous uncertainties. The assessment of the quality of the review is an initial step in this process. Quality assessments of systematic reviews are important as was explained in a recent investigation by Ioannidis.23 He suggested that 'possibly the large majority of produced systematic reviews and meta-analyses are unnecessary, misleading, and/or conflicted'.23 These three limitations were also found in the systematic review by Antoszewska-Smith and co-workers.1 First, the review reported no information on potential conflicts of interests. Second, assessments with both the AMSTAR and ROBIS tools identified numerous limitations. These shortcomings could have been resolved either during the protocol phase or during the conduct and reporting of the review. Supplementary files published online should have expanded on many of the underreported items. A good collaboration between editors, peer-reviewers, and the authors of this review could also have significantly raised its quality. Now only a new unbiased review team can address these shortcomings in a new systematic review. Third, before developing and conducting a new review one should first assess whether addressing the current research questions is really necessary, because the effectiveness of non-moving implants for anchorage purposes is obvious and probably does not need further research. Clinicians and patients are possibly more interested in reviews that assess how displacements of OMIs and therefore anchorage loss can be avoided. Prioritising review questions with pertinent stakeholders is the first step when considering a new review.6

Practice point

  • Critical appraisals with the AMSTAR and ROBIS tools conducted by two reviewers independently, identified numerous limitations in this systematic review. These shortcomings should be carefully weighed prior to implementing the findings of this low quality review to our patients

  • Research on the obvious effectiveness of non-moving implants for anchorage purposes is probably not what patients and clinicians want. Prioritising review questions with pertinent stakeholders is indicated before designing and conducting a new review on orthodontic mini-implants and comparator interventions.