Original article
Clinical endoscopy
Comparing the number and relevance of false activations between 2 artificial intelligence computer-aided detection systems: the NOISE study

https://doi.org/10.1016/j.gie.2021.12.031Get rights and content

Background and Aims

Artificial intelligence has been shown to be effective in polyp detection, and multiple computer-aided detection (CADe) systems have been developed. False-positive (FP) activation emerged as a possible way to benchmark CADe performance in clinical practice. The aim of this study was to validate a previously developed classification of FPs comparing the performances of different brands of approved CADe systems.

Methods

We compared 2 different consecutive video libraries (40 video per arm) collected at Humanitas Research Hospital with 2 different CADe system brands (CADe A and CADe B). For each video, the number of CADe false activations, cause, and time spent by the endoscopist to examine the area erroneously highlighted were reported. The FP activations were classified according to the previously developed classification of FPs (the NOISE classification) according to their cause and relevance.

Results

In CADe A 1021 FP activations were registered across the 40 videos (25.5 ± 12.2 FPs per colonoscopy), whereas in CADe B 1028 were identified (25.7 ± 13.2 FPs per colonoscopy; P = .53). Among them, 22.9 ± 9.9 (89.8% in CADe A) and 22.1 ± 10.0 (86.0% in CADe B) were because of artifacts from the bowel wall. Conversely, 2.6 ± 1.9 (10.2% in CADe A) and 3.5 ± 2.1 (14% in CADe B) were caused by bowel content (P = .45). Within CADe A each false activation required .2 ± .9 seconds, with 1.6 ± 1.0 FPs (6.3%) requiring additional time for endoscopic assessment. Comparable results were reported within CADe B with .2 ± .8 seconds spent per false activation and 1.8 ± 1.2 FPs per colonoscopy requiring additional inspection.

Conclusions

The use of a standardized nomenclature provided comparable results with either of the 2 recently approved CADe systems. (Clinical trial registration number: NCT04399590.)

Section snippets

Methods

The study was approved by the local Institutional Review Board and registered on the ClinicalTrial.gov (NCT04399590). This was a no-profit study, and no funding was received or solicited, except the loan of the equipment by Fujifilm and Medtronic (CAD-EYE by Fuji and GI-Genius by Medtronic). All authors had access to the study data and reviewed and approved the final manuscript.

Results

Patient and procedural characteristics were comparable between the 2 groups (Table 2). At least 1 polyp was detected in 32 and 33 colonoscopies (P = .77) in the CADe A (CAD-EYE) and CADe B (GI Genius) groups, respectively. Overall, 80 polyps were detected in CADe A and 76 polyps were detected in CADe B with mean withdrawal times of 11.2 ± 4.2 minutes and 11.4 ± 3.9 minutes, respectively (P = .84).

A total of 1021 FP activations were registered across the 40 videos of CADe A, corresponding to a

Discussion

When comparing 2 different CADe brands, no difference in the rate of FP activations was observed, supporting the clinical equivalence of different CADe systems applied to screening and diagnostic colonoscopy. In addition, we showed a very narrow range of FP activations between 19 and 41 that may be adopted as a benchmark for other CADe systems in the future.

The clinical relevance of the equivalent performances of the 2 different CADe brands is primarily related to differences in terms of

References (28)

  • J.R. Su et al.

    Impact of a real-time automatic quality control system on colorectal polyp and adenoma detection: a prospective randomized controlled study (with videos)

    Gastrointest. Endosc

    (2020)
  • W.N. Liu et al.

    Study on detection rate of polyps and adenomas in artificial-intelligence-aided colonoscopy

    Saudi J Gastroenterol

    (2020)
  • A. Repici et al.

    Efficacy of real-time computer-aided detection of colorectal neoplasia in a randomized trial

    Gastroenterology

    (2020)
  • A. Repici et al.

    Artificial intelligence and colonoscopy experience: lessons from two randomised trials

    Gut

    (2022)
  • Cited by (0)

    DISCLOSURE: The following authors disclosed financial relationships: C. Hassan, A. Repici: Consultant for Medtronic and Fuji. R. Maselli: Consultant for Fuji. S. Carrara, A. Anderloni: Consultant for Olympus Corp. Y. Mori: Consultant for Olympus Corp; ownership in Cybernet Corp. M. B. Wallace: Research funding from Medtronic; Research grants from Fujifilm, Boston Scientific, Olympus, Medtronic, Ninepoint Medical, and Cosmo/Aries Pharmaceuticals; Stock options in Virgo Inc; Minor food/beverage at meetings from Synergy Pharmaceuticals, Boston Scientific, and Cook Medical; Consultant on behalf of Mayo Clinic, paid to Mayo Clinic, for GI Supply (2018), Endokey, Endostart, Boston Scientific, Microtek, and Verily. All other authors disclosed no financial relationships.

    View full text