Artificial intelligence−enhanced white-light colonoscopy with attention guidance predicts colorectal cancer invasion depth

doi:10.1016/j.gie.2021.03.936

Gastrointestinal Endoscopy

Volume 94, Issue 3, September 2021, Pages 627-638.e1

https://doi.org/10.1016/j.gie.2021.03.936 Get rights and content

Background and Aims

Endoscopic submucosal dissection (ESD) and EMR are applied in treating superficial colorectal neoplasms but are contraindicated by deeply invasive colorectal cancer (CRC). The invasion depth of neoplasms can be examined by an automated artificial intelligence (AI) system to determine the applicability of ESD and EMR.

Methods

A deep convolutional neural network with a tumor localization branch to guide invasion depth classification was constructed on the GoogLeNet architecture. The model was trained using 7734 nonmagnified white-light colonoscopy (WLC) images supplemented by image augmentation from 657 lesions labeled with histopathologic analysis of invasion depth. An independent testing dataset consisting of 1634 WLC images from 156 lesions was used to validate the model.

Results

For predicting noninvasive and superficially invasive neoplasms, the model achieved an overall accuracy of 91.1% (95% confidence interval [CI], 89.6%-92.4%), with 91.2% sensitivity (95% CI, 88.8%-93.3%) and 91.0% specificity (95% CI, 89.0%-92.7%) at an optimal cutoff of .41 and the area under the receiver operating characteristic (AUROC) curve of .970 (95% CI, .962-.978). Inclusion of the advanced CRC data significantly increased the sensitivity in differentiating superficial neoplasms from deeply invasive early CRC to 65.3% (95% CI, 61.9%-68.8%) with an AUROC curve of .729 (95% CI, .699-.759), similar to experienced endoscopists (.691; 95% CI, .624-.758).

Conclusions

We have developed an AI-enhanced attention-guided WLC system that differentiates noninvasive or superficially submucosal invasive neoplasms from deeply invasive CRC with high accuracy, sensitivity, and specificity.

Graphical abstract

Section snippets

Data preparation and sample distribution

WLC images of nonpolypoid advanced colorectal adenoma and CRC from the Department of Gastroenterology of Nanfang Hospital, Southern Medical University, China, were retrospectively collected. The study protocol was approved by the Nanfang Hospital Institutional Review Board (NFEC-2019-160). There was no restriction on the instruments used. However, all images used in this study are nonmagnified. Ethics approval was provided by Nanfang Hospital, Southern Medical University, China.

Lesions were

Histopathologic characteristics of the patient cohort

A summary of patient clinical profiles in the training and testing datasets are shown in Table 1. Of 657 lesions included in the training dataset, with patient ages ranging from 20 to 90 years, 424 (64.5%) were group 1 cases, 42 (6.4%) group 2 CRC, and 191 (29.1%) group 3 cases. Four hundred twenty four superficial lesions (64.5%) were defined as P0 and 233 (35.5%) as P1. To assess the classification performance of the AI model, we used an independent testing dataset consisting of 1634

Discussion

Endoscopic discrimination of noninvasive or superficially invasive tumors from deeply invasive CRC is essential to determine the optimal treatment strategy for patients. Unfortunately, the accurate identification of lesion features associated with deep submucosal invasion is challenging in clinical practice even for experienced endoscopists. In recent years, an increasing body of work has demonstrated the potential of computer-aided systems for improving clinicians’ diagnosis and work

Acknowledgment

Data can be acquired upon request to the authors.

References (38)

Y. Saito et al.
A prospective, multicenter study of 1111 colorectal endoscopic submucosal dissections (with video)
Gastrointest Endosc
(2010)
Y. Tateishi et al.
Pathological prognostic factors predicting lymph node metastasis in submucosal invasive (T1) colorectal carcinoma
Mod Pathol
(2010)
M. Tokunaga et al.
Computer-aided diagnosis system using only white-light endoscopy for the prediction of invasion depth in colorectal cancer
Gastrointest Endosc
(2021)
M. Mukae et al.
Diagnostic performance of EUS for evaluating the invasion depth of early colorectal cancers
Gastrointest Endosc
(2015)
N. Hayashi et al.
Endoscopic prediction of deep submucosal invasive carcinoma: validation of the narrow-band imaging international colorectal endoscopic (NICE) classification
Gastrointest Endosc
(2013)
N. Kobayashi et al.
Determining the treatment strategy for colorectal neoplastic lesions: endoscopic assessment or the non-lifting sign for diagnosing invasion depth?
Endoscopy
(2007)
S. Tanaka et al.
JGES guidelines for colorectal endoscopic submucosal dissection/endoscopic mucosal resection
Dig Endosc
(2015)
P. Pimentel-Nunes et al.
Endoscopic submucosal dissection: European Society of Gastrointestinal Endoscopy (ESGE) guideline
Endoscopy
(2015)
K. Niimi et al.
Long-term outcomes of endoscopic submucosal dissection for colorectal epithelial neoplasms
Endoscopy
(2010)
S.R. Puli et al.
Meta-analysis and systematic review of colorectal endoscopic mucosal resection
World J Gastroenterol
(2009)

N. Yoshida et al.

Importance of histological evaluation in endoscopic resection of early colorectal cancer

World J Gastrointest Pathophysiol

(2012)

Y. Hashiguchi et al.

Japanese Society for Cancer of the Colon and Rectum (JSCCR) guidelines 2019 for the treatment of colorectal cancer

Int J Clin Oncol

(2020)

C. Hassan et al.

A resect and discard strategy would improve cost-effectiveness of colorectal cancer screening

Clin Gastroenterol Hepatol

(2010)

M. Li et al.

Kudo's pit pattern classification for colorectal neoplasms: a meta-analysis

World J Gastroenterol

(2014)

H. Ikehara et al.

Diagnosis of depth of invasion for early colorectal cancer using magnifying colonoscopy

J Gastroenterol Hepatol

(2010)

S. Tanaka et al.

Endoscopic assessment of colorectal cancer with superficial or deep submucosal invasion using magnifying colonoscopy

Clin Endosc

(2013)

N. Kakushima

Endoscopic diagnosis for the depth of early colorectal cancer

J Gastroenterol Hepatol

(2010)

T. Matsuda et al.

Efficacy of the invasive/non-invasive pattern by magnifying chromoendoscopy to estimate the depth of invasion of early colorectal neoplasms

Am J Gastroenterol

(2008)

Y. Sano et al.

Japan NBI Expert Team classification: Narrow-band imaging magnifying endoscopic classification of colorectal tumors

Dig Endosc

(2018)

Cited by (26)

Application of the convolution neural network in determining the depth of invasion of gastrointestinal cancer: a systematic review and meta-analysis
2024, Journal of Gastrointestinal Surgery
With the development of endoscopic technology, endoscopic submucosal dissection (ESD) has been widely used in the treatment of gastrointestinal tumors. It is necessary to evaluate the depth of tumor invasion before the application of ESD. The convolution neural network (CNN) is a type of artificial intelligence that has the potential to assist in the classification of the depth of invasion in endoscopic images. This meta-analysis aimed to evaluate the performance of CNN in determining the depth of invasion of gastrointestinal tumors.
A search on PubMed, Web of Science, and SinoMed was performed to collect the original publications about the use of CNN in determining the depth of invasion of gastrointestinal neoplasms. Pooled sensitivity and specificity were calculated using an exact binominal rendition of the bivariate mixed-effects regression model. I² was used for the evaluation of heterogeneity.
A total of 17 articles were included; the pooled sensitivity was 84% (95% CI, 0.81-0.88), specificity was 91% (95% CI, 0.85-0.94), and the area under the curve (AUC) was 0.93 (95% CI, 0.90-0.95). The performance of CNN was significantly better than that of endoscopists (AUC: 0.93 vs 0.83, respectively; P = .0005).
Our review revealed that CNN is one of the most effective methods of endoscopy to evaluate the depth of invasion of early gastrointestinal tumors, which has the potential to work as a remarkable tool for clinical endoscopists to make decisions on whether the lesion is feasible for endoscopic treatment.
The role of artificial intelligence in colonoscopy
2024, Seminars in Colon and Rectal Surgery
Recent advancements in artificial intelligence (AI) have had a profound impact on the field of gastrointestinal endoscopy, particularly in the realm of colonoscopy. Recent studies have shown excellent performance of AI models in both computer-aided detection (CADe) and computer-aided diagnosis (CADx) of polyps. Moreover, AI has shown promising results in other aspects of colonoscopy such as prediction of depth of tumor invasion, colonoscopy quality assurance, and AI applications in inflammatory bowel disease.
In this review, we summarize the recent literature on the application of AI in colonoscopy, and review the limitations and challenges of existing AI technologies and future directions for this field.
Role of the artificial intelligence in the management of T1 colorectal cancer
2024, Digestive and Liver Disease
Approximately 10% of submucosal invasive (T1) colorectal cancers demonstrate extraintestinal lymph node metastasis, necessitating surgical intervention with lymph node dissection. The ability to identify T1b (submucosal invasion depth ≥ 1000 µm) as a risk factor for lymph node metastasis via pre-treatment endoscopy is crucial in guiding treatment strategies. Accurately distinguishing T1b from T1a (submucosal invasion depth < 1000 µm) or dysplasia remains a significant challenge for artificial intelligence (AI) systems, which require high and consistent diagnostic capabilities. Moreover, as endoscopic therapies like endoscopic full-thickness resection and endoscopic intermuscular dissection evolve, and the focus on reducing unnecessary surgeries intensifies, the initial management of T1 colorectal cancers via endoscopic treatment is anticipated to increase. Consequently, the development of highly accurate and reliable AI systems is essential, not only for pre-treatment depth assessment but also for post-treatment risk stratification of lymph node metastasis. While such AI diagnostic systems are still under development, significant advancements are expected in the near future to improve decision-making in T1 colorectal cancer management.
pT1 colorectal cancer: A treatment dilemma
2023, Best Practice and Research: Clinical Gastroenterology
The implementation of population screening programs for colorectal cancer (CRC) has led to a considerable increase in the prevalence pT1-CRC originating on polyps amenable by local treatments. However, a high proportion of patients are referred for unnecessary oncological surgeries without a clear benefit in terms of survival. Selecting the appropriate endoscopic resection technique in the moment of diagnosis becomes crucial to provide the best treatment alternative to each individual polyp and patient. For this, it is imperative to increase the optical diagnostic skill for differentiating pT1-CRCs and decide the appropriate initial therapy. En bloc resection is crucial to obtain an adequate histological specimen that might allow organ preserving therapeutic management. In this review, we address key challenges in T1 CRC management, explore the efficacy and safety of the available diagnostic and therapeutic approaches, and shed light on upcoming advances in the field.
Computer-aided diagnosis of early-stage colorectal cancer using nonmagnified endoscopic white-light images (with videos)
2023, Gastrointestinal Endoscopy
Differentiation of colorectal cancers (CRCs) with deep submucosal invasion (T1b) from CRCs with superficial invasion (T1a) or no invasion (Tis) is not straightforward. This study aimed to develop a computer-aided diagnosis (CADx) system to establish the diagnosis of early-stage cancers using nonmagnified endoscopic white-light images alone.
From 5108 images, 1513 lesions (Tis, 1074; T1a, 145; T1b, 294) were collected from 1470 patients at 10 academic hospitals and assigned to training and testing datasets (3:1). The ResNet-50 network was used as the backbone to extract features from images. Oversampling and focal loss were used to compensate class imbalance of the invasive stage. Diagnostic performance was assessed using the testing dataset including 403 CRCs with 1392 images. Two experts and 2 trainees read the identical testing dataset.
At a 90% cutoff for the per-lesion score, CADx showed the highest specificity of 94.4% (95% confidence interval [CI], 91.3-96.6), with 59.8% (95% CI, 48.3-70.4) sensitivity and 87.3% (95% CI, 83.7-90.4) accuracy. The area under the characteristic curve was 85.1% (95% CI, 79.9-90.4) for CADx, 88.2% (95% CI, 83.7-92.8) for expert 1, 85.9% (95% CI, 80.9-90.9) for expert 2, 77.0% (95% CI, 71.5-82.4) for trainee 1 (vs CADx; P = .0076), and 66.2% (95% CI, 60.6-71.9) for trainee 2 (P < .0001). The function was also confirmed on 9 short videos.
A CADx system developed with endoscopic white-light images showed excellent per-lesion specificity and accuracy for T1b lesion diagnosis, equivalent to experts and superior to trainees. (Clinical trial registration number: UMIN000037053.)
Role of Artificial Intelligence in Colonoscopy: A Literature Review of the Past, Present, and Future Directions
2023, Techniques and Innovations in Gastrointestinal Endoscopy
Colonoscopy remains one of the most common procedures performed by gastroenterologists and is critical for early detection and management of precursors to colorectal cancer (CRC). Although CRC remains one of the deadliest malignancies, earlier detection of precancerous polyps is directly associated with increased patient survival. As such, quality metrics for colonoscopy, such as polyp detection and mucosal visualization, are key parameters that are directly tied to patient outcomes. Over the past 2 decades, artificial intelligence and machine learning (AI/ML) tools have been tested and developed to augment colonoscopy performance and in 2021 resulted in the first-ever FDA-approved computer-aided detection (CADe) tool. This narrative review begins by reviewing the evidence behind the use of CADe that led to FDA approval. Next, the review discusses the current evidence and technological approaches for computer-aided diagnosis for optical in situ histopathological differentiation of colorectal polyps, including narrow-band imaging, blue light imaging, and endocytoscopy. Studies are ongoing to develop systems to predict the depth of submucosal invasion and to assess endoscopic disease activity among patients with inflammatory bowel disease. The applications of AI/ML to quality improvement are explored, including real-time assessment of bowel preparation, detection of cecal intubation, and automated polyp reporting and surveillance recommendations using natural language processing. Despite initial cost concerns, models have suggested that CADe systems could result in long-term cost savings and are generally accepted by patients and gastroenterologists. There is some reservation in adopting computer-aided diagnosis systems among gastroenterologists due to medico-legal concerns. Future directions for AI/ML in colonoscopy include health system improvements, such as automating note writing, optimizing procedural scheduling, and predicting sedation needs.

View all citing articles on Scopus

: DISCLOSURE: All authors disclosed no financial relationships. Research support for this study was provided in part by Guangdong Provincial Science and Technology Research Program (2020A1414010265,) funding to Xiaobei Luo and Guangdong Provincial Science and Technology Research Program (2019A141405016, and 2017B020209003) funding to Side Liu. This work was also supported in part by the Institute of Bioengineering and Nanotechnology, Biomedical Research Council, Agency for Science, Technology and Research (A∗STAR; Project Number IAF-PPH18/01/a0/014, IAF-PP H18/01/a0/K14, MedCaP-LOA-18-02); MOE ARC (MOE2017-T2-1-149); IAF (H18/01/a0/017); SMART CAMP; The Institute for Digital Medicine (WisDM); and Mechanobiology Institute of Singapore (R-714-106-004-135) funding to Hanry Yu.

: If you would like to chat with an author of this article, you may contact Dr Side at [email protected] and Dr Hanry at [email protected].

^∗: Drs Luo, Wang and Han contributed equally to this article.

View full text

Original articleClinical endoscopyArtificial intelligence−enhanced white-light colonoscopy with attention guidance predicts colorectal cancer invasion depth

Background and Aims

Methods

Results

Conclusions

Graphical abstract

Section snippets

Data preparation and sample distribution

Histopathologic characteristics of the patient cohort

Discussion

Acknowledgment

Gastrointest Endosc

Mod Pathol

Gastrointest Endosc

Gastrointest Endosc

Gastrointest Endosc

Determining the treatment strategy for colorectal neoplastic lesions: endoscopic assessment or the non-lifting sign for diagnosing invasion depth?

Endoscopy

JGES guidelines for colorectal endoscopic submucosal dissection/endoscopic mucosal resection

Dig Endosc

Endoscopic submucosal dissection: European Society of Gastrointestinal Endoscopy (ESGE) guideline

Endoscopy

Long-term outcomes of endoscopic submucosal dissection for colorectal epithelial neoplasms

Endoscopy

Meta-analysis and systematic review of colorectal endoscopic mucosal resection

World J Gastroenterol

Importance of histological evaluation in endoscopic resection of early colorectal cancer

World J Gastrointest Pathophysiol

Japanese Society for Cancer of the Colon and Rectum (JSCCR) guidelines 2019 for the treatment of colorectal cancer

Int J Clin Oncol

A resect and discard strategy would improve cost-effectiveness of colorectal cancer screening

Clin Gastroenterol Hepatol

Kudo's pit pattern classification for colorectal neoplasms: a meta-analysis

World J Gastroenterol

Diagnosis of depth of invasion for early colorectal cancer using magnifying colonoscopy

J Gastroenterol Hepatol

Endoscopic assessment of colorectal cancer with superficial or deep submucosal invasion using magnifying colonoscopy

Clin Endosc

Endoscopic diagnosis for the depth of early colorectal cancer

J Gastroenterol Hepatol

Efficacy of the invasive/non-invasive pattern by magnifying chromoendoscopy to estimate the depth of invasion of early colorectal neoplasms

Am J Gastroenterol

Japan NBI Expert Team classification: Narrow-band imaging magnifying endoscopic classification of colorectal tumors

Dig Endosc

Original article
Clinical endoscopy
Artificial intelligence−enhanced white-light colonoscopy with attention guidance predicts colorectal cancer invasion depth