Abstract
Haustral folds are colon wall protrusions implicated for high polyp miss rate during optical colonoscopy procedures. If segmented accurately, haustral folds can allow for better estimation of missed surface and can also serve as valuable landmarks for registering pre-treatment virtual (CT) and optical colonoscopies, to guide navigation towards the anomalies found in pre-treatment scans. We present a novel generative adversarial network, FoldIt, for feature-consistent image translation of optical colonoscopy videos to virtual colonoscopy renderings with haustral fold overlays. A new transitive loss is introduced in order to leverage ground truth information between haustral fold annotations and virtual colonoscopy renderings. We demonstrate the effectiveness of our model on real challenging optical colonoscopy videos as well as on textured virtual colonoscopy videos with clinician-verified haustral fold annotations. All code and scripts to reproduce the experiments of this paper will be made available via our Computational Endoscopy Platform at https://github.com/nadeemlab/CEP.
S. Mathew and S. Nadeem—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Supplementary Video: https://youtu.be/_iWBJnDMXjo.
- 2.
Supplementary Video: https://youtu.be/_iWBJnDMXjo.
References
Amodio, M., Krishnaswamy, S.: Travelgan: image-to-image translation by transformation vector learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8983–8992 (2019)
Bae, G., Budvytis, I., Yeung, C.K., Cipolla, R.: Deep multi-view stereo for dense 3D reconstruction from monocular endoscopic video. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 774–783 (2020)
Chen, R.J., Bobrow, T.L., Athey, T., Mahmood, F., Durr, N.J.: Slam endoscopy enhanced by adversarial depth prediction. arXiv preprint arXiv:1907.00283 (2019)
Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.: Stargan: unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8789–8797 (2018)
Fang, H., Deng, W., Zhong, Y., Hu, J.: Triple-GAN: progressive face aging with triple translation loss. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 804–805 (2020)
Freedman, D., et al.: Detecting deficient coverage in colonoscopies. arXiv preprint arXiv:2001.08589 (2020)
Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
İncetan, K., et al.: VR-Caps: a virtual environment for capsule endoscopy.Med. Image Anal. 70, 101990 (2021)
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)
Liu, X., et al.: Reconstructing sinus anatomy from endoscopic video-towards a radiation-free approach for quantitative longitudinal assessment. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 3–13 (2020)
Ma, R., Wang, R., Pizer, S., Rosenman, J., McGill, S.K., Frahm, J.M.: Real-time 3D reconstruction of colonoscopic surfaces for determining missing regions. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 573–582 (2019)
Mahmood, F., Chen, R., Durr, N.J.: Unsupervised reverse domain adaptation for synthetic medical images via adversarial training. IEEE Trans. Med. Imaging 37(12), 2572–2581 (2018)
Mathew, S., Nadeem, S., Kaufman, A.: Visualizing missing surfaces in colonoscopy videos using shared latent space representations. In: IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 329–333 (2021)
Mathew, S., Nadeem, S., Kumari, S., Kaufman, A.: Augmenting colonoscopy using extended and directional cyclegan for lossy image translation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4696–4705 (2020)
Nadeem, S., Kaufman, A.: Computer-aided detection of polyps in optical colonoscopy images. SPIE Med. Imaging 9785, 978525 (2016)
Nadeem, S., Marino, J., Gu, X., Kaufman, A.: Corresponding supine and prone colon visualization using eigenfunction analysis and fold modeling. IEEE Trans. Vis. Comput. Gr. 23(1), 751–760 (2016)
Rau, A., et al.: Implicit domain adaptation with conditional generative adversarial networks for depth prediction in endoscopy. Int. J. Comput. Assist. Radiol. Surg. 14(7), 1167–1176 (2019). https://doi.org/10.1007/s11548-019-01962-w
Xu, J., et al.: Ofgan: realistic rendition of synthetic colonoscopy videos. in: international Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 732–741 (2020)
Zhu, J.Y., Park, T., Isola, P., Efros, A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)
Acknowledgements
This project was supported by MSK Cancer Center Support Grant/Core Grant (P30 CA008748), and NSF grants CNS1650499, OAC1919752, and ICER1940302.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (mp4 90818 KB)
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Mathew, S., Nadeem, S., Kaufman, A. (2021). FoldIt: Haustral Folds Detection and Segmentation in Colonoscopy Videos. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12903. Springer, Cham. https://doi.org/10.1007/978-3-030-87199-4_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-87199-4_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87198-7
Online ISBN: 978-3-030-87199-4
eBook Packages: Computer ScienceComputer Science (R0)
