Skip to main content
SpringerLink
Log in

Real-time analysis of cataract surgery videos using statistical models

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The automatic analysis of the surgical process, from videos recorded during surgeries, could be very useful to surgeons, both for training and for acquiring new techniques. The training process could be optimized by automatically providing some targeted recommendations or warnings, similar to the expert surgeon’s guidance. In this paper, we propose to reuse videos recorded and stored during cataract surgeries to perform the analysis. The proposed system allows to automatically recognize, in real time, what the surgeon is doing: what surgical phase or, more precisely, what surgical step he or she is performing. This recognition relies on the inference of a multilevel statistical model which uses 1) the conditional relations between levels of description (steps and phases) and 2) the temporal relations among steps and among phases. The model accepts two types of inputs: 1) the presence of surgical tools, manually provided by the surgeons, or 2) motion in videos, automatically analyzed through the Content Based Video retrieval (CBVR) paradigm. Different data-driven statistical models are evaluated in this paper. For this project, a dataset of 30 cataract surgery videos was collected at Brest University hospital. The system was evaluated in terms of area under the ROC curve. Promising results were obtained using either the presence of surgical tools (A z = 0.983) or motion analysis (A z = 0.759). The generality of the method allows to adapt it to other kinds of surgeries. The proposed solution could be used in a computer assisted surgery tool to support surgeons during the surgery.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. http://opencv.org/

  2. https://wapiti.limsi.fr/

  3. Limited-memory Broyden-Fletcher-Goldfarb-Shanno.

  4. http://dlib.net/

References

  1. André B, Vercauteren T, Buchner AM, Wallace MB, Ayache N (2012) Learning semantic and visual similarity for endomicroscopy video retrieval. IEEE Trans Med Imaging 31(6):1276–1288

    Article  Google Scholar 

  2. Cao Y, Li M, Baang S, Hu S et al (2008) Medical video event classification using shared features. In: Tenth IEEE international symposium on multimedia, 2008. ISM 2008. IEEE, pp 266–273

  3. Charrière K, Quellec G, Lamard M, Coatrieux G, Cochener B, Cazuguel G (2014) Automated surgical step recognition in norMalized cataract surgery videos. In: 36th annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 4647–4650

  4. Charrière K, Quellec G, Lamard M, Martiano D, Cazuguel G, Coatrieux G, Cochener B (2016) Real-time multilevel sequencing of cataract surgery videos. In: 2016 14th international workshop on content-based multimedia indexing (CBMI). IEEE, pp 1–6

  5. Chattopadhyay T, Chaki A, Bhowmick B, Pal A (2008) An application for retrieval of frames from a laparoscopic surgical video based on image of query instrument. tencon 2008. In: IEEE region 10 conference, vol 11, pp 1–5

  6. Fine S, Singer Y, Tishby N (1998) The hierarchical hidden markov model: analysis and applications. Mach Learn 32(1):41–62

    Article  MATH  Google Scholar 

  7. Forestier G, Riffaud L, Jannin P (2015) Automatic phase prediction from low-level surgical activities. Int J Comput Assist Radiol Surg 1–9

  8. Forney Jr GD (1973) The viterbi algorithm. Proc IEEE 61(3):268–278

    Article  MathSciNet  Google Scholar 

  9. Lafferty J, McCallum A, Pereira F (2001) Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of the eighteenth international conference on machine learning, ICML, vol 1, pp 282–289

  10. Lalys F, Riffaud L, Bouget D, Jannin P (2012) A framework for the recognition of high-level surgical tasks from video images for cataract surgeries. IEEE Trans Biomed Eng 59(4):966–976

    Article  Google Scholar 

  11. Lalys F, Bouget D, Riffaud L, Jannin P (2013) Automatic knowledge-based recognition of low-level tasks in ophthalmological procedures. Int J Comput Assist Radiol Surg 8(1):39–49

    Article  Google Scholar 

  12. Laptev I (2005) On space-time interest points. Int J Comput Vis 64(2-3):107–123

    Article  Google Scholar 

  13. Loukas C, Nikiteas N, Schizas D, Georgiou E (2016) Shot boundary detection in endoscopic surgery videos using a variational bayesian framework. Int J CARS 1–13

  14. Lucas BD, Kanade T et al (1981) An iterative image registration technique with an application to stereo vision. In: IJCAI, vol 81, pp 674–679

  15. Padoy N, Blum T, Ahmadi SA, Feussner H, Berger MO, Navab N (2012) Statistical modeling and recognition of surgical workflow. Med Image Anal 16(3):632–641

    Article  Google Scholar 

  16. Pearl J, Russell S (1998) Bayesian networks. Computer Science Department, University of California

  17. Quellec G, Charrière K, Lamard M, Cochener B, Cazuguel G (2014) NorMalizing videos of anterior eye segment surgeries, In: Engineering in medicine and biology society (EMBC), 2014 36th annual international conference of the IEEE. IEEE, pp 122–125

  18. Quellec G, Charrière K, Lamard M, Droueche Z, Roux C, Cochener B, Cazuguel G (2014) Real-time recognition of surgical tasks in eye surgery videos. Med Image Anal 18(3):579–590

    Article  Google Scholar 

  19. Quellec G, Lamard M, Cochener B, Cazuguel G (2014) Real-time segmentation and recognition of surgical tasks in cataract surgery videos. IEEE Trans Med Imaging 33(12)

  20. Quellec G, Lamard M, Cochener B, Cazuguel G (2015) Real-time task recognition in cataract surgery videos using adaptive spatiotemporal polynomials. IEEE Trans Med Imaging 34(4):877–887

    Article  Google Scholar 

  21. Roberts CM (2006) Radio frequency identification (rfid). Comput Secur 25 (1):18–26

    Article  Google Scholar 

  22. Sha F, Pereira F (2003) Shallow parsing with conditional random fields. In: Proceedings of the 2003 conference of the north american chapter of the association for computational linguistics on human language technology, vol 1, pp 134–141. Association for Computational Linguistics

  23. Singh A, Strauss GH (2014) High-fidelity cataract surgery simulation and third world blindness. Surgical Innovation

  24. Stanek SR, Tavanapong W, Wong J, Oh JH, De Groen PC (2012) Automatic real-time detection of endoscopic procedures using temporal features. Comput Methods Prog Biomed 108(2):524–535

    Article  Google Scholar 

  25. Tao L, Zappella L, Hager GD, Vidal R (2013) Surgical gesture segmentation and recognition. In: Medical image computing and computer-assisted intervention–MICCAI 2013. Springer, pp 339–346

  26. Twinanda AP, Shehata S, Mutter D, Marescaux J, de Mathelin M, Padoy N (2016)

  27. Wang H, Ullah MM, Klaser A, Laptev I, Schmid C (2009) Evaluation of local spatio-temporal features for action recognition. In: BMVC 2009 - British machine vision conference. London, Royaume-Uni. http://hal.inria.fr/inria-00439769. CLASS

  28. Yao W, Chu CH, Li Z (2010) The use of rfid in healthcare: benefits and barriers. In: IEEE international conference on RFID-technology and applications (RFID-TA), 2010, pp 128–134

  29. Zappella L, Béjar B, Hager G, Vidal R (2013) Surgical gesture classification from video and kinematic data. Med Image Anal 17(7):732–745

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the Urban Community of Brest (Brest Métropole Océane) and the “Institut Mines-Télecom” for funding this project.

Author information

Authors and Affiliations

  1. Institut Mines-Telecom; Telecom Bretagne; UEB; Dpt ITI, Brest, 29200, France

    Katia Charrière, Guy Cazuguel & Gouenou Coatrieux

  2. LaTIM - INSERM UMR 1101, Brest, 29200, France

    Katia Charrière, Gwénolé Quellec, Mathieu Lamard, David Martiano, Guy Cazuguel, Gouenou Coatrieux & Béatrice Cochener

  3. University Bretagne Occidentale, Brest, 29200, France

    Mathieu Lamard & Béatrice Cochener

  4. CHRU Brest, Service d’Ophtalmologie, Brest, 29200, France

    David Martiano & Béatrice Cochener

Authors
  1. Katia Charrière
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gwénolé Quellec
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mathieu Lamard
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David Martiano
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Guy Cazuguel
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Gouenou Coatrieux
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Béatrice Cochener
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Katia Charrière.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Charrière, K., Quellec, G., Lamard, M. et al. Real-time analysis of cataract surgery videos using statistical models. Multimed Tools Appl 76, 22473–22491 (2017). https://doi.org/10.1007/s11042-017-4793-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-4793-8

Keywords

Search

Navigation