Sophie Maria
Helena M. Mentis
Ignacio Avellino
ABSTRACT
Although Augmented Reality (AR) has been touted as the future of surgery, its contribution to distributed collaboration such as in surgical teleconsulting has not been articulated. We propose AR-Head Mounted Displays (AR-HMD) to tackle two previously-identified challenges: operating surgeons needing to view and interact with imaging systems that reside away from the operative field, and, their lack of gesturing tools to point and annotate on the shared images and physical environment. We report on a controlled lab experiment where 12 expert gynecology surgeons perform a tumor localisation task guided by a remote radiologist (confederate) via an AR-HMD. We find that bringing the shared images to the place of work reduces the need for clarifications and provides opportunistic access to information when required, and, that pointing and annotating provides opportunities to further support verbal instruction in deictic communication. Our results inform the design of intraoperative AR-HMD systems for surgical telecollaboration.
Footnotes
Supplemental Material
- Rahul Agarwal, Adam W. Levinson, Mohamad Allaf, Danil Makarov, Alex Nason, and Li-Ming Su. 2007. The RoboConsultant: telementoring and remote presence in the operating room during minimally invasive urologic surgeries using a novel mobile robotic interface. Urology 70, 5 (Nov. 2007), 970–974. https://doi.org/10.1016/j.urology.2007.09.053Google Scholar
Cross Ref
- Daniel Andersen, Voicu Popescu, Maria Eugenia Cabrera, Aditya Shanghavi, Gerardo Gomez, Sherri Marley, Brian Mullis, and Juan P. Wachs. 2016. Medical telementoring using an augmented reality transparent display. Surgery 159, 6 (June 2016), 1646–1653. https://doi.org/10.1016/j.surg.2015.12.016Google ScholarCross Ref
- Steven Arild Wuyts Andersen, Peter Trier Mikkelsen, Lars Konge, Per Cayé-Thomasen, and Mads Sølvsten Sørensen. 2016. Cognitive load in distributed and massed practice in virtual reality mastoidectomy simulation. The Laryngoscope 126, 2 (2016), E74–E79.Google ScholarCross Ref
- R Anthony Jr 2008. Cognitive load theory and the role of learner experience: An abbreviated review for educational practitioners. AACE Review (formerly AACE Journal) 16, 4 (2008), 425–439.Google Scholar
- Ignacio Avellino, Gilles Bailly, Geoffroy Canlorbe, Jérémie Belgihti, Guillaume Morel, and Marie-Aude Vitrani. 2019. Impacts of Telemanipulation in Robotic Assisted Surgery. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems(CHI ’19). ACM, New York, NY, USA, 583:1–583:15. https://doi.org/10.1145/3290605.3300813 event-place: Glasgow, Scotland, UK.Google ScholarDigital Library
- Ignacio Avellino, Sheida Nozari, Geoffroy Canlorbe, and Yvonne Jansen. 2021. Surgical Video Summarization: Multifarious Uses, Summarization Process and Ad-Hoc Coordination. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (April 2021), 140:1–140:23. https://doi.org/10.1145/3449214Google ScholarDigital Library
- H. Borgmann, M. Rodríguez Socarrás, J. Salem, I. Tsaur, J. Gomez Rivas, E. Barret, and L. Tortolero. 2017. Feasibility and safety of augmented reality-assisted urological surgery using smartglass. World Journal of Urology 35, 6 (June 2017), 967–972. https://doi.org/10.1007/s00345-016-1956-6Google ScholarCross Ref
- Caroline GL Cao, Mi Zhou, Daniel B Jones, and Steven D Schwaitzberg. 2007. Can surgeons think and operate with haptics at the same time?Journal of Gastrointestinal Surgery 11, 11 (2007), 1564–1569.Google Scholar
- Herbert H. Clark. 1996. Using Language. Cambridge University Press, Cambridge. https://doi.org/10.1017/CBO9780511620539Google ScholarCross Ref
- Herbert H. Clark and Susan E. Brennan. 1991. Grounding in communication. In Perspectives on socially shared cognition, L. B. Resnick, J. M. Levine, and S. D. Teasley (Eds.). American Psychological Association, Washington, DC, US, 127–149.Google Scholar
- Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20, 1 (April 1960), 37–46. https://doi.org/10.1177/001316446002000104 Publisher: SAGE Publications.Google ScholarCross Ref
- Gregorio Convertino, Helena M Mentis, Aleksandra Slavkovic, Mary Beth Rosson, and John M Carroll. 2011. Supporting common ground and awareness in emergency management planning: A design research project. ACM Transactions on Computer-Human Interaction (TOCHI) 18, 4(2011), 1–34.Google ScholarDigital Library
- Matthew Christopher Davis, Dang D. Can, Jonathan Pindrik, Brandon G. Rocque, and James M. Johnston. 2016. Virtual Interactive Presence in Global Surgical Education: International Collaboration Through Augmented Reality. World Neurosurgery 86 (Feb. 2016), 103–111. https://doi.org/10.1016/j.wneu.2015.08.053Google ScholarCross Ref
- Barrett Ens, Joel Lanir, Anthony Tang, Scott Bateman, Gun Lee, Thammathip Piumsomboon, and Mark Billinghurst. 2019. Revisiting collaboration through mixed reality: The evolution of groupware. International Journal of Human-Computer Studies 131 (Nov. 2019), 81–98. https://doi.org/10.1016/j.ijhcs.2019.05.011Google ScholarDigital Library
- Yuanyuan Feng, Katie Li, Azin Semsar, Hannah McGowan, Jacqueline Mun, H. Reza Zahiri, Ivan George, Adrian Park, Andrea Kleinsmith, and Helena M. Mentis. 2019. Communication Cost of Single-user Gesturing Tool in Laparoscopic Surgical Training. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems(CHI ’19). ACM, New York, NY, USA, 611:1–611:12. https://doi.org/10.1145/3290605.3300841 event-place: Glasgow, Scotland Uk.Google ScholarDigital Library
- Yuanyuan Feng, Hannah McGowan, Azin Semsar, Hamid R. Zahiri, Ivan M. George, Adrian Park, Andrea Kleinsmith, and Helena Mentis. 2019. Virtual pointer for gaze guidance in laparoscopic surgery. Surgical Endoscopy 34 (Oct. 2019), 3533–3539. https://doi.org/10.1007/s00464-019-07141-xGoogle ScholarCross Ref
- Yuanyuan Feng and Helena M. Mentis. 2018. Improving Common Ground Development in Surgical Training through Talk and Action. AMIA Annual Symposium Proceedings 2017 (April 2018), 696–705. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5977723/Google Scholar
- Susan R. Fussell, Leslie D. Setlock, Jie Yang, Jiazhi Ou, Elizabeth Mauer, and Adam D. I. Kramer. 2004. Gestures over Video Streams to Support Remote Collaboration on Physical Tasks. Hum.-Comput. Interact. 19, 3 (Sept. 2004), 273–309. https://doi.org/10.1207/s15327051hci1903_3Google ScholarDigital Library
- Danilo Gasques, Janet G. Johnson, Tommy Sharkey, Yuanyuan Feng, Ru Wang, Zhuoqun Robin Xu, Enrique Zavala, Yifei Zhang, Wanze Xie, Xinming Zhang, Konrad Davis, Michael Yip, and Nadir Weibel. 2021. ARTEMIS: A Collaborative Mixed-Reality System for Immersive Surgical Telementoring. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–14. https://doi.org/10.1145/3411764.3445576Google ScholarDigital Library
- Steffen Gauglitz, Benjamin Nuernberger, Matthew Turk, and Tobias Höllerer. 2014. World-stabilized annotations and virtual scene navigation for remote collaboration. In Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST ’14. ACM Press, Honolulu, Hawaii, USA, 449–459. https://doi.org/10.1145/2642918.2647372Google ScholarDigital Library
- Kori M. Inkpen, Wai-ling Ho-Ching, Oliver Kuederle, Stacey D. Scott, and Garth B. D. Shoemaker. 1999. This is fun! we’re all best friends and we’re all playing: supporting children’s synchronous collaboration. In Proceedings of the 1999 conference on Computer support for collaborative learning(CSCL ’99). International Society of the Learning Sciences, Palo Alto, California, 31–es.Google Scholar
- Ellen A. Isaacs and Herbert H. Clark. 1987. References in conversation between experts and novices.Journal of Experimental Psychology: General 116, 1 (March 1987), 26–37. https://doi.org/10.1037/0096-3445.116.1.26Google ScholarCross Ref
- Rose Johnson, Kenton O’Hara, Abigail Sellen, Claire Cousins, and Antonio Criminisi. 2011. Exploring the potential for touchless interaction in image-guided interventional radiology. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Vancouver BC Canada, 3323–3332. https://doi.org/10.1145/1978942.1979436Google ScholarDigital Library
- Seungwon Kim, Gun Lee, Mark Billinghurst, and Weidong Huang. 2020. The combination of visual communication cues in mixed reality remote collaboration. Journal on Multimodal User Interfaces 14, 4 (Dec. 2020), 321–335. https://doi.org/10.1007/s12193-020-00335-xGoogle ScholarCross Ref
- Seungwon Kim, Gun A. Lee, Nobuchika Sakata, Andreas Dunser, Elina Vartiainen, and Mark Billinghurst. 2013. Study of augmented gesture communication cues and view sharing in remote collaboration. In 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, Adelaide, Australia, 261–262. https://doi.org/10.1109/ISMAR.2013.6671795Google ScholarCross Ref
- David Kirk, Tom Rodden, and Danaë Stanton Fraser. 2007. Turn it this way: grounding collaborative action with remote gestures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’07). Association for Computing Machinery, New York, NY, USA, 1039–1048. https://doi.org/10.1145/1240624.1240782Google ScholarDigital Library
- Naveen Madapana, Daniela Chanci, Glebys Gonzalez, Lingsong Zhang, and Juan P. Wachs. 2022. Touchless Interfaces in the Operating Room: A Study in Gesture Preferences. International Journal of Human–Computer Interaction 0, 0 (April 2022), 1–11. https://doi.org/10.1080/10447318.2022.2041896Google ScholarCross Ref
- Sophie Maria, Solène Lambert, and Ignacio Avellino. 2022. From Déjà vu to Déjà vécu: Reliving Surgery in Post-Operative Debriefing. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE Computer Society, Los Alamitos, CA, USA, 462–465. https://doi.org/10.1109/VRW55335.2022.00102Google ScholarCross Ref
- Helena M. Mentis. 2017. Collocated Use of Imaging Systems in Coordinated Surgical Practice. Proc. ACM Hum.-Comput. Interact. 1, CSCW (Dec. 2017), 78:1–78:17. https://doi.org/10.1145/3134713Google ScholarDigital Library
- Helena M. Mentis, Ignacio Avellino, and Jwawon Seo. 2022. AR HMD for Remote Instruction in Healthcare. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE Computer Society, Los Alamitos, CA, USA, 437–440. https://doi.org/10.1109/VRW55335.2022.00096Google ScholarCross Ref
- Helena M. Mentis, Yuanyuan Feng, Azin Semsar, and Todd A. Ponsky. 2020. Remotely Shaping the View in Surgical Telementoring. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–14. https://doi.org/10.1145/3313831.3376622Google ScholarDigital Library
- Helena M. Mentis, Kenton O’Hara, Gerardo Gonzalez, Abigail Sellen, Robert Corish, Antonio Criminisi, Rikin Trivedi, and Pierre Theodore. 2015. Voice or Gesture in the Operating Room. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM, Seoul Republic of Korea, 773–780. https://doi.org/10.1145/2702613.2702963Google ScholarDigital Library
- Helena M. Mentis, Kenton O’Hara, Abigail Sellen, and Rikin Trivedi. 2012. Interaction Proxemics and Image Use in Neurosurgery. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’12). ACM, New York, NY, USA, 927–936. https://doi.org/10.1145/2207676.2208536Google ScholarDigital Library
- Helena M. Mentis and Alex S. Taylor. 2013. Imaging the body: embodied vision in minimally invasive surgery. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Paris France, 1479–1488. https://doi.org/10.1145/2470654.2466197Google ScholarDigital Library
- Edgar Rojas-Muñoz, Dan Andersen, Maria Eugenia Cabrera, Voicu Popescu, Sherri Marley, Ben Zarzaur, Brian Mullis, and Juan P. Wachs. 2019. Augmented Reality as a Medium for Improved Telementoring. Military Medicine 184, Suppl 1 (March 2019), 57–64. https://doi.org/10.1093/milmed/usy300Google ScholarCross Ref
- Jeremy Roschelle and Stephanie D. Teasley. 1995. The Construction of Shared Knowledge in Collaborative Problem Solving. In Computer Supported Collaborative Learning, Claire O’Malley (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 69–97. https://doi.org/10.1007/978-3-642-85098-1_5Google ScholarCross Ref
- Abigail J. Sellen. 1995. Remote Conversations: The Effects of Mediating Talk with Technology. Hum.-Comput. Interact. 10, 4 (Dec. 1995), 401–444. https://doi.org/10.1207/s15327051hci1004_2Google ScholarDigital Library
- Azin Semsar, Hannah McGowan, Yuanyuan Feng, Hamid R. Zahiri, Ivan M. George, Timothy Turner, Adrian Park, Helena M. Mentis, and Andrea Kleinsmith. 2019. Effects of a Virtual Pointer on Trainees’ Cognitive Load and Communication Efficiency in Surgical Training. AMIA... Annual Symposium proceedings. AMIA Symposium 2019 (2019), 1197–1206.Google Scholar
- Azin Semsar, Hannah McGowan, Yuanyuan Feng, H. Reza Zahiri, Adrian Park, Andrea Kleinsmith, and Helena Mentis. 2019. How Trainees Use the Information from Telepointers in Remote Instruction. Proc. ACM Hum.-Comput. Interact. 3, CSCW (Nov. 2019), 93:1–93:20. https://doi.org/10.1145/3359195Google ScholarDigital Library
- Azin Semsar, Hannah McGowan, Yuanyuan Feng, H. Reza Zahiri, Adrian Park, Andrea Kleinsmith, and Helena M. Mentis. 2020. Quality of and Attention to Instructions in Telementoring. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (Oct. 2020), 165:1–165:21. https://doi.org/10.1145/3415236Google ScholarDigital Library
- A Semsar, J Ton, N Maharoof, I Avellino, HR Zahiri, F Guckes, and H Mentis. 2021. Effect of Training the Mentor on Quality of Instruction and Trainees’ Performance in Laparoscopic Oophorectomy Telementoring. Journal of Minimally Invasive Gynecology 28, 11, Supplement (Nov. 2021), S60–S61. https://doi.org/10.1016/j.jmig.2021.09.452Google ScholarCross Ref
- Jwawon Seo, Ignacio Avellino, Damaruka Priya Rajasagi, Anita Komlodi, and Helena M. Mentis. 2022. HoloMentor: Enabling Remote Instruction through Augmented Reality Mobile Views. Proceedings of the ACM on Human-Computer Interaction 7, GROUP (Dec. 2022), 11:1–11:29. https://doi.org/10.1145/3567561Google ScholarDigital Library
- Mahesh B. Shenai, Marcus Dillavou, Corey Shum, Douglas Ross, Richard S. Tubbs, Alan Shih, and Barton L. Guthrie. 2011. Virtual interactive presence and augmented reality (VIPAR) for remote surgical assistance. Neurosurgery 68, 1 Suppl Operative (March 2011), 200–207; discussion 207. https://doi.org/10.1227/NEU.0b013e3182077efdGoogle ScholarCross Ref
- Angelina M. Vera, Michael Russo, Adnan Mohsin, and Shawn Tsuda. 2014. Augmented reality telementoring (ART) platform: a randomized controlled trial to assess the efficacy of a new surgical education technology. Surgical Endoscopy 28, 12 (Dec. 2014), 3467–3472. https://doi.org/10.1007/s00464-014-3625-4Google ScholarCross Ref
Index Terms
- Supporting Collaborative Discussions In Surgical Teleconsulting Through Augmented Reality Head Mounted Displays
Recommendations
Collaborative Augmented Reality: Multi-user Interaction in Urban Simulation
IVIC '09: Proceedings of the 1st International Visual Informatics Conference on Visual Informatics: Bridging Research and PracticeAugmented reality (AR) environment allows user or multi-user to interact with 2D and 3D data. AR simply can provide a collaborative interactive AR environment for urban simulation, where users can interact naturally and intuitively. AR collaboration ...
Development of a surgical navigation system based on augmented reality using an optical see-through head-mounted display
Display Omitted An Augmented Reality (AR)-based surgical navigation system was developed.The real- and virtual-world views are fused in the HMD during the surgery.Some disadvantages of the traditional surgical navigation will be overcome.The accuracy ...
An augmented reality system for surgical pre-operative planning
BEBI'08: Proceedings of the 1st WSEAS international conference on Biomedical electronics and biomedical informaticsAugmented Reality technology can be used in minimally invasive surgery to increase the physician's view by showing information gathered from patient medical images. The developed AR system can improve the performances of a surgeon involved in a ...
Comments