Skip to main content

Advertisement

SpringerLink
Log in

Nail it! vision-based drift correction for accurate mixed reality surgical guidance

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

A Correction to this article was published on 05 August 2023

This article has been updated

Abstract

Purpose

Mixed reality-guided surgery through head-mounted displays (HMDs) is gaining interest among surgeons. However, precise tracking of HMDs relative to the surgical environment is crucial for successful outcomes. Without fiducial markers, spatial tracking of the HMD suffers from millimeter- to centimeter-scale drift, resulting in misaligned visualization of registered overlays. Methods and workflows capable of automatically correcting for drift after patient registration are essential to assuring accurate execution of surgical plans.

Methods

We present a mixed reality surgical navigation workflow that continuously corrects for drift after patient registration using only image-based methods. We demonstrate its feasibility and capabilities using the Microsoft HoloLens on glenoid pin placement in total shoulder arthroplasty. A phantom study was conducted involving five users with each user placing pins on six glenoids of different deformity, followed by a cadaver study by an attending surgeon.

Results

In both studies, all users were satisfied with the registration overlay before drilling the pin. Postoperative CT scans showed 1.5 mm error in entry point deviation and 2.4\(^\circ \) error in pin orientation on average in the phantom study and 2.5 mm and 1.5\(^\circ \) in the cadaver study. A trained user takes around 90 s to complete the workflow. Our method also outperformed HoloLens native tracking in drift correction.

Conclusion

Our findings suggest that image-based drift correction can provide mixed reality environments precisely aligned with patient anatomy, enabling pin placement with consistently high accuracy. These techniques constitute a next step toward purely image-based mixed reality surgical guidance, without requiring patient markers or external tracking hardware.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Change history

References

  1. Fida B, Cutolo F, di Franco G, Ferrari M, Ferrari V (2018) Augmented reality in open surgery. Updat Surg 70(3):389–400

    Article  Google Scholar 

  2. Pérez-Pachón L, Poyade M, Lowe T, Gröning F (2020) Image overlay surgery based on augmented reality: a systematic review. Biomed Vis 8:175–195

    Article  Google Scholar 

  3. Andress S, Johnson A, Unberath M, Winkler AF, Yu K, Fotouhi J, Weidert S, Osgood GM, Navab N (2018) On-the-fly augmented reality for orthopedic surgery using a multimodal fiducial. J Med Imaging 5(2):021209

    Article  Google Scholar 

  4. Deib G, Johnson A, Unberath M, Yu K, Andress S, Qian L, Osgood G, Navab N, Hui F, Gailloud P (2018) Image guided percutaneous spine procedures using an optical see-through head mounted display: proof of concept and rationale. J Neurointerventional Surg 10(12):1187–1191

    Article  Google Scholar 

  5. Gu W, Shah K, Knopf J, Navab N, Unberath M (2021) Feasibility of image-based augmented reality guidance of total shoulder arthroplasty using microsoft hololens 1. Comput Method Biomech Biomed Eng Imaging Vis 9(3):261–270

    Article  Google Scholar 

  6. Gu W, Shah K, Knopf J, Josewski C, Unberath M (2022) A calibration-free workflow for image-based mixed reality navigation of total shoulder arthroplasty. Comput Method Biomech Biomed Eng Imaging Vis 10(3):243–251

    Article  Google Scholar 

  7. Liebmann F, Roner S, von Atzigen M, Scaramuzza D, Sutter R, Snedeker J, Farshad M, Fürnstahl P (2019) Pedicle screw navigation using surface digitization on the microsoft hololens. IJCARS 14(7):1157–1165

    Google Scholar 

  8. Unberath M, Fotouhi J, Hajek J, Maier A, Osgood G, Taylor R, Armand M, Navab N (2018) Augmented reality-based feedback for technician-in-the-loop c-arm repositioning. Healthcare Technol Lett 5(5):143–147

    Article  Google Scholar 

  9. Fotouhi J, Unberath M, Song T, Gu W, Johnson A, Osgood G, Armand M, Navab N (2019) Interactive flying frustums (iffs): spatially aware surgical data visualization. IJCARS 14(6):913–922

    Google Scholar 

  10. Creighton FX, Unberath M, Song T, Zhao Z, Armand M, Carey J (2020) Early feasibility studies of augmented reality navigation for lateral skull base surgery. Otol Neurotol 41(7):883–888

    Article  PubMed  Google Scholar 

  11. Hübner P, Clintworth K, Liu Q, Weinmann M, Wursthorn S (2020) Evaluation of hololens tracking and depth sensing for indoor mapping applications. Sensors 20(4):1021

    Article  PubMed  PubMed Central  Google Scholar 

  12. Vassallo R, Rankin A, Chen EC, Peters TM (2017) Hologram stability evaluation for microsoft hololens. In: Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment, vol. 10136, pp. 295–300. SPIE

  13. Frantz T, Jansen B, Duerinck J, Vandemeulebroucke J (2018) Augmenting microsoft’s hololens with vuforia tracking for neuronavigation. Healthcare Technol Lett 5(5):221–225

    Article  Google Scholar 

  14. de Oliveira ME, Debarba HG, Lädermann A, Chagué S, Charbonnier C (2019) A hand-eye calibration method for augmented reality applied to computer-assisted orthopedic surgery. Int J Med Robot Comput Assist Surg 15(2):1969

  15. Zhou Z, Yang Z, Jiang S, Zhang F, Yan H (2019) Design and validation of a surgical navigation system for brachytherapy based on mixed reality. Med Phys 46(8):3709–3718

    Article  PubMed  Google Scholar 

  16. Zhou Z, Yang Z, Jiang S, Zhuo J, Zhu T, Ma S (2022) Augmented reality surgical navigation system based on the spatial drift compensation method for glioma resection surgery. Med Phys 49(6):3963–3979

    Article  PubMed  Google Scholar 

  17. Martin-Gomez A, Li H, Song T, Yang S, Wang G, Ding H, Navab N, Zhao Z, Armand M (2022) Sttar: Surgical tool tracking using off-the-shelf augmented reality head-mounted displays. arXiv preprint arXiv:2208.08880

  18. Arun KS, Huang TS, Blostein SD (1987) Least-squares fitting of two 3-d point sets. IEEE PAMI 5:698–700

    Article  Google Scholar 

  19. Schönberger JL, Frahm J-M (2016) Structure-from-motion revisited. In: CVPR

  20. Lindenberger P, Sarlin P-E, Larsson V, Pollefeys M (2021) Pixel-Perfect Structure-from-Motion with Featuremetric Refinement. In: ICCV

  21. DeTone D, Malisiewicz T, Rabinovich A (2018) Superpoint: Self-supervised interest point detection and description. In: CVPR, pp. 224–236

  22. Cernea D (2020) OpenMVS: Multi-View Stereo Reconstruction . https://cdcseacave.github.io/openMVS

  23. Granger S, Pennec X, Roche A (2001) Rigid point-surface registration using an em variant of icp for computer guided oral implantology. In: MICCAI, pp. 752–761. Springer

  24. Iannotti J, Baker J, Rodriguez E, Brems J, Ricchetti E, Mesiha M, Bryan J (2014) Three-dimensional preoperative planning software and a novel information transfer technology improve glenoid component positioning. JBJS 96(9):71

  25. Liu H, Auvinet E, Giles J, Rodriguez y, Baena F (2018) Augmented reality based navigation for computer assisted hip resurfacing: a proof of concept study. Ann Biomed Eng 46:1595–1605

    Article  CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

This work was funded in part by a sponsored research agreement between Arthrex Inc. and the Johns Hopkins University. We would like to thank Stephen Herrington, Michael Moreland, Nick Metcalfe and Leonardo Guibert for assistance with study setup.

Author information

Authors and Affiliations

  1. Johns Hopkins University, Baltimore, MD, USA

    Wenhao Gu, John Cast & Mathias Unberath

  2. Arthrex Inc., 1 Arthrex Way, Naples, FL, USA

    Jonathan Knopf, Laurence D. Higgins & David Knopf

Authors
  1. Wenhao Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jonathan Knopf
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John Cast
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Laurence D. Higgins
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. David Knopf
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mathias Unberath
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wenhao Gu.

Ethics declarations

Conflict of interest

The authors have no conflict of interest to declare.

Ethical approval

Studies are approved and performed in accordance with ethical standards.

Informed consent

Informed consents were obtained from all participants in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original online version of this article was revised: The third subtitle under Method section should be “3D reconstruction and registration” rather than “3D reconstruction and”.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (mp4 43688 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gu, W., Knopf, J., Cast, J. et al. Nail it! vision-based drift correction for accurate mixed reality surgical guidance. Int J CARS 18, 1235–1243 (2023). https://doi.org/10.1007/s11548-023-02950-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-023-02950-x

Keywords

Search

Navigation