Augmented reality navigation for spinal pedicle screw instrumentation using intraoperative 3D imaging

doi:10.1016/j.spinee.2019.10.012

The Spine Journal

Volume 20, Issue 4, April 2020, Pages 621-628

https://doi.org/10.1016/j.spinee.2019.10.012 Get rights and content

Abstract

BACKGROUND CONTEXT

Due to recent developments in augmented reality with head-mounted devices, holograms of a surgical plan can be displayed directly in the surgeon's field of view. To the best of our knowledge, three dimensional (3D) intraoperative fluoroscopy has not been explored for the use with holographic navigation by head-mounted devices in spine surgery.

PURPOSE

To evaluate the surgical accuracy of holographic pedicle screw navigation by head-mounted device using 3D intraoperative fluoroscopy.

STUDY DESIGN

In this experimental cadaver study, the accuracy of surgical navigation using a head-mounted device was compared with navigation with a state-of-the-art pose-tracking system.

METHODS

Three lumbar cadaver spines were embedded in nontransparent agar gel, leaving only commonly visible anatomy in sight. Intraoperative registration of preoperative planning was achieved by 3D fluoroscopy and fiducial markers attached to lumbar vertebrae. Trackable custom-made drill sleeve guides enabled real-time navigation. In total, 20 K-wires were navigated into lumbar pedicles using AR-navigation, 10 K-wires by the state-of-the-art pose-tracking system. 3D models obtained from postexperimental CT scans were used to measure surgical accuracy. MF is the founder and shareholder of Incremed AG, a Balgrist University Hospital start-up focusing on the development of innovative techniques for surgical executions. The other authors declare no conflict of interest concerning the contents of this study. No external funding was received for this study.

RESULTS

No significant difference in accuracy was measured between AR-navigated drillings and the gold standard with pose-tracking system with mean translational errors between entry points (3D vector distance; p=.85) of 3.4±1.6 mm compared with 3.2±2.0 mm, and mean angular errors between trajectories (3D angle; p=.30) of 4.3°±2.3° compared with 3.5°±1.4°.

CONCLUSIONS

In conclusion, holographic navigation by use of a head-mounted device achieve accuracy comparable to the gold standard of high-end pose-tracking systems.

CLINICAL SIGNIFICANCE

These promising results could result in a new way of surgical navigation with minimal infrastructural requirements but now have to be confirmed in clinical studies.

Introduction

The proximity of vital and neural structures in spine surgery requires very high accuracy from the surgeons [1]. Recently, there has been increasing interest in augmented reality (AR) and its potential to improve accuracy of orthopedic surgeries [2], [3], [4], [5]. To perform surgical navigation by holographic guidance, a three-dimensional (3D) preoperative plan can be uploaded to the AR head-mounted device (HMD) and registered (superimposed) with the intraoperative anatomy [6]. HMDs enable visualization of a 3D hologram directly into the surgeon's field of view. Compared with current state-of-the-art pose-tracking systems (PTS) working with two-dimensional (2D) monitors, HMDs have the advantage of direct overlay of navigation on anatomy and that the surgeon can stay focused on the operating field without having to turn away.

A systematic review of most recent AR advancements reported a surgical accuracy between 1 and 5 mm and attested improvements in safety and efficacy of surgical procedures [2,6,7]. In a comparison of three different HMDs, Qian et al. presented the HoloLens (Microsoft, Redmond, USA) to be most suitable for supporting surgical interventions [8].

Since the position of the patient's anatomy in surgery is different from the position in the preoperatively acquired computed tomography (CT), the anatomy can be registered intraoperatively in order to align it with the hologram of the 3D preoperative plan [9]. Several registration techniques such as manual or landmark-based surface registration, surface digitalization or intraoperative ultrasound have been employed [[5], [6], [7],9,10]. However, in conventional navigation of spine surgeries, imaging-based registration remains the gold standard to perform the registration task precisely, and 3D fluoroscopy has been shown to be a reliable imaging technique in this regard [11]. Surprisingly, to the best of our knowledge, imaging-based registration has not been investigated for holographic navigation and compared with state-of-the-art real-time PTS.

In this cadaver study, AR-enabled holographic navigation of pedicle screw instrumentation by use of an HMD (HoloLens, Microsoft, Redmond, USA) was compared with gold standard navigation with a state-of-the-art PTS (fusionTrack500, Atracsys LLC, Puidoux, Switzerland). Intraoperative registration by 3D fluoroscopy was used in both methods to ensure identical experimental setups and comparable results.

Section snippets

Material and methods

This experimental cadaver study was approved by the Cantonal Ethics Committee on research involving humans (ID: KEK 2017-00874). Consent to use a person's body for research purposes was given by the person itself or his/her representative. After the experiments, all exogenic material was extracted and cadavers were cremated.

3D evaluation method

Postexperimentally, CT scans with 1 mm slice thickness of the three cadaver specimens were conducted in prone position embedded within agar gel. Vertebrae and K-wires were semiautomatically segmented and registered with the 3D planning, as explained before [6].

With the CASPA software, cylinders of 2 mm diameter were aligned with the 3D models of the K-wires and sphere centers (diameter 2 mm), representing the entry points, were placed where the cylinder passed through the posterior surface of

Results

The AR and PTS groups were navigated with 20 K-wires and 10 K-wires, respectively.

The evaluation of the surgical accuracy resulted in a mean 3D TE in the AR and PTS groups of 3.4±1.6 mm and 3.2±2.0 mm, respectively (Fig. 6a). Mean 3D AE was 4.3°±2.3° in the AR group and 3.5°±1.4° in the PTS group (Fig. 6b). No statistically significant difference between the groups was found for TE (p=.85) or AE (p=.30).

In 2D, the axial TE was 1.8±1.3 mm (AR group) versus 1.6±1.0 mm (PTS group) and the sagittal

Discussion

Although being the gold standard in surgical navigation [11], to our knowledge, image-based registration has not been explored for the use with AR-based HMD and compared with state-of-the-art navigation systems in spine surgery. In the here performed cadaveric pilot study, we found surgical accuracy achieved by AR navigation using HMD with image-based registration to be within the range of commercially available navigation systems.

Various different clinical grading systems have been applied to

Conclusions

In conclusion, navigation for pedicle instrumentation by augmented reality-based head-mounted devices achieves comparable results to state-of-the-art high precision navigation systems when using 3D image-based registration.

Acknowledgments

This work is part of “SURGENT” under the auspices of University Medicine Zurich/Hochschulmedizin Zürich. Imaging was performed with the support of the Swiss Center for Musculoskeletal Imaging, SCMI, Balgrist Campus AG, Zurich. We thank Fabio Carrillo and Jonas Walker for their great support throughout the experiments, especially during the design of custom-made guides and markers as well as illustrations.

References (26)

TM Urakov et al.
Workflow caveats in augmented reality–assisted pedicle instrumentation: Cadaver Lab
World Neurosurg
(2019)
JP Du et al.
Accuracy of pedicle screw insertion among 3 image-guided navigation systems: systematic review and meta-analysis
World Neurosurg
(2018)
D Guha et al.
Spinal intraoperative three-dimensional navigation: correlation between clinical and absolute engineering accuracy
Spine J
(2017)
JE Mathew et al.
Pedicle violation and navigational errors in pedicle screw insertion using the intraoperative O-arm: a preliminary report
Int J Spine Surg
(2013)
YR Rampersaud et al.
Accuracy requirements for image-guided spinal pedicle screw placement
Spine (Phila Pa 1976)
(2001)
P Vávra et al.
Recent development of augmented reality in surgery: a review
J Healthc Eng
(2017)
M Fischer et al.
Preclinical usability study of multiple augmented reality concepts for K-wire placement
Int J Comput Assist Radiol Surg
(2016)
S Belmustakov et al.
Augmented and virtual reality navigation for interventions in the musculoskeletal system
Curr Radiol Rep
(2018)
L Ma et al.
Augmented reality surgical navigation with ultrasound-assisted registration for pedicle screw placement: a pilot study
Int J Comput Assist Radiol Surg
(2017)
F Liebmann et al.
Pedicle screw navigation using surface digitization on the Microsoft HoloLens
Int J Comput Assist Radiol Surg
(2019)

JT Gibby et al.

Head-mounted display augmented reality to guide pedicle screw placement utilizing computed tomography

Int J Comput Assist Radiol Surg

(2019)

L Qian et al.

Comparison of optical see-through head-mounted displays for surgical interventions with object-anchored 2D-display

Int J Comput Assist Radiol Surg

(2017)

K-M Scheufler et al.

Accuracy of image-guided pedicle screw placement using intraoperative computed tomography- based navigation with automated referencing. Part II: thoracolumbar spine

Neurosurgery

(2011)

Cited by (75)

Facial augmented reality based on hierarchical optimization of similarity aspect graph
2024, Computer Methods and Programs in Biomedicine
The existing face matching method requires a point cloud to be drawn on the real face for registration, which results in low registration accuracy due to the irregular deformation of the patient's skin that makes the point cloud have many outlier points.
This work proposes a non-contact pose estimation method based on similarity aspect graph hierarchical optimization. The proposed method constructs a distance-weighted and triangular-constrained similarity measure to describe the similarity between views by automatically identifying the 2D and 3D feature points of the face. A mutual similarity clustering method is proposed to construct a hierarchical aspect graph with 3D pose as nodes. A Monte Carlo tree search strategy is used to search the hierarchical aspect graph for determining the optimal pose of the facial 3D model, so as to realize the accurate registration of the facial 3D model and the real face.
The proposed method was used to conduct accuracy verification experiments on the phantoms and volunteers, which were compared with four advanced pose calibration methods. The proposed method obtained average fusion errors of 1.13 ± 0.20 mm and 0.92 ± 0.08 mm in head phantom and volunteer experiments, respectively, which exhibits the best fusion performance among all comparison methods.
Our experiments proved the effectiveness of the proposed pose estimation method in facial augmented reality.
Automatic registration with continuous pose updates for marker-less surgical navigation in spine surgery
2024, Medical Image Analysis
Established surgical navigation systems for pedicle screw placement have been proven to be accurate, but still reveal limitations in registration or surgical guidance. Registration of preoperative data to the intraoperative anatomy remains a time-consuming, error-prone task that includes exposure to harmful radiation. Surgical guidance through conventional displays has well-known drawbacks, as information cannot be presented in-situ and from the surgeon’s perspective. Consequently, radiation-free and more automatic registration methods with subsequent surgeon-centric navigation feedback are desirable. In this work, we present a marker-less approach that automatically solves the registration problem for lumbar spinal fusion surgery in a radiation-free manner. A deep neural network was trained to segment the lumbar spine and simultaneously predict its orientation, yielding an initial pose for preoperative models, which then is refined for each vertebra individually and updated in real-time with GPU acceleration while handling surgeon occlusions. An intuitive surgical guidance is provided thanks to the integration into an augmented reality based navigation system. The registration method was verified on a public dataset with a median of 100% successful registrations, a median target registration error of 2.7 mm, a median screw trajectory error of 1.6°and a median screw entry point error of 2.3 mm. Additionally, the whole pipeline was validated in an ex-vivo surgery, yielding a 100% screw accuracy and a median target registration error of 1.0 mm. Our results meet clinical demands and emphasize the potential of RGB-D data for fully automatic registration approaches in combination with augmented reality guidance.
Glenoid component placement in reverse shoulder arthroplasty assisted with augmented reality through a head-mounted display leads to low deviation between planned and postoperative parameters
2023, Journal of Shoulder and Elbow Surgery
Navigated augmented reality (AR) through a head-mounted display (HMD) may lead to accurate glenoid component placement in reverse shoulder arthroplasty (RSA). The purpose of this study is to evaluate the deviation between planned, intra- and postoperative inclination, retroversion, entry point, depth, and rotation of the glenoid component placement assisted by a navigated AR through HMD during RSA.
Both shoulders of 6 fresh frozen human cadavers, free from fractures or other bony pathologies, were used. Preoperative computed tomography (CT) scans were used for the 3-dimensional (3D) planning. The glenoid component placement was assisted using a navigated AR system through an HMD in all specimens. Intraoperative inclination, retroversion, depth, and rotation were measured by the system. A postoperative CT scan was performed. The pre- and postoperative 3D CT scan reconstructions were superimposed to calculate the deviation between planned and postoperative inclination, retroversion, entry point, depth, and rotation of the glenoid component placement. Additionally, a comparison between intra- and postoperative values was calculated. Outliers were defined as >10° inclination, >10° retroversion, >3 mm entry point.
The registration algorithm of the scapulae prior to the procedure was correctly completed for all cases. The deviations between planned and postoperative values were 1.0° ± 0.7° for inclination, 1.8° ± 1.3° for retroversion, 1.1 ± 0.4 mm for entry point, 0.7 ± 0.6 mm for depth, and 1.7° ± 1.6° for rotation. The deviation between intra- and postoperative values were 0.9° ± 0.8° for inclination, 1.2° ± 1.1° for retroversion, 0.6 ± 0.5 mm for depth, and 0.3° ± 0.2° for rotation. There were no outliers between planned and postoperative parameters.
In this study, the use of a navigated AR system through an HMD for RSA led to low deviation between planned and postoperative values and between intra- and postoperative parameters.
Augmented Reality in Minimally Invasive Spinal Surgery: A Narrative Review of Available Technology
2023, World Neurosurgery
Spine surgery has undergone significant changes in approach and technique. With the adoption of intraoperative navigation, minimally invasive spinal surgery (MISS) has arguably become the gold standard. Augmented reality (AR) has now emerged as a front-runner in anatomical visualization and narrower operative corridors. In effect, AR is poised to revolutionize surgical training and operative outcomes. Our study examines the current literature on AR-assisted MISS, synthesizes findings, and creates a narrative highlighting the history and future of AR in spine surgery.
Relevant literature was gathered using the PubMed (Medline) database from 1975 to 2023. Pedicle screw placement models were the primary intervention in AR. These were compared to the outcomes of traditional MISS
We found that AR devices on the market show promising clinical outcomes in preoperative training and intraoperative use. Three prominent systems were as follows: XVision, HoloLens, and ImmersiveTouch. In the studies, surgeons, residents, and medical students had opportunities to operate AR systems, showcasing their educational potential across each phase of learning. Specifically, one facet described training with cadaver models to gauge accuracy in pedicle screw placement. AR-MISS exceeded free-hand methods without unique complications or contraindications.
While still in its infancy, AR has already proven beneficial for educational training and intraoperative MISS applications. We believe that with continued research and advancement of this technology, AR is poised to become a dominant player within the fundamentals of surgical education and MISS operative technique.
Virtual and Augmented Reality in Spine Surgery: A Systematic Review
2023, World Neurosurgery
Augmented reality (AR) and virtual reality (VR) implementation in spinal surgery has expanded rapidly over the past decade. This systematic review summarizes the use of AR/VR technology in surgical education, preoperative planning, and intraoperative guidance.
A search query for AR/VR technology in spine surgery was conducted through PubMed, Embase, and Scopus. After exclusions, 48 studies were included. Included studies were then grouped into relevant subsections. Categorization into subsections yielded 12 surgical training studies, 5 preoperative planning, 24 intraoperative usage, and 10 radiation exposure.
VR-assisted training significantly reduced penetration rates or increased accuracy rates compared to lecture-based groups in 5 studies. Preoperative VR planning significantly influenced surgical recommendations and reduced radiation exposure, operating time, and estimated blood loss. For 3 patient studies, AR-assisted pedicle screw placement accuracy ranged from 95.77% to 100% using the Gertzbein grading scale. Head-mounted display was the most common interface used intraoperatively followed by AR microscope and projector. AR/VR also had applications in tumor resection, vertebroplasty, bone biopsy, and rod bending. Four studies reported significantly reduced radiation exposure in AR group compared to fluoroscopy group.
AR/VR technologies have the potential to usher in a paradigm shift in spine surgery. However, the current evidence indicates there is still a need for 1) defined quality and technical requirements for AR/VR devices, 2) more intraoperative studies that explore usage outside of pedicle screw placement, and 3) technological advancements to overcome registration errors via the development of an automatic registration method.
The HoloLens in medicine: A systematic review and taxonomy
2023, Medical Image Analysis
The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality (AR) display, is the main player in the recent boost in medical AR research. In this systematic review, we provide a comprehensive overview of the usage of the first-generation HoloLens within the medical domain, from its release in March 2016, until the year of 2021. We identified 217 relevant publications through a systematic search of the PubMed, Scopus, IEEE Xplore and SpringerLink databases. We propose a new taxonomy including use case, technical methodology for registration and tracking, data sources, visualization as well as validation and evaluation, and analyze the retrieved publications accordingly. We find that the bulk of research focuses on supporting physicians during interventions, where the HoloLens is promising for procedures usually performed without image guidance. However, the consensus is that accuracy and reliability are still too low to replace conventional guidance systems. Medical students are the second most common target group, where AR-enhanced medical simulators emerge as a promising technology. While concerns about human–computer interactions, usability and perception are frequently mentioned, hardly any concepts to overcome these issues have been proposed. Instead, registration and tracking lie at the core of most reviewed publications, nevertheless only few of them propose innovative concepts in this direction. Finally, we find that the validation of HoloLens applications suffers from a lack of standardized and rigorous evaluation protocols. We hope that this review can advance medical AR research by identifying gaps in the current literature, to pave the way for novel, innovative directions and translation into the medical routine.

View all citing articles on Scopus

: FDA device/drug status: Not applicable.

: Author disclosures: FM: Nothing to disclose. SR: Nothing to disclose. FL: Nothing to disclose. JMS: Nothing to disclose. PF: Nothing to disclose. MF: Stock Ownership: Incremed AG (amount not disclosed); Consulting: Incremed AG (amount not disclosed); Scientific Advisory Board/Other Office: Incremed AG, Prognosyst AG; Research Support (Investigator Salary, Staff/Materials): Medacta, Depuy Synthes (paid directly to institution); Fellowship Support: Depuy Synthes (paid directly to institution).

View full text

Basic ScienceAugmented reality navigation for spinal pedicle screw instrumentation using intraoperative 3D imaging

Abstract

BACKGROUND CONTEXT

PURPOSE

STUDY DESIGN

METHODS

RESULTS

CONCLUSIONS

CLINICAL SIGNIFICANCE

Introduction

Section snippets

Material and methods

3D evaluation method

Results

Discussion

Conclusions

Acknowledgments

World Neurosurg

World Neurosurg

Spine J

Int J Spine Surg

Accuracy requirements for image-guided spinal pedicle screw placement

Spine (Phila Pa 1976)

Recent development of augmented reality in surgery: a review

J Healthc Eng

Preclinical usability study of multiple augmented reality concepts for K-wire placement

Int J Comput Assist Radiol Surg

Augmented and virtual reality navigation for interventions in the musculoskeletal system

Curr Radiol Rep

Augmented reality surgical navigation with ultrasound-assisted registration for pedicle screw placement: a pilot study

Int J Comput Assist Radiol Surg

Pedicle screw navigation using surface digitization on the Microsoft HoloLens

Int J Comput Assist Radiol Surg

Head-mounted display augmented reality to guide pedicle screw placement utilizing computed tomography

Int J Comput Assist Radiol Surg

Comparison of optical see-through head-mounted displays for surgical interventions with object-anchored 2D-display

Int J Comput Assist Radiol Surg

Accuracy of image-guided pedicle screw placement using intraoperative computed tomography- based navigation with automated referencing. Part II: thoracolumbar spine

Neurosurgery

Basic Science
Augmented reality navigation for spinal pedicle screw instrumentation using intraoperative 3D imaging