Introduction

In oro-maxillofacial surgeries, the complexity of the involved anatomical structures often hampers direct visualization of the surgical site. Therefore, it is important to correctly grasp steric positional relations, such as the osteotomy line or the mandibular canal, while performing surgical procedures. Although most surgeons have precise knowledge of the underlying tissues, a better understanding of the three-dimensional (3D) relationships between these internal structures would greatly facilitate surgical procedures. Efforts in this direction have resulted in the advent of various techniques and instruments that enable direct visualization and manipulation of the surgical site, such as a conventional navigation system.1,2 However, a disadvantage of such systems is that the images obtained are two-dimensional, and therefore, necessitate constant comparisons between the surgical field and the displayed image, and consequently, frequent hand-eye transformations.3 The two-dimensional information gathered is thus indirectly recognized and understood, resulting in a potentially large gap between the actual patient morphology and its interpretation through the imaging system. With 3D models too, smaller surface details might be smoothened away and information may be lost. Additionally, fixed planes of whatever orientation are no longer considered state-of-the-art 3D image analysis. Therefore, it is imperative that full access to the volume dataset itself is provided to the person and planes are interactively reconstructed per optimal visualization needs.4

Innovative imaging technology in the form of augmented reality (AR) technique, wherein the virtual and real scenes are merged into one environment, has been recently introduced to help oral surgeons visualize sites that cannot be directly observed. AR systems can provide navigated support whereby interpreted information from preoperative data can be superimposed on the surgical site in order to enhance the perception of the physical environment. Therefore, alternate viewing of the displayed image and the actual surgical area is no longer required, as both fields can be simultaneously observed. Furthermore, in situ visualization, which is possible with these systems, is advantageous as it is intuitive and improves the speed of execution of procedures.5

This study examines the possible application of the novel AR technique in the field of oral and maxillofacial surgery. The algorithms and conditions required for 3D virtual representation of the surgical area have been defined by previous research, in which integral videographic (IV) images of only the mandibular component were included.3 However, this is the first study in which dentomaxillofacial 3D computed tomography (CT) data (both maxillary and mandibular jaws along with teeth (crown and root apices)) generated by a 3D IV image display system were superimposed on a human volunteer. In addition, the study evaluated if real-time in situ AR allowed accurate display of the stereoscopic images of the maxilla, mandible and teeth, even when viewed from different angles. Furthermore, a solid model was used to test the accuracy of this system by simulating tasks such as bone resection.

Materials and methods

Configuration of the AR system

The configuration of the AR system is shown in Figure 1. The IV image display, which has been developed at The University of Tokyo and previously described,6,7,8 consists of a high-density rear liquid crystal display and microarray glass. By creating a lens array fixed to the display surface, non-distorted 3D images can be displayed. The specifications of the IV image display system are as follows: a Pentium Core 2 Duo, 2.4-GHz processor; a NVIDIA Quadro Plex model 4 (Quadro FX 5600×2) graphics processing unit (GPU); a 6.5-inch, 1 024×768 pixel display (pixel pitch=0.125 mm); and a fly’s eye lens array. The pixels on the display were computed such that the light rays generated by the pixels and the microarray lenses could accurately reconstruct 3D objects in real space, allowing the operator to observe the 3D IV image with the naked eye. The IV image was generated as shown in previous studies.6,7,8

Figure 1
figure 1

Augmented reality system configuration, which consists of an IV display device, 3D data source collection equipment (computed tomography), 3D optical tracking system, and a computer. A half-silvered mirror is attached to the IV display. Through the half mirror, surgeons can see the IV image without the need for special glasses. (a) Tracker attached to the teeth; (b) tracker attached to the surgical instrument. 3D, three-dimensional; IV, integral videographic.

Full size image

Generation of 3D IV images from preoperative CT data

Briefly, the IV image of the 3D CT was constructed as an assembly of reconstructed light sources (Figure 2). No special glasses were required for viewing the IV image display, and motion parallax could be reproduced in all directions. The system was able to render IV images at a rate of 5 frames per second.

Figure 2
figure 2

The principle of IV (generating and reproducing a 3D object). Each point, shown in a 3D space, is reconstructed at the same position as the actual object by the convergence of rays from the pixels of the element images on the computer display after they pass through the lenses in a microconvex lens array. The surgeon can see any point on the display from various directions, as though it were fixed in 3D space. Each point appears as a different light source. 3D, three-dimensional; IV, integral videographic.

Full size image

Feasibility study on a volunteer

This feasibility study of the AR system, performed on a volunteer, was conducted in accordance with the Good Clinical Practice guidelines and the Declaration of Helsinki, and the study protocol was approved by the medical ethics committee of the Graduate School of Medicine, The University of Tokyo. Informed consent was obtained from the volunteer before study initiation. X-ray CT diagnostic equipment (Aquilion ONE; Toshiba, Tokyo, Japan) was used to acquire CT scans (slice thickness: 0.5 mm) of the orofacial region of the volunteer. Based on the obtained CT data, 3D surface models of the upper and lower jaw bones of the volunteer were constituted using the medical image processing software (Mimics; Materialise, Leuven, Belgium). In addition to these models, the function of medical image processing open source software (3D Slicer; http://www.slicer.org) was extended, and the GPU was set to an algorithm to produce IV images and perform real-time superimposition of the 3D image on the upper and lower jaw bones. The time required for preparing the 3D models within Mimics and/or Slicer was 5–10 min.

Registration and patient tracking

A 3D coordinate system is a prerequisite for image-guided surgeries.1 To integrate the coordinate systems between the subject and IV images, the Polaris Spectra optical tracking system (Northern Digital Inc., Waterloo, Ontario, Canada) was used. This system measured the position and direction of the subject’s movements in real time. The tracking marker was noninvasively attached to the subject’s tooth by using a dental splint (Figure 1a). The coordinate system of the IV images was obtained by measuring the position of the characteristics of these images in space. Similarly, the coordinates of the subject and those of the image were correlated by measuring the coordinate system of patient characteristics. By attaching an infrared reflective ball to the teeth and by measuring the 3D position of the infrared ball, the subject’s movements could be followed; thus, superimposition of stereoscopic images on the appropriate physical area was enabled in real time.

Instrument calibration

The tracking marker frames were designed to meet the following conditions:

  1. 1

    preserving the geometric uniqueness of the markers;

  2. 2

    ensuring that the surgical operations were not disturbed.

This system required calibration using point registration for the instruments. In order to measure the tracking marker position from the optical tracking system, the calibration was performed by careful pivot motion.

Since the tracking marker frames can be rigidly mounted onto the body of the surgical instruments, the calibration needs to be performed less frequently (e.g., once every few months), which can be predetermined.

Real-time surgical navigation using AR

Using the same AR assembly, superimposed 3D images of the surgical instrument (SUCCESS-40MV; OSADA, Tokyo, Japan) in real time were displayed. The accuracy of the system when used in a clinical case was evaluated by performing the drilling task on a solid model imitating a bone surface. The solid model was a plastic block of size 90 mm×40 mm×10 mm. To integrate the coordinate systems between the object and IV images, the Polaris Spectra optical tracking system was used. However, in this case, the tracking marker was attached to the surgical instrument (Figure 1b). The coordinate system of the IV images was obtained by measuring the position of the characteristics of these images in space. While performing the feasibility study in the volunteer, by attaching an infrared reflective ball to the teeth and by measuring the 3D position of the infrared ball, the surgical instrument’s movements could be followed. Three infrared balls were mounted to the splint to track patient movements. In order to register the position of the solid model, instead of using an external marker frame, a landmark-based transformation was performed on the model. A surgical path, defined by a point of insertion and a direction, was pre-determined and its 3D IV image was overlaid onto the surgical scene. This surgical path works as the navigation tool that can inform surgeons of the direction in which drilling should be initiated. Additionally, to help the surgeon perform precise movements, two parameters were added to the navigation screen. The first parameter was the Euclidean distance from the tip of the Lindemann drill (Stryker Corporation, Kalamazoo, MI, USA) to the surgical path (in mm), and the second was the angle between the tool’s main axis and the surgical path. When the instrument was correctly aligned to the surgical path, the color of the instrument automatically changed to alert the surgeon that it was safe to proceed. The surgeon could then initiate the task. The simulation ended when the drill reached the target point on the other side of the model. By measuring the position of the drill holes on both sides, it was possible to compute the positional and angular errors during the simulated surgery. Another similar surgical navigation trial was performed using a surgical instrument that included a reciprocating saw (Stryker Corporation, Kalamazoo, MI, USA). The videos documenting all procedures (on the volunteer as well as those on the solid model) were filmed using an EOS Kiss X4 digital camera (Olympus Optical Co., Tokyo, Japan).

Results

Positional registration was attained by identifying and tracking the subject’s position, and the IV images of the maxillary and mandibular bones and teeth could be superimposed in the correct position. In this system, the surgeon could view the internal structures in the 3D format, the data for which were initially obtained from the preoperative CT data and superimposed onto the actual subject’s anatomy through a half-silvered mirror. Supplementary Movies 1–3 show the surgical sites, as viewed by the surgeon. The movies demonstrate that by using the AR technique, stereoscopy is possible from every position and that the stereoscopic images are accurately displayed across different viewing angles. Supplementary Movie 3 demonstrates that this novel technique can also appropriately detect the subject’s movements. Since the IV images were updated at the speed of 5 frames per second, the stereoscopic images can be considered to be updated in real time. Furthermore, it was observed that change in the viewing angle, laterally or vertically, did not negatively affect the surgeon’s ability to observe the 3D images and visualize the subject, and allowed the surgeon to observe images as if the target areas were being viewed in real space, without special glasses.

Positional registration was also attained by identifying and tracking the position of the surgical instrument, and its IV images were superimposed appropriately. Supplementary Movies 4 and 5 show IV images of surgical navigation using the Lindemann drill and reciprocating saw, respectively. Accuracy of the system is indicated by the positional error (in mm) and angular error (°) displayed at the bottom of the screen. The positional error, using the Lindemann drill, ranged from 0.45–1.34 mm (average 0.77; standard deviations 0.19), whereas the angular error was 2°. Similarly, in the trial using the reciprocating saw, the positional error ranged from 0.02–1.26 mm (average 0.68; standard deviations 0.26), and the angular error was 2°–4° (average 3.13; standard deviations 0.65), while the saw was cutting through the solid model. Since these values are negligible, this AR system can be considered as highly accurate. Additionally, the direction of movement of the surgical instruments was also correctly depicted in this study.

Discussion

The use of image guidance in oral and maxillofacial surgery has offered better patient outcomes.2 The present study evaluated the application of a 3D imaging AR display system, incorporating IV, in oral and maxillofacial surgery. The IV technique displays 3D images, without requiring the user to wear special glasses, irrespective of the viewing position. Thus, it enables observation of stereoscopic images as if the objects were being viewed in real space. Although conventional image-guided surgery provides a 3D view of the patient’s anatomy, it is difficult to accurately correlate the displayed image to the surgical field.2 The AR display system transparently displays the 3D image directly on the surgical site, and the GPU-based rendering algorithm enables real-time tracking of patient’s movements, making it possible to update the 3D images in real time during an operation. The prime advantage of AR is that the surgeon does not need to alternate his/her attention between the surgical site and monitor, as is required in other systems, thus allowing for better focus, improved hand-eye coordination, and faster execution of surgical procedures.

Liao et al.6 showed that the real-time IV algorithm for calculating 3D images of surgical instruments is as effective as that for both surgical instruments and organs in representing the real-time location during a surgery. The present study demonstrated the real time use of AR technology in surgical navigation as well as to generate IV images. Since the 3D image is superimposed in real space, it is possible to immediately recognize a discrepancy between the image and the actual patient appearance. Results from a previous study show that these positional errors were in the range of 2–3 mm,6 but a recent study showed that use of the AR technique, as described in this paper, could decrease the mean positional errors to 0.7 mm.3 The positional error and angular error calculated in this study were 0.77 mm and 0.68°, respectively, which is almost negligible. Thus, the technique demonstrated in this paper can be considered as highly accurate. This could be attributed to the fact that the Liao et al.’s6 study used a computer with specifications that were as follows: CPU, 800-MHz dual Pentium III processor with 1 GB memory; Windows 2000 Operating System; and old IV display (lens pitch, 2.32 mm; pixel pitch, 0.125 mm; and display size, 1 024×768 pixels). The system was then upgraded to Pentium Core 2 Duo and 2.4 GHz; GPU, NVIDIA Quadro Plex 1000 Model 4 with 4 GB memory; Linux Fedora Operating System; and new IV display (lens pitch, 1.0 mm×0.8 mm; pixel pitch, 0.125 mm; and display size, 1 024×768 pixels) in the study by Tran et al.3 The same upgraded system was used in the present study. Modern navigation systems reference patient anatomy and obtain data sets, mainly from skin surface scanning by using laser or by using anatomic landmarks or fiducial markers. A previous study used splints attached to the lower jaw, with titanium screws mounted on them to serve as fiducial markers for the registration process,9 whereas another study used two splints to fix the mandible to the viscerocranium—one each in the open and closed position.10 Although the mobile nature of the mandible is known to generally pose problems for its intraoperative synchronization with preacquired imaging data,1 the IV images obtained with the AR technique in the present study were an exact reproduction of the patient’s internal structures. Additionally, with the use of the novel AR technology, the position of the reflected IV image does not change even if the observer moves the eyes (i.e., changes the viewing angle). Furthermore, the IV image provides depth perception and allows multiple viewers to have different perspectives while observing from different directions.

Results from previous studies using the AR technique for intraoperative navigation have shown its effectiveness, accuracy and feasibility in interstitial brachytherapy11 and maxillofacial surgeries.12 However, these studies employed projection of the image in real space, using a two-dimensional display projector and a 3D display by binocular stereopsis, which were set ups different from the one used in the present study. Other studies used a completely different approach to AR—some used a stereo video see-through head-mounted display as an individual navigation device,5,13 whereas others used the binocular stereopsis technology, whereby the observed video does not change in accordance with the change in viewing position, and thus, allows the surgeon to only determine relative depth.5,11,12,13 Without motion parallax, accurate 3D positioning cannot be reproduced. The IV technique, on the other hand, is based on a different principle than binocular stereopsis, and allows for both binocular parallax, a major component of depth perception, and motion parallax, where depth cues result from the motion of the observer. The new IV technology described in this paper allows the surgeon to determine exact 3D positional relationships by using both binocular parallax and motion parallax without the need for special viewing apparatus; this is not possible with conventional binocular stereopsis technology.

The present study evaluated the application of a 3D image AR display system in the field of oral and maxillofacial surgery. The results showed that a surgical site that cannot be observed directly could be visualized with correct steric positioning; thus, this AR system can be extremely beneficial while performing surgeries and can be a huge asset in the fields of dentistry and oral and craniomaxillofacial surgery. This system can be used to track 3D movements and 3D positions of blood vessels, nerves, tumors, cysts and impacted teeth intraoperatively. In orthodontics, this system can be used to visualize the 3D position and orientation of teeth at the time of correction. In conservative dentistry, the AR system may help determine the 3D location and relationship between an apical lesion and the root canal or the condition of a root canal. In dental implantology, this technology would be particularly useful for preventing damage to the underlying nerves and cortical plate perforations during implant placement.

The novel AR technology provides a complete 3D output and would therefore add to the precision and confidence of the surgeon. Using this system, the movement of the surgeon's head does not require the image to be adapted and updated for the altered perspective, which is an advantage compared to other systems. Future research based on this study will focus on improving some of the technical aspects with regard to the accuracy and tracking system, such as a high-definition back display or lens array for the densification of pixels in the AR display to achieve high levels of precision and geometrical accuracy, which is important because the system provides a 3D image containing accurate spatial information. Further, the intended next step is to initiate clinical trials of this technology in patients to assess a multitude of applications of this technique.

Conclusion

In conclusion, this study presents an accurate AR system for use in oral and maxillofacial dentistry that provides a real-time, in situ, stereoscopic visualization of 3D-CT IV images overlaid onto the surgical site with the naked eye.