Augmented reality visualization: A review of civil infrastructure system applications

Abstract

In Civil Infrastructure System (CIS) applications, the requirement of blending synthetic and physical objects distinguishes Augmented Reality (AR) from other visualization technologies in three aspects: (1) it reinforces the connections between people and objects, and promotes engineers’ appreciation about their working context; (2) it allows engineers to perform field tasks with the awareness of both the physical and synthetic environment; and (3) it offsets the significant cost of 3D Model Engineering by including the real world background. This paper reviews critical problems in AR and investigates technical approaches to address the fundamental challenges that prevent the technology from being usefully deployed in CIS applications, such as the alignment of virtual objects with the real environment continuously across time and space; blending of virtual entities with their real background faithfully to create a sustained illusion of co-existence; and the integration of these methods to a scalable and extensible computing AR framework that is openly accessible to the teaching and research community. The research findings have been evaluated in several challenging CIS applications where the potential of having a significant economic and social impact is high. Examples of validation test beds implemented include an AR visual excavator-utility collision avoidance system that enables workers to “see” buried utilities hidden under the ground surface, thus helping prevent accidental utility strikes; an AR post-disaster reconnaissance framework that enables building inspectors to rapidly evaluate and quantify structural damage sustained by buildings in seismic events such as earthquakes or blasts; and a tabletop collaborative AR visualization framework that allows multiple users to observe and interact with visual simulations of engineering processes.

Introduction

In several science and engineering applications, visualization can enhance a user’s cognition or learning experience and help communicate information about a complex phenomenon or to demonstrate the applicability of an abstract concept to real world circumstances. An important category of visualization is termed Virtual Reality (VR), which replaces the user’s physical world with a totally synthetic environment and isolates the user’s sensory receptors (eyes and ears) from the real physical world. The cost and effort of constructing a faithful synthetic environment includes tasks such as model engineering (the process of creating, refining, archiving, and maintaining 3D models), scene management, and graphics rendering and can thus be enormous [1].

In contrast to VR, another category of visualization techniques, called Augmented Reality (AR), preserves the user’s awareness of the real environment by compositing the real world and the virtual contents in a mixed (blended) 3D space [2]. For this purpose, AR must not only maintain a correct and consistent spatial relation between the virtual and real objects, but also sustain the illusion that they coexist in the augmented space. In addition, the awareness of the real environment in AR and the information conveyed by the virtual objects provide users with hints to discover their surroundings and help them perform real-world tasks [3]. Furthermore, AR offers a promising alternative to the model engineering challenge inherent in VR by only including entities that capture the essence of the study [4]. These essential entities usually exist in a complex and dynamic context that is necessary to the model, but costly to replicate in VR. However, reconstructing the context is rarely a problem in AR, where modelers can take full advantage of the real context (e.g., terrains and existing structures), and render them as backgrounds, thereby saving a considerable amount of effort and resources.

Fig. 1 shows the virtuality continuum, a concept first defined by Milgram and Kishino [5] that represents the mixture of classes of objects presented in any particular display situation. Within this continuum, the real environment (i.e., reality) and the virtual environment (i.e., VR) are shown at the two ends. While reality defines environments consisting solely of real objects, VR defines environments consisting solely of virtual objects, an example of which would be a conventional computer graphic simulation. As illustrated in this continuum, Mixed Reality (MR) is an environment in which real world and virtual world objects are presented together, that is, anywhere between the extremes of the virtuality continuum. AR and Augmented Virtuality (AV) fall in this category.

The application of visualization techniques such as AR for planning, analysis, and design of Architecture, Engineering, and Construction (AEC) projects is relatively new compared to the sizeable amount of AR-related research conducted for diverse applications in fields such as manufacturing, medical operations, military, and gaming. A thorough statistical review of recent AR-related research studies in AEC and potential future trends in this area was recently conducted by Rankohi and Waugh [6]. Their work showed that field workers and project managers have high interest in using non-immersive and desktop standalone AR technologies during project construction phase mainly to monitor progress and detect defective work. In another study, the potential of AR applications in AEC including eight work tasks (i.e., layout, excavation, positioning, inspection, coordination, supervision, commenting, and strategizing) was discussed [7]. A general overview of AR technology and its use in construction and civil engineering applications, as well as applications in other fields can be found in [8]. Examples of major undertakings in AR research and development that have at one point set the tone for future endeavors are provided in the following paragraphs. Section 1.2 will contain detailed discussions about more recent value-adding applications of AR within the AEC domain.

In order to assist with utility inspection and maintenance, researchers have used AR to visualize underground utilities, provide the ability to look beneath the ground, and inspect the subsurface utilities [9]. Some further exploration can be found in [10], [11] where the work has been extended to improve visual perception for excavation safety and subsurface utilities, respectively. AR can serve as a useful inspection assistance method in the sense that it supplements a user’s normal experience with context-related or georeferenced virtual objects. For example, an AR system was developed by Webster et al. [12] for improving the inspection and renovation of architectural structures by allowing users to see columns behind a finished wall and re-bars inside the columns. A “discrepancy check” tool has been developed by Georgel et al. [13] that allows users to readily obtain an augmentation in order to find differences between an as-design 3D model and an as-built facility. Other researchers implemented a system for visualizing performance metrics that aims to represent progress deviations through the superimposition of 4D as-planned models over time-lapsed real jobsite photographs [14]. In another example, overlaying as-built drawings onto site photos for the purpose of continuous quality investigation of a bored pile construction was presented [15].

Some examples of coordinating and strategizing are the visualization of construction simulations and architectural designs. ARVISCOPE, an AR framework for visualization of simulated outdoor construction operations was designed by Behzadan and Kamat [16] to facilitate the verification and validation of the results generated by Discrete Event Simulation (DES). Another mobile AR platform called TINMITH2 was developed by Thomas et al. [17] and used to visualize the design of an extension to a building. Some other construction tasks which feature high complexity may also benefit from AR. For example, the quality of welding normally depends on the welders’ experience and skill. By developing a welding helmet that augmented visual information such as paper drawings and online quality assistance, researchers were able to improve welders’ working conditions as well as the quality control [18].

Recent applications of AR technology in AEC domain have helped improve performance in areas such as virtual site visits, comparing as-built and as-planned status of projects, preempting work package schedule disputes, enhancing collaboration opportunities, and planning and training for similar projects [6], [19]. Examples of such application areas include but are not limited to a framework for analyzing, visualizing, and assessing architectural/construction progress with unordered photo collections and 3D building models [20], [21], a client/server AR system for viewing complex assembly models on mobile phones [22], a tangible MR-based virtual design prototype as a distributed virtual environment (DVE) for the purpose of improving remote design review collaboration [23], an AR interior design service which combines features of social media, AR and 3D modeling to ambient home design [24], an interactive speech and gesture recognition-based, immersive AR model designed to visualize and interact with buildings and their thermal environments [25], an integrated AR-based framework for indoor thermal performance data visualization that utilizes a mobile robot to generate environment maps [26], a tabletop AR system for collaboratively visualizing computer-generated models [27], and a mobile AR application capable of delivering context-aware visual project information to students and trainees to improve the quality and pace of learning [28].

The Laboratory for Interactive Visualization in Engineering (LIVE) at the University of Michigan has been engaged in AR research with applications related to construction operations planning, inspection, safety, and education. These AEC applications include visual excavator-collision avoidance systems, rapid reconnaissance systems for measuring earthquake-induced building damage, and visualization of operations-level construction processes in both outdoor AR and the collaborative tabletop AR environments (Fig. 2). The developed visual collision avoidance system allows excavator operators to persistently “see” what utilities lie buried in the vicinity of a digging machine or a human spotter, thus helping prevent accidents caused by utility strikes [29]. With the aid of AR, the rapid post-disaster reconnaissance system for building damage assessment superimposes previously stored building baselines onto the corresponding images of a real structure. The on-site inspectors can then estimate the damage by evaluating discrepancies between the baselines and the real building edges [30]. Moreover, research was conducted to enable the visualization of construction operations in outdoor AR to facilitate the verification and validation of the results of simulated construction processes, with minimum effort spent on creating 3D models of the surrounding environment [31]. Lastly, the tabletop collaborative AR visualization helps to bridge the gap between paper-based static information and computer-based graphical models. It reflects the dynamic nature of a jobsite, and preserves the convenience of face-to-face collaboration [27].