Keywords

1 Introduction

Following the development of mobile augmented reality (AR) in the 2010s, AR technology has been confirmed to have applications in various fields. Because AR technology effectively transmits various additional information related to reality, it is used as an immersive delivery method in the commercial field, and it is utilized as a new medium for enhancing the learning effect of users in the field of education and training.

Following these trends, AR technologies are actively applied in the field of cultural heritage. In general, an in-depth understanding of cultural heritage requires a historical, economic, and cultural context of the cultural heritage, apart from a description of itself. For this purpose, information that can improve understanding is provided to visitors in various forms, such as text-based description and various audiovisual materials. Consequently, AR technology, which increases human recognition abilities, is attracting attention as the next generation technology in the field of cultural heritage, as a means for effectively providing a large amount of additional information to visitors [1].

Major research projects that use AR technology in the cultural heritage domain are classified into applications in museum-oriented indoor environments and applications in site-oriented outdoor environments. For example, the ARCO System [2], ARTSense Project [3], and Project CHESS [4] are representative examples that use AR technology in indoor environments. The ARCO System was developed in the early stages of the applications of VR and AR in museums. Although it was an early attempt, the ARCO System established a data structure that considers users, technologies, interoperability, and operational practices to provide reusability and interoperability. The ARCO system is considered a novel precedent for the later AR projects that were executed in Europe. In the cases of the ARTSense Project and Project CHESS, these projects attempted to apply AR in indoor environments such as European museums. The ARTSense Project is characterized by the provision of user adaptive services based on user context information acquisition. On the other hand, the CHESS project provided a storytelling user experience and supported a content authoring tool, which was used by the general public to create and share the content.

On the other hand, there have been several projects that attempted to deploy AR in outdoor environments. Archeologuide [5] was the first attempt to apply outdoor AR in the cultural heritage domain; it visualized the 3D model of an ancient Greek temple as an augmented reality from an outdoor site based on a PC-based hardware platform. Archeologuide is considered a pioneering example of proving the hardware feasibility of augmented reality in an outdoor environment. Since then, Mobi-AR [6] has been providing urban tourism support applications using augmented reality technology in a mobile environment. Mobi-AR detects the location of users using embedded sensors and vision algorithms and provides related information through the mobile device. Particularly, to address the issue of computing power and battery depletion, which are considered inherent limits of mobile platforms, a remote server is dedicated to the calculation of user localization. Project ORCHESTRA is the first example of AR implementation based on wearable platforms of optical see-thorough HMD devices in the cultural heritage domain. Although it focused on user interaction that is suitable for wearable environments based on vision technology [7] and user evaluation through demonstration [8], this project proved that it is still difficult to utilize wearable devices for augmented reality due to the technical limitations of hardware. In view of this technological trend, present augmented reality technologies are in a transition period from mobile platforms to wearable platforms.

Moreover, we have tried to apply AR technology in the cultural heritage domain through K-Culture Time Machine Project [9, 10]. The K-Culture Time Machine Project aims to integrate heterogeneous cultural heritage databases in Korea and create time-space correlations between cultural heritages and provide time-space correlated content through AR technology. For this purpose, we have studied the integration of heterogeneous cultural heritage databases according to time and space, and we have standardized this aggregation method, while providing users with an immersive experience through AR technology. In this paper, we introduce the wearable AR platform, which is the latest development in the K-Culture Time Machine Project.

2 Wearable AR Platform for K-Culture Time Machine

The wearable platform of K-Culture Time Machine Project consists of two part. The first part retrieves time-space correlated content and multimedia data and incorporates the information into a standardized metadata schema. To provide these contents and multimedia data, heterogeneous databases have been aggregated with mitigating ontology. The second part provides the AR experience to user in outdoor environment. To support AR visualization in outdoor environments, spatial data composition component constructs spatial information of a heritage site based on vision and sensor information. In addition, an experience provisioning module is developed to provide immersive experiences to users. The following sections will provide a detailed description of each part of the project.

2.1 Standardized Metadata Schema-Based Data Retrieval

Standardized metadata schema-based data retrieval module performs the dynamic content retrieval for the wearable AR visualization and its overall process is described in Fig. 1. The content retrieval process consists of two main flows. First, an ontology, constructed according to the time-space correlation defined by existing Korean Cultural Heritage Data Model (KCHDM) [11], is parsed by an SPARQL query. Second, a multimedia content DB, which contains the multimedia content related to the cultural heritage, is established and parsed with a MySQL query. Such information is integrated into the revised standardized metadata for AR content [12], and visualization of the AR content is performed based on this data structure.

Fig. 1.
figure 1

Overall workflow of the data retrieval process

Full size image

For the time-space correlated content integration, the Korean Cultural Heritage Data Model (KCHDM) [13], which is the ontology standard for aggregating heterogeneous data from Korean cultural heritage institutions, was developed and adapted into the retrieval process. KCHDM consists of 31 classes and 41 properties to link cultural heritage entities semantically. Through the time-space correlation generation technique proposed in the previous study [10], a conceptual relationship model among cultural heritage entities based on KCHDM can be developed semi-automatically.

However, the conceptual relationship model from the previous process needs to be modified in order to develop the cultural heritage knowledge base for the wearable AR. Although KCHDM has 31 classes, we used only 5 super classes, including actor, event, thing, time-span, and place, due to the simple and intuitive user interface of the wearable platform. Figure 2 is an example of a knowledge graph for the wearable AR platform of K-Culture Time Machine.

Fig. 2.
figure 2

An example of a knowledge graph for injeongjeon hall of the Changdeokgung palace [13]

Full size image

In addition to modifying classes, the process of mapping web resources to the relationship model is needed. The conceptual relationship model does not include information from web resources that present a certain cultural heritage entity, but includes information on only the relationship between cultural heritage entities. Therefore, we used the properties “has description” for text descriptions and “has representation” for multimedia content in order to map web resources onto the relationship model. In addition, the data-type property “has url” was used for presenting the locations of web resources in order to load content in the wearable AR platform. After these modification processes, the final relationship model is stored in the web database as Web Ontology Language (OWL) format.

To parse the information from the OWL format, the SPARQL Query method is utilized in the data retrieval process. Using the dotNefRdf library [14], which is an open source.NET Library for Resource Description Framework (RDF), ontology of OWL format can be parsed into C# data structures, and it can be utilized in visualization platforms by parsing the ontology into standardized 5W1H metadata schema implemented with C# data structures. Using ontology data parsing, detailed descriptions and media content for each cultural heritage, which are in raw database of the concerned authorities, and the spatio-temporal correlation between cultural heritages are parsed into the standardized metadata schema.

Multimedia database includes video content related to cultural heritages such as historic TV dramas and movies. To support the interoperability and reuse of AR content, existing metadata schema and its design principles [15, 16] are referred for building our multimedia database. The features of the referred existing metadata sets are as follows. First, the metadata set of W3C [17], the de facto standard of metadata schema, aims at enhancing the interoperability of different descriptions of media resources. It has 28 critical multimedia metadata properties proposed by W3C by analyzing 18 multimedia metadata formats and 6 multimedia container formats. Second, existing metadata schema for the AR content is used for considering the characteristics of AR and supporting the AR visualization [18]. It supports expandability and context-aware features for AR visualization, based on the 5W1H-based metadata schema. Third, the metadata set for broadcast content distribution [19] is used for designing a data structure for the metadata schema. For covering the video resources thoroughly, we adopted a multi-layered hierarchical structure for the video [20]. In the hierarchy, video and sequence have the same meaning of program and episode as shown in [19]. Therefore, it was used for designing the video and sequence classes to implement the multimedia database. Figure 3 shows the process of establishing a multimedia database.

Fig. 3.
figure 3

Implementing a multimedia database

Full size image

To parse video content from the database, a multimedia–parsing module is developed using SQL query with POST. When the name of the cultural heritage entity is given, the module retrieves scenes and shot data with the same entity value. These parsed data are restructured into the standardized 5W1H metadata schema with C# data structure. Through this process, the content can be retrieved and visualized in the AR system.

As an integrating container, standardized metadata schema is adapted for background data structure of the wearable AR platform. We had worked on the standardizations of an aggregation method for heterogeneous cultural heritage database [11] and integrating metadata for AR content [12]. Currently, we are revising the AR content metadata standard to integrate the two previous standards. The main improvement is adding the MAR ontology structure to include the KCHDM and other elements of ontology. To support the service, user information is added and a lot of common information is arranged. The result of the revision is shown in Figs. 4 and 5.

Fig. 4.
figure 4

Revised 5W1H metadata schema for AR content

Full size image
Fig. 5.
figure 5

Revised 5W1H metadata structure for AR content

Full size image

According to the schema, background content data structure of the visualization module is developed and information is parsed as discussed above. From the KCHDM ontology, information on cultural heritages and their relevance and related content are parsed. On the basis of that, contemporary video content of multimedia database complements the data of the information content. After this process, the visualization module can provide the AR/VR scene and related contents from the contents data.

2.2 Wearable AR Platform Development

Vision- and Sensor-based Spatial Data Composition

To support outdoor AR environment, we proposed a vision-based outdoor AR framework for cultural heritage sites in previous research [21]. The framework was designed to incorporate computer vision-based camera tracking and ontology-based data-authoring technology. Our framework design includes: (1) visual data generation, (2) AR content authoring, (3) mobile optimization, and (4) AR content visualization.

In the visual data generation module, 3D keypoints and keyframes are automatically generated by the SfM (structure-from-motion) pipeline. For the feature extraction and matching process, we use ORB [22] to reduce computational costs for real-time camera tracking. In the AR content authoring process, we connect associated 3D coordinates of a PoI (Point of Interest) with virtual content. Figure 6 shows the content authoring process. In order to stabilize the camera position estimation and tracking, our framework was designed using the multi-threading architecture. Finally, the content visualization module integrates 3D visual data and content authoring results to visualize the AR content with the current 6DoF (Degree-of-freedom) camera position of the mobile device. Figure 7 shows real-time AR content visualization results.

Fig. 6.
figure 6

AR content authoring process

Full size image
Fig. 7.
figure 7

AR content visualization

Full size image

Immersive Visualization Provision

The immersive visualization module provides an enhanced experience to the user by generating an AR/VR scene for cultural heritages. The AR module provides mobile-based AR experience to the visitor with related information and multimedia content of cultural heritages. On the other hand, the VR module provides mobile - and wearable-based remote experience to a user exploring sites of cultural heritage from remote environment based on 360-degree panoramic images.

The AR visualization module aims to provide enhanced experience to the visitors of the cultural heritage site. Based on the vision - and sensor-based spatial data composition, user localization of object-level accuracy in large-scale outdoor environment is present due to the hybrid tracking method. Moreover, a user can search for related information and multimedia content with dynamic access to raw databases through the user interface.

The VR visualization module was developed to address the space constraints of the AR where users must be on the site and for immersive VR experience. The VR visualization module provides users with a VR experience through mobile and wearable platforms based on 360-degree panoramic images. The user’s time-space constraints required by the AR module are solved via the VR module through a remote experience of the cultural heritage site.

On the mobile platform, users can explore cultural heritage sites like the street view feature of the existing maps service such as Google maps. In addition, the same information and multimedia content provided by the AR module are also supported by the VR module. Through this feature, a user can explore a cultural heritage site in a remote environment. On the other hand, on the wearable platform, immersive experience on the cultural heritage site and 3D reconstruction of the cultural heritage, which is currently perished, are provided.

3 Prototype and Future Expansion

A prototype was developed to verify the proposed platform. This prototype targets an area from Sejong-ro to Changdeokgung Palace in Seoul (Fig. 8). The prototype supports mobile AR visualization, and mobile and wearable VR experience based on 360-degree panoramic image. The prototype provides AR visualization based on detailed object-level robust tracking in the outdoor environment and provides a virtual experience from the Sejong-ro to the Changdeok Palace based on 360-degree panoramic images. The smartphone VR HMD was utilized to solve issues such as limited viewing angle of the current optical see-through HMD, battery depletion and insufficient computation power of devices, and barriers to user entry that requires purchasing expensive HMD. To provide the 360-degree panoramic view, Google VR SDK was implemented.

Fig. 8.
figure 8

360-degree panoramic image-based VR experience using the prototype

Full size image

However, there are some issues that need to be addressed in order to visualize assorted content on mobile and wearable platforms. First, for a digital cultural heritage, all content has authenticity issues. If the digital contents of cultural heritage are not validated by historical records, there could be a risk of conveying false information to users. In this prototype, we solved the issue of authenticity by directly utilizing the contents of the existing cultural heritage database built on the basis of historical data, or developing content by referring to them. On the other hand, 3D model data and large volume of contents may consume a lot of resources to be visualized on mobile and wearable devices. For this case, we optimized the content with the appropriate level of data for mobile and wearable devices and made it available to users.

For future expansion, we plan to develop several features such as: (1) advanced video content retrieval and (2) view-point guidance. First, using the integrated metadata schema and content parsing module, advanced video content retrieval can be enabled. Until now, when searching for video content, users were able to use limited information of a video such as title, actor, and genre. Moreover, it has been impossible to search for video clips based on the content of a scene or shot unit. However, based on the integrated metadata schema, users can search related multimedia based on not only the user’s location and position but also the temporal and spatial data of the content. As shown in Fig. 9, a user can watch various video clips related to certain PoIs, watch video clips filmed in one location sequentially, and watch video clips based on the same time background in PoI. This would provide new media experience in cultural heritage sites by providing various TV drama or movie clips, which are temporally and spatially related to the historical site.

Fig. 9.
figure 9

Advanced video contents retrieval based on multimedia database

Full size image

In addition, using the integrated metadata schema could provide immersive experience with view-point guidance (Fig. 10). Each frame of a video contains various information such as date, GPS, compass, and altitude, a service can guide users to the exact point of the video clip to be watched. At this point, users can watch the real world as the background of the video clip on the device screen. Furthermore, when a tourist visits the filming location, oftentimes, it looks different from the scene because of several reasons such as different seasons, different time (day or night), and computer graphics. If the database has data related to the filming date or object, it would be possible to provide more immersive experience by augmenting the scene to the real world based on the context-awareness of an environment, content, and a user.

Fig. 10.
figure 10

View-point guidance based on metadata of contents

Full size image

4 Conclusion

In this paper, we presented a wearable AR platform for the cultural heritage of the K-Culture Time Machine project. In the back-end, heterogeneous cultural heritage databases were aggregated with standardized metadata schema, and extensive spatial data composition was performed to support vision-based rigorous tracking in an outdoor environment. In the front-end, the platform provided AR/VR visualization to provide the additional information, historic knowledge, and assorted multimedia content for cultural heritages.

To verify the platform, we developed a prototype targeting the cultural heritage of Seoul. With several additional functionality expansions discussed above, a usability test would be part of a future study.