Keywords

1 Introduction

Storytelling content allows users to experience various stories through a character’s active behaviors [1,2,3]. By matching a real space, the user can feel as if the character is in the same world. If the user can experience an Mixed Reality (MR) story of their room as a character world setting, this may increase the reality of the experience. It is thought that interaction, such as being able to change to the story through room remodeling, is possible. For example, when routes to obtain items differ depending on the shape of the room or the furniture arrangement, it is possible to use furniture to place a bridge (Fig. 1). Everything in the room can be used as a tool to experience the content. This study proposes a method to build an MR environment tailored to the room in preparation for developing a system that can realize an experience in which the user senses that they are living in the same room as the character.

Fig. 1.
figure 1

(left, right) Image that makes room a character world stage. (center) Creating a road using furniture.

Full size image

2 Related Work

One study into the story experience is Steven’s AR Façade [4], which explored communicating with characters and experiencing a story in real space using a transparent HMD based on Facade [3]. Façade is the story content in the house where the character lives. The AR Façade reproduces the same placement in real space. Nakevska used the CAVE environment and projected images onto the walls of a room in order to perform the story experience of Alice in Wonderland [5]. In a room, the user can experience a story by acting on objects in the room or projected images; however, such methods cannot be incorporated into a user’s own room because such a real environment is special.

Another method sets a virtual object that matches a real environment measured using a depth camera to create a character world tailored to an arbitrary room. Misha’s Oasis [6] detects a walkable region from three-dimensional information of walls and floors to build a VR space adapted to that region. That is not story content. The proposed method also measures a real environment.

With the proposed method, we aim to create MR story experiences in arbitrary environments by setting virtual objects according to measured information.

3 Construction of Mixed Reality Space

3.1 Overview

We construct an MR space that matches the shape of the three-dimensional data of a real space acquired by scanning. We define the floor, tables, etc. as a “horizontal plane,” walls, shelves, etc. as a “vertical plane,” and inclined planes as a “slope.” We define this classification as a “region.” We place a CG object on the surface classified into each region based on the rule. A CG object placed in a certain region is defined as a “stage.” In this procedure, three-dimensional data of the physical space are classified into each region, and the stage is arranged in a region according to the rule.

3.2 Obtaining Surface Meshes

Note that the mesh shape of the obtained three-dimensional data is incomplete, and there are extra meshes and holes caused by the ceiling and measurement error. Data resampling is performed because it is difficult to judge the appropriate region. Note that subsequent processes can be simplified by acquiring data as a single plane mesh. We performed data resampling so that we could handle only the confirmed surface from the top of the room model (Fig. 2). First, we sufficiently cover the room model and prepare a fine rectangular mesh relative to the room model mesh. Move from the top of the room model to the bottom and fix each vertex at the point where it hits the mesh of the room model. Thus, it is possible to acquire a mesh of the surface.

Fig. 2.
figure 2

Retrieving the surface mesh by resampling.

Full size image

3.3 Region Classification

To classify the re-sampled three-dimensional data into each region, clustering is performed based on the inner product of the normal vectors of the mesh vertices. The inner product of the normal vectors of adjacent vertices is taken to find the angle between the normal vectors. Clustering performs vertices with angles less than a threshold as the same region. To make vertices classified in each cluster a mesh, a Delaunay triangulation is created from the group of vertices. Thus, it is possible to divide the three-dimensional data of the real space into planes. The plane is estimated using the RANSAC algorithm on the divided planes, and the inclination of the plane is obtained from the estimated plane equations. Each region is classified using the inclination and area of the plane (Table 1). For visibility, we change the color according to the region (Fig. 3).

Table 1. Region classification.
Full size table
Fig. 3.
figure 3

Region classification

Full size image

3.4 Stage Placement

We place stages based on the rules in each region, and a stage is placed in a region or between regions. Note that placement is performed based on different rules.

Rules for placing stages in regions. Region: Region type. (e.g. FLOOR, HPLANE, VPLANE…), Area: Sort by Region area. (e.g. 1, 2, 3…, unspecified), Height: Sort by Region height. (e.g. 1, 2, 3…, unspecified), Number of Regions: Number of regions to place the stage. (e.g. 1, 2, 3…, unspecified), Number of Stages: Number of Stages placed in the Region. (e.g. 1, 2, 3…, unspecified).

Rules for placing a stage between regions. Target Stages: Stages where areas are placed. (e.g. Tent-Lab), Number of placement: Number of stages to be connected from one stage. (e.g. 1, 2, 3…, unspecified).

The placement rule is determined by combining these. When multiple stages satisfy the condition, they are placed in the same region. In addition, the placement position is determined randomly within a range not outside a region but also not in contact with another mesh; however, in the case of a very narrow area, it does not depend on it.

3.5 Result of Placement

Stages (Fig. 4) were placed in two real environments (room1 and room2) according to the rules (Tables 2 and 3). The scan data are shown in Fig. 5, and the placement results are shown in Fig. 6. Note that they were also confirmed using Microsoft HoloLens (Fig. 7).

Fig. 4.
figure 4

Stage models.

Full size image
Table 2. Placement rule of the stage on the region.
Full size table
Table 3. Placement rule of the stage between regions.
Full size table
Fig. 5.
figure 5

Scanned data. (left: room1, right: room2)

Full size image
Fig. 6.
figure 6

Result of stage placement. (left: room1, right: room2)

Full size image
Fig. 7.
figure 7

Confirmation by Microsoft HoloLens (left: room1, center and right: room2)

Full size image

4 Discussion and Future Work

The purpose of this study was to experience MR in an arbitrary environment by scanning a real space. Note that we did not achieve completely automatic placement; however, we were able to place objects according to the environment. Note that not all components of this system work automatically. For example, parameters must be changed according to the given environment. Therefore, it is necessary to acquire an appropriate threshold and automatically change the shape and size of the model at the time of placement according to the given environment. In addition, this work represents an incomplete implementation of an environment construction system; thus, this work does not represent a method that can provide a true story experience. Therefore, it is necessary to implement a story system that changes with user interactions and the environment. We think that it is possible to experience an MR story as if the character actually lives with the user by enhancing the completeness of the system.