1 Introduction

Self-driving cars is an emerging technology which is expected to give a huge impact in the transportation and mobility field. However, concerns regarding the various and complex riding situations require the self-driving technology to be highly accurate due to the serious consequences in case of any kind of errors. Hence, implementation of the state-of-art technology has started in a slow but a steady and reliable approach. Especially nowadays, assistive technologies are implemented to assist the driver in various ways which provide practical and convenient functions. On the other hand, technologies to provide autonomous functions are yet not practical enough, though some prototype tests or practical tests have recently started. These technologies are named as Advanced Driver Assistance Systems (ADAS) is under research and tests, although, on-road studies are highly limited to avoid serious accidents. Especially, tests to verify or validate under certain circumstances, such as testing with elder driver, needs to take care of extreme caution. Therefore, this paper proposes a practical approach to conduct safe and reliable tests for such circumstances. In particular, this paper proposes utilizing an affordable and highly customizable immersive virtual reality environment to conduct various human-in-the-loop simulation test.

2 Recent Approach for Advanced Driver Assistance System Tests

Recent developments ofnself-driving cars have reached to a level where an autonomous car can fairly drive through cities in normal traffic condition, though the system still needs special driver operations in case of difficult situations. In the field of self-driving cars, the action of moving the privilege from the driver to the system, or vise-versa, starts with an action named Take-Over-Request (TOR). While the design of TOR carefully respects the principle of Human-Machine Interface (HMI), it is normally done under simulated environment, typically using a driving simulator. Studies use reaction time [1] to measure the performance, though there are various studies using other indexes since there are various types of HMI display. Some studies of HMI display examples use visual display [2], auditory display [3], vibrotactical display [4] and the combination providing multimodal display [5, 6]. It is also to be noted that the TOR occurs continuously while driving, depending on traffic situations or drivers will [7], and the performance of reaction to TOR may differs depending on various aspects like drowsiness [9], alcohol consumption [8] and distractions from non-driving related tasks [10] or related tasks [11]. Nonetheless, the driver’s driving experience and habits are traditionally known to have meaningful relations with crash risks.

While the driver’s characteristics and the crash risks tends to indicate some correlations [12], it typically is biased indicating other fundamental issues underlying [14]. For example, from the perspective of the aging society that Japan is facing, the increase of elder driver’s crash risk is attributed to age-related and disease-related cognitive deficits rather than the term age itself. Therefore, the crash risk may be lowered if the driver focus on maintaining and strengthening situation awareness skills. Although one of the most effective act to strengthen the situation awareness skill is actual driving experience [15, 16], driving on the road without having enough skill and experience is obviously dangerous. Hence, driving simulator is typically used for such kind of training which can expect certain amount of efficiency, even though there are limitations based on the configuration of the simulator [17].

Driving simulators can also be utilized as an environment to understand driving behaviour in general, critical use cases, or to test HMI prototypes. Due to various resource limitations, simulators typically have its resource weight to realize the main purpose of the simulator. Considering the fact that the actual proportion of how much visual information drivers rely to drive seems high as 80% or more [18], immersive virtual reality environments such as CAVE [19] are known to be effective in such simulators. Compared to Head-Mounted Display (HMD) environment, CAVE can easily integrate physical interfaces in the virtual reality environment where it can physically include a driving seat or a full-scale automobile. While CAVE has the ability to realize immersive virtual reality, the configuration scale becomes relatively large and complex which also has significant issues of the very high cost, whereas the original CAVE [20] and CAVE2 [21] are estimated to have cost $2,000,000 and $926,000 respectively. Therefore, the complexity and difficulty to control the simulation increases exponentially when making and deploying scenarios or actors within the driving simulator.

Recent research and commercial development has significantly reduced the cost to construct the CAVE, especially on the software. Low-cost CAVE has been introduced by using recent game technologies such as CryEngine [22] or Unity Engine [23, 24]. There is also free low-level OpenGL library [25, 26]. Low-level implementation has pros of capability with high customization and configuration, although, has cons of requirement of heavy programming when developing an application. Additionally, sensor integration for head tracking and bridging to the application uses many proprietary closed-source hardware and software which consumes high budget and human resource for configuration. Therefore, while trying to realize an immersive virtual reality environment, it is typical to configure the system for such main purpose. Also, it is not always easy to overcome the difficulties in financial and spatial resource to construct an immersive virtual reality environment. Therefore, the process of splitting the software development and usage of running the software within the virtual environment needs to be considered, composed by local software development and remote use.

Immersive virtual reality environment can be considered for use as a driving simulator with the aim to conduct safe and reliable tests of self-driving technology for elder drivers. Regarding that software development and running the application are not always capable in one physical location, systems engineering methods shall be adapted to keep the simulation entity and functions clear which is allocated to whichever software or hardware. Similar approach can be observed within the kinect-based KAVE [24]. However, this paper adopted a slightly different approach due to constraint of realizing a driving simulator application configured for elder driver. Considering the complexity of implementing ADAS functions, this paper utilized a proprietary physics-based simulation platform for driving simulator. In particular, we propose and demonstrate a practical method and approach to develop a CAVE driving simulator as an ADAS HMI test environment for elder driver.

3 Design of Human-in-the-Loop Simulation Using Immersive Virtual Reality Environmenmt

Human consideration in the process of engineering is important especially in human-centered domains [27]. It is commonly referred as human-in-the-loop simulation when a human can operate or interact within the simulation in real time. The concept of using virtual reality environment for human-in-the-loop simulation can be observed in fields of system evaluation within the automobile field. For example, evaluation on steer-by-wire system [28], braking-based rollover prevention [29], and wheel loader [30]. The virtual reality environment can also act as an enabling system, in occasion when there is a different system of interest. This paper proposes the method to enable the virtual reality environment for both occasions with ease of switching in both directions to simulate within or without the system for testing.

3.1 Human-in-the-Loop Simulation for Advanced Driver Assistance System

In order to conduct practical simulation, it is important to understand the basic system architecture operated in the real world. Since this paper focuses on testing ADAS for elderly driver, therefore, understanding the system architecture of ADAS shall be the first step. Figure 1 illustrates the basic system structure of ADAS domain, based on HAVEit (Highly Automated Vehicle for Intelligent Transport) [31, 32] perspectives. Described in a block definition diagram based on SysML (System Modeling Language) [33], the figure indicates that the domain of automobile ADAS is composed of ego vehicle driver, ego vehicle, ADAS, pedestrian, surround mobility, transport system, and physical environments. Since Fig. 1 only indicates the top level hierarchy, it is noted that some blocks shall be specified when necessary, for example, the Surrounding Mobility can be specified into Rear Vehicle, Side Vehicle, Leading Vehicle, and Potential Vehicle. Components such as roads and traffic signals, are also generalized as Transport System, and other objects are generalized as the Physical Environment.

Fig. 1.
figure 1

Block definition diagram of automobile Advanced Driver Assistance System domain based on HAVEit.

Full size image
Fig. 2.
figure 2

Use case diagram of Advanced Driver Assistance System from HAVEit.

Full size image
Fig. 3.
figure 3

Overall illustration with some basic physical components with indication what needs to be simulated to run a simulation.

Full size image

Regarding the basic system architecture, HAVEit address 1 important key functional task repartition between the driver and system, which are described with 3 functional definitions included and 1 functional definition to extend. The 3 included functionals are ensure driver can react properly, ensure driver can react properly, recognize scenery, and the 1 extend functional as perform safe motion control. Figure 2 illustrates this functionality using a use case diagram, based on some early research analysis on HAVEit [34].

According to accident analysis done by HAVEit [32], it addresses the fact that 95% of all accidents are driver related and more than 22% are related especially to lack of driver’s alertness. Therefore, HAVEit especially focus on including the driver as a key element when considering ADAS since there are occasions when ADAS cannot handle the situation and the driver needs to take over. Hence, following HAVEit’s philosophy, the driver is considered included inside the system loop, which means the ADAS and the driver will both need to be included when performing research or development.

Overall, the approach of HAVEit can be illustrated as Fig. 3. Original overall block diagram from HAVEit [32] is described based on four layers within the HAVEit architecture. The four layers are Perception layer, Command layer, Execution layer, and HMI layer, though the layers are only explainable from the function definition and requirements phase, which it too much to describe within this paper. Hence, rather not completely correct illustration, though this paper illustrates Fig. 3 colored and described based on the functionality, described in Fig. 2.

Based on Fig. 3, it is understandable that three main functionality is required within simulation for ADAS, and while one functionality (perform safe motion control) extends the main functional task repartition between the driver and system is not directly necessary. The block diagram also indicates the interactions between the components, which can also explains that the Ego Vehicle Driver is included through a loop consisted by HMI and Driver Monitoring, derived from the ensure driver can react properly. Hence, consideration of the Ego Vehicle Driver is extremelly important, which can be realized by using either a driver model for automated scenario based simulation, or a real human for human-in-the-loop simulation. Meanwhile, for the components other than the driver, preparation for the simulator shall be focused on the three functionality, allocating the funtional to how it shall be virtually realized in the simulation. In particular, within this paper, the issue is to how shall the immersive virtual reality environment can be utilized.

3.2 Configuration for Immersive Virtual Reality Environment

There are several well known definitions or principles for virtual reality environment such as \( I^3 \) (Immersion, Interaction, Imagination) [35] and AIP (Autonomy, Interaction, Presence) [36]. Within the topic of this paper, the key concept can be summarized as the following: iteraction in real-time, immersion in real-time, and environment in realistic 3D. Running the simulation based on human-in-the-loop simulation in real-time will basically provide the real-time-ness. Hence, the key aspects to consider for immersive virtual reality environment are on interaction, immersion, realistic 3D environment. In particular, an operative hardware or devices for interaction, big screens surrounding the driver’s field of view for immersion, and a realistic vehicle model within a realistic simulation world for realistic 3D environment.

For immersion, if the costs are not a big issue, utilizing the CAVE [19] and enabling it as an immersive CAVE simulator shall be on technical approach to consider. This approach has been taken by previous research, for observation of drivers’ behavior on narrow roads [37], and design of motorcycle head-up display [38]. Although both approach use real-time simulation and real driver realizing human-in-the-loop simulation, the loop does not consist ADAS.

For interaction, use of real controller or virtual (game-type) controller can be utilized within virtual reality environments. Although, when virtual reality environments are realized using a ead-mounted display, it normally uses dedicated controllers since the display is directly attached to the head, making the controllers physically non-visible. Therefore, again, CAVE type virtual reality environment has the advantage of capability of placing real or virtual controllers. Some CAVE type simulator, especially dedicated for driving simulator, have a full-scale real model of automobile or a driver seats are placed inside the screens. However, CAVE has many advantages as a virtual reality environment, it also has disadvantages of being a large system consuming both physical space and budget. Normally, trade-off decisions are made based on constraints and try to keep some minimum requirements based on the purpose of using the CAVE [24].

For realistic 3D environment, recent approach typically adapts use of game engines [22,23,24]. While building scenery of 3D environment can be relatively easy compared to making the full world from scratch, normally, the physical model of engine or calculation of the automobile is simplified [37]. In terms of realizing a realistic 3D environment for driving simulator, the precision of automobile model shall be virtually accurate as possible. Hence, this paper proposes to adopt a PLM software intended to test and design real automobiles. In particular, PreScan by Siemens which is a dedicated simulation software for ADAS and active safety was used to develop a realistic 3D environment [39]. PreScan uses Simulink, a well known component of Matlab to run the simulation model. Therefore, implementation of original software system and or hardware components are easily done through Simulink.

3.3 Considerations of Elder Drivers

There are many researches addressing the situations with elder drivers. Unless the individual lives in an urban city with sufficient public transportation, maintaining their mobility is important to support their quality of life [40]. However, it is clear that the spatial cognition ability declines with increase in age indicating that at some point the individual will become not capable to drive by themself. Furthermore, if there is no public transportation available, the loss of the ability to drive will also mean the connection loss of social activities, affecting the individuals’ quality of life. To solve this issue, self-driving cars or automobiles with highly advanced ADAS are expected to contribute. Although, unless the self-driving car can drive on its own completely, issues regarding the Take-Over-Request (TOR) need to consider about the drivers’ ability to drive upon the request. Therefore, understanding the characteristics with elder driver and their behaviours of interaction with the ADAS is important.

To understand the characteristics of elder driver, an approach of using a driving simulator is a typical approach since it is relatively safe. However, it is known that elders acceptance against the simulator is somewhat different with the young [41]. Early research about elder participation in virtual reality environments also argues with the decline of spatial cognition. Many clinical associated with clear evidence of decline in physical functions affecting negatively on general display recognition and 3D recognition [42,43,44]. It also has been reported with the sickness can occur even with simple driving simulators using only one CRT monitor [45, 46]. Although the are arguments that the driving performance can be unaffected [48], though some elderly feel extreme sickness which leads to dropout from the simulation [47]. Therefore, to minimize but ensuring the simulation environments, control of stereoscopic view and or fixing the head-tracking may be necessary in case of utilizing an immersive virtual environment.

4 Findings and Discussion

Based on the approach designed test ADAS interface for elder driver, the simulator to realize an immersive virtual reality environment was configured as follows. First, for designing the scenarios and use of simple virtual reality environment composed of three 65-inch LCD displays (Panasonic, TH-L65WT600), and a computer (Table 1). Using two GPU (Zotac, ZOTAC GAMING GeForce RTX 2080 Blower), the computer has full ability of rendering 60fps 4K for all three display using DisplayPort cables (DisplayPort 1.2a).

Table 1. Specification of the computer for simple virtual reality environment.
Full size table
Fig. 4.
figure 4

Scenery of the virtual world built using PreScan.

Full size image
Fig. 5.
figure 5

View from the top to see the whole virtual world built using PreScan.

Full size image

While using a proprietary function within PreScan called Remote Viewer plugin, viewports can be easily placed within a vehicle’s driver seat, or an actor sitting or standing in the virtual world (Fig. 4). While the GPU load relies heavely on the modeled world, a basic world with few cars was successfully rendered in real-time. The top view of the modeled world is shown in Fig. 5, whereas it also indicates how easily the objects are realized including the lightings and the shadows. However, as observable in Fig. 4, the stereoscopic view has been disabled in this picture. Furthermore, in case of CAVE type display environment for immersive virtual reality environment, visual port can be placed on human models based near human eye-point for both left eye and right eye. Theoretically, if the display is compatible with 3D view, for example, side-by-side, rendering left eye and right eye on one display may realize a 3D view though it will need additional hardware or software configuration.

Head-tracking can also be integrated through Matlab and Simulink configuration, where it can directly specify the head movements within the simulation loop. While it can be implemented easily for head-mounted display, in case of CAVe type projection, it will need to consider and will need adjustment calculations. The calculation itself is nothing new considering it is already implemented with Uni-CAVE [23] and KAVE [24]. However, this implementation is theoretically possible, it has not yet been tested which shall be done as future work.

5 Conclusion

Regarding the background of the aging society and the upcoming technology of self-driving cars, this paper proposed the use and process of immersive virtual reality environment to test the Advanced Driver Assistance Systems (ADAS). Following the HAVEit system architecture, this paper proposes a simple driving simulator which can easily be adopted to CAVE type immersive virtual reality environment. Moreover, consideration of elderly was taken in within the configuration of the virtual reality environment. Controlling the immersiveness is difficult based on the elderly’s acceptability against virtual reality environment, which will require an easy interface to activate and deactivate the stereoscopic view and head-tracking for comfort. Given the ease and the low requirement of setup for the simulation, extended use with CAVE type display environment, we expect that this paper address some solutions for approach in developing a safe and reliable ADAS system capable for elderly driver.