1 Introduction

The emergence of consumer-friendly head mounted displays with high frame rates and wide angle, such as Oculus Rift, easily provides the general public with an immersive virtual reality experience. Thanks to the progress in video technology and the recent spread of video presentation equipment, we can watch stereoscopic movies and large-screen high-definition videos in our private living rooms much more easily than before. Other sensory information, such as tactile, olfactory, or vestibular information, could be added to further enhance the presence of these audiovisual contents [1, 2]. If sensory stimuli can be fully optimized, it is expected that a highly effective system can be developed with inexpensive, simple, and small equipment.

For contents such as driving games, it is very hard to create a highly realistic experience just with audio-visual high-definition technologies because the sense of body motion is strongly involved in the experience. Thus, we must consider building a new framework to present a body motion stimulus when we design contents related to body motion. In driving simulators, humans generally detect velocity information using visual cues, and acceleration and angular acceleration information using mechanical ones (vestibular and tactile sensations). The organs that sense acceleration can be stimulated by mechanical (e.g., motion chairs [3]), electrical (e.g., galvanic vestibular stimulation [4]), and thermal means (e.g., caloric tests). Electrical stimulation can be achieved with a more inexpensive configuration than other methods can. However, although it affects anteroposterior and lateral directions differently, there have been no reports that it affects the vertical direction. In addition, its effect is changed by the electrical impedance of the skin. In thermal stimulation, cold water is poured directly into the ear, which is not a suitable experimental stimulus with computer systems. Therefore, motion chairs have been the de facto standard to create a sensation of full body motion in vehicles.

A driving experience is generally created by vehicle velocity and body motion caused by the topography of the road. Much research has been reported that velocity perception can be modulated by visually induced self-motion illusions [5] or tactile motion information [6, 7]. On the other hand, the perception of the topography has not been well studied. Most conventional research has used motion chairs with six degrees of freedom to reproduce exact physical information, but such chairs tend to be expensive and large.

In this study, to move the user’s body and induce a shape perception, we choose a motion chair (Kawada Industries, Inc.; Joy Chair-R1) with two degrees of freedom in roll and pitch rotations; it does not move in the vertical direction. However, one study has shown that humans more strongly perceive the shape of an object during active touch from the force profile applied to the finger than from the position profile of the finger [8]. This implies that the shape perception could be induced by local changes in topographic information without vertical movement, although the shape perception by the finger and by the body will differ. To verify the hypothesis that shape perception could be induced by body tilt, we constructed an experimental system using the simple two-DOF motion chair to present body tilt with optical flow and conducted a user study to classify the perceived shape based on visual and vestibular cues.

2 User Study

2.1 Participants

Ten male participants, aged 19–33 years, participated in the experiments. Because it has been reported that women experience motion sickness more often than men [9], only male participants were recruited. They had no recollection of ever experiencing motion sickness. All participants had normal or corrected-to-normal vision. They had no known abnormalities of their vestibular and tactile sensory systems. Informed consent was obtained from the naive participants before the experiment started. Recruitment of the participants and experimental procedures were approved by the NTT Communication Science Laboratories Research Ethics Committee, and the procedures were conducted in accordance with the Declaration of Helsinki.

2.2 Apparatus

Visual stimuli were radial expansions of 700 random dots. The size of each dot was \(81.28 \times 10^{-3} \mathrm {m}\). Resolution was \(1024 \times 768\) pixels (XGA). A visual stimulus was presented by a projector on the floor (NEC; WT600J). We used a 100-in. screen. The distance between the participant and the screen was 1.72 m. Participants wore an earmuff (Peltor Optime II Ear Defenders; 3M, Minnesota, USA) to mask the sound of the motion chair.

The motion chair and the visual stimulus were controlled by different computers on a network with distributed processing. Stimulus presentation was controlled by Matlab (The Mathworks, Inc., Natick, MA, USA) using Cogent Graphics Toolbox developed by John Romaya at the LON, Wellcome Department of Imaging Neuroscience, and Psychophysics Toolbox [10]. Synchronization of the stimuli was performed over the network. Position control was adopted to drive the motion chair by applying a voltage proportional to the desired angle with a microprocessor (Microchip Inc.; PIC18F252) and a 10-bit D/A converter (MAXIM; MAX5141).

2.3 Procedure

The experimental task was to report whether the shape they ran over was a bump (convex upward), a hole (concave), or a flat surface (plane) by pressing keys of a numeric keyboard labeled ‘bump’, ‘hole’ and ‘flat’. No feedback was given during the experiment. Ratings of motion sickness on a seven-point scale (1: not at all, 4: neither agree nor disagree, 7: very much) were also collected. Three optical-flow conditions (Bump/Hole/Flat) \(\times \) 3 motion-chair conditions (Bump/Hole/Flat) \(\times \) 3 velocity conditions (20, 30, or 40 m/s) \(\times \) 10 trials (a total of 270 trials) were conducted. Subjects had 15-minute breaks after every 28 trials, but could rest at any time. A typical experiment lasted about three hours and thirty minutes.

In each trial, a stimulus combination of optical-flow and motion-chair conditions was presented in a random order. Participants were seated in the motion chair with their body secured with a belt. They were instructed to keep their heads on the headrest of the chair and not to move the neck. Figure 1 shows the experimental procedure. Participants were instructed to watch the fixation point on the screen during the trial. After five seconds, the stimuli were presented for 20 s.

Fig. 1.
figure 1

Experimental procedure.

Full size image
Fig. 2.
figure 2

Tilt of the motion chair when running over a bump (a), hole (b), and flat (c).

Full size image

The shape was expressed by titling the chair forwards and backwards as shown in Fig. 2, i.e., modifying the pitch rotation which corresponded to the tangential angle on the surface. We adopted the shape profile (\(y=f(x)\)) as a Gaussian surface (\(N(\mu , \sigma ^2)\)). The maximum of the tilt of the motion chair,

$$\begin{aligned} \theta _{max} = \tan ^{-1}(\frac{d}{dx}f(x)|_{x=\mu \pm \sigma }), \end{aligned}$$
(1)

was set to \(13.5^\circ \). After ten seconds from the start, the height was at the maximum (i.e., \(x=\mu \)). The translational velocity was calculated by \(v=dx/dt\). The slope of the shape was set as \(\sigma =1.1\), which was determined by the rotational velocity of the actuators in the motion chair.

3 Results and Discussion

3.1 Shape Classification

The average probabilities of classifying perceived shapes by visuo-vestibular stimulation are shown in Fig. 3. Each row of graphs is the condition of bump, hole, or flat of vestibular stimuli and each column of graphs is the probability of classifying the surface as bump, hole, or flat. Note that we merged the velocity conditions (20, 30, and 40 m/s) because there were not large difference across them.

We applied an arcsine transformation to the probabilities P as \(\phi =\arcsin (\sqrt{P})\) to meet the initial assumption of an analysis of variance (ANOVA) test. Then, a two-way repeated measures ANOVA was performed on \(\phi \). The result showed main effects of the motion-chair condition (F(2, 18) = 13.46, \(p{<}.001\), \(\eta ^2_p=.599\) for the bump stimuli; F(2, 18) = 12.05, \(p{<}.001\), \(\eta ^2_p=.572\) for the hole; F(2, 18) = 16.10, \(p{<}.001\), \(\eta ^2_p =.641\) for the flat) and of the optical-flow condition (F(2, 18) =9.28, \(p{<}.005\), \(\eta ^2_p=.508\) for bump; F(2, 18) =7.24, \(p{<}.005\), \(\eta ^2_p=.446\) for hole; F(2, 18) =17.48, \(p{<}.001\), \(\eta ^2_p=.660\) for flat), but the interaction between motion-chair and optical-flow conditions was not significant (\(p{>}.10\)), except for flat (F(4, 216) = 7.72, \(p{<}.001\), \(\eta ^2_p=.125\)). This would be because participants mostly classified the shape as ‘flat’ due to a lack of information for judging the shape in the condition where both conditions were flat.

The comparison of the effect sizes between conditions shows that vestibular stimulation (i.e., stimulus by the motion chair) affected shape perception greater than visual stimulation (i.e., stimulus by optical flow). This suggests that the tilt of the chair, \(13.5^\circ \), was large enough to judge the shape independent of visual stimuli since the threshold of tilt perception was \(2.2^\circ \) [11].

It has been argued that all motion sickness arises from either visual or vestibular rearrangements [12]. It would be possible that a sensory conflict caused by our experimental stimuli produces motion sickness, resulting in misjudgment of shape perception. However, subjective ratings of motion sickness from all subjects were not larger than 2, which means that the experimental stimuli did not generate motion sickness.

Fig. 3.
figure 3

Probabilities of classifying surfaces as bump, hole, or flat.

Full size image

3.2 Perceptual Model

Since we did not observe an interaction between motion-chair and optical-flow conditions, we built a linear model of multimodal integration from unimodal shape perception as

$$\begin{aligned} \phi _{12}= w_{1}\phi _{1}+w_{2}\phi _{2}+b \end{aligned}$$
(2)

where \(\phi _{i}\) is the angular transformation value of the response ratio, \(w_{i}\) is the weight of each factor of the modality (vision, \(i=1\); vestibular sense, \(i=2\); multimodal of vision and vestibular sense, \(i=12\)), and b is the intercept. Independent variables are the responses from visual stimuli of either a bump or hole under the flat motion-chair condition (\(\phi _{1}\)) and from motion-chair stimuli of either a bump or hole under the flat visual condition (\(\phi _{2}\)). Dependent variable \(\phi _{12}\) is the response from visual and motion-chair stimuli of either a bump or hole.

The result shows that the coefficients of determination were \(R^2 = 0.57\) for the shape perception of a bump and \(R^2 = 0.58\) for that of a hole and that the responses from visual and motion-chair stimuli were able to be explained with the linear regression (ps\(<.001\)). In both cases, the weight of vestibular sense was larger than that of vision (bump \(w_{1} = 0.45\), \(w_{2} = 0.68\); hole \(w_{1} = 0.43\), \(w_{2} = 0.64)\). This is in line with the effect sizes of the ANOVA we mentioned above. Figure 4 shows a scatter plot of values predicted by the obtained regression formula and actual values of the multisensory effect.

Fig. 4.
figure 4

Scatter plot of values predicted by the obtained regression formula and actual values of the multisensory effect.

Full size image

4 Conclusions and Future Work

In this paper, we reported a perceptual integration of visuo-vestibular stimulation to generate convex or concave perception of a topographic surface. We used a motion chair and optic flow on a wide screen and conducted a user study to assess the influence of congruence or incongruence of visual and vestibular shape stimuli on the perception of a topographic surface. Results show that the vestibular shape cue contributed to the shape perception more than the visual one. Finally, we built a perceptual model of sensory integration, and the results of a linear regression analysis showed that performance with visual unimodal and vestibular unimodal cues could account for that with visuo-vestibular multimodal cues. Future work includes conducting a further experiment with different parameters in an attempt to augment the effect of visual stimuli or weaken the effect of the vestibular sensory stimuli.