FocusFlow: 3D Gaze-Depth Interaction in Virtual Reality Leveraging Active Visual Depth Manipulation

1 INTRODUCTION

Gaze interaction has gained popularity as an input method for 3D interaction in mixed reality (XR) headsets [2, 19, 38], where the eye movements and gestures would be used as an input. Current gaze-based XR interaction methods often rely on gaze as a means of pointing, supplemented by other inputs such as hand gestures [49], head movement [38], or dwell time [28, 33] for selection confirmation to avoid the so-called "Midas Touch" problem [42] – the problem of distinguishing intentional gaze inputs from involuntary fixations as the users perceive the scene. However, these works primarily utilize the direction of gaze and overlook the valuable visual depth information, representing an additional free input dimension along the z-axis (Figure 1). To see how, imagine a window through which you can see scenery outside. By bringing your gaze focus closer to the window, you blur out the distant scenery and are able to perceive dust particles on the window surface instead. Another example is autostereograms [41], where slightly adjusting your visual depth allows you to perceive a 3D image (see example in Appendix A). In both of these examples, users actively manipulate their visual depth to select which object to observe. In this research, we aim to leverage this natural eye behavior and convert it into an intuitive and learnable interaction input in Virtual Reality (VR).

Figure 1:

View Figure

More recent works on gaze-based VR/AR interaction demonstrate the great potential of visual depth as an interaction input to solve the Midas touch problem. These methods either guide the user to look at physical or virtual objects at different depths [3, 27, 37, 44, 45] or rely on voluntary eye convergence and divergence [13, 15] by asking the users focusing on the nose or imagining to fixate on some point behind the display plane. These works prove the feasibility of leveraging visual depth as an input for VR headset, however, the usability and learnability of this new interaction design are not fully explored. Failure-prone and difficult-to-learn experience can lead to user frustration and fatigue, making the interaction unusable. How to guide the users to learn to manipulate their visual depth as a reliable input is still a research gap.

In particular, leveraging visual depth as an interaction input is challenging due to several practical factors. First, the natural eye movement is inherently unstable, resulting in noisy visual depth estimates. Moreover, the eye tracking modules in the headsets suffer from limited frame rate and jitters, adding extra noise to depth estimates. So, a usable interaction mechanism is required that is robust against imprecise eye input signal. Second, user’s visual depth fixation naturally relies on the presence of objects. So, it may feel counter-intuitive if the user is asked to transition to a depth without any visual referent. So, it is crucial to ensure that novice users can intuitively and effortlessly learn and use this interaction and a carefully designed learning procedure is needed to guide the users on how to master this new interaction mechanism. To tackle these challenges, we employed an empirical research approach. Initially, we devised a visual depth detection method for off-the-shelf VR headsets and conducted experiments to investigate the properties of visual depth input. Building upon the insights gained from these experiments, we subsequently developed our proposed user interface and incorporated a learning procedure with visual cues to help users adapt to the depth-control process.

As such, we present FocusFlow, a novel gaze-depth interaction method that realizes the concept of Virtual Window (just like the examples mentioned above) by utilizing the 3D nature of immersive scenes and the innate human capability to shift focus across different visual planes. FocusFlow employs the eye trackers in VR headsets to estimate the user’s visual depth. It then proposes a layer-based user interface, where each layer acts as a transparent Virtual Window positioned at varying depths. These windows become interactive when a user’s visual depth aligns with their location, allowing for seamless interaction without visual clutter. Specifically, FocusFlow selects interaction targets based on the user’s gaze direction and activates Virtual Windows—initially hidden within the VR scene—through voluntary changes in visual depth (as illustrated in Figure 1). This mechanism facilitates a range of applications, from quick previews and system panel activation to detailed zoom lenses for accessibility, enhancing the VR experience without overwhelming the user interface.

To help users master this new interaction mechanism, we incorporate visual cues in the UI design and a learning process that allows users to establish a clear connection between the eye behavior and visual depth control, facilitating an intuitive and effortless interaction for novice users. The key intuition is that users can develop muscle memory for freely shifting their focus between different visual planes even if empty of objects. We propose two learning strategies, in-stages learning, and adaptive learning. The in-stages learning uses a series of strong to weak visual cues at a certain depth (e.g. where the transparent virtual window is located) so the user can practice visual depth control by just moving her gaze from the target object to the cue. While these cues are effective in practicing visual depth control, they also cause visual interference for the user during non-interaction times. The proposed adaptive visual cues addresses this issue by adaptively changing the transparency of visual cues as the user gets closer to the intended visual depth. The transparency level of the cue also serves as a feedback for the user to better control their visual depth changes. To help master this interaction, we design the adaptive learning process gradually reduces the guided depth range until reaching no visual cue.

To evaluate the usability and learnability of FocusFlow, we conducted quantitative and qualitative evaluations, considering input efficiency, accuracy, and cognitive load. Results from 24 participants indicate that FocusFlow offers an efficient, easy-to-learn experience for hands-free selection tasks, effectively addressing the Midas Touch problem while maintaining natural interaction. Furthermore, our findings suggest that the adaptive visual cues in our design enhance users’ depth perception during visual depth transitions. Additionally, the adaptive learning strategy improves users’ proficiency with our gaze-depth interaction method.

The contribution of this work comprise:

2 RELATED WORK

In this section, we first summarize different eye behaviors and the corresponding input techniques which can be used for gaze interactions. Based on this, we then review the evolution of gaze-based interaction in VR and some representative methods to identify the research gap. Lastly, we discussed design space research on user interfaces that integrate gaze input.

3 EXAMINING GAZE DEPTH CHARACTERISTICS

In order to design an intuitive and robust gaze-depth interaction, we first conducted a user study to explore the performance of built-in eye trackers in commodity VR headsets and visual depth characteristics.

Figure 2:

View Figure

3.1 Visual Depth Estimation

In order to calculate the user’s visual depth, we use the built-in eye trackers in VR headsets, which capture the positions of eyeballs and converts them into binocular gaze rays originating from eyes and intersecting at a specific depth (see Figure 2a). However, two rays in 3 dimensions may not intersect at a point specially with a small error in gaze direction estimations. To address this issue, we leverage the projection of these two rays onto a plane which ensures the intersection of the two rays. We define the projection plane along the two gaze ray origins (x-axes) and z-axes. As such, the distance between the eye line and the intersection point in this projection plane will be equal to the visual depth. To deal with random eye movements [26, 51] and the resulting depth jitters, we apply a moving average technique with 0.2 seconds of time window to filter out the high frequency depth fluctuations in the depth estimates.

Figure 3:

View Figure

3.2 Experiment Design For Pilot Study

For the apparatus, we use an HTC VIVE Pro Eye headset, which has an FOV of 100° and a 90Hz frame rate. The eye tracking data is recorded by a built-in eye tracker camera at a frame rate of 120Hz and 0.5-1.1° tracking precision. To evaluate the accuracy of visual depth estimates, we invited 7 participants from the campus. For each test, the participant is sitting on a chair while wearing the headset and follows the instructions on looking at certain targets.

We designed two sub-tasks in this experiment. In the first task, participants are asked to stare at a static VR target for five seconds located straight ahead at different distances from the user (0.5m, 1.0m, and 2.0m respectively). The target will be highlighted to capture participants’ attention. In the second task, they are asked to look at a moving target that moves back and forth toward the user at depths between 0.5m to 2.5m. The participants are allowed to blink and take a rest if they feel tired in both stages. Figure 3 shows all the tasks in the experiment.

Figure 4:

View Figure

4 FOCUSFLOW: SYSTEM OVERVIEW

This paper proposes FocusFlow, which consists of three main components to satisfy the design goals mentioned in the previous section:

5 FOCUSFLOW: LAYER-BASED UI DESIGN

Figure 5:

View Figure

Our vision perception undergoes a significant transformation when there is a change in visual depth. More specifically, a change in the perceived depth causes objects within the original field of view to appear out-of-focus and blurred since they are no longer aligned with our current focal plane. Our goal is to mimic the same intrinsic user experience in the virtual environments by aligning the in-focus objects with the user’s perceived depth. To achieve this, we propose a “layer-based” spatial user interface that organizes virtual content into distinct layers. We also propose an activation logic that defines the visibility index of each layer based on the user’s visual depth. These visibility indexes are dynamically adjusted based on shifts in the user’s visual depth. To create an immersive experience, the layer that is matched with user’s visual depth is visible with low transparency so that the user can perceive its position relative to further layers, while the closer layers are completely invisible. As such, we can enhance the user’s sense of depth perception. Figure 5 shows a high-level overview of the proposed layer-based UI.

Figure 6:

View Figure

5.1 Virtual Window: A Depth-based Selection Widget

Leveraging the proposed layer-based UI, we design Virtual Window, a hands-free selection widget with visual depth input. The high level idea is to replicate the user experience of seeing through a “window” or looking at the “window” itself in the virtual environment. Our vision system is capable of ignoring the presence of window and only focus on scenery behind it. At the same time, by bringing your visual focus closer to the window, you blur out the distant scenery and zoom into dust particles on the window surface instead.

Similarly, this virtual Window is an information layer, floating in front of the user, and it is invisible by default. The user can activate the window by bringing their gaze point closer. When the visual depth is matched with the window distance, the information on the Virtual Window will be activated, as shown in Figure 6. This transition process is highly intuitive, as it provides users a sense of “grabbing in” or “taking a closer look” at detailed information when they bring the visual depth closer. Similarly, to send the window to its default transparent status, the user only need to push their visual depth further, back to the portal layer. The window’s visibility change is a response indicating the activation is successful, and also naturally helps users to maintain the visual depth as they are looking at information on the window.

One challenge with this layer-based UI design is potential false triggers or no triggers due to gaze-depth estimation errors. To avoid false triggers, we employ our findings in the pilot study, described in Section 3, that showed a higher depth estimation accuracy at shorter distances. Therefore, we select the virtual window depth within the 2m range of the user at a distance that no virtual object is present. To avoid false negatives and no triggers, we define an activation zone, shown in Figure 6, which is a predefined spatial range around the actual virtual window depth. If the user’s visual depth falls inside this activation zone, the virtual window will be activated.

Figure 7:

View Figure

5.2 Visual Cues

While visual depth transition comes naturally to our vision system, it highly relies on the presence of physical/virtual content at that particular depth. So, it may be at first counter-intuitive for the users to perceive the depth and switch their gaze to a point in the space with no visual feature. Previous works on gaze-depth interaction address this issue by asking users to look at their nose to create vergence [3, 13]. However, this may results in user discomfort and fatigue. Instead, FocusFlow defines three types of visual cues that offer intuitive interaction with minimal visual interference.

The first design is a strong guidance (Figure 7 a), which is a static green circle in the center of the user’s view at the close depth. With this, users can transition their visual depth from far to nearby by directly looking at the circle. However, since the circle is always in the user’s field of view, it may be a strong visual interference for some users. The second design is a frame layout in the corners of the view, called weak guidance (Figure 7 b). Users can perceive the layer depth through their peripheral vision to transition their visual depth. As such, the strong guidance is a direct referent for the gaze point to focus, but the weak guidance is an indirect referent to the layer depth. The third design is an adaptive guidance (Figure 7 c), which is also a circle in the center, but its visibility actively changes based on the user’s visual depth. We map its opacity to a certain range of the user’s visual depth. When users are looking at distant objects, the circle is not visible so there is no visual interference. As users look closer, its opacity increases so users can find the circle and manipulate their gaze depth easier. In this adaptive process, the visual cue is not only a depth referent but also an indicator of the user’s visual depth, which could serve as a feedback to the user; from the opacity or transparency of the circle: the closer the visual depth, the more obvious the circle will be. This helps users understand the gaze-depth interaction mechanism and better perceive the depth information in virtual scenes.

Figure 8:

View Figure

6 FocusFlow: Interaction Design

Figure 8 illustrates the overall control logic of FocusFlow’s interaction design. The process is initiated with the eye tracker module in the headsets, which continuously monitors the user’s binocular gaze directions. From the gaze direction information, it can then compute the gaze depth to determine the exact point the user is looking at (as described in Section 3.1). The Eye Tracker also employs a ’moving average’ de-noising technique to refine the gaze data, ensuring that the data is smooth and free from random fluctuations that could lead to depth estimation errors.

The gaze data is then passed to the Interactive Object Selection module, which determines whether the user’s gaze is aligned with an interactive object in the environment. The module mainly searches for intersection of the gaze rays with individual objects in the virtual scene and moves forward to layer selection module if the an object is selected.

Upon selection of the interactive object, FocusFlow continuesly tracks the gaze depth estimates and evaluates whether the visual depth coincides with the activation zone, which is a predefined spatial area where user interaction is intended to occur. If the visual depth falls outside this zone, it is interpreted as a non-interactive state. However, if the gaze depth is within the zone, indicated by a "Yes" decision, it triggers the interaction signal.

Upon satisfying both the direction and depth conditions, the interaction signal is then conveyed to the Virtual Window Activation module, which is responsible for managing the active window elements on the user interface. It reacts to the interaction signal by activating or deactivating window elements. For instance, if the user’s gaze is directed at and focused on an interactive object within the activation zone, the corresponding virtual window element becomes active with full visibility. Conversely, if the user’s gaze shifts away, the virtual window will disappear and return to an ’Inactive’ state, as depicted in Figure 8.

Figure 9:

View Figure

6.1 Interaction Applications

The proposed layer-based UI and interaction offers a flexible design that can support different functionalities for the Virtual Windows. We implement and demonstrate FocusFlow functionality across several example applications:

Quick Preview. Given the expected short depth transition delays in our visual systems (i.e.160-200 milliseconds [16]), users can activate the Virtual Window very fast, making it suitable for quick information preview. As Figure 9a

Safe Activation. Compared to actions such as gaze direction change, blinking, and finger movement, wide range visual depth shift is a relatively low-frequency and low-randomness action. Therefore, we can apply the depth activation method to scenarios where we don’t want false touches to occur, which we call “safe activation”. For example, when browsing a web page or watching a video in full screen, we can activate a toolbar by shifting the visual depth. Since our visual depth is relatively stable, we can avoid false touches that could interfere with our viewing experience. (Figure 9b)

Zoom Lens. Visual depth is an input dimension along z-axis, where occlusion happens. This constraint actually offers logic connection between different layers, for example, the Virtual Window can also be used as zoom lens to magnify the object behind it. (Figure 9c) This design connects the information layer (Virtual Window) and portal layer (environment), i.e. providing a zoomed view.

Figure 10:

View Figure

7 FOCUSFLOW: INTERACTION LEARNING PROCESS

Section 5.2 defines several visual cues that could help the users to intuitively shift their gaze depth. However, we argue that depth change is a learnable input method and can become a muscle memory with a proper learning strategy without the need for strong or even any visual cues. To help users develop this new muscle memory, we propose two learning processes based on different series of visual cues: (1) in-stages learning (Figure 10-I), which walks the user through a sequence of strong to weak visual cues, gradually weakening the cue to remove the user reliance on visual cues for gaze depth transitions. (2) adaptive learning (Figure 10-II), which leverages the adaptive visual cue with different transparency indexes for the circular visual cue based on the user’s visual depth. We describe the technical details of these two learning strategies in the next sections.

In-stages Learning During in-stages learning process, the UI walks the user through a sequence of gaze-depth interactions with strong, then weak, and finally no visual cues. The learning process mainly relies on repetition and gradually weakening the cues in a plane dimension. In addition a fixed learning sequence is expected to be used for different users. This learning process is simple and efficient but lacks feedback to users on their depth control and perception quality.

Adaptive Learning The second learning strategy uses the adaptive guidance. At first, a depth range will be defined for activating the visual cue with different transparency indexes assigned to the visual cue based on the user’s visual depth. It should be noted that the visual cue’s depth range is much larger than the activation zone for virtual window activation. While the adaptive transparency of the visual cue serves as a feedback to the user, we further expand this feedback-based learning process by gradually reducing the depth range of the visual cue until we reach the depth at which the Virtual Window is located. In addition to feedback offering, this learning process can be flexibly modified for different users by changing the rate of visual cue’s depth range.

8 USER STUDY

In this section, we focus on two questions: First, can a gaze-depth input method be intuitively learned to the extent of developing a muscle memory? Second, how efficient is our method and what are the advantages compared with the non-depth hands-free methods? To prove the learnability and efficiency of FocusFlow, we conducted a user study with 24 participants, analyzed the effects of two learning strategies, and compared the performance with the baseline dwell-based selection method as detailed below.

8.1 Participants and Apparatus

We recruited two groups of participants from the campus with 12 people for each group (Group 1 and Group 2). All of these 24 individuals only have occasional or no VR experience. Out of these 24 individuals, 6 of them reported slight shortsightedness (3 in Group 1 and 3 in Group 2), while 18 of them had a standard uncorrected vision. All the participants did not wear glasses during the test. In addition, we invited 2 expert users (E1, E2) who had long-term participation in this project as a reference result. The expert users finished all the studies for Group 1 and Group 2. We used the same apparatus as in study 3.2.

Figure 11:

View Figure

Figure 12:

View Figure

Figure 13:

View Figure

8.3 Learning Strategy Analysis

In this section, we discuss the detailed performance of the two proposed two learning procedures and their learning capabilities by comparing the activation time, users’ gaze depth behavior, and the workload during the learning process. With better learning performance and user experience, the adaptive learning strategy is shown as a more effective way for learning gaze-depth interaction than the in-stages learning strategy.

8.3.1 Learning Procedure.

Participants in Group 1 are asked to go through the in-stages learning process. As Figure 12 shows, they first do 5 selections with strong guidance, followed by 10 selection attempts with weak guidance, and finally 3 attempts without any visual cues. On the other hand, participants in Group 2 are asked to go through the adaptive learning process. As shown in Figure 13, they perform 15 selection attempts using the adaptive visual cue. The depth range of adaptive guidance start from 5 meters, shrinks 1 meter every 3 attempts, and ends at 1 meter visual depth range which overlaps with the activation zone of Virtual Window. After this adaptive process, the users are asked to perform selections without any visual cue.

Figure 14:

View Figure

8.3.2 Activation Time.

We visualize the activation time of all the participant attempts from both groups in the learning stage in Figure 12 and 13, where the color intensity encodes the activation time. Both figures show a trend of decreasing activation time as learning progresses, demonstrated in lighter colors from left to right for each line/participant. We can also see that 7 participants from Group 1 cannot successfully activate the Virtual Window under no guidance setting, while all the participants from Group 2 can successfully do this. This indicates that learning strategy 2 is more capable of helping users form muscle memory during the learning process.

We further analyze the effectiveness of learning by comparing the activation time changes in a statistical way. The box plot in Figure 14 shows the distribution of activation times for each round of attempt within two groups. We can see that the activation time of Group 1 (Figure 14a) shows a decreasing trend in median activation time specifically under the strong and weak guidance setting, but this trend does not extend to the no guidance setting. In comparison, Group 2 (Figure 14b) shows a consistent decreasing trend under the adaptive guidance and no guidance setting, proving its superiority over learning strategy 1.

8.3.3 Gaze Depth Behavior.

We recorded the gaze depth data during the learning process to investigate the effect of visual cues on the interaction behavior. Figure 15 shows the raw gaze depth values of two representative subjects (P8 and P26).

Figure 15:

View Figure

P8 adopted the in-stages learning strategy and experienced strong guidance, weak guidance, and no guidance activation successively. As Figure 15-A shows, the gaze depth transition of P8 is smooth under the strong guidance and weak guidance setting. However, when entering the no guidance setting from weak guidance (Figure 15-B), P8 made several attempts but can not effectively bring his gaze depth close enough to reach the activation threshold. The reason is it’s hard to adjust the gaze depth accurately without any depth referent or muscle memory. The in-stages learning strategy fails to help P8 to develop the muscle memory, so as the system switched to the no guidance setting and there was nothing to be a referent, P8 can not shift his gaze depth to the target distance.

The adaptive learning strategy shows a better effectiveness on the transition from adaptive guidance to no guidance. As Figure 15-C shows, P26 can still stably shift the gaze depth between close distance and far distance, proving the feasibility of forming a muscle memory. This is because the shrinking assisted range is continuously reduce users’ dependence on visual cues, and the changing opacity provides a clear feedback to users to help them understand their input status.

Figure 16:

View Figure

8.3.4 Workload.

We employed the NASA Task Load Index (NASA-TLX) to assess the cognitive and physical workload of two learning processes. The result in Figure 16 shows that the no guidance method imposes higher cognitive and physical demands on the users than the guided methods, indicating the muscle memory development requires some effort. In addition, the cognitive load scores of adaptive guidance is slightly higher than the weak guidance method, because its changing capacity brings more information to users to process. However, it also shows a lower performance load score (lower is better), which means eventually the adaptive guidance can offer better activation performance.

Figure 17:

View Figure

8.4 Performance Analysis

Figure 18:

View Figure

The analysis in Section 8.3 revealed that the adaptive learning strategy enhances participants’ proficiency with the gaze-depth interaction method in FocusFlow. Building on this insight, this section delves into the performance of Group 2 during their test stages, comparing it to the baseline dwell method to assess the efficiency and accuracy of FocusFlow.

8.4.1 Activation Time.

We define ’activation time’ as the duration from the moment a user’s gaze targets an object to when the virtual window appears. Figure 17a 12 for the in-stage learning strategy. Additionally, their activation times in the no-guidance test are comparable to the time in the adaptive-guidance test, suggesting that Group 2 participants have effectively mastered the gaze-depth interaction method without any visual guidance. Moreover, we also compare the activation time between the gaze-depth method (FocusFlow) and the baseline dwell method in Figure 18 a. We could find that the two settings in FocusFlow achieve a similar activation time of around 1.3s, which is shorter than the 2-second dwell method, while the half-second dwell method achieved the shortest time.

8.4.2 Failure Rate.

We define ’failure rate’ as the ratio of successful selection attempts to the total number of attempts in each stage, considering an attempt a failure if it’s not completed within the time limit. Figure 18 b reveals that all four settings exhibit a failure rate below 5%, which suggests that the reliability of FocusFlow is comparable to that of the traditional dwell method.

8.4.3 False Trigger Rate.

We define the ’false trigger rate’ as the proportion of incorrect selections relative to the total number of selections, with ’incorrect selections’ being those where the wrong targets are chosen. As illustrated in Figure 18 c, the half-second dwell method exhibits the highest false trigger rate. This may be attributed to the method’s overly sensitive nature, leading to an increased likelihood of unintentional target activations. In contrast, both settings in FocusFlow demonstrate a considerably lower false trigger rate, approximately 5%. This rate is marginally higher than that observed in the 2-second dwell method.

8.4.4 Overall Performance.

In the preceding paragraphs, we evaluated various metrics of FocusFlow, comparing them against two baseline dwell time settings as depicted in Figure 18. Overall, FocusFlow demonstrates commendable performance with an average activation time of 1.3 seconds, a failure rate under 5%, and a relatively low false trigger rate of approximately 5%. While the half-second dwell method offers a shorter average activation time and a comparably low failure rate, its high incidence of false triggers could negatively impact user experience. Conversely, the 2-second dwell method, despite its stability and low rates of failure and false triggers, may lead to user frustration due to longer activation times, especially in certain scenarios. Additionally, the dwell method restricts users from continuously observing the target object, which might detract from the overall user experience. Thus, FocusFlow is a reliable and efficient interaction choice with an appropriate learning strategy applied to enhance the user experience in the VR space.

Figure 19:

View Figure

8.4.5 User Experience.

Besides the quantitative metrics we discussed above, we also designed a Likert questionnaire to evaluate the user experience of FocusFlow, focusing on its usability, efficiency, comfort, overall preference, and potential applicability in various VR contexts, and compared the results with the dwell-based baseline method. The result in Figure 19 shows that FocusFlow receives positive feedback from participants, which proves its usability and learnability. Compared to the baseline method, FocusFlow is overall better and obviously outperforms it in terms of efficiency (Q.4, Q.5, Q.6) and overall experience (Q.9, Q.10). It also shows that the eye fatigue when using FocusFlow is more serious than the baseline dwell-based method, which should be an issue that needs to be emphasized and improved upon. In addition, users are wary of using hands-free gaze interaction as an alternative to other VR interaction methods. This suggests that gaze-only interaction cannot cope with all VR interaction scenarios, so compatibility and extendibility with other interaction methods should also be an important design consideration.

9 CASE STUDY

Figure 20:

View Figure

FocusFlow facilitates intuitive and accurate hands-free selections in VR. To demonstrate how users interact with FocusFlow and their perceptions of the process, we present a case study set in a virtual fashion store shopping environment that mimics real-world scenarios. (Figure 20a)

In this environment, each product can trigger a preview window that showcases its profile. This is consistent with the Virtual Window setting encountered during user training. Items within the store vary in distance from the user, ranging between 2 to 7 meters. Users are encouraged to freely explore these products, mirroring a genuine shopping experience. The dwell method and our depth-based method are employed successively to ascertain the experiential difference conferred by the addition of a depth input dimension.

Utilizing FocusFlow, users reported a seamless and efficient experience. Initially, they would pan their heads around to acquaint themselves with the virtual space. Gradually, their attention would focus on specific items of interest. At this juncture, visual cues become evident (Figure 20b). “The appearance of adaptive visual cues hinted at the target depth and gave me some feedback on my gaze depth, allowing me to confidently engage in depth adjustments to activate the virtual window.”“I didn’t experience significant visual disruption since the depth of the adaptive visual cues didn’t match the depth of the object I was observing. Thus, I could easily overlook its presence when focused on an item.” (P19) To access detailed information about an item, users adjusted their gaze to the depth of the virtual window (Figure 20c). This action promptly displayed the product details. Subsequently, by shifting their focus, users could effortlessly continue to survey the environment. “The activation and deactivation mechanism was very fluid. It felt as though a virtual window was consistently present; and whenever I looked at it, information would display.” (P8)

However, during free exploration using the dwell method, users faced considerable challenges. When their gaze lingered on a product, the sudden appearance of the information panel – triggered by surpassing the observation time threshold – obstructed their view. “The most significant flaw of this method is that I can’t freely gaze at an object indefinitely. I constantly have to be cautious to not stare too long, lest the window pops up, hindering my observation. I believe the freedom to observe the world undisturbed is crucial for user experience.” (P21) The lag between intention to inspect and the system’s response was also disruptive. “During that waiting period, even if it’s just a second, I felt anxious. It felt like wasted time since the system didn’t promptly capture my intent.” (P11) After inspecting a product, to exit the information panel, users had to move their gaze to its periphery, an action they deemed superfluous. “The dwell method required me to shift my gaze to the edge to exit. There was a continual need to oscillate between the center and periphery. In contrast, with the depth method, I merely had to naturally look past the translucent panel towards the distant environment to exit. Essentially, with the depth method, I wasn’t burdened with unnecessary actions since my goal was to resume observation, which in itself served as the exit mechanism.” (P9)

In summary, users welcomed the depth input dimension. “The depth method gave me a stronger sense of control. I could distinctly decide whether to activate a target.” (P6) “I preferred the depth method. Though time-based techniques are simpler, once I got the hang of the depth method, I noticed an improved experience.” (P8) However, they also highlighted that the depth method was more straining on the eyes after frequent use. “Compared to the dwell method, a downside of the depth approach is the fatigue it induces in my eyes due to the constant need for depth adjustments.” (P13) This feedback implies that the depth method might not be ideal for extended use in short durations, as continuous adjustments could tire the eyes.

10 LIMITATION AND FUTURE WORK

FocusFlow features a hands-free gaze-depth-based input method that is efficient and intuitive for selection tasks after a minimal learning process. We proposed a layer-based user interface design according to the visual depth estimation characteristics and learning strategies to help users learn this gaze-depth input method. Although our evaluation results demonstrate FocusFlow is superior to baseline in terms of operational efficiency and the effectiveness of the adaptive learning strategy in helping users master this gaze-depth input method, there are still some aspects that could be improved or extended in future works:

Statistical Analysis: The current paper does not include any major statistical significance analysis due to small participant population. In our future work, we plan to perform a long-term experiment with a larger user population to fully demonstrate the effectiveness and intuitiveness of the proposed approach.

Multi-modal Interaction Inputs: Visual depth is a unique input dimension with a marginal likelihood of clashing with other eye behaviors. However, it only offers limited interaction inputs such as selection or zoom-in. To improve the number of inputs, FocusFlow can be seamlessly integrated with other eye input methods, such as eyelid movements or blinks. Moreover, the visual depth modality can be harmoniously coupled with other VR interaction modes like hand and head inputs. For instance, a user might activate a virtual window through gaze-depth interaction and subsequently interact with components within the window using direct or indirect hand gestures [29]. Such compound interactions can be explored in detail in future research.

Gaze Depth as a Continuous Input: The evaluations in Section 3.3 show that high-accuracy depth estimations are limited to close distances. While this limits our accurate depth detection over long distances, we can still utilize depth data at close range for more fine-grained inputs. For example, we can expand our gaze-depth interaction from binary selection to continuous inputs by continuously tracking exact visual depth changes and mapping the depth to some operational values. This will enable various manipulations such as scrolling on a webpage [36] or rotating a 3D object [17] by looking at micro gaze-depth changes.

Virtual Window Depth Adaptability: To elevate user experience and reduce erroneous virtual window activation, the position of the interaction layer could be adapted to a target’s location. For example, when the target is a few meters away from the user, the interaction layer could be set to one meter in front of the user. However, if the target distance is less than one meter, the position of the interaction layer will be automatically set to half a meter from the user. However, such adaptability necessitates visual cues to inform users of the adjusted window position, ensuring seamless interactions.

Figure 21:

View Figure

11 CONCLUSION

We present FocusFlow, a novel gaze-depth interaction technique that leverages the potential of visual depth as an intuitive input dimension. Our approach introduces a layer-based user interface that resonates with the concept of a “Virtual Window”, offering an efficient and hands-free selection method. By doing so, we address not only the traditional Midas Touch problem but also enhance the overall user experience in virtual reality (VR) interactions.

Considering that visual depth is a new input dimension for VR users, we place emphasis on the learnability of our design. To evaluate this aspect, we conducted both a user study and a case study to observe how novice users adapt to this new interaction method. The results demonstrate that gaze-depth interaction can be learned smoothly with adaptive visual cues that facilitate users’ perception of depth and develop muscle memory.

We hope this study will inspire more researchers to refine theoretical frameworks related to gaze-depth interaction. For instance, future work could focus on designing more effective learning processes, assisting users’ depth perception during transition periods, and integrating inputs from other modalities. These efforts would provide valuable theoretical support for promoting widespread adoption of this intuitive interaction method.

ACKNOWLEDGMENTS

This research was partially funded by UIUC-Insper program. We thank Professor Andrew Kurauchi and Professor Luciano Soares for their guidance in eye-tracking characterization and Arnav Shah for designing and implementing the surgical training application.