Design and Evaluation of an Augmented Reality Head-mounted Display Interface for Human Robot Teams Collaborating in Physically Shared Manufacturing Tasks

1.1 Article Evolution

Early, partial results of a smaller user study on our AR system were published in Reference [4]. This article provides the following key new contributions:

2.1 AR Robot Surrogates

One of the earliest applications of AR in robotics was to enable interaction and control of a real robot, through a virtual proxy of the robot visualized in AR. Chong et al. [5] created a robot guidance system enabling users to control a real robot through interaction with a virtual copy. Another system was created by Fang et al. [11]. They implemented their system using a monitor display, and users could program a real robot by moving a virtual robot rendered onto the monitor. In their study, it was found that the use of a monitor display reduced depth perception and distracted the user’s attention from the task. An AR system for controlling drones was created by Walker et al. [39]. Their system enabled robot control via virtual robot surrogates visualized in AR. They reported improved completion time and subjectively reduced stress levels with the use of their AR system.

2.2 AR as In Situ Displays

Another group of literature focuses on the use of AR for displaying robot information to the user. Andersen et al. constructed a system for visualizing task and robot information using projection AR for a vehicle door assembling task [2]. Ro et al. equipped their robot with the capability of projecting arrows onto the floor for guiding users to their destination [31], while Reardon et al. created a similar guidance system using a head-mounted AR device [30]. Lim et al. presented a system for mini-car driving, combining the use of both a smart phone and a projector to provide an enhanced user-immersion experience [23]. Kemmoku and Komuro considered applications requiring a large effective area of display, and they built an AR display system using a projector mounted onto the user’s head to achieve this [19]. Projection-based AR systems, however, suffer from occlusions by objects in the environment or by the user themselves, rendering them unsuitable for human-robot collaboration scenarios. To avoid occlusion issues, Hanson et al. built an AR system using a head-mounted device [15]. Their system targeted an assembly task and was used to display assembling instructions to the user. They reported that their AR interfaces increased task efficiency and accuracy; however, there were no robots involved in this study. Rosen et al. created a system that uses AR displays for bidirectional human-robot communication [33]. Their application focused on communication and did not involve collaboration in shared space. Walker et al. created a system using a head-mounted display to show a drone’s future path/waypoints. They compared different types of visualization in a task involving human and robot sharing the same workspace, but not the same goal [38]. In summary, studies in this category focus on the use of AR as a communication tool for providing information to the user. However, they do not involve shared-space interaction or collaboration between human and robot partners working together toward a shared goal.

2.3 AR for Teleoperation and Programming

The majority of literature on AR systems for human-robot interaction have used AR as a robot teleoperation or programming interface. Rosen et al. presented an AR system that allows users to set robot arm goal poses and provides robot motion preview [34]. Ni et al. presented a system that uses AR and a haptic device for a welding task application [27]. Their system enabled users to specify a remote robot’s welding trajectory through a monitor displaying the remote scene and an overlaid virtual robot. Gregory et al. presented a system that utilizes a head-mounted AR display along with a gesture glove that allows the user to command a mobile robot [14]. The robot was equipped with a certain level of autonomy for reconnaissance missions. Stadler et al. analyzed the workload on industrial robot programmers as they controlled a robot with a tablet-based AR interface [37]. They reported a decrease in mental demand, but increase in completion time. Their study involved a miniature robot (Sphero). Quintero et al. also created an AR system that allows user to program, preview, and edit robot trajectories [29]. They tested their system with a table-top robotic arm and instead they found that their AR system resulted in increased mental demand but reduced robot teaching time. Similarly, Ong et al. also found that an AR-based interface allows users to more quickly and intuitively program a table-top robot for welding and pick and place tasks [28], while Frank et al. found that using a tablet-based AR system to instruct a table-top robot for a pick and place task yields more efficient performances [12]. The variations in results reported from these studies suggest that robot size and task type can affect the performance of AR interfaces.

Although the benefits of AR for improving task efficiency in various robotic applications have been demonstrated, most studies have been limited to robot teleoperation, control, or programming tasks, where users are restricted to interaction with AR objects in virtual space, while the robots operate separately in the physical workspace. Furthermore, existing studies have been largely limited to table-top-sized robots and low-physical-demand tasks. There have not yet been studies focusing on human-robot teaming applications requiring both human and robot partner to collaborate in the same shared workspace, working on the same physical workpiece simultaneously. Further, there have not been studies focusing on larger-scale robots or high-physical-demand tasks. Industry 5.0 focuses on human-machine cooperation, and there are numerous industrial applications where human-robot collaboration in a shared workspace is of particular interest [13, 42]. Hence, towards enabling effective human-robot teams in CFRP manufacturing, we conducted a user study to investigate and evaluate the performance of an AR interface we developed for human-robot teams, working collaboratively on a high-physical-demand shared task with a shared workpiece and shared workspace.

4.1 Robot Platform

Our robot platform is shown in Figure 1(c). We constructed a robotic test bench with a two-axes movable platform. The test bench measures approximately 1.8 m \( \times \) 1.7 m \( \times \) 1.9 m. A KUKA IIWA LBR14 robot arm is mounted on the movable platform, and below the robot arm is the workpiece. The KUKA IIWA has impedance control capabilities, providing a suitable platform for performing tasks requiring physical contact with the environment and safe interaction with humans. The two-axis movable platform serves to move and position the robot arm around the workspace, extending the robot arm’s reachable range, such that it can reach all parts of the workpiece. At our partner DLR’s factory, the KUKA IIWA will be mounted on their large ceiling-hanging robots (Figure 1(b)), which will serve to move and position the KUKA IIWA arm. In our user study, we attached a 3D printed spring-loaded mechanism onto the wrist of the KUKA IIWA arm for mounting the tool (a marker pen). The spring-loaded mechanism serves to provide passive compliance for safe contact with the workpiece.

4.2 AR Human Robot Collaboration Interface

Figure 2 shows the block diagram of our AR human-robot collaboration system. Our system consists of three main components. A Microsoft HoloLens AR head-mounted display [21], a robot control computer, and communication library ROS Bridge [6]. The HoloLens is used to create an immersive, intuitive user interface. Unity, along with the programming library ROS# [32] is used to develop the AR interface. The Hololens renders 3D AR displays and user interface elements, co-located with the real robot and workspace, for user interaction. It permits user inputs including speech, arm gesture, and gaze. During operation, our AR systems renders a geometrically accurate robot and workpiece model to show to the user. Different models can be loaded, depending on the current robot partner and workpiece. A fiducial marker (ARTag) placed at a known position relative to the robot and the workpiece is used for calibrating the positions of the virtual models with the real counterparts. The rendered virtual robot and workpiece models provide task context and visual feedback of the calibration result to the user. Our system allows the user to specify the robot path, visualize the motion, and execute robot trajectories. The robot control computer processes user commands, completes the necessary trajectory planning and inverse kinematics, and sends low-level commands to the real robot for motion execution. The Robot Operating System (ROS) is used to implement these functionalities. The communication library ROS Bridge is used to transmit data between the Hololens and the robot control computer through a wireless connection and manages the necessary translation between different data formats.

Fig. 2.

View Figure

Figure 3 shows the workflow for using our AR system to specify and execute the robot’s motion. Unlike standard teach pendants that distract user attention away from the robot, our system facilitates human-robot interaction through interface components rendered in AR, co-located in the same physical space as the robot and the workspace (Figure 3(A)). A virtual model of the robot and the workpiece is rendered over the real robot and workpiece to provide a visual indication of the positional calibration result and task context (Figure 3(B)). At any time, if the user notices any positional mismatch (due to drift in HoloLen’s spatial tracking), they can look at the ARTag marker to recalibrate the system. The system infers the gaze location of the user by tracking a ray from the person’s head orientation. A ring marker is rendered on the workpiece at the intersecting point with this ray to indicate the gaze location of the user. The user can set trajectory way points at the marker location by using the speech command “set point.” A green sphere with a blue arrow indicating the surface normal is then rendered to indicate the set point location (Figure 3(C)). By repeating this process, the user can set successive trajectory way points on the workpiece (Figure 3(D)). At any time, the user can use the speech command “reset path” to clear any points that have been set. Once the user has finished setting all way points of the trajectory, they can then use the speech command “execute” to instruct the robot to execute the trajectory they have set (Figure 3(E)). During trajectory execution, the virtual model reflects the motion of the real robot. This provides a confirmation of the communication connectivity between the HoloLens and the control computer to the user.

Fig. 3.

View Figure

4.3 Joystick Trajectory Programming Interface

We also created a joystick interface for controlling and programming our robot, to serve as a proxy for the teach pendants used as the current industry standard interface [7]. We used a standard PS3 controller to implement the joystick interface. Using this interface, the user controls the robot end effector’s forward/backward, left/right motions by pushing the controller joystick up/down, left/right. A trajectory way point can be set at the current end effector location by pressing a set point button. A second button enables clearing of set way points. Pressing a third button commands the robot arm to execute the set trajectory. This robot programming modality provided by the joystick is analogous to the current programming methods supported by teach pendants.

5.1 Participants

We recruited participants for our user study through word-of-mouth, advertisements posted on the university campus, and social media. A total of 26 participants (19 male, 7 female) participated in our user study. Prior to conducting the study, we obtained research ethics approval from the University of British Columbia Behavioural Research Ethics Board (application ID H10-00503). We obtained informed consent from each participant before commencing each experiment session.

5.2 Experiment Task

In our experiment, participants carried out a physical collaborative task, simulating CFRP manufacturing procedure.¹ In a CFRP manufacturing task, the pleating procedure involves two main tasks. The first task is to start forming pleats around the edges of the mould by gathering and neatly folding excess membrane material. The second task is to move and align the pleats along stringers that extend from the edges of the mould to the center. The edge tasks require more dexterity, while the center tasks are more physically demanding. As the moulds we considered can reach 4 m in diameter and be concave, for the workers to reach the center parts of the mould, they need to set up scaffolds and climb over them to reach the center parts (Figure 1(a)). To alleviate the labor demand and risk associated with the task, in the envisioned human-robot collaboration scenario, it is intended that a human worker would perform the edge tasks, while the robot assistant would perform the center tasks.

For our experimental task, to simulate the physically demanding nature of the task, we used a mould that is 1.6 m in diameter and 0.6 m in depth. The robot setup and the mould used in our experiment is shown in Figure 1(c). This necessitated participants to move around the mould to perform all the edge task and set up scaffolds to reach the center parts of the mould. As the pleating procedure is a complex task requiring expertise, for our experiment, we placed a flat whiteboard over the mould, and we asked participants to color pre-defined lines on the whiteboard instead. Edges tasks were simulated with coloring of zig-zag lines, while center tasks were simulated with coloring of lines running from the edges to the center. To ensure safety, participants were asked to set up a scaffold as a stepping stool for reaching/performing the center tasks, instead of climbing over the scaffold or the mould. We asked participants to perform a total of four sets of pleating paths (edge path + center path) in a predefined order in each experiment. In a real CFRP manufacturing task, the worker must assemble and insert a vacuum tube under the membrane material in preparation for the next step following the pleating procedure. Participants were asked to assemble the actual tube as the last step in the experiment task as well.

5.3 Experiment Conditions

We wanted to test whether AR-enabled human-robot teams for physically shared tasks would provide benefits over the current manual method, and we wanted to test if our AR interface would provide benefits over existing alternative interfaces. Furthermore, we wanted to test whether our AR interface provides an effective interaction method that can encourage human-robot teaming and promote the utilization of the robotic assistant partner. Hence, we designed the following five conditions for our experiment:

H: Human-only Condition. The participant performs the entire task by themselves, without the use of the robot assistant. This condition serves as a baseline comparison and represents the current manual pleating process for CFRP fabrication.

J1: Joystick Condition - Task Division Predefined. The participant performs the experiment task collaborating with the robot assistant using the joystick interface. The joystick interface is analogous to current standard teach pendants. This allows us to compare our AR interface with an alternative standard interface. Considering the target application scenario as mentioned in the previous subsection, the human is designated to take on the easier-to-reach edge tasks (half of the task) and use the robot for the harder-to-reach, physically demanding center tasks (the other half of the task).

AR1: AR Condition - Task Division Predefined. The participant performs the experiment task collaborating with the robot assistant using our AR interface. Again, considering the target application scenario, participants are to complete half of the tasks, the edge tasks, while using the robot assistant to complete the other half of the tasks, the center tasks.

J2: Joystick Condition - Task Division Unspecified. The participant performs the experiment task collaborating with the robot assistant using the joystick interface. However, participants are given the freedom to choose which tasks to perform themselves and which tasks to use the robot to perform. This freedom to decide when and how much to use the robot allows us to examine if the joystick interface affects robot utilization.

AR2: AR Condition - Task Division Unspecified. The participant performs the experiment task collaborating with the robot assistant using our AR interface. However, participants are given the freedom to choose which tasks to perform themselves and which tasks to use the robot assistant’s help to perform. This freedom to decide when and how much to use the robot allows us to examine if our AR interface affects robot utilization.

5.4 Procedure

At each experiment session, we first gave an overview of the pleating procedure and explained our experiment task to the participant. We explained to the participant how to use the joystick and the AR interfaces to control and program the robot. Then, we allowed the participant to try using the two interfaces. After the participant familiarized themselves with interfaces, we then proceeded to the experiment trials.

Each participant first performed the task in the H (Human Only) condition as a baseline. After that, the participant performed the task in the J1 and AR1 conditions. We counterbalanced the ordering of J1 and AR1 among participants to mitigate carryover effects. Finally, the participant performed the task in the J2 and AR2 conditions. We again counterbalanced the ordering of J2 and AR2 among participants to mitigate carryover effects. After performing the task in each condition, each participant was asked to complete a set of questionnaires (details in Section 7) to evaluate the system. After experiencing all conditions, we also asked the participant to provide any additional comments they have at the end of the experiment.

7.1 Objective Measures

Task Completion Time. We measured the total time, t, required by the user to complete the experiment task in each condition. This provides a measurement of task efficiency.

Robot Utilization. For the conditions where the participant was given freedom to choose how much to utilize the robot assistant (J2, AR2), we measured robot utilization by calculating the percentage of discrete paths (edge or center) executed by the robot among all eight paths to be executed by the human-robot team. This provides an indication of robot/system acceptance by the user.

We compared completion time among conditions using ANOVA. Post hoc analysis was carried out using t-tests with Holm-Bonferroni method. To compare robot utilization, we used t-tests to determine if robot utilization in J2 and AR2 has either increased or decreased significantly relative to J1 and AR1, when robot utilization was pre-defined at 50%. The alpha level was set to 0.05 for all analyses.

7.2 Subjective Measures

NASA Task Load Index (NASA-TLX). We used the NASA-TLX [16] to measure the subjective task load on participants in each condition. The NASA-TLX is a questionnaire composed of six questions asking the participant to rate their experienced task load in six different aspects on a 21-point scale.

System Usability Scale (SUS). To evaluate the usability of the system in each condition, we used the SUS [3]. The SUS includes 10 questions asking the participant to rate the system’s usability in different aspects on a 10-point scale. An overall score is then calculated. The benefits of using the SUS is that it is a well-established scale with benchmarks from studies in different types of user interfaces and applications, allowing us to obtain a general indication of system usability relative to other existing interfaces, not limited to AR or robot [35].

User Experience Questionnaire (UEQ). We also evaluated the user’s experience on using the system in each condition via the UEQ [22]. The UEQ contains 26 questions on a seven-point scale, evaluating the participant’s user experience on the systems from different aspects relating to six attributes, including attractiveness, dependability, and novelty. Existing studies on the use of AR for robots do not often measure novelty. However, as wearable AR systems have not yet become widespread, it is likely that there will be novelty effects with AR-based interfaces. Hence, it is important to evaluate these related aspects as well. Like the SUS, the UEQ is a well-established questionnaire with existing benchmarks, allowing comparison with other types of systems [36].

Following existing studies [8, 29, 37], we performed ANOVA on the subjective measures to identify significant differences among conditions. Post hoc analysis was conducted using paired t-tests with Holm-Bonferroni method. The alpha level was set to 0.05 for all analyses.

8.1 Objective Measures

Completion Time. Table 1 reports the measured completion time, t, in each condition, while Figure 4(a) shows the box plots. Overall, AR1 and AR2 were found to have the shortest completion times. ANOVA revealed that there were significant differences among the measured completion times (\( F(4, 125) = 24.6, p\lt 0.001 \)). Pairwise t-test indicated that completion times are significantly shorter in J1, J2, AR1, and AR2 when compared to H (\( t(25) = 6.15,p\lt 0.001 \); \( t(25) = 7.00,p\lt 0.001 \); \( t(25) = 10.2,p\lt 0.001 \); \( t(25) = 9.74,p\lt 0.001 \)). Furthermore, completion times in AR1, AR2 are also significantly shorter than that in J1, J2, respectively (\( t(25) = 8.10,p\lt 0.001 \); \( t(25) = 7.36,p\lt 0.001 \)), and completion times in J2, AR2 are significantly shorter than that in J1, AR1, respectively (\( t(25) = 5.14,p\lt 0.001 \); \( t(25) = 4.40,p\lt 0.001 \)).

Fig. 4.

View Figure

Robot Utilization. Figure 4(b) reports the measured robot utilization, R, in J2 and AR2. For J2, robot utilization ranged from 0% to 100%, and for 8 out of 26 participants, we measured a robot utilization of 50% or less (with 4 having a robot utilization =50% and 4 having a robot utilization <50%). With the joystick interface, a number of participants decided to utilize the robot assistant less, or even not at all. For AR2, robot utilization ranged from 50% to 100%, and for 24 out of 26 participants, we measured a robot utilization of >50%. This indicates that with the AR interface, all but two participants utilized the robot assistant more. A t-test revealed that, compared with J1 and AR1, where robot utilization was set at 50%, robot utilization overall increased in both J2 (\( R = 75.5\%\pm 33.6 \), \( t(25) = 11.4,p\lt 0.001 \)) and AR2 (\( R = 84.6\%\pm 19.5 \), \( t(25) = 22.2,p\lt 0.001 \)).

8.2 Subjective Measures

Perceived Task Load. Table 2 reports the NASA-TLX results, while Figure 5 shows the box plots. Overall, AR2 had the smallest physical demand, temporal demand, effort, and frustration scores, while AR1 had the largest performance score. H was found to have the smallest mental demand score. ANOVA indicated significant differences in physical demand (\( F(4, 125) = 16.74, p\lt 0.001 \)), temporal demand (\( F(4, 125) = 10.98, p\lt 0.001 \)), and effort (\( F(4, 125) = 2.55, p\lt 0.05 \)), but not in mental demand (\( F(4, 125) = 1.50, p = 0.206 \)), performance (\( F(4, 125) = 0.86, p = 0.493 \)), and frustration (\( F(4, 125) = 1.92, p = 0.112 \)). Post hoc analysis revealed that physical demand in J1, J2, AR1, and AR2 were significantly lower than that in H (\( t(25) = 6.94,p\lt 0.001 \); \( t(25) = 6.23,p\lt 0.001 \); \( t(25) = 6.50,p\lt 0.001 \); \( t(25) = 6.28,p\lt 0.001 \)). Temporal demand in J1, J2, AR1, and AR2 were also significantly lower than H (\( t(25) = 4.86,p\lt 0.001 \); \( t(25) = 4.79,p\lt 0.001 \); \( t(25) = 4.71,p\lt 0.001 \); \( t(25) = 6.84,p\lt 0.001 \)). Meanwhile, temporal demand in AR2 was significantly lower than that in AR1 (\( t(25) = 3.76,p\lt 0.001 \)). The effort in AR2 is significantly lower than that in H (\( t(25) = 3.47,p\lt 0.002 \)).

Fig. 5.

View Figure

System Usability. Table 3 shows the SUS score for the five tested conditions, while Figure 6 shows the box plot. Results showed that AR2 achieved the highest SUS score. ANOVA revealed that there were significant differences among the measured SUS score across the five conditions (\( F(4,129) = 3.586, p \le 0.01 \)). The pairwise t-tests show a significant difference between H and both J2 (\( p\le 0.001 \)) and AR2 (\( p\le 0.01 \)), as well as a significant difference between J1 and J2 (\( p\le 0.0001 \)). According to the global benchmark for SUS created by Sauro and Lewis through surveying 446 studies spanning different types of systems, the mean given score is \( 68 \pm 12.5 \) [35]. Comparing this global mean score from the benchmark with those obtained for our five conditions tested, we found that the SUS score for H and J1 were significantly lower than the global mean (\( t(25) = 3.785,p = 0.00086; t(25) = 3.067,p = 0.0051 \)), while the SUS score for the other conditions J2, AR1, AR2 had no significant difference from the global mean (\( t(25) = 0.158,p = 0.876;t(25) = 1.128,p = 0.270;t(25) = 0.441,p = 0.663 \)).

Fig. 6.

View Figure

User Experience. Table 4 shows the UEQ scores, while Figure 7 shows the box plots for the six measured aspects in the five tested conditions. Figure 8 shows comparisons of the UEQ scores with established benchmark [36]. (Reported UEQ scores are after data transform [22], and has a range of [\( - \)3, 3].) Overall, the AR interface achieved the highest score in Attractiveness and Novelty (AR2) and highest score in Efficiency and Stimulation (AR1). The joystick interface (J2) achieved the highest score in Dependability and Perspicuity. ANOVA indicated significant differences among the five conditions for Attractiveness (\( F(4,100) = 24.15 , p \le 0.0001 \)), Efficiency (\( F(4, 100) = 22.748 , p \le 0.0001 \)), Stimulation (\( F(4, 100) = 48.057, p\le 0.0001 \)), and Novelty (\( F(4, 100) = 73.996, p\le 0.0001 \)). Pairwise t-tests revealed that condition H had significantly lower scores compared to the other four conditions in terms of Attractiveness (\( p\le 0.0001 \)), Efficiency (\( p\le 0.0001 \)), Stimulation (\( p\le 0.0001 \)), and Novelty (\( p\le 0.0001 \)). Furthermore, comparison between J1 and AR1 showed a statistical difference in terms of Stimulation (\( p\le 0.05 \)) and Novelty (\( p\le 0.001 \)). Similarly, comparison between J2 and AR2 showed a statistical difference in terms of Novelty (\( p\le 0.001 \)). Also, comparison between J1 and J2 showed a statistical difference in terms of Dependability (\( p\le 0.05 \)). Compared against benchmarks, condition H received Bad ratings for all measures except perspicuity. Joystick conditions (J1, J2) received mostly ratings around Average (Above Average or Below Average) for the six measures. AR conditions (AR1, AR2) received Excellent ratings for stimulation and novelty, but received Below Average or Bad ratings for Dependability.

Fig. 7.

View Figure

Fig. 8.

View Figure

9.1 Benefits of a Robotic Assistant

In relation to our hypothesis H1 regarding benefits of a robotic assistant, we found that regardless of user interface, in conditions J1, J2, AR1, and AR2, the introduction of a robotic assistant reduced physical and temporal demand on the participant and improved task efficiency by reducing task time when compared to condition H. These results directly support H1. As current CFRP manufacturing pleating processes are still performed fully manually, these results are encouraging and they show promise that by introducing robot assistants and enabling human-robot teams in high physical-demand manufacturing tasks, we can potentially mitigate strain injuries and increase task efficiency.

9.2 Comparison of AR vs. Joystick Interface

In relation to our hypothesis H2 comparing our AR interface with a standard joystick interface, we found that when using our AR interface (AR1, AR2), the completion time was significantly shorter compared to when using the joystick interface (J1, J2, respectively). These results supported the first part of H2, where we hypothesized that task time would be reduced. These results confirmed findings from the literature [12, 28, 29, 37]. Inspecting the NASA-TLX results, we found that the use of our AR interface (AR2) yielded the lowest physical and mental demand. However, neither differences in physical or mental demand reached significance compared with joystick interfaces. Hence, our results did not support the second part of H2, which stated that task load would be lower with the use of AR. Comparing with existing studies, some have found that use of AR reduces mental load [37], while others found that use of AR increases mental load and decreases physical load [29]. The discrepancy between results from our current study and existing studies may be due to different task types being considered in each study. Hence, when implementing the use of AR for robotic applications, consideration for the task at hand may be required. Existing studies tend to focus on comparing the use of AR versus alternative interfaces, but not the use of AR for different tasks. Further investigation is needed to determine how different task types influence the effects of AR on human-robot collaboration outcome.

9.3 System Usability and User Experience

Regarding hypothesis H3 stating that AR improves system usability and user experience compared to joystick, we found that both interfaces improved SUS score when compared to human-only condition (H). Our AR interface (AR2) achieved highest SUS score, which was significantly higher than human-only condition (H). Considering the UEQ scores, the human condition received Bad ratings for all but one aspect, indicating that significant improvement is needed for current mode of manual operation. Comparing the joystick interface conditions (J1, J2), which received roughly Average ratings for all aspects, with the AR interface (AR1, AR2), we found that the AR interface has improved ratings (Excellent) for Stimulation and Novelty aspects, but worse rating (Below Average or Bad) for Dependability. Hence, we found only partial support for H3.

It is interesting to note that the SUS scores for conditions AR2 and J2, when task division was unspecified, with participants given freedom to decide how and when to use the system, were higher than the scores for conditions AR1 and J1, when task division was predetermined, with participants told which tasks to perform and which tasks to use the robot for. This may be an indication that freedom for user to choose how to use a given system is also important towards increasing system usability and perhaps more so than the interface itself. Conversation with our industry expert partner indeed confirms that it is important to allow the user freedom to choose how and when they like to use a robotic assistant, from a worker’s perspective.

9.4 AR Interface Promotes Robot Utilization

Regarding hypothesis H4 stating that robot utilization will increase with the use of AR, our results showed that with the use of our AR interface (AR2), robot utilization increased significantly. While the same is observed for J2, the increase in robot utilization in AR2 is larger than that in J2. These results provided support for H4.

A more detailed examination of results revealed that our AR interface encouraged all but two participants to increase their utilization of the robot (with the remaining two participants still utilizing the robot 50%). However, with a joystick interface, in J2, the minimum robot utilization decreased for 4 out of 26 (15%) participants, (including 2 participants with 0% robot utilization), despite the fact that a reduced task completion time was observed when comparing J1 to H. This meant that although utilization of robot assistant can improve task efficiency (J1 compared to H), given the joystick interface, participants may opt to not use the robot at all. This demonstrates that a beneficial system may be left abandoned by users if the interface design is inadequate. Indeed, many existing studies have pointed out the importance of designing robotics systems that are accepted and adopted by users, as it affects the continual use and further development of the technology [17, 20, 25]. Experiment results demonstrated that our AR interface can encourage the adoption and utilization of a robotic assistant technology.

9.5 Novelty Effects

With new technologies such as AR, there are potentially novelty effects. Existing studies on the use of AR often do not examine such effects. The UEQ results from our study, however, showed that there are novelty effects from the AR interface. Participants found the AR interface to be more novel and more stimulating. They also perceived the AR interface to be less dependable, despite increased task efficiency (reduced task time) and robot utilization. These results suggested that the novelty effects may have caused a mismatch between user perception and actual task outcome. As users become more familiarized with the AR interface, we expect this mismatch to eventually fade and task performance to further improve.

9.6 Participant Comments

From the participants’ feedback, we found that our AR system was generally the most preferred interface. Participants indicated that our AR interface as “easier,” “faster,” and “more convenient” to use, and they indicated that our AR interface was “the best one” compared to the fully manual or joystick alternatives. A few participants indicated that the use of the joystick interface was slower but provided better accuracy, as it was more difficult to position the way points using gaze with the AR interface, and there is sometimes misalignment between the virtual and real robot. One participant had trouble with the speech commands, as HoloLens’ standard speech recognition was not able to accurately recognize his spoken commands. The participant expressed that this affected how much he would have used the AR system by 70%; although, the participant still opted to utilized the robot for more than 50% of the task in the AR2 condition.

9.7 Limitations

To allow participants without expertise in CFRP fabrication to perform the experiment, we have substituted the real pleating task with a coloring task. While our experiment task simulates the key aspects of a real pleating task, comparison of our findings with literature have suggested that task type may influence human-robot interaction outcome. Hence, subsequent studies on the use of our AR system for real pleating tasks would be worthwhile. Our target task of CFRP fabrication intends to utilize large-scale factory robots. While we have conducted our study using a robot test bench setup that is much larger than typical table-top robots used in existing studies [28, 29, 37], the target robots at our industry partner’s facility is still of much larger scale. We intend on eventually testing our system with those robots. As our study results suggested that there are novelty effects with the the use of AR, longer-term studies, or studies involving longer training sessions, may help better understand their effects on task performance and user experience. With increased familiarity, we expect AR to be able to further increase task efficiency and robot utilization.

In our study design, we have a partially counter-balanced ordering of the test conditions. The H condition was always first, since we wanted participants to be able to first experience the analog to the current manual method of CFRP pleating. As a result, the performance increase observed in the subsequent conditions (J1, J2, AR1, AR2) may have been due to task familiarization. In future work, we would propose to ask participants to perform another H trial (H2) at the end of the experiment to measure the effect of familiarization. However, given the length of the experimental task, it was not desirable to conduct an additional H2 trial for all participants, as we were constrained to a one-hour session with participants to avoid fatigue. However, we were able to conduct an H2 trial for two of the participants to provide some insight. The resulting task completion times in H2 for these two participants were found to be shorter than that in H, but comparable to those in J1 and J2, and still longer than those in AR1 and AR2. This suggests that while task familiarity will improve task completion time, our AR system indeed provided benefit to these users.

9.8 Recommendations

Based on our study findings, we provided the following recommendations towards implementation of AR interfaces for human-robot teams:

• As AR is a fairly new technology, we recommend longer training or familiarization periods to reduce novelty effects, reduce negative perception on dependability, and maximize task performance gain.

• The system should permit user freedom in deciding how and when to utilize the robotic assistant, as it may otherwise negatively impact system usability.

• The effects of AR interfaces on task performance outcomes may depend on the actual task type. Hence, consideration for the target task type should be given when considering the application of AR. For example, in some studies, mental demand was found to increase with use of AR, while in some others it was found to decrease. Longer training and familiarization periods may potentially help reduce some of such negative effects.