Keywords

1 Introduction

An interest of Human-Robot Interaction (HRI) lies in the investigation of robots and their emotional abilities through an interaction with peers or colleagues [6]. Such proximate interaction includes factors (e.g., gaze, expressions, gestures, speed, distance) which are perceived to elicit affect in humans, and used to attribute emotional states to robots [11]. Robots have been found to be as engaging as humans [7]. Furthermore, engaging physical non-humanoid robot collaborators have been found to elicit emotional responses [11]. Evidence shows that emotions critically influence human decision-making and performance [20]. Humans use the mechanisms from Human-Human Interaction (HHI) to perceive robots as autonomous social agents [11]. These propositions motivate this investigation to take into consideration both HHI and HRI, in the investigation of the elicited physiological affect.

Various emotional models have been reported in the literature, while Russell [18] classified emotions through a combination of their independent components, arousal and valence. In his model, the level of excitement has been represented by arousal, while valence defined whether the current emotional state is positive or negative. Studies have found a strong correlation between electrocardiography (ECG) and physiological valence [24]. There have been multiple findings of a linear correlation between galvanic skin response (GSR) and physiological arousal [5]. Evidence suggests that people are sensitive to the proximate interaction factors of collaborating robots [13] (i.e., gestured motion and speed [9]), where they have been found to elicit emotions in their human partners [25]. Previous investigations employed gestured motion and speed of the collaborating robots, from a direct path at a fixed speed to a variable speed in gestured motions [25].

Many traditional games have been played in the physical world and require a tangible interaction, in contrast to a certain popularity of electronic games in the current research methods [23]. This study uses a traditional Tower of Hanoi (ToH) game which provides an easy measurement of performance through a sequential set of steps. Physical collaborators might support higher motivation and better performance in contrast to the traditional collaboration-based digital serious games [15], especially for the robot collaborators in serious games [14]. In contrast to the traditional collaboration-based digital serious games where one is playing together with a computer (a virtual entity), HRI-enhanced serious games present a physical entity eliciting diverse behaviors and stronger emotional responses in participants [14].

The purpose of this study is to investigate the decision performance of human participants, collaborating with robot partners in their proximate interaction on a shared serious game task. This paper attempts to investigate how a small subset of proximate interaction factors elicits physiological affect underlying such human-robot proximate collaboration, in an attempt to investigate how these influence performance on a serious game task. Moreover, it aims to understand the role of affect in decision–making performance by mapping the participants’ physiological responses towards the collaborating robot. Following this goal, this paper sets out to investigate the effects of the elicited physiological affect on proximate interaction, in an attempt to inform the design of robot collaborators in serious games and to optimize the decision performance on collaborative tasks.

2 Hypothesis

Following up on given propositions, this study extends the previous research [11] to take into consideration the human collaborator condition, taking into account the elicited physiological affect in response to the human and robot collaborators. Therefore, the following hypothesis is presented:

Hypothesis 1: The collaborator condition (i.e., human, robot) will affect the performance on the game task (H1).

This study manipulated the collaborator conditions by varying the proximate interaction factors eliciting physiological arousal, through speed and gestured motions. Following on the previous findings [8] that the conditions of proxemic collaborators would influence elicited physiological arousal in participants and therefore performance on a game task, the following hypothesis is presented:

Hypothesis 2: Elicited physiological arousal will be affected by the collaborator condition (H2a), which in turn will affect the performance on the game task (H2b).

While physical collaborators have been found to elicit a higher motivation on a task, which is correlated with positive emotions [10], previous studies have not found a strong correlation between physiological valence and performance [8]. To expand on these findings, the following hypothesis is postulated:

Hypothesis 3: Elicited physiological valence will be affected by the collaborator condition (H3a), which in turn will have no significant effect on the performance on the game task (H3b).

3 Methodology

3.1 Participants

This study included 70 participants, 58 were males and 12 females. The age of participants ranged between 19 and 31, with a mean of (\(23.60 \pm 2.34\)). Demographic data (i.e., familiarity with the ToH game task, board games in general, and solving mathematical problems) were collected and they were given a movie ticket as a reward for participating. Participants were students of Blekinge Institute of Technology. The Ethical Review Board in Lund (reference number 2012/737), Sweden, has approved all experiments conducted in this PsyIntEC ECHORD project (FP7-ICT-231143).

3.2 Experimental Setup

Fig. 1.
figure 1

Experiment conditions: (a) the Solo condition in which the participant is playing the game on his own; (b) playing with the Human Collaborator emulating the direct robot collaborator condition, with direct path at fixed speed; (c) the Direct Robot Collaborator condition, always moving in a similar fashion with a direct path at fixed speed; (d) Non-Direct Robot Collaborator condition which had one additional non-direct random point inserted in its path when performing the moves at varying speeds.

Full size image

A crossover study with controlled experiments has been conducted in a laboratory setting. The lighting and temperature conditions were controlled in such a way that artificial fixture light was used throughout the experiment while the temperature was held constant at 23 \(^{\circ }\)C ± 1 \(^{\circ }\)C. The participants were seated in a chair with a fixed height and a predefined position. The height and position were constant during the experiment. The two experimenters were always present in the laboratory room to monitor the experiments, but they were completely hidden behind the screen.

Study Stimuli. The ToH game (Fig. 2(d)) was used as the serious game for the study to investigate the elicited human physiological affect in collaborative HHI and HRI, and bring it in relation to the performance on a collaborative serious game task. Most of the participants were naive to the ToH serious game. The game was easy for the robot to handle since an optimal solution to the game exists, and it was a reasonable challenge for most of the participants. ToH was originally a single player game. In a collaborative gameplay, human-human or human-robot took turns to complete the game. The rules were explained in [17]. In short, the ToH is a mathematical game consisting of three rods and a number of disks of different sizes that can slide onto any rod. The goal of the game is to start from a given configuration of the disks on the leftmost peg and to arrive in a minimal number of moves at the same configuration on the rightmost peg [17]. In this study, the serious game started from a given configuration of the four disks, which was the same for all participants and referred to the beginning configuration in the game definition above. The individual trials consisted of moving any single disc to a next legal position, interchangeably between a participant and a collaborator until the final configuration of disks was reached on the opposite peg from the start. The participants always started first. At every move in the ToH game, the participants had an option to take a binary decision placing a disc on one of the two available legal pegs. This decision consisted of just one possible optimal step to move a disk towards the final configuration, and a non-optimal step which would be corrected by the collaborator, providing an immediate feedback on the outcome of a decision. The participants always had an option to take this optimal step as their next move, while it was mandatory for the collaborators. Therefore, only the participants had an option to take the non-optimal step to move a disk in any other legal position, which would not necessarily lead towards the final configuration.

The elicitation of physiological affect was achieved by the gestured motions and the speed of the collaborating robot. In particular, humans prefer that a robot moves at a speed slower than that of a walking human [9]. The gestured motions were composed from a direct path at fixed speed of 30 cm/s (30% of robot speed) between the two endpoints of a current disc movement, for the Direct Robot Collaborator, to a random path and speed between 5 cm/s up to 70 cm/s (5% to 70% of robot speed). A random path and speed were generated online for the Non-Direct Robot Collaborator. A non-direct path is generated using the two endpoints of a current disc movement in between which a random point in space above the disks was inserted, randomized on each game move robot arm makes, which totals in three virtual positions the robot arm has to follow while making its move. The robot was passing through all the specified positions before having arrived at a final disc movement position.

Main Manipulations. The main manipulations (see Fig. 1) were: (a) the Solo condition where no collaborators were present and in which a participant was playing the game on his own; (b) the Human Collaborator condition emulating the direct robot collaborator condition, with a direct path at fixed speed; (c) the Direct Robot Collaborator control condition, where the participants played together with the robot which was always moving in a similar fashion with a direct path at fixed speed; (d) the Non-Direct Robot Collaborator condition which had one additional non-direct random point inserted in its path moving at varying speeds. The experiment setup was identical between the trials and participants, where human participants played the turn-taking ToH game together with a robot or human collaborator, which allows for a pace participants feel most comfortable with. The goal of the game task was to move the disks from the starting to the final configuration. The collaborators were playing optimally on each move, following the algorithm. The experimenter has been trained to interact the same way with every participant according to a well–rehearsed procedure.

3.3 Experiment Procedure

The participants were allowed for six minutes of rest in total between the trials. As performed in the study from [8], the four conditions (Solo, Human Collaborator, Direct Robot Collaborator and Non-Direct Robot Collaborator) were presented to each participant. For each of the four conditions a participant repeated each condition three times one after the other (thus in all, a total of 12 ToH games were played per participant). Each experimental session took around 90 min to complete.

3.4 Data Collection

The physiological signals were acquired using Biosemi Active TwoFootnote 1 physiological data acquisition system and its accompanying ActiView 9 software. ECG was measured at the chest using two 16-mm Ag/AgCl spot electrodes in a three-lead unipolar modified chest configuration: the two active electrodes were placed on the right collarbone and the lowest rib on the left side, and the ground electrode was placed on the left earlobe. GSR was measured using surface electrodes attached to the palmar surface of the middle phalanges from the middle finger and the index finger of the non-dominant hand (to reduce mechanical pressure susceptibility).

3.5 Data Reduction and Analysis

The data reduction was performed using Ledalab software for GSR [3]. Furthermore, Kubios software [21] and the HRV ToolkitFootnote 2 were used for ECG. This data were compared across the condition differences (Solo, Human Collaborator, Direct Robot Collaborator and Non-Direct Robot Collaborator) and the individual differences for the same trials (comparing the responses across individual moves for each condition).

In continuous stimulus settings, the most common measures of GSR are skin conductance level (SCL) and skin conductance response (SCR), where their changes are thought to reflect general changes in autonomic arousal [5]. The authors stated that the SCR signal is suitable for assessing the intensity of single (phasic) emotions, but changes in the overall (tonic) level are rather inert, thus valid for the trials longer than two minutes, such as the overall session in this experiment. Changes in arousal within periods shorter than two minutes are not likely indicated using the SCL. This problem is particularly limiting in trials shorter than two minutes. When the SCL temporal precision is insufficient, the rapidly reacting phasic changes (NS-SCR) seem to indicate a more promising focus: their number during a given time period is a prominent phasic-based indicator of arousal [4], such as a collaborator condition trial in this experiment.

Heart rate variability (HRV) is highly correlated with emotions [16]. Two measures of HRV are the standard deviation of normal-to-normal heartbeat intervals in the time domain (SDNN) and the ratio of low and high frequency powers (LF/HF) [1]. Wang and Huang [22] stated that SDNN and LF/HF were employed as two dimensions in the physiological valence/arousal model, where evidence revealed that SDNN was a good physiological indicator of valence [12]. The total variance of HRV increased with the length of analyzed recordings [19]. Thus, in practice, it was inappropriate to compare the SDNN measures obtained from the recordings of different durations. However, the duration of recordings used to determine the SDNN values (and similarly the other HRV measures) were standardized to a minimum of 5 min recordings for the short-term. Generally, the SDNN levels for the participants with positive affect were found to be higher than for negative one [12]. For even shorter recordings, main spectral components of the LF/HF ratio were distinguished in a spectrum calculated for the short-term recordings from 2 to 5 min [2].

Fig. 2.
figure 2

Results and ToH serious game demonstration

Full size image

4 Results

The participants reported previous experience with robotics on a seven-point Likert scale, where 1 meant “no experience” and 7 meant “familiar experience” (\(\mu \) = 1.7 \(\sigma \) = 1.095 N = 70). The differences between the participants were analyzed based on the reported values and the experienced outliers were identified. Six participants from the experienced outliers group were removed from the analysis to exclude the effects of participants’ familiarity on the experience with the robot collaborators, which resulted in the 64 valid data samples. Moreover, 43 of 70 participants have not had any previous experience with the ToH.

4.1 Collaborator Conditions and Performance (H1)

There was no statistically significant difference in the total number of moves between the human and robot collaborator condition groups (F(1,569) = 3.705, p > .05), shown in Fig. 2(a), where a higher value reflects worse performance. However, a higher number of moves in the Solo condition in contrast to any of the collaborator conditions was expected since the participants were not expected to know the most optimal solution for the ToH game task. The difference was statistically significant between the groups (F(3,751) = 20.807, p < .001).

4.2 Physiological Arousal (H2)

The serious game used in this study was found to elicit the relative physiological arousal value (measured with SCL) of 765.209 muS (\(\sigma \) = 835.545 muS) where the overall arousal value was normalized against the baseline. Therefore, this result suggested that the serious game elicited a high physiological arousal overall. Furthermore, there was a statistically significant difference between the collaborator conditions (F(3,744 = 58.881, p < .001). A Tukey post-hoc test revealed that the physiological arousal indicator NS-SCR was statistically significantly higher for the Non-Direct Robot Collaborator (\(19.59 \pm 8.05\), p = .03) compared to the Direct Robot Collaborator (\(17.43 \pm 6.76\)) condition. Both robot collaborator conditions were statistically significantly higher (\(18.51 \pm 7.49\), p < .001) than the Solo condition (\(11.62 \pm 9.11\)) and the Human Collaborator condition (\(11.01 \pm 6.08\)), as shown in Fig. 2(b). From these results the participants seemed to elicit a higher physiological arousal in the Non-Direct Robot Collaborator condition.

The worse performing participants reported higher arousal values after each round, as the Pearson product-moment correlation was run to determine the relationship between the participant’s number of moves and their physiological arousal indicator NS-SCR. There was a strong, positive correlation between the number of moves per round and the physiological arousal indicator NS-SCR values, which was statistically significant (r = .179, N = 739, p < .001).

There was a significant interaction between the effects of the collaborator conditions and physiological arousal on the number of moves (F(2,556) = 8.902, p < .001). The simple main effects analysis showed that the collaborator conditions significantly affected the performance when physiological arousal was lower (p < .001), with better performance associated with both robot collaborators compared to the Human Collaborator. Between the robot collaborators, better performance was associated with the Non-Direct Robot Collaborator, after which comes the Direct Robot Collaborator one (p < .001).

4.3 Physiological Valence (H3)

The serious game task environment presented in this study was found to elicit the relative physiological valence score (measured with SDNN) of .624 s (\(\sigma \) = 1.07 s), where the overall physiological valence value was normalized against the baseline. Therefore, this result suggested that the serious game elicited a high (positive) physiological valence overall. Furthermore, there was a statistically significant difference between the collaborator conditions (F(3,732) = 3.575, p = .014). A Tukey post-hoc test revealed that the valence indicator LF/HF was statistically significantly higher for the Non-Direct Robot Collaborator (\(3.081 \pm 1.869\), p = .008) compared to the Human Collaborator (\(2.453 \pm 1.724\)). Both robot collaborator conditions were statistically significantly higher (\(2.967 \pm 1.841\), p = .007) than the Human Collaborator condition (\(2.453 \pm 1.724\)), as shown in Fig. 2(c). There were no statistically significant differences between the Solo and the Direct Robot Collaborator (p > .05), as shown in Fig. 2(c).

Overall in the experiment, the participants were found to perform the task equally well regardless of the physiological valence found, as no significant correlation was found between the valence indicator LF/HF and the number of moves (p > .05).

5 Discussion

Considering the H1, as the difference in the participants’ performance was not statistically significant between any of the collaborator conditions, indicating that the collaboration with robot partners might be as effective as collaboration with human ones. It is possible that the participants may have been highly focused on the game task since the collaboration with a physical entity eliciting diverse behaviors and strong emotional responses might have promoted a higher focus on the task [10, 14]. As the worst performance was found in the Solo condition, it is possible that the participants left to their own skills had more room for the non-optimal moves, since the help of the collaborators was not available.

The relevance of the previous claims is further supported through the H2a and the H2b, exploring the performance in regard to the elicited physiological arousal on the collaborative serious game task. The robot collaborators elicited a higher physiological arousal than the human ones, while the Non-Direct Robot Collaborator elicited a higher physiological arousal when compared to the Human Collaborator. The results indicate that people are sensitive to the robots’ proximate interaction factors regarding physiological arousal in the context of collaborative serious games, supported by the previous investigations on proximate interaction factors in the context of HRI [9, 13]. The results indicate that high physiological arousal is associated with worse performance in the context of collaborative serious games, supported by the previous investigations on the connection between arousal and performance on a task [10]. The authors state that the performance is positively correlated with physiological arousal up to the point when the level of arousal becomes too high and the performance decreases. Possibly there was a high physiological arousal elicited in this study, especially if we consider the “lower” physiological arousal group which was the only one that showed the statistically significant effect of collaborator conditions on the performance. The other “higher” physiological arousal group had no statistically significant effect on the performance. These findings indicate that serious games might elicit high physiological arousal which may have disrupting effects on the performance on the game task.

Regarding the H3a and the H3b, investigating on the performance in regard to the elicited physiological valence on a collaborative serious game task. The Non-Direct Robot Collaborator elicited higher (positive) physiological valence compared to the Human Collaborator, indicating as well that people are sensitive to robots’ proximate interaction factors regarding physiological valence in the context of collaborative serious games. As the participants performed equally well regardless of the elicited physiological valence, this may suggest that physiological arousal has a more profound effect on the performance than physiological valence in the context of collaborative serious games, as supported by the previous studies exploring characteristics of robot behavior in HRI [9, 13].

6 Conclusion

Overall, the collaborators in this study created a physiologically arousing, high (positive) physiologically valenced serious game environment. As a number of moves per trial in the serious game were consistent across all the collaborator conditions studied, these findings indicate that the participants’ performance on the serious game task is comparable between the human and robot collaborator conditions. Regarding autonomous robots, this study found evidence of higher physiological affect elicited (arousal and valence) in contrast to their human collaborator counterparts, while still indicated a comparable performance on the game task between them. These findings motivate the introduction of autonomous robots as partners in the context of collaborative serious games, where the same performance benefits may be achieved as with using human ones. The Non-Direct Robot Collaborator condition elicited a higher physiological arousal and a (positive) valence, compared to the Human Collaborator. Moreover, it elicited a higher physiological arousal than the Direct Robot Collaborator condition, indicating that the careful design of Direct Robot partners can leverage different social cues to elicit target physiological arousal in the context of collaborative serious games. Furthermore, such context my witness a more positive valence elicited when using Non-Direct Robot partners instead of human ones. This study found evidence that the better performance was associated with the robot collaborators compared to the human ones, only for the “lower” physiological arousal group which was the only one showing the statistically significant effect of the collaborator conditions on the performance. The current study supports the notion that understanding physiological affect underlying such collaborative HRI from the human perspective, it would be possible to design more personalized serious games with intelligent robots which act together with human partners eliciting relevant physiological affect. This may contribute to improving the quality of HRI informing the design of such collaborative serious games. On the other hand, one has to be careful when designing serious games which elicit high physiological arousal, as such high levels of physiological arousal may be correlated with lower performance [10]. In contrast, physiological valence may not have such a significant effect.

Future studies should investigate the recognition of participants’ emotions on–line using physiological measurements to adapt the robots’ behavior in a closed-loop social interaction. Moreover, such future study should consider further physiological changes in participants by allowing for a non-optimal solution to the game, which would increase participants’ autonomy and involve a more relaxed collaborator’s behavior insisted of an optimal one. Furthermore, the Human Collaborator condition might have been too restrictive with emulating the direct robot collaborator condition. While this allowed for comparison between the collaborator conditions, it may miss the elements of HHI collaborative play. Future studies should allow for a more natural HHI collaborative interaction.