Smile and the world will smile with you—The effects of a virtual agent‘s smile on users’ evaluation and behavior

https://doi.org/10.1016/j.ijhcs.2012.09.006Get rights and content

Abstract

Recent studies have demonstrated that people show social reactions when interacting with human-like virtual agents. For instance, human users behave in a socially desirable way, show increased cooperation or apply human-like communication. It has, however, so far not been tested whether users are prone to mimic the artificial agent’s behavior although this is a widely cited phenomenon of human–human communication that seems to be especially indicative of the sociality of the situation. We therefore conducted an experiment, in which we analyzed whether humans reciprocate an agent’s smile. In a between-subjects design, 104 participants conducted an 8-min small-talk conversation with an agent that either did not smile, showed occasional smiles, or displayed frequent smiles. Results show that although smiling did not have a distinct impact on the evaluation of the agent, the human interaction partners themselves smiled longer when the agent was smiling.

Highlights

► Results show that participants smile more when a virtual agent smiles more. ► Paper provides a rich theoretical background on social effects of embodied virtual agents. ► Whether mimicry can be responsible for the phenomena is discussed thoroughly.

Introduction

Embodied conversational agents are advocated as one future possibility to overcome the problems and pitfalls of human–technology interaction. Researchers have therefore developed and tested embodied agents in different application contexts, for example as pedagogical agents (Graesser et al., 1999, Lester et al., 2000, Moreno, 2001), as digital real estate agents (Cassell et al., 1999), as information kiosks (Cassell et al., 2002, Kopp et al., 2005, Kopp et al., 2004), as health counselors (Bickmore and Giorgino, 2006, Pelachaud et al., 2002) or as TV/VCR interfaces (Krämer et al., 2003a, Krämer et al., 2003b). So far, however, only a few of the systems adhere to all of the requirements posited by Cassell et al. (2000) for such agents, namely the ability to recognize and respond to verbal and nonverbal input, to generate verbal and nonverbal output, to deal with conversational functions (e.g., turn-taking), and to recognize the state of the conversation.

Although the systems are still in their infancy, the research field is already characterized by a comparatively large amount of evaluation studies. Although not unequivocal, the findings are mostly encouraging for the proponents of agents: in most laboratory studies, acceptance is demonstrated as being high (Dehn and van Mulken, 2000), and the efficiency of the agents in terms of being helpful has predominantly been proven (Cassell et al., 2000, Krämer, 2008). Most astonishingly, numerous studies yield social effects, demonstrating that humans’ reactions towards virtual agents are remarkably similar to those towards human interlocutors (Krämer, 2008, Nass and Moon, 2000, Sproull et al., 1996). It has been found that virtual, anthropomorphic agents interacting with users trigger impression management on the part of the human user (Krämer et al., 2003a, Krämer et al., 2003b, Sproull et al., 1996), foster cooperation (Parise et al., 1999) lead to social inhibition (Rickenberg and Reeves, 2000) or evoke communication behavior that is equivalent to that which would be expected in a human face-to-face conversation (Krämer, 2005, Kopp et al., 2005). However, it has not been studied so far to what extent the human interlocutor would reciprocate the behavior of the agent. Here, one common mechanism of reciprocal behavior in interactions has been termed behavioral mimicry (Van Baaren, 2003). This behavior is especially interesting for social effects within human–agent interaction as it has been shown to be especially indicative for the sociality of the interaction (Lakin and Chartrand, 2003). Moreover, studies on the reactions towards virtual agents usually do not focus on subtle nonverbal behavior on the part of the user (for exceptions, see Bickmore, 2002, Kaiser and Wehrle, 2005, Sidner et al., 2006). For the present study, we therefore opted to examine whether a human reciprocates the smiling shown by a virtual human. In a preliminary approach we tested whether further research in this area would be worthwhile by, in a first step, analyze whether varying amounts of smiling by a virtual agent would result in more smiling of the participants on an aggregated level. In a between-subjects design, we tested whether participants who conduct an 8-min small-talk conversation with an agent that either does not smile, engages in occasional smiles or displays frequent smiles will also smile more and evaluate the frequently-smiling agent more favorably.

In the following, we will present previous findings on social effects of virtual agents and discuss theoretical explanations. We will then give an overview of the phenomenon of reciprocation and mimicry, and the effects of smiling in particular, before describing our study in detail and discussing its results.

Early evaluation studies of conventional computers characterized by human-like attributes (Reeves and Nass, 1996, Nass et al., 1994) as well as with embodied conversational agents (Sproull et al., 1996, Takeuchi and Naito, 1995) showed that machines and agents are readily perceived as social entities: Minimal cues and conditions in terms of similarity with humans are sufficient to lead users to show behavior that would be expected in human–human interaction (for an overview, see Krämer, 2008). Besides the fact that virtual humans elicit attention just as real humans do (Dehn and van Mulken, 2000, Takeuchi and Naito, 1995), person perception was shown to be like that of real humans (Bente et al., 2001), cooperation and trust is fostered (Parise et al., 1999, Sproull et al., 1996, Milewski and Lewis, 1997), tasks are facilitated or inhibited by the “social” presence of a virtual agent (Rickenberg and Reeves, 2000), and socially desirable behavior is triggered (; Sproull et al., 1996). Blascovich et al. (2002) and Bailenson et al. (2003) demonstrated that interactions with artificial humans in virtual environments follow the rules of spatial use in human–human interaction: People approach a non-human-like figure more closely than a human-like virtual person, especially if the virtual person shows realistic behavior such as eye gaze. With regard to users’ communicative behavior, it has been demonstrated that the presence of an agent triggers behavior known from human to human dialogue: Users are inclined to interact more using natural language (instead of, for example, a remote control; Cassell and Bickmore, 2000, Krämer et al., 2003a, Krämer et al., 2003b, Krämer and Nitschke, 2002, Smith, 2000) and employ conversational politeness strategies such as small talk and etiquette (Kopp et al., 2005, Krämer, 2005).

Some of these results suggest that people’s reactions to embodied agents are automatic in the sense that they are largely unaware, efficient, uncontrollable (i.e., cannot be stopped) and unintentional (see Bargh’s (1994) definition of automaticity). In fact, Nass et al. (1997); see also Nass et al. (1994) propose that users automatically and unconsciously apply social rules to their interactions with computers—due to the fact that humans are inherently social. Nass and Moon (2000) term this phenomenon “ethopoeia” and refer to the theory by Langer (1989), who coined the term “mindless behavior” for behaviors observable in social situations in reaction to a group of contextual cues that trigger scripts and expectations. Nass and Moon (2000) thus expected and empirically demonstrated that people would consciously object to the suggestion that it is appropriate to react socially towards artificial agents. Nevertheless, on an unconscious level, these reactions are indeed shown. Similarly, Blascovich et al. (2002) state in their threshold model of social influence that virtual humans will exert social influence as long as they show behavioral realism. Although they assume that the virtual agent has to engage in realistic behavior (thus disagreeing with Nass and Moon (2000), who state that minimal social cues are sufficient), they ultimately expect subtle social influences. On the other hand, Kiesler and Sproull (1997) challenge these assumptions and put forward a more cautious explanation for these seemingly social reactions. They suggest that “demand characteristics” of the situation are causative and that people merely act “as though”, without attributing humanness and really acting social. They assume that when confronted with an anthropomorphic virtual being, people conclude that they are expected to behave as if they were interacting with a human (for an overview on the debate, see Krämer, 2008). To the present date, the debate is still open as neither side was able to provide convincing empirical results that do not allow for alternative interpretations.

Additionally, it has to be noted that although there is a large number of studies that give evidence that people react socially towards artificial entities (see above), other studies do not yield results that are in line with the media equation approach (Reeves and Nass, 1996) or the so-called persona effect (Lester et al., 1997). For example, Chiasson and Gutwin (2005) replicated two of the studies that were presented in the media equation research program and found neither for adults nor for children consistent evidence for the existence of media equation phenomena. Moreover, especially studies which directly compare behavior towards fellow humans versus computers show that the persistence of the media equation can be doubted: For example Bhatt et al. (2004) demonstrated that computer-tutors are treated very differently from human tutors. Also, Shechtman and Horowitz (2003) gave evidence in chat interactions that the behavior towards humans clearly differs from the behavior that people exert towards computers. It therefore still seems to be desirable and a valuable contribution to test whether social reactions towards artificial entities occur. Our approach, however, differs from the media equation approach (manipulating the extent to which a computer interacts human-like) and the direct comparison of humans and artificial entities. Instead, we manipulate the behavior of a virtual agent and observe the users’ reactions.

A smile is the shortest distance between two people”. This famous quote by the Danish pianist Victor Borge aptly describes the special function that a smile has in interpersonal interactions. Indeed, the smile has been described not only as a means to express emotions but first and foremost as an important communication “device” (Kraut and Johnston, 1979). For instance, a smile is seen as consent to start social contact (Eibl-Eibesfeld, 1997).

Most consistently, effects on impression formation in interpersonal encounters have been described. In general, smiling people are considered more attractive, sincere, and sociable as well as competent than those who do not smile (Reis et al., 1990). Moreover, people who smile are in general perceived in more positive terms and are rated as more friendly, honest, warm and humorous (Deutsch et al., 1987). More specifically, LaFrance and Hecht (1995) demonstrated that smiling delinquents received significantly less punishment than non-smiling delinquents—probably due to the fact that smiling leads to an evaluation as more trustworthy, sincere, and admirable. Additionally, unknown faces appear more familiar when smiling (Baudouin et al., 2000). However, smiling should not only be seen as influencing likeability in a unidirectional manner; a reciprocal relationship between likeability and smiling has been repeatedly demonstrated (see Kanning, 1999, Katz, 2001, Reis et al., 1990, Rosemann and Kerres, 1986).

Although these effects are pervasive in human–human interaction, the question of whether smiling will also affect the evaluation of an embodied agent remains open. Numerous studies have provided evidence that an agent’s nonverbal behavior matters and is a crucial variable influencing the agent’s effects on the user (Bailenson et al., 2003, Cowell and Stanney, 2003, Rickenberg and Reeves, 2000, Slater and Steed, 2002). Even subtle effects of nonverbal behavior are pervasive (Krämer, 2001), and Slater and Steed (2002) aptly state that “it is behavior that matters rather than visual appearance” (p. 168). An overview of the effects of an agent’s nonverbal behavior is given in Krämer et al. (2007). Smiling, however, has thus far not been systematically varied and tested in human–agent interaction.

When behavior of one interaction partner is reciprocated by the other, one potential mechanism is behavioral mimicry which has widely been described in social psychology. A broad definition of mimicry is given by Van Baaren (2003; p. 11): “mimicry is having one individual doing what another individual does”. Numerous studies have demonstrated that the phenomenon is pervasive in human interaction and that there is a bi-directional relationship between nonconscious mimicry and liking, rapport, or affiliation (Lakin and Chartrand, 2003). Various terms for the phenomenon have been put forth, which differ with regard to whether dynamic or static similarities are focused on and whether identical and/or mirror-image positions and movements are targeted: reciprocity und compensation (Argyle and Cook, 1976), mirroring (Bernieri and Rosenthal, 1991), conversational adaptation (Burgoon et al., 1993), simulation patterning (Cappella, 1991), synchrony (Condon and Ogston, 1966), congruence (Scheflen, 1964, Kendon, 1973), motor mimicry (Bavelas et al., 1988, Bavelas et al., 1986, Lipps, 1907) or accommodation (Giles, 1980, Giles et al., 1987, for a review see Manusov, 1995, Wallbott, 1995). Some of these phenomena explicitly only refer to behaviors that are direct reactions to the interaction partner’s behavior, for some it is sufficient to show that on an aggregated level (e.g., during the course of the interaction), the specific behavior of the receiver is mimicked. Recently, the automaticity of many of these phenomena has been stressed (Lakin et al., 2003, Van Baaren, 2003), i.e., they are seen as unaware, efficient, uncontrollable (i.e., cannot be stopped) and unintentional.

Interactional synchrony and mimicry are often assumed to be connected to rapport or positive evaluations of the interaction partner. However, this has not been proven consistently. Drawing on Tickle-Degnen and Rosenthal (1987), who theoretically linked interpersonal coordination, attentiveness and positivity to rapport, Bernieri and Rosenthal (1991) provide evidence that we coordinate our behavior to a greater degree when interacting with others whom we like. Empirically, interpersonal coordination was found to be an important predictor of self-reports of rapport (see also Bernieri et al., 1996, LaFrance, 1982, Scheflen, 1964). In sum, Hess et al. (1999) suggest that certain types of mimicry, in particular postural congruence, are markers of rapport during an interaction. Moreover, within recent approaches that conceptualize mimicry and in line with social cognitive assumptions of automaticity as non-conscious, passive, and unintentional (Chartrand and Bargh, 1999, Van Baaren et al., 2004, Van Baaren, 2003), it was demonstrated that mimicry facilitates the smoothness of interactions, increases liking between interaction partners, and fosters prosocial behavior. The latter findings indicate that, instead of rapport leading to mimicry, it may also be the other way around, i.e., that it is the synchronization that causes positive relations. However, so far, only correlations rather than causal relations have been established. Lakin et al. (2003) as well as Chartrand et al. (2005) assume a bidirectional relationship. Against this background, the function of mimicry is seen as fostering social contact: mimicry increases similarity and adaptation, which permit social bonding. Lakin et al. (2003) thus coin the term “chameleon effect” and fittingly speak of “social glue” (p. 145). In line with this, mimicry has long been described and demonstrated to be a means of communication instead of an expression of emotions (see a study by Bavelas et al., 1986 that was aptly named “I show how you feel”).

Chartrand et al. (2005) as well as van Baaren et al. (2004) suggest that the phenomenon is rooted in evolution and might be connected to mirror neurons. They see mirror neurons, which are activated both when the individual is conducting an action him/herself and when he/she is observing somebody else performing the action, as evidence for a direct perception-behavior link: “Perceiving a behavior […] leads to muscular responses that are the same as those displayed while performing that same behavior” (Chartrand et al., 2005, p. 354).

Empirically, it has been demonstrated that all kinds of different behaviors are reciprocated. In an experimental study, Chartrand and Bargh (1999) provided evidence that rubbing one’s face or shaking one’s foot is mimicked (see also Lakin and Chartrand, 2003, Van Baaren, 2003). In addition, the adoption of congruent arm, body and leg postures have been demonstrated (LaFrance and Broadbent, 1976, LaFrance and Ickes, 1981). Fewer studies have looked into mimicry of facial actions (Hess et al., 1999) such as smiling, tongue protruding, mouth opening and yawning (see Chartrand et al., 2005, for a summary). Likewise, the observation of an emotional face leads to congruent facial expressions in the observer’s face (e.g., Dimberg, 1990, Dimberg and Thunberg, 1998). The reactions occur very quickly (300–500 ms after stimulus exposure; Dimberg and Thunberg, 1998) and can even be detected after subliminal presentation of the facial stimuli (Dimberg et al., 2001). Most consistently, it has been shown that a smile is reciprocated in a very immediate way (Eibl-Eibesfeld, 1997). Chartrand and Bargh (1999) demonstrated that participants smiled more frequently (1.03 times per minute) when the investigator smiled compared to when he displayed a neutral face (0.36 times per minute; see van Baaren, 2003, for a replication). Additionally, it has been shown that facial mimicry might even occur in the form of micro-mimicry, i.e., when looking at facial expressions, muscles that are responsible for these movements are minimally activated (Chartrand et al., 2005; see Hess et al., 1998).

The question of whether reciprocation and mimicry effects also arise when humans interact with virtual embodied agents remains open. Generally speaking, mimicry has also been shown to occur when another person is presented via media (Bavelas et al., 1986, Hsee et al., 1990, Lakin and Chartrand, 2003, Neumann and Strack, 2000). With regard to interactive agents, however, mostly anecdotal evidence has been reported, suggesting that the behavior of an agent or robot is mimicked (Cañamero, 2002, Thórisson, 1996). Thórisson (1996), for example, reported that “in general, participants tended to mimic the agents’ behaviours: If the agent was rigid, they tended to stand still; if the agent was more animated, they tended to be animated” (p. 175). Moreover, in a controlled study, children adapted the acoustic–prosodic attributes of their speech to those of virtual interaction partners (Oviatt et al., 2004). However, in another controlled experimental study, mimicry neither occurred with regard to the eyebrow movements of the agent nor concerning the agent’s hand movements (Krämer et al., 2007). In the opposite direction, mimicry by agents was shown to affect humans: When the head movements of human participants were mimicked by the virtual interaction partner, the latter was not only more influential and persuasive but was also rated more favorably (Bailenson and Yee, 2005). Indeed, this effect persisted even when subjects were aware that the mimicker was a nonhuman, artificial agent. Furthermore, a virtual listening agent that encompassed mimicry of the human speaker was shown to increase speaker fluency and engagement (Gratch et al., 2006). Recent research on fundamental issues of mimicry further highlights the fact that virtual figures in general are capable of evoking mimicry. Various studies in which non-interactive virtual faces (developed by means of the Poser Software, http://poser.smithmicro.com/poser.html) were utilized to show that humans react to virtual stimuli in the same way as would be expected with human stimuli—especially when the stimuli are dynamic (Likowski et al., 2008, Weyers et al., 2009, Weyers et al., 2006).

Against this theoretical and empirical background, we put forth the following hypotheses:

H1: A virtual agent who smiles is evaluated more positively than a virtual agent who does not smile.

H2: A virtual agent who smiles evokes more smiling in a human interaction partner than an agent who does not smile.

Additionally, we also take gender differences into consideration. With regard to general smiling activity it is well known that women smile more than men do (Deutsch et al., 1987). In line with this, we expect a main effect of gender with regard to the smiling behavior of the participants:

H3: Women smile more than men when interacting with a virtual agent.

With regard to the more interesting question whether men and women will be affected differently by the virtual agent´s smiling activity no specific hypothesis can be derived from the literature. We do not know of any study which explicitly considered gender differences within a study on the reciprocation of smiling. Thus, it is an open question whether men and women will differ and whether men or female participants are more prone to react towards the agent with smiling depending on the conditions.

RQ1: Do men and women differ in their amount of smiling depending on the amount of smiling Max shows?

Section snippets

Design and procedure

In order to test the hypotheses and answer the research question we conducted a 3×2 between subjects experimental study in which we varied the smiling behavior of a virtual agent (no smiles, infrequent smiles, frequent smiles, see Section 2.3) and included gender of the participants as independent variables. As dependent variables we assessed the participants’ ratings of the agent (see Section 2.4.1) as well as the participants’ smiling behavior (see Section 2.4.2).

Participants were requested

Preparatory analyses

Before beginning the analyses, we carried out preparatory analyses. By means of a 5-point Likert scale (1=very frequently to 5=not at all), we had asked the participants whether Max had smiled and whether Max had laughed. Descriptive data including all conditions show that people gave average ratings with regard to smiling (M=3.08; SD=1.02). With regard to audible laughter, which Max is not able to produce with his synthetic voice, they were more certain that Max did not engage in that kind of

Discussion

The present study was conducted in order to analyze the effects of an agent’s smile on participants’ evaluations as well as their behavior. Against the background of numerous results on social effects of embodied agents, we expected that an agent’s smiling behavior would result in more smiling by a human interaction partner. As a prerequisite to test this, we conducted preparatory analyses in order to establish whether participants would be able to consciously notice whether the agent had

References (111)

  • J.N. Bailenson et al.

    Digital chameleons. Automatic assimilation of nonverbal gestures in immersive virtual environments

    Psychological Science

    (2005)
  • J.N. Bailenson et al.

    Interpersonal distance in immersive virtual environments

    Personality and Social Psychology Bulletin

    (2003)
  • J.A. Bargh

    The four horsemen of automaticity: awareness, intention, efficiency, and control in social cognition

  • J.B. Bavelas et al.

    Form and function in motor mimicry: topographic evidence that the primary function is communicative

    Human Communication Research

    (1988)
  • J.B. Bavelas et al.

    I show how you feel: motor mimicry as a communicative act

    Journal of Personality and Social Psychology

    (1986)
  • J. Baudouin et al.

    When the smile is a cue to familiarity

    Memory

    (2000)
  • Becker, C., Kopp, S., Wachsmuth, I., 2004. Simulating the emotion dynamics of a multimodal conversational agent. In:...
  • C. Becker-Asano et al.

    Why emotions should be integrated into conversational agents

  • G. Bente et al.

    Computer animated movement and person perception. methodological advances in nonverbal behavior research

    Journal of Nonverbal Behavior

    (2001)
  • F.J. Bernieri et al.

    Dyad rapport and the accuracy of its judgment across situations: a lens model analysis

    Journal of Personality and Social Psychology

    (1996)
  • F.J. Bernieri et al.

    Interpersonal coordination: behavioral matching and interactional synchrony

  • Bhatt, K., Evens, M. & Argamon, S., 2004. Hedged responses and expressions of affect in human/human and human/computer...
  • Bickmore, T., 2002. Towards the design of multimodal interfaces for handheld conversational characters. Proceedings of...
  • J. Blascovich et al.

    Immersive virtual environment technology as a methodological tool for social psychology

    Psychological Inquiry

    (2002)
  • J.K. Burgoon et al.

    Adaptation in dyadic interaction: defining and operationalizing patterns of reciprocity and compensation

    Communication Theory

    (1993)
  • Cañamero, L., 2002. Playing the emotion game with Feelix: What can a LEGO robot tell us about emotion?, in: Dautenhahn,...
  • J.N. Cappella

    Mutual adaptation and relativity of measurement

  • J. Cassell et al.

    External manifestations of trustworthiness in the interface

    Communications of the ACM

    (2000)
  • Cassell, J., Bickmore, T., Billinghurst, M., Campbell, L., Chang, K., Vilhjálmsson, H., Yan, H., 1999. Embodiment in...
  • J. Cassell et al.

    Human conversation as a system framework: designing embodied conversational agents

  • Cassell, J., Stocky, T., Bickmore, T., Gao, Y., Nakano, Y., Ryokai, K., Tversky, D., Vaucelle, C., Vilhjálmsson, H.,...
  • T.L. Chartrand et al.

    The chameleon effect: the perception-behavior link and social interaction

    Journal of Personality and Social Psychology

    (1999)
  • T.L. Chartrand et al.

    Beyond the perception-behavior link: the ubiquitous utility and motivational moderators of nonconscious mimicry

  • Chiasson, S. & Gutwin, C., 2005. Testting the media equation with children. Proceedings of the SIGCHI Conference on...
  • W.S. Condon et al.

    Sound-film analysis of normal and pathological behavior patterns

    Journal of Nervous and Mental Disease

    (1966)
  • Cowell, A.J., Stanney, K.M., 2003. Embodiment and interaction guidelines for designing credible, trustworthy embodied...
  • Dahlbäck, N., Jönsson, A., Ahrenberg, L., 1993. Wizard of Oz studies—why and how. In Proceedings of the ACM...
  • F.M. Deutsch et al.

    What is in a smile?

    Psychology of Women Quarterly

    (1987)
  • U. Dimberg

    Facial electromyography and emotional responses

    Psychophysiology

    (1990)
  • U. Dimberg et al.

    Rapid facial reactions to different emotionally relevant stimuli

    Scandinavian Journal of Psychology

    (1998)
  • U. Dimberg et al.

    Unconscious facial reactions to emotional facial expressions

    Psychological Science

    (2001)
  • Eibl-Eibesfeld, I., 1997. Die Biologie des menschlichen Verhaltens: Grundriss der Humanethologie. Piper,...
  • P. Ekman et al.

    The Facial Action Coding System

    (1978)
  • S. Frey et al.

    A unified approach to the investigation of nonverbal and verbal behavior in communication research

  • Garson, G. David (2010). “Inter-rater reliability”, from Statnotes: Topics in Multivariate Analysis. Retrieved...
  • H. Giles

    Accommodation theory: some new directions

  • H. Giles et al.

    Speech accommodation theory: the first decade and beyond

  • Gratch, J., Okhmatovskaia, A., Lamothe, F., Marsella, S., Morales, M., van der Werf, R., Morency, L.P., 2006. Virtual...
  • U. Hess et al.

    Mimicry: facts and fiction

  • U. Hess et al.

    Facial reactions to emotional facial expressions: affect or cognition?

    Cognition and Emotion

    (1998)
  • Cited by (0)

    1

    Tel.: +49 521 106 12144; fax: +49 521 106 6560.

    2

    Tel.: +49 0761 203 8251.

    3

    Tel.: +49 203 3792919; fax: +49 203 3793670.

    View full text