Nonverbal Cues in Human–Robot Interaction: A Communication Studies Perspective

4.1.1.1 Body type.

Early research on body type distinguishes between three typical forms: (1) endomorph ( short, round, and fat), (2) ectomorph (tall and thin), and (3) mesomorph (muscular and athletic) [36]. Peoples’ aesthetic preferences as well as expectations on a robot's capabilities based on a specific body type must be considered. A wealth of research exists on robot appearance and form factors, including body type; see Fink [37] for a review that remains relevant to this day. As Fink [33] found, robots that are more humanlike in shape tend to perform better in social interactions with people. But this can be a double-edged sword: If humanoid robots are placed in social situations but are not equipped with the behaviors necessary to meet human expectations of the humans that they represent, then user experience will suffer. Additionally, the heads and faces of robots will receive the most attention, so nonverbal codes involving these areas of the robot body may deserve the most attention. Another common theme is the relationship between body shape and perceived gender of the robot (e.g., References [38, 39]); notably, cultural context can affect interpretations of gender, as found in the cases of Wakamaru [40] and Pepper [41]. What does not exist is a link to communication studies. One exception is the work of Jacquemont and colleagues [42], who applied a communication studies perspective on robotic body types so as to match people's impressions of their temperament. Specifically, they created an explicitly endomorphic robot form, with many round and large components to its shape. However, to the best of our knowledge, results on the actual user experience of this robot have not been explored.

Applying established communication studies frameworks in HRI studies could enhance conceptual rigor and provide a way of doing systematic literature surveys on different body types across different robots, contexts, user groups, and so on. Non-human or proto-human (even alien) forms could be explored, with inspiration drawn from animated characters. For instance, Pokémon, the fictional pocket-holdable monster characters from the series of the same name, are perceived as cute and engaging despite being generally non-humanoid and even non-animaloid in body type.

4.1.1.2 Clothing.

Clothing or dress is another nonverbal code and especially important in creating a first impression. For people, clothing functions as comfort-protection and preserves modesty but also serves as cultural displays about the social class, gender, age, profession, or status in the hierarchy, to name a few. To our knowledge, research has not explored so far how clothing might change people's perception of a robot. However, we believe cloths could be beneficial especially for robots working in social contexts. Clothing can be quickly created and changed. As such, it may be one of the quickest and easiest nonverbal cues related to physical appearance to explore. Many robots, such as KASPAR [43], have removable clothing, but the use of clothing, varieties of clothing, and even manner of wear, such as topless versus bottomless, has not been well studied, despite the possibility of important effects according to human models. Future work can start to explore the possibility of designing clothing for robots, i.e., robot-centered robot clothing and robot fashion [44]. Furthermore, clothing can be an easy way to clarify the role of a specific robot in public spaces or its affiliation, similarly to how people wear a specific type of uniform or clothing that features the company logo across a variety of jobs.

4.1.1.3 Kinesics: Body Movement and Gestures.

Body movements and gestures are part of the kinesic category of nonverbal cues. These include all nonverbal cues that involve motion of certain parts of the body or the entire body [45], such as emblems, adaptors, body posture, and body language. Various taxonomies for gestures, as a particular type of body movement, have been described in the communication literature [46–48]. For example, a well-used framework by McNeill [46] categorizes gestures as iconic, metaphoric, or deictic. Gesture taxonomies are provisional working tools tailored to particular research interests [46]. Therefore, classification systems for gestures in human communication can only be partially applied to HRI. Additionally, some are limited to certain parts of the body, such as the hands and arms in the case of McNeill [46]. As such, we chose to anchor this work on the taxonomy of Ekman and Friesen [48], which offers the following categories: emblems, or gestures, that are a direct verbal translation; illustrators, or movements, associated with speech; regulators that regulate speech acts; and adaptors, or repetitive movements for self-satisfaction, based on the earlier work of Efron [47]. Efron's categorization system refers to the whole body and thus seems well suited as a basis for HRI.

4.1.1.4 Emblems.

Emblems are gestures that have a direct literal translation shared by the members of a social group. These are often actions or behaviors that have a certain meaning in certain contexts. Miming gestures, for instance, illustrate a certain action that typically is taken, such as opening a cupboard door or performing a dance move. As such, emblems are often culturally defined, varying over time and across generational cohorts [49]. In most cases, it is better to avoid using such cues in HRI, because receivers might have difficulty decoding them. Conversely, robots designed for specific cultural contexts may need to use emblems to be accepted and understood. Nevertheless, while some have recognized that emblems may be culturally sensitive, most have not considered this directly within the context of their own work. We now cover some examples. Cabibihan et al. [50] compared the performance of a person and a robot at performing iconic and emblematic gestures, such as hugging, carrying a baby, and nodding. However, these were not distinguished as iconic, emblematic, or another type of nonverbal code. Carter et al. [51] built emblematic gestures into the design of a robot that plays catch with a person (throwing up its arms or shrugging on a near miss and shaking its head back-and-forth if the ball was thrown out of range), whereas Gross et al. [52] explored the use of emblematic gestures in place of speech, e.g., using a thumbs up gesture for “great.” Zheng, Liu, and Meng [53] explored how people interpret 10 “common” emblematic gestures performed by the Nao robot (Aldebaran/SoftBank Robotics), including clapping and handshaking. We may be able to guess what culture(s) are relevant to the results, based on the location of the work conducted. However, this should be stated clearly. Indeed, we should not shy away from exploring emblems in HRI. For example, a robot able to demonstrate mastery of the emblems of a specific sub-culture might be more readily accepted by that group, with its demonstration facilitating its integration into the group or work-team. Therefore, we must be conscious of cultural rules and justify our design decisions with and for the cultural group involved. We recommend taking a human-centered design approach, where the needs, expectations, and desires of the particular cultural and/or generational group are mapped out and then applied to the robot. Participatory and co-design methodologies, as well as iterative testing with members of the cultural group, are examples of how to achieve this.

4.1.1.5 Illustrators.

Illustrators are gestures that provide a description of what is being related vocally, such as the size or shape of an object demonstrated by drawing the outline of the object (iconic gestures) or an indication of the speed of an object by way of fast or slow hand movements. Illustrators are also used to highlight important points in a speech; see Streeck [30] for examples. They may be the most common type of gestures used in human communication. Salem et al. [54] evaluated how well people perceive illustrators, such as the size and shape of a vase, performed by a Honda robot. Adding pointing gestures to a Nao robot delivering vocal directions improved its performance [55]. To ensure the interpretability of the illustrators used, human gestures can be analyzed in a specific context, and these can then be transferred into a robotic hand, as was done by Sheikholeslami et al. [56]. So far, much work in robotics has focused on developing robots that perceive human gestures (e.g., [57, 58]). More work is needed the other way around: whether people can perceive illustrative gestures performed by robots accurately and the degree to which such gestures are useful. Illustrators add redundancy to speech by encoding what is being said into gestures, thus providing additional cues of how to interpret a message. Work has also started on two-way interaction between people and robots enabled by other technologies, such as wearables (e.g., Reference [59]). This sets the stage for new forms of illustrator-based bidirectional communication.

4.1.1.6 Regulators.

Regulators are body movements and gestures that manage and direct a conversation. Regulators, such as head nods, eye contact, and hand gestures pointing to a specific communication partner (deictic gestures), are used to manage the conversation flow and turn-taking between speakers. Their absence can make conversations in larger groups especially difficult. Consider the case of Zoom meetings where several people try to speak at the same time. Regulators are especially necessary in HRI when more than one person interacts with a robot simultaneously or when more than one robot interacts with one or more people simultaneously. Some previous research has incorporated regulators into a robot's nonverbal cue set. For example, Sugiyama et al. [60] developed a “facilitation approach” to increasing the effectiveness of deictic pointing gestures with the Robovie humanoid robot. Regulators, such as pointing to direct the actions of a human partner, are especially effective in combination with speech and eye gaze [54], a finding also supported in collaborative tasks [61]. Other examples of regulators used in HRI are hesitation gestures in the form of the robot slightly withdrawing its hand when it and a person reach for the same object at the same time [62] or the robot nodding while maintaining eye contact during back-channeling (verbal interjections) and turn-taking scenarios in response to more or less extraverted people [63]. In general, regulators as gestures are easy to implement into a robot's gesture repertoire. The challenge is that the timing of such gestures requires a certain situational awareness. The first stages of implementation could start with a robot using regulators to signal the beginning or end of speech. More advanced systems should be able to manage the flow of the conversation by encouraging a person to respond to specific cues or even being able to encourage one person out of a larger group of people to join the discussion, making sure everybody contributes evenly. Thus, robots could become mediators in group discussions. They might even be more readily accepted as mediators because of their perceived neutrality to specific topics.

4.1.1.7 Adaptors.

Adaptors (sometimes “adapters”) are repetitive gestures and body motions that satisfy physiological and psychological needs. People use adaptors like touching their hair or bouncing their legs to manage stress. Thus, adaptors reveal information about a person's internal emotional state. Adaptors may be conscious or unconscious, controllable or uncontrollable. As such, they could be useful in HRI for improving humanlikeness and a sense of reciprocity, especially in terms of social emotions or emotional contagion. Very few examples on the use of adaptors in HRI exist. Vocal and body adaptors (an “umm” and touching the chin) have been explored in awkward conversation settings [64] or added to increase social appropriateness at the end of a conversation [65]. However, few studies have explored adaptors as part of a robot's regular gesture repertoire. We can imagine many possibilities that should not be technically difficult to implement given what has so far been achieved with other codes. We can work up to a range of small cues working in concert to produce a larger impression intended to reveal a robot's internal state, such as a nervous-looking robot that breathes, shifts side-to-side, clutches its fingers, cowers a little, blinks rapidly, and so on. Robots displaying adaptors could also communicate a certain degree of vulnerability and insecurity that might make them more relatable and approachable, especially in first-contact situations or with inexperienced users. A robot development kit made up of the smallest gestures and behaviors may be useful for crafting a range of adaptors.

4.1.1.8 Posture and Body Language.

Posture refers to a special position of the body or the way that a person holds their body. The four basic body postures are standing, sitting, squatting, and lying down [66]. There are many variations in postures, but a basic distinction can be made between dynamic postures (when the body is in motion, e.g., walking) and static postures (when the body is not moving, e.g., sleeping or standing). Position and posture can communicate the attitude and relationship status of a sender [67]. For example, Mehrabian [68] found that when communicators who were women adopted an open posture, they were interpreted as having a more positive attitude than when they adopted a closed posture. Dynamic postures are also a rich source to express emotions, e.g., shying away when anxious, raising one's arms in an outburst of happiness. Such “emotional body language” involves the whole body and coordinated movements that facilitate the interpretation of a person's emotions [69]. Using the body language of robots to express emotions appears to be an effective tool, as well. Robotic emotional body language can be interpreted similarly to that of humans, and people can correctly identify emotions displayed in key poses [70]. A review of studies on affective-expressive movements for robots by Karg et al. [71] revealed that work so far has mainly focused on a small set of movement types restricted to the upper half of the body. They suggest that more research on the relationship between movement type and style of expressiveness is needed, e.g., whether walking up and down, slow or fast, will result in completely different impressions of the robot.

Full body motion, like jumping up and down while waving your hands toward a friend who is trying to find you in a crowd of people, is generally an effective way to attract attention. In HRI, body posture can communicate the activation status of a robot. An upright, straight posture can indicate high activation, while a relaxed, crouched posture can indicate inactivity. Similarly, lifting or lowering the head can indicate activation or inactivation. These behaviors can be seen in the Cozmo and Vector robots (Digital Dream Labs) and the Nao robot. Notably, robotic motion and movement of body parts is often accompanied by the sound of the motors involved in these actions, which can be distracting to the receiver [29]. We can explore whether subtle social human phenomena apply to robots. For instance, a robot can incline toward a speaker or mimic the conversation partner's posture to promote feelings of interest and sympathy [72]. This could have implications for power and social hierarchies, as well as robot acceptance and affect. Therefore, robots taking on different roles could communicate their status via specific postures, such as a robot teacher using power poses [73], e.g., standing upright with legs slightly apart, to create an aura of authority, or a service robot using a submissive pose, e.g., slightly bending its body or leaning toward a customer to communicate respect and attentiveness. Some work has also started to adapt human measures and instruments. In particular, McColl and Nejat [74] applied the Position Accessibility Scale from the Nonverbal Interaction States Scale [75] to an HRI context. Others may be adapted for HRI.

4.1.1.9 Proxemics.

Proxemics refers to the perception and management of space, including distance from others. For robots, proxemics is as part of its kinesthetic body language. People create boundaries and control areas of space with their body language. Cultural groups have different rules for what kind of behavior is accepted depending on the type of territory (e.g., References [76, 77]). Certain intimate behaviors that are exhibited in the home might not be acceptable in public, and people are usually expected to follow these rules. This applies to dress codes as well. A special type of perceptual or psychological space is the personal space, a kind of invisible bubble that surrounds a person. People are sensitive about violations of personal space, as it defines the acceptable distance to other people based on many contextual factors ranging from gender, relationship status, social hierarchy, and so on. Violations of personal space can trigger strong reactions, such as high arousal, anxiety, and physical aggression [78].

Proxemics has been widely studied within HRI. In an early work, Walters et al. [79] used Eysenck's model of personality traits to explore spatial zones in HRI, finding that people who were more “proactive” kept greater distance compared to others. Stark et al. [80] explored comfort levels when a robot violated the person's personal space during a collaboration activity. People's approach behavior toward robots is influenced by whether they perceive the robot as acting autonomously or as teleoperated [81]. It is also influenced by perceived liking and gaze patterns, such that people keep away from unlikable robots with sharp gazes [82]. Furthermore, previous studies by Kim and Mutlu [83] revealed a relationship between proxemic distance (close versus far), task distance (cooperative versus competitive), and power distance (supervisor versus subordinate), showing that consistency in these various kinds of distancing is key. Peters et al. [84] found that people responded akin to human–human interaction contexts when walking with virtual human characters and a humanoid robot (in this case, Pepper) in terms of social distance.

Many other factors, nonverbal and otherwise, can influence proxemics between people, and this is sure to be true in HRI contexts, as well. The experience of near-future applications on exoskeletons and exorobotics (e.g., Reference [85]), where the robot is intimately close to the human body, will need to be evaluated. While people usually have some understanding about the size of another person's personal bubble and try to avoid intrusion into this space, a similar awareness might not exist for or by robots. Clear visualizations of a robot's personal space could prevent violent, inappropriate, and/or dangerous inter/actions with and/or toward robots. Another question is whether humanlike robots are more likely to be perceived as having something like a personal bubble compared to machinelike robots or if the size of this perceived space differs. These questions might be especially useful for explaining people's approach and avoidance behaviors toward different types of robots.