Soumia Dermouche
Catherine Pelachaud
ABSTRACT
A social interaction implies a social exchange between two or more persons, where they adapt and adjust their behaviors in response to their interaction partners. With the growing interest in human-agent interactions, it is desirable to make these interactions more natural and human like. In this context, we aim at enhancing the quality of the interaction between user and Embodied Conversational Agent (ECA) by endowing ECA with the capacity to adapt its behavior in real time according the user’s behavior. The novelty of our approach is to model the agent’s nonverbal behaviors as a function of both agent’s and user’s behaviors jointly with the agent’s communicative intentions creating a dynamic loop between both interactants. Moreover, we encompass the variation of behavior over time through a LSTM-based model. Our model IL-LSTM (Interaction Loop LSTM) predicts the next agent’s behavior taking into account the behavior that both, the agent and the user, have displayed within a time window. We have conducted an evaluation study involving an agent interacting with visitors in a science museum. Results of our study show that participants have better experience and are more engaged in the interaction when the agent adapts its behaviors to theirs, thus creating an interactive loop.
- Jens Allwood and Loredana Cerrato. 2003. A study of gestural feedback expressions. In First Nordic Symposium on Multimodal Communication. 7–22.Google Scholar
- Fumihito Arai and Yasuhisa Hasegawa. 2004. Facial Expressive Robotic Head System for Human – Robot Communication and Its. December (2004). https://doi.org/10.1109/JPROC.2004.835355Google Scholar
- Tadas Baltrusaitis, Peter Robinson, and Louis Philippe Morency. 2016. OpenFace: An open source facial behavior analysis toolkit. 2016 IEEE Winter Conference on Applications of Computer Vision, WACV 2016 (2016). https://doi.org/10.1109/WACV.2016.7477553Google ScholarCross Ref
- Timothy Bickmore, Daniel Schulman, and Langxuan Yin. 2012. Maintaining Engagement in Long-term Interventions with Relational Agents. International Society of Differentiation 83, 2 (2012), 1–29. https://doi.org/10.1158/0008-5472.CAN-10-4002.BONE arxiv:NIHMS150003Google Scholar
- C. Breazeal.2004. Function meets style: insights from emotion theory applied to HRI. IEEE Transactions on Systems, Man, and Cybernetics 34(2) (2004).Google Scholar
- Judee K. Burgoon, Lesa A. Stern, and Leesa Dillman. 2010. Adaptation in Dyadic Interaction: Defining and Operationalizing Patterns of Reciprocity and Compensation.Communication Theory1993(2010). https://doi.org/10.1017/cbo9780511720314Google Scholar
- Angelo Cafaro, Brian Ravenet, Magalie Ochs, Hannes Högni Vilhjálmsson, and Catherine Pelachaud. 2016. The Effects of Interpersonal Attitude of a Group of Agents on User’s Presence and Proxemics Behavior. ACM Transactions on Interactive Intelligent Systems 6, 2 (2016), 1–33. https://doi.org/10.1145/2914796Google ScholarDigital Library
- Angelo Cafaro, Johannes Wagner, Tobias Baur, Soumia Dermouche, Mercedes Torres Torres, Catherine Pelachaud, Elisabeth Andr, and Michel Valstar. 2017. The NoXi Database : Multimodal Recordings of Mediated Novice-Expert Interactions. In ICMI’17,. ACM, Glasgow, Scotland, 350–359. https://doi.org/10.1145/3136755.3136780Google ScholarDigital Library
- Ginevra Castellano, André Pereira, Iolanda Leite, Ana Paiva, and Peter W. McOwan. 2009. Detecting user engagement with a robot companion using task and social interaction-based features. Proceedings of the 2009 international conference on Multimodal interfaces - ICMI-MLMI ’09January 2009 (2009), 119. https://doi.org/10.1145/1647314.1647336Google ScholarDigital Library
- Paul Ekman and Wallace V Friesen. 1976. Mesauring facial movement.pdf.Google Scholar
- Will Feng, Anitha Kannan, Georgia Gkioxari, and C. Lawrence Zitnick. 2017. Learn2Smile: Learning non-verbal interaction through observation. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Vol. 2017-Septe. 4131–4138. https://doi.org/10.1109/IROS.2017.8206272Google ScholarDigital Library
- Terrence Fong, Illah Nourbakhsh, and Kerstin Dautenhahn. 2002. A Survey of Socially Interactive Robots : Concepts , Design , and Applications Terrence Fong , Illah Nourbakhsh , and Kerstin Dautenhahn. Technical Report CMU-RI-TR-02-29, Robotics Institute, Pittsburgh, PANovember(2002).Google Scholar
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in neural information processing systems (2014), 2672–2680. https://doi.org/10.1016/B978-0-408-00109-0.50001-8 arxiv:arXiv:1011.1669v3Google Scholar
- Goren Gordon, Samuel Spaulding, Kory Westlund, Joo Lee, Luke Plummer, Marayna Martinez, and Madhurima Das. 2016. Affective Personalization of a Social Robot Tutor for Children ’ s Second Language Skills. 2011 (2016), 3951–3957.Google Scholar
- H. L. O’Brien and E. G. Toms. 2010. What is User Engagement? A Conceptual Framework for Defining User Engagement with Technology Heather. Journal of the American Society for Information Science 1, 6(2010), 2581–2583. https://doi.org/10.1002/asiGoogle Scholar
- Dirk Heylen, Stefan Kopp, Stacy C. Marsella, Catherine Pelachaud, and Hannes Vilhjálmsson. 2008. The next step towards a function markup language. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 5208 LNAI (2008), 270–280. https://doi.org/10.1007/978-3-540-85483-8_28Google ScholarDigital Library
- Yuchi Huang and Saad M. Khan. 2017. DyadGAN: Generating Facial Expressions in Dyadic Interactions. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops 2017-July(2017), 2259–2266. https://doi.org/10.1109/CVPRW.2017.280Google Scholar
- Elly A. Konijn and Johan F. Hoorn. 2005. Some like it bad: Testing a model for perceiving and experiencing fictional characters. Media Psychology 7, 2 (2005), 107–144. https://doi.org/10.1207/S1532785XMEP0702_1Google ScholarCross Ref
- Caroline Langlet and Chloé Clavel. 2018. Detecting User’s Likes and Dislikes for a Virtual Negotiating Agent. In Proceedings of the 20th ACM International Conference on Multimodal Interaction(ICMI ’18). ACM, 103–110.Google ScholarDigital Library
- A S Mackenzie, C Beaumont, R Boutilier, J Rullkotter, S A F Murrell, R Mason, G Eglinton, and D P McKenzie. 1985. The Aromatization and Isomerization of Hydrocarbons and the Thermal and Subsidence History of the Nova Scotia Margin. Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences 315, 1531 (1985), 203–232. https://doi.org/10.1103/PhysRevB.44.2085Google Scholar
- Erik Murphy-chutorian, Student Member, and Mohan Manubhai Trivedi. 2009. Head Pose Estimation in Computer Vision :. Analysis 31, 4 (2009), 607–626. https://doi.org/10.1109/TPAMI.2008.106Google ScholarDigital Library
- Behnaz Nojavanasghari, Yuchi Huang, and Saad Khan. 2018. Interactive Generative Adversarial Networks for Facial Expression Generation in Dyadic Interactions. (2018). arxiv:1801.09092http://arxiv.org/abs/1801.09092Google Scholar
- T. Nomura, T. Kanda, and T. Suzuki. 2006. Experimental Investigation into Influence of Negative Attitudes toward Robots on Human–Robot Interaction. AI & Society 20, 2 (2006).Google Scholar
- Igor S Pandzic and Robert Forchheimer. [n.d.]. MPEG-4 Facial Animation The Standard , Implementation. John Wiley & Sons.Google Scholar
- Florian Pecune, Angelo Cafaro, Mathieu Chollet, Pierre Philippe, and Catherine Pelachaud. 2014. Suggestions for Extending SAIBA with the VIB Platform. In Workshop on Architectures and Standards for IVAs, held at the ’14th International Conference on Intelligent Virtual Agents (IVA 2014). Bielefeld eCollections, Boston, MA, USA, 16–20.Google Scholar
- Evangelos Sariyanidi, Hatice Gunes, and Andrea Cavallaro. 2015. Automatic analysis of facial affect: A survey of registration, representation, and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 37, 6(2015), 1113–1133. https://doi.org/10.1109/TPAMI.2014.2366127Google ScholarDigital Library
- Candace L Sidner, Christopher Lee, and Neal Lesh. 2003. Engagement by Looking: Behaviours for Robots when Collaborating with People. Proceedings of DiaBruck (the 7th Workshop on Semantics and Pragmatics of Dialogue) (2003), 123–130.Google Scholar
- Henriette C. van Vugt, Johan F. Hoorn, Elly A. Konijn, and Athina de Bie Dimitriadou. 2006. Affective affordances: Improving interface character engagement through interaction. International Journal of Human Computer Studies 64, 9 (2006), 874–888. https://doi.org/10.1016/j.ijhcs.2006.04.008Google ScholarDigital Library
- Jelte van Waterschoot, Merijn Bruijnes, Jan Flokstra, Dennis Reidsma, Daniel Davison, Mariët Theune, and Dirk Heylen. 2018. Flipper 2.0. In Proceedings of the 18th International Conference on Intelligent Virtual Agents. 43–50. https://doi.org/10.1145/3267851.3267882Google ScholarDigital Library
- Hannes Vilhjalmsson, Nathan Cantelmo, Justine Cassell, Nicolas E. Chafai, Michael Kipp, Stefan Kopp, Maurizio Mancini, Stacy Marsella, Andrew N. Marshall, Catherine Pelachaud, Zsofi Ruttkay, Kristinn R. Thórisson, Herwin Van Welbergen, and Rick J. Van Der Werf. 2007. The behavior markup language: Recent developments and challenges. Lecture Notes in Artificial Intelligence 4722, 1 (2007), 99–111. https://doi.org/10.1007/978-3-540-74997-4Google Scholar
- G. Volpe, P. Alborno, A. Camurri, P. Coletta, S. Ghisio, M. Mancini, R. Niewiadomski, and S. Piana. 2016. Designing Multimodal Interactive Systems Using EyesWeb XMI. SERVE@AVI (2016), 49–56.Google Scholar
- Nannan Wang, Xinbo Gao, Dacheng Tao, Heng Yang, and Xuelong Li. 2018. Facial feature point detection: A comprehensive survey. Neurocomputing 275(2018), 50–65. https://doi.org/10.1016/j.neucom.2017.05.013 arxiv:1410.1037Google ScholarDigital Library
- J. S. Wiggins. 1979. A psychological taxonomy of trait-descriptive terms: The interpersonal domain.Journal of Personality and Social Psychology37 (1979), 395–412.Google Scholar
- Beverly Woolf and Winslow Burleson. 2009. Affect-aware tutors : recognising and responding to student affect Ivon Arroyo , Toby Dragon and David Cooper Rosalind Picard. 4 (2009).Google Scholar
- Zhou Yu, Xinrui He, Alan W Black, and Alexander I Rudnicky. 2016. User Engagement Study with Virtual Agents Under Different Cultural Contexts. In Intelligent Virtual Agents - 16th International Conference, IVA2016. Los Angeles, CA, USA, 364–368.Google Scholar
Recommendations
The Optimum Rate of Mimicry in Human-Agent Interaction
HAI '16: Proceedings of the Fourth International Conference on Human Agent InteractionThe importance of building rapport between a human and an agent is increasing with the burgeoning development of robot technology. Several recent studies have focused on the chameleon effect, using psychological concepts to investigate human-agent ...
Interruptions in Human-Agent Interaction
IVA '21: Proceedings of the 21st ACM International Conference on Intelligent Virtual AgentsTurn management is one of the necessary social interactions skills. In human-human interactions, turn changes are naturally completed by interruption, "cooperatively" or "competitively". Interruptions are inherent in conversation. They can be considered ...
Communicative Listener Feedback in Human-Agent Interaction: Artificial Speakers Need to Be Attentive and Adaptive
AAMAS '18: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent SystemsIn human dialogue, listener feedback is a pervasive phenomenon that serves important functions in the coordination of the conversation, both in regulating its flow, as well as in creating and ensuring understanding between interlocutors. This make ...
Comments