Skip to main content
SpringerLink
Log in

Modeling and Evaluating Emotion, Contextual Head Movement and Voices for a Social Robot Storyteller

  • S.I.: Embodied Interactive Robots
  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

Storytelling bears great potential as prospective application for social robots. Through their embodiment and potential for multi-modal behavior, they can enhance current storytelling media, for example books or audio books. We combined theoretical knowledge about designing storytelling robots with an analysis of human storytellers to implement the behavior of a robot storyteller. We tested the following modalities in three perception studies: emotional behavior, contextual head movements, and voice acting through different synthetic voices. Our results suggest, that an emotional robot storyteller is able to transport participants equally well as the traditional audio book, contextual head movement has to be carefully modeled to be recognized, and voice acting can improve narrative presence and can make the robot more interesting. With the present contribution, we provide guidance for modeling and evaluating the behavior of a social robot storyteller, and suggest that the careful integration of emotional display and different voices can enhance the perception of a robot storytelling experience.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. Robopec, www.reeti.fr.

  2. www.youtube.com/watch?v=Km6ICdi9-b4.

  3. https://github.com/DanielSWolf/rhubarb-lip-sync.

  4. Loquendo TTS 7 by www.nuance.com.

References

  1. Adam C, Cavedon L (2013) A companion robot that can tell stories. In: Proceedings of intelligent virtual agents. Springer

  2. Appel M, Gnambs T, Richter T, Green MC (2015) The transportation scale–short form (TS–SF). Media Psychol 18(2):243–266 test

    Article  Google Scholar 

  3. Bartneck C, Kulić D, Croft E, Zoghbi S (2009) Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. Int J Soc Robot 1(1):71–81

    Article  Google Scholar 

  4. Bates J (1994) The role of emotion in believable agents. Commun ACM 37(7):122–125

    Article  Google Scholar 

  5. Bradley MM, Lang PJ (1994) Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psychiatry 25(1):49–59

    Article  Google Scholar 

  6. Brandtzæg PB, Følstad A, Heim J (2018) Enjoyment: lessons from Karasek. Springer, Cham, pp 331–341. https://doi.org/10.1007/978-3-319-68213-6_21

  7. Breazeal C, Dautenhahn K, Kanda T (2016) Social robotics. In: Springer handbook of robotics. Springer, pp 1935–1972

  8. Breazeal CL (2004) Designing sociable robots. MIT Press, Cambridge

    Book  Google Scholar 

  9. Busselle R, Bilandzic H (2008) Fictionality and perceived realism in experiencing stories: a model of narrative comprehension and engagement. Commun Theory 18(2):255–280

    Article  Google Scholar 

  10. Busselle R, Bilandzic H (2009) Measuring narrative engagement. Media Psychol 12(4):321–347

    Article  Google Scholar 

  11. Cassell J, McNeill D (1991) Gesture and the poetics of prose. Poet Today 12(3):375–404

    Article  Google Scholar 

  12. Chella A, Barone RE, Pilato G, Sorbello R (2008) An emotional storyteller robot. In: AAAI spring symposium: emotion, personality, and social behavior, pp 17–22

  13. Clair AS, Mead R, Mataric MJ (2011) Investigating the effects of visual saliency on deictic gesture production by a humanoid robot. In: 2011 RO-MAN, pp 210–216

  14. Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Taylor & Francis Inc

  15. Coles S, Dautenhahn K (2000) A robotic story-teller. In: Proceedings of SIRS2000, 8th symposium on intelligent robotic systems, grundlage Soziale Robotik, Erste Experimente. The University of Reading, England, pp 18–20

  16. Costa S, Brunete A, Bae BC, Mavridis N (2016) Emotional storytelling using virtual and robotic agents. arXiv preprint arXiv:1607.05327

  17. Cuijpers RH, Knops MAMH (2015) Motions of robots matter! The social effects of idle and meaningful motions. In: Tapus A, André E, Martin JC, Ferland F, Ammi M (eds) Social robotics. Springer, Cham, pp 174–183

    Chapter  Google Scholar 

  18. Damiano R, Pizzo A (2008) Emotions in drama characters and virtual agents. In: AAAI spring symposium: emotion, personality, and social behavior, pp 30–37

  19. Dautenhahn K, Nehaniv C (1998) Artificial life and natural stories. In: International symposium on artificial life and robotics, pp 435–439

  20. DiSalvo CF, Gemperle F, Forlizzi J, Kiesler S (2002) All robots are not created equal: the design and perception of humanoid robot heads. In: Proceedings of the 4th conference on designing interactive systems: processes, practices, methods, and techniques. ACM, pp 321–326

  21. Ekman P, Friesen WV (2003) Unmasking the face: a guide to recognizing emotions from facial clues. Ishk

  22. Ekman P, Friesen WV, Hager JC (2002) Facial action coding system: the manual. Research Nexus, Salt Lake City, UT

  23. Francisco V, Gervás P, González M, León C (2007) Expressive synthesis of read aloud tales. In: Olivier P, Kray C (eds) AISB’07, Artificial and ambient intelligence. pp 179–186

  24. Francisco V, Hervás R, Peinado F, Gervás P (2012) Emotales: creating a corpus of folk tales with emotional annotations. Lang Resources Eval 46(3):341–381

    Article  Google Scholar 

  25. Gelin R, d’Alessandro C, Le QA, Deroo O, Doukhan D, Martin JC, Pelachaud C, Rilliard A, Rosset S (2010) Towards a storytelling humanoid robot. In: AAAI fall symposium: dialog with robots

  26. Goetz J, Kiesler S, Powers A (2003) Matching robot appearance and behavior to tasks to improve human–robot cooperation. In: Proceedings of the 12th IEEE international workshop on robot and human interactive communication. IEEE Press Piscataway, NJ, pp 55–60

  27. Green MC, Brock TC (2000) The role of transportation in the persuasiveness of public narratives. J Personal Soc Psychol 79(5):701

    Article  Google Scholar 

  28. Ham J, Bokhorst R, Cuijpers R, van der Pol D, Cabibihan JJ (2011) Making robots persuasive: the influence of combining persuasive strategies (gazing and gestures) by a storytelling robot on its persuasive power. In: Mutlu B, Bartneck C, Ham J, Evers V, Kanda T (eds) Social robotics. Springer, Berlin, pp 71–83

    Chapter  Google Scholar 

  29. Ham J, Cuijpers RH, Cabibihan JJ (2015) Combining robotic persuasive strategies: the persuasive power of a storytelling robot that uses gazing and gestures. Int J Soc Robot 7(4):479–487. https://doi.org/10.1007/s12369-015-0280-4

    Article  Google Scholar 

  30. Hegel F, Muhl C, Wrede B, Hielscher-Fastabend M, Sagerer G (2009) Understanding social robots. In: 2nd International conferences on advances in computer–human interactions, 2009. ACHI’09. IEEE, pp 169–174

  31. Huang CM, Mutlu B (2013) Modeling and evaluating narrative gestures for humanlike robots. In: Robotics: science and systems, pp 57–64

  32. Jara-Rivera I, Castro-González Á, Castillo JC, Alonso-Martín F, Malfaz M, Salichs MA (2017) Creating lively behaviors in social robots. In: International conference on social robotics. Springer, pp 283–292

  33. Kory Westlund J, Breazeal C (2014) Storytelling with robots: learning companions for preschool children’s language development. In: the 23rd IEEE international symposium on robot and human interactive communication, 2014 RO-MAN. IEEE, pp 643–648

  34. Kory Westlund J, Gordon G, Spaulding S, Lee JJ, Plummer L, Martinez M, Das M, Breazeal C (2016) Lessons from teachers on performing HRI studies with young children in schools. In: 11th ACM/IEEE international conference on human robot interaction. IEEE Press, Piscataway, NJ, USA, HRI ’16, pp 383–390

  35. Kory Westlund J, Jeong S, Park HW, Ronfard S, Adhikari A, Harris PL, DeSteno D, Breazeal CL (2017) Flat vs. expressive storytelling: young children’s learning and retention of a social robot’s narrative. Front Hum Neurosci 11:295

    Article  Google Scholar 

  36. KPMG (2016) Social robots—2016’s new breed of social robots is ready to enter your world. https://assets.kpmg/content/dam/kpmg/pdf/2016/06/social-robots.pdf. Accessed 06 Feb 2019

  37. Lang A (2000) The limited capacity model of mediated message processing. J Commun 50(1):46–70

    Article  Google Scholar 

  38. Lombardo V, Damiano R (2012) Semantic annotation of narrative media objects. Multimed Tools Appl 59(2):407–439

    Article  Google Scholar 

  39. Lugmayr A, Sutinen E, Suhonen J, Sedano CI, Hlavacs H, Montero CS (2017) Serious storytelling—a first definition and review. Multimed Tools Appl 76(14):15707–15733. https://doi.org/10.1007/s11042-016-3865-5

    Article  Google Scholar 

  40. Montaño R, Alías F, Ferrer J (2013) Prosodic analysis of storytelling discourse modes and narrative situations oriented to text-to-speech synthesis. In: 8th ISCA workshop on speech synthesis

  41. Mutlu B, Forlizzi J, Hodgins J (2006) A storytelling robot: modeling and evaluation of human-like gaze behavior. In: 6th IEEE-RAS international conference on humanoid robots, 2006. IEEE, pp 518–523

  42. Niculescu A, Van Dijk B, Nijholt A, See SL (2011) The influence of voice pitch on the evaluation of a social robot receptionist. In: 2011 International conference on user science and engineering (i-USEr). IEEE, pp 18–23

  43. Paradeda RB, Martinho C, Paiva A (2017) Persuasion based on personality traits: using a social robot as storyteller. In: Companion of the 2017 ACM/IEEE international conference on human–robot interaction, HRI 2017, Vienna, Austria, 6–9 March 2017, pp 367–368. https://doi.org/10.1145/3029798.3034824

  44. Park HW, Gelsomini M, Lee JJ, Breazeal C (2017) Telling stories to robots: the effect of backchanneling on a child’s storytelling. In: Proceedings of the 2017 ACM/IEEE international conference on human–robot interaction. ACM, pp 100–108

  45. Pelachaud C, Gelin R, Martin JC, Le QA (2010) Expressive gestures displayed by a humanoid robot during a storytelling application. In: New frontiers in human-robot interaction (AISB), Leicester, GB

  46. Pr̆ibil J, Pr̆ibilová A (2008) Application of expressive speech in TTS system with cepstral description. In: Verbal and nonverbal features of human–human and human–machine interaction. Springer, pp 200–212

  47. Picard RW (1997) Affective computing. MIT Press, ISBN 978-0262661157

  48. Plutchik R (2001) The nature of emotions: human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. Am Sci 89(4):344–350

    Article  Google Scholar 

  49. Pratchett T, Stewart I, Cohen JS (2003) The science of discworld: the globe, vol 2. Random House, New York

    Google Scholar 

  50. Russell JA (1980) A circumplex model of affect. J Personal Soc Psychol 39(6):1161

    Article  Google Scholar 

  51. Striepe H, Lugrin B (2017) There once was a robot storyteller: measuring the effects of emotion and non-verbal behaviour. In: Kheddar A, Yoshida E, Ge SS, Suzuki K, Cabibihan JJ, Eyssel F, He H (eds) Social robotics. Springer, Cham, pp 126–136

    Chapter  Google Scholar 

  52. Theune M, Meijs K, Heylen D, Ordelman R (2006) Generating expressive speech for storytelling applications. IEEE Trans Audio Speech Lang Process 14(4):1137–1144

    Article  Google Scholar 

  53. Tolkien JRR (1947) On fairy-stories. Oxford University Press, Oxford

    Google Scholar 

  54. Van Laer T, De Ruyter K, Visconti LM, Wetzels M (2013) The extended transportation–imagery model: a meta-analysis of the antecedents and consequences of consumers’ narrative transportation. J Consum Res 40(5):797–817

    Article  Google Scholar 

  55. Walters ML, Syrdal DS, Koay KL, Dautenhahn K, Te Boekhorst R (2008) Human approach distances to a mechanical-looking robot with different robot voice styles. In: 17th IEEE international symposium on robot and human interactive communication, 2008. RO-MAN 2008. IEEE, pp 707–712

  56. Xu J, Broekens J, Hindriks K, Neerincx MA (2015) Effects of a robotic storyteller’s moody gestures on storytelling perception. In: 2015 International conference on affective computing and intelligent interaction (ACII). IEEE, pp 449–455

  57. Zipes J (2012) Fairy tales and the art of subversion. Routledge, London

    Book  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Media Informatics, Human-Computer Interaction, University of Wuerzburg, Würzburg, Germany

    Hendrik Striepe, Melissa Donnermann, Martina Lein & Birgit Lugrin

Authors
  1. Hendrik Striepe
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Melissa Donnermann
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Martina Lein
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Birgit Lugrin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Birgit Lugrin.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical Standards

All procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national). All subjects were voluntarily part of the presented studies.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Striepe, H., Donnermann, M., Lein, M. et al. Modeling and Evaluating Emotion, Contextual Head Movement and Voices for a Social Robot Storyteller. Int J of Soc Robotics 13, 441–457 (2021). https://doi.org/10.1007/s12369-019-00570-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-019-00570-7

Keywords

Search

Navigation