Skip to main content
SpringerLink
Log in

Understanding of the Biological Process of Nonverbal Communication: Facial Emotion and Expression Recognition

  • Chapter
  • First Online:
Video Bioinformatics

Part of the book series: Computational Biology ((COBO,volume 22))

  • 1407 Accesses

Abstract

Facial emotion and expression recognition is the study of facial expressions to infer the emotional state of a person. A camera captures video or images of a person’s face and algorithms automatically, without the help of a human operator, detect his/her expressions to infer his/her underlying emotional state. There has been an increased interest in this field in the past decade, and a system that accomplishes these tasks in unconstrained settings is a realizable goal. In this chapter, we will discuss the process by which a human expresses an emotion; how it is perceived by the human visual system at a low level; how prediction of emotion is made by a human; and publicly available datasets currently used by researchers in the field.

Portions of this chapter are © IEEE 2013, 2014 and appeared in “Background suppressing Gabor energy filtering,” “Score-based facial emotion recognition,” and “Vision and attention theory-based sampling for continuous facial emotion recognition”.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Darwin C (1872) The expression of the emotions in man and animals. John Murray

    Google Scholar 

  2. el Kaliouby R, Robinson P (2005) The emotional hearing aid: an assistive tool for children with asperger syndrome. Univ Access Inf Soc 4(2):121–134

    Google Scholar 

  3. Schuller B, Marchi E, Baron-Cohen S, O’Reilley H, Robinson P, Davies I, Golan O, Friedenson S, Friedenson S, Tal S, Newman S, Meir N, Shillo R, Camurri A, Piana S (2013) ASC-Inclusion: interactive emotion games for social inclusion of children with autism spectrum conditions. In: Intelligent digital games for empowerment and inclusion

    Google Scholar 

  4. Shotton J, Fitzgibbon A, Cook M, Finocchio M, Moore R, Kipman A, Blake A (2011) Real-Time human pose recognition in parts from single depth images. In: Proceedings of the IEEE computer vision and pattern recognition, Colorado Springs

    Google Scholar 

  5. McKeown G, Valstar M, Cowie R, Pantic M, Schröder M (2012) The SEMAINE database: annotated multimodal records of emotionally colored conversations between a person and a limited agent. IEEE Trans Affect Comput 3(1):5–17

    Article  Google Scholar 

  6. Elkins AC, Sun Y, Zafeiriou S, Pantic M (2013) The face of an imposter: computer vision for deception detection. In: Proceedings of the Hawaii international conference on system sciences, Grand Wailea

    Google Scholar 

  7. Valstar MF, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the MMI facial expression database. In: Proceedings of the international language resources and evaluation conference, Malta

    Google Scholar 

  8. Valstar MF, Mehu M, Jiang B, Pantic M, Scherer K (2012) Meta-analysis of the first facial expression recognition challenge. IEEE Trans Syst Man Cybern Part B 42(4):966–979

    Article  Google Scholar 

  9. Shuller B, Valstar M, Eyben F, Cowie R, Pantic M (2012) AVEC 2012—the continuous audio/visual emotion challenge. In: Proceedings of the ACM international conference on multimodal interaction, Santa Monica

    Google Scholar 

  10. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: IEEE CVPR

    Google Scholar 

  11. Yang S, Bhanu B (2012) Understanding discrete facial expressions in video using an emotion avatar image. IEEE Trans Syst, Man, Cybern, Part B 42(4):980–992

    Article  Google Scholar 

  12. Heikkila J, Ojansivu V (2008) Blur insensitive texture classification using local phase quantization. In: Image and signal processing. Springer, New York, pp 236–243

    Google Scholar 

  13. Grigorescue C, Petkov N, Westenberg MA (2003) Contour detection based on nonclassical receptive field inhibition. IEEE Trans Image Process 12(7):729–739

    Article  Google Scholar 

  14. Jiang B, Valstar MF, Pantic M (2012) Facial action detection using block-based pyramid appearance descriptors. In: Proceedings of the IEEE international conference on social computing, Amsterdam

    Google Scholar 

  15. Pietikainen M, Zhao G (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Article  Google Scholar 

  16. Pietikainen T, Ahonen A, Hadid M (2006) Face description with local binary patterns: application to face recognition. IEEE Trans Pattern Recogn Anal 28(12):2037–2041

    Article  Google Scholar 

  17. Wolf L, Hassner T, Taigman Y (2011) Effective unconstrained face recognition by combining multiple descriptors and learned background statistic. IEEE Trans Pattern Recogn and Anal 33(10):1978–1990

    Article  Google Scholar 

  18. Cruz AC, Bhanu B, Thakoor NS (2013) Facial emotion recognition with expression energy. In: Proceedings of the ACM international conference on multimodal interaction, Santa Monica

    Google Scholar 

  19. Glodek M, Tschechne S, Layher G, Schels M, Brosch T, Scherer S, Kächele M, Schmidt M, Neumann H, Palm G, Schwenker F (2011) Multiple classifier systems for the classification of audio-visual emotional states. In: Proceedings of the affective computing and intelligent interaction, Memphis

    Google Scholar 

  20. Gupta MD, Jing X (2011) Non-negative matrix factorization as a feature selection tool for maximum margin classifiers. In: IEEE CVPR

    Google Scholar 

  21. Brunzell H, Eriksson J (2000) Feature reduction for classification of multidimensional data. Pattern Recogn 33(10):1741–1748

    Article  Google Scholar 

  22. Cruz AC, Bhanu B, Yang S (2011) A psychologically inspired match-score fusion model for video-based facial expression recognition. In: Proceedings of the affective computing and intelligent interaction, Memphis

    Google Scholar 

  23. Lyons M, Akamatsu S (1998) Coding facial expressions with Gabor wavelets. In: Proceedings of the IEEE conference on automatic face and gesture recognition, Nara

    Google Scholar 

  24. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z (2010) The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit. In: IEEE CVPR

    Google Scholar 

  25. Ekman P, Friesen W (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto

    Google Scholar 

  26. Ekman P (1999) Basic emotions. In: The handbook of cognition and emotion. Wiley, New York, pp 45–60

    Google Scholar 

  27. Schuller B, Valstar M, Eyben F, Cowie R, Pantic M (2012) AVEC 2012—the continuous audio/visual emotion challenge. In: Proceedings of the ACM international conference on multimodal interaction, Santa Monica

    Google Scholar 

  28. McKeown G (2013) Youtube, 24 February 2011. http://www.youtube.com/watch?v=6KZc6e_EuCg. Accessed 21 June 2013 (Online)

  29. Fontaine J, Scherer K, Roesch E, Ellsworth P (2007) The world of emotions is not two-dimensional. Psychol Sci 18(12):2050–1057

    Article  Google Scholar 

  30. Soladie C, Salam H, Pelachaud C, Nicolas Stoiber RS (2012) A multimodal fuzzy inference system using a continuous facial expression representation for emotion detection. In: Proceedings of the ACM international conference on multimodal interaction, Santa Monica

    Google Scholar 

Download references

Acknowledgment

This work was supported in part by the National Science Foundation Integrative Graduate Education and Research Traineeship (IGERT) in Video Bioinformatics (DGE-0903667). Alberto Cruz is an IGERT Fellow.

Author information

Authors and Affiliations

  1. Center for Research, Intelligent Systems, University of California, Riverside, 900 University Ave, Riverside, CA, 92521, USA

    Alberto C. Cruz, B. Bhanu & N. S. Thakoor

Authors
  1. Alberto C. Cruz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. B. Bhanu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. N. S. Thakoor
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alberto C. Cruz .

Editor information

Editors and Affiliations

  1. University of California, Riverside, California, USA

    Bir Bhanu

  2. University of California, Riverside, California, USA

    Prue Talbot

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Cruz, A.C., Bhanu, B., Thakoor, N.S. (2015). Understanding of the Biological Process of Nonverbal Communication: Facial Emotion and Expression Recognition. In: Bhanu, B., Talbot, P. (eds) Video Bioinformatics. Computational Biology, vol 22. Springer, Cham. https://doi.org/10.1007/978-3-319-23724-4_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-23724-4_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-23723-7

  • Online ISBN: 978-3-319-23724-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation