Skip to main content
Log in

Enhancing intelligence in multimodal emotion assessments

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Computer systems are a part of everyday life, since they influence human behavior and stimulate changes in the emotional states of the users. The assessment of users’ emotions during their interaction with computer systems can help to provide tailorable website interfaces and better recommendations systems. However, emotions are complex and difficult to identify or assess. Previous studies have shown that, in a real-world scenario, the use of single sensors do not provide an accurate emotional assessment. Hence, in this study, we propose a framework that takes into account multiple sensors so that conclusions can be drawn about the emotional state of the user at the time of interaction. The proposed multi-sensing approach includes several inputs from users (such as speech, facial movements, and everyday activities), and uses an artificial intelligent strategy to map these different responses into one or more emotional states. The Componential Emotion Theory and Scherer’s Emotional Semantic Space are used to underpin the theoretical framework. The experimental results show that the combination of outputs generated by multiple sensors provides a more accurate assessment of emotional states than when the sensors are treated individually.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. An emotional episode usually refers to an experience that the individual can recall when questioned about it [14].

  2. The Kappa coefficient is a statistical approach that measures the degree to which specialists concur in a given classification [5]. We obtained a Kappa coefficient was 0.80 when comparing the opinion of three specialists.

References

  1. Bailenson JN, Pontikakis ED, Mauss IB, Gross JJ, Jabon ME, Hutcherson CA, Nass C, John O (2008) Real-time classification of evoked emotions using facial feature tracking and physiological responses. Int J Human Comput Stud 66(5):303– 317

    Article  Google Scholar 

  2. de Carvalho AC, Freitas AA (2009) A tutorial on multi-label classification techniques. In: Foundations of computational intelligence, vol 5, pp 177–195. Springer

  3. Chanel G, Kierkels JJ, Soleymani M, Pun T (2009) Short-term emotion assessment in a recall paradigm. Int J Human-Comput Stud 67(8):607–627

    Article  Google Scholar 

  4. Chen J, Fang Hr, Saad Y (2009) Fast approximate knn graph construction for high dimensional data via recursive lanczos bisection. J Mach Learn Res 10:1989–2012

    MathSciNet  MATH  Google Scholar 

  5. Cohen J et al (1960) A coefficient of agreement for nominal scales. Educ Psychol Measur 20(1):37–46

    Article  Google Scholar 

  6. Desmet P (2004) Measuring emotion: development and application of an instrument to measure emotional responses to products. In: Blythe MA, Overbeeke K, Monk AF, Wright PC (eds) Funology. Kluwer Academic Publishers, Norwell, pp 111–123

    Google Scholar 

  7. Ekman P, Friesen WV (1969) The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica 1(1):49–98

    Article  Google Scholar 

  8. Ellsworth PC, Scherer KR (2003) Appraisal processes in emotion. Handbook Affect Sci 572:V595

    Google Scholar 

  9. Ericsson K, Simon H (1993) Protocol analysis; verbal reports as data (revised edition). Bradfordbooks

  10. Filho GPR, Ueyama J, Faical BS, Pessin Gd, Farias CM, Pazzi RW, Guidoni DL, Villas LA (2015) An energy-aware system for decision-making in a residential infrastructure using wireless sensors and actuators. In: 2015 IEEE 14th International symposium on network computing and applications (NCA), pp 9–16

  11. Filho GPR, Ueyama J, Villas LA, Pinto AR, Goncalves VP, Pessin G, Pazzi RW, Braun T (2014) Nodepm: a remote monitoring alert system for energy consumption using probabilistic techniques. Sensors 14(1):848–867

    Article  Google Scholar 

  12. Fontaine JR, Poortinga YH, Setiadi B, Markam SS (2002) Cognitive structure of emotion terms in indonesia and the netherlands. Cogn Emot 16(1):61–86

    Article  Google Scholar 

  13. Frantzidis CA, Bratsas C, Klados MA, Konstantinidis E, Lithari CD, Vivas AB, Papadelis CL, Kaldoudi E, Pappas C, Bamidis PD (2010) On the classification of emotional biosignals evoked while viewing affective pictures: an integrated data-mining-based approach for healthcare applications. IEEE Trans Inf Technol Biomed 14(2):309–318

    Article  Google Scholar 

  14. Frijda NH (2008) The Psychologist‘s point of view. Handbook of Emotions

  15. Gao X, Xiao B, Tao D, Li X (2010) A survey of graph edit distance. Pattern Anal Appl 13(1):113–129

    Article  MathSciNet  Google Scholar 

  16. Gonċalves VP, de Almeida Neris VP, Seraphini S, Dias TC, Pessin G, Johnson T, Ueyama J (2015) Providing adaptive smartphone interfaces targeted at elderly people: an approach that takes into account diversity among the elderly, pp 1–21. Universal Access in the Information Society

  17. Gonċalves VP, Giancristofaro GT, Rocha Filho GP, Johnson T, Carvalho V, Pessin G, de Almeida Neris VP, Ueyama J (2016) Assessing users’ emotion at interaction time: a multimodal approach with multiple sensors. Soft Comput:1–15

  18. James W (1884) What is an emotion? Mind 34:188–205

    Article  Google Scholar 

  19. Kukolja D, Popović S, Horvat M, Kovaċ B, Ćosić K (2014) Comparative analysis of emotion estimation methods based on physiological measurements for real-time applications. Int J Human-Comput Stud 72 (10):717–727

    Article  Google Scholar 

  20. Lang PJ, Greenwald MK, Bradley MM, Hamm AO (1993) Looking at pictures: affective, facial, visceral, and behavioral reactions. Psychophysiology 30:261–261

    Article  Google Scholar 

  21. Lathia N, Rachuri K, Mascolo C, Roussos G (2013) Open source smartphone libraries for computational social science. In: Proceedings of the 2013 ACM conference on pervasive and ubiquitous computing adjunct publication, pp 911–920. ACM

  22. Li T, An C, Campbell AT, Zhou X (2015) Hilight: hiding bits in pixel translucency changes. ACM SIGMOBILE Mob Comput Commun Rev 18(3):62–70

    Article  Google Scholar 

  23. Lichtenstein A, Oehme A, Kupschick S, Jürgensohn T (2008) Comparing two emotion models for deriving affective states from physiological data. In: Affect and emotion in human-computer interaction, pp 35–50. Springer

  24. LiKamWa R, Liu Y, Lane ND, Zhong L (2013) Moodscope: building a mood sensor from smartphone usage patterns. In: Proceeding of the 11th annual international conference on Mobile systems, applications, and services, pp 389–402. ACM

  25. Littlewort G, Whitehill J, Wu T, Fasel I, Frank M, Movellan J, Bartlett M (2011) The computer expression recognition toolbox (cert). In: 2011 IEEE International conference on automatic face and gesture recognition and workshops (FG 2011), pp 298–305. IEEE

  26. Mahlke S, Minge M (2008) Consideration of multiple components of emotions in human-technology interaction. In: Peter C, Beale R (eds) Affect and emotion in human-computer interaction, lecture notes in computer science, vol 4868. Springer, pp 51–62

  27. Mahlke S, Minge M (2008) Consideration of multiple components of emotions in human-technology interaction. In: Affect and emotion in human-computer interaction, pp 51–62. Springer

  28. Martínez HP, Yannakakis GN (2014) Deep multimodal fusion: combining discrete events and continuous signals. In: Proceedings of the 16th International conference on multimodal interaction, pp 34–41. ACM

  29. Nahin ANH, Alam JM, Mahmud H, Hasan K (2014) Identifying emotion by keystroke dynamics and text pattern analysis. Behav Inf Technol 33(9):987–996

    Article  Google Scholar 

  30. Newman MEJ (2003) The structure and function of complex networks. SIAMR 45(2):167–256

    Article  MathSciNet  MATH  Google Scholar 

  31. Newman M (2010) Networks: an introduction. Oxford University Press, Inc., New York

    Book  MATH  Google Scholar 

  32. Niedenthal PM, Krauth-Gruber S, Ric F (2006) Psychology of emotion: interpersonal, experiential, and cognitive approaches. Psychology Press

  33. Øhrn A, Rowland T (2000) Rough sets: a knowledge discovery technique for multifactorial medical outcomes. Amer J Phys Med Rehab 79(1):100–108

    Article  Google Scholar 

  34. Pedersen C, Togelius J, Yannakakis GN (2010) Modeling player experience for content creation. IEEE Trans Comput Intell AI Games 2(1):54–67

    Article  Google Scholar 

  35. Peter C, Urban B (2012) Emotion in human-computer interaction. In: Expanding the frontiers of visual analytics and visualization

  36. Ramakrishnan S, El Emary IM (2013) Speech emotion recognition approaches in human computer interaction. Telecommun Syst 52(3):1467–1478

    Article  Google Scholar 

  37. Rodrigues K, Garcia F, Bocanegra L, Gonċalves V, Carvalho V, Neris V (2015) Personas-driven design for mental health therapeutic applications. SBC 6(1):18

    Google Scholar 

  38. Rosales GCM, Borges de Araujo R, Otsuka JL, da Rocha RV (2011) Using logical sensors network to the accurate monitoring of the learning process in distance education courses. In: 2011 11th IEEE International conference on advanced learning technologies (ICALT), pp 573–575. IEEE

  39. Russell J (1980) A circumplex model of affect. J Person Soc Psychol 39(6):1161–1178

    Article  Google Scholar 

  40. Santos V, Coca S, Libralon G, Romero R (2013) Imitation of facial expressions for learning emotions in social robotics. In: Proceedings of the 2013 Simposio Brasileiro de Automacao Inteligente

  41. Saragih JM, Lucey S, Cohn JF (2011) Deformable model fitting by regularized landmark mean-shift. Int J Comput Vis 91(2):200–215

    Article  MathSciNet  MATH  Google Scholar 

  42. Scherer K (2005) What are emotions? and how can they be measured? Social Science Information

  43. Scherer KR (2001) Appraisal considered as a process of multilevel sequential checking. Apprais Process Emot Theory Methods Res 92:120

    Google Scholar 

  44. Stemmler G (2003) Methodological considerations in the psychophysiological study of emotion. Handbook Affect Sci:225–255

  45. Van Someren MW, Barnard YF, Sandberg JA et al (1994) The think aloud method: a practical guide to modelling cognitive processes, vol 2. Academic Press London

  46. Van Someren MW, Barnard YF, Sandberg JA et al (1994) The think aloud method: a practical guide to modelling cognitive processes. Academic Press London

  47. Wang R, Chen F, Chen Z, Li T, Harari G, Tignor S, Zhou X, Ben-Zeev D, Campbell AT (2014) Studentlife: assessing mental health, academic performance and behavioral trends of college students using smartphones. In: Proceedings of the 2014 ACM International joint conference on pervasive and ubiquitous computing, pp 3–14. ACM

  48. Wrzus C, Mehl MR (2015) Lab and/or field? measuring personality processes and their social consequences. Eur J Person 29(2):250–271

    Article  Google Scholar 

  49. Xavier RAC, de Almeida Neris VP (2014) Measuring the emotional experience of users through a hybrid semantic approach. In: Proceedings of the 13th Brazilian symposium on human factors in computing systems, pp 226–235. Sociedade Brasileira de Computaċão

  50. Yik MS, Russell JA, Barrett LF (1999) Structure of self-reported current affect: integration and beyond. J Person Social Psychol 77(3):600

    Article  Google Scholar 

  51. Yuan G, Lim TS, Juan WK, Ringo HMH, Li Q (2010) A gmm based 2-stage architecture for multi-subject emotion recognition using physiological responses. In: Proceedings of the 1st augmented human international conference, p 3. ACM

  52. Zhou F, Qu X, Helander MG, Jiao JR (2011) Affect prediction from physiological measures via visual stimuli. Int J Human Comput Stud 69(12):801–819

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding authors

Correspondence to Vinícius P. Gonçalves, Gustavo Pessin or Jó Ueyama.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gonçalves, V.P., Costa, E.P., Valejo, A. et al. Enhancing intelligence in multimodal emotion assessments. Appl Intell 46, 470–486 (2017). https://doi.org/10.1007/s10489-016-0842-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-016-0842-7

Keywords

Navigation