Abstract
Computer systems are a part of everyday life, since they influence human behavior and stimulate changes in the emotional states of the users. The assessment of users’ emotions during their interaction with computer systems can help to provide tailorable website interfaces and better recommendations systems. However, emotions are complex and difficult to identify or assess. Previous studies have shown that, in a real-world scenario, the use of single sensors do not provide an accurate emotional assessment. Hence, in this study, we propose a framework that takes into account multiple sensors so that conclusions can be drawn about the emotional state of the user at the time of interaction. The proposed multi-sensing approach includes several inputs from users (such as speech, facial movements, and everyday activities), and uses an artificial intelligent strategy to map these different responses into one or more emotional states. The Componential Emotion Theory and Scherer’s Emotional Semantic Space are used to underpin the theoretical framework. The experimental results show that the combination of outputs generated by multiple sensors provides a more accurate assessment of emotional states than when the sensors are treated individually.
Similar content being viewed by others
Notes
An emotional episode usually refers to an experience that the individual can recall when questioned about it [14].
The Kappa coefficient is a statistical approach that measures the degree to which specialists concur in a given classification [5]. We obtained a Kappa coefficient was 0.80 when comparing the opinion of three specialists.
References
Bailenson JN, Pontikakis ED, Mauss IB, Gross JJ, Jabon ME, Hutcherson CA, Nass C, John O (2008) Real-time classification of evoked emotions using facial feature tracking and physiological responses. Int J Human Comput Stud 66(5):303– 317
de Carvalho AC, Freitas AA (2009) A tutorial on multi-label classification techniques. In: Foundations of computational intelligence, vol 5, pp 177–195. Springer
Chanel G, Kierkels JJ, Soleymani M, Pun T (2009) Short-term emotion assessment in a recall paradigm. Int J Human-Comput Stud 67(8):607–627
Chen J, Fang Hr, Saad Y (2009) Fast approximate knn graph construction for high dimensional data via recursive lanczos bisection. J Mach Learn Res 10:1989–2012
Cohen J et al (1960) A coefficient of agreement for nominal scales. Educ Psychol Measur 20(1):37–46
Desmet P (2004) Measuring emotion: development and application of an instrument to measure emotional responses to products. In: Blythe MA, Overbeeke K, Monk AF, Wright PC (eds) Funology. Kluwer Academic Publishers, Norwell, pp 111–123
Ekman P, Friesen WV (1969) The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica 1(1):49–98
Ellsworth PC, Scherer KR (2003) Appraisal processes in emotion. Handbook Affect Sci 572:V595
Ericsson K, Simon H (1993) Protocol analysis; verbal reports as data (revised edition). Bradfordbooks
Filho GPR, Ueyama J, Faical BS, Pessin Gd, Farias CM, Pazzi RW, Guidoni DL, Villas LA (2015) An energy-aware system for decision-making in a residential infrastructure using wireless sensors and actuators. In: 2015 IEEE 14th International symposium on network computing and applications (NCA), pp 9–16
Filho GPR, Ueyama J, Villas LA, Pinto AR, Goncalves VP, Pessin G, Pazzi RW, Braun T (2014) Nodepm: a remote monitoring alert system for energy consumption using probabilistic techniques. Sensors 14(1):848–867
Fontaine JR, Poortinga YH, Setiadi B, Markam SS (2002) Cognitive structure of emotion terms in indonesia and the netherlands. Cogn Emot 16(1):61–86
Frantzidis CA, Bratsas C, Klados MA, Konstantinidis E, Lithari CD, Vivas AB, Papadelis CL, Kaldoudi E, Pappas C, Bamidis PD (2010) On the classification of emotional biosignals evoked while viewing affective pictures: an integrated data-mining-based approach for healthcare applications. IEEE Trans Inf Technol Biomed 14(2):309–318
Frijda NH (2008) The Psychologist‘s point of view. Handbook of Emotions
Gao X, Xiao B, Tao D, Li X (2010) A survey of graph edit distance. Pattern Anal Appl 13(1):113–129
Gonċalves VP, de Almeida Neris VP, Seraphini S, Dias TC, Pessin G, Johnson T, Ueyama J (2015) Providing adaptive smartphone interfaces targeted at elderly people: an approach that takes into account diversity among the elderly, pp 1–21. Universal Access in the Information Society
Gonċalves VP, Giancristofaro GT, Rocha Filho GP, Johnson T, Carvalho V, Pessin G, de Almeida Neris VP, Ueyama J (2016) Assessing users’ emotion at interaction time: a multimodal approach with multiple sensors. Soft Comput:1–15
James W (1884) What is an emotion? Mind 34:188–205
Kukolja D, Popović S, Horvat M, Kovaċ B, Ćosić K (2014) Comparative analysis of emotion estimation methods based on physiological measurements for real-time applications. Int J Human-Comput Stud 72 (10):717–727
Lang PJ, Greenwald MK, Bradley MM, Hamm AO (1993) Looking at pictures: affective, facial, visceral, and behavioral reactions. Psychophysiology 30:261–261
Lathia N, Rachuri K, Mascolo C, Roussos G (2013) Open source smartphone libraries for computational social science. In: Proceedings of the 2013 ACM conference on pervasive and ubiquitous computing adjunct publication, pp 911–920. ACM
Li T, An C, Campbell AT, Zhou X (2015) Hilight: hiding bits in pixel translucency changes. ACM SIGMOBILE Mob Comput Commun Rev 18(3):62–70
Lichtenstein A, Oehme A, Kupschick S, Jürgensohn T (2008) Comparing two emotion models for deriving affective states from physiological data. In: Affect and emotion in human-computer interaction, pp 35–50. Springer
LiKamWa R, Liu Y, Lane ND, Zhong L (2013) Moodscope: building a mood sensor from smartphone usage patterns. In: Proceeding of the 11th annual international conference on Mobile systems, applications, and services, pp 389–402. ACM
Littlewort G, Whitehill J, Wu T, Fasel I, Frank M, Movellan J, Bartlett M (2011) The computer expression recognition toolbox (cert). In: 2011 IEEE International conference on automatic face and gesture recognition and workshops (FG 2011), pp 298–305. IEEE
Mahlke S, Minge M (2008) Consideration of multiple components of emotions in human-technology interaction. In: Peter C, Beale R (eds) Affect and emotion in human-computer interaction, lecture notes in computer science, vol 4868. Springer, pp 51–62
Mahlke S, Minge M (2008) Consideration of multiple components of emotions in human-technology interaction. In: Affect and emotion in human-computer interaction, pp 51–62. Springer
Martínez HP, Yannakakis GN (2014) Deep multimodal fusion: combining discrete events and continuous signals. In: Proceedings of the 16th International conference on multimodal interaction, pp 34–41. ACM
Nahin ANH, Alam JM, Mahmud H, Hasan K (2014) Identifying emotion by keystroke dynamics and text pattern analysis. Behav Inf Technol 33(9):987–996
Newman MEJ (2003) The structure and function of complex networks. SIAMR 45(2):167–256
Newman M (2010) Networks: an introduction. Oxford University Press, Inc., New York
Niedenthal PM, Krauth-Gruber S, Ric F (2006) Psychology of emotion: interpersonal, experiential, and cognitive approaches. Psychology Press
Øhrn A, Rowland T (2000) Rough sets: a knowledge discovery technique for multifactorial medical outcomes. Amer J Phys Med Rehab 79(1):100–108
Pedersen C, Togelius J, Yannakakis GN (2010) Modeling player experience for content creation. IEEE Trans Comput Intell AI Games 2(1):54–67
Peter C, Urban B (2012) Emotion in human-computer interaction. In: Expanding the frontiers of visual analytics and visualization
Ramakrishnan S, El Emary IM (2013) Speech emotion recognition approaches in human computer interaction. Telecommun Syst 52(3):1467–1478
Rodrigues K, Garcia F, Bocanegra L, Gonċalves V, Carvalho V, Neris V (2015) Personas-driven design for mental health therapeutic applications. SBC 6(1):18
Rosales GCM, Borges de Araujo R, Otsuka JL, da Rocha RV (2011) Using logical sensors network to the accurate monitoring of the learning process in distance education courses. In: 2011 11th IEEE International conference on advanced learning technologies (ICALT), pp 573–575. IEEE
Russell J (1980) A circumplex model of affect. J Person Soc Psychol 39(6):1161–1178
Santos V, Coca S, Libralon G, Romero R (2013) Imitation of facial expressions for learning emotions in social robotics. In: Proceedings of the 2013 Simposio Brasileiro de Automacao Inteligente
Saragih JM, Lucey S, Cohn JF (2011) Deformable model fitting by regularized landmark mean-shift. Int J Comput Vis 91(2):200–215
Scherer K (2005) What are emotions? and how can they be measured? Social Science Information
Scherer KR (2001) Appraisal considered as a process of multilevel sequential checking. Apprais Process Emot Theory Methods Res 92:120
Stemmler G (2003) Methodological considerations in the psychophysiological study of emotion. Handbook Affect Sci:225–255
Van Someren MW, Barnard YF, Sandberg JA et al (1994) The think aloud method: a practical guide to modelling cognitive processes, vol 2. Academic Press London
Van Someren MW, Barnard YF, Sandberg JA et al (1994) The think aloud method: a practical guide to modelling cognitive processes. Academic Press London
Wang R, Chen F, Chen Z, Li T, Harari G, Tignor S, Zhou X, Ben-Zeev D, Campbell AT (2014) Studentlife: assessing mental health, academic performance and behavioral trends of college students using smartphones. In: Proceedings of the 2014 ACM International joint conference on pervasive and ubiquitous computing, pp 3–14. ACM
Wrzus C, Mehl MR (2015) Lab and/or field? measuring personality processes and their social consequences. Eur J Person 29(2):250–271
Xavier RAC, de Almeida Neris VP (2014) Measuring the emotional experience of users through a hybrid semantic approach. In: Proceedings of the 13th Brazilian symposium on human factors in computing systems, pp 226–235. Sociedade Brasileira de Computaċão
Yik MS, Russell JA, Barrett LF (1999) Structure of self-reported current affect: integration and beyond. J Person Social Psychol 77(3):600
Yuan G, Lim TS, Juan WK, Ringo HMH, Li Q (2010) A gmm based 2-stage architecture for multi-subject emotion recognition using physiological responses. In: Proceedings of the 1st augmented human international conference, p 3. ACM
Zhou F, Qu X, Helander MG, Jiao JR (2011) Affect prediction from physiological measures via visual stimuli. Int J Human Comput Stud 69(12):801–819
Author information
Authors and Affiliations
Corresponding authors
Rights and permissions
About this article
Cite this article
Gonçalves, V.P., Costa, E.P., Valejo, A. et al. Enhancing intelligence in multimodal emotion assessments. Appl Intell 46, 470–486 (2017). https://doi.org/10.1007/s10489-016-0842-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-016-0842-7