Abstract
Facial expressions convey nonverbal cues which play an important role in interpersonal relations, and are widely used in behavior interpretation of emotions, cognitive science, and social interactions. In this paper we analyze different ways of representing geometric feature and present a fully automatic facial expression recognition (FER) system using salient geometric features. In geometric feature-based FER approach, the first important step is to initialize and track dense set of facial points as the expression evolves over time in consecutive frames. In the proposed system, facial points are initialized using elastic bunch graph matching (EBGM) algorithm and tracking is performed using Kanade-Lucas-Tomaci (KLT) tracker. We extract geometric features from point, line and triangle composed of tracking results of facial points. The most discriminative line and triangle features are extracted using feature selective multi-class AdaBoost with the help of extreme learning machine (ELM) classification. Finally the geometric features for FER are extracted from the boosted line, and triangles composed of facial points. The recognition accuracy using features from point, line and triangle are analyzed independently. The performance of the proposed FER system is evaluated on three different data sets: namely CK+, MMI and MUG facial expression data sets.
Similar content being viewed by others
References
Aifanti N, Delopoulos A (2014) Linear subspace for facial expression recognition. Signal Process Image Commun 29:177–188
Aifanti N, Papachristou A, Delpoulos A (2010) The MUG facial expression database. In Proceeding of 11th international workshop on image analysis for multimedia interactive services, pp 1–4
Asthana A, Saragih J, Wagner M, Goecke R (2009) Evaluating AAM fitting methods for facial expression recognition. In Proceeding of the international conference on affective computing and intelligent interaction, pp 1–8
Blome DS (2003) Elastic bunch graph matching. M.Sc. Thesis, Colorado State University: Fort Collins, CO, USA
Bouguet JY (1999) Pyramidal implementation of the Lucas-Kanade feature tracker. Technological Report, Intel Corporation, Microprocessor Research Lab
Chang Y, Hu C, Feris R, Turk M (2006) Manifold based analysis of facial expression. Image Vis Comput 24:605–614
Chang CC, Lin CJ (2001) LIBSVM: a library for support vector machines. Available online: http://www.csie.ntu.edu.tw/~cjlin/libsvm. Accessed on 16 Jan 2015
Choi HC, Oh SY (2006) Realtime facial expression recognition using active appearance model and multilayer perceptron. In Proceeding of the international joint conference SICE-ICASE, pp 5924–5927
Cid F, Moreno J, Bustos P, Nunez P (2014) Muecas: a multi-sensor robotic head for affective human robot interaction and imitation. Sensors 14:7711–7737
Cruz AC, Bhanu B, Thakoor NS (2014) Vision and attention theory based sampling for continuous facial emotion recognition. IEEE Trans Affect Comput 5:418–431
Ekman P (1994) Strong evidence of universal in facial expressions: a reply to Russell’s mistaken critique. Psychol Bull 115:268–287
Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recogn 36:259–275
Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55:119–139
Ghimire D, Lee J (2013) Geometric feature-based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines. Sensors 13:7714–7734
Ghimire D, Lee J (2014) Extreme learning machine ensemble using bagging for facial expression recognition. J Inf Process Syst 10:443–458
Hsu CW, Chang CC, Lin CJ (2010) A practical guide to support vector classification. Technical Report. Department of Computer Science, National Taiwan University, Taiwan
Huang GB, Zhu QY, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70:489–501
Kotisa I, Buciu I, Pitas I (2008) An analysis of facial expression recognition under partial facial image occlusion. Image Vis Comput 26:1033–1046
Kotisa I, Pitas I (2007) Facial expression recognition in image sequence using geometric deformation features and support vector machines. IEEE Trans Image Process 16:172–187
Li Y, Wang S, Zhao Y, Ji Q (2013) Simultaneous facial feature tracking and facial expression recognition. IEEE Trans Image Process 22:2559–2573
Liu W, Lu J, Wang Z, Song H (2008) An expression space model for facial expression analysis. In Proceeding of IEEE congress in image and signal processing, pp 680–684
Mehrabian A (1968) Communication without words. Psychol Today 2:53–56
Moore S, Bowden R (2011) Local binary patterns for multi-view facial expression recognition. Comput Vis Image Underst 115:541–558
Pantic M, Rothkrantz L (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell 22:1424–1445
Pantic M, Valster R, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In Proceeding of IEEE international conference multimedia and expo, pp. 317–321
Pantic M, Valster M, Rademaker R, Maat L (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specific expression. In Proceeding of 3rd IEEE workshop on CVPR for human communication behavior analysis, pp. 94–101
Rahulmathavan Y, Phan RCW, Chambers JA, Parish DJ (2013) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans Affect Comput 4:83–92
Rudovic O, Pantic M, Patras I (2012) Coupled Gaussian processes for pose-invariant facial expression recognition. IEEE Trans Pattern Anal Mach Intell 25:1357–1369
Saeed A, Al-Hamadi A, Niese R, Elzobi M (2014) Frame-based facial expression recognition using geometric features. Adv Hum Comput Interact 2014:1–13
Samal A, Iyenger PA (1994) Automatic recognition of human face and facial expressions: a survey. Pattern Recogn 25:65–77
Schels M, Schwenker F (2010) A multiple classifier system approach for facial expressions in image sequence utilizing GMM Supervectors. In Proceeding of the 2010 20th international conference on pattern recognition, pp 4251–4254
Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27:803–816
Siddiqi MH, Lee S, Lee YK, Khan AM, Truc P (2013) Hierarchical recognition scheme for human facial expression recognition systems. Sensors 13:16682–16713
Soyel H, Demirel H (2012) Localized discriminative scale invariant feature transform based facial expression recognition. Comput Electr Eng 38:1299–1309
Sung J, Kin D (2009) Real-time facial expression recognition using STAAM and layered GDA classifiers. Image Vis Comput 27:1313–1325
Tian Y-L, Kanade T, Cohn JF (2005) Handbook of face recognition. Springer, Berlin, pp 247–275
Uddin M, Lee J, Kim T (2009) An enhanced independent component-based human facial expression recognition from video. IEEE Trans Consum Electron 55:2216–2224
Uhls YT, Michikyan M, Morris J, Garcia D, Small GW, Zgourou E, Greenfield PM (2014) Five days at outdoor education camp without screen improves pattern skills with nonverbal emotion cues. Comput Hum Behav 39:387–392
Valster MF, Mehu M, Jiang B, Pantic M, Scherer K (2012) Meta-analysis of the first facial expression recognition challenge. IEEE Trans Syst Man Cybern B Cybern 42:966–979
Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57:137–154
Wiskott L, Fellous JM, Krüger N (1997) Face recognition by elastic bunch graph matching. IEEE Trans Pattern Anal Mach Intell 19:775–779
Wright J, Yang AY, Ganesh A, Sastry SS, Ma Y (2009) Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell 31:210–227
Yeasin M, Bullot B, Sharma R (2006) Recognition of facial expressions and measurements of levels of interest from video. IEEE Trans Multimed 8:500–508
Zafeirius S, Pitas I (2008) Discriminant graph structures for facial expression recognition. IEEE Trans Multimed 10:1528–1540
Zeng Z, Pantic M, Roisman GI, Huang TS (2009) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans Pattern Anal Mach Intell 31:39–58
Zhang S, Zhao X, Lei B (2012) Robust facial expression recognition via compressive sensing. Sensors 12:3747–3761
Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29:915–928
Zhao X, Zhang S (2011) Facial expression recognition based on local binary patterns and kernel discriminant isomap. Sensors 11:9573–9588
Zhi R, Flierl M, Ruan Q, Kleijn WB (2011) Graph-preserving sparse nonnegative matrix factorization with application to facial expression recognition. IEEE Trans Syst Man Cybern B Cybern 41:38–52
Zhu J, Zou H, Rosset S, Hastie T (2009) Multi-class AdaBoost. Stat Its Interface 2:349–360
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Ghimire, D., Lee, J., Li, ZN. et al. Recognition of facial expressions based on salient geometric features and support vector machines. Multimed Tools Appl 76, 7921–7946 (2017). https://doi.org/10.1007/s11042-016-3428-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-016-3428-9