Summary
Vision-based human computer interaction is an emerging field of science and industry to provide natural way to communicate with computer. In that sense, one of the skills is to infer the emotional state of the person based on the facial expression recognition. In this paper, we present a novel approach to recognize facial expression from a sequence of input images using HMM (Hidden Markov Model) and facial motion tracking based on optical flow. Conventionally, in the HMM which consists of seven basic emotional states, it is considered natural that transitions between emotions are imposed to pass through neutral state. However, in this work we propose an enhanced transition framework model which consists of transitions between each emotional state without passing through neutral state in addition to a traditional transition model. For the localization of facial features from video sequence we exploit template matching and optical flow. The facial feature displacements traced by the optical flow are used for input parameters to HMM for facial expression recognition. From the experiment, we can prove that the proposed framework can effectively recognize the facial expression in real time.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Fasel, B., Luettin, J.: Automatic Facial Expression Analysis: A Survey. Pattern Recognition 36(1), 259–275 (2003)
Chien, C.C., Chang, Y.J., Chen, Y.C.: Facial Expression Analysis Under Various Head Poses. In: Proceedings of 3rd IEEE Pacific-Rim Conf. on Multimedia, pp. 16–18 (2002)
Cohen, I., Sebe, N., Garg, A., Chen, L.S., Huang, T.S.: Facial expression recognition from video sequences: temporal and static modeling. Computer Vision and Image Understanding 91, 160–187 (2003)
Chun, J.C., Kwon, O.R., Park, P.: A Robust 3D Face Pose Estimation and Facial Expression Control for Vision-Based Animation. In: Cham, T.-J., Cai, J., Dorai, C., Rajan, D., Chua, T.-S., Chia, L.-T. (eds.) MMM 2007. LNCS, vol. 4351, pp. 700–708. Springer, Heidelberg (2007)
Huang, J., Blanz, V., Heisele, B.: Face Recognition with Support Vector Machines and 3D Head Models. In: International Workshop on Pattern Recognition with Support Vector Machines, Niagara Falls, Canada, pp. 334–341 (2002)
Jonsson, K., Matas, J., Kittler, J., Li, Y.P.: Learning Support Vectors for Face Verification and Recognition. In: Procedings of 4th IEEE International Conference on AFGAGF 2000, pp. 208–213 (2000)
Min, K.P., Chun, J.C., Park, G.R.: A Nonparametric Skin Color Model for Face Detection from Color Images. In: Liew, K.-M., Shen, H., See, S., Cai, W. (eds.) PDCAT 2004. LNCS, vol. 3320, pp. 115–119. Springer, Heidelberg (2004)
Rabiner, L.R.: A Tutorial on Hidden Markov Models and selected Applications in Speech Processing. Proc. IEEE 77(2), 257–286 (1989)
Black, M.: Robust incremental optical flow, PhD thesis. Yale University (1992)
Kwon, O.R., Chun, J.C., Park, P.: Cylindrical Model-Based Head Tracking and 3D Pose Recovery from Sequential Face Images. In: IEEE International Conference on Hybrid Information Technology, pp. 135–139 (2006)
Ekman, P., Friesen, W.V.: Facial Action Coding System (FACS). Consulting Psychologist Press, Inc. (1978)
Michel, P., El Kaliouby, R.: Real Time Facial Expression Recognition in Video using Support Vector Machines. In: ICMI 2003, pp. 258–264 (2003)
Levinson, S.E., Rabiner, L.R., Sondhi, M.M.: An Introduction to the Application of the Theory of Probilitic functions of a Markov Process to Automatic Speech Recognition. Bell Lab System Technical J. 62(4), 1035–1072 (1983)
Zhu, Y., De Silva, L.C., Ko, C.C.: A Solution for Facial Expression Representation, and Recognition. Signal Processing: Image Communication 17, 657–673 (2002)
Zhu, Y., De Silva, L.C., Ko, C.C.: Using Moment Invariants and HMM in Facial Expression Recognition. Pattern Recognition Letters Archive 23, 83–91 (2002)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Shin, G., Chun, J. (2008). Spatio-temporal Facial Expression Recognition Using Optical Flow and HMM. In: Lee, R. (eds) Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing. Studies in Computational Intelligence, vol 149. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-70560-4_3
Download citation
DOI: https://doi.org/10.1007/978-3-540-70560-4_3
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-70559-8
Online ISBN: 978-3-540-70560-4
eBook Packages: EngineeringEngineering (R0)