ScienceDirect - International Journal of Human-Computer Studies : Real-time classification of evoked emotions using facial feature tracking and physiological responses

International Journal of Human-Computer Studies
Volume 66, Issue 5, May 2008, Pages 303-317

Font Size:

Abstract - selected

E-mail Article

Add to my Quick Links

Bookmark and share in 2collab (opens in new window)

Request permission to reuse this article

Cited By in Scopus (0)

Related Articles in ScienceDirect

Recognition of individual faces and average face protot...
Cognitive Development

What's so special about the average face?
Trends in Cognitive Sciences

The 3-Dimensional Construction of the Average 11-Year-O...
Journal of Oral and Maxillofacial Surgery

Anthropometric proportions in the upper lip-lower lip-c...
American Journal of Orthodontics

Facial anatomy
Clinics in Dermatology

View More Related Articles

View Record in Scopus

doi:10.1016/j.ijhcs.2007.10.011

Real-time classification of evoked emotions using facial feature tracking and physiological responses

Jeremy N. Bailenson^a^,^,, Emmanuel D. Pontikakis^b^,, Iris B. Mauss^c^,, James J. Gross^d^,, Maria E. Jabon^e^,, Cendri A.C. Hutcherson^d^,, Clifford Nass^a^, and Oliver John^f^,

^aDepartment of Communication, Stanford University, Stanford, CA 94305, USA ^bDepartment of Computer Science, Stanford University, Stanford, CA 94305, USA ^cDepartment of Psychology, 2155 South Race Street, University of Denver, Denver, CO 80208, USA ^dDepartment of Psychology, Stanford University, Stanford, CA 94305, USA ^eDepartment of Electrical Engineering, Stanford University, Stanford, CA 94305, USA ^fDepartment of Psychology, University of California, Berkeley, CA 94720, USA

Received 15 February 2007;

revised 28 October 2007;

accepted 29 October 2007.

Communicated by S. Brave.

Available online 1 November 2007.

References and further reading may be available for this article. To view references and further reading you must purchase this article.

Abstract

We present automated, real-time models built with machine learning algorithms which use videotapes of subjects’ faces in conjunction with physiological measurements to predict rated emotion (trained coders’ second-by-second assessments of sadness or amusement). Input consisted of videotapes of 41 subjects watching emotionally evocative films along with measures of their cardiovascular activity, somatic activity, and electrodermal responding. We built algorithms based on extracted points from the subjects’ faces as well as their physiological responses. Strengths of the current approach are (1) we are assessing real behavior of subjects watching emotional videos instead of actors making facial poses, (2) the training data allow us to predict both emotion type (amusement versus sadness) as well as the intensity level of each emotion, (3) we provide a direct comparison between person-specific, gender-specific, and general models. Results demonstrated good fits for the models overall, with better performance for emotion categories than for emotion intensity, for amusement ratings than sadness ratings, for a full model using both physiological measures and facial tracking than for either cue alone, and for person-specific models than for gender-specific or general models.

Keywords: Affective computing; Facial tracking; Emotion; Computer vision