Reactive memories: an interactive talking-head

Abstract

We demonstrate a novel method for producing a synthetic talking-head. The method is based on earlier work in which the behaviour of a synthetic individual is generated by reference to a probabilistic model of interactive behaviour within the visual domain—such models are learnt automatically from typical interactions. We extend this work into a combined visual and auditory domain and employ a state-of-the-art facial appearance model. The result is a real-time synthetic talking-head that responds appropriately and with realistic timing to simple forms of greeting.

Introduction

The screen-based ‘talking-head’ is a powerful device for mediating interaction between humans and machines, enabling a form of interaction that mimics direct communication between humans [2], [5], [8], [16], [17], [18], [19], [22]. The experience of realism is further enhanced when the computer is equipped with visual and auditory senses with which to perceive the user [1], [3], [6], [9], [10], [12], [20], [21], [23], [26], [27], [28]. In this symmetric situation, both the human and synthetic head can see and be seen, and can hear and be heard.

Of paramount importance within face to face conversation is of course the content of what is said. However, the sequence and timing of accompanying facial expressions is also important; mistimed or inappropriate expressions may convey unintended meaning and can therefore be disruptive. It is reasonable to suppose the same requirements will apply for human interaction with a synthetic talking-head.

An approach that begins to meet these requirements is proposed in Refs. [12], [14]. Their idea is based on the common notion of a state space, in which each vector represents the instantaneous configuration of a participant in an interaction. Such vectors are the end-point of a perceptual process within the computer, sensing the human party, and the start-point for a graphical process generating the synthetic individual. An interaction can be thought of as a pathway through the joint configuration space corresponding to the human and synthetic party in an interaction. The range of possible interactions is represented as a stochastic process over the joint configuration space, which is learnt through observation of real interactions captured on video. Johnson [13] modelled the profiles of two people shaking hands. Jebara and Pentland [12] modelled head and hand gestures. In both cases, the models were used to drive a synthetic individual in response to past joint behaviour.

In the current paper, we construct a simple synthetic talking-head by adopting the same approach together with combined modelling of speech and facial expression. The principal objective has been to produce a reactive head in which speech utterances and facial expressions are both appropriate and timely. We handle only a few kinds of simple verbal interactions. Nevertheless, the resulting system is indicative of a new kind of medium that could complement the photograph in an album or the home video with a reactive icon of a familiar person—hence the title of the paper.