We describe a system to detect the speaker's face and mouth in videophone sequences. A statistical scheme based on a subspace method is described for detecting and tracking faces under varying poses. A matching criterion based on a Generalized Likelihood Ratio is optimized efficiently with respect to a perspective transformation using a coarse-to-fine search strategy combined with a simulated annealing algorithm. Moreover, we analyze the amplitude projections around the speaker's mouth to describe the shape of the lips. All computations are performed on lossy H263-coded images. The proposed algorithms are well-suited to a further real-time implementation.