Color active shape models for tracking non-rigid objects

Abstract

Active shape models can be applied to tracking non-rigid objects in video image sequences. Traditionally these models do not include color information in their formulation. In this paper, we present a hierarchical realization of an enhanced active shape model for color video tracking and we study the performance of both hierarchical and non-hierarchical implementations in the RGB, YUV, and HSI color spaces.

Introduction

The problem of tracking people and recognizing their actions in video sequences is of increasing importance to many applications (Haritaoglu et al., 2000; McKenna et al., 1999; Plänkers and Fua, 2001). Examples include video surveillance, human computer interaction, and motion capture for animation, to name a few. Special considerations for digital image processing are required when tracking objects whose forms (and/or their silhouettes) change between consecutive frames. For example, cyclists in a road scene and people in an airport terminal belong to this class of objects denoted as non-rigid objects. Active shape models (ASMs) can be applied to the tracking of non-rigid objects in a video sequence. Most existing ASMs do not consider color information (Pardàs and Sayrol, 2001). We present several extensions of the ASM for color images using different color-adapted objective functions.

In this paper, tracking an object means to identify the object in a video sequence and to calculate its position in every image frame during the analysis of successive images. Using color information as a feature to describe a moving object or person can support these tasks. Brock-Gunn et al. (1994) suggested the use of four-dimensional templates for tracking objects in color image sequences. However, if the observation is accomplished over a long period of time and with many single objects, then both the memory requirements for the templates in the database and the time requirements for the search of a template in the database increase. Deng and Manjunath (2001) used a segmentation scheme based on color quantization to track regions. The regions represent image areas of similar colors and they do not necessarily represent objects. Their approach also differs from other techniques in that it does not estimate exact object motion. In contrast to this approach, active contour models have been investigated for interactive interpretation of features in an image by Kass et al. (1988). The active contour model is an energy-minimizing spline, which is pulled toward features such as lines and edges. This model has been considered as a framework for low-level feature interpretation, such as stereo matching and motion tracking. A more comprehensive description and several applications of the active contour model are given by Blake and Isard (1998). The ASM is a compact model for which the form variety and the color distribution of an object class can both be taught in a training phase (Cootes et al., 1995). Compactness of the ASM results from principal component analysis (PCA) and a priori shape information from the training set.

Several systems use skin color information for tracking faces and hands (e.g. Comaniciu and Ramesh, 2000; Kim et al., 2001; Lee et al., 2001; Li et al., 2000; Marqués and Vilaplana, 2002). The basic idea is to limit the search complexity to one single color cluster (representing skin color) and to identify pixels based on their membership to this cluster. Several problems affect these approaches. First, skin colors are not easy to define for different ethnic groups under varying illumination conditions (Störring and Granum, 2002). Second, it is difficult to track individuals in a crowd of people if these individuals have similar skin colors and; in addition, a person cannot be identified based on skin color when seen from behind. Tracking clothes instead of skin is more appropriate in this situation (Roh et al., 2000). Third, color distributions are sensitive to occlusions, shadows, and changing illumination. Addressing the problem occurring with shadows and occlusions, Lu and Tan (2001) assume that the only moving objects in the scene are people. This assumption does not hold for many applications. Most of the approaches mentioned above cannot be easily extended to multi-colored objects other than people. In this paper, we present a general technique to track colored, non-rigid objects (including people).

A very efficient technique for the recognition of colored objects is color indexing (Swain and Ballard, 1991). An object in the image is assigned to an object stored in a database based on comparisons between color distributions. When applying color indexing to video tracking, the color distribution of the tracked object in frame i can be treated similar to the data stored in the database in “classical” indexing. In this context, tracking becomes the identification and localization of the object to be tracked in frame i+1 based on comparison with its color distribution in frame i. Several modifications of the color indexing algorithm have been proposed to make this technique more robust with regard to illumination changes (e.g. Adjeroh and Lee, 2001; Berens et al., 2000; Finlayson and Xu, 2002; Finlayson et al., 1996; Funt and Finlayson, 1995; Healey and Slater, 1994). However, this technique usually requires multiple views of the object to be recognized, which is not always ensured when the people to be tracked are in a road scene, for example. Furthermore, color indexing partly fails with partial occlusions of the object. ASMs do not need multiple views of an object, since by using energy functions they can be adapted to the silhouette of an object represented in the image. However, the outlier problem, which can occur particularly with partial object occlusion, represents a difficulty for these models.

Vandenbroucke et al. (1997) presented a snake-based approach for tracking soccer players. They used a supervised scheme to learn the jersey colors of each team in a hybrid color space. Based on the results of a color classification in the images, each player who is present in the images is modeled by a snake. In our approach, we do not apply color segmentation to the images. Update of the ASM position in the next frame is based on the minimization of energy functions in the color components. Vandenbroucke et al. (1997) assume that there is only a small change in illumination during the entire soccer game. In our approach, we assume that there is only a small change in illumination between two successive frames.

In addition to earlier results presented in (Koschan et al., 2002), we study the performance of a hierarchical technique in the RGB, YUV, and HSI color spaces. The contributions of the paper are

• an extension of the ASM to color images by incorporating color information into the minimization of the energy functions,

• a hierarchical implementation of the tracking scheme in a color image pyramid applying different color spaces, and

• an investigation of the influence of the length of the search profiles and the number of landmark points on the results.

The remaining part of this paper is organized as follows. In Section 2, the fundamentals of ASMs are described. A hierarchical realization of the tracking algorithm is introduced in Section 3. In Section 4, we discuss the extension of ASMs to color images. Experimental results are provided in Section 5 and Section 6 concludes the paper.