A study of Gaussian mixture models of color and texture features for image classification and segmentation

Abstract

The aims of this paper are two-fold: to define Gaussian mixture models (GMMs) of colored texture on several feature spaces and to compare the performance of these models in various classification tasks, both with each other and with other models popular in the literature. We construct GMMs over a variety of different color and texture feature spaces, with a view to the retrieval of textured color images from databases. We compare supervised classification results for different choices of color and texture features using the Vistex database, and explore the best set of features and the best GMM configuration for this task. In addition we introduce several methods for combining the ‘color’ and ‘structure’ information in order to improve the classification performances. We then apply the resulting models to the classification of texture databases and to the classification of man-made and natural areas in aerial images. We compare the GMM model with other models in the literature, and show an overall improvement in performance.

Introduction

In many domains of image processing, there is a strong correspondence between entities in the scene and textures¹ in the image. This implies that the ability to recognize these textures can furnish important semantic information about the scene. Consequently, the problems of texture description and classification, and the closely related problem of segmentation, have received considerable attention, with numerous approaches being proposed (Refs. [1], [2] and references therein). In particular, in the field of content-based image retrieval, the ability to answer the question: “Is there a significant amount of such-and-such texture in this image?”, can be the basis for many types of query.

Two variations on the problem exist: supervised and unsupervised segmentation. In the former, models of the texture associated with different entities in the scene are assumed known, and are then applied to the image in the hope of segmenting it into regions corresponding to those entities. Clearly this requires a training stage in which human beings group texture exemplars into classes, corresponding to the entities involved, from which the corresponding model parameters are then learnt. In the unsupervised case, no models are known a priori. Instead, the aim is to discover similarities in the data that betray the existence of one or more distinct classes into which the data can be divided. This may or may not involve explicitly learning the model parameters. When the entities in the scene into which the image should be segmented are not decided upon beforehand, as they often are not, unsupervised segmentation is methodologically ill-defined, since no specification of the ideal result is given. In supervised segmentation on the other hand, texture classes necessarily correspond to distinct entities in the scene, and the success or failure of the segmentation can be decided on this basis. In this paper, we consider the supervised texture segmentation problem.

Many kinds of statistical models have been applied to texture classification. These include Bayes classifiers assuming multivariate Gaussian distributions for the features [3], [4], [5], [6]; Fisher transformation [7], [8]; nonparametric nearest-neighbor classification [9], [10], [11], [12]; classification trees [8]; learning vector quantization [13], [14]; feed-forward neural networks [15]; and recently support vector machine [16], [17] and multiple histogram combined with self-organized map [18] . In some earlier cases, the statistical modelling after the feature extraction is just thresholding [19], [20], [21], [22]; or simple extremum picking [23], [24], [25]. Markov random fields, and especially Gaussian Markov random fields have been extensively used for texture modelling and segmentation since the early work in Ref. [26]. For a good review, see the paper by Geman and Graffigne [27]. Li and Gray [28] proposed a 2D hidden Markov model (HMM) for image classification, while a somewhat different model is the noncausal HMM described in Ref. [29].

Another recent class of models uses hidden Markov trees (HMTs) to model the joint statistics of wavelet coefficients. Tree models sacrifice some descriptive power (usually only inter- rather than intra-scale dependencies) to ease of implementation (many algorithms that work in the case of linear graphs, also work on trees, but not on more complicated models). HMT models were first introduced in Ref. [30], and were applied to texture analysis in Refs. [31], [32]. They are typically used, even in texture applications, with binary-valued hidden state variables that switch between high and low variance Gaussian distributions for the wavelet coefficients. This behavior is intended to capture the difference between edges and noisy but otherwise smooth regions in images, an important distinction for ‘edge-preserving denoising’. Indeed, for denoising, HMTs result in state of the art algorithms. It is not clear however, that they remain appropriate for single textures, whose statistics may differ markedly from those for natural images considered as a whole. In particular, the division into ‘edges’ and ‘noise’ seems strange in this context. HMTs are used in Ref. [33], where texture and color are combined in an HMT model. Texture is described using HMTs of grayscale wavelet coefficient magnitudes, while color is described using independent Gaussian distributions at each scale for the colored scaling coefficients.

In recent work [34], we proposed the use of Gaussian mixture models (GMMs) for texture classification, demonstrating improved performance over other, computationally more expensive methods. This paper is an extension of the work presented there. In related but differently directed work, Gray et al. [35] also used GMMs for image classification.