EURASIP Journal on Applied Signal Processing 
Volume 2003 (2003), Issue 2, Pages 170-185
doi:10.1155/S1110865703211173

Semantic Indexing of Multimedia Content Using Visual, Audio, and Text Cues

W. H. Adams,1 Giridharan Iyengar,1 Ching-Yung Lin,2 Milind Ramesh Naphade,2 Chalapathy Neti,1 Harriet J. Nock,1 and John R. Smith2

 1IBM T. J. Watson Research Center, Yorktown Heights 10598, NY, USA
2IBM T. J. Watson Research Center, Hawthorne 10532, NY, USA
Received 2 April 2002; Revised 15 November 2002

Abstract

We present a learning-based approach to the semantic indexing of multimedia content using cues derived from audio, visual, and text features. We approach the problem by developing a set of statistical models for a predefined lexicon. Novel concepts are then mapped in terms of the concepts in the lexicon. To achieve robust detection of concepts, we exploit features from multiple modalities, namely, audio, video, and text. Concept representations are modeled using Gaussian mixture models (GMM), hidden Markov models (HMM), and support vector machines (SVM). Models such as Bayesian networks and SVMs are used in a late-fusion approach to model concepts that are not explicitly modeled in terms of features. Our experiments indicate promise in the proposed classification and fusion methodologies: our proposed fusion scheme achieves more than 10% relative improvement over the best unimodal concept detector.