Techniques for automatic video parsing and annotation are crucial to turn enormous volumes of video data into a rich and structured data type, and to facilitate video content-based search and retrieval. In this paper, we present a generic video parser with a Scene Description Language (SDL). The SDL enables the human operator to model a video clip in terms of a relatively high abstract level. The video parser is equipped with various algorithms that are common and essential to general video analyses. To handle the video domain with virtually unlimited sets of unanticipated and variable objects and events efficiently, an object-orientated, processing-on-demand approach is devised to perform the video parsing. The video parser first interprets the video model defined by the operator identifies the prominent video properties to be parsed, and then creates an entity for each of the video properties. Each entity knows how to find a match for itself from the video properties extracted from the video image. The video parser interacts with these entities, and performs the feature extraction operations with processing-on-demand basis. Each entity has a self-diagnostic function that is able to turn itself into an inert state when it fails to find the necessary matches during the video parsing process. The inert entities will be excluded from subsequent operations, and will no longer consume any system resources. Our experiments have shown that our generic video parser is effective and efficient in handling a large variety of video images.