Describing Images with Hierarchical Concepts and Object Class Localization

Current research into automatic generation of semantic descriptions centers mainly on improving the annotation accuracy for individual tag or attributes. In this paper, we focus on the generation of more informative descriptions for images. We proposes to generate layered, semantically meaningful descriptions and create summaries of key aspects of the data from the component detectors. In particular, the output descriptions include superclass, class, attributes, and the location of the area of the object which may interest users. We propose to integrate ROI (Region of Interest) identification and hierarchical semantic elements detection into a joint framework. The joint optimization of the ROI localizer and the hierarchical concept detection make them mutually beneficial and reciprocal. In this way, we create a discriminative image description generation framework based on a tightly coupled multi-layer optimization. The output descriptions contain richer information of the image content with layered contextual information, thereby enabling better management and usage of image data. Experiments on two public open benchmark datasets demonstrate that the proposed method obtains state of the art performance.