Qualitative evaluation of automatic assignment of keywords to images

Abstract

In image retrieval, most systems lack user-centred evaluation since they are assessed by some chosen ground truth dataset. The results reported through precision and recall assessed against the ground truth are thought of as being an acceptable surrogate for the judgment of real users. Much current research focuses on automatically assigning keywords to images for enhancing retrieval effectiveness. However, evaluation methods are usually based on system-level assessment, e.g. classification accuracy based on some chosen ground truth dataset. In this paper, we present a qualitative evaluation methodology for automatic image indexing systems. The automatic indexing task is formulated as one of image annotation, or automatic metadata generation for images. The evaluation is composed of two individual methods. First, the automatic indexing annotation results are assessed by human subjects. Second, the subjects are asked to annotate some chosen images as the test set whose annotations are used as ground truth. Then, the system is tested by the test set whose annotation results are judged against the ground truth. Only one of these methods is reported for most systems on which user-centred evaluation are conducted. We believe that both methods need to be considered for full evaluation. We also provide an example evaluation of our system based on this methodology. According to this study, our proposed evaluation methodology is able to provide deeper understanding of the system’s performance.

Introduction

Evaluation is a critical issue for Information Retrieval (IR). Assessment of the performance or the value of an IR system for its intended task is one of the distinguishing features of the subject. The type of evaluation to be considered depends on the objectives of the retrieval system. In general, retrieval performance evaluation is based on a test reference collection, e.g. TREC, and on an evaluation measure, e.g. precision and recall (Baeza-Yates & Ribeiro-Neto, 1999).

Saracevic (1995) reviews the history and nature of evaluation in IR and describes six different levels of IR evaluation from system to user levels. However, most IR evaluations are only based on the system level(s) and lack user-centred evaluation. To achieve a more comprehensive picture of IR performance and users’ needs, both system- and user-centred evaluations are needed. That is, we need to evaluate at different levels as appropriate and/or against different types of relevance (Dunlop, 2000). Examples of some recent studies focusing on user judgments are Belkin et al., 2001, Hersh et al., 2001, and Spink (2002).

Due to the advances in computing and multimedia technologies, the size of image collections is increasing rapidly. Content-Based Image Retrieval (CBIR) has been an active research area for the last decade whose main goal is to design mechanisms for searching large image collections. Similar to traditional IR, studies on user issues of image retrieval are lacking (Fidel, 1997, Rasmussen, 1997).

Current CBIR systems index and retrieve images based on their low-level features, such as colour, texture, and shape, and it is difficult to find desired images based on these low-level features, because they have no direct correspondence to high-level concepts in humans’ minds. This is the so-called semantic gap problem. Bridging the semantic gap in image retrieval has attracted much work generally focussing on making systems more intelligent and automatically understanding image contents in terms of high-level concepts (Eakins, 2002). Image annotation systems, i.e. automatic assignment of one or multiple keywords to an image, have been developed for this purpose (Barnard et al., 2003, Kuroda and Hagiwara, 2002, Li and Wang, 2003, Park et al., 2004, Tsai et al., 2003, Vailaya et al., 2001).

To evaluate the annotation results, most of these systems are only based on some chosen dataset with ground truth, such as Corel. However, the problem is that currently there is no standard image dataset for evaluation, like the web track of TREC for IR (Craswell, Hawking, Wilkinson, & Wu, 2003). As IR systems also need to consider human subjects for evaluation, quantitative evaluation of current annotation systems are insufficient to validate their performances. Therefore, user-centred evaluation of image annotation systems is also necessary.

This paper is organised as follows. Section 2 reviews related work on conducting qualitative evaluation for image retrieval related algorithms, systems, etc. Section 3 presents our qualitative evaluation methodology for image annotation systems. Section 4 shows an example of assessing our image annotation system based on the proposed methodology. Section 5 provides some discussion of the user-centred evaluation. Finally, some conclusions are drawn in Section 6.