Mark J. Huiskes
ABSTRACT
In this paper we review the evaluation of relevance feedback methods for content-based image retrieval systems. We start out by presenting an overview of current common practice, and argue that the evaluation of relevance feedback methods differs from evaluating CBIR systems as a whole. Specifically, we identify the challenging issues that are particular to the evaluation of retrieval employing relevance feedback.
Next, we propose three guidelines to move toward more effective evaluation benchmarks. We focus particularly on assessing feedback methods more directly in terms of their goal of identifying the relevant target class with a small number of samples, and show how to compensate for query targets of varying difficulty by measuring efficiency at generalization.
- L. Armitage and P. Enser. Analysis of user need in image archives. Journal of Information Science, 23(4):287--299, 1997.]]Google Scholar
Cross Ref
- Y. Chen, X. Zhou, and T. Huang. One-class SVM for learning in image retrieval. Proc. IEEE ICIP 2001, Thessaloniki, Greece, 1:34--37, 2001.]]Google Scholar
- G. Ciocca and R. Schettini. A relevance feedback mechanism for content-based image retrieval. Information Processing and Management, 35(5):605--632, 1999.]] Google ScholarDigital Library
- P. Clough, H. Mueller, and M. Sanderson. The clef cross language image retrieval track (imageclef) 2004. In Fifth Workshop of the Cross-Language Evaluation Forum (CLEF 2004), LNCS, New York, NY, USA, 2004. ACM Press.]] Google ScholarDigital Library
- I. Cox, M. Miller, T. Minka, and T. Papathomas. The Bayesian image retrieval system, PicHunter: Theory, implementation, and psychophysical experiments. IEEE Trans. Image Processing, 9(1):20--37, 2000.]] Google ScholarDigital Library
- R. Datta, J. Li, and J. Z. Wang. Content-based image retrieval: approaches and trends of the new age. In MIR '05: Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval, pages 253--262, New York, NY, USA, 2005. ACM Press.]] Google ScholarDigital Library
- A. Dimai. Assessment of effectiveness of content based image retrieval systems. In VISUAL '99: Proceedings of the Third International Conference on Visual Information and Information Systems, pages 525--532, London, UK, 1999. Springer-Verlag.]] Google ScholarDigital Library
- A. Dong and B. Bhanu. A new semi-supervised em algorithm for image retrieval. cvpr, 02:662, 2003.]]Google Scholar
- P. Enser. Query analysis in a visual information retrieval context. Journal of Document and Text Management, 01(1):662, 1993.]]Google Scholar
- M. Everingham, A. Zisserman, C. K. I. Williams, and L. Van Gool. The PASCAL Visual Object Classes Challenge 2006 (VOC2006) Results. http://www.pascalnetwork.org/challenges/VOC/voc2006/results.pdf, 2006.]]Google Scholar
- C. Fluhr, P.-A. Moéllic, and P. Hede. Usage-oriented multimedia information retrieval technological evaluation. In MIR '06: Proceedings of the 8th ACM international workshop on Multimedia information retrieval, pages 301--306, New York, NY, USA, 2006. ACM Press.]] Google ScholarDigital Library
- P. H. Gosselin and M. Cord. Semantic kernel updating for content-based image retrieval. In ISMSE '04: Proceedings of the IEEE Sixth International Symposium on Multimedia Software Engineering (ISMSE'04), pages 537--544, Washington, DC, USA, 2004. IEEE Computer Society.]] Google ScholarDigital Library
- Y. Hastie, R. Tibshirani, and J. Friedman. Elements of Statistical Learning -- Datamining, Inference, and Prediction . Springer, 2001.]]Google Scholar
- J. He, H. Tong, M. Li, H.-J. Zhang, and C. Zhang. Mean version space: a new active learning method for content-based image retrieval. In MIR '04: Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval, pages 15--22, New York, NY, USA, 2004. ACM Press.]] Google ScholarDigital Library
- X. He, W.-Y. Ma, O. King, M. Li, and H. Zhang. Learning and inferring a semantic space from user's relevance feedback for image retrieval. In MULTIMEDIA '02: Proceedings of the tenth ACM international conference on Multimedia, pages 343--346, New York, NY, USA, 2002. ACM.]] Google ScholarDigital Library
- D. R. Heisterkamp and J. Peng. Kernel vector approximation files for relevance feedback retrieval in large image databases. Multimedia Tools Appl., 26(2):175--189, 2005.]] Google ScholarDigital Library
- C. Hoi and M. Lyu. Biased support vector machine for relevance feedback in image retrieval. In Proc. Intl. Joint Conf. on Neural Networks, Budapest, Hungary, pages 3189--3194, 2004.]]Google Scholar
- C.-H. Hoi and M. R. Lyu. A novel log-based relevance feedback technique in content-based image retrieval. In MULTIMEDIA '04: Proceedings of the 12th annual ACM international conference on Multimedia, pages 24--31, New York, NY, USA, 2004. ACM Press.]] Google ScholarDigital Library
- D. P. Huijsmans and N. Sebe. How to complete performance graphs in content-based image retrieval: Add generality and normalize scope. IEEE Trans. Pattern Anal. Mach. Intell., 27(2):245--251, 2005.]] Google ScholarDigital Library
- M. J. Huiskes. Aspect-based relevance learning for image retrieval. In W. Leow, editor, Proceedings of CIVR05, LNCS 3568, pages 639--649. Springer, 2005.]] Google ScholarDigital Library
- M. J. Huiskes. Image searching and browsing by active aspect-based relevance learning. In Proceedings of CIVR06, LNCS 4071, pages 211--220. Springer, 2006.]] Google ScholarDigital Library
- C. H. C. Leung and H. H.-S. Ip. Benchmarking for content-based visual information search. In VISUAL'00: Proceedings of the 4th International Conference on Advances in Visual Information Systems, pages 442--456, London, UK, 2000. Springer-Verlag.]] Google ScholarDigital Library
- M. S. Lew, N. Sebe, C. Djeraba, and R. Jain. Content-based multimedia information retrieval: State of the art and challenges. ACM Trans. Multimedia Comput. Commun. Appl., 2(1):1--19, 2006.]] Google ScholarDigital Library
- Y.-Y. Lin, T.-L. Liu, and H.-T. Chen. Semantic manifold learning for image retrieval. In MULTIMEDIA '05: Proceedings of the 13th annual ACM international conference on Multimedia, pages 249--258, New York, NY, USA, 2005. ACM Press.]] Google ScholarDigital Library
- J. Luo and M. A. Nascimento. Content-based sub-image retrieval using relevance feedback. In MMDB '04: Proceedings of the 2nd ACM international workshop on Multimedia databases, pages 2--9, New York, NY, USA, 2004. ACM Press.]] Google ScholarDigital Library
- S. Marchand-Maillet. The Benchathlon Network. http://www.benchathlon.net, 2005.]]Google Scholar
- S. Marchand-Maillet and M. Worring. Benchmarking image and video retrieval: an overview. In MIR '06: Proceedings of the 8th ACM international workshop on Multimedia information retrieval, pages 297--300, New York, NY, USA, 2006. ACM Press.]] Google ScholarDigital Library
- G. Marchionini. Human performance measures for video retrieval. In MIR '06: Proceedings of the 8th ACM international workshop on Multimedia information retrieval, pages 307--312, New York, NY, USA, 2006. ACM Press.]] Google ScholarDigital Library
- M. Markkula and E. Sormunen. Searching for photos - journalists - practices in pictorial ir, 1998.]]Google Scholar
- H. Müller, S. Marchand-Maillet, and T. Pun. The truth about corel - evaluation in image retrieval. In CIVR '02: Proceedings of the International Conference on Image and Video Retrieval, pages 38--49, London, UK, 2002. Springer-Verlag.]] Google ScholarDigital Library
- H. Müller, W. Müller, D. M. Squire, S. Marchand-Maillet, and T. Pun. Performance evaluation in content-based image retrieval: overview and proposals. Pattern Recogn. Lett., 22(5):593--601, 2001.]] Google ScholarDigital Library
- M. Naphade, J. R. Smith, J. Tesic, S.-F. Chang, W. Hsu, L. Kennedy, A. Hauptmann, and J. Curtis. Large-scale concept ontology for multimedia. IEEE MultiMedia, 13(3):86--91, 2006.]] Google ScholarDigital Library
- J. Peng. Multi-class relevance feedback content-based image retrieval. Comput. Vis. Image Underst., 90(1):42--67, 2003.]] Google ScholarDigital Library
- V. Raghavan, P. Bollmann, and G. S. Jung. A critical investigation of recall and precision as measures of retrieval system performance. ACM Trans. Inf. Syst., 7(3):205--229, 1989.]] Google ScholarDigital Library
- Y. Rui, T. Huang, M. Ortega, and S. Mehrotra. Relevance feedback: A power tool for interactive content-based image retrieval. IEEE Trans. Circuits and Systems for Video Technology, 8(5):644--655, 1998.]] Google ScholarDigital Library
- L. Schamber, M. Eisenberg, and M. S. Nilan. A re-examination of relevance: toward a dynamic, situational definition. Inf. Process. Manage., 26(6):755--776, 1990.]] Google ScholarDigital Library
- A. F. Smeaton, P. Over, and W. Kraaij. Evaluation campaigns and trecvid. In MIR '06: Proceedings of the 8th ACM international workshop on Multimedia information retrieval, pages 321--330, New York, NY, USA, 2006. ACM Press.]] Google ScholarDigital Library
- A. Smeulders, M. Worring, S. Santini, and R. Jain. Content-based image retrieval at the end of the early years. IEEE Trans. on Pattern Analysis and Machine Intelligence, 22(12):20--37, 2000.]] Google ScholarDigital Library
- J. Smith, M. Naphade, A. Natsev, and J. Tesic. Multimedia research challenges for industry. Proceedings of the International Conference on Image and Video Retrieval (CIVR 2005), Singapore., Lecture Notes in Computer Science(3568):28--37, 2005.]] Google ScholarDigital Library
- C. G. M. Snoek, M. Worring, J. C. van Gemert, J.-M. Geusebroek, and A. W. M. Smeulders. The challenge problem for automated detection of 101 semantic concepts in multimedia. In MULTIMEDIA '06: Proceedings of the 14th annual ACM international conference on Multimedia, pages 421--430, New York, NY, USA, 2006. ACM Press.]] Google ScholarDigital Library
- K. Sparck Jones and C. van Rijsbergen. Report on the need for and provision of an ideal information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975.]]Google Scholar
- D. M. Squire, W. Müller, H. Müller, and T. Pun. Content-based query of image databases: inspirations from text retrieval. Pattern Recogn. Lett., 21(13-14):1193--1198, 2000.]] Google ScholarDigital Library
- K. Tieu and P. Viola. Boosting image retrieval. International Journal of Computer Vision, 56(1/2):17--36, 2004.]] Google ScholarDigital Library
- S. Tong and E. Chang. Support vector machine active learning for image retrieval. In MULTIMEDIA '01: Proceedings of the ninth ACM international conference on Multimedia, pages 107--118, New York, NY, USA, 2001. ACM Press.]] Google ScholarDigital Library
- E. Voorhees. The philosophy of information retrieval evaluation. In Proceedings of the The Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems, pages 355--370, 2001.]] Google ScholarDigital Library
- T. Westerveld. Trecvid as a re-usable test-collection for video retrieval. In Proceedings of the Multimedia Information Retrieval Workshop 2005, Salvador, Brazil, August 2005.]]Google Scholar
- Y. Wu and A. Zhang. A feature re-weighting approach for relevance feedback in image retrieval, 2002.]]Google Scholar
- T. Yoshizawa and H. Schweitzer. Long-term learning of semantic grouping from relevance-feedback. In MIR'04: Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval, pages 165--172, New York, NY, USA, 2004. ACM Press.]] Google ScholarDigital Library
- H. Zhang, C. Zheng, M. Li, and Z. Su. Relevance feedback and learning in content-based image search. WWW: Internet and web information systems, 6:131--155, 2003.]] Google ScholarDigital Library
- X. Zhou and T. Huang. Relevance feedback in image retrieval: a comprehensive review. ACM Multimedia Systems Journal, 8(6):536--544, April 2003.]]Google ScholarCross Ref
Index Terms
- Performance evaluation of relevance feedback methods
Recommendations
The effect of low-level image features on pseudo relevance feedback
Relevance feedback (RF) is a technique popularly used to improve the effectiveness of traditional content-based image retrieval systems. However, users must provide relevant and/or irrelevant images as feedback for their queries, which is a tedious ...
Relevance feedback methods for logo and trademark image retrieval on the web
SAC '06: Proceedings of the 2006 ACM symposium on Applied computingRelevance feedback is the state-of-the-art approach for adjusting query results to the needs of the users. This work extends the existing framework of image retrieval with relevance feedback on the Web by incorporating text and image content into the ...
Multimodal retrieval with relevance feedback based on genetic programming
This paper presents a framework for multimodal retrieval with relevance feedback based on genetic programming. In this supervised learning-to-rank framework, genetic programming is used for the discovery of effective combination functions of (multimodal)...
Comments