Abstract
We investigate possible assessment trends and inconsistencies within the collected relevance assessments of the INEX’02 test collection in order to provide a critical analysis of the employed relevance criterion and assessment procedure for the evaluation of content-oriented XML retrieval approaches.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Voorhees, E.M., Harman, D.K. (eds.): The tenth Text Retrieval Conference (TREC-2001), Gaithersburg, MD, USA. NIST (2002)
Harter, S.P.: Variations in Relevance Assessments and the Measurement of Retrieval Effectiveness. Journal of the American Society for Information Science 47(1), 37–47 (1996)
Saracevic, T.: http://www.scils.rutgers.edu/~tefko/Courses/610/Lectures/
Vakkari, P., Hakala, N.: Changes in Relevance Criteria and Problem Stages in Task Performance. Journal of Documentation 56(5), 540–562 (2000)
Schamber, L.: Relevance and Information Behaviour. Annual Review of Information Science and Technology (ARIST) 29, 3–48 (1994)
Borlund, P.: The Concept of Relevance in IR. Journal of the American Society for Information Science 54(10), 913–925 (2003)
Cosijn, E., Ingwersen, P.: Dimensions of Relevance. Information Processing and Management 36, 533–550 (2000)
Saracevic, T.: Relevance Reconsidered. In: Proceedings of the 2nd International Conference on Conceptions of Library and Information Science (COLIS), Copenhagen, pp. 201–218 (1996)
Mizzaro, S.: Relevance: the whole history. Journal of the American Society for Information Science 48(9), 810–832 (1997)
Tang, R., Shaw, W.M., Vevea, J.L.: Towards the identification of the optimal numbers of relevance categories. Journal of the American Society for Information Science 50(3), 254–264 (1999)
Kekäläinen, J., Järvelin, K.: Using graded relevance assessments in IR evaluation. Journal of the American Society for Information Science 53(13), 1120–1129 (2002)
Voorhees, E.: Evaluation by highly relevant documents. In: Proceedings of the 24th ACM-SIGIR conference on research and development in information retrieval, New York, pp. 74–82 (2001)
Vorhees, E.M.: Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness. In: Proceedings of the 21st ACM-SIGIR conference on research and development in information retrieval, Melbourne, pp. 315–323 (1998)
Fuhr, N., Lalmas, M., Kazai, G., Gövert, N. (eds.): Proceedings of the first workshop of the INitiative for the Evaluation of XML Retrieval (INEX), Dagstuhl. ERCIM workshop proceedings (2003)
Järvelin, K., Kekäläinen, J.: IR evaluation methods for retrieving highly relevant documents. In: Proceedings of the 23rd ACM-SIGIR conference on research and development in information retrieval, Athens, pp. 41–48 (2000)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kazai, G., Masood, S., Lalmas, M. (2004). A Study of the Assessment of Relevance for the INEX’02 Test Collection. In: McDonald, S., Tait, J. (eds) Advances in Information Retrieval. ECIR 2004. Lecture Notes in Computer Science, vol 2997. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-24752-4_22
Download citation
DOI: https://doi.org/10.1007/978-3-540-24752-4_22
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-21382-6
Online ISBN: 978-3-540-24752-4
eBook Packages: Springer Book Archive