Laurence A. F. Park
ABSTRACT
Information retrieval metrics that provide uncertainty intervals when faced with unjudged documents, such as Rank-Biased Precision (RBP), provide us with an indication of the upper and lower bound of the system score. Unfortunately, the uncertainty is disregarded when examining the mean over a set of queries. In this article, we examine the distribution of the uncertainty per query and averaged over all queries, under the assumption that each unjudged document has the same probability of being relevant. We also derive equations for the mean, variance, and distribution of Mean RBP uncertainty. Finally, the impact of our assumption is assessed using simulation. We find that by removing the assumption of equal probability of relevance, we obtain a scaled form of the previously defined mean and standard deviation for the distribution of Mean RBP uncertainty.
- Chris Buckley and Ellen M Voorhees. Retrieval evaluation with incomplete information. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 25--32. ACM, 2004. Google Scholar
Digital Library
- Alistair Moffat and Justin Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inf. Syst., 27(1):2:1--2:27, December 2008. Google ScholarDigital Library
- Laurence A. F. Park and Yuye Zhang. On the distribution of user persistence for rank-biased precision. In Proceedings of the 12th Australasian document computing symposium, pages 17--24, 2007.Google Scholar
- Kenneth T Wallenius. Biased sampling; the noncentral hypergeometric probability distribution. Technical report, DTIC Document, 1963.Google Scholar
- Yuye Zhang, Laurence A. F. Park, and Alistair Moffat. Click-based evidence for decaying weight distributions in search effectiveness metrics. Information Retrieval, 13(1):46--69, 2010. Google ScholarDigital Library
- Justin Zobel. How reliable are the results of large-scale information retrieval experiments? In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, pages 307--314. ACM, 1998. Google ScholarDigital Library
Recommendations
From a User Model for Query Sessions to Session Rank Biased Precision (sRBP)
ICTIR '19: Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information RetrievalTo satisfy their information needs, users usually carry out searches on retrieval systems by continuously trading off between the examination of search results retrieved by under-specified queries and the refinement of these queries through ...
An Investigation into the Use of Document Scores for Optimisation over Rank-Biased Precision
Information Retrieval TechnologyAbstractWhen a Document Retrieval system receives a query, a Relevance model is used to provide a score to each document based on its relevance to the query. Relevance models have parameters that should be tuned to optimise the accuracy of the relevance ...
Sketch retrieval and relevance feedback with biased SVM classification
This paper proposes an effective approach for content-based sketch retrieval. It addresses three characteristics as follows. Firstly, both structural relations and global shape descriptors are combined to represent sketch content. Secondly, feature ...
Comments