Low-Cost Preference Judgment via Ties

Abstract

Preference judgment, as an alternative to graded judgment, leads to more accurate labels and avoids the need to define relevance levels. However, it also requires a larger number of judgments. Prior research has successfully reduced that number to $\mathcal {O}(N_d\,\log {N_d})$ for $N_d$ documents by assuming transitivity, which is still too expensive in practice. In this work, by analytically deriving the number of judgments and by empirically simulating the ground-truth ranking of documents from Trec Web Track, we demonstrate that the number of judgments can be dramatically reduced when allowing for ties.