Experiments with the 4 Query Sets of WebCLEF 2006

Abstract

In the WebCLEF mixed monolingual retrieval task of the Cross-Language Evaluation Forum (CLEF) 2006, the system was given 1939 known-item queries, and the goal was to find the desired page in the 82GB EuroGOV collection (3.4 million pages crawled from government sites of 27 European domains). The 1939 queries included 124 new manually-created queries, 195 manually-created queries from last year, and two sets of 810 automatically-generated queries. In our experiments, the results on the automatically-generated queries were not always predictive of the results on the manually-created queries; in particular, our title-weighting and duplicate-filtering techniques were fairly effective on the manually-created queries but were detrimental on the automatically-generated queries. Further investigation uncovered serious encoding issues with the automatically-generated queries; for instance, the queries labelled as Greek actually used Latin characters.