doi:10.1016/j.ipm.2007.08.002
Copyright © 2007 Elsevier Ltd All rights reserved.
Contextual factors affecting the utility of surrogates within exploratory search
Ian Ruthven
, a,
, Mark Bailliea, Leif Azzopardia, Ralf Bieriga, Emma Nicola, Simon Sweeneya and Murat Yacikia
aUniversity of Strathclyde, Department of Computer and Information Sciences, 26 Richmond Street, Glasgow G1 1XH, United Kingdom
Received 21 December 2006;
revised 9 August 2007;
accepted 10 August 2007.
Available online 19 September 2007.
References and further reading may be available for this article. To view references and further reading you must
purchase this article.
Abstract
In this paper we investigate how information surrogates might be useful in exploratory search and what information it is useful for a surrogate to contain. By comparing assessments based on artificially created information surrogates, we investigate the effect of the source of information, the quality of an information source and the date of information upon the assessment process. We also investigate how varying levels of topical knowledge, assessor confidence and prior expectation affect the assessment of information surrogates. We show that both types of contextual information affect how the information surrogates are judged and what actions are performed as a result of the surrogates.
Keywords: Surrogates; Exploratory search; Ciqa; Relevance assessment
Fig. 1. Google surrogates.
Fig. 2. ciqa topic number 32.
Fig. 4. Answer template in Interaction Form 2.
Fig. 5. Questions on quality of answer set.
Fig. A.1. “High quality” sources.
Fig. A.2. “Low quality” sources.
Fig. A.3. Interaction Form 1.
Fig. A.4. Interaction Form 2.
Table 1.
Summary of variables and measurement

Table 2.
Sub-questions and entity count by relationship type

Highest value shown in bold.
Table 3.
Distribution of answers to answer quality variable

Highest value shown in bold.
Table 4.
Distribution of answers to answer expectation variable

Highest value shown in bold.
Table 5.
Distribution of answers to answer expectation variable

Highest value shown in bold.
Table 6.
Distribution of answers to answer quality variable

Highest value shown in bold.
Table 7.
Distribution of predicted next actions over answer quality

Highest value within each answer assessment category shown in bold.
Table 8.
Percentage of supported/unsupported answers rated under different assessment categories

Highest value within each answer assessment category shown in bold.
Table 9.
Predicted next actions for supported/unsupported answers

Highest value within each predicted next action category shown in bold.
Table 10.
Percentage of recent/older answers rated under different assessment categories

Highest value within each answer assessment category shown in bold.
Table 11.
Predicted next actions for recent/older answers

Highest value within each predicted next action category shown in bold.
Table 12.
Percentage of answers presented as from good/weak sources under different assessment categories

Highest value within each answer assessment category shown in bold.
Table 13.
Predicted next actions for answers presented as coming from good/weak sources

Highest value within each predicted next action category shown in bold.
Table 14.
Summary of most likely predicted next actions

Table 15.
Distribution of answers over predicted next action

Highest value within each answer assessment category shown in bold.
Table 16.
Knowledge and answer quality

Highest value within each answer assessment category shown in bold.
a n in this table and subsequent tables in this section refer to the number of answers not the number of assessors in each category.
Table 17.
Knowledge and next action

Highest value within each topical knowledge category shown in bold.
Table 18.
Pre-assessment confidence and answer quality

Highest value within each answer assessment category shown in bold.
Table 19.
Pre-assessment confidence and next action

Highest value within each predicted next action category shown in bold.
Table 20.
Prior expectation and answer quality

Highest value within each answer assessment category shown in bold.
Table 21.
Prior expectation and next action

Highest value within each predicted next action category shown in bold.