Abstract

Exploratory search increasingly becomes an important research topic. Our interests focus on task-based information exploration, a specific type of exploratory search performed by a range of professional users, such as intelligence analysts. In this paper, we present an evaluation framework designed specifically for assessing and comparing performance of innovative information access tools created to support the work of intelligence analysts in the context of task-based information exploration. The motivation for the development of this framework came from our needs for testing systems in task-based information exploration, which cannot be satisfied by existing frameworks. The new framework is closely tied with the kind of tasks that intelligence analysts perform: complex, dynamic, and multiple facets and multiple stages. It views the user rather than the information system as the center of the evaluation, and examines how well users are served by the systems in their tasks. The evaluation framework examines the support of the systems at users’ major information access stages, such as information foraging and sense-making. The framework is accompanied by a reference test collection that has 18 tasks scenarios and corresponding passage-level ground truth annotations. To demonstrate the usage of the framework and the reference test collection, we present a specific evaluation study on CAFÉ, an adaptive filtering engine designed for supporting task-based information exploration. This study is a successful use case of the framework, and the study indeed revealed various aspects of the information systems and their roles in supporting task-based information exploration.

Keywords: Task-based information exploration; Exploratory search; Evaluation framework; Adaptive filtering; CAFE; User study; Intelligence analysts

Article Outline

1. Introduction

2. Related work

3. An evaluation framework for task-based information exploration

3.1. Reasons for developing the framework

3.2. Methodology

3.3. Metrics

4. A reference test collection

4.1. Document set

4.2. Simulated task scenarios

4.3. Ground truth assessments

5. Evaluating CAFÉ: A sample study

5.1. Data selection and preparation

5.2. Subjects

5.3. Experimental and baseline systems

5.4. Experimental procedure

6. Result analysis

6.1. System comparison based on output ranked lists

6.2. System comparison based on usage profiles

6.3. System comparison based on user feedbacks

7. General discussion and conclusion

Acknowledgements

References