ABSTRACT
One of the most challenging aspects of designing an interactive information retrieval (IIR) study is the development of search tasks. In this paper, we present preliminary results of a study designed to evaluate a set of search tasks that were developed for use in IIR studies. We created 20 search tasks using five levels of cognitive complexity and four domains, and conducted a laboratory evaluation of these tasks with 48 undergraduate subjects. We describe preliminary results from an analysis of data from 24 subjects for 10 search tasks. Initial results show that, in general, as cognitive complexity increased, subjects issued more queries, clicked on more search results, viewed more URLs and took more time to complete the task. Subjects' expected and experienced difficulty ratings of tasks generally increased as cognitive complexity increased with some exceptions. When subjects were asked to rank tasks according to difficulty and engagement, tasks with higher cognitive complexity were rated as more difficult than tasks with lower cognitive complexity, but not necessarily as more engaging. These preliminary results suggest that behaviors and ratings are fairly consistent with the differences one might expect among the search tasks and provide initial evidence of the usefulness of these tasks in IIR studies.
- Anderson, L. W. & Krathwohl, D. A. (2001). A taxonomy for learning, teaching and assessing: A revision of Bloom's taxonomy of educational objectives. New York: Longman.Google Scholar
- Arguello, J., Wu, W. C., Kelly, D., & Edwards, A. (2012). Task complexity, vertical display and user interaction in aggregated search. Proceedings of SIGIR '12, Portland, OR. Google ScholarDigital Library
- Borlund, P. (2003). The IIR evaluation model: A framework for the evaluation of interactive information retrieval systems. Information Research, 8(3), paper 152.Google Scholar
- Byström, K. & Jäärvelin, K. (1995). Task complexity affects information seeking and use. IP&M, 31(2), 191--213. Google ScholarDigital Library
- Jansen, B. J., Booth, D., & Smith, D. (2009). Using the taxonomy of cognitive learning to model online searching. IP&M, 45, 643--663. Google ScholarDigital Library
- Li, Y. & Belkin, N. J. (2008). A faceted approach to conceptualizing tasks in information seeking. IP&M, 44, 1822--1837. Google ScholarDigital Library
- Toms, E. G. (2011). Task-based information searching and retrieval. In I. Ruthven & D. Kelly (Eds.) Interactive Information Seeking, Behaviour and Retrieval. Facet Publishing, 43--59.Google Scholar
- Vakkari, P. (2003). Task-based information searching. ARIST, 37, 413--464.Google Scholar
- Whittaker, S., Terveen, L., & Nardi, B. (2000). Let's stop pushing the envelope and start addressing it: a reference task agenda for HCI. Human Computer Interaction, 15, 75--106. Google ScholarDigital Library
- Wildemuth, B. M. & Freund, L. (2009) Search tasks and their role in studies of search behaviors. Paper presented at the Third Annual Workshop on HCIR, Washington D.C.Google Scholar
Index Terms
Grannies, tanning beds, tattoos and NASCAR: evaluation of search tasks with varying levels of cognitive complexity
Recommendations
Development and Evaluation of Search Tasks for IIR Experiments using a Cognitive Complexity Framework
ICTIR '15: Proceedings of the 2015 International Conference on The Theory of Information RetrievalOne of the most challenging aspects of designing interactive information retrieval (IIR) experiments with users is the development of search tasks. We describe an evaluation of 20 search tasks that were designed for use in IIR experiments and developed ...
An empirical study of interest, task complexity, and search behaviour on user engagement
Highlights- User engagement (UE) is an important outcome measure in interactive information retrieval.
AbstractUser engagement has become an important outcome measure in interactive information retrieval (IIR) research, as commercial (e.g., search engines and e-commerce companies) and educational (e.g., libraries) enterprises focus on capturing ...
Representing Tasks with a Graph-Based Method for Supporting Users in Complex Search Tasks
CHIIR '23: Proceedings of the 2023 Conference on Human Information Interaction and RetrievalDespite the considerable advancements in modern search systems for assisting users in search tasks of varying types, support for complex tasks that call for multi-round interactions remains challenging. Identifying users’ tasks is essential to ...
Comments