research-article

A Systematic Review of Cost, Effort, and Load Research in Information Search and Retrieval, 1972–2020

Authors:
Molly McGregor

University of Strathclyde

University of Strathclyde

0000-0002-3329-3829
View Profile

,
Leif Azzopardi

University of Strathclyde

University of Strathclyde

0000-0002-6900-0557
View Profile

,
Martin Halvey

University of Strathclyde

University of Strathclyde

0000-0001-6387-8679
View Profile

Authors Info & Claims

ACM Transactions on Information Systems Volume 42 Issue 1Article No.: 2pp 1–39https://doi.org/10.1145/3583069

Published:18 August 2023Publication History

ACM Transactions on Information Systems

Abstract

During the information search and retrieval (ISR) process, user-system interactions such as submitting queries, examining results, and engaging with information impose some degree of demand on the user’s resources. Within ISR, these demands are well recognised, and numerous studies have demonstrated that the cost, effort, and load (CEL) experienced during the search process are affected by a variety of factors. Despite this recognition, there is no universally accepted definition of the constructs of CEL within the field of ISR. Ultimately, this has led to problems with how these constructs have been interpreted and subsequently measured. This systematic review contributes a synthesis of literature, summarising key findings relating to how researchers have been defining and measuring CEL within ISR over the past 50 years. After manually screening 1,109 articles, we detailed and analysed 91 articles which examine CEL within ISR. The discussion focuses on comparing the similarities and differences between CEL definitions and measures before identifying the limitations of the current state of the nomenclature. Opportunities for future research are also identified. Going forward, we propose a CEL taxonomy that integrates the relationships between CEL and their related constructs, which will help focus and disambiguate future research in this important area.

REFERENCES

[1] Amento Brian, Hill Will, Hix Deborah, Schulman Robert, and Terveen Loren. 2003. Experiments in social data mining. ACM Transactions on Computer-Human Interaction 10, 1 (2003), 54–85. Google ScholarDigital Library
[2] Arapakis Ioannis, Leiva Luis A., and Cambazoglu B. Barla. 2015. Know your onions: Understanding the user experience with the knowledge module in web search. In Proceedings of the International Conference on Information and Knowledge Management. 1695–1698. Google ScholarDigital Library
[3] Arguello Jaime and Choi Bogeum. 2019. The effects of working memory, perceptual speed, and inhibition in aggregated search. ACM Transactions on Information Systems 37, 3 (2019), Article 36, 34 pages. Google ScholarDigital Library
[4] Arguello Jaime and Crescenzi Anita. 2019. Using principal component analysis to better understand behavioral measures and their effects. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR’19). ACM, New York, NY, 177–184. Google ScholarDigital Library
[5] Ariza Ferran, Kalra Dipak, and Potts Henry W. W.. 2015. How do clinical information systems affect the cognitive demands of general practitioners? Usability study with a focus on cognitive workload. Journal of Innovation in Health Informatics 22, 4 (2015), 379–390. Google ScholarCross Ref
[6] Avula Sandeep, Arguello Jaime, Capra Robert, Dodson Jordan, Huang Yuhui, and Radlinski Filip. 2019. Embedding search into a conversational platform to support collaborative search. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval (CHIIR’19).15–23. Google ScholarDigital Library
[7] Azzopardi Leif. 2011. The economics in interactive information retrieval. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’11).15–24. Google ScholarDigital Library
[8] Azzopardi Leif, Kelly Diane, and Brennan Kathy. 2013. How query cost affects search behavior categories and subject descriptors. In Proceedings of the 36th Annual ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’13). 23–32. Google ScholarDigital Library
[9] Azzopardi Leif, Thomas Paul, and Craswell Nick. 2018. Measuring the utility of search engine result pages: An information foraging based measure. In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’18).605–614. Google ScholarDigital Library
[10] Azzopardi Leif and Zuccon Guido. 2016. An analysis of the cost and benefit of search interactions. In Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval (ICTIR’16). ACM, New York, NY, 59–68. Google ScholarDigital Library
[11] Bailey Earl and Kelly Diane. 2011. Is amount of effort a better predictor of search success than use of specific search tactics? Proceedings of the American Society for Information Science and Technology 48, 1 (2011), 1–10. Google ScholarCross Ref
[12] Boksem Maarten A. S. and Tops Mattie. 2008. Mental fatigue: Costs and benefits. Brain Research Reviews 59, 1 (2008), 125–139. Google ScholarCross Ref
[13] Bota Horatiu, Zhou Ke, and Jose Joemon M.. 2016. Playing your cards right: The effect of entity cards on search behaviour and workload. In Proceedings of the 2016 ACM Conference on Human Information Interaction and Retrieval (CHIIR’16). ACM, New York, NY, 131–140. Google ScholarDigital Library
[14] Brennan Kathy, Kelly Diane, and Arguello Jaime. 2014. The effect of cognitive abilities on information search for tasks of varying levels of complexity. In Proceedings of the 5th Information Interaction in Context Symposium (IIiX’14). ACM, New York, NY, 165–174. Google ScholarDigital Library
[15] Brünken Roland, Plass Jan L., and Leutner Detlev. 2003. Direct measurement of cognitive load in multimedia learning. Educational Psychologist 38, 1 (2003), 53–61. Google ScholarCross Ref
[16] Brutlag Jake. 2009. Speed Matters for Google Web Search. Retrieved February 19, 2023 from https://www.services.google.com/fh/files/blogs/google_delayexp.pdf.Google Scholar
[17] Capra Robert, Arguello Jaime, O’Brien Heather, Li Yuan, and Choi Bogeum. 2018. The effects of manipulating task determinability on search behaviors and outcomes. In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’18).445–454. Google ScholarDigital Library
[18] Capra Rob, Arguello Jaime, and Zhang Yinglong. 2017. The effects of search task determinability on search behaviour. In Advances in Information Retrieval. Lecture Notes in Computer Science, Vol. 10193. Springer, 108–121. Google ScholarCross Ref
[19] Capra Robert, Marchionini Gary, Oh Jung Sun, Stutzman Fred, and Zhang Yan. 2007. Effects of structure and interaction style on distinct search tasks. In Proceedings of the ACM International Conference on Digital Libraries.442–451. Google ScholarDigital Library
[20] Charles Rebecca L. and Nixon Jim. 2019. Measuring mental workload using physiological measures: A systematic review. Applied Ergonomics 74 (2019), 221–232. Google ScholarCross Ref
[21] Chen Hsinchun, Fan Haiyan, Chau Michael, and Zeng Daniel. 2003. Testing a cancer meta spider. International Journal of Human Computer Studies 59, 5 (2003), 755–776. Google ScholarDigital Library
[22] Chen Ye, Liu Yiqun, Zhou Ke, Wang Meng, Zhang Min, and Ma Shaoping. 2015. Does vertical bring more satisfaction? Predicting search satisfaction in a heterogeneous environment. In Proceedings of the International Conference on Information and Knowledge Management.1581–1590. Google ScholarDigital Library
[23] Chevalier Aline and Kicka Maud. 2006. Web designers and web users: Influence of the ergonomic quality of the web site on the information search. International Journal of Human Computer Studies 64, 10 (2006), 1031–1048. Google ScholarDigital Library
[24] Choi Bogeum, Capra Robert, and Arguello Jaime. 2019. The effects of working memory during search tasks of varying complexity. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval (CHIIR’19). ACM, New York, NY, 261–265. Google ScholarDigital Library
[25] Choi Hwan Hee, Merriënboer Jeroen J. G. van, and Paas Fred. 2014. Effects of the physical environment on cognitive load and learning: Towards a new model of cognitive load. Educational Psychology Review 26, 2 (2014), 225–244. Google ScholarCross Ref
[26] Claypool M., Le P., Wased M., and Brown D.. 2001. Implicit interest indicators. In Proceedings of the International Conference on Intelligent User Interfaces (IUI’01).33–40. Google ScholarDigital Library
[27] Cole Michael J., Gwizdka Jacek, Liu Chang, and Belkin Nicholas J.. 2011. Dynamic assessment of information acquisition effort during interactive search. Proceedings of the American Society for Information Science and Technology 48, 1 (2011), 1–10. Google ScholarCross Ref
[28] Cole Michael J., Gwizdka Jacek, Liu Chang, Belkin Nicholas J., and Zhang Xiangmin. 2013. Inferring user knowledge level from eye movement patterns. Information Processing and Management 49, 5 (2013), 1075–1091. Google ScholarDigital Library
[29] Cooper Michael D.. 1972. A cost model for evaluating information retrieval systems. Journal of the American Society for Information Science 23, 5 (1972), 306–312. Google ScholarCross Ref
[30] Jong Ton de. 2010. Cognitive load theory, educational research, and instructional design: Some food for thought. Instructional Science 38 (2010), 105–134. Google ScholarCross Ref
[31] Winter J. C.F. de. 2014. Controversy in human factors constructs and the explosive use of the NASA-TLX: A measurement perspective. Cognition, Technology and Work 16, 3 (2014), 289–297. Google ScholarDigital Library
[32] Dennis Alan R. and Taylor Nolan J.. 2006. Information foraging on the web: The effects of “acceptable” Internet delays on multi-page information search behavior. Decision Support Systems 42, 2 (2006), 810–824. Google ScholarDigital Library
[33] Dennis Simon, Bruza Peter, and McArthur Robert. 2002. Web searching: A process-oriented experimental study of three interactive search paradigms. Journal of the American Society for Information Science and Technology 53, 2 (2002), 120–133. Google ScholarDigital Library
[34] Stasi Leandro L. Di, Antolí Adoración, Gea Miguel, and Cañas José J.. 2011. A neuroergonomic approach to evaluating mental workload in hypermedia interactions. International Journal of Industrial Ergonomics 41, 3 (2011), 298–304. Google ScholarCross Ref
[35] Dubiel Mateusz, Halvey Martin, Azzopardi Leif, and Daronnat Sylvain. 2020. Interactive evaluation of conversational agents: Reflections on the impact of search task design. In Proceedings of the 2020 ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR’20).85–88. Google ScholarDigital Library
[36] Sarah E. Crudge and Frances C. Johnson. 2007. Using the repertory grid and laddering technique to determine the user’s evaluative model of search engines. Journal of Documentation 63, 2 (2007), 259–280. Google ScholarCross Ref
[37] Edwards Ashlee, Kelly Diane, and Azzopardi Leif. 2015. The impact of query interface design on stress, workload and performance. In Proceedings of the European Conference of Information Retrieval (ECIR’15). 691–702. Google ScholarCross Ref
[38] Egeth Howard and Kahneman Daniel. 1975. Attention and effort. American Journal of Psychology 88, 2 (1975), 339. Google ScholarCross Ref
[39] Feldon David F., Callan Gregory, Juth Stephanie, and Jeong Soojeong. 2019. Cognitive load as motivational cost. Educational Psychology Review 31, 2 (2019), 319–337. Google ScholarCross Ref
[40] Fenichel Carol Hansen. 1981. Online searching: Measures that discriminate among users with different types of experiences. Journal of the American Society for Information Science 32, 1 (1981), 23–32. Google ScholarCross Ref
[41] Flynn T., Holohan P. A., Magson M. S., and Munro J. D.. 1979. Cost effectiveness comparison of online and manual bibliographic information retrieval. Journal of Information Science 1, 2 (1979), 77–84. Google ScholarCross Ref
[42] Fox Steve, Karnawat Kuldeep, Mydland Mark, Dumais Susan, and White Thomas. 2005. Evaluating implicit measures to improve web search. ACM Transactions on Information Systems 23, 2 (2005), 147–168. Google ScholarDigital Library
[43] Gäde Maria, Koolen Marijn, Hall Mark, Bogers Toine, and Petras Vivien. 2021. A manifesto on resource re-use in interactive information retrieval. In Proceedings of the 2021 Conference on Human Information Interaction and Retrieval (CHIIR’21).141–149. Google ScholarDigital Library
[44] Galletta Dennis, Henry Raymond, McCoy Scott, and Polak Peter. 2004. Web site delays: How tolerant are users? Journal of the Association for Information Systems 5, 1 (2004), 1–28. Google ScholarCross Ref
[45] Gerwe Paul and Viles Charles L.. 2000. User effort in query construction and interface selection. In Proceedings of the 5th ACM Conference on Digital Libraries (DL’00). ACM, New York, NY, 246–247. Google ScholarDigital Library
[46] González-Ibáñez Roberto, Varela-Otárola José Luis, and Barrera-Pulgar Carlos. 2016. Evaluating body-centered interactions in an image search task. In Proceedings of the 2016 ACM Conference on Human Information Interaction and Retrieval (CHIIR’16). ACM, New York, NY, 269–272. Google ScholarDigital Library
[47] González-Ibáñez Roberto, Proaño-Ríos Verónica, Fuenzalida Gary, and Martinez-Ramirez Gonzalo. 2017. Effects of a visual representation of search engine results on performance, user experience and effort. Proceedings of the Association for Information Science and Technology 54, 1 (2017), 128–138. Google ScholarCross Ref
[48] Grant Maria J. and Booth Andrew. 2009. A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal 26, 2 (2009), 91–108. Google ScholarCross Ref
[49] Guo Qi and Agichtein Eugene. 2012. Beyond dwell time: Estimating document relevance from cursor movements and other post-click searcher behavior. In Proceedings of the 21st Annual Conference on World Wide Web (WWW’12).569–578. Google ScholarDigital Library
[50] Gwizdka Jacek. 2008. Revisiting search task difficulty: Behavioral and individual difference measures. Proceedings of the American Society for Information Science and Technology 45, 1 (2008), 1–12. Google ScholarCross Ref
[51] Gwizdka Jacek. 2010. Assessing cognitive load on web search tasks. Ergonomics Open Journal 2, 2 (2010), 114–123. Google ScholarCross Ref
[52] Gwizdka Jacek. 2010. Distribution of cognitive load in web search. Journal of the American Society for Information Science and Technology 61, 11 (2010), 2167–2187. Google ScholarCross Ref
[53] Gwizdka Jacek. 2013. Effects of working memory capacity on users’ search effort. In Proceedings of the International Conference on Multimedia, Interaction, Design, and Innovation (MIDI’13). Article 11, 8 pages. Google ScholarDigital Library
[54] Gwizdka Jacek. 2014. Characterizing relevance with eye-tracking measures. In Proceedings of the 5th Information Interaction in Context Symposium (IIiX’14). ACM, New York, NY, 58–67. Google ScholarDigital Library
[55] Gwizdka Jacek. 2017. I can and so I search more: Effects of memory span on search behavior. In Proceedings of the 2017 Conference on Human Information Interaction and Retrieval (CHIIR’17). ACM, New York, NY, 341–344. Google ScholarDigital Library
[56] Gwizdka J. and Cole M. J.. 2011. Least effort? Not if I can search more. In Proceedings of the 5th Workshop on Human-Computer Interaction and Information Retrieval. 1–4. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.306.8297&rep=rep1&type=pdf.Google Scholar
[57] Gwizdka Jacek and Lopatovska Irene. 2009. The role of subjective factors in the information search process. Journal of the American Society for Information Science and Technology 60, 12 (2009), 2452–2464. Google ScholarCross Ref
[58] Gwizdka Jacek and Spence Ian. 2006. What can searching behavior tell us about the difficulty of information tasks? A study of web navigation. Proceedings of the American Society for Information Science and Technology 43, 1 (2006), 1–22. Google ScholarCross Ref
[59] Halvey Martin and Villa Robert. 2014. Evaluating the effort involved in relevance assessments for images. In Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’14). 887–890. Google ScholarDigital Library
[60] Hart Sandra G.. 2006. NASA-task load index (NASA-TLX); 20 years later. Proceedings of the Human Factors and Ergonomics Society 50, 9 (2006), 904–908. Google ScholarCross Ref
[61] Hart Sandra G. and Staveland Lowell E.. 1988. Development of NASA-TLX (task load index): Results of empirical and theoretical research. In Human Mental Workload, Hancock Peter A. and Meshkati Najmedin (Eds.). Advances in Psychology, Vol. 52. North-Holland, 139–183. Google ScholarCross Ref
[62] He Jiyin, Bron Marc, Vries Arjen De, Azzopardi Leif, and Rijke Maarten De. 2015. Untangling result list refinement and ranking quality: A framework for evaluation and prediction. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’15).293–302. Google ScholarDigital Library
[63] Heale Roberta and Twycross Alison. 2015. Validity and reliability in quantitative studies. Evidence-Based Nursing 18, 3 (2015), 66–67. Google ScholarCross Ref
[64] Inzlicht Michael, Shenhav Amitai, and Olivola Christopher Y.. 2018. The effort paradox: Effort is both costly and valued. Trends in Cognitive Sciences 22, 4 (2018), 337–349. Google ScholarCross Ref
[65] Jansen Bernard J., Booth Danielle, and Smith Brian. 2009. Using the taxonomy of cognitive learning to model online searching. Information Processing and Management 45, 6 (2009), 643–663. Google ScholarDigital Library
[66] Jiang Jiepu, Awadallah Ahmed Hassan, Shi Xiaolin, and White Ryen W.. 2015. Understanding and predicting graded search satisfaction. In Proceedings of the 8th ACM International Conference on Web Search and Data Mining (WSDM’15). ACM, New York, NY, 57–66. Google ScholarDigital Library
[67] Jiang Jiepu, He Daqing, and Allan James. 2014. Searching, browsing, and clicking in a search session: Changes in user behavior by task and over time. In Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’14).607–616. Google ScholarDigital Library
[68] Jiang Jiepu, He Daqing, and Allan James. 2017. Comparing in situ and multidimensional relevance judgments. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’17).405–414. Google ScholarDigital Library
[69] Jiang Jiepu, He Daqing, Kelly Diane, and Allan James. 2017. Understanding ephemeral state of relevance. In Proceedings of the 2017 Conference on Human Information Interaction and Retrieval (CHIIR’17).137–146. Google ScholarDigital Library
[70] Jimenez-Molina Angel, Retamal Cristian, and Lira Hernan. 2018. Using psychophysiological sensors to assess mental workload during web browsing. Sensors (Basel) 18, 2 (2018), 458. Google ScholarCross Ref
[71] Kamvar Maryam and Baluja Shumeet. 2008. Query suggestions for mobile search: Understanding usage patterns. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’08). ACM, New York, NY, 1013–1016. Google ScholarDigital Library
[72] Ke Weimao, Sugimoto Cassidy R., and Mostafa Javed. 2009. Dynamicity vs. effectiveness: Studying online clustering for scatter/gather. In Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’09).19–26. Google ScholarDigital Library
[73] Kelly Diane. 2009. Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval 3, 1-2 (2009), 1–224. Google ScholarDigital Library
[74] Kelly Diane and Azzopardi Leif. 2015. How many results per page? A study of SERP size, search behavior and user experience. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’15).183–192. Google ScholarDigital Library
[75] Kelly Diane and Belkin Nicholas J.. 2004. Display time as implicit feedback: Understanding task effects. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’04).377–384.Google Scholar
[76] Kelly Diane and Sugimoto Cassidy R.. 2013. A systematic review of interactive information retrieval evaluation studies, 1967–2006. Journal of the American Society for Information Science and Technology 64, 4 (2013), 745–770. Google ScholarCross Ref
[77] Kim Yong-Mi and Rieh Soo Young. 2006. Dual-task performance as a measure of mental effort in searching a library system and the web. Proceedings of the American Society for Information Science and Technology 42, 1 (2006), 1–18. Google ScholarCross Ref
[78] Kirschner Paul A.. 2002. Cognitive load theory: Implications of cognitive load theory on the design of learning. Learning and Instruction 12, 1 (2002), 1–10. Google ScholarCross Ref
[79] Kiseleva Julia, Williams Kyle, Jiang Jiepu, Awadallah Ahmed Hassan, Crook Aidan C., Zitouni Imed, and Anastasakos Tasos. 2016. Understanding user satisfaction with intelligent assistants. In Proceedings of the 2016 ACM Conference on Human Information Interaction and Retrieval (CHIIR’16). ACM, New York, NY, 121–130. Google ScholarDigital Library
[80] Kittur Aniket, Peters Andrew M., Diriye Abdigani, Telang Trupti, and Bove Michael R.. 2013. Costs and benefits of structured information foraging. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’13). ACM, New York, NY, 2989–2998. Google ScholarDigital Library
[81] Klepsch Melina, Schmitz Florian, and Seufert Tina. 2017. Development and validation of two instruments measuring intrinsic, extraneous, and germane cognitive load. Frontiers in Psychology (2017), 1997. Google ScholarCross Ref
[82] Martinic Marina Krnic, Pieper Dawid, Glatt Angelina, and Puljak Livia. 2019. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Medical Research Methodology 19, 1 (2019), 1–12. Google ScholarCross Ref
[83] Kwok Cody C. T., Etzioni Oren, and Weld Daniel S.. 2001. Scaling question answering to the web. In Proceedings of the 10th International Conference on World Wide Web (WWW’01).150–161. Google ScholarDigital Library
[84] Lastres. Steven A.2012. Legal Research Comes of Age. Retrieved February 19, 2023 from http://www.lexisnexis.com/documents/pdf/20130806061418_large.pdf?sf17880724=1.Google Scholar
[85] Leppink Jimmie, Paas Fred, Vleuten Cees P. M. Van der, Gog Tamara Van, and Merriënboer Jeroen J. G. Van. 2013. Development of an instrument for measuring different types of cognitive load. Behavior Research Methods 45, 4 (2013), 1058–1072. Google ScholarCross Ref
[86] Li Yuelin and Belkin Nicholas J.. 2010. An exploration of the relationships between work task and interactive information search behavior. Journal of the American Society for Information Science and Technology 61, 9 (2010), 1771–1789. https://econpapers.repec.org/RePEc:bla:jamist:v:61:y:2010:i:9:p:1771-1789.Google ScholarDigital Library
[87] Liu Qiaoling, Liu Yandong, and Agichtein Eugene. 2010. Exploring web browsing context for collaborative question answering. In Proceedings of the 3rd Symposium on Information Interaction in Context (IIiX’10). ACM, New York, NY, 305–310. Google ScholarDigital Library
[88] Liu Yiqun, Chen Ye, Tang Jinhui, Sun Jiashen, Zhang Min, Ma Shaoping, and Zhu Xuan. 2015. Different users, different opinions: Predicting search satisfaction with mouse movement information. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’15).493–502. Google ScholarDigital Library
[89] Longo Luca and Dondio Pierpaolo. 2016. On the relationship between perception of usability and subjective mental workload of web interfaces. In Proceedings of the 2015 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT’15). 345–352.Google ScholarDigital Library
[90] Luo Cheng, Li Xue, Liu Yiqun, Sakai Tetsuya, Zhang Fan, Zhang Min, and Ma Shaoping. 2017. Investigating users’ time perception during web search. In Proceedings of the 2017 Conference on Human Information Interaction and Retrieval (CHIIR’17). ACM, New York, NY, 127–136. Google ScholarDigital Library
[91] Luo Cheng, Liu Yiqun, Sakai Tetsuya, Zhou Ke, Zhang Fan, Li Xue, and Ma Shaoping. 2017. Does document relevance affect the searcher’s perception of time? In Proceedings of the 10th ACM International Conference on Web Search and Data Mining (WSDM’17). ACM, New York, NY, 141–150. Google ScholarDigital Library
[92] Maior Horia A., Pike Matthew, Wilson Max L., and Sharples Sarah. 2013. Directly evaluating the cognitive impact of search user interfaces: A two-pronged approach with fNIRS. Euro HCIR Workshop Proceedings 1033 (2013), 43–46.Google Scholar
[93] Makri Stephann, Blandford Ann, and Cox Anna L.. 2008. Investigating the information-seeking behaviour of academic lawyers: From Ellis’s model to design. Information Processing and Management 44, 2 (2008), 613–634. Google ScholarDigital Library
[94] Mansourian Yazdan and Ford Nigel. 2007. Search persistence and failure on the web: A “bounded rationality” and “satisficing” analysis. Journal of Documentation 63, 5 (2007), 680–701. Google ScholarCross Ref
[95] Matthews Gerald, Winter Joost De, and Hancock P. A.. 2020. What do subjective workload scales really measure? Operational and representational solutions to divergence of workload measures. Theoretical Issues in Ergonomics Science 21, 4 (2020), 369–396. Google ScholarCross Ref
[96] Maxwell David and Azzopardi Leif. 2014. Stuck in traffic: How temporal delays affect search behaviour. In Proceedings of the 5th Information Interaction in Context Symposium (IIiX’14).155–164. Google ScholarDigital Library
[97] McGregor Molly, Azzopardi Leif A., and Halvey Martin. 2021. Untangling cost, effort, and load in information seeking and retrieval. In Proceedings of the 2021 Conference on Human Information Interaction and Retrieval. 151–161.Google ScholarDigital Library
[98] Mikkonen Anna. 2015. Books’ interest grading and fiction readers’ search actions during query reformulation intervals categories and subject descriptors. In Proceedings of the 15th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL’15). 27–36.Google Scholar
[99] Miller G.. 1956. The magical number seven, plus minus two: Some limits on our capacity for processing information. Psychological Review 63 (1956), 81–97.Google ScholarCross Ref
[100] Miller Robert B.. 1968. Response time in man-computer conversational transactions: Introductions and major concepts. In Proceedings of the December 9-11, 1968, Fall Joint Computer Conference, Part I.267–277.Google Scholar
[101] Mosaly Prithima Reddy, Mazur Lukasz, and Marks Lawrence B.. 2016. Usability evaluation of electronic health record system (EHRs) using subjective and objective measures. In Proceedings of the 2016 ACM Conference on Human Information Interaction and Retrieval (CHIIR’16).313–316. Google ScholarDigital Library
[102] Mulrow Cynthia, Cook Deborah, Meade Maureen O., and Richardson W. Scott. 1997. Systematic review series series editors: Selecting and appraising studies for a systematic review selecting studies for systematic reviews. Annals of Internal Medicine 127 (1997), 531–537.Google ScholarCross Ref
[103] Nah Fiona Fui Hoon. 2004. A study on tolerable waiting time: How long are web users willing to wait? Behaviour and Information Technology 23, 3 (2004), 153–163. Google ScholarCross Ref
[104] Nordlie-Oslo Ragnar and Pharo Nils. 2013. Search transition as a measure of effort in information retrieval interaction. Proceedings of the American Society for Information Science and Technology 50, 1 (2013), 1–7. Google ScholarCross Ref
[105] O’Connor Denise, Green Sally Elizabeth, and Higgins Julian P. T.. 2008. Defining the review question and developing criteria for including studies. In Cochrane Handbook for Systematic Reviews of Interventions,Higgins Julian P. T. and Green Sally (Eds.). John Wiley & Sons, 8–94.Google Scholar
[106] Oksanen Suvi and Vakkari Pertti. 2012. Emphasis on examining results in fiction searches contributes to finding good novels. In Proceedings of the 12th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL’12). ACM, New York, NY, 199–202. Google ScholarDigital Library
[107] Ong Kevin, Sanderson Mark, Järvelin Kalervo, and Scholer Falk. 2018. QWERTY: The effects of typing on web search behavior. In Proceedings of the 2018 Conference on Human Information Interaction and Retrieval (CHIIR’18).281–284. Google ScholarDigital Library
[108] (Eds.). L. Longo and G. Orru2019. Human Mental Workload: Models and Applications. Communications in Computer and Information Science, Vol. 1318. Springer.Google ScholarCross Ref
[109] Otto A. Ross and Daw Nathaniel D.. 2019. The opportunity cost of time modulates cognitive effort. Neuropsychologia 123, May 2018 (2019), 92–105. Google ScholarCross Ref
[110] Paas Fred, Tuovinen Juhani E., Tabbers Huib, and Gerven Pascal W. M. Van. 2003. Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist 38, 1 (2003), 63–71. Google ScholarCross Ref
[111] Paas F. G., Merriënboer J. J. Van, and Adam J. J.. 1994. Measurement of cognitive load in instructional research. Perceptual and Motor Skills 79, 1 Pt 2 (1994), 419–430. Google ScholarCross Ref
[112] Pothirattanachaikul Suppanut, Yamamoto Yusuke, Yamamoto Takehiro, and Yoshikawa Masatoshi. 2019. Analyzing the effects of document’s opinion and credibility on search behaviors and belief dynamics. In Proceedings of the International Conference on Information and Knowledge Management.1653–1662. Google ScholarDigital Library
[113] Price Paul, Jhangiani Rajiv, and Chiang I-Chant. 2015. Research methods in psychology. In Research Methods in Psychology (2nd ed.). BCcampus Open Education, 322. Google ScholarCross Ref
[114] Rath Manasa, Ghosh Souvick, and Shah Chirag. 2018. Exploring online and offline search behavior based on the varying task complexity. In Proceedings of the 2018 Conference on Human Information Interaction and Retrieval (CHIIR’18). ACM, New York, NY, 285–288. Google ScholarDigital Library
[115] Rieh Soo Young, Kim Yong Mi, and Markey Karen. 2012. Amount of invested mental effort (AIME) in online searching. Information Processing and Management 48, 6 (2012), 1136–1150. Google ScholarDigital Library
[116] Schmutz Peter, Heinz Silvia, Métrailler Yolanda, and Opwis Klaus. 2009. Cognitive load in ecommerce applications—Measurement and effects on user satisfaction. Advances in Human-Computer Interaction 2009 (2009), Article 3, 9 pages. Google ScholarDigital Library
[117] Shah Chirag and González-Ibáñez Roberto. 2011. Evaluating the synergic effect of collaboration in information seeking. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’11).913–922. Google ScholarDigital Library
[118] Shenhav Amitai, Musslick Sebastian, Lieder Falk, Kool Wouter, Griffiths Thomas L., Cohen Jonathan D., and Botvinick Matthew M.. 2017. Toward a rational and mechanistic account of mental effort. Annual Review of Neuroscience 40 (2017), 99–124. Google ScholarCross Ref
[119] Singer Georg, Norbisrath Ulrich, and Lewandowski Dirk. 2012. Ordinary search engine users assessing difficulty, effort, and outcome for simple and complex search tasks. In Proceedings of the 4th Information Interaction in Context Symposium: Behaviors, Interactions, Interfaces, Systems (IIiX’12).110–119. Google ScholarDigital Library
[120] Speier Cheri and Morris Micheal G.. 2003. The influence of query interface design on decision-making performance. MIS Quarterly 27, 3 (2003), 397–423. https://www.jstor.org/stable/30036539.Google ScholarDigital Library
[121] Sweller John. 1988. Cognitive load during problem solving: Effects on learning. Cognitive Science 12, 2 (1988), 257–285. http://onlinelibrary.wiley.com/doi/10.1207/s15516709cog1202_4/abstract.Google Scholar
[122] Sweller John. 1994. Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction 4, 4 (1994), 295–312. Google ScholarCross Ref
[123] Sweller John. 2018. Measuring cognitive load. Perspectives on Medical Education 7, 1 (2018), Article 1, 2 pages. Google ScholarCross Ref
[124] Vakkari Pertti and Huuskonen Saila. 2012. Search effort degrades search output but improves task outcome. Journal of the American Society for Information Science and Technology 63, 4 (2012), 657–670. https://econpapers.repec.org/RePEc:bla:jamist:v:63:y:2012:i:4:p:657-670.Google ScholarCross Ref
[125] Acker Bram B. Van, Parmentier Davy D., Vlerick Peter, and Saldien Jelle. 2018. Understanding mental workload: From a clarifying concept analysis toward an implementable framework. Cognition, Technology and Work 20, 3 (2018), 351–365. Google ScholarDigital Library
[126] Merriënboer Jeroen J. G. Van and Sweller John. 2005. Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review 17, (2005), 147–177. Google ScholarCross Ref
[127] Opijnen Marc van and Santos Cristiana. 2017. On the concept of relevance in legal information retrieval. Artificial Intelligence and Law 25, 1 (2017), 65–87. Google ScholarDigital Library
[128] Vassar Matt, Atakpo Paul, and Kash Melissa J.. 2016. Manual search approaches used by systematic reviewers in dermatology. Journal of the Medical Library Association 104, 4 (2016), 302–304. Google ScholarCross Ref
[129] Verma Manisha and Yilmaz Emine. 2017. Search costs vs. user satisfaction on mobile. In Advances in Information Retrieval. Lecture Notes in Computer Science, Vol. 10193. Springer, 698–704.Google ScholarCross Ref
[130] Villa Robert and Halvey Martin. 2013. Is relevance hard work? Evaluating the effort of making relevant assessments. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’13). 765–768. Google ScholarDigital Library
[131] Villa Robert and Jose Joemon M.. 2012. A study of awareness in multimedia search. Information Processing and Management 48, 1 (2012), 32–46. Google ScholarDigital Library
[132] Vuong Tung, Saastamoinen Miamaria, Jacucci Giulio, and Ruotsalo Tuukka. 2019. Understanding user behavior in naturalistic information search tasks. Journal of the Association for Information Science and Technology 70, 11 (2019), 1248–1261. Google ScholarDigital Library
[133] Wang Qiuzhen, Yang Sa, Liu Manlu, Cao Zike, and Ma Qingguo. 2014. An eye-tracking study of website complexity from cognitive load perspective. Decision Support Systems 62 (2014), 1–10. Google ScholarCross Ref
[134] Ward Austin R. and Capra Rob. 2020. Immersive search: Using virtual reality to examine how a third dimension impacts the searching process. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’20). 1621–1624. Google ScholarDigital Library
[135] Westbrook Andrew and Braver Todd S.. 2015. Cognitive effort: A neuroeconomic approach. Cognitive, Affective and Behavioral Neuroscience 15, 2 (2015), 395–415. Google ScholarCross Ref
[136] Westbrook Andrew, Kester Daria, and Braver Todd S.. 2013. What is the subjective cost of cognitive effort? Load, trait, and aging effects revealed by economic preference. PLOS ONE 8, 7 (2013), 1–8. Google ScholarCross Ref
[137] Wickens Christopher D.. 2002. Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science 3, 2 (2002), 159–177. Google ScholarCross Ref
[138] Wilson Max L.. 2011. Evaluating the cognitive impact of search user interface design decisions. Euro HCIR Workshop Proceedings 763 (2011), 27–30.Google Scholar
[139] Wu I. Chin and Vakkari Pertti. 2014. Supporting navigation in Wikipedia by information visualization: Extended evaluation measures. Journal of Documentation 70, 3 (2014), 392–424. Google ScholarCross Ref
[140] Yang Beverly and Jeh Glen. 2006. Retroactive answering of search queries. In Proceedings of the 15th International Conference on World Wide Web (WWW’06).457–466. Google ScholarDigital Library
[141] Yilmaz Emine, Verma Manisha, Craswell Nick, Radlinski Filip, and Bailey Peter. 2014. Relevance and effort: An analysis of document utility. In Proceedings of the 2014 ACM International Conference on Information and Knowledge Management (CIKM’14).91–100. Google ScholarDigital Library
[142] Young Mark S., Brookhuis Karel A., Wickens Christopher D., and Hancock Peter A.. 2015. State of science: Mental workload in ergonomics. Ergonomics 58, 1 (2015), 1–17. Google ScholarCross Ref
[143] Zhang Yinglong and Gwizdka Jacek. 2014. Effects of tasks at similar and different complexity levels. Proceedings of the American Society for Information Science and Technology 51, 1 (2014), 1–4. Google ScholarCross Ref
[144] Zhang Yinglong and Gwizdka Jacek. 2016. Rethinking the cost of information search behavior. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’16).969–972. Google ScholarDigital Library
[145] Zipf G. K.. 1949. Human Behaviour and the Principle of Least Effort. Addison-Wesley, Reading, MA.Google Scholar

Index Terms

A Systematic Review of Cost, Effort, and Load Research in Information Search and Retrieval, 1972–2020
1. Information systems
  1. Information retrieval
    1. Users and interactive retrieval

Recommendations

Defining and Measuring Cost, Effort, and Load in Information Retrieval
SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

The demands imposed on the user during the Information Retrieval (IR) process are well-recognised, with numerous studies highlighting the influence and importance of cost, effort, and load in explaining user search behaviour and experience. Despite this ...
Read More
Untangling Cost, Effort, and Load in Information Seeking and Retrieval
CHIIR '21: Proceedings of the 2021 Conference on Human Information Interaction and Retrieval

When performing Information Seeking and Retrieval (ISR) activities, people submit queries, examine results, assess documents and engage with the information to make decisions and complete tasks. All these activities come at a "cost'', but within the ...
Read More
On Obtaining Effort Based Judgements for Information Retrieval
WSDM '16: Proceedings of the Ninth ACM International Conference on Web Search and Data Mining

Document relevance has been the primary focus in the design, optimization and evaluation of retrieval systems. Traditional testcollections are constructed by asking judges the relevance grade for a document with respect to an input query. Recent work of ...
Read More

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Published in
ACM Transactions on Information Systems Volume 42, Issue 1
January 2024
924 pages
ISSN:1046-8188
EISSN:1558-2868
DOI:10.1145/3613513
Editor:
Min Zhang
Tsinghua University, China
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
- Published: 18 August 2023
- Online AM: 9 February 2023
- Accepted: 25 January 2023
- Revised: 18 January 2023
- Received: 1 February 2022
Published in tois Volume 42, Issue 1

Permissions
Request permissions about this article.
Request Permissions

Check for updates
Author Tags
Cost
effort
cognitive load
workload
Qualifiers
- research-article
Conference
Funding Sources
Other Metrics
View Article Metrics

Article Metrics
- 1
  Total Citations
  View Citations
- 410
  Total Downloads
- Downloads (Last 12 months)310
- Downloads (Last 6 weeks)27
Other Metrics
View Author Metrics
Cited By
View all

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

View Full Text

A Systematic Review of Cost, Effort, and Load Research in Information Search and Retrieval, 1972–2020

ACM Transactions on Information Systems

Abstract

REFERENCES

Cited By

Index Terms

Recommendations

Defining and Measuring Cost, Effort, and Load in Information Retrieval

Untangling Cost, Effort, and Load in Information Seeking and Retrieval

On Obtaining Effort Based Judgements for Information Retrieval