Abstract
Competency measurement typically focuses on task outcomes. Taking process data into account (i.e., processing time and steps) can provide new insights into construct-related solution behavior, or confirm assumptions that govern task design. This chapter summarizes four studies to illustrate the potential of behavioral process data for explaining task success. It also shows that generic process measures such as time on task may have different relations to task success, depending on the features of the task and the test-taker. The first study addresses differential effects of time on task on success across tasks used in the OECD Programme for the International Assessment of Adult Competencies (PIAAC). The second study, also based on PIAAC data, investigates at a fine-grained level, how the time spent on automatable subtasks in problem-solving tasks relates to task success. The third study addresses how the number of steps taken during problem solving predicts success in PIAAC problem-solving tasks. In a fourth study, we explore whether successful test-takers can be clustered on the basis of various behavioral process indicators that reflect information problem solving. Finally, we address how to handle unstructured and large sets of process data, and briefly present a process data extraction tool.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Naumann et al. (2014) included a third predictor in their model: the openness of the task. For the results for this predictor, not reported here, please refer to the original source.
References
Asseburg, R., & Frey, A. (2013). Too hard, too easy, or just right? The relationship between effort or boredom and ability-difficulty fit. Psychological Test and Assessment Modeling, 55, 92–104.
Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59, 390–412. doi:10.1016/j.jml.2007.12.005.
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validty. Psychological Review, 111, 1061–1071. doi:10.1037/0033-295X.111.4.1061.
Brand-Gruwel, S., Wopereis, I., & Walraven, A. (2009). A descriptive model of information problem solving while using internet. Computers & Education, 53, 1207–1217. doi:10.1016/j.compedu.2009.06.004.
Ericsson, K. A., & Kintsch, W. (1995). Long-term working memory. Psychological Review, 102, 211–245. doi:10.1037/0033-295X.102.2.211.
Funke, J. (2006). Komplexes Problemlösen [Complex problem solving]. In J. Funke (Ed.), Denken und Problemlösen (Enzyklopädie der Psychologie, Themenbereich C: Theorie und Forschung, Serie II: Kognition, Band 8) (pp. 375–446). Göttingen: Hogrefe.
Goldhammer, F., & Klein Entink, R. H. (2011). Speed of reasoning and its relation to reasoning ability. Intelligence, 39, 108–119. doi:10.1016/j.intell.2011.02.001.
Goldhammer, F., Naumann, J., Stelter, A., Tóth, K., Rölke, H., & Klieme, E. (2014). The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. Journal of Educational Psychology, 106, 608–626. doi:10.1037/a0034716.
Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from Internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47, 356–381. doi:10.1002/RRQ.027.
Lawless, K. A., & Kulikowich, J. M. (1996). Understanding hypertext navigation through cluster analysis. Journal of Educational Computing Research, 14, 385–399. doi:10.2190/dvap-de23-3xmv-9mxh.
Naumann, J. (2012). Belastungen und Ressourcen beim Lernen aus Text und Hypertext. [Costs and resources in learning from text and hypertext]. (Unpublished habilitation thesis). Goethe Universität Frankfurt, Frankfurt, Germany.
Naumann, J., Goldhammer, F., Rölke, H., & Stelter, A. (2014). Erfolgreiches Problemlösen in technologiereichen Umgebungen: Wechselwirkungen zwischen Interaktionsschritten und Aufgabenkomplexität [Successful problem solving in technology-rich environments: Interactions between the number of actions and task complexity]. Zeitschrift für Pädagogische Psychologie/German Journal of Educational Psychology, 28, 193–203. doi:10.1024/1010-0652/a000134.
Naumann, J., Richter, T., Christmann, U., & Groeben, N. (2008). Working memory capacity and reading skill moderate the effectiveness of strategy training in learning from hypertext. Learning and Individual Differences, 18, 197–213.
OECD (Organisation for Economic Co-Operation and Development). (2011). PISA 2009 results Vol VI: Student on line. Digital technologies and performance. Paris: Author.
OECD (Organisation for Economic Co-Operation and Development). (2013). OECD skills outlook 2013: First results from the survey of Adult Skills. Paris: Author.
Pfaff, Y., & Goldhammer, F. (2011, September). Measuring individual differences in ICT literacy: Evaluating online information. Paper presented at the 14th biennial EARLI conference, Exeter, UK.
Richter, T., Naumann, J., Brunner, M., & Christmann, U. (2005). Strategische Verarbeitung beim Lernen mit Text und Hypertext [Strategic processing in learning from text and hypertext]. German Journal of Educational Psychology, 19, 5–22. doi:10.1024/1010-0652.19.12.5.
Roskam, E. E. (1997). Models for speed and time-limit tests. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 187–208). New York: Springer.
Salmerón, L., Cañas, J. J., Kintsch, W., & Fajardo, I. (2005). Reading strategies and hypertext comprehension. Discourse Processes, 40, 171–191. doi:10.1207/s15326950dp4003_1.
Schroeder, E. E., & Grabowski, B. L. (1995). Patterns of exploration and learning with hypermedia. Journal of Educational Computing Research, 13, 313–335.
Shiffrin, R. M., & Schneider, W. (1977). Controlled and automatic human information processing: II. Perceptual learning, automatic attending, and a general theory. Psychological Review, 84, 127–190. doi:10.1037/0033-295X.84.2.127.
Sireci, S. G., & Zenisky, A. L. (2006). Innovative item formats in computer-based testing: In pursuit of improved construct representation. In S. M. Downing & T. S. Haladyna (Eds.), Handbook of test development (pp. 329–348). Mahwah: Erlbaum.
Stelter, A., Goldhammer, F., Naumann, J., & Rölke, H. (2015). Die Automatisierung prozeduralen Wissens: Eine Analyse basierend auf Prozessdaten [The automation of procedural knowledge: An analysis based on process data]. In J. Stiller & C. Laschke (Eds.), Berlin-Brandenburger Beitrage zur Bildungsforschung 2015: Herausforderungen, Befunde und Perspektiven Interdisziplinärer Bildungsforschung (pp. 111–132). Frankfurt am Main: Lang.
Tóth, K., Rölke, H., Goldhammer, F. Kröhne, U. (2013, January/February). Investigating students’ ICT-skills with process data. Paper presented at the DAILE13 workshop, Villard-de-Lans, France.
van der Linden, W. J. (2005). Linear models for optimal test design. New York: Springer.
van der Linden, W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287–308. doi:10.1007/s11336-006-1478-z.
van der Linden, W. J. (2009). Conceptual issues in response-time modeling. Journal of Educational Measurement, 46, 247–272. doi:10.1111/j.1745-3984.2009.00080.x.
van Merriënboer, J. J. G., & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17, 147–177. doi:10.1007/s10648-005-3951.
Acknowledgments
The preparation of this manuscript was supported by a grant from the German Research Foundation (DFG), awarded to Frank Goldhammer, Johannes Naumann, and Heiko Rölke (GO 1979/1-1) in the Priority Programme “Competence Models for Assessing Individual Learning Outcomes and Evaluating Educational Processes” (SPP 1293). Frank Goldhammer and Johannes Naumann contributed equally to this chapter.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this chapter
Cite this chapter
Goldhammer, F., Naumann, J., Rölke, H., Stelter, A., Tóth, K. (2017). Relating Product Data to Process Data from Computer-Based Competency Assessment. In: Leutner, D., Fleischer, J., Grünkorn, J., Klieme, E. (eds) Competence Assessment in Education. Methodology of Educational Measurement and Assessment. Springer, Cham. https://doi.org/10.1007/978-3-319-50030-0_24
Download citation
DOI: https://doi.org/10.1007/978-3-319-50030-0_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-50028-7
Online ISBN: 978-3-319-50030-0
eBook Packages: EducationEducation (R0)