Skip to main content
SpringerLink
Log in

Cognitive engagement with technology scale: a validation study

  • Research Article
  • Published:
Educational technology research and development Aims and scope Submit manuscript

Abstract

Quantitative studies on technology integration often examined general quantity of classroom technology use or teacher-reported accounts of integration practices. There is a current need for a measurement tool that links students’ technology use and cognitive engagement, which will allow researchers to better illustrate how technology is woven into the learning process. The purpose of this study is to develop a scale to measure how students use technology for different cognitive tasks, following theoretical conceptions from Bloom’s Digital Taxonomy and Multiple-Document Task-based Relevance Assessment and Content Extraction. We employed Confirmatory Factor Analysis, as well as both classical test theory and item response theory over three studies to validate our newly created scale. The new Cognitive Engagement with Technology (CET) scale showed good psychometric properties, item functioning, and construct validity. The CET scale can be used to triangulate students’ technology use patterns with other research methods. It can also help extend past findings by taking into account how students use technology to aid in the cognitive processes of learning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  • Ajzen, I. (1985). From intentions to actions: a theory of planned behavior. In J. Kuhl & J. Beckmann (Eds.), Action control: From cognition to behavior (pp. 11–39). Springer.

    Chapter  Google Scholar 

  • Ajzen, I. (2002). Perceived behavioral control, self-efficacy, locus of control, and the theory of planned behavior. Journal of Applied Social Psychology, 32(4), 665–683. https://doi.org/10.1111/j.1559-1816.2002.tb00236.x

    Article  Google Scholar 

  • Al-Adwan, A. A., Al-Adwan, A. A., & Smedley, J. (2013). Exploring students acceptance of e-learning using Technology Acceptance Model in Jordanian universities. International Journal of Education and Development Using ICT, 9(2), 2.

    Google Scholar 

  • Anderson, L. W., & Krathworthl, D. R. (Eds.). (2001). A taxonomy for learning, teaching and assisting: A revision of Bloom’s taxonomy of education objectives. Longman.

    Google Scholar 

  • Anmarkrud, Ø., McCrudden, M. T., Bråten, I., & Strømsø, H. I. (2013). Task-oriented reading of multiple documents: Online comprehension processes and offline products. Instructional Science, 41(5), 873–894. https://doi.org/10.1007/s11251-013-9263-8

    Article  Google Scholar 

  • Barron, A. E., Kemker, K., Harmes, C., & Kalaydjian, K. (2003). Large-scale research study on technology in K–12 schools: Technology integration as it relates to the National Technology Standards. Journal of Research on Technology in Education, 35(4), 489–507.

    Article  Google Scholar 

  • Böckenholt, U., & Meiser, T. (2017). Response style analysis with threshold and multi-process IRT models: A review and tutorial. British Journal of Mathematical and Statistical Psychology, 70(1), 159–181.

    Article  Google Scholar 

  • Bråten, I., Ferguson, L. E., Strømsø, H. I., & Anmarkrud, Ø. (2014). Students working with multiple conflicting documents on a scientific issue: Relations between epistemic cognition while reading and sourcing and argumentation in essays. British Journal of Educational Psychology, 84(1), 58–85. https://doi.org/10.1111/bjep.12005

    Article  Google Scholar 

  • Bråten, I., Stadtler, M., & Salmerón, L. (2017). The role of sourcing in discourse comprehension. The Routledge Handbook of Discourse Processes, Second Edition. https://doi.org/10.4324/9781315687384

    Article  Google Scholar 

  • Cameron, C., Van Meter, P., & Long, V. A. (2017). The effects of instruction on students’ generation of self-questions when reading multiple documents. Journal of Experimental Education, 85(2), 334–351. https://doi.org/10.1080/00220973.2016.1182884

    Article  Google Scholar 

  • Chen, R. J. (2010). Investigating models for preservice teachers’ use of technology to support student-centered learning. Computers and Education, 55(1), 32–42. https://doi.org/10.1016/j.compedu.2009.11.015

    Article  Google Scholar 

  • Cheng, E. W. L. (2018). Choosing between the theory of planned behavior (TPB) and the technology acceptance model (TAM). Educational Technology Research and Development, 67(1), 1–17. https://doi.org/10.1007/s11423-018-9598-6

    Article  Google Scholar 

  • Cheung, R., & Vogel, D. (2013). Predicting user acceptance of collaborative technologies: An extension of the technology acceptance model for e-learning. Computers and Education, 63, 160–175. https://doi.org/10.1016/j.compedu.2012.12.003

    Article  Google Scholar 

  • Churches, A. (2009). Bloom's digital taxonomy. Retrieved 6th June, 2020, from http://edorigami.wikispaces.com/Bloom%27s+Digital+Taxonomy.

  • Clarebout, G., & Elen, J. (2009). The complexity of tool use in computer-based learning environments. Instructional Science, 37(5), 475–486. https://doi.org/10.1007/s11251-008-9068-3

    Article  Google Scholar 

  • Crompton, H., Burke, D., & Lin, Y. C. (2019). Mobile learning and student cognition: A systematic review of PK-12 research using Bloom’s Taxonomy. British Journal of Educational Technology, 50(2), 684–701. https://doi.org/10.1111/bjet.12674

    Article  Google Scholar 

  • Crook, S. J., & Sharma, M. D. (2013). Bloom-ing heck! the activities of australian science teachers and students two years into a 1:1 laptop program across 14 high schools. International Journal of Innovation in Science and Mathematics Education, 21(1), 54–69.

    Google Scholar 

  • Crow, S. R. (2015). The information-seeking behavior of intrinsically motivated elementary school children of a collectivist culture. School Library Media Research, 18.

  • Davis Jr, F. D. (1986). A technology acceptance model for empirically testing new end-user information systems: Theory and results [Doctoral dissertation, Massachusetts Institute of Technology]. https://dspace.mit.edu/handle/1721.1/15192#files-area.

  • D’Mello, S. (2013). A selective meta-analysis on the relative incidence of discrete affective states during learning with technology. Journal of Educational Psychology, 105(4), 1082–1099. https://doi.org/10.1037/a0032674

    Article  Google Scholar 

  • Diacopoulos, M. M. (2015). Untangling web 2.0: Charting web 2.0 tools, the NCSS guidelines for effective use of technology, and bloom’s taxonomy. The Social Studies, 106(4), 139–148. https://doi.org/10.1080/00377996.2015.1015711

    Article  Google Scholar 

  • Dietrich, T., & Balli, S. J. (2014). Digital natives: Fifth-grade students’ authentic and ritualistic engagement with technology. International Journal of Instruction, 7(2), 21–34.

    Google Scholar 

  • Dugdale, S., Legare, O., Matthews, J. I., & Ju, M. K. (1998). Mathematical problem solving and computers: A study of learner-initiated application of technology in a general problem-solving context. Journal of Research on Computing in Education, 30(3), 239–252. https://doi.org/10.1080/08886504.1998.10782225

    Article  Google Scholar 

  • Edmunds, R., Thorpe, M., & Conole, G. (2012). Student attitudes towards and use of ICT in course study, work and social activity: A technology acceptance model approach. British Journal of Educational Technology, 43(1), 71–84. https://doi.org/10.1111/j.1467-8535.2010.01142.x

    Article  Google Scholar 

  • Embretson, S. E., & Reise, S. P. (2000). Item response theory. Psychology Press.

    Google Scholar 

  • Ertmer, P. A. (1999). Addressing first- and second-order barriers to change: strategies for technology integration. Educational Technology Research and Development, 47(4), 47–61.

    Article  Google Scholar 

  • Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299.

    Article  Google Scholar 

  • Gašević, D., Jovanović, J., Pardo, A., & Dawson, S. (2017). Detecting learning strategies with analytics: Links with self-reported measures and academic performance. Journal of Learning Analytics, 4(2), 113–128. https://doi.org/10.18608/jla.2017.42.10

    Article  Google Scholar 

  • Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47(4), 356–381. https://doi.org/10.1002/RRQ.027

    Article  Google Scholar 

  • Heflin, H., Shewmaker, J., & Nguyen, J. (2017). Impact of mobile technology on student attitudes, engagement, and learning. Computers and Education, 107, 91–99. https://doi.org/10.1016/j.compedu.2017.01.006

    Article  Google Scholar 

  • Henrie, C. R., Halverson, L. R., & Graham, C. R. (2015). Measuring student engagement in technology-mediated learning: A review. Computers and Education, 90, 36–53. https://doi.org/10.1016/j.compedu.2015.09.005

    Article  Google Scholar 

  • Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43(1), 115–135.

    Article  Google Scholar 

  • Hew, K. F., & Brush, T. (2007). Integrating technology into K-12 teaching and learning: Current knowledge gaps and recommendations for future research. Educational Technology Research and Development, 55(3), 223–252.

    Article  Google Scholar 

  • Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55.

    Article  Google Scholar 

  • Inan, F. A., Lowther, D. L., Ross, S. M., & Strahl, D. (2010). Pattern of classroom activities during students’ use of computers: Relations between instructional strategies and computer applications. Teaching and Teacher Education, 26(3), 540–546. https://doi.org/10.1016/j.tate.2009.06.017

    Article  Google Scholar 

  • Jiang, L., Elen, J., & Clarebout, G. (2009). The relationships between learner variables, tool-usage behaviour and performance. Computers in Human Behavior, 25(2), 501–509. https://doi.org/10.1016/j.chb.2008.11.006

    Article  Google Scholar 

  • Kim, M., Cheng, S. L., & Xie, K. (2017). The validation of a systemic evaluation framework to investigate the multi-layered impacts of technology integration projects. Paper presented at the annual meeting of the American Educational Research Association, San Antonio, TX.

  • Kline, R. B. (2011). Principles and practice of structural equation modeling. Guilford Publications.

    Google Scholar 

  • Kopcha, T. J. (2012). Teachers’ perceptions of the barriers to technology integration and practices with technology under situated professional development. Computers & Education, 59, 1109–1121. https://doi.org/10.1016/j.compedu.2012.05.014

    Article  Google Scholar 

  • Linderholm, T., Kwon, H., & Therriault, D. J. (2014). Instructions that enhance multiple-text comprehension for college readers. Journal of College Reading and Learning, 45(1), 3–19. https://doi.org/10.1080/10790195.2014.906269

    Article  Google Scholar 

  • Lust, G., Elen, J., & Clarebout, G. (2013). Students’ tool-use within a web enhanced course: Explanatory mechanisms of students’ tool-use pattern. Computers in Human Behavior, 29(5), 2013–2021. https://doi.org/10.1016/j.chb.2013.03.014

    Article  Google Scholar 

  • Makransky, G., Lilleholt, L., & Aaby, A. (2017). Development and validation of the Multimodal Presence Scale for virtual reality environments: A confirmatory factor analysis and item response theory approach. Computers in Human Behavior, 72, 276–285. https://doi.org/10.1016/j.chb.2017.02.066

    Article  Google Scholar 

  • Miranda, H. P., & Russell, M. (2012). Understanding factors associated with teacher-directed student use of technology in elementary classrooms: A structural equation modeling approach. British Journal of Educational Technology, 43(4), 652–666.

    Article  Google Scholar 

  • Mueller, J., Wood, E., Willoughby, T., Ross, C., & Specht, J. (2008). Identifying discriminating variables between teachers who fully integrate computers and teachers with limited integration. Computers and Education, 51(4), 1523–1537. https://doi.org/10.1016/j.compedu.2008.02.003

    Article  Google Scholar 

  • Park, S. Y., Nam, M. W., & Cha, S. B. (2012). University students’ behavioral intention to use mobile learning: Evaluating the technology acceptance model. British Journal of Educational Technology, 43(4), 592–605. https://doi.org/10.1111/j.1467-8535.2011.01229.x

    Article  Google Scholar 

  • Pellas, N. (2014). The influence of computer self-efficacy, metacognitive self-regulation and self-esteem on student engagement in online learning programs: Evidence from the virtual world of Second Life. Computers in Human Behavior, 35, 157–170. https://doi.org/10.1016/j.chb.2014.02.048

    Article  Google Scholar 

  • Pittman, T., & Gaines, T. (2015). Technology integration in third, fourth and fifth grade classrooms in a Florida school district. Educational Technology Research and Development, 63(4), 539–554. https://doi.org/10.1007/s11423-015-9391-8

    Article  Google Scholar 

  • Purcell, K., Heaps, A., Buchanan, J., & Friedrich, L. (2013). How teachers are using technology at home and in their classrooms. Pew Research Center.

    Google Scholar 

  • Rashid, T., & Asghar, H. M. (2016). Technology use, self-directed learning, student engagement and academic performance: Examining the interrelations. Computers in Human Behavior, 63, 604–612. https://doi.org/10.1016/j.chb.2016.05.084

    Article  Google Scholar 

  • Rouet, J. F., & Britt, A. M. (2011). Relevance processes in multiple document comprehension. Text Relevance and Learning from Text, June, 19–52.

  • Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8(2), 23–74.

    Google Scholar 

  • Schmid, R. F., Bernard, R. M., Borokhovski, E., Tamim, R., Abrami, P. C., Wade, C. A., & Lowerison, G. (2009). Technology’s effect on achievement in higher education: A Stage I meta-analysis of classroom applications. Journal of Computing in Higher Education, 21(2), 95–109.

    Article  Google Scholar 

  • Sun, J. C. Y., & Rueda, R. (2012). Situational interest, computer self-efficacy and self-regulation: Their impact on student engagement in distance education. British Journal of Educational Technology, 43(2), 191–204. https://doi.org/10.1111/j.1467-8535.2010.01157.x

    Article  Google Scholar 

  • Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study. Review of Educational Research, 81(1), 4–28.

    Article  Google Scholar 

  • Teo, T. (2009). Modelling technology acceptance in education: A study of pre-service teachers. Computers & Education, 52(2), 302–312. https://doi.org/10.1016/j.compedu.2008.08.006

    Article  Google Scholar 

  • Teo, T. S., Srivastava, S. C., & Jiang, L. (2008). Trust and electronic government success: An empirical study. Journal of Management Information Systems, 25(3), 99–132.

    Article  Google Scholar 

  • Tondeur, J., Aesaert, K., Prestridge, S., & Consuegra, E. (2018). A multilevel analysis of what matters in the training of pre-service teacher’s ICT competencies. Computers and Education, 122, 32–42. https://doi.org/10.1016/j.compedu.2018.03.002

    Article  Google Scholar 

  • Tondeur, J., van Keer, H., van Braak, J., & Valcke, M. (2008). ICT integration in the classroom: Challenging the potential of a school policy. Computers and Education, 51(1), 212–223. https://doi.org/10.1016/j.compedu.2007.05.003

    Article  Google Scholar 

  • Vannatta, R. A., & O’Bannon, B. (2002). Beginning to put the pieces together: A technology infusion model for teacher education. Journal of Computing in Teacher Education, 18(4), 112–123.

    Google Scholar 

  • Vongkulluksn, V. W., Xie, K., & Bowman, M. A. (2018). The role of value on teachers’ internalization of external barriers and externalization of personal beliefs for classroom technology integration. Computers & Education, 118, 70–81.

    Article  Google Scholar 

  • Wang, S. K., Hsu, H. Y., Campbell, T., Coster, D. C., & Longhurst, M. (2014). An investigation of middle school science teachers and students use of technology inside and outside of classrooms: Considering whether digital natives are more technology savvy than their teachers. Educational Technology Research and Development, 62(6), 637–662.

    Article  Google Scholar 

  • Xie, K., & Luthy, N. (2017). Textbooks in the digital world. The Conversation. Retrieved from https://theconversation.com/textbooks-in-the-digital-world-78299.

Download references

Acknowledgements

The study reported in this paper is based upon work in the EDCITE: Evaluating Digital Content for Instructional and Teaching Excellence project and the College Ready Ohio project supported by the Straight A fund from the Ohio Department of Education. The grant organization had no involvement in the design, data collection, data analysis, writing, or decision on article submission. The conclusions and recommendations within this article do not necessarily reflect the views of the Ohio Department of Education.

Author information

Authors and Affiliations

  1. University of Nevada Las Vegas, 4505 South Maryland Parkway, Las Vegas, NV, 89154-3003, USA

    Vanessa W. Vongkulluksn & Lin Lu

  2. Cleveland Clinic Education Institute, Cleveland, OH, USA

    Michael J. Nelson

  3. The Ohio State University, Columbus, OH, USA

    Kui Xie

Authors
  1. Vanessa W. Vongkulluksn
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lin Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael J. Nelson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kui Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vanessa W. Vongkulluksn.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vongkulluksn, V.W., Lu, L., Nelson, M.J. et al. Cognitive engagement with technology scale: a validation study. Education Tech Research Dev 70, 419–445 (2022). https://doi.org/10.1007/s11423-022-10098-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11423-022-10098-9

Keywords

Search

Navigation