Guojing Zhou
ABSTRACT
Video viewing is an important component of online learning, yet little is known about what information about learning outcomes can be derived from students’ video control actions. We investigate the extent to which information on student learning is contained in their video-watching clickstreams (e.g. pausing, playing) immediately after watching a video. We analyzed data from 10,492 students who used an online learning platform for their Algebra 1 course. Our experiments encode students’ video-control clickstreams into sequences in several ways (e.g. aggregate actions, shuffle actions, and merge action types), and train Long Short-term Memory (LSTM) neural network models to predict after-video quiz scores (N = 32,482) from the sequences in a student-independent fashion. The results suggest that the action sequences contain a limited amount of information about student learning (r = 0.108 between model-predicted- and actual- quiz scores), with most of the information in simple counts of actions (r = 0.081) rather than the temporal ordering of actions. Combining information from video action sequences and traditional knowledge estimates from item-response theory (IRT) outperformed (r = 0.224) either approach independently. Implications for student modeling and adaptive learning support for viewing lecture videos are discussed.
Supplemental Material
- Everaldo Aguiar, Saurabh Nagrecha, and Nitesh V Chawla. 2015. Predicting online video engagement using clickstreams. In 2015 IEEE international conference on data science and advanced analytics (DSAA). IEEE, 1–10.Google Scholar
Cross Ref
- Vincent Aleven and Kenneth R Koedinger. 2000. Limitations of student control: Do students know when they need help?. In International conference on intelligent tutoring systems. Springer, 292–303.Google ScholarCross Ref
- Giora Alexandron, Qian Zhou, and David Pritchard. 2015. Discovering the Pedagogical Resources that Assist Students in Answering Questions Correctly – A Machine Learning Approach. In In: Proceedings of the 8th International Conference on Educational Data Mining (EDM 2015).Google Scholar
- Roger Azevedo, Amy Johnson, Amber Chauncey, Arthur Graesser, B Zimmerman, and D Schunk. 2011. Use of hypermedia to assess and convey self-regulated learning. Handbook of self-regulation of learning and performance 32 (2011), 102–121.Google Scholar
- Lucy Barnard-Brak, Valerie Osland Paton, and William Y Lan. 2010. Profiles in self-regulated learning in the online learning environment. International Review of Research in Open and Distributed Learning 11, 1(2010), 61–80.Google ScholarCross Ref
- A Lord Birnbaum. 1968. Some latent trait models and their use in inferring an examinee’s ability. Statistical theories of mental test scores(1968).Google Scholar
- Christopher G Brinton and Mung Chiang. 2015. MOOC performance prediction via clickstream data and social learning networks. In 2015 IEEE conference on computer communications (INFOCOM). IEEE, 2299–2307.Google ScholarCross Ref
- R Philip Chalmers 2012. mirt: A multidimensional item response theory package for the R environment. Journal of statistical Software 48, 6 (2012), 1–29.Google ScholarCross Ref
- Zhenghua Chen, Min Wu, Rui Zhao, Feri Guretno, Ruqiang Yan, and Xiaoli Li. 2020. Machine remaining useful life prediction via an attention-based deep learning approach. IEEE Transactions on Industrial Electronics 68, 3 (2020), 2521–2531.Google ScholarCross Ref
- Francois Chollet 2015. Keras. https://github.com/fchollet/kerasGoogle Scholar
- Ryan SJ d Baker, Albert T Corbett, and Vincent Aleven. 2008. More accurate student modeling through contextual estimation of slip and guess probabilities in bayesian knowledge tracing. In International conference on intelligent tutoring systems. Springer, 406–415.Google ScholarDigital Library
- Thanasis Daradoumis, Roxana Bassi, Fatos Xhafa, and Santi Caballé. 2013. A review on massive e-learning (MOOC) design, delivery and assessment. In 2013 eighth international conference on P2P, parallel, grid, cloud and internet computing. IEEE, 208–213.Google Scholar
- Birk Diedenhofen and Jochen Musch. 2015. cocor: A comprehensive solution for the statistical comparison of correlations. PloS one 10, 4 (2015), e0121945.Google ScholarCross Ref
- Asmaa Elbadrawy, Agoritsa Polyzou, Zhiyun Ren, Mackenzie Sweeney, George Karypis, and Huzefa Rangwala. 2016. Predicting student performance using personalized analytics. Computer 49, 4 (2016), 61–69.Google ScholarDigital Library
- Susan E Embretson and Steven P Reise. 2013. Item response theory. Psychology Press.Google Scholar
- Andrew Emerson, Fernando J Rodríguez, Bradford Mott, Andy Smith, Wookhee Min, Kristy Elizabeth Boyer, Cody Smith, Eric Wiebe, and James Lester. 2019. Predicting Early and Often: Predictive Student Modeling for Block-Based Programming Environments.International Educational Data Mining Society (2019).Google Scholar
- Jensen Emily, Umada Tetsumichi, C. Hunkins Nicholas, Hutt Stephen, Huggins-Manley A. Corinne, and K. D’Mello Sidney. 2021. What You Do Predicts How You Do: Prospectively Modeling Student Quiz Performance Using Activity Features in anOnline Learning Environment. In In: Proceedings of the 11th international conference on learning analytics and knowledge.Google Scholar
- Lianli Gao, Zhao Guo, Hanwang Zhang, Xing Xu, and Heng Tao Shen. 2017. Video captioning with attention-based LSTM and semantic consistency. IEEE Transactions on Multimedia 19, 9 (2017), 2045–2055.Google ScholarDigital Library
- Aritra Ghosh, Neil Heffernan, and Andrew S Lan. 2020. Context-aware attentive knowledge tracing. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2330–2339.Google ScholarDigital Library
- Philip J Guo and Katharina Reinecke. 2014. Demographic differences in how students navigate through MOOCs. In Proceedings of the first ACM conference on Learning@ scale conference. 21–30.Google ScholarDigital Library
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780.Google ScholarDigital Library
- Binxuan Huang, Yanglan Ou, and Kathleen M Carley. 2018. Aspect level sentiment classification with attention-over-attention neural networks. In International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation. Springer, 197–206.Google ScholarCross Ref
- Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991(2015).Google Scholar
- Stephen Hutt, Joseph F Grafsgaard, and Sidney K D’Mello. 2019. Time to scale: Generalizable affect detection for tens of thousands of students across an entire school year. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–14.Google ScholarDigital Library
- Michael J Jacobson. 2008. A design framework for educational hypermedia systems: Theory, research, and learning emerging scientific conceptual perspectives. Educational technology research and development 56, 1(2008), 5–28.Google Scholar
- Emily Jensen, Stephen Hutt, and Sidney K D’Mello. 2019. Generalizability of Sensor-Free Affect Detection Models in a Longitudinal Dataset of Tens of Thousands of Students.EDM (2019).Google Scholar
- Hamid Karimi, Tyler Derr, Jiangtao Huang, and Jiliang Tang. 2020. Online Academic Course Performance Prediction Using Relational Graph Convolutional Neural Network.International Educational Data Mining Society (2020).Google Scholar
- Tanja Käser, Severin Klingler, Alexander G Schwing, and Markus Gross. 2017. Dynamic Bayesian networks for student modeling. IEEE Transactions on Learning Technologies 10, 4 (2017), 450–462.Google ScholarDigital Library
- Byung-Hak Kim, Ethan Vizitei, and Varun Ganapathi. 2018. GritNet: Student performance prediction with deep learning. arXiv preprint arXiv:1804.07405(2018).Google Scholar
- Gábor Kőrösi and Richard Farkas. 2020. MOOC performance prediction by deep learning from raw clickstream data. In International Conference on Advances in Computing and Data Sciences. Springer, 474–485.Google ScholarCross Ref
- Geza Kovacs. 2016. Effects of in-video quizzes on MOOC lecture viewing. In Proceedings of the third (2016) ACM conference on Learning@ Scale. 31–40.Google ScholarDigital Library
- Sébastien Lallé and Cristina Conati. 2020. A data-driven student model to provide adaptive support during video watching across moocs. In International Conference on Artificial Intelligence in Education. Springer, 282–295.Google ScholarDigital Library
- Nan Li, Łukasz Kidziński, Patrick Jermann, and Pierre Dillenbourg. 2015. MOOC video interaction patterns: What do they tell us?. In European Conference on Technology Enhanced Learning. Springer, 197–210.Google ScholarDigital Library
- Allison Littlejohn, Nina Hood, Colin Milligan, and Paige Mustain. 2016. Learning in MOOCs: Motivations and self-regulated learning in MOOCs. The Internet and Higher Education 29 (2016), 40–48.Google ScholarCross Ref
- G Maragatham and Shobana Devi. 2019. LSTM model for prediction of heart failure in big data. Journal of medical systems 43, 5 (2019), 1–13.Google ScholarDigital Library
- J Patrick Meyer and Shi Zhu. 2013. Fair and equitable measurement of student learning in MOOCs: An introduction to item response theory, scale linking, and score equating.Research & Practice in Assessment 8 (2013), 26–39.Google Scholar
- Ahmed Ali Mubarak, Han Cao, and Salah AM Ahmed. 2021. Predictive learning analytics using deep learning model in MOOCs’ courses videos. Education and Information Technologies 26, 1 (2021), 371–392.Google ScholarDigital Library
- Ahmed A Mubarak, Han Cao, and Weizhen Zhang. 2020. Prediction of students’ early dropout based on their interaction logs in online learning environment. Interactive Learning Environments(2020), 1–20.Google Scholar
- Sahba Akhavan Niaki, Clint P George, George Michailidis, and Carole R Beal. 2019. Investigating the Usage Patterns of Algebra Nation Tutoring Platform. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge. 481–490.Google ScholarDigital Library
- David S Park, Robert W Schmidt, Charankumar Akiri, Stephanie Kwak, and David A Joyner. 2020. Affordable Degrees at Scale: New Phenomenon or New Hype?. In Proceedings of the Seventh ACM Conference on Learning@ Scale. 25–35.Google ScholarDigital Library
- F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830.Google ScholarDigital Library
- Nikolaos Pellas. 2014. The influence of computer self-efficacy, metacognitive self-regulation and self-esteem on student engagement in online learning programs: Evidence from the virtual world of Second Life. Computers in Human Behavior 35 (2014), 157–170.Google ScholarCross Ref
- Chris Piech, Jonathan Spencer, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas Guibas, and Jascha Sohl-Dickstein. 2015. Deep knowledge tracing. arXiv preprint arXiv:1506.05908(2015).Google Scholar
- VR Raghuveer, BK Tripathy, Taranveer Singh, and Saarthak Khanna. 2014. Reinforcement learning approach towards effective content recommendation in MOOC environments. In 2014 IEEE International Conference on MOOC, Innovation and Technology in Education (MITE). IEEE, 285–289.Google ScholarCross Ref
- Ido Roll, Ryan SJ d Baker, Vincent Aleven, and Kenneth R Koedinger. 2014. On the benefits of seeking (and avoiding) help in online problem-solving environments. Journal of the Learning Sciences 23, 4 (2014), 537–560.Google ScholarCross Ref
- John Ruscio. 2008. A probability-based measure of effect size: robustness to base rates and other factors.Psychological methods 13, 1 (2008), 19.Google Scholar
- Tanmay Sinha, Patrick Jermann, Nan Li, and Pierre Dillenbourg. 2014. Your click decides your fate: Inferring information processing and attrition behavior from mooc video clickstream interactions. arXiv preprint arXiv:1407.7131(2014).Google Scholar
- Tanmay Sinha, Nan Li, Patrick Jermann, and Pierre Dillenbourg. 2014. Capturing ”attrition intensifying” structural traits from didactic interaction sequences of MOOC learners. arXiv preprint arXiv:1409.5887(2014).Google Scholar
- Kurt VanLehn. 2006. The behavior of tutoring systems. International journal of artificial intelligence in education 16, 3(2006), 227–265.Google Scholar
- Howard Wainer and Robert J Mislevy. 1990. Item response theory, item calibration, and proficiency estimation. Computerized adaptive testing: A primer 4 (1990), 65–102.Google Scholar
- Xiaolu Xiong, Siyuan Zhao, Eric G Van Inwegen, and Joseph E Beck. 2016. Going deeper with deep knowledge tracing.International Educational Data Mining Society (2016).Google Scholar
- Chen-Hsiang Yu, Jungpin Wu, and An-Chi Liu. 2019. Predicting learning outcomes with MOOC clickstreams. Education sciences 9, 2 (2019), 104.Google Scholar
- Robert Zemsky. 2014. With a MOOC MOOC here and a MOOC MOOC there, here a MOOC, there a MOOC, everywhere a MOOC MOOC. The Journal of General Education 63, 4 (2014), 237–243.Google ScholarCross Ref
- Hao Zhang, Tao Huang, Zhihan Lv, Sanya Liu, and Heng Yang. 2019. MOOCRC: A highly accurate resource recommendation model for use in MOOC environments. Mobile Networks and Applications 24, 1 (2019), 34–46.Google ScholarDigital Library
- Hao Zhang, Heng Yang, Tao Huang, and Gaoqiang Zhan. 2017. DBNCF: Personalized courses recommendation system based on DBN in MOOC environment. In ISET. IEEE, 106–108.Google Scholar
- Jiani Zhang, Xingjian Shi, Irwin King, and Dit-Yan Yeung. 2017. Dynamic key-value memory networks for knowledge tracing. In Proceedings of the 26th international conference on World Wide Web. 765–774.Google ScholarDigital Library
- Liang Zhang, Xiaolu Xiong, Siyuan Zhao, Anthony Botelho, and Neil T Heffernan. 2017. Incorporating rich features into deep knowledge tracing. In Proceedings of the fourth (2017) ACM conference on learning@ scale. 169–172.Google ScholarDigital Library
- Yingbin Zhang, Luc Paquette, Ryan S Baker, Jaclyn Ocumpaugh, Nigel Bosch, Anabil Munshi, and Gautam Biswas. 2020. The relationship between confusion and metacognitive strategies in Betty’s Brain. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge. 276–284.Google ScholarDigital Library
- Siqian Zhao, Chunpai Wang, and Shaghayegh Sahebi. 2020. Modeling knowledge acquisition from multiple learning resource types. arXiv preprint arXiv:2006.13390(2020).Google Scholar
Index Terms
- What do Students’ Interactions with Online Lecture Videos Reveal about their Learning?
Index terms have been assigned to the content through auto-classification.
Recommendations
Students' Social Presence in Online Learning System
LATICE '14: Proceedings of the 2014 International Conference on Teaching and Learning in Computing and EngineeringThe purposes of this study are to identify students' perceptions on social presence on online learning and to investigate students' social presence in online learning. Next, this study will also identify students' social presence in online learning in ...
Developing a diagnostic tool for assessing tertiary students' readiness for online learning
The increasing investment in developing high quality systems for online learning has focused more on technology with little regard for learners' readiness to learn through such systems. One of the reasons for this was an assumption that most students ...
Applying online competency-based learning and design-based learning to enhance the development of students’ skills in using PowerPoint and Word, self-directed learning readiness, and experience of online learning
AbstractOnline learning has been widely adopted in higher education, because it can help both teachers and students to achieve educational goals through better accessibility, flexibility, and interaction. As the Internet and educational technologies have ...
Comments