Skip to main content

Advertisement

SpringerLink
Log in

Knowledge-Based Design Analytics for Authoring Courses with Smart Learning Content

  • Article
  • Published:
International Journal of Artificial Intelligence in Education Aims and scope Submit manuscript

Abstract

Over the last 10 years, learning analytics have provided educators with both dashboards and tools to understand student behaviors within specific technological environments. However, there is a lack of work to support educators in making data-informed design decisions when designing a blended course and planning appropriate learning activities. In this paper, we introduce knowledge-based design analytics that uncover facets of the learning activities that are being created. A knowledge-based visualization is integrated into edCrumble, a (blended) learning design authoring tool. This new approach is explored in the context of a higher education programming course, where instructors design labs and home practice sessions with online smart learning content on a weekly basis. We performed a within-subjects user study to compare the use of the design tool both with and without visualization. We studied the differences in terms of cognitive load, controllability, confidence and ease of choice, design outcomes, and user actions within the system to compare both conditions with the objective of evaluating the impact of using design analytics during the decision-making phase of course design. Our results indicate that the use of a knowledge-based visualization allows the teachers to reduce the cognitive load (especially in terms of mental demand) and that it facilitates the choice of the most appropriate activities without affecting the overall design time. In conclusion, the use of knowledge-based design analytics improves the overall learning design quality and helps teachers avoid committing design errors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Albó L., & Hernández-Leo D. (2018). edCrumble: Designing for learning with data analytics. In V. Pammer-Schindler., M. Pérez-Sanagustín., H. Drachsler., R. Elferink & M. Scheffel (Eds.), Lifelong technology-enhanced learning. EC-TEL 2018. Lecture notes in computer science, vol. 11082 (pp. 605–608). Springer. https://doi.org/10.1007/978-3-319-98572-5_55.

  • Albó, L., Barria-Pineda, J., Brusilovsky, P., & Hernández-Leo, D. (2019). Concept-level design analytics for blended courses. In M. Scheffel, J. Broisin, V. Pammer-Schindler, A. Ioannou & J. Schneider (Eds.), Transforming Learning with Meaningful Technologies. EC-TEL 2019. Lecture notes in computer science, vol. 11722, (pp. 541–554). Springer. https://doi.org/10.1007/978-3-030-29736-7_40.

  • Bodily, R., & Verbert, K. (2017). Review of research on student-facing learning analytics dashboards and educational recommender systems. IEEE Transactions on Learning Technologies, 10(4), 405–418. https://doi.org/10.1109/TLT.2017.2740172.

    Article  Google Scholar 

  • Bodily, R., Kay, J., Aleven, V., Jivet, I., Davis, D., Xhakaj, F., & Verbert, K. (2018). Open learner models and learning analytics dashboards: A systematic review. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ‘18) (pp. 41–50). ACM. https://doi.org/10.1145/3170358.3170409.

  • Brusilovsky, P. (2001). WebEx: Learning from examples in a programming course. In: WebNet, 1, 124–129.

    Google Scholar 

  • Brusilovsky, P., Edwards, S., Kumar, A., Malmi, L., Benotti, L., Buck, D., Ihantola, P., Prince, R., Sirkiä, T., Sosnovsky, S., Urquiza, J., Vihavainen, A, Wollowski, M. (2014). Increasing adoption of smart learning content for computer science education. In Proceedings of the Working Group Reports of the 2014 on Innovation and Technology in Computer Science Education Conference (ITiCSE-WGR ‘14) (pp. 31-57). ACM. https://doi.org/10.1145/2713609.2713611.

  • Bull, S., & Kay, J. (2007). Student models that invite the learner in the SMILI:() open learner Modelling framework. International Journal of Artificial Intelligence in Education, 17(2), 89–120.

    Google Scholar 

  • Bull, S., Brusilovsky, P., Guerra, J., and Araujo, R. (2016). Individual and peer comparison open learner model visualizations to identify what to work on next. In: Extended Proceedings of 24th ACM Conference on User Modeling, Adaptation and Personalisation, UMAP 2016, Extended proceedings, Halifax, Canada, July 13-17, 2016. http://ceur-ws.org/Vol-1618/LBR4.pdf

  • Bull, S., Brusilovsky, P., & Guerra, J. (2018). Which learning Visualisations to offer students? In: 13th European conference on technology enhanced learning, EC-TEL 2018. Leeds: Springer.

    Book  Google Scholar 

  • Bull, S. (2020). There are open learner models about! IEEE Transactions on Learning Technologies (Early Access)., 13, 425–448. https://doi.org/10.1109/TLT.2020.2978473.

    Article  Google Scholar 

  • Cleveland, W. S., & McGill, R. (1984). Graphical perception: Theory, experimentation, and application to the development of graphical methods. Journal of the American Statistical Association, 79(387), 531–554. https://doi.org/10.1080/01621459.1984.10478080.

  • Corbett, A., McLaughlin, M., & Scarpinatto, C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10(2–3), 81–108. https://doi.org/10.1023/A:1026505626690.

    Article  Google Scholar 

  • Cross, S., Galley, R., Brasher, A., & Weller, M. (2012). OULDI-JISC project evaluation report: The impact of new curriculum design tools and approaches on institutional process and design cultures. OULDI Project, at http://oro.open.ac.uk/34140/

  • Csikszentmihalyi, M. (2008) Flow: The psychology of optimal experience. Harper Perennial Modern Classics.

  • Dillenbourg, P., & Hong, F. (2008). The mechanics of CSCL macro scripts. International Journal of Computer-Supported Collaborative Learning, 3(1), 5–23. https://doi.org/10.1007/s11412-007-9033-1.

    Article  Google Scholar 

  • Goodyear, P., & Carvalho, L. (2014). Framing the analysis of learning network architectures. In The architecture of productive learning networks (pp. 48-70). Routledge.

  • Grann, J., & Bushway, D. (2014). Competency map: Visualizing student learning to promote student success. In Proceedings of the Fourth International Conference on Learning Analytics and Knowledge (LAK ‘14) (pp.168-172). ACM. https://doi.org/10.1145/2567574.2567622.

  • Grier, R. A. (2015). How high is high? A meta-analysis of NASA-TLX global workload scores. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 59(1), 1727–1731. https://doi.org/10.1177/1541931215591373.

    Article  Google Scholar 

  • Guerra-Hollstein, J., Barria-Pineda, J., Schunn, C., Bull, S., & Brusilovsky, P. (2017). Fine-grained open learner models: Complexity versus support. In Proceedings of Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization (pp.41-49). ACM. https://doi.org/10.1145/3079628.3079682.

  • Guerra, J., Schunn, C., Bull, S., Barria-Pineda, J., & Brusilovsky, P. (2018). Navigation support in complex open learner models: Assessing visual design alternatives. New Review of Hypermedia and Multimedia, 24(3), 160–192. https://doi.org/10.1080/13614568.2018.1482375.

    Article  Google Scholar 

  • Hart, S. G. (2012). NASA-task load index (NASA-TLX): 20 years later. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 50(9), (pp. 904–908). https://doi.org/10.1177/154193120605000909.

  • Hernández-Leo, D., Martinez-Maldonado, R., Pardo, A., Muñoz-Cristóbal, J. A., & Rodríguez-Triana, M. J. (2019). Analytics for learning design: A layered framework and tools. British Journal of Educational Technology, 51(1), 139–152. https://doi.org/10.1111/bjet.12645.

    Article  Google Scholar 

  • Hosseini, R., Sirkiä, T., Guerra, J., Brusilovsky, P., Malmi, L. (2016). Animated examples as practice content in a Java programming course. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education - SIGCSE ‘16 (pp. 540–545). ACM. https://doi.org/10.1145/2839509.2844639.

  • Hosseini, R., Akhuseyinoglu, K., Petersen, A., Schunn, C. D., Brusilovsky, P. (2018). PCEX: Interactive program construction examples for learning programming. In Proceedings of the 18th Koli Calling International Conference on Computing Education Research (Koli Calling ‘18) (pp. 1-9). ACM. https://doi.org/10.1145/3279720.3279726.

  • Hsiao, I.-H., Sosnovsky, S., & Brusilovsky, P. (2010). Guiding students to the right questions: Adaptive navigation support in an E-learning system for Java programming. Journal of Computer Assisted Learning, 26(4), 270–283. https://doi.org/10.1111/j.1365-2729.2010.00365.x.

    Article  Google Scholar 

  • Joksimović, S., Kovanović, V., & Dawson, S. (2019). The journey of learning analytics. HERDSA Review of Higher Education, 6, 37–63. http://www.herdsa.org.au/herdsa-review-higher-education-vol-6/37-63

  • Laurillard, D., Charlton, P., Craft, B., Dimakopoulos, D., Ljubojevic, D., Magoulas, G., Masterman, E., Pujadas, R., Whitley, E. A., & Whittlestone, K. (2013). A constructionist learning environment for teachers to model learning designs. Journal of Computer Assisted Learning, 29(1), 15–30. https://doi.org/10.1111/j.1365-2729.2011.00458.x.

    Article  Google Scholar 

  • Laurillard, D., Kennedy, E., Charlton, P., Wild, J., & Dimakopoulos, D. (2018). Using technology to develop teachers as designers of TEL: Evaluating the learning designer. British Journal of Educational Technology, 49(6), 1044–1058. https://doi.org/10.1111/bjet.12697.

    Article  Google Scholar 

  • Loboda, T. D., Guerra, J., Hosseini, R., & Brusilovsky, P. (2014). Mastery grids: An open source social educational progress visualization. In European conference on technology enhanced learning (pp. 235–248). Springer, Cham. https://doi.org/10.1007/978-3-319-11200-8_18.

  • Lockyer, L., & Dawson, S. (2011). Learning designs and learning analytics. In Proceedings of the 1st International Conference on Learning Analytics and Knowledge (LAK ‘11) (pp. 153-156). ACM. https://doi.org/10.1145/2090116.2090140.

  • Martinez-Maldonado, R., Goodyear, P., Carvalho, L., Thompson, K., Hernández-Leo, D., Dimitriadis, Y., Prieto, L., & Wardak, D. (2017). Supporting collaborative design activity in a multi-user digital design ecology. Computers in Human Behavior, 71, 327–342. https://doi.org/10.1016/j.chb.2017.01.055.

    Article  Google Scholar 

  • Michos, K., & Hernández-Leo, D. (2020). CIDA: A collective inquiry framework to study and support teachers as designers in technological environments. Computers & Education, 143(January 2020), 103679. https://doi.org/10.1016/j.compedu.2019.103679.

    Article  Google Scholar 

  • Milligan, S., Corrin, L., Law, N., & Ringtved, U. (2020). DesignLAK20: Developing quality standards for analytic measures of learning for learning design. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 375–378). ACM.

  • Ondov, B. D., Jardine, N., Elmqvist, N., & Franconeri, S. L. (2018). Face to face: Evaluating visual comparison. IEEE Transactions on Visualization and Computer Graphics, 25(1), 861–871. https://doi.org/10.1109/TVCG.2018.2864884.

    Article  Google Scholar 

  • Palavitsinis, N., Manouselis, N., & Sanchez-Alonso, S. (2014). Metadata quality in learning object repositories: A case study. The Electronic Library, 32(1), 62–82. https://doi.org/10.1108/EL-12-2011-0175.

  • Papamitsiou, Z., Giannakos, M. N., & Ochoa, X. (2020). From childhood to maturity: Are we there yet? Mapping the intellectual progress in learning analytics during the past decade. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (LAK ‘20) (pp. 559–568) ACM. https://doi.org/10.1145/3375462.3375519.

  • Papanikolaou, K. A., Grigoriadou, M., Kornilakis, H., & Magoulas, G. D. (2003). Personalising the interaction in a web-based educational hypermedia system: The case of INSPIRE. User Modeling and User Adapted Interaction, 13(3), 213–267. https://doi.org/10.1023/A:1024746731130.

    Article  Google Scholar 

  • Persico, D., Pozzi, F., Anastopoulou, S., Conole, G., Craft, B., Dimitriadis, Y., Hernández-Leo, D., Kali, Y., Mor, Y., Pérez Sanagustín, M., Walmsley, H. (2013). Learning design Rashomon I – Supporting the design of one lesson through different approaches. Journal of Research in Learning Technologies, 21. https://doi.org/10.3402/rlt.v21i0.20224.

  • Reimann, P. (2016). Connecting learning analytics with learning research: The role of design-based research. Learning. Research and Practice, 2, 130–142. https://doi.org/10.1080/23735082.2016.1210198.

  • Saket, B., Endert, A., & Demiralp, Ç. (2018). Task-based effectiveness of basic visualizations. IEEE Transactions on Visualization and Computer Graphics, 25(7), 2505–2512. https://doi.org/10.1109/TVCG.2018.2829750.

    Article  Google Scholar 

  • Sergis, S., & Sampson, D. G. (2017). Teaching and learning analytics to support teacher inquiry: A systematic literature review. In Learning Analytics: Fundaments, Applications, and Trends (pp. 25-63). Springer. https://doi.org/10.1007/978-3-319-52977-6_2.

  • Sosnovsky, S., & Brusilovsky, P. (2015). Evaluation of Topic-based Adaptation & Student Modeling in QuizGuide. User Modeling and User-Adapted Interaction, 25(4), 371–424. https://doi.org/10.1007/s11257-015-9164-4.

    Article  Google Scholar 

  • Toker, D., Conati, C., Carenini, G., & Haraty, M. (2012). Towards adaptive information visualization: On the influence of user characteristics. In International Conference on User Modeling, Adaptation, and Personalization UMAP’12 (pp. 274–285). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-31454-4_23.

  • Treder, M. S. (2010). Behind the looking-glass: A review on human symmetry perception. Symmetry, 2(3), 1510–1543. https://doi.org/10.3390/sym2031510.

    Article  Google Scholar 

  • Villasclaras-Fernández, E. D., Hernández-Leo, D., Asensio-Pérez, J. I., & Dimitriadis, Y. (2013). Web collage: An implementation of support for assessment design in CSCL macro-scripts. Computers & Education, 67, 79–97. https://doi.org/10.1016/j.compedu.2013.03.002.

    Article  Google Scholar 

  • Zingaro, D., Cherenkova, Y., Karpova, O., & Petersen, A. (2013). Facilitating code-writing in PI classes. In The 44th ACM Technical Symposium on Computer Science Education, SIGCSE '13 (pp. 585-590). ACM. https://doi.org/10.1145/2445196.2445369.

Download references

Acknowledgments

The authors would like to thank all the instructors who participated in the study.

Funding

This work is a result of a collaboration within a mobility grant for research funded by the SEBAP, Societat Econòmica Barcelonesa d’Amics del País. This work has also been partially funded by NSF DRL 1740775, “la Caixa Foundation” (CoT project, 100010434) and FEDER, the National Research Agency of the Spanish Ministry of Science, Innovations and Universities MDM-2015-0502, TIN2014–53199-C3–3-R, TIN2017–85179-C3–3-R. D. Hernández-Leo acknowledges the support by ICREA under the ICREA Academia programme.Additionally, the work of one of the authors was funded by CONICYT PFCHA/ Doctorado Becas Chile/ 2018 - 72190680.

Author information

Authors and Affiliations

  1. Department of Information and Communications Technologies, Universitat Pompeu Fabra, Roc Boronat, 138, 08018, Barcelona, Spain

    Laia Albó & Davinia Hernández-Leo

  2. School of Computing and Information, University of Pittsburgh, 135 N. Bellefield Ave, Pittsburgh, 15213, USA

    Jordan Barria-Pineda & Peter Brusilovsky

Authors
  1. Laia Albó
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jordan Barria-Pineda
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Peter Brusilovsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Davinia Hernández-Leo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Laia Albó.

Ethics declarations

Conflicts of Interest/Competing Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. This paper is an extension of the experimental study published in:

Albó, L., Barria-Pineda, J., Brusilovsky, P., & Hernández-Leo, D. (2019). Concept-level design analytics for blended courses. In M. Scheffel, J. Broisin, V. Pammer-Schindler, A. Ioannou & J. Schneider (Eds.), Transforming Learning with Meaningful Technologies. EC-TEL 2019. Lecture Notes in Computer Science, vol. 11,722, (pp. 541–554). Springer. 10.1007/978-3-030-29736-7_40

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Albó, L., Barria-Pineda, J., Brusilovsky, P. et al. Knowledge-Based Design Analytics for Authoring Courses with Smart Learning Content. Int J Artif Intell Educ 32, 4–27 (2022). https://doi.org/10.1007/s40593-021-00253-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40593-021-00253-3

Keywords

Search

Navigation