Skip to main content
SpringerLink
Log in

Using Evaluation to Shape ITS Design: Results and Experiences with SQL-Tutor

  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

This paper presents the results of three evaluation studies performed during 1998 and 1999 on SQL-Tutor, an intelligent tutoring system for the SQL database language. We have evaluated the system in the context of genuine courses, and used the results to further refine the system. The main goal of our research has been the exploration and extension of Constraint-Based Modeling (CBM), a student modeling approach proposed by Ohlsson (1994). SQL-Tutor provided us with experiences of using CBM, and we used it to extend the approach in several important ways. The main goal of all three evaluation studies was to determine how well CBM supported student learning. We have obtained positive results. The students who learnt with SQL-Tutor in the first study performed significantly better than those who did not when assessed by a subsequent classroom examination. Furthermore, the analysis of students' learning shows that CBM has a sound psychological foundation.

Besides the evaluation of CBM, we also evaluated the improvements in terms of student assessments of the usefulness of the system and evaluated various techniques used in SQL-Tutor. In the second study, we evaluated the effectiveness of feedback provided to the students. This study showed that high-level advice is most beneficial to students' learning. The focus of the third study was different. We extended CBM to support long-term modeling of student knowledge, and used this extension to develop an adaptive problem-selection strategy. The study revealed the benefits of this strategy in comparison with a simple heuristic strategy. We also reflect on our experiences in evaluating SQL-Tutor.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Albacete, P. L. and Van Lehn, K.: 2000, The Conceptual Helper: an Intelligent Tutoring System for Teaching Fundamental Physics Concepts. 5th In ternational Conference on Intelligent Tutoring Systems, Montreal, pp. 564–573.

  • Anderson, J. R, Corbett, A. T., Koedinger, K. R. and Pelletier, R.: 1995, Cognitive Tutors: Lessons Learned. Journal of the Learning Sciences 4(2), 167–207.

    Google Scholar 

  • Bloom, B. S.: 1984, The 2 Sigma Problem: the Search for Methods of Group Instruction as Effective as one-to-one Tutoring. Educational Researcher 13, 4–16.

    Google Scholar 

  • Chao-Lin Liu and Wellman, M.: 1998, Incremental Tradeoff Resolution in Qualitative Probability Networks. 14th Conference on Uncertainty in Artificial Intelligence, Madison, WI, pp. 346–353.

  • Charniak, E.: 1991, Bayesian Networks without Tears. AI Magazine, Winter 1991, 50–63.

  • Chin, D. N.: 2000, Empirical Evaluation of User Models and User-adapted Systems. User-Modeling and User Adapted Interaction, 12(2-3), 1–5.

    Google Scholar 

  • Collins, J. A., Greer, J. and Huang, S. X.: 1996, Adaptive Assessment Using Granularity Hierarchies and Bayesian Nets. 3rd International Conference on Intelligent Tutoring Systems, Montreal, pp. 569-577.

  • Conati, C., Gertner, A., Van Lehn, K. and Druzdzel, M. J.: 1997, On-Line Student Modeling for Coached Problem Solving Using Bayesian Networks. 6th International Conference on User Modeling, Sardinia, pp. 231–242.

  • Forgy, C. L.: 1982, Rete: a Fast Algorithm for theMany Pattern/Many Object Pattern Match Problem. Artificial Intelligence 19, 17–37.

    Google Scholar 

  • Gertner, A. S.: 1998, Providing Feedback to Equation Entries in an Intelligent Tutoring System for Physics. 4th International Conference on Intelligent Tutoring Systems, San Antonio, TX, pp. 254–263.

  • Mayo. M.: 2000, A Normative Framework for Intelligent Tutoring Systems, 5th International Conference on Intelligent Tutoring Systems, Montreal (poster).

  • Mayo, M., Mitrovic, A. and McKenzie, J.: 2000, CAPIT: an Intelligent Tutoring System for Capitalization and Punctuation. International Workshop on Advanced Learning Technologies, Palmerston North, NZ, pp. 151–154.

  • Mayo, M. and Mitrovic, A.: 2001, Optimising ITS Behavior with Bayesian Networks and Decision Theory. Accepted for International Journal on Artificial Intelligence in Education.

  • Millan, E. and Perez-de-la-Cruz, J. L.: 2001, A Bayesian Diagnostic Algorithm for Student Modeling and its Evaluation. UMUAI, this volume.

  • Mitrovic, A.: 1998a, A Knowledge-Based Teaching System for SQL. ED-MEDIA'98, Freiburg, pp. 1027–1032.

  • Mitrovic, A.: 1998b, Experiences in Implementing Constraint-Based Modeling in SQL-Tutor. 4th International Conference on Intelligent Tutoring Systems, San Antonio, TX, pp. 414–423.

  • Mitrovic, A. and Hausler, K.: 2000, Porting SQL-Tutor to the Web. ITS'2000 Workshop on Adaptive and Intelligent Web-based Education Systems, Montreal, pp. 37–44.

  • Mitrovic, A. and Ohlsson, S.: 1999, Evaluation of a Constraint-Based Tutor for a Database Language. Int. Journal. on Artificial Intelligence in Education 10(3–4), 238–256.

    Google Scholar 

  • Mitrovic, A. and Suraweera, P.: 2000, Evaluating an Animated Pedagogical Agent. 5th International Conference on Intelligent Tutoring Systems, Montreal, pp. 73–82.

  • Newell, A. and Rosenblum, P. S.: 1981, Mechanisms of skill acquisition and the law of practice. In: J.R. Anderson (ed.), Cognitive Skills and Their Acquisition, pp. 1–55.

  • Ohlsson, S.: 1994, Constraint-based Student Modeling. In: J. E. Greer and G.I. McCalla (eds.), Student Modeling: the Key to Individualized Knowledge-based Instruction, pp. 167–189.

  • Ohlsson, S.: 1996, Learning from Performance Errors. Psychological Review 103(2) 241–262.

    Google Scholar 

  • Pearl, J.: 1988, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann.

  • Reye, J.: 1998. Two-Phase Updating of Student Models Based on Dynamic Belief Networks. 4th International Conference on Intelligent Tutoring Systems, San Antonio, TX, pp. 274–283.

  • Stern, L. and Sterling, L.: 1997, Teaching AI Algorithms using Animations Reinforced by Interactive Exercises. Australian Computer Science Education Conference, Sydney, pp. 78–83.

  • Suraweera, P. and Mitrovic, A.: 2001,Designing a Constraint-based Tutor for Database Design. To be presented at the 9th Int. Conf. on Human-Computer Interaction, New Orleans.

  • Vigotsky, L. S.: 1978, The development of higher psychological processes. Cambridge, MA: Harvard University Press.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Intelligent Computer Tutoring Group, Department of Computer Science, University of Canterbury, Private Bag 4800, Christchurch, New Zealand

    Antonija Mitrovic, Brent Martin & Michael Mayo

Authors
  1. Antonija Mitrovic
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brent Martin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael Mayo
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Mitrovic, A., Martin, B. & Mayo, M. Using Evaluation to Shape ITS Design: Results and Experiences with SQL-Tutor. User Modeling and User-Adapted Interaction 12, 243–279 (2002). https://doi.org/10.1023/A:1015022619307

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1015022619307

Search

Navigation