ABSTRACT
Encountering the same compiler error repeatedly, particularly several times consecutively, has been cited as a strong indicator that a student is struggling with important programming concepts. Despite this, there are relatively few studies which investigate repeated errors in isolation or in much depth. There are also few data-driven metrics for measuring programming performance, and fewer for measuring repeated errors. This paper makes two contributions. First we introduce a new metric to quantify repeated errors, the repeated error density (RED). We compare this to Jadud's Error Quotient (EQ), the most studied metric, and show that RED has advantages over EQ including being less context dependent, and being useful for short sessions. This allows us to answer two questions posited by Jadud in 2006 that have until now been unanswered. Second, we compare the EQ and RED scores using data from an empirical control/intervention group study involving an editor which enhances compiler error messages. This intervention group has been previously shown to have a reduced overall number of student errors, number of errors per student, and number of repeated student errors per compiler error message. In this research we find a reduction in EQ, providing further evidence that error message enhancement has positive effects. In addition we find a significant reduction in RED providing evidence that this metric is valid.
- A. Ahadi, R. Lister, H. Haapala, and A. Vihavainen. Exploring machine learning methods to automatically identify students in need of assistance. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research, pages 121--130. ACM, 2015. Google ScholarDigital Library
- A. Altadmri and N. C. Brown. 37 million compilations: Investigating novice programming mistakes in large-scale student data. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education, pages 522--527. ACM, 2015. Google ScholarDigital Library
- B. A. Becker. An exploration of the e ects of enhanced compiler error messages for computer programming novices. Master's thesis, Dublin Institute of Technology, 2015.Google Scholar
- B. A. Becker. An e ective approach to enhancing compiler error messages. In Proceedings of the 47th ACM Technical Symposium on Computer Science Education, pages 126--131. ACM, 2016. Google ScholarDigital Library
- J. S. Brown and K. VanLehn. Repair theory: A generative theory of bugs in procedural skills. Cognitive science, 4(4):379--426, 1980.Google ScholarCross Ref
- N. C. Brown and A. Altadmri. Investigating novice programming mistakes: Educator beliefs vs. student data. In Proceedings of the tenth annual Conference on International Computing Education Research, pages 43--50. ACM, 2014. Google ScholarDigital Library
- N. C. C. Brown, M. Kölling, D. McCall, and I. Utting. Blackbox: A large scale repository of novice programmers' activity. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education, pages 223--228. ACM, 2014. Google ScholarDigital Library
- R. Chmiel and M. C. Loui. Debugging: from novice to expert. ACM SIGCSE Bulletin, 36(1):17--21, 2004. Google ScholarDigital Library
- P. Denny, A. Luxton-Reilly, E. Tempero, and J. Hendrickx. Codewrite: supporting student-driven practice of java. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education, pages 471--476. ACM, 2011. Google ScholarDigital Library
- M. C. Jadud. A rst look at novice compilation behaviour using bluej. Computer Science Education, 15(1):25--40, 2005.Google ScholarCross Ref
- M. C. Jadud. An exploration of novice compilation behaviour in BlueJ. PhD thesis, University of Kent, 2006.Google Scholar
- M. C. Jadud and B. Dorn. Aggregate compilation behavior: Findings and implications from 27,698 users. In Proceedings of the eleventh annual International Conference on International Computing Education Research, pages 131--139. ACM, 2015. Google ScholarDigital Library
- W. L. Johnson. Intention-based diagnosis of novice programming errors. Morgan Kaufmann, 1986. Google ScholarDigital Library
- I. R. Katz and J. R. Anderson. Debugging: An analysis of bug-location strategies. Human-Computer Interaction, 3(4):351--399, 1987. Google ScholarDigital Library
- A. Petersen, J. Spacco, and A. Vihavainen. An exploration of error quotient in multiple contexts. In Proceedings of the 15th Koli Calling Conference on Computing Education Research, pages 77--86. ACM, 2015. Google ScholarDigital Library
- M. M. T. Rodrigo, E. Tabanao, M. B. E. Lahoz, and M. C. Jadud. Analyzing online protocols to characterize novice java programmers. Philippine Journal of Science, 138(2):177--190, 2009.Google Scholar
- J. C. Spohrer and E. Soloway. Novice mistakes: Are the folk wisdoms correct? Communications of the ACM, 29(7):624--632, 1986. Google ScholarDigital Library
- E. S. Tabanao, M. M. Rodrigo, and M. C. Jadud. Predicting at-risk novice java programmers through the analysis of online protocols. In Proceedings of the seventh international workshop on Computing education research, pages 85--92. ACM, 2011. Google ScholarDigital Library
- H. Tamada, A. Ogino, and H. Ueda. A framework for programming process measurement and compiling error interpretation for novice programmers. In Software Measurement, 2011 Joint Conference of the 21st Int'l Workshop on and 6th International Conference on Software Process and Product Measurement (IWSM-MENSURA), pages 233--238. IEEE, 2011. Google ScholarDigital Library
- C. Watson, F. W. Li, and J. L. Godwin. Predicting performance in an introductory programming course by logging and analyzing student programming behavior. In Advanced Learning Technologies (ICALT), 2013 IEEE 13th International Conference on, pages 319--323. IEEE, 2013 Google ScholarDigital Library
Index Terms
- A New Metric to Quantify Repeated Compiler Errors for Novice Programmers
Recommendations
An Effective Approach to Enhancing Compiler Error Messages
SIGCSE '16: Proceedings of the 47th ACM Technical Symposium on Computing Science EducationOne of the many challenges novice programmers face from the time they write their first program is inadequate compiler error messages. These messages report details on errors the programmer has made and are the only feedback the programmer gets from the ...
Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research
ITiCSE-WGR '19: Proceedings of the Working Group Reports on Innovation and Technology in Computer Science EducationDiagnostic messages generated by compilers and interpreters such as syntax error messages have been researched for over half of a century. Unfortunately, these messages which include error, warning, and run-time messages, present substantial difficulty ...
Enhancing syntax error messages appears ineffectual
ITiCSE '14: Proceedings of the 2014 conference on Innovation & technology in computer science educationDebugging is an important skill for novice programmers to acquire. Error messages help novices to locate and correct errors, but compiler messages are frequently inadequate. We have developed a system that provides enhanced error messages, including ...
Comments