Brett A. Becker
Paul Denny
Raymond Pettit
Durell Bouchard
Brian Harrington
Amey Karkare
Peter-Michael Osera
Janice L. Pearce
James Prather
ABSTRACT
Diagnostic messages generated by compilers and interpreters such as syntax error messages have been researched for over half of a century. Unfortunately, these messages which include error, warning, and run-time messages, present substantial difficulty and could be more effective, particularly for novices. Recent years have seen an increased number of papers in the area including studies on the effectiveness of these messages, improving or enhancing them, and their usefulness as a part of programming process data that can be used to predict student performance, track student progress, and tailor learning plans. Despite this increased interest, the long history of literature is quite scattered and has not been brought together in any digestible form.
In order to help the computing education community (and related communities) to further advance work on programming error messages, we present a comprehensive, historical and state-of-the-art report on research in the area. In addition, we synthesise and present the existing evidence for these messages including the difficulties they present and their effectiveness. We finally present a set of guidelines, curated from the literature, classified on the type of evidence supporting each one (historical, anecdotal, and empirical). This work can serve as a starting point for those who wish to conduct research on compiler error messages, runtime errors, and warnings. We also make the bibtex file of our 300+ reference corpus publicly available. Collectively this report and the bibliography will be useful to those who wish to design better messages or those that aim to measure their effectiveness, more effectively.
- Alireza Ahadi, Arto Hellas, Petri Ihantola, Ari Korhonen, and Andrew Petersen. 2016. Replication in Computing Education Research: Researcher Attitudes and Experiences. In Proceedings of the 16th Koli Calling International Conference on Computing Education Research (Koli Calling '16). ACM, New York, NY, USA, 2--11. https://doi.org/10.1145/2999541.2999554Google Scholar
Digital Library
- Marzieh Ahmadzadeh, Dave Elliman, and Colin Higgins. 2005. An Analysis of Patterns of Debugging Among Novice Computer Science Students. In Proceedings of the 10th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (ITiCSE '05). ACM, New York, NY, USA, 84--88. https://doi.org/ 10.1145/1067445.1067472Google ScholarDigital Library
- Umair Z. Ahmed, Pawan Kumar, Amey Karkare, Purushottam Kar, and Sumit Gulwani. 2018. Compilation Error Repair: For the Student Programs, from the Student Programs. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET '18). ACM, New York, NY, USA, 78--87. https://doi.org/10.1145/3183377.3183383Google ScholarDigital Library
- Umair Z. Ahmed, Renuka Sindhgatta, Nisheeth Srivastava, and Amey Karkare. 2019. Targeted Example Generation for Compilation Errors. In Proceedings of the 34th ACM/IEEE International Conference on Automated Software Engineering (ASE '19). ACM, New York, NY, USA, 12.Google ScholarDigital Library
- Amjad Altadmri, Michael Kolling, and Neil C. C. Brown. 2016. The Cost of Syntax and How to Avoid It: Text versus Frame-Based Editing. In 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC '16). IEEE, 748--753. https://doi.org/10.1109/COMPSAC.2016.204Google ScholarCross Ref
- Paul Ayres and John Sweller. 1990. Locus of Difficulty in Multistage Mathematics Problems. The American Journal of Psychology 103, 2 (1990), 167--193. http: //www.jstor.org/stable/1423141Google ScholarCross Ref
- Paul L. Ayres. 2001. Systematic Mathematical Errors and Cognitive Load. Contemporary Educational Psychology 26, 2 (2001), 227 -- 248. https://doi.org/10.1006/ ceps.2000.1051Google ScholarCross Ref
- Titus Barik. 2016. How Should Static Analysis Tools Explain Anomalies to Developers? A Communication Theory of Computationally Supporting Developer Self-Explanations for Static Analysis Anomalies. (2016). http://static.barik.net/ barik/proposal/barik_proposal_approved.pdfGoogle Scholar
- Titus Barik. 2018. Error Messages as Rational Reconstructions. Ph.D. Dissertation. North Carolina State University, Raleigh. https://repository.lib.ncsu.edu/handle/ 1840.20/35439Google Scholar
- Titus Barik, Denae Ford, Emerson Murphy-Hill, and Chris Parnin. 2018. How Should Compilers Explain Problems to Developers?. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2018). ACM, New York, NY, USA, 633--643. https://doi.org/10.1145/3236024.3236040Google ScholarDigital Library
- Titus Barik, Kevin Lubick, Samuel Christie, and Emerson Murphy-Hill. 2014. How Developers Visualize Compiler Messages: A Foundational Approach to Notification Construction. In 2014 Second IEEE Working Conference on Software Visualization. IEEE, 87--96. https://doi.org/10.1109/VISSOFT.2014.24Google ScholarDigital Library
- Titus Barik, Justin Smith, Kevin Lubick, Elisabeth Holmes, Jing Feng, Emerson Murphy-Hill, and Chris Parnin. 2017. Do Developers Read Compiler Error Messages?. In Proceedings of the 39th International Conference on Software Engineering (ICSE '17). IEEE Press, Piscataway, NJ, USA, 575--585. https: //doi.org/10.1109/ICSE.2017.59Google ScholarDigital Library
- D. W. Barron. 1975. A Note on APL. Comput. J. 19, 1 (1975), 93. https://academic. oup.com/comjnl/article-pdf/19/1/93/1058172/190093.pdfGoogle ScholarCross Ref
- Brett A. Becker. 2015. An Exploration Of The Effects Of Enhanced Compiler Error Messages For Computer Programming Novices. Masters Thesis. Dublin Institute of Technology. https://doi.org/10.13140/RG.2.2.26637.13288Google Scholar
- Brett A. Becker. 2016. An Effective Approach to Enhancing Compiler Error Messages. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education (SIGCSE '16). ACM, New York, NY, USA, 126--131. https://doi. org/10.1145/2839509.2844584Google ScholarDigital Library
- Brett A. Becker. 2016. A New Metric to Quantify Repeated Compiler Errors for Novice Programmers. In Proceedings of the 21st ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '16). ACM, New York, NY, USA, 296--301. https://doi.org/10.1145/2899415.2899463Google ScholarDigital Library
- Brett A. Becker. 2019. Parlez-vous Java? Bonjour La Monde != Hello World: Barriers to Programming Language Acquisition for Non-Native English Speakers. In Proceedings of the 30th Annual Conference of the Psychology of Programming Interest Group (PPIG '19). http://www.ppig.org/sites/ppig.org/files/ 2019-PPIG-30th-becker.pdfGoogle Scholar
- Brett A. Becker, Graham Glanville, Ricardo Iwashima, Claire McDonnell, Kyle Goslin, and Catherine Mooney. 2016. Effective Compiler Error Message Enhancement for Novice Programming Students. Computer Science Education 26, 2--3 (2016), 148--175. https://doi.org/10.1080/08993408.2016.1225464Google ScholarCross Ref
- Brett A. Becker, Kyle Goslin, and Graham Glanville. 2018. The Effects of Enhanced Compiler Error Messages on a Syntax Error Debugging Test. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (SIGCSE '18). ACM, New York, NY, USA, 640--645. https://doi.org/10.1145/3159450.3159461Google ScholarDigital Library
- Brett A. Becker and Catherine Mooney. 2016. Categorizing Compiler Error Messages with Principal Component Analysis. In Proceedings of the 12th China- Europe International Symposium on Software Engineering Education (CEISEE '16). Shenyang, China, 1--8. https://researchrepository.ucd.ie/handle/10197/7889Google Scholar
- Brett A. Becker, Cormac Murray, Tianyi Tao, Changheng Song, Robert McCartney, and Kate Sanders. 2018. Fix the First, Ignore the Rest: Dealing with Multiple Compiler Error Messages. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (SIGCSE '18). ACM, New York, NY, USA, 634--639. https://doi.org/10.1145/3159450.3159453Google ScholarDigital Library
- Brett A. Becker and Keith Quille. 2019. 50 Years of CS1 at SIGCSE: A Review of the Evolution of Introductory Programming Education Research. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (SIGCSE '19). ACM, New York, NY, USA, 338--344. https://doi.org/10.1145/3287324.3287432Google Scholar
- T.R. Beelders and Jean-Pierre L. du Plessis. 2016. The Influence of Syntax Highlighting on Reading and Comprehending Code. Journal of Eye Movement Research Beelders 91, 1 (2016), 1--11. https://doi.org/10.16910/jemr.9.1.1Google Scholar
- Mordechai (Moti) Ben-Ari. 2007. Compile and Runtime Errors in Java. (2007). http://www.weizmann.ac.il/sci-tea/benari/sites/sci-tea.benari/files/ uploads/softwareAndLearningMaterials/errors.pdfGoogle Scholar
- Joe Bergin, Achla Agarwal, and Krishna Agarwal. 2003. Some Deficiencies of C++ in Teaching CS1 and CS2. ACM SIGPLAN Notices 38, 6 (2003), 9--13. https://doi.org/10.1145/885638.885642Google ScholarDigital Library
- Sahil Bhatia and Rishabh Singh. 2016. Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks. In 2nd Indian Workshop on Machine Learning (IWML '16). https://www2.cse.iitk.ac.in/~iwml/ 2016/papers/iWML_2016_paper_5.pdfGoogle Scholar
- Michael W. Bigrigg, Russell Bortz, Shyamal Chandra, David Reed, Jared Sheehan, and Sara Smith. 2003. An Evaluation of the Usefulness of Compiler Error Messages. Technical Report. http://www.ices.cmu.edu/reports/040903.pdfGoogle Scholar
- Dennis Bouvier, Ellie Lovellette, John Matta, Bedour Alshaigy, Brett A. Becker, Michelle Craig, Jana Jackova, Robert McCartney, Kate Sanders, and Mark Zarb. 2016. Novice Programmers and the Problem Description Effect. In Proceedings of the 2016 ITiCSE Working Group Reports (ITiCSE-WGR '16). ACM, New York, NY, USA, 103--118. https://doi.org/10.1145/3024906.3024912Google ScholarDigital Library
- Bernd Braßel. 2004. Typehope: There is Hope for Your Type Errors. In Int. Workshop on Implementation of Functional Languages.Google Scholar
- Pearl Brereton, Barbara A. Kitchenham, David Budgen, Mark Turner, and Mohamed Khalil. 2007. Lessons from Applying the Systematic Literature Review Process Within the Software Engineering Domain. J. Syst. Softw. 80, 4 (April 2007), 571--583. https://doi.org/10.1016/j.jss.2006.07.009Google ScholarDigital Library
- Neil C. C. Brown and Amjad Altadmri. 2017. Novice Java Programming Mistakes: Large-Scale Data vs. Educator Beliefs. ACM Transactions on Computing Education 17, 2, Article 7 (May 2017), 21 pages. https://doi.org/10.1145/2994154Google Scholar
- Neil C. C. Brown, Amjad Altadmri, Sue Sentance, and Michael Kölling. 2018. Blackbox, Five Years On. In Proceedings of the 2018 ACM Conference on International Computing Education Research - ICER '18. ACM, Espoo, Finland, 196--204. https://doi.org/10.1145/3230977.3230991Google ScholarDigital Library
- Neil C. C. Brown, Michael Kölling, Davin McCall, and Ian Utting. 2014. Blackbox: A Large Scale Repository of Novice Programmers' Activity. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (SIGCSE '14). ACM, New York, NY, USA, 223--228. https://doi.org/10.1145/2538862.2538924Google ScholarDigital Library
- Neil C. C. Brown and Greg Wilson. 2018. Ten Quick Tips For Teaching Programming. PLoS Computational Biology 14, 4 (Apr 2018), e1006023. https: //doi.org/10.1371/journal.pcbi.1006023Google ScholarCross Ref
- P. J. Brown. 1982. My System Gives Excellent Error Messages - Or Does It? Software: Practice and Experience 12, 1 (Jan 1982), 91--94. https://doi.org/10.1002/ spe.4380120110Google ScholarCross Ref
- P. J. Brown. 1983. Error Messages: The Neglected Area of the Man/Machine Interface. Commun. ACM 26, 4 (Apr 1983), 246--249. https://doi.org/10.1145/2163. 358083Google ScholarDigital Library
- Andrew Buxton and Lesley Trenner. 1987. An Experiment to Assess the Friendliness of Error Messages from Interactive Information Retrieval Systems. Journal of Information Science 13, 4 (Aug 1987), 197--209. https://doi.org/10.1177/ 016555158701300403Google ScholarDigital Library
- Adam S. Carter, Christopher D. Hundhausen, and Olusola Adesope. 2015. The Normalized Programming State Model: Predicting Student Performance in Computing Courses Based on Programming Behavior. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research (ICER '15). ACM, New York, NY, USA, 141--150. https://doi.org/10.1145/2787622. 2787710Google ScholarDigital Library
- A. T. Chamillard and William C. Hobart, Jr. 1997. Transitioning to Ada in an Introductory Course for Non-majors. In Proceedings of the Conference on TRI-Ada (TRI-Ada '97). ACM, New York, NY, USA, 37--40. https://doi.org/10.1145/269629. 269634Google Scholar
- G. V. Cormack. 1989. An LR Substring Parser for Noncorrecting Syntax Error Recovery. In Proceedings of the ACM SIGPLAN 1989 Conference on Programming Language Design and Implementation (PLDI '89). ACM, New York, NY, USA, 161--169. https://doi.org/10.1145/73141.74832Google ScholarDigital Library
- Natalie J Coull. 2008. SNOOPIE: Development of a Learning Support Tool for Novice Programmers within a Conceptual Framework. Ph.D. Dissertation. University of St Andrews, St Andrews, Scotland. http://hdl.handle.net/10023/522Google Scholar
- Natalie J. Coull and Ishbel M.M. Duncan. 2011. Emergent Requirements for Supporting Introductory Programming. Innovations in Teaching and Learning in Information and Computer Sciences (ITaLICS) 10, 1 (2011), 78--85. https://doi.org/ 10.11120/ital.2011.10010078Google Scholar
- Paul Cress, Paul Dirksen, andWesley J Graham. 1970. FORTRAN IV WithWATFOR and WATFIV. Prentice-Hall, Englewood Cliffs, New Jersey.Google Scholar
- Edgar Dale and Jeanne S Chall. 1949. The Concept of Readability. Elementary English 26, 1 (1949), 19--26.Google Scholar
- E A Davis, M C Linn, and M Clancy. 1995. Learning to Use Parentheses and Quotes in LISP. Computer Science Education 6, 1 (1995), 15--31. https://doi.org/10. 1080/0899340950060102Google ScholarCross Ref
- Carla De Lira. 2017. Improving the Learning Experiences of First-Year Computer Science Students with Empathetic IDEs. In Proceedings of the 2017 ACM Conference on International Computing Education Research (ICER '17). ACM, New York, NY, USA, 293--294. https://doi.org/10.1145/3105726.3105742Google ScholarDigital Library
- Morris Dean. 1982. How a Computer Should Talk To People. IBM Systems Journal 21, 4 (1982), 424--453. https://doi.org/10.1147/sj.214.0424Google ScholarDigital Library
- Paul Denny, Brett A. Becker, Michelle Craig, Greg Wilson, and Piotr Banaszkiewicz. 2019. Research This! Questions That Computing Educators Most Want Computing Education Researchers to Answer. In Proceedings of the 2019 ACM Conference on International Computing Education Research (ICER '19). ACM, New York, NY, USA, 259--267. https://doi.org/10.1145/3291279.3339402Google ScholarDigital Library
- Paul Denny, Brian Hanks, and Beth Simon. 2010. PeerWise: Replication Study of a Student-collaborative Self-testing Web Service in a U.S. Setting. In Proceedings of the 41st ACM Technical Symposium on Computer Science Education (SIGCSE '10). ACM, New York, NY, USA, 421--425. https://doi.org/10.1145/1734263.1734407Google ScholarDigital Library
- Paul Denny, Andrew Luxton-Reilly, and Dave Carpenter. 2014. Enhancing Syntax Error Messages Appears Ineffectual. In Proceedings of the 19th Conference on Innovation and Technology in Computer Science Education (ITiCSE '14). ACM, New York, NY, USA, 273--278. https://doi.org/10.1145/2591708.2591748Google ScholarDigital Library
- Paul Denny, Andrew Luxton-Reilly, and Ewan Tempero. 2012. All Syntax Errors Are Not Equal. In Proceedings of the 17th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE '12). ACM, New York, NY, USA, 75--80. https://doi.org/10.1145/2325296.2325318Google ScholarDigital Library
- Paul Denny, Andrew Luxton-Reilly, Ewan Tempero, and Jacob Hendrickx. 2011. CodeWrite: Supporting Student-driven Practice of Java. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education (SIGCSE '11). ACM, New York, NY, USA, 471--476. https://doi.org/10.1145/1953163.1953299Google ScholarDigital Library
- Paul Denny, Andrew Luxton-Reilly, Ewan Tempero, and Jacob Hendrickx. 2011. Understanding the Syntax Barrier for Novices. In Proceedings of the 16th Annual Joint Conference on Innovation and Technology in Computer Science Education (ITiCSE '11). ACM, New York, NY, USA, 208--212. https://doi.org/10.1145/1999747. 1999807Google ScholarDigital Library
- Paul Denny, James Prather, Brett A. Becker, Zachary Albrecht, Dastyni Loksa, and Raymond Pettit. 2019. A Closer Look at Metacognitive Scaffolding: Solving Test Cases Before Programming. In Proceedings of the 19th Koli Calling International Conference on Computing Education Research (Koli Calling '19). ACM, New York, NY, USA. https://doi.org/10.1145/3364510.3366170Google ScholarDigital Library
- Gergely Dévai, Dániel Leskó, and Máté Tejfel. 2013. The EDSL's Struggle for Their Sources. In: Zsók V., Horváth Z., Csató L. (eds) Central European Functional Programming School. CEFP 2013. Lecture Notes in Computer Science, Vol. 8606. Springer, Cham. 300--335 pages. https://doi.org/10.1007/978--3--319--15940--9_7Google Scholar
- Tao Dong and Kandarp Khandwala. 2019. The Impact of "Cosmetic" Changes on the Usability of Error Messages. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1--6. https://doi.org/10.1145/3290607. 3312978Google ScholarDigital Library
- Benedict du Boulay and Ian Matthew. 1984. Fatal Error in Pass Zero: How Not to Confuse Novices. Behaviour and Information Technology 3, 2 (1984), 109--118. https://doi.org/10.1080/01449298408901742Google ScholarCross Ref
- William H DuBay. 2007. Smart Language: Readers, Readability, and the Grading of Text. ERIC.Google Scholar
- Thomas Dy and Ma. Mercedes Rodrigo. 2010. A Detector for Non-literal Java Errors. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research (Koli Calling '10). ACM, New York, NY, USA, 118--122. https://doi.org/10.1145/1930464.1930485Google ScholarDigital Library
- Eddie Antonio Santos, Joshua Charles Campbell, Dhvani Patel, Abram Hindle and José Nelson Amaral. 2018. Syntax and Sensibility: Using Language Models to Detect and Correct Syntax Errors. 311--322 pages.Google Scholar
- Marc Eisenstadt and Matthew W. Lewis. 2018. Errors in an Interactive Programming Environment: Causes and Cures. In Novice Programming Environments, Mark Eisenstadt, Mark T. Keane, and Tim Rajan (Eds.). Routledge, London, Chapter 5, 111--131.Google Scholar
- Nabil El Boustani and Jurriaan Hage. 2010. Corrective Hints for Type Incorrect Generic Java Programs. In Proceedings of the 2010 ACM SIGPLAN Workshop on Partial Evaluation and Program Manipulation (PEPM '10). ACM, New York, NY, USA, 5--14. https://doi.org/10.1145/1706356.1706360Google ScholarDigital Library
- Nabil El Boustani and Jurriaan Hage. 2011. Improving Type Error Messages for Generic Java. In Higher-Order and Symbolic Computation, Vol. 24. Savannah, GA, 3--39. https://doi.org/10.1007/s10990-011--9070--3Google ScholarDigital Library
- U. Engelmann and H. P. Meinzer. 1985. Rules for the Design of End User Languages. In Medical Informatics Europe 85, F. H. Roger, P. Grönroos, R. Tervo- Pellikka, and R. O'Moore (Eds.). Springer, Berlin, Heidelberg, Helsinki, Finland, 240--245. https://doi.org/10.1007/978--3--642--93295--3_48Google Scholar
- Anneli Eteläpelto. 1993. Metacognition and the Expertise of Computer Program Comprehension. Scandinavian Journal of Educational Research 37, 3 (1993), 243-- 254. https://doi.org/10.1080/0031383930370305Google ScholarCross Ref
- Andrew Ettles, Andrew Luxton-Reilly, and Paul Denny. 2018. Common Logic Errors Made by Novice Programmers. In Proceedings of the 20th Australasian Computing Education Conference (ACE '18). ACM, New York, NY, USA, 83--89. https://doi.org/10.1145/3160489.3160493Google ScholarDigital Library
- Georgios Evangelidis, Vassilios Dagdilelis, Maria Satratzemi, and Vassilios Efopoulos. 2001. X-compiler: Yet Another Integrated Novice Programming Environment. In Proceedings IEEE International Conference on Advanced Learning Technologies. IEEE Comput. Soc, 166--169. https://doi.org/10.1109/ICALT.2001.943890Google ScholarCross Ref
- S. Fincher. 1999. What are we doing when we teach programming?. In FIE'99 Frontiers in Education. 29th Annual Frontiers in Education Conference. Designing the Future of Science and Engineering Education. Conference Proceedings (IEEE Cat. No.99CH37011, Vol. 1. 12A4/1--12A4/5 vol.1. https://doi.org/10.1109/FIE.1999. 839268Google ScholarCross Ref
- Robert Bruce Findler, John Clements, Cormac Flanagan, Matthew Flatt, Shriram Krishnamurthi, Paul Steckler, and Matthias Felleisen. 2002. DrScheme: A Programming Environment for Scheme. Journal of Functional Programming 12, 2 (March 2002), 159--182. https://doi.org/10.1017/S0956796801004208Google ScholarDigital Library
- Allan Fisher and Jane Margolis. 2002. Unlocking the Clubhouse: The Carnegie Mellon Experience. SIGCSE Bull. 34, 2 (June 2002), 79--83. https://doi.org/10.1145/ 543812.543836Google ScholarDigital Library
- Julie Fisher. 1999. The Importance of User Message Text and Why Professional Writers Should Be Involved. Australian Computer Journal 31 (Nov 1999), 118--123.Google Scholar
- Thomas Flowers, Curtis Carver, and James Jackson. 2004. Empowering students and building confidence in novice programmers through gauntlet. In 34th ASEE/IEEE Annual Frontiers in Education, 2004. FIE 2004. IEEE, Savannah, GA, USA, T3H10--13. https://doi.org/10.1109/fie.2004.1408551Google ScholarCross Ref
- Edward B Fry. 2006. Readability: Reading Hall of Fame Book. Newark. DE: International Reading Association (2006).Google Scholar
- Richard Furuta and P. Michael Kemp. 1979. Experimental Evaluation of Programming Language Features: Implications for Introductory Programming Languages. In Proceedings of the 10th SIGCSE Technical Symposium on Computer Science Education (SIGCSE '79). ACM, New York, NY, USA, 18--21. https: //doi.org/10.1145/800126.809544Google Scholar
- Susan L. Graham and Steven P. Rhodes. 1973. Practical Syntactic Error Recovery in Compilers. In Proceedings of the 1st Annual ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages (POPL '73). ACM, New York, NY, USA, 52--58. https://doi.org/10.1145/512927.512932Google Scholar
- David Gries. 1968. Use of Transition Matrices in Compiling. Commun. ACM 11, 1 (Jan. 1968), 26--34. https://doi.org/10.1145/362851.362872Google Scholar
- David Gries. 1974. What Should We Teach in an Introductory Programming Course?. In Proceedings of the Fourth SIGCSE Technical Symposium on Computer Science Education (SIGCSE '74). ACM, New York, NY, USA, 81--89. https://doi.org/ 10.1145/800183.810447Google ScholarDigital Library
- Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017. DeepFix: Fixing Common C Language Errors by Deep Learning. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI'17). AAAI Press, 1345--1351. http://dl.acm.org/citation.cfm?id=3298239.3298436Google ScholarCross Ref
- Jurriaan Hage and Heeren Bastiaan. 2006. Heuristics for Type Error Discovery and Recovery. In 18th International Conference on Implementation and Application of Functional Languages (IFL '06). 199--216. https://link.springer.com/chapter/10. 1007/978--3--540--74130--5_12Google Scholar
- Devon Harker. 2017. Examining the Effects of Enhanced Compilers on Student Productivity. Masters Thesis. University of Northern British Columbia. https: //unbc.arcabc.ca/islandora/object/unbc%3A58897Google Scholar
- Jan Lee Harrington. 1984. The Effect of Error Messages on Learning Computer Programming by Individuals Without Prior Programming Experience. PhD Thesis. Drexel University.Google Scholar
- Theodore L Harris and Richard E Hodges. 1995. The Literacy Dictionary: The Vocabulary of Reading and Writing. ERIC.Google Scholar
- Björn Hartmann, Daniel MacDougall, Joel Brandt, and Scott R. Klemmer. 2010. What Would Other Programmers Do: Suggesting Solutions to Error Messages. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 1019--1028. https://doi.org/10.1145/1753326. 1753478Google Scholar
- J. Hartz, Adam. 2012. CAT-SOOP: A Tool for Automatic Collection and Assessment of Homework Exercises. Master's thesis. Massachusetts Institute of Technology. https://dspace.mit.edu/bitstream/handle/1721.1/77086/825763362-MIT.pdf? sequence=2Google Scholar
- Brian Harvey. 1982. Why Logo? Byte 7, 8 (Aug 1982), 163--195. http://cmkfutures. com/wp-content/uploads/2017/06/Why-Logo-by-Brian-Harvey.pdfGoogle Scholar
- Bastiaan Heeren, Jurriaan Hage, and S. Doaitse Swierstra. 2003. Scripting the Type Inference Process. In Proceedings of the 8th ACM SIGPLAN International Conference on Functional Programming (ICFP '03). ACM, New York, NY, USA, 3--13. https://doi.org/10.1145/944705.944707Google Scholar
- Bastiaan Heeren, Daan Leijen, and Arjan van IJzendoorn. 2003. Helium, for Learning Haskell. In Proceedings of the 2003 ACM SIGPLAN Workshop on Haskell (Haskell '03). ACM, New York, NY, USA, 62--71. https://doi.org/10.1145/871895. 871902Google ScholarDigital Library
- Zef Hemel, Danny M. Groenewegen, Lennart C.L. Kats, and Eelco Visser. 2011. Static Consistency Checking of Web Applications with WebDSL. Journal of Symbolic Computation 46, 2 (2011), 150--182. https://doi.org/10.1016/j.jsc.2010.08. 006Google ScholarDigital Library
- Richard Hill. 2008. Developing a Teaching Compiler for Students Learning the C Programming Language. Bachelor of Science Dissertation. University of Bath.Google Scholar
- James J Horning. 1976. What the Compiler Should Tell the User. In Compiler Construction: An Advanced Course, G Goos and J Hartmanis (Eds.). Springer- Verlag, Berlin-Heidelberg, 525--548.Google Scholar
- C. D. Hundhausen, D. M. Olivares, and A. S. Carter. 2017. IDE-Based Learning Analytics for Computing Education: A Process Model, Critical Review, and Research Agenda. ACM Transactions on Computing Education 17, 3, Article 11 (Aug. 2017), 26 pages. https://doi.org/10.1145/3105759Google ScholarDigital Library
- Petri Ihantola, Arto Vihavainen, Alireza Ahadi, Matthew Butler, Jürgen Börstler, Stephen H. Edwards, Essi Isohanni, Ari Korhonen, Andrew Petersen, Kelly Rivers, Miguel Ángel Rubio, Judy Sheard, Bronius Skupas, Jaime Spacco, Claudia Szabo, and Daniel Toll. 2015. Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies. In Proceedings of the 2015 ITiCSE Working Group Reports (ITICSE-WGR '15). ACM, New York, NY, USA, 41--63. https://doi.org/10.1145/2858796.2858798Google ScholarDigital Library
- Barbara S. Isa, James M. Boyle, Alan S. Neal, and Roger M. Simons. 1983. A Methodology for Objectively Evaluating Error Messages. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '83). ACM, New York, NY, USA, 68--71. https://doi.org/10.1145/800045.801583Google Scholar
- ISO/IEC 14882:2011 2011. Information Technology -- Programming languages -- C++. Technical Report. https://www.iso.org/standard/50372.htmlGoogle Scholar
- ISO/IEC TS 19217:2015 2015. Information Technology -- Programming Languages -- C++ Extensions for Concepts. Technical Report. https://www.iso.org/standard/ 64031.htmlGoogle Scholar
- J. Jackson, M. Cobb, and C. Carver. 2005. Identifying Top Java Errors for Novice Programmers. In 35th Annual Frontiers in Education Conference (FIE '05). T4C--24 -- T4C--27. https://doi.org/10.1109/fie.2005.1611967Google Scholar
- Matthew C Jadud. 2005. A First Look at Novice Compilation Behaviour Using BlueJ. Computer Science Education 15, 1 (2005), 25--40. https://doi.org/10.1080/ 08993400500056530 arXiv:https://doi.org/10.1080/08993400500056530Google ScholarCross Ref
- Matthew C. Jadud. 2006. An Exploration of Novice Compilation Behaviour in BlueJ. Ph.D. Dissertation. University of Kent at Canterbury. https://jadud.com/dl/pdf/ jadud-dissertation.pdfGoogle Scholar
- Mathias Johan Johansen. 2015. Errors and Misunderstandings Among Novice Programmers Assessing the Student Not the Program. Masters Thesis. University of Oslo. https://www.duo.uio.no/handle/10852/49045Google Scholar
- Eliezer Kantorowitz and H. Laor. 1986. Automatic Generation of Useful Syntax Error Messages. Software: Practice and Experience 16, 7 (1986), 627--640.Google ScholarDigital Library
- Ioannis Karvelas. 2019. Investigating Novice Programmers' Interaction with Programming Environments. In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '19). ACM, New York, NY, USA, 336--337. https://doi.org/10.1145/3304221.3325596Google ScholarDigital Library
- Caitlin Kelleher, Dennis Cosgrove, and David Culyba. 2002. Alice2: Programming Without Syntax Errors. User Interface Software and Technology - UIST 2002 (2002), 3--4. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.60.4640&rep= rep1&type=pdfGoogle Scholar
- B. Kitchenham and S. Charters. 2007. Guidelines for Performing Systematic Literature Reviews in Software Engineering, version 2.3. (2007).Google Scholar
- Amy J Ko, Brad A Myers, and Htet Htet Aung. 2004. Six Learning Barriers in End-User Programming Systems Prior Research on Learning Barriers A Study of Visual Basic . NET 2003. In Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing (VLHCC '04). 199--206. https://doi.org/10.1109/ VLHCC.2004.47Google Scholar
- Tobias Kohn. 2017. Teaching Python Programming to Novices: Addressing Misconceptions and Creating a Development Environment. PhD Thesis. ETH Zürich.Google Scholar
- Tobias Kohn. 2019. The Error Behind The Message: Finding the Cause of Error Messages in Python. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (SIGCSE '19). ACM, New York, NY, USA, 524--530. https://doi.org/10.1145/3287324.3287381Google ScholarDigital Library
- Michael Kölling. 1999. The Design of an Object-Oriented Environment and Language for Teaching. Ph.D. Dissertation. University of Sydney. https://kar.kent.ac. uk/21868/1/the_design_of_an_object-oriented_kolling.pdfGoogle Scholar
- Michael Kölling. 2015. Lessons from the Design of Three Educational Programming Environments: Blue, BlueJ and Greenfoot. International Journal of People- Oriented Programming 4, 1 (2015), 5--32. https://doi.org/10.4018/ijpop.2015010102Google ScholarDigital Library
- Michael Kölling, Bruce Quig, Andrew Patterson, and John Rosenberg. 2003. The BlueJ System and its Pedagogy. Computer Science Education 13, 4 (Dec 2003), 249--268. https://doi.org/10.1076/csed.13.4.249.17496Google ScholarCross Ref
- Sarah K. Kummerfeld and Judy Kay. 2003. The Neglected Battle Fields of Syntax Errors. In Proceedings of the Fifth Australasian Conference on Computing Education - Volume 20 (ACE '03). Australian Computer Society, Inc., Darlinghurst, Australia, Australia, 105--111. http://dl.acm.org/citation.cfm?id=858403.858416Google Scholar
- Thomas Kurtz. 1978. BASIC. ACM SIGPLAN Notices - Special issue: History of programming languages conference 13, 8 (1978), 103--118. https://doi.org/10.1145/ 960118.808376Google Scholar
- Nicolas Laurent. 2017. Red Shift: Procedural Shift-reduce Parsing (Vision Paper). In Proceedings of the 10th ACM SIGPLAN International Conference on Software Language Engineering (SLE '17). ACM, New York, NY, USA, 38--42. https://doi. org/10.1145/3136014.3136036Google ScholarDigital Library
- Michael J. Lee and Amy J. Ko. 2011. Personifying Programming Tool Feedback Improves Novice Programmers' Learning. In Proceedings of the Seventh International Workshop on Computing Education Research (ICER '11). ACM, New York, NY, USA, 109--116. https://doi.org/10.1145/2016911.2016934Google Scholar
- Oukseh Lee and Kwangkeun Yi. 1998. Proofs About a Folklore Let-polymorphic Type Inference Algorithm. ACM Trans. Program. Lang. Syst. 20, 4 (July 1998), 707--723. https://doi.org/10.1145/291891.291892Google ScholarDigital Library
- Ronald Paul Leinius. 1970. Error Detection and Recovery for Syntax Directed Compiler Systems. Ph.D. Dissertation. AAI7024758.Google Scholar
- Benjamin S. Lerner, Matthew Flower, Dan Grossman, and Craig Chambers. 2007. Searching for Type-error Messages. In Proceedings of the 28th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '07). ACM, New York, NY, USA, 425--434. https://doi.org/10.1145/1250734.1250783Google ScholarDigital Library
- Stuart Lewis and Gaius Mulley. 1998. A Comparison Between Novice and Experienced Compiler Users in a Learning Environment. In Proceedings of the 6th Annual Conference on the Teaching of Computing and the 3rd Annual Conference on Integrating Technology into Computer Science Education: Changing the Delivery of Computer Science Education (ITiCSE '98). ACM, New York, NY, USA, 157--161. https://doi.org/10.1145/282991.283106Google Scholar
- William Lidwell, Kritina Holden, and Jill Butler. 2010. Universal Principles of Design, Revised and Updated: 125 Ways to Enhance Usability, Influence Perception, Increase Appeal, Make Better Design Decisions, and Teach through Design. Rockport Publishers.Google Scholar
- Derrell Lipman. 2014. LearnCS! a Browser-Based Research Platform for CS1 and Studying the Role of Instruction of Debugging from Early in the Course. Ph.D. Dissertation. University of Massachusetts Lowell.Google Scholar
- Charles R. Litecky and Gordon B. Davis. 1976. A Study of Errors, Errorproneness, and Error Diagnosis in Cobol. Commun. ACM 19, 1 (1976), 33--38. https://doi.org/10.1145/359970.359991Google ScholarDigital Library
- Dastyni Loksa, Amy J. Ko, Will Jernigan, Alannah Oleson, Christopher J. Mendez, and Margaret M. Burnett. 2016. Programming, Problem Solving, and Self- Awareness: Effects of Explicit Guidance. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1449--1461. https://doi.org/10.1145/2858036.2858252Google Scholar
- Glenn R Luecke, James Coyle, James Hoekstra, Marina Kraeva, and Ying Xu. 2009. The Importance of Run-time Error Detection. In Tools for High Performance Computing 2009. 145--155. https://doi.org/10.1007/978--3--642--11261--4Google Scholar
- Glenn R. Luecke, James Coyle, James Hoekstra, Marina Kraeva, Ying Xu, Elizabeth Kleiman, and Olga Weiss. 2009. Evaluating Error Detection Capabilities of UPC Run-time Systems. In Proceedings of the Third Conference on Partitioned Global Address Space Programing Models (PGAS '09). ACM, New York, NY, USA, Article 7, 4 pages. https://doi.org/10.1145/1809961.1809971Google ScholarDigital Library
- Harri Luoma, Essi Lahtinen, and Hannu-Matti Järvinen. 2007. CLIP, a Command Line Interpreter for a Subset of C++. In Proceedings of the Seventh Baltic Sea Conference on Computing Education Research - Volume 88 (Koli Calling '07). Australian Computer Society, Inc., Darlinghurst, Australia, Australia, 199--202. http://dl.acm.org/citation.cfm?id=2449323.2449351Google ScholarDigital Library
- Andrew Luxton-Reilly. 2016. Learning to Program is Easy. In Proceedings of the 21st ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '16). ACM, New York, NY, USA, 284--289. https://doi.org/10.1145/2899415. 2899432Google ScholarDigital Library
- Andrew Luxton-Reilly, Brett A. Becker, Yingjun Cao, Roger McDermott, Claudio Mirolo, Andreas Mühling, Andrew Petersen, Kate Sanders, Simon, and Jacqueline Whalley. 2017. Developing Assessments to Determine Mastery of Programming Fundamentals. In Proceedings of the 2017 ITiCSE Conference on Working Group Reports (ITiCSE-WGR '17). ACM, New York, NY, USA, 47--69. https://doi.org/10. 1145/3174781.3174784Google ScholarDigital Library
- Andrew Luxton-Reilly, Simon, Ibrahim Albluwi, Brett A. Becker, Michail Giannakos, Amruth N. Kumar, Linda Ott, James Paterson, Michael James Scott, Judy Sheard, and Claudia Szabo. 2018. Introductory Programming: A Systematic Literature Review. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '18). ACM, New York, NY, USA, 55--106. https://doi.org/10.1145/3293881.3295779Google ScholarDigital Library
- Celeste S. Magers. 1983. An Experimental Evaluation of On-line HELP for Non-programmers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '83). ACM, New York, NY, USA, 277--281. https: //doi.org/10.1145/800045.801626Google ScholarDigital Library
- Qusay H. Mahmoud, Wlodek Dobosiewicz, and David Swayne. 2004. Redesigning Introductory Computer Programming with HTML, JavaScript, and Java. SIGCSE Bull. 36, 1 (March 2004), 120--124. https://doi.org/10.1145/1028174.971344Google ScholarDigital Library
- John Maloney, Mitchel Resnick, Natalie Rusk, Brian Silverman, and Evelyn Eastmond. 2010. The Scratch Programming Language and Environment. ACM Transactions on Computing Education 10, 4, Article 16 (Nov. 2010), 15 pages. https://doi.org/10.1145/1868358.1868363Google ScholarDigital Library
- Murali Mani and Quamrul Mazumder. 2013. Incorporating Metacognition into Learning. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education (SIGCSE '13). ACM, New York, NY, USA, 53--58. https://doi.org/10. 1145/2445196.2445218Google ScholarDigital Library
- Guillaume Marceau, Kathi Fisler, and Shriram Krishnamurthi. 2011. Measuring the Effectiveness of Error Messages Designed for Novice Programmers. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education (SIGCSE '11). ACM, New York, NY, USA, 499--504. https://doi.org/10.1145/1953163.1953308Google ScholarDigital Library
- Guillaume Marceau, Kathi Fisler, and Shriram Krishnamurthi. 2011. Mind Your Language: On Novices' Interactions with Error Messages. In Proceedings of the 10th SIGPLAN Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward! 2011). ACM, New York, NY, USA, 3--18. https://doi.org/10.1145/2048237.2048241Google ScholarDigital Library
- Samiha Marwan, Nicholas Lytle, Joseph Jay Williams, and Thomas Price. 2019. The Impact of Adding Textual Explanations to Next-step Hints in a Novice Programming Environment. In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '19). ACM, New York, NY, USA, 520--526. https://doi.org/10.1145/3304221.3319759Google ScholarDigital Library
- Richard E. Mayer. 2004. Teaching of Subject Matter. Annual Review of Psychology 55, 1 (Feb 2004), 715--744. https://doi.org/10.1146/annurev.psych.55.082602.133124Google ScholarCross Ref
- G Harry Mc Laughlin. 1969. SMOG Grading -- A New Readability Formula. Journal of reading 12, 8 (1969), 639--646.Google Scholar
- Bruce J. McAdam. 1998. On the Unification of Substitutions in Type Inference. Technical Report. 1--23 pages. http://www.lfcs.inf.ed.ac.uk/reports/98/ ECS-LFCS-98--384/ECS-LFCS-98--384.pdfGoogle Scholar
- Davin Mccall. 2016. Novice Programmer Errors-Analysis and Diagnostics. Ph.D. Dissertation. The University of Kent. https://kar.kent.ac.uk/id/eprint/61340Google Scholar
- Davin McCall and Michael Kölling. 2014. Meaningful Categorisation of Novice Programmer Errors. In IEEE Frontiers in Education Conference (FIE '14). 1--8. https://doi.org/10.1109/FIE.2014.7044420Google Scholar
- Davin McCall and Michael Kölling. 2019. A New Look at Novice Programmer Errors. ACM Transactions on Computing Education 19, 4 (2019), 1--30. https: //doi.org/10.1145/3335814 arXiv:10.1145/3335814Google ScholarDigital Library
- Michael McCracken, Vicki Almstrum, Danny Diaz, Mark Guzdial, Dianne Hagan, Yifat Ben-David Kolikant, Cary Laxer, Lynda Thomas, Ian Utting, and Tadeusz Wilusz. 2001. A Multi-national, Multi-institutional Study of Assessment of Programming Skills of First-year CS Students. SIGCSE Bull. 33, 4 (Dec. 2001), 125--180. https://doi.org/10.1145/572139.572181Google ScholarDigital Library
- L. McIver and D. Conway. 1996. Seven Deadly Sins of Introductory Programming Language Design. In International Conference on Software Engineering: Education and Practice (SEEP '96). IEEE Computer Society, Dunedin, New Zealand, 309--316. https://doi.org/10.1109/SEEP.1996.534015Google Scholar
- Linda Kathryn McIver. 2001. Syntactic and Semantic Issues in Introductory Programming Education. PhD Thesis. Monash University.Google Scholar
- R. P. Medeiros, G. L. Ramalho, and T. P. Falcão. 2019. A Systematic Literature Review on Teaching and Learning Introductory Programming in Higher Education. IEEE Transactions on Education 62, 2 (May 2019), 77--90. https: //doi.org/10.1109/TE.2018.2864133Google ScholarDigital Library
- Rolf Molich and Jakob Nielsen. 1990. Improving a Human-computer Dialogue. Commun. ACM 33, 3 (March 1990), 338--348. https://doi.org/10.1145/77481.77486Google ScholarDigital Library
- P. G. Moulton and M. E. Muller. 1967. DITRAN - A Compiler Emphasizing Diagnostics. Commun. ACM 10, 1 (1967), 45--52. https://doi.org/10.1145/363018. 363060Google ScholarDigital Library
- F Mulder. 2016. Awesome Error Messages for Dotty. (Oct 2016). https://www. scala-lang.org/blog/2016/10/14/dotty-errors.htmlGoogle Scholar
- Christian Murphy, Eunhee Kim, Gail Kaiser, and Adam Cannon. 2008. Backstop: A Tool for Debugging Runtime Errors. In Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education (SIGCSE '08). ACM, New York, NY, USA, 173--177. https://doi.org/10.1145/1352135.1352193Google ScholarDigital Library
- Emerson Murphy-Hill, Titus Barik, and Andrew P. Black. 2013. Interactive Ambient Visualizations For Soft Advice. Information Visualization 12, 2 (2013), 107--132. https://doi.org/10.1177/1473871612469020Google ScholarCross Ref
- Emerson Murphy-Hill and Andrew P. Black. 2012. Programmer-Friendly Refactoring Errors. IEEE Transactions on Software Engineering 38, 6 (2012), 1417--1431. https://doi.org/10.1109/TSE.2011.110Google ScholarDigital Library
- Scott Nesbitt. 2017. How to Write Better Error Messages. (Aug 2017). https: //opensource.com/article/17/8/write-effective-error-messagesGoogle Scholar
- Eric Niebler. 2007. Proto: A Compiler Construction Toolkit for DSELs. In Proceedings of the 2007 Symposium on Library-Centric Software Design (LCSD '07). ACM, New York, NY, USA, 42--51. https://doi.org/10.1145/1512762.1512767Google ScholarDigital Library
- Marie-Hélène Nienaltowski, Michela Pedroni, and Bertrand Meyer. 2008. Compiler Error Messages: What Can Help Novices?. In Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education (SIGCSE '08). ACM, New York, NY, USA, 168--172. https://doi.org/10.1145/1352135.1352192Google ScholarDigital Library
- Sagar Parihar, Ziyaan Dadachanji, Praveen Kumar Singh, Rajdeep Das, Amey Karkare, and Arnab Bhattacharya. 2017. Automatic Grading and Feedback Using Program Repair for Introductory Programming Courses. In Proceedings of the 22nd ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '17). ACM, New York, NY, USA, 92--97. https://doi.org/10.1145/3059009. 3059026Google ScholarDigital Library
- Miranda C. Parker, Mark Guzdial, and Shelly Engleman. 2016. Replication, Validation, and Use of a Language Independent CS1 Knowledge Assessment. In Proceedings of the 2016 ACM Conference on International Computing Education Research (ICER '16). ACM, New York, NY, USA, 93--101. https://doi.org/10.1145/ 2960310.2960316Google ScholarDigital Library
- Terence Parr and Kathleen Fisher. 2011. LL(*): The Foundation of the ANTLR Parser Generator. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '11). ACM, New York, NY, USA, 425--436. https://doi.org/10.1145/1993498.1993548Google ScholarDigital Library
- D. N. Perkins and Fay Martin. 1986. Fragile Knowledge and Neglected Strategies in Novice Programmers. In Papers Presented at the First Workshop on Empirical Studies of Programmers on Empirical Studies of Programmers. Ablex Publishing Corp., Norwood, NJ, USA, 213--229. http://dl.acm.org/citation.cfm?id=21842. 28896Google Scholar
- Raymond S. Pettit, John Homer, and Roger Gee. 2017. Do Enhanced Compiler Error Messages Help Students?: Results Inconclusive.. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE '17). ACM, New York, NY, USA, 465--470. https://doi.org/10.1145/3017680.3017768Google ScholarDigital Library
- Phitchaya Mangpo Phothilimthana and Sumukh Sridhara. 2017. High-Coverage Hint Generation for Massive Courses: Do Automated Hints Help CS1 Students?. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '17). ACM, New York, NY, USA, 182--187. https://doi.org/10.1145/3059009.3059058Google ScholarDigital Library
- James Prather. 2018. Beyond Automated Assessment: Building Metacognitive Awareness in Novice Programmers in CS1. Ph.D. Dissertation. Nova Southeastern University.Google Scholar
- James Prather, Raymond Pettit, Brett A. Becker, Paul Denny, Dastyni Loksa, Alani Peters, Zachary Albrecht, and Krista Masci. 2019. First Things First: Providing Metacognitive Scaffolding for Interpreting Problem Prompts. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (SIGCSE '19). ACM, New York, NY, USA, 531--537. https://doi.org/10.1145/3287324.3287374Google ScholarDigital Library
- James Prather, Raymond Pettit, Kayla McMurry, Alani Peters, John Homer, and Maxine Cohen. 2018. Metacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools. In Proceedings of the 2018 ACM Conference on International Computing Education Research (ICER '18). ACM, New York, NY, USA, 41--50. https://doi.org/10.1145/3230977.3230981Google ScholarDigital Library
- James Prather, Raymond Pettit, Kayla Holcomb McMurry, Alani Peters, John Homer, Nevan Simone, and Maxine Cohen. 2017. On Novices' Interaction with Compiler Error Messages: A Human Factors Approach. In Proceedings of the 2017 ACM Conference on International Computing Education Research (ICER '17). ACM, New York, NY, USA, 74--82. https://doi.org/10.1145/3105726.3106169Google ScholarDigital Library
- David Pritchard. 2015. Frequency Distribution of Error Messages. In Proceedings of the 6th Workshop on Evaluation and Usability of Programming Languages and Tools. 1--8. https://doi.org/10.1145/nnnnnnn.nnnnnnn arXiv:1509.07238v1Google ScholarDigital Library
- Yizhou Qian and James Lehman. 2017. Students' Misconceptions and Other Difficulties in Introductory Programming: A Literature Review. ACM Transactions on Computing Education 18, 1, Article 1 (Oct. 2017), 24 pages. https: //doi.org/10.1145/3077618Google ScholarDigital Library
- Keith Quille, Roisin Faherty, Susan Bergin, and Brett A. Becker. 2018. Second Level Computer Science: The Irish K-12 Journey Begins. In Proceedings of the 18th Koli Calling International Conference on Computing Education Research (Koli Calling '18). ACM, New York, NY, USA, Article 22, 5 pages. https://doi.org/10. 1145/3279720.3279742Google Scholar
- Timothy Rafalski, P. Merlin Uesbeck, Cristina Panks-Meloney, Patrick Daleiden, William Allee, Amelia Mcnamara, and Andreas Stefik. 2019. A Randomized Controlled Trial on the Wild Wild West of Scientific Computing with Student Learners. In Proceedings of the 2019 ACM Conference on International Computing Education Research (ICER '19). ACM, New York, NY, USA, 239--247. https://doi. org/10.1145/3291279.3339421Google ScholarDigital Library
- Vincent Rahli, Joe Wells, John Pirie, and Fairouz Kamareddine. 2015. Skalpel: A Type Error Slicer for Standard ML. Electronic Notes in Theoretical Computer Science 312 (2015), 197--213. https://doi.org/10.1016/j.entcs.2015.04.012Google ScholarDigital Library
- Vincent Rahli, Joe Wells, John Pirie, and Fairouz Kamareddine. 2017. Skalpel: A Constraint-based Type Error Slicer for Standard ML. Journal of Symbolic Computation 80 (May 2017), 164--208. https://doi.org/10.1016/j.jsc.2016.07.013Google ScholarDigital Library
- Kyle Reestman and Brian Dorn. 2019. Native Language's Effect on Java Compiler Errors. In Proceedings of the 2019 ACM Conference on International Computing Education Research (ICER '19). ACM, New York, NY, USA, 249--257. https://doi. org/10.1145/3291279.3339423Google ScholarDigital Library
- Charles Reis and Robert Cartwright. 2004. Taming a Professional IDE for the Classroom. In Proceedings of the 35th SIGCSE Technical Symposium on Computer Science Education (SIGCSE '04). ACM, New York, NY, USA, 156--160. https://doi. org/10.1145/971300.971357Google ScholarDigital Library
- H. G. Rice. 1953. Classes of Recursively Enumerable Sets and Their Decision Problems. Trans. Amer. Math. Soc. 74, 2 (1953), 358--366. http://www.jstor.org/ stable/1990888Google ScholarCross Ref
- Liam Rigby, Paul Denny, and Andrew Luxton-Reilly. 2020. A Miss is as Good as a Mile: Off-By-One Errors and Arrays in an Introductory Programming Course. In Proceedings of the 22nd Australasian Computing Education Conference (ACE '20). ACM, New York, NY, USA. https://doi.org/10.1145/3373165.3373169Google ScholarDigital Library
- Peter C. Rigby and Suzanne Thompson. 2005. Study of Novice Programmers Using Eclipse and Gild. In Proceedings of the 2005 OOPSLA Workshop on Eclipse Technology eXchange (eclipse '05). ACM, New York, NY, USA, 105--109. https: //doi.org/10.1145/1117696.1117718Google ScholarDigital Library
- Anthony Robins, Janet Rountree, and Nathan Rountree. 2003. Learning and Teaching Programming: A Review and Discussion. Computer Science Education 13, 2 (2003), 137--172. https://doi.org/10.1076/csed.13.2.137.14200Google ScholarCross Ref
- Christine Rogerson and Elsje Scott. 2017. The Fear Factor: How It Affects Students Learning to Program in a Tertiary Environment. Journal of Information Technology Education: Research 9 (2017), 147--171. https://doi.org/10.28945/1183Google ScholarCross Ref
- Saul Rosen, Robert A. Spurgeon, and Joel K. Donnelly. 1965. PUFFT - The Purdue University Fast FORTRAN Translator. Commun. ACM 8, 11 (nov 1965), 661--666. https://doi.org/10.1145/365660.365671Google ScholarDigital Library
- Caitlin Sadowski, Jeffrey van Gogh, Ciera Jaspan, Emma Söderberg, and Collin Winter. 2015. Tricorder: Building a Program Analysis Ecosystem. In Proceedings of the 37th International Conference on Software Engineering - Volume 1 (ICSE '15). IEEE Press, Piscataway, NJ, USA, 598--608. http://dl.acm.org/citation.cfm?id= 2818754.2818828Google ScholarDigital Library
- Advait Sarkar. 2015. The Impact of Syntax Colouring on Program Comprehension. Proceedings of the 26th Annual Conference of the Psychology of Programming Interest Group (PPIG '15) (2015), 49--58. http://www.ppig.org/library/paper/ impact-syntax-colouring-program-comprehensionGoogle Scholar
- Thomas Schilling. 2012. Constraint-Free Type Error Slicing. In Proceedings of the 12th International Conference on Trends in Functional Programming (TFP'11). Springer-Verlag, Berlin, Heidelberg, 1--16. https://doi.org/10.1007/ 978--3--642--32037--8_1Google ScholarDigital Library
- Jean Scholtz and Susan Wiedenbeck. 1993. Using Unfamiliar Programming Languages: The Effects on Expertise. Interacting with Computers 5, 1 (1993), 13--30.Google ScholarCross Ref
- Tom Schorsch. 1995. CAP: An Automated Self-assessment Tool to Check Pascal Programs for Syntax, Logic and Style Errors. In Proceedings of the Twenty-sixth SIGCSE Technical Symposium on Computer Science Education (SIGCSE '95). ACM, New York, NY, USA, 168--172. https://doi.org/10.1145/199688.199769Google ScholarDigital Library
- Hyunmin Seo, Caitlin Sadowski, Sebastian Elbaum, Edward Aftandilian, and Robert Bowdidge. 2014. Programmers' Build Errors: A Case Study (at Google). In Proceedings of the 36th International Conference on Software Engineering - ICSE 2014. ACM Press, New York, New York, USA, 724--734. https://doi.org/10.1145/ 2568225.2568255Google ScholarDigital Library
- Alejandro Serrano and Jurriaan Hage. 2016. Type Error Diagnosis for Embedded DSLs by Two-Stage Specialized Type Rules. In Proceedings of the 25th European Symposium on Programming Languages and Systems - Volume 9632. Springer- Verlag New York, Inc., New York, NY, USA, 672--698. https://doi.org/10.1007/ 978--3--662--49498--1_26Google ScholarCross Ref
- Dale Shaffer, Wendy Doube, and Juhani Tuovinen. 2003. Applying Cognitive Load Theory to Computer Science Education.. In Proceedings of the 15th Annual Workshop of the Psychology of Programming Interest Group (PPIG '03). 333--346. http://ppig.org/library/paper/ applying-cognitive-load-theory-computer-science-educationGoogle Scholar
- W. J. Shaw. 1989. Making APL Error Messages Kinder and Gentler. In Conference Proceedings on APL As a Tool of Thought (APL '89). ACM, New York, NY, USA, 320--324. https://doi.org/10.1145/75144.75188Google Scholar
- Ben Shneiderman. 1982. Designing Computer System Messages. Commun. ACM 25, 9 (1982), 610--611. https://doi.org/10.1145/358628.358639Google ScholarDigital Library
- Ben Shneiderman and Catherine Plaisant. 2004. Designing the User Interface: Strategies for Effective Human-Computer Interaction (4th ed.). Pearson Addison Wesley.Google Scholar
- M. E. Sime, A. T. Arblaster, and T. R. Green. 1977. Structuring the Programmer's Task. Journal of Occupational Psychology 50, 3 (sep 1977), 205--216. https://doi. org/10.1111/j.2044--8325.1977.tb00376.xGoogle ScholarCross Ref
- Jaime Spacco, Paul Denny, Brad Richards, David Babcock, David Hovemeyer, James Moscola, and Robert Duvall. 2015. Analyzing StudentWork Patterns Using Programming Exercise Data. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (SIGCSE '15). ACM, New York, NY, USA, 18--23. https://doi.org/10.1145/2676723.2677297Google ScholarDigital Library
- Andreas Stefik and Stefan Hanenberg. 2014. The Programming Language Wars: Questions and Responsibilities for the Programming Language Community. In Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software (Onward! 2014). ACM, New York, NY, USA, 283--299. https://doi.org/10.1145/2661136.2661156Google ScholarDigital Library
- Andreas Stefik, Bonita Sharif, Brad. A. Myers, and Stefan Hanenberg. 2018. Evidence About Programmers for Programming Language Design (Dagstuhl Seminar 18061). Dagstuhl Reports 8, 2 (2018), 1--25. https://doi.org/10.4230/ DagRep.8.2.1Google Scholar
- Andreas Stefik and Susanna Siebert. 2013. An Empirical Investigation into Programming Language Syntax. ACM Transactions on Computing Education 13, 4, Article 19 (Nov. 2013), 40 pages. https://doi.org/10.1145/2534973Google ScholarDigital Library
- S Suhailan, S Abdul Samad, and M A Burhanuddin. 2014. A Perspective of Automated Programming Error Feedback Approaches in Problem Solving Exercises. Journal of Theoretical and Applied Information Technology 70, 1 (2014), 121--129. http://www.jatit.org/volumes/Vol70No1/16Vol70No1.pdfGoogle Scholar
- John Sweller. 1988. Cognitive Load During Problem Solving: Effects on Learning. Cognitive science 12, 2 (1988), 257--285. https://www.sciencedirect.com/science/ article/pii/0364021388900237Google Scholar
- Emily S. Tabanao, Ma. Mercedes T. Rodrigo, and Matthew C. Jadud. 2011. Predicting At-risk Novice Java Programmers Through the Analysis of Online Protocols. In Proceedings of the 7th International Workshop on Computing Education Research (ICER '11). ACM, New York, NY, USA, 85--92. https: //doi.org/10.1145/2016911.2016930Google Scholar
- Tim Teitelbaum and Thomas Reps. 1981. The Cornell Program Synthesizer: A Syntax-Directed Programming Environment. Commun. ACM 24, 9 (sep 1981), 563--573. https://doi.org/10.1145/358746.358755Google ScholarDigital Library
- Warren Teitelman and Larry Masinter. 1981. The Interlisp Programming Environment. Computer 14, 4 (1981), 25--33. https://doi.org/10.1109/C-M.1981.220410Google ScholarDigital Library
- Emillie Thiselton and Christoph Treude. 2019. Enhancing Python Compiler Error Messages via Stack Overflow. In Proceedings of the 19th International Symposium on Empirical Software Engineering and Measurement (ESEM '19). arXiv:1906.11456. http://arxiv.org/abs/1906.11456Google Scholar
- Suzanne Marie Thompson. 2006. An Exploratory Study of Novice Programming Experiences and Errors. Masters Thesis. University of Victoria.Google Scholar
- Warren Toomey. 2011. Quantifying The Incidence of Novice Programmers' Errors. (2011), 5 pages. https://minnie.tuhs.org/Programs/BlueJErrors/arjen_ draft.pdfGoogle Scholar
- V. Javier Traver. 2010. On Compiler Error Messages: What They Say and What They Mean. Advances in Human-Computer Interaction 2010, Article 3 (Jan. 2010), 26 pages. https://doi.org/10.1155/2010/602570Google Scholar
- Kota Uchida and Katsuhiko Gondow. 2016. C-Helper: C Latent-error Static/Heuristic Checker for Novice Programmers. In Proceedings of the 8th International Conference on Computer Supported Education (CSEDU 2016). SciTePress - Science and Technology Publications, Lda, Portugal, 321--329. https://doi.org/10. 5220/0005797703210329Google ScholarDigital Library
- Miguel Ulloa. 1983. A survey of run-time and logic errors in a classroom environment. ACM SIGCUE Outlook 17, 3 (1983), 21--25. https://doi.org/10.1145/ 1045078.1045081Google ScholarDigital Library
- Leo C. Ureel II and Charles Wallace. 2019. Automated Critique of Early Programming Antipatterns. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. 738--744. https://doi.org/10.1145/3287324.3287463Google Scholar
- Christopher Watson, Frederick W.B. Li, and Jamie L. Godwin. 2014. No Tests Required: Comparing Traditional and Dynamic Predictors of Programming Success. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (SIGCSE '14). ACM, New York, NY, USA, 469--474. https://doi.org/10.1145/2538862.2538930Google Scholar
- Christopher Watson, Frederick W. B. Li, and Jamie L. Godwin. 2012. BlueFix: Using Crowd-sourced Feedback to Support Programming Students in Error Diagnosis and Repair. In Proceedings of the 11th International Conference on Advances in Web-Based Learning (ICWL'12). Springer-Verlag, Berlin, Heidelberg, 228--239. https://doi.org/10.1007/978--3--642--33642--3_25Google Scholar
- Richard L.Wexelblat. 1976. Maxims for Malfeasant Designers, or How to Design Languages to Make Programming As Difficult As Possible. In Proceedings of the 2nd International Conference on Software Engineering (ICSE '76). IEEE Computer Society Press, Los Alamitos, CA, USA, 331--336. http://dl.acm.org/citation.cfm? id=800253.807695Google ScholarDigital Library
- Emily Wilska. 2004. Non-Fatal Errors : Creating Usable , Effective Error Messages. http://www.writersua.com/articles/message/index.html. (2004).Google Scholar
- Niklaus Wirth. 1968. PL360, a Programming Language for the 360 Computers. J. ACM 15, 1 (Jan. 1968), 37--74. https://doi.org/10.1145/321439.321442Google Scholar
- Alexander William Wong, Amir Salimi, Shaiful Chowdhury, and Abram Hindle. 2019. Syntax and Stack Overflow: A Methodology for Extracting a Corpus of Syntax Errors and Fixes. (Jul 2019). arXiv:1907.07803 http://arxiv.org/abs/1907. 07803Google Scholar
- John Wrenn and Shriram Krishnamurthi. 2017. Error Messages Are Classifiers: A Process to Design and Evaluate Error Messages. In Proceedings of the 2017 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward! 2017). ACM, New York, NY, USA, 134--147. https://doi.org/10.1145/3133850.3133862Google ScholarDigital Library
- Baijun Wu, John Peter Campora III, and Sheng Chen. 2017. Learning User Friendly Type-error Messages. Proc. ACM Program. Lang. 1, OOPSLA, Article 106 (Oct. 2017), 29 pages. https://doi.org/10.1145/3133930Google ScholarDigital Library
- Stelios Xinogalos, Maya Satratzemi, and Vassilios Dagdilelis. 2006. An Introduction to Object-Oriented Programming with a Didactic Microworld: objectKarel. Computers and Education 47, 2 (2006), 148--171. https://doi.org/10.1016/j.compedu. 2004.09.005Google ScholarDigital Library
- Stelios Xinogalos, Maya Satratzemi, and Christos Malliarakis. 2017. Microworlds, Games, Animations, Mobile Apps, Puzzle Editors and More: What is Important for an Introductory Programming Environment? Education and Information Technologies 22, 1 (Jan. 2017), 145--176. https://doi.org/10.1007/s10639-015--9433--1Google ScholarDigital Library
- Andreas Zeller. 1999. Yesterday, My Program Worked. Today, It Does Not. Why?. In Proceedings of the 7th European Software Engineering Conference Held Jointly with the 7th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE '99). Springer-Verlag, Berlin, Heidelberg, 253--267. http://dl.acm.org/citation.cfm?id=318773.318946Google ScholarDigital Library
- Danfeng Zhang, Andrew C. Myers, Dimitrios Vytiniotis, and Simon Peyton- Jones. 2015. Diagnosing Type Errors with Class. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '15). ACM, New York, NY, USA, 12--21. https://doi.org/10.1145/2737924.2738009Google ScholarDigital Library
- Lu Zhang. 2012. Empirical Design and Analysis of a Defect Taxonomy for Novice Programmers. Masters Thesis (by Research). The University ofWestern Australia. https://research-repository.uwa.edu.au/files/3232803/Zhang_Lu_2012.pdfGoogle Scholar
- Daniel Zingaro, Michelle Craig, Leo Porter, Brett A. Becker, Yingjun Cao, Phill Conrad, Diana Cukierman, Arto Hellas, Dastyni Loksa, and Neena Thota. 2018. Achievement Goals in CS1: Replication and Extension. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (SIGCSE '18). ACM, New York, NY, USA, 687--692. https://doi.org/10.1145/3159450.3159452Google ScholarDigital Library
Index Terms
- Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research
Recommendations
On Designing Programming Error Messages for Novices: Readability and its Constituent Factors
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing SystemsProgramming error messages play an important role in learning to program. The cycle of program input and error message response completes a loop between the programmer and the compiler/interpreter and is a fundamental interaction between human and ...
An Effective Approach to Enhancing Compiler Error Messages
SIGCSE '16: Proceedings of the 47th ACM Technical Symposium on Computing Science EducationOne of the many challenges novice programmers face from the time they write their first program is inadequate compiler error messages. These messages report details on errors the programmer has made and are the only feedback the programmer gets from the ...
Unexpected Tokens: A Review of Programming Error Messages and Design Guidelines for the Future
ITiCSE '19: Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science EducationDiagnostic messages generated by compilers and interpreters such as syntax error messages have been researched for decades. Unfortunately these messages which include error, warning, and runtime messages, present substantial difficulty and could be more ...
Comments