ABSTRACT
Research on students learning to program has produced studies on both compile-time errors (syntax errors) and run-time errors (exceptions). Both of these types of errors are natural targets, since detection is built into the programming language. In this paper, we present an empirical investigation of static analysis errors present in syntactically correct code. Static analysis errors can be revealed by tools that examine a program's source code, but this error detection is typically not built into common programming languages and instead requires separate tools. Static analysis can be used to check formatting or commenting expectations, but it also can be used to identify problematic code or to find some kinds of conceptual or logic errors. We study nearly 10 million static analysis errors found in over 500 thousand program submissions made by students over a five-semester period. The study includes data from four separate courses, including a non-majors introductory course as well as the CS1/CS2/CS3 sequence for CS majors. We examine the differences between the error rates of CS major and non-major beginners, and also examine how these patterns change over time as students progress through the CS major course sequence. Our investigation shows that while formatting and Javadoc issues are the most common, static checks that identify coding flaws that are likely to be errors are strongly correlated with producing correct programs, even when students eventually fix the problems. With experience, students produce fewer errors, but the errors that are most frequent are consistent between both computer science majors and non-majors, and across experience levels. These results can highlight student struggles or misunderstandings that have escaped past analyses focused on syntax or run-time errors.
- M. Ahmadzadeh, D. Elliman, and C. Higgins. An analysis of patterns of debugging among novice computer science students. In Proceedings of the 10th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, ITiCSE '05, pages 84{88, New York, NY, USA, 2005. ACM. Google ScholarDigital Library
- A. Altadmri and N. C. C. Brown. 37 Million Compilations: Investigating Novice Programming Mistakes in Large-Scale Student Data. SIGCSE '15 Proceedings of the 46th ACM Technical Symposium on Computer Science Education. Pages 522--527Google Scholar
- D. McCall and M. Kolling. Meaningful categorisation of novice programmer errors. In Frontiers In Education Conference, pages 2589--2596, 2014. Google ScholarCross Ref
- P. Denny, A. Luxton-Reilly, and E. Tempero. All syntax errors are not equal. In Proceedings of the 17th ACM Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE '12, pages 75--80, New York, NY, USA, 2012. ACM. Google ScholarDigital Library
- M. C. Jadud and B. Dorn. Aggregate Compilation Behavior: Findings and Implications from 27,698 Users. ICER '15 Proceedings of the eleventh annual International Conference on International Computing Education Research. Pages 131--139Google Scholar
- E. S. Tabanao, M. M. T. Rodrigo, and M. C. Jadud. Predicting at-risk novice Java programmers through the analysis of online protocols. In Proceedings of the Seventh International Workshop on Computing Education Research, ICER '11, pages 85--92, NewYork, NY, USA, 2011. ACM. Google ScholarDigital Library
- M. M. T. Rodrigo, T. C. S. Andallaza, F. E. V. G. Castro, M. L. V. Armenta, T. T. Dy, and M. C. Jadud. An analysis of java programming behaviors, affect, perceptions, and syntax errors among low-achieving, average, and high-achieving novice programmers. Journal of Educational Computing Research, 49(3):293--325, 2013. Google ScholarCross Ref
- S. H. Edwards and M. A. Perez-Quinones. Web-cat: automatically grading programming assignments. In ACM SIGCSE Bulletin, volume 40, pages 328--328. ACM, 2008. Google ScholarDigital Library
- N. Truong, P. Roe and P. Bancroft. Static analysis of students' Java programs. ACE '04 Proceedings of the Sixth Australasian Conference on Computing Education - Volume 30 Pages 317--325.Google Scholar
- C. Artho and A. Biere. Applying Static Analysis to Large-Scale, Multi-Threaded Java Programs. ASWEC '01 Proceedings of the 13th Australian Conference on Software Engineering. Page 68Google Scholar
- Fenwick, J.B.Jr., Norris, C., Barry, F.E., Rountree,J., Spicer,C.J. and Cheek, S.D. 2009. Another look at the behaviors of novice programmers. SIGCSE Bull. 41, 1 (March 2009), 296--300. DOI=10.1145/1539024.1508973Google Scholar
- A. Stefik and S. Siebert. An empirical investigation into programming language syntax. Trans. Comput. Educ., 13(4):19:1{19:40, Nov. 2013.Google ScholarDigital Library
- Mengel, S. and Yerramilli, V. (1999): A Case Study Of The Static Analysis Of the Quality Of Novice Student Programs. Proc. Thirtieth SIGCSE technical symposium on Computer science education, New Orleans, Louisiana, United States, 13:78--82. Google ScholarDigital Library
- N. Ayewah, W. Pugh, J. David Morgenthaler, J. Penix and Y. Zhou. Evaluating Static Analysis Defect Warnings On Production Software. PASTE '07 Proceedings of the 7th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, San Diego, California, United States. Pages 1--8. June 2007 Google ScholarDigital Library
- N. Nagappan and T. Ball. Static Analysis Tools as Early Indicators of Pre-Release Defect Density. ICSE '05 Proceedings of the 27th international conference on Software engineering. Pages 580--586. St. Louis, MO, USA - May 15 - 21, 2005.Google Scholar
Index Terms
- Investigating Static Analysis Errors in Student Java Programs
Recommendations
Finding bugs in java native interface programs
ISSTA '08: Proceedings of the 2008 international symposium on Software testing and analysisIn this paper, we describe static analysis techniques for finding bugs in programs using the Java Native Interface (JNI). The JNI is both tedious and error-prone because there are many JNI-specific mistakes that are not caught by a native compiler. This ...
Improving your software using static analysis to find bugs
OOPSLA '06: Companion to the 21st ACM SIGPLAN symposium on Object-oriented programming systems, languages, and applicationsFindBugs looks for bugs in Java programs. It is based on the concept of bug patterns. A bug pattern is a code idiom that is often an error. Bug patterns arise for a variety of reasons, such as difficult language features, misunderstood API semantics, ...
Static Detection of Bugs Caused by Incorrect Exception Handling in Java Programs
QSIC '11: Proceedings of the 2011 11th International Conference on Quality SoftwareException handling is a vital but often poorly tested part of a program. Static analysis can spot bugs on exceptional paths without actually making the exceptions happen. However, the traditional methods only focus on null dereferences on exceptional ...
Comments