Abstract
Bug finding tools can find defects in software source code using an automated static analysis. This automation may be able to reduce the time spent for other testing and review activities. For this we need to have a clear understanding of how the defects found by bug finding tools relate to the defects found by other techniques. This paper describes a case study using several projects mainly from an industrial environment that were used to analyse the interrelationships. The main finding is that the bug finding tools predominantly find different defects than testing but a subset of defects found by reviews. However, the types that can be detected are analysed more thoroughly. Therefore, a combination is most advisable if the high number of false positives of the tools can be tolerated.
This research was supported in part by the Deutsche Forschungsgemeinschaft (DFG) within the project InTime.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Ball, T., Rajamani, S.K.: The SLAM Project: Debugging System Software via Static Analysis. In: Proc. 29th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (2002)
Beizer, B.: Software Testing Techniques, 2nd edn. Thomson Learning (1990)
Bush, W.R., Pincus, J.D., Sielaff, D.J.: A static analyzer for finding dynamic programming errors. Softw. Pract. Exper. 30, 775–802 (2000)
Chillarege, R.: Orthogonal Defect Classification. In: Lyu, M.R. (ed.) Handbook of Software Reliability Engineering, ch. 9. IEEE Computer Society Press and McGraw-Hill (1996)
Csallner, C., Smaragdakis, Y.: CnC: Combining Static Checking and Testing. In: Inverardi, P., Jazayeri, M. (eds.) ICSE 2005. LNCS, vol. 4309. Springer, Heidelberg (2006) (to appear)
Engler, D., Musuvathi, M.: Static Analysis versus Model Checking for Bug Finding. In: Steffen, B., Levi, G. (eds.) VMCAI 2004. LNCS, vol. 2937, pp. 191–210. Springer, Heidelberg (2004)
Flanagan, C., Leino, K.R.M., Lillibridge, M., Nelson, G., Saxe, J.B., Stata, R.: Extended Static Checking for Java. In: Proc. 2002 ACM SIGPLAN Conference on Programming Language Design and Implementation (2002)
Hovemeyer, D., Pugh, W.: Finding Bugs is Easy. SIGPLAN Notices 39(12) (2004) (to appear)
IEEE. IEEE Standard Classification for Software Anomalies (1993) (IEEE Std 1044-1993)
Johnson, R., Wagner, D.: Finding User/Kernel Pointer Bugs With Type Inference. In: Proc. 13th USENIX Security Symposium (2004)
Jones, C.: Applied Software Measurement: Assuring Productivity and Quality. McGraw-Hill, New York (1991)
Koller, C.: Vergleich verschiedener Methoden zur analytischen Qualitätssicherung. Diploma Thesis, Technische Universität München (2004) (in German)
Myers, G.J.: The Art of Software Testing. John Wiley & Sons, Chichester (1979)
Palsberg, J.: Type-Based Analysis and Applications. In: Proc. 2001 ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering (PASTE 2001), pp. 20–27. ACM Press, New York (2001)
PMD (February 2005), http://pmd.sourceforge.net
Pretschner, A., Prenninger, W., Wagner, S., Kühnel, C., Baumgartner, M., Sostawa, B., Zölch, R., Stauner, T.: One Evaluation of Model-Based Testing and its Automation. In: Proc. 27th International Conference on Software Engineering (ICSE 2005) (2005) (to appear)
QJ Pro (February 2005), http://qjpro.sourceforge.net
Rutar, N., Almazan, C.B., Foster, J.S.: A Comparison of Bug Finding Tools for Java. In: Proc. 15th IEEE International Symposium on Software Reliability Engineering (ISSRE 2004), pp. 245–256 (2004)
S. Wagner. Efficiency Analysis of Defect-Detection Techniques. Technical Report TUMI-0413, Institut für Informatik, Technische Universität München, 2004.
Wagner, S.: Reliability Efficiency of Defect-Detection Techniques: A Field Study. In: Suppl. Proc. 15th IEEE International Symposium on Software Reliability Engineering (ISSRE 2004), pp. 294–301 (2004)
Wagner, S.: Towards Software Quality Economics for Defect-Detection Techniques. In: Proc. 29th Annual IEEE/NASA Software Engineering Workshop (2005) (to appear)
Zitser, M., Lippmann, R., Leek, T.: Testing Static Analysis Tools using Exploitable Buffer Overflows from Open Source Code. In: Proc. 12th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT 2004/FSE-12), pp. 97–106. ACM Press, New York (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 IFIP International Federation for Information Processing
About this paper
Cite this paper
Wagner, S., Jürjens, J., Koller, C., Trischberger, P. (2005). Comparing Bug Finding Tools with Reviews and Tests. In: Khendek, F., Dssouli, R. (eds) Testing of Communicating Systems. TestCom 2005. Lecture Notes in Computer Science, vol 3502. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11430230_4
Download citation
DOI: https://doi.org/10.1007/11430230_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-26054-7
Online ISBN: 978-3-540-32076-0
eBook Packages: Computer ScienceComputer Science (R0)