Shay Artzi
Julian Dolby
Marco Pistoia
ABSTRACT
Fault-localization techniques that apply statistical analyses to execution data gathered from multiple tests are quite effective when a large test suite is available. However, if no test suite is available, what is the best approach to generate one? This paper investigates the fault-localization effectiveness of test suites generated according to several test-generation techniques based on combined concrete and symbolic (concolic) execution. We evaluate these techniques by applying the Ochiai fault-localization technique to generated test suites in order to localize 35 faults in four PHP Web applications. Our results show that the test-generation techniques under consideration produce test suites with similar high fault-localization effectiveness, when given a large time budget. However, a new, "directed" test-generation technique, which aims to maximize the similarity between the path constraints of the generated tests and those of faulty executions, reaches this level of effectiveness with much smaller test suites. On average, when compared to test generation based on standard concolic execution techniques that aims to maximize code coverage, the new directed technique preserves fault-localization effectiveness while reducing test-suite size by 86.1% and test-suite generation time by 88.6%.
- R. Abreu, P. Zoeteweij, and A. J. van Gemund. On the accuracy of spectrum-based fault localization. In Proceedings of Testing: Academia and Industry Conference - Practice and Research Techniques (TAIC PART'07), pages 89--98, September 2007. Google Scholar
Digital Library
- R. Abreu, P. Zoeteweij, and A. J. C. van Gemund. An evaluation of similarity coefficients for software fault localization. In PRDC 2006, pages 39--46, 2006. Google ScholarDigital Library
- H. Agrawal, J. R. Horgan, S. London, and W. E. Wong. Fault localization using execution slices and dataflow tests. In ISSRE, pages 143--151, Toulouse, France, 1995.Google ScholarCross Ref
- S. Artzi, J. Dolby, F. Tip, and M. Pistoia, editors. Practical Fault Localization for Dynamic Web Applications, 2010. To appear.Google Scholar
- S. Artzi, A. Kieżun, J. Dolby, F. Tip, D. Dig, A. Paradkar, and M. D. Ernst. Finding bugs in dynamic web applications. In ISSTA, pages 261--272, 2008. Google ScholarDigital Library
- S. Artzi, A. Kieżun, J. Dolby, F. Tip, D. Dig, A. Paradkar, and M. D. Ernst. Finding bugs in web applications using dynamic test generation and explicit state model checking. IEEE Transactions on Software Engineering, 2010. To appear. Google ScholarDigital Library
- B. Baudry, F. Fleurey, and Y. L. Traon. Improving test suites for efficient fault localization. In L. J. Osterweil, H. D. Rombach, and M. L. Soffa, editors, Proceedings of the 28th International Conference on Software Engineering (ICSE'06), pages 82--91. ACM, 2006. Google ScholarDigital Library
- J. Burnim and K. Sen. Heuristics for scalable dynamic test generation. In ASE, pages 443--446. IEEE, 2008. Google ScholarDigital Library
- C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Engler. EXE: automatically generating inputs of death. In CCS, 2006. Google ScholarDigital Library
- T. M. Chilimbi, B. Liblit, K. K. Mehra, A. V. Nori, and K. Vaswani. Holmes: Effective statistical debugging via efficient path profiling. In Proceedings of the 31st International Conference on Software Engineering (ICSE'09), pages 34--44, Vancouver, Canada, May 2009. Google ScholarDigital Library
- H. Cleve and A. Zeller. Locating causes of program failures. In ICSE, pages 342--351, May 2005. Google ScholarDigital Library
- P. Godefroid, N. Klarlund, and K. Sen. DART: Directed automated random testing. In PLDI, 2005. Google ScholarDigital Library
- P. Godefroid, M. Y. Levin, and D. Molnar. Automated whitebox fuzz testing. In NDSS, 2008.Google Scholar
- S. Horwitz, T. Reps, and D. Binkley. Interprocedural slicing using dependence graphs. ACM Trans. Program. Lang. Syst., 12(1):26--60, 1990. Google ScholarDigital Library
- M. Hutchins, H. Foster, T. Goradia, and T. Ostrand. Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria. In ICSE, pages 191--200, 1994. Google ScholarDigital Library
- B. Jiang, Z. Zhang, T. Tse, and T. Y. Chen. How well do test case prioritization techniques support statistical fault localization. In Proceedings of the 33rd Annual IEEE International Computer Software and Applications Conference (COMPSAC 2009), July 2009. Google ScholarDigital Library
- J. A. Jones and M. J. Harrold. Empirical evaluation of the tarantula automatic fault-localization technique. In ASE, pages 273--282, 2005. Google ScholarDigital Library
- J. A. Jones, M. J. Harrold, and J. Stasko. Visualization of test information to assist fault localization. In ICSE, pages 467--477, 2002. Google ScholarDigital Library
- B. Liblit, A. Aiken, A. X. Zheng, and M. I. Jordan. Bug isolation via remote program sampling. In PLDI, pages 141--154, 2003. Google ScholarDigital Library
- B. Liblit, M. Naik, A. X. Zheng, A. Aiken, and M. I. Jordan. Scalable statistical bug isolation. In PLDI'05, pages 15--26, 2005. Google ScholarDigital Library
- C. Liu, X. Yan, L. Fei, J. Han, and S. P. Midkiff. Sober: statistical model-based bug localization. In FSE, pages 286--295, 2005. Google ScholarDigital Library
- J. Lyle and M. Weiser. Automatic bug location by program slicing. In Proceedings of the Second International Conference on Computers and Applications, pages 877--883, Beijing (Peking), China, 1987.Google Scholar
- H. Pan and E. H. Spafford. Heuristics for automatic localization of software faults. Technical Report SERC-TR-116-P, Purdue University, July 1992.Google Scholar
- X. Ren and B. G. Ryder. Heuristic ranking of java program edits for fault localization. In D. S. Rosenblum and S. G. Elbaum, editors, Proceedings of the ACM/SIGSOFT International Symposium on Software Testing and Analysis (ISSTA'07), pages 239--249. ACM, 2007. Google ScholarDigital Library
- M. Renieris and S. P. Reiss. Fault localization with nearest neighbor queries. In ASE, pages 30--39, 2003.Google ScholarCross Ref
- R. Santelices, J. A. Jones, Y. Yu, and M. J. Harrold. Lightweight fault-localization using multiple coverage types. In Proceedings of the 31st International Conference on Software Engineering (ICSE'09), pages 56--66, Vancouver, Canada, May 2009. Google ScholarDigital Library
- K. Sen, D. Marinov, and G. Agha. CUTE: A concolic unit testing engine for C. In FSE, 2005. Google ScholarDigital Library
- M. Stoerzer, B. G. Ryder, X. Ren, and F. Tip. Finding Failure-inducing Changes in Java Programs Using Change Classification. In FSE, pages 57--68, Portland, OR, USA, Nov. 7--9, 2006. Google ScholarDigital Library
- F. Tip. A survey of program slicing techniques. Journal of Programming Languages, 3(3):121--189, 1995.Google Scholar
- T. Wang and A. Roychoudhury. Automated path generation for software fault localization. In ASE '05: Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering, pages 347--351, New York, NY, USA, 2005. ACM. Google ScholarDigital Library
- G. Wassermann, D. Yu, A. Chander, D. Dhurjati, H. Inamura, and Z. Su. Dynamic test input generation for web applications. In ISSTA, pages 249--260, 2008. Google ScholarDigital Library
- C. Yilmaz, A. M. Paradkar, and C. Williams. Time will tell: fault localization using time spectra. In W. Schäfer, M. B. Dwyer, and V. Gruhn, editors, Proceedings of the 30th International Conference on Software Engineering (ICSE'08), pages 81--90. ACM, 2008. Google ScholarDigital Library
- Y. Yu, J. A. Jones, and M. J. Harrold. An empirical study of the effects of test-suite reduction on fault localization. In ICSE, pages 201--210, 2008. Google ScholarDigital Library
- A. Zeller. Yesterday, my program worked. today, it does not. why? SIGSOFT Softw. Eng. Notes, 24(6):253--267, 1999. Google ScholarDigital Library
Index Terms
- Directed test generation for effective fault localization
Recommendations
Directed Test Suite Augmentation
APSEC '09: Proceedings of the 2009 16th Asia-Pacific Software Engineering ConferenceAs software evolves, engineers use regression testing to evaluate its fitness for release. Such testing typically begins with existing test cases, and many techniques have been proposed for reusing these cost-effectively. After reusing test cases, ...
Directed test suite augmentation: an empirical investigation
Test suite augmentation techniques are used in regression testing to identify code elements in a modified program that are not adequately tested and to generate test cases to cover those elements. A defining feature of test suite augmentation techniques ...
Directed test suite augmentation: techniques and tradeoffs
FSE '10: Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineeringTest suite augmentation techniques are used in regression testing to identify code elements affected by changes and to generate test cases to cover those elements. Our preliminary work suggests that several factors influence the cost and effectiveness ...
Comments