Shin Yoo
Mark Harman
Skip Abstract Section
Abstract
Test case prioritization techniques seek to maximize early fault detection. Fault localization seeks to use test cases already executed to help find the fault location. There is a natural interplay between the two techniques; once a fault is detected, we often switch focus to fault fixing, for which localization may be a first step. In this article we introduce the Fault Localization Prioritization (FLP) problem, which combines prioritization and localization. We evaluate three techniques: a novel FLP technique based on information theory, FLINT (Fault Localization using INformation Theory), that we introduce in this article, a standard Test Case Prioritization (TCP) technique, and a “test similarity technique” used in previous work. Our evaluation uses five different releases of four software systems. The results indicate that FLP and TCP can statistically significantly reduce fault localization costs for 73% and 76% of cases, respectively, and that FLINT significantly outperforms similarity-based localization techniques in 52% of the cases considered in the study.
- Abreu, R., Zoeteweij, P., and Van Gemund, A. J. C. 2007. On the accuracy of spectrum-based fault localization. In Proceedings of the Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION. IEEE Computer Society, 89--98. Google Scholar
Digital Library
- Allen, E. B. and Khoshgoftaar, T. M. 1999. Measuring coupling and cohesion: An information-theory approach. In Proceedings of the 6th International Symposium on Software Metrics (METRICS'99). IEEE. Google ScholarDigital Library
- Arbuckle, T. 2011. Studying software evolution using artefacts' shared information content. Sci. Comput. Program. 76, 12, 1078--1097. Google ScholarDigital Library
- Artzi, S., Dolby, J., Tip, F., and Pistoia, M. 2010. Directed test generation for effective fault localization. In Proceedings of the 19th International Symposium on Software Testing and Analysis (ISSTA'10). ACM Press, New York, 49--60. Google ScholarDigital Library
- Baah, G. K., Podgurski, A., and Harrold, M. J. 2010. Causal inference for statistical fault localization. In Proceedings of the 19th International Symposium on Software Testing and Analysis (ISSTA'10). ACM Press, New York, 73--84. Google ScholarDigital Library
- Boreale, M., Clark, D., and Gorla, D. 2010. A semiring-based trace semantics for processes with applications to information leakage analysis. In Theoretical Computer Science, C. Calude and V. Sassone, Eds., IFIP Advances in Information and Communication Technology, vol. 323, Springer, 340--354.Google ScholarCross Ref
- Clark, D., Hunt, S., and Malacaria, P. 2002. Quantitative analysis of the leakage of confidential data. Electron. Not. Theor. Comput. Sci. 59, 3, 1--14.Google Scholar
- Clark, D., Hunt, S., and Malacaria, P. 2007. A static analysis for quantifying information flow in a simple imperative language. J. Comput. Secur. 15, 3, 321--372. Google ScholarDigital Library
- Clarkson, M. R., Myers, A. C., and Schneider, F. B. 2009. Quantifying information flow with beliefs. J. Comput. Secur. 17, 5, 655--701. Google ScholarDigital Library
- Cleve, H. and Zeller, A. 2005. Locating causes of program failures. In Proceedings of the 27th International Conference on Software Engineering (ICSE'05). ACM Press, New York, 342--351. Google ScholarDigital Library
- Cover, T. M. and Thomas, J. A. 1991. Elements of Information Theory. Wiley Interscience. Google ScholarDigital Library
- Denning, D. E. R. 1982. Cryptography and Data Security. Addison-Wesley.Google Scholar
- Do, H., Elbaum, S. G., and Rothermel, G. 2005. Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empirical Softw. Engin. 10, 4, 405--435. Google ScholarDigital Library
- Elbaum, S. G., Malishevsky, A. G., and Rothermel, G. 2000. Prioritizing test cases for regression testing. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA'00). ACM Press, New York, 102--112. Google ScholarDigital Library
- Gonzalez-Sanchez, A., Piel, E., Abreu, R., Gross, H.-G., and Van Gemund, A. J. C. 2011. Prioritizing tests for software fault diagnosis. Softw. Pract. Exper. 41, 10, 1105--1129. Google ScholarDigital Library
- Hao, D., Zhang, L., Pan, Y., Mei, H., and Sun, J. 2008. On similarity-awareness in testing-based fault localization. Autom. Softw. Engin. 15, 207--249. Google ScholarDigital Library
- Jiang, B., Zhang, Z., Tse, T. H., and Chen, T. Y. 2009. How well do test case prioritization techniques support statistical fault localization. In Proceedings of the 33rd Annual International Computer Software and Applications Conference (COMPSAC'09). IEEE Computer Society Press, 99--106. Google ScholarDigital Library
- Jones, J. A. and Harrold, M. J. 2005. Empirical evaluation of the tarantula automatic fault-localization technique. In Proceedings of the 20th International Conference on Automated Software Engineering (ASE'05). ACM Press, New York, 273--282. Google ScholarDigital Library
- Jones, J. A., Harrold, M. J., and Stasko, J. 2002. Visualization of test information to assist fault localization. In Proceedings of the 24th International Conference on Software Engineering. ACM Press, New York, 467--477. Google ScholarDigital Library
- Jones, J. A., Harrold, M. J., and Stasko, J. T. 2001. Visualization for fault localization. In Proceedings of the ICSE Workshop on Software Visualization. 71--75.Google Scholar
- Kim, J.-M. and Porter, A. 2002. A history-based test prioritization technique for regression testing in resource constrained environments. In Proceedings of the 24th International Conference on Software Engineering. ACM Press, New York, 119--129. Google ScholarDigital Library
- Leon, D. and Podgurski, A. 2003. A comparison of coverage-based and distribution-based techniques for filtering and prioritizing test cases. In Proceedings of the IEEE International Symposium on Software Reliability Engineering (ISSRE'03). IEEE Computer Society Press, 442--456. Google ScholarDigital Library
- Li, Z., Harman, M., and Hierons, R. M. 2007. Search algorithms for regression test case prioritization. IEEE Trans. Softw. Engin. 33, 4, 225--237. Google ScholarDigital Library
- Liblit, B., Naik, M., Zheng, A. X., Aiken, A., and Jordan, M. I. 2005. Scalable statistical bug isolation. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI'05). ACM Press, New York, 15--26. Google ScholarDigital Library
- Renieres, M. and Reiss, S. 2003. Fault localization with nearest neighbor queries. In Proceedings of the 18th International Conference on Automated Software Engineering. 30--39.Google Scholar
- Rothermel, G., Untch, R. J., and Chu, C. 2001. Prioritizing test cases for regression testing. IEEE Trans. Softw. Engin. 27, 10, 929--948. Google ScholarDigital Library
- Shannon, C. E. 1948. A mathematical theory of communication. Bell Syst. Tech. J. 27, 379--423 and 623--656.Google ScholarCross Ref
- Tonella, P., Avesani, P., and Susi, A. 2006. Using the case-based ranking methodology for test case prioritization. In Proceedings of the 22nd International Conference on Software Maintenance (ICSM'06). IEEE Computer Society, 123--133. Google ScholarDigital Library
- Yoo, S. and Harman, M. 2012. Regression testing minimisation, selection and prioritisation: A survey. Softw. Testing Verif. Reliabil. 22, 2, 67--120. Google ScholarDigital Library
- Yoo, S., Harman, M., Tonella, P., and Susi, A. 2009. Clustering test cases to achieve effective and scalable prioritization incorporating expert knowledge. In Proceedings of International Symposium on Software Testing and Analysis (ISSTA'09). ACM Press, New York, 201--211. Google ScholarDigital Library
- Yu, Y., Jones, J. A., and Harrold, M. J. 2008. An empirical study of the effects of test-suite reduction on fault localization. In Proceedings of the International Conference on Software Engineering (ICSE'08). ACM Press, New York, 201--210. Google ScholarDigital Library
Index Terms
- Fault localization prioritization: Comparing information-theoretic and coverage-based approaches
Recommendations
An effective fault aware test case prioritization by incorporating a fault localization technique
ESEM '10: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and MeasurementPrior coverage-based test case prioritization techniques aim to increase fault detection rates by ordering the test cases according to some coverage criteria. However, in practice, since detected faults are typically removed, test cases that already ...
Test case selection using multi-criteria optimization for effective fault localization
As spectra-based fault localization techniques report suspicious statements by analyzing the coverage of test cases, the effectiveness of the results is highly dependent on the composition of test suites. This paper proposes an approach for selecting a ...
How Well Do Test Case Prioritization Techniques Support Statistical Fault Localization
COMPSAC '09: Proceedings of the 2009 33rd Annual IEEE International Computer Software and Applications Conference - Volume 01In continuous integration, a tight integration of test case prioritization techniques and fault-localization techniques may both expose failures faster and locate faults more effectively. Statistical fault-localization techniques use the execution ...
Comments