Lu Zhang
Tao Xie
ABSTRACT
Due to the expensiveness of compiling and executing a large number of mutants, it is usually necessary to select a subset of mutants to substitute the whole set of generated mutants in mutation testing and analysis. Most existing research on mutant selection focused on operator-based mutant selection, i.e., determining a set of sufficient mutation operators and selecting mutants generated with only this set of mutation operators. Recently, researchers began to leverage statistical analysis to determine sufficient mutation operators using execution information of mutants. However, whether mutants selected with these sophisticated techniques are superior to randomly selected mutants remains an open question. In this paper, we empirically investigate this open question by comparing three representative operator-based mutant-selection techniques with two random techniques. Our empirical results show that operator-based mutant selection is not superior to random mutant selection. These results also indicate that random mutant selection can be a better choice and mutant selection on the basis of individual mutants is worthy of further investigation.
- A. T. Acree, T. A. Budd, R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Mutation analysis. Technical report, Georgia Institute of Technology, 1979.Google Scholar
- H. Agrawal, R. A. DeMillo, B. Hathaway, W. Hsu, W. Hsu, E. W. Krauser, R. J. Martin, A. P. Mathur, and E. Spafford. Design of mutant operators for the C programming language. Technical report, Purdue University, 2006.Google Scholar
- J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments? In Proc. ICSE, pages 402--411, 2005. Google ScholarDigital Library
- E. F. Barbosa, J. C. Maldonado, and A. M. R. Vincenzi. Toward the determination of sufficient mutant operators for C. STVR, 11(2):113--136, 2001.Google Scholar
- L. C. Briand, Y. Labiche, and M. M. Sówka. Automated, contract-based user testing of commercial-off-the-shelf components. In Proc. ICSE, pages 92--101, 2006. Google ScholarDigital Library
- L. C. Briand, Y. Labiche, and Y. Wang. Using simulation to empirically investigate test coverage criteria based on statechart. In Proc. ICSE, pages 86--95, 2004. Google ScholarDigital Library
- M. E. Delamaro, J. C. Maidonado, and A. P. Mathur. Interface mutation: An approach for integration testing. IEEE TSE, 27(3):228--247, 2001. Google ScholarDigital Library
- M. E. Delamaro and J. C. Maldonado. Proteum - a tool for the assessment of test adequacy for C programs. In Proc. Conference on Performability in Computing Systems, pages 79--95, 1996.Google Scholar
- R. A. DeMillo, E. W. Krauser, and A. P. Mathur. Compiler-integrated program mutation. In Proc. COMPSAC, pages 351--356, 1991.Google ScholarCross Ref
- R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on test data selection: Help for the practicing programmer. Computer, 11(4):34--41, 1978. Google ScholarDigital Library
- R. A. DeMillo and R. J. Martin. The Mothra software testing environment user's manual. Technical report, Software Engineering Research Center, 1987.Google Scholar
- R. A. DeMillo and A. J. Offutt. Constraint-based automatic test data generation. IEEE TSE, 17(9):900--910, 1991. Google ScholarDigital Library
- H. Do and G. Rothermel. On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE TSE, 32(9):733--752, 2006. Google ScholarDigital Library
- S. Elbaum, A. Malishevsky, and G. Rothermel. Prioritizing test cases for regression testing. In Proc. ISSTA, pages 102--112, 2000. Google ScholarDigital Library
- J. B. Goodenough and S. L. Gerhart. Toward a theory of test data selection. IEEE TSE, 1(2):156--173, 1975.Google ScholarDigital Library
- B. Grün, D. Schuler, and A. Zeller. The impact of equivalent mutants. In Proc. International Workshop on Mutation Analysis, pages 192--199, 2009.Google ScholarDigital Library
- R. G. Hamlet. Testing programs with the aid of a compiler. IEEE TSE, 3(4):279--290, 1977. Google ScholarDigital Library
- R. Hierons and M. Harman. Using program slicing to assist in the detection of equivalent mutants. STVR, 9(4):233--262, 1999.Google Scholar
- W. E. Howden. Weak mutation testing and completeness of test sets. IEEE TSE, 8(4):371--379, 1982. Google ScholarDigital Library
- M. Hutchins, H. Foster, T. Goradia, and T. Ostrand. Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria. In Proc. ICSE, pages 191--200, 1994. Google ScholarDigital Library
- Y. Jia and M. Harman. An analysis and survey of the development of mutation testing. Technical Report TR-09-06, CREST Centre, King's College London, London, UK, 2009.Google Scholar
- J. A. Jones and M. J. Harrold. Empirical evaluation of the Tarantula automatic fault-localization technique. In Proc. ASE, pages 273--282, 2005. Google ScholarDigital Library
- E. W. Krauser, A. P. Mathur, and V. Rego. High performance testing on SIMD machines. In Proc. Second Workshop on Software Testing, Verification, and Analysis, pages 171--177, 1988.Google ScholarCross Ref
- Z. Li, M. Harman, and R. Hierons. Search algorithms for regression test case prioritization. IEEE TSE, 33(4):225--237, 2007. Google ScholarDigital Library
- M.-H. Liu, Y.-F. Gao, J.-H. Shan, J.-H. Liu, L. Zhang, and J.-S. Sun. An approach to test data generation for killing multiple mutants. In Proc. ICSM, pages 113--122, 2006. Google ScholarDigital Library
- Y.-S. Ma, J. Offutt, and Y. R. Kwon. MuJava: An automated class mutation system. STVR, 15(2):97--133, 2005. Google ScholarDigital Library
- A. P. Mathur. Performance, effectiveness, and reliability issues in software testing. In Proc. COMPSAC, pages 604--605, 1991.Google Scholar
- J. Mayer and C. Schneckenburger. An empirical analysis and comparison of random testing techniques. In Proc. ISESE, pages 105--114, 2006. Google ScholarDigital Library
- A. J. Offutt and W. M. Craft. Using compiler optimization techniques to detect equivalent mutants. STVR, 4(3):131--154, 1994.Google Scholar
- A. J. Offutt, Z. Jin, and J. Pan. The dynamic domain reduction approach to test data generation. Software Practice and Experience, 29(2):167--193, 1999. Google ScholarDigital Library
- A. J. Offutt, A. Lee, G. Rothermel, R. H. Untch, and C. Zapf. An experimental determination of sufficient mutation operators. ACM TOSEM, 5(2):99--118, 1996. Google ScholarDigital Library
- A. J. Offutt and J. Pan. Automatically detecting equivalent mutants and infeasible paths. STVR, 7(3):165--192, 1997.Google Scholar
- A. J. Offutt, R. P. Pargas, S. V. Fichter, and P. K. Khambekar. Mutation testing of software using mimd computer. In Proc. International Conference on Parallel Processing, pages 257--266, August 1992.Google Scholar
- G. Rothermel, M. J. Harrold, J. Ostrin, and C. Hong. An empirical study of the effects of minimization on the fault detection capabilities of test suites. In Proc. ICSM, pages 34--43, 1998. Google ScholarDigital Library
- G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Test case prioritization: an empirical study. In Proc. ICSM, pages 179--188, 1999. Google ScholarDigital Library
- D. Schuler, V. Dallmeier, and A. Zeller. Efficient mutation testing by checking invariant violations. In Proc. ISSTA, pages 69--80, 2009. Google ScholarDigital Library
- A. Siami Namin, J. H. Andrews, and D. J. Murdoch. Sufficient mutation operators for measuring test effectiveness. In Proc. ICSE, pages 351--360, 2008. Google ScholarDigital Library
- J. Tuya, M. J. Suárez-Cabal, and C. de la Riva. Mutating database queries. IST, 49(4):398--417, 2007. Google ScholarDigital Library
- R. Untch, A. J. Offutt, and M. J. Harrold. Mutation analysis using program schemata. In Proc. ISSTA, pages 139--148, 1993. Google ScholarDigital Library
- W. E. Wong. On mutation and data flow. PhD thesis, Purdue University, December 1993. Google ScholarDigital Library
- W. E. Wong and A. P. Mathur. Reducing the cost of mutation testing: An empirical study. JSS, 31(3):185--196, 1995. Google ScholarDigital Library
- M. R. Woodward and K. Halewood. From weak to strong, dead or alive? An analysis of some mutation testing issues. In Proc. Second Workshop on Software Testing, Verification, and Analysis, pages 152--158, 1988.Google ScholarCross Ref
- H. Zhu. A formal analysis of the subsume relation between software test adequacy criteria. IEEE TSE, 22(4):248--255, 1996. Google ScholarDigital Library
Recommendations
Are mutants a valid substitute for real faults in software testing?
FSE 2014: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software EngineeringA good test suite is one that detects real faults. Because the set of faults in a program is usually unknowable, this definition is not useful to practitioners who are creating test suites, nor to researchers who are creating and evaluating tools that ...
Predictive mutation testing
ISSTA 2016: Proceedings of the 25th International Symposium on Software Testing and AnalysisMutation testing is a powerful methodology for evaluating test suite quality. In mutation testing, a large number of mutants are generated and executed against the test suite to check the ratio of killed mutants. Therefore, mutation testing is widely ...
Comments