Jackson A. Prado Lima
ABSTRACT
The use of Higher-Order Mutants (HOMs) presents some advantages concerning the traditional use of First-Order Mutants (FOMs). HOMs can better simulate real and subtle faults, reduce the number of generated mutants and test cases, and so on. However, the HOM space is potentially huge, and an efficient strategy to generate the best HOMs is fundamental. In the literature different strategies were proposed and evaluated, mainly to generate Second-Order Mutants (SOMs), but none has been proved to perform better in different situations. Due to this, the selection of the best strategy is an important task. Most times a lot of experiments need to be conducted. To help the tester in this task and to allow the use of HOMs in practice, this paper proposes a hyper-heuristic approach. Such approach is based on NSGA-II and uses the selection method Choice Function to automatically choose among different Low-Level Heuristics (LLHs), which, in this case, are search-operators related to existing SOM generation strategies. The performance of each LLH is related to some objectives such as the number of SOMs generated, the capacity to capture subtler faults and replace the constituent FOMs. In comparison with existing strategies, our approach obtained better results considering the used objectives, and statistically equivalent results considering mutation score with respect to the FOMs.
- A. Arcuri and L. Briand. A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing, Verification and Reliability, 24(3):219--250, 2014. Google Scholar
Digital Library
- M. P. Basgalupp, R. C. Barros, T. S. da Silva, and A. C. P. L. F. de Carvalho. Software effort prediction: A hyper-heuristic decision-tree based approach. In 28th Annual Symposium on Applied Computing, SAC, pages 1109--1116, Mar. 2013. Google ScholarDigital Library
- E. K. Burke, M. Gendreau, M. Hyde, G. Kendall, G. Ochoa, E. Özcan, and R. Qu. Hyper-heuristics: a survey of the state of the art. Journal of the Operational Research Society, 64(12):1695, Dec. 2013. Google ScholarCross Ref
- E. K. Burke, M. Hyde, G. Kendall, G. Ochoa, E. Özcan, and J. R. Woodward. A classification of hyper-heuristic approaches. In Handbook of Metaheuristics, pages 449--468. Springer, Jan. 2010. Google ScholarCross Ref
- P. Cowling, G. Kendall, and E. Soubeiga. A hyperheuristic approach to scheduling a sales summit. In E. Burke and W. Erben, editors, 3rd Practice and Theory of Automated Timetabling (PATAT'00), volume 2079 of Lecture Notes in Computer Science, pages 176--190. Springer, 2001. Google ScholarCross Ref
- K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. on Evolutionary Computation, 6(2):182--197, 2002. Google ScholarDigital Library
- J. J. Durillo and A. J. Nebro. jMetal: A Java framework for multi-objective optimization. Advances in Engineering Software, 42(10):760--771, 2011. Google ScholarDigital Library
- E. L. Féderle, G. Guizzo, T. E. Colanzi, S. R. Vergilio, and E. Spinosa. Seleção de produto baseada em algoritmos multiobjetivos para o teste de mutação de variabilidades. In IV Workshop de Engenharia de Software Baseada em Busca, 2013.Google Scholar
- T. N. Ferreira, J. N. Kuk, A. T. R. Pozo, and S. R. Vergilio. Product selection based on upper confidence bound MOEA/D-DRA for testing software product lines. In Congress on Evolutionary Computation, CEC, pages 4135--4142, July 2016.Google Scholar
- T. N. Ferreira, J. A. Prado Lima, A. Strickler, J. N. Kuk, S. R. Vergilio, and A. Pozo. Hyper-heuristic Based Product Selection for Software Product Line Testing. IEEE Computational Intelligence Magazine, 12(2):34--45, May 2017. Google ScholarDigital Library
- G. Guizzo, G. M. Fritsche, S. R. Vergilio, and A. T. R. Pozo. A hyper-heuristic for the multi-objective integration and test order problem. In Genetic and Evolutionary Computation Conference, GECCO, pages 1343--1350, 2015.Google Scholar
- M. Harman, E. Burke, J. Clark, and X. Yao. Dynamic adaptive search based software engineering. In International Symposium on Empirical Software Engineering and Measurement, ESEM, pages 1--8, Sept. 2012. Google ScholarDigital Library
- M. Harman, Y. Jia, and W. B. Langdon. A manifesto for higher order mutation testing. In Third International Conference on Software Testing, Verification, and Validation Workshops (ICSTW), pages 80--89, 2010. Google ScholarDigital Library
- M. Harman, Y. Jia, P. R. Mateo, and M. Polo. Angels and monsters: An empirical investigation of potential test effectiveness and efficiency improvement from strongly subsuming higher order mutation. In International conference on Automated Software Engineering, pages 397--408, 2014. Google ScholarDigital Library
- M. Harman, Y. Jia, and Y. Zhang. Achievements, open problems and challenges for search based software testing. IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pages 1--12, 2015. Google ScholarCross Ref
- M. Harman, S. A. Mansouri, and Y. Zhang. Search-based software engineering: Trends, techniques and applications. ACM Computing Surveys (CSUR), 45(1):11, 2012. Google ScholarDigital Library
- Y. Jia, M. B. Cohen, M. Harman, and J. Petke. Learning combinatorial interaction test generation strategies using hyperheuristic search. In 37th International Conference on Software Engineering, volume 1 of ICSE'15, pages 540--550, May 2015. Google ScholarCross Ref
- Y. Jia and M. Harman. Higher order mutation testing. Information and Software Technology, 51(10):1379--1393, 2009. Google ScholarDigital Library
- Y. Jia and M. Harman. An analysis and survey of the development of mutation testing. IEEE Trans. on Software Engineering, 37(5):649--678, 2011. Google ScholarDigital Library
- G. Kendall, E. Soubeiga, and P. Cowling. Choice function and random hyperheuristics. In 4th Asia-Pacific Conference on Simulated Evolution And Learning (SEAL'02), pages 667--671. Springer, 2002.Google Scholar
- M. Kintis, M. Papadakis, and N. Malevris. Isolating first order equivalent mutants via second order mutation. In IEEE Fifth International Conference on Software Testing, Verification and Validation (ICST), 2012, pages 701--710. IEEE, 2012. Google ScholarDigital Library
- A. C. Kumari, K. Srinivas, and M. P. Gupta. Software module clustering using a hyper-heuristic based multi-objective genetic algorithm. In 3rd International Advance Computing Conference, IACC'13, pages 813--818, 2013. Google ScholarCross Ref
- W. Langdon, M. Harman, and Y. Jia. Multi objective higher order mutation testing with GP. In Genetic and Evolutionary Computation Conference (GECCO), pages 1945--1946, 2009. Google ScholarDigital Library
- W. B. Langdon, M. Harman, and Y. Jia. Efficient multi-objective higher order mutation testing with genetic programming. Journal of systems and Software, 83(12):2416--2430, 2010. Google ScholarDigital Library
- K. Li, A. Fialho, S. Kwong, and Q. Zhang. Adaptive operator selection with bandits for a multiobjective evolutionary algorithm based on decomposition. IEEE Trans. on Evolutionary Computation, 18(1):114--130, Feb. 2014. Google ScholarCross Ref
- M. López-Ibánez, J. Dubois-Lacoste, L. P. Cáceres, M. Birattari, and T. Stützle. The irace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives, 3:43--58, 2016. Google ScholarCross Ref
- Y. Ma, J. Offutt, and Y. Kwon. MuJava: an automated class mutation system. Software Testing, Verification and Reliability, 15(2):97--133, 2005. Google ScholarDigital Library
- M. Maashi, E. Özcan, and G. Kendall. A multi-objective hyper-heuristic based on choice function. Expert Systems with Applications, 41(9):4475--4493, 2014. Google ScholarDigital Library
- L. Madeyski, W. Orzeszyna, R. Torkar, and M. Józala. Overcoming the equivalent mutant problem: A systematic literature review and a comparative experiment of second order mutation. IEEE Trans. on Software Engineering, 40(1):23--42, 2014. Google ScholarDigital Library
- T. Mariani, G. Guizzo, S. R. Vergilio, and A. T. R. Pozo. Grammatical Evolution for the Multi-Objective Integration and Test Order Problem. In Genetic and Evolutionary Computation Conference, GECCO, pages 1069--1076, 2016. Google ScholarDigital Library
- R. P. Mateo, P. U. Macario, F. Aleman, and J. Luis. Validating second-order mutation at system level. IEEE Trans. on Software Engineering, 39(4):570--587, 2013. Google ScholarDigital Library
- E. Omar, S. Ghosh, and D. Whitley. Constructing subtle higher order mutants for Java and AspectJ programs. In IEEE 24th International Symposium on Software Reliability Engineering (ISSRE), pages 340--349, 2013. Google ScholarCross Ref
- E. Omar, S. Ghosh, and D. Whitley. Comparing search techniques for finding subtle higher order mutants. In Genetic and evolutionary computation Conference, GECCO, pages 1271--1278, 2014. Google ScholarDigital Library
- M. Papadakis and N. Malevris. An empirical evaluation of the first and second order mutation testing strategies. In third International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 2010, pages 90--99, 2010. Google ScholarDigital Library
- V. Pareto. Manuel D'économie Politique, Paris. Ams Press, 1927.Google Scholar
- M. Polo, M. Piattini, and I. García-Rodríguez. Decreasing the cost of mutation testing with second-order mutants. Software Testing, Verification and Reliability, 19(2):111--131, 2009. Google ScholarDigital Library
- A. Strickler, J. A. Prado Lima, S. R. Vergilio, and A. T. R. Pozo. Deriving products for variability test of feature models with a hyper-heuristic approach. Applied Soft Computing, 49:1232--1242, Dec. 2016. Google ScholarDigital Library
- E. Zitzler and L. Thiele. Multiobjective optimization using evolutionary algorithms --- a comparative case study. In Parallel Problem Solving from Nature, PPSN V, pages 292--301. Springer, Sept. 1998. Google ScholarCross Ref
Index Terms
- A Multi-objective optimization approach for selection of second order mutant generation strategies
Recommendations
A path-aware approach to mutant reduction in mutation testing
A path-aware approach to mutant reduction which explores mutant reduction from the perspective of the path depth in the program under test.Four mutant reduction strategies which are developed with different priorities among the four heuristic rules for ...
Multi-objective optimization using teaching-learning-based optimization algorithm
Two major goals in multi-objective optimization are to obtain a set of nondominated solutions as closely as possible to the true Pareto front (PF) and maintain a well-distributed solution set along the Pareto front. In this paper, we propose a teaching-...
Adaptive weights generation for decomposition-based multi-objective optimization using Gaussian process regression
GECCO '17: Proceedings of the Genetic and Evolutionary Computation ConferenceBy transforming a multi-objective optimization problem into a number of single-objective optimization problems and optimizing them simultaneously, decomposition-based evolutionary multi-objective optimization algorithms have attracted much attention in ...
Comments