Abstract
Covering arrays are increasingly being used by test engineers to derive test cases to test complex engineered systems. This approach to testing is known as combinatorial testing and has proven to be a cost-efficient way to determine test cases that are highly effective at identifying faults in the system that are due to the combination of several inputs. However, when such faults are encountered and failures occur, the test engineer is tasked with determining the inputs and associated values that triggered the failures. This exercise typically involves examining a long list of potential causes and may even require performing follow-up tests to reduce the number of potential causes. This paper addresses this issue by considering the prior knowledge of the system under test that test engineers often have. We show how this knowledge can be used to evaluate and improve the effectiveness of a suite of test cases before any test cases are executed. Finally, we address the case where failures occur and show how this prior knowledge can aid in determining the inputs, and associated values, that triggered the failures. In addition, throughout the paper, we compare and contrast the use of covering arrays for testing complex engineered systems to the use of factorial experiments in traditional experimental design settings.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Box G, Tyssedal J (1996) Projective properties of certain orthogonal arrays. Biometrika 83(4):950–955
Bryce RC, Colbourn CJ (2006) Prioritized interaction testing for pair-wise coverage with seeding and constraints. Inf Softw Technol 48(10):960–970. https://doi.org/10.1016/j.infsof.2006.03.004 (Advances in model-based testing)
Chen T, He T, Benesty M, Khotilovich V, Tang Y (2015) Xgboost: extreme gradient boosting. R package version 0.4-2, pp 1–4
Colbourn CJ, Martirosyan SS, Mullen GL, Shasha D, Sherwood GB, Yucas JL (2006) Products of mixed covering arrays of strength two. J Comb Des 14(2):124–138
Colbourn CJ, McClary DW (2008) Locating and detecting arrays for interaction faults. J Comb Optim 15(1):17–48. https://doi.org/10.1007/s10878-007-9082-4
Colbourn CJ, Syrotiuk VR (2016) Coverage, location, detection, and measurement. In: IEEE ninth international conference on software testing, verification and validation workshops (ICSTW), pp 19–25. https://doi.org/10.1109/ICSTW.2016.38
Dalal SR, Mallows CL (1998) Factor-covering designs for testing software. Technometrics 40(3):234–243. https://doi.org/10.1080/00401706.1998.10485524
Elbaum S, Rothermel G, Kanduri S, Malishevsky AG (2004) Selecting a cost-effective test case prioritization technique. Softw Qual J 12(3):185–210. https://doi.org/10.1023/B:SQJO.0000034708.84524.22
Ghandehari LS, Lei Y, Kung D, Kacker R, Kuhn R (2013) Fault localization based on failure-inducing combinations. In: IEEE 24th international symposium on software reliability engineering (ISSRE), pp 168–177. https://doi.org/10.1109/ISSRE.2013.6698916
Gonzalez-Hernandez L, Torres-Jiménez J, Rangel-Valdez N (2011) An exact approach to maximize the number of wild cards in a covering array. In: Batyrshin I, Sidorov G (eds) Advances in artificial intelligence. MICAI 2011. Lecture notes in computer science, vol 7094. Springer, Berlin, Heidelberg
Katona GO (1973) Two applications (for search theory and truth functions) of sperner type theorems. Period Math Hung 3(1–2):19–26
Kim Y, Jang DH, Anderson-Cook CM (2017) Selecting the best wild card entries in a covering array. Qual Reliab Eng Int 33(7):1615–1627
Kleitman DJ, Spencer J (1973) Families of k-independent sets. Discret Math 6(3):255–262
Kuhn DR, Okum V (2006) Pseudo-exhaustive testing for software. In: 30th Annual IEEE/NASA software engineering workshop. IEEE, pp 153–158
Kuhn DR, Reilly MJ (2002) An investigation of the applicability of design of experiments to software testing. In: Proceedings 27th annual NASA Goddard/IEEE software engineering workshop. IEEE, pp 91–95
Kuhn DR, Wallace DR, Gallo AM (2004) Software fault interactions and implications for software testing. IEEE Trans Softw Eng 30(6):418–421. https://doi.org/10.1109/TSE.2004.24
Lekivetz R, Morgan J (2018) Evaluation and construction of covering arrays utilizing prior information. In: IEEE international symposium on software reliability engineering workshops (ISSREW). IEEE, pp 132–133
Lekivetz R, Morgan J (2018) Fault localization: analyzing covering arrays given prior information. In: IEEE international conference on software quality, reliability and security companion (QRS-C). IEEE, pp 116–121
Morgan J (2018) Combinatorial testing: an approach to systems and software testing based on covering arrays. In: Kenett RS, Ruggeri F, Faltin FW (eds) Analytic methods in systems and software testing. Wiley, pp 131–158
Qu X, Cohen MB, Rothermel G (2008) Configuration-aware regression testing: an empirical study of sampling and prioritization. In: Proceedings of the 2008 international symposium on software testing and analysis. ACM, pp 75–86
Rényi A (2007) Foundations of probability. Courier Corporation, North Chelmsford
Wu CJ, Hamada MS (2011) Experiments: planning, analysis, and optimization, vol 552. Wiley, Hoboken
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Part of special issue guest edited by Pritam Ranjan and Min Yang—Algorithms, Analysis and Advanced Methodologies in the Design of Experiments.
Rights and permissions
About this article
Cite this article
Lekivetz, R., Morgan, J. Covering Arrays: Using Prior Information for Construction, Evaluation and to Facilitate Fault Localization. J Stat Theory Pract 14, 7 (2020). https://doi.org/10.1007/s42519-019-0075-2
Published:
DOI: https://doi.org/10.1007/s42519-019-0075-2