Abstract
Highly configurable systems (HCSs), such as software product lines, have complex configuration spaces. Combinatorial Testing and Random Testing are the main approaches to testing of HCSs. In this paper, we empirically compare their strengths with respect to scalability and diversity of sampled configurations (i.e., tests). We choose Icpl and QuickSampler to respectively represent Combinatorial Testing and Random Testing. Experiments are conducted to evaluate the t-way coverage criterion of generated test suites for HCS benchmarks.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Arcuri, A., Briand, L.C.: Formal analysis of the probability of interaction fault detection using random testing. IEEE Trans. Softw. Eng. 38(5), 1088–1099 (2012)
Chakraborty, S., Fremont, D.J., Meel, K.S., Seshia, S.A., Vardi, M.Y.: On parallel scalable uniform sat witness generation. In: Baier, C., Tinelli, C. (eds.) TACAS 2015. LNCS, vol. 9035, pp. 304–319. Springer, Heidelberg (2015). https://doi.org/10.1007/978-3-662-46681-0_25
Chen, T.Y., Leung, H., Mak, I.K.: Adaptive random testing. In: Maher, M.J. (ed.) ASIAN 2004. LNCS, vol. 3321, pp. 320–329. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30502-6_23
Dutra, R., Laeufer, K., Bachrach, J., Sen, K.: Efficient sampling of SAT solutions for testing. In: Proceedings of ICSE 2018, pp. 549–559 (2018)
Ermon, S., Gomes, C., Selman, B.: Uniform solution sampling using a constraint solver as an oracle. In: Proceedings of UAI 2012, pp. 255–264 (2012)
Gargantini, A., Radavelli, M.: Migrating combinatorial interaction test modeling and generation to the web. In: 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 308–317, April 2018. https://doi.org/10.1109/ICSTW.2018.00066
Garvin, B.J., Cohen, M.B., Dwyer, M.B.: Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empirical Softw. Eng. 16(1), 61–102 (2011)
Hirasaki, Y., Kojima, H., Tsuchiya, T.: Applying random testing to constrained interaction testing. In: Proceedings of SEKE 2013, pp. 193–198 (2014)
Johansen, M.F., Haugen, O., Fleurey, F.: An algorithm for generating t-wise covering arrays from large feature models. In: Proceedings of SPLC 2012, pp. 46–55 (2012)
Kuhn, D.R., Bryce, R., Duan, F., Ghandehari, L.S., Lei, Y., Kacker, R.N.: Chapter one - combinatorial testing: theory and practice. In: Advances in Computers, vol. 99, pp. 1–66. Elsevier (2015)
Kuhn, D.R., Kacker, R.N., Lei, Y.: Introduction to Combinatorial Testing. CRC Press, Boca Raton (2013)
Lin, J., Cai, S., Luo, C., Lin, Q., Zhang, H.: Towards more efficient meta-heuristic algorithms for combinatorial test generation. In: Proceedings of ESEC/FSE 2019, pp. 212–222 (2019)
Lynce, I., Silva, J.P.M.: On computing minimum unsatisfiable cores. In: Proceedings of SAT 2004 (2004)
Nanba, T., Tsuchiya, T., Kikuno, T.: Using satisfiability solving for pairwise testing in the presence of constraints. IEICE Trans. 95-A(9), 1501–1505 (2012)
Oh, J., Batory, D.S., Myers, M., Siegmund, N.: Finding near-optimal configurations in product lines by random sampling. In: Proceedings of ESEC/FSE 2017, pp. 61–71 (2017)
Wu, H., Nie, C., Petke, J., Jia, Y., Harman, M.: An empirical comparison of combinatorial testing, random testing and adaptive random testing. IEEE Trans. Softw. Eng. (2018)
Yamada, A., Biere, A., Artho, C., Kitamura, T., Choi, E.: Greedy combinatorial test case generation using unsatisfiable cores. In: Proceedings of ASE 2016, pp. 614–624 (2016)
Yu, L., Lei, Y., Kacker, R.N., Kuhn, D.R.: ACTS: a combinatorial test generation tool. In: Proceedings of ICST 2013, pp. 370–375 (2013)
Yu, L., Duan, F., Lei, Y., Kacker, R., Kuhn, D.R.: Combinatorial test generation for software product lines using minimum invalid tuples. In: Proceedings of HASE 2014, pp. 65–72 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 IFIP International Federation for Information Processing
About this paper
Cite this paper
Jin, H., Kitamura, T., Choi, EH., Tsuchiya, T. (2020). A Comparative Study on Combinatorial and Random Testing for Highly Configurable Systems. In: Casola, V., De Benedictis, A., Rak, M. (eds) Testing Software and Systems. ICTSS 2020. Lecture Notes in Computer Science(), vol 12543. Springer, Cham. https://doi.org/10.1007/978-3-030-64881-7_20
Download citation
DOI: https://doi.org/10.1007/978-3-030-64881-7_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-64880-0
Online ISBN: 978-3-030-64881-7
eBook Packages: Computer ScienceComputer Science (R0)
