Elnatan Reisner
Jeffrey S. Foster
ABSTRACT
Many modern software systems are designed to be highly configurable, which increases flexibility but can make programs hard to test, analyze, and understand. We present an initial empirical study of how configuration options affect program behavior. We conjecture that, at certain levels of abstraction, configuration spaces are far smaller than the worst case, in which every configuration is distinct. We evaluated our conjecture by studying three configurable software systems: vsftpd, ngIRCd, and grep. We used symbolic evaluation to discover how the settings of run-time configuration options affect line, basic block, edge, and condition coverage for our subjects under a given test suite. Our results strongly suggest that for these subject programs, test suites, and configuration options, when abstracted in terms of the four coverage criteria above, configuration spaces are in fact much smaller than combinatorics would suggest and are effectively the composition of many small, self-contained groupings of options.
- R. Brownlie, J. Prowse, and M. S. Phadke. Robust testing of AT&T PMX/StarMAIL using OATS. AT&T Technical Journal, 71(3):41--7, 1992.Google Scholar
Cross Ref
- C. Cadar, D. Dunbar, and D. R. Engler. KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, pages 209--224, 2008. Google ScholarDigital Library
- C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Engler. EXE: automatically generating inputs of death. In CCS, pages 322--335, 2006. Google ScholarDigital Library
- D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton. The AETG system: an approach to testing based on combinatorial design. TSE, 23(7):437--44, 1997. Google ScholarDigital Library
- I. S. Dunietz, W. K. Ehrlich, B. D. Szablak, C. L. M. ws, and A. Iannino. Applying design of experiments to software testing. In ICSE, pages 205--215, 1997. Google ScholarDigital Library
- V. Ganesh and D. L. Dill. A decision procedure for bit-vectors and arrays. In CAV, July 2007. Google ScholarDigital Library
- P. Godefroid, N. Klarlund, and K. Sen. DART: directed automated random testing. In PLDI, pages 213--223, 2005. Google ScholarDigital Library
- P. Godefroid, M. Y. Levin, and D. A. Molnar. Automated whitebox fuzz testing. In NDSS. Internet Society, 2008.Google Scholar
- J. C. King. Symbolic execution and program testing. Commun. ACM, 19(7):385--394, 1976. Google ScholarDigital Library
- D. Kuhn and M. Reilly. An investigation of the applicability of design of experiments to software testing. In NASA Goddard/IEEE Software Engineering Workshop, pages 91--95, 2002. Google ScholarDigital Library
- G. C. Necula, S. McPeak, S. P. Rahul, and W. Weimer. CIL: Intermediate language and tools for analysis and transformation of C programs. In CC, pages 213--228, 2002. Google ScholarDigital Library
- A. Porter, C. Yilmaz, A. M. Memon, D. C. Schmidt, and B. Natarajan. Skoll: A process and infrastructure for distributed continuous quality assurance. TSE, 33(8):510--525, August, 2007. Google ScholarDigital Library
- E. Reisner, C. Song, K.-K. Ma, J. S. Foster, and A. Porter. Using symbolic evaluation to understand behavior in configurable software systems. Technical Report CS-TR-4946, Department of Computer Science, University of Maryland, 2009.Google Scholar
Recommendations
PrefFinder: getting the right preference in configurable software systems
ASE '14: Proceedings of the 29th ACM/IEEE International Conference on Automated Software EngineeringHighly configurable software, such as web browsers,databases or office applications, have a large number of preferences that the user can customize, but documentation of them may be scarce or distributed. A user, tester or service technician may have to ...
Symbolic execution and program testing
This paper describes the symbolic execution of programs. Instead of supplying the normal inputs to a program (e.g. numbers) one supplies symbols representing arbitrary values. The execution proceeds as in a normal execution except that values may be ...
An Empirical Study on Performance Bugs for Highly Configurable Software Systems
ESEM '16: Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and MeasurementModern computer systems are highly-configurable, complicating the testing and debugging process. The sheer size of the configuration space makes the quality of software even harder to achieve. Performance is one of the key aspects of non-functional ...
Comments