Talia Ringer
Serdar Tasiran
Skip Abstract Section
Abstract
Developing a small but useful set of inputs for tests is challenging. We show that a domain-specific language backed by a constraint solver can help the programmer with this process. The solver can generate a set of test inputs and guarantee that each input is different from other inputs in a way that is useful for testing.
This paper presents Iorek: a tool that empowers the programmer with the ability to express to any SMT solver what it means for inputs to be different. The core of Iorek is a rich language for constraining the set of inputs, which includes a novel bounded enumeration mechanism that makes it easy to define and encode a flexible notion of difference over a recursive structure. We demonstrate the flexibility of this mechanism for generating strings.
We use Iorek to test real services and find that it is effective at finding bugs. We also build Iorek into a random testing tool and show that it increases coverage.
- Saswat Anand, Corina S. Păsăreanu, and Willem Visser. 2006. Symbolic Execution with Abstract Subsumption Checking (SPIN). Google Scholar
Digital Library
- Saswat Anand, Corina S. Păsăreanu, and Willem Visser. 2007. JPF-SE: A Symbolic Execution Extension to Java PathFinder (TACAS).Google Scholar
- Clark Barrett, Roberto Sebastiani, Sanjit A. Seshia, and Cesare Tinelli. 2008a. Handbook of Satisfiability. Chapter Satisfiability Modulo Theories, 127–149.Google Scholar
- Clark Barrett, Roberto Sebastiani, Sanjit A. Seshia, and Cesare Tinelli. 2008b. Handbook of Satisfiability. Chapter Satisfiability Modulo Theories, 737–797.Google Scholar
- Nikolaj Bjørner, Vijay Ganesh, Raphaël Michel, and Margus Veanes. 2012. An SMT-LIB format for sequences and regular expressions (SMT Workshop).Google Scholar
- Rastislav Bodik, Satish Chandra, Joel Galenson, Doug Kimelman, Nicholas Tung, Shaon Barman, and Casey Rodarmor. 2010. Programming with Angelic Nondeterminism (POPL). Google ScholarDigital Library
- James Bornholt, Emina Torlak, Dan Grossman, and Luis Ceze. 2016. Optimizing Synthesis with Metasketches (POPL). Google ScholarDigital Library
- Chandrasekhar Boyapati, Sarfraz Khurshid, and Darko Marinov. 2002. Korat: Automated Testing Based on Java Predicates (ISSTA). Google ScholarDigital Library
- Cristian Cadar, Vijay Ganesh, Peter M. Pawlowski, David L. Dill, and Dawson R. Engler. 2006. EXE: Automatically Generating Inputs of Death (CCS). Google ScholarDigital Library
- Jens R. Calamé, Natalia Ioustinova, and Jaco van de Pol. 2007. Automatic Model-Based Generation of Parameterized Test Cases Using Data Abstraction. ENTCS 191 (2007), 25–48. Google ScholarDigital Library
- Vitaly Chipounov, Volodymyr Kuznetsov, and George Candea. 2011. S2E: A Platform for In-vivo Multi-path Analysis of Software Systems (Proceedings of the Sixteenth International Conference on Architectural Support for Programming Languages and Operating Systems).Google Scholar
- Koen Claessen and John Hughes. 2000. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs (ICFP). Google ScholarDigital Library
- Brett Daniel, Danny Dig, Kely Garcia, and Darko Marinov. 2007. Automated Testing of Refactoring Engines (ESEC-FSE). Google ScholarDigital Library
- Jeremy Dick and Alain Faivre. 1993. Automating the generation and sequencing of test cases from model-based specifications (International Symposium of Formal Methods Europe).Google Scholar
- Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software (ESEC/FSE). Google ScholarDigital Library
- Patrice Godefroid, Adam Kiezun, and Michael Y. Levin. 2008. Grammar-based Whitebox Fuzzing (PLDI). Google ScholarDigital Library
- Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed Automated Random Testing (PLDI). Google ScholarDigital Library
- Patrice Godefroid, Michael Y Levin, and David A Molnar. 2008. Automated Whitebox Fuzz Testing (NDSS).Google Scholar
- John B. Goodenough and Susan L. Gerhart. 1975. Toward a Theory of Test Data Selection. In Proceedings of the International Conference on Reliable Software. Google ScholarDigital Library
- Sumit Gulwani. 2012. Synthesis from examples: Interaction models and algorithms (SYNASC).Google Scholar
- Susmit Jha and Sanjit A. Seshia. 2014. Are there good mistakes? A theoretical analysis of CEGIS (3rd Workshop on Synthesis (SYNT)).Google Scholar
- Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. 2013. Why Don’t Software Developers Use Static Analysis Tools to Find Bugs? (ICSE). Google ScholarCross Ref
- Sarfraz Khurshid and Darko Marinov. 2004. TestEra: Specification-Based Testing of Java Programs Using SAT. ASE (2004).Google Scholar
- Ali Sinan Köksal, Viktor Kuncak, and Philippe Suter. 2012. Constraints As Control (POPL). Google ScholarDigital Library
- Ivan Kuraj, Viktor Kuncak, and Daniel Jackson. 2015. Programming with Enumerable Sets of Structures (OOPSLA). Google ScholarDigital Library
- Leonidas Lampropoulos, Diane Gallois-Wong, Cătălin Hriţcu, John Hughes, Benjamin C. Pierce, and Li-yao Xia. 2017. Beginner’s Luck: A Language for Property-based Generators (POPL).Google Scholar
- Nuo Li, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. 2009. Reggae: Automated Test Generation for Programs Using Complex Regular Expressions (ASE).Google Scholar
- Tianyi Liang, Andrew Reynolds, Cesare Tinelli, Clark Barrett, and Morgan Deters. 2014. A DPLL(T) Theory Solver for a Theory of Strings and Regular Expressions (CAV). Google ScholarDigital Library
- Tim Mackinnon, Steve Freeman, and Philip Craig. 2001. Extreme Programming Examined. Chapter Endo-testing: Unit Testing with Mock Objects, 287–301.Google Scholar
- Phil McMinn, Muzammil Shahbaz, and Mark Stevenson. 2012. Search-Based Test Input Generation for String Data Types Using the Results of Web Queries (International Conference on Software Testing, Verification and Validation).Google Scholar
- Kuldeep S. Meel, Moshe Y. Vardi, Supratik Chakraborty, Daniel J. Fremont, Sanjit A. Seshia, Dror Fried, Alexander Ivrii, and Sharad Malik. 2015. Constrained Sampling and Counting: Universal Hashing Meets SAT Solving. CoRR (2015).Google Scholar
- Simeon Ntafos. 1998. On Random and Partition Testing (ISSTA). Google ScholarDigital Library
- Carlos Pacheco and Michael D. Ernst. 2007. Randoop: Feedback-directed Random Testing for Java (OOPSLA). Google ScholarDigital Library
- Pablo Ponzio, Nazareno Aguirre, Marcelo F. Frias, and Willem Visser. 2016. Field-exhaustive Testing (FSE). Google ScholarDigital Library
- Nicolás Rosner, Valeria Bengolea, Pablo Ponzio, Shadi Abdul Khalek, Nazareno Aguirre, Marcelo F. Frias, and Sarfraz Khurshid. 2014. Bounded Exhaustive Test Input Generation from Hybrid Invariants (OOPSLA). Google ScholarDigital Library
- Hesam Samimi, Rebecca Hicks, Ari Fogel, and Todd Millstein. 2013. Declarative Mocking (ISSTA). Google ScholarDigital Library
- Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: A Concolic Unit Testing Engine for C (ESEC/FSE). Google ScholarCross Ref
- Valerio Senni and Fabio Fioravanti. 2012. Generation of Test Data Structures Using Constraint Logic Programming (TAP). Google ScholarDigital Library
- Ali Shahbazi and James Miller. 2016. Black-Box String Test Case Generation Through a Multi-Objective Optimization. IEEE Transactions on Software Engineering (2016), 361–378. Google ScholarDigital Library
- Armando Solar Lezama. 2008. Program Synthesis By Sketching. Ph.D. Dissertation. EECS Department, University of California, Berkeley.Google Scholar
- Nikolai Tillmann and Jonathan De Halleux. 2008. Pex–white box test generation for. net (TAP). Google ScholarCross Ref
- Emina Torlak and Rastislav Bodik. 2013. Growing Solver-aided Languages with Rosette (Onward!). Google ScholarDigital Library
- Emina Torlak and Daniel Jackson. 2007. Kodkod: A Relational Model Finder (TACAS).Google Scholar
- Minh-Thai Trinh, Duc-Hiep Chu, and Joxan Jaffar. 2014. S3: A Symbolic String Solver for Vulnerability Detection in Web Applications (CCS). Google ScholarDigital Library
- Richard Uhler and Nirav Dave. 2013. Smten: Automatic Translation of High-level Symbolic Computations into SMT Queries (CAV).Google Scholar
- Yunhui Zheng, Xiangyu Zhang, and Vijay Ganesh. 2013. Z3-str: A Z3-based String Solver for Web Application Analysis (FSE).Google Scholar
Index Terms
- A solver-aided language for test input generation
Recommendations
Growing solver-aided languages with rosette
Onward! 2013: Proceedings of the 2013 ACM international symposium on New ideas, new paradigms, and reflections on programming & softwareSAT and SMT solvers have automated a spectrum of programming tasks, including program synthesis, code checking, bug localization, program repair, and programming with oracles. In principle, we obtain all these benefits by translating the program (once) ...
A lightweight symbolic virtual machine for solver-aided host languages
PLDI '14Solver-aided domain-specific languages (SDSLs) are an emerging class of computer-aided programming systems. They ease the construction of programs by using satisfiability solvers to automate tasks such as verification, debugging, synthesis, and non-...
A lightweight symbolic virtual machine for solver-aided host languages
PLDI '14: Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and ImplementationSolver-aided domain-specific languages (SDSLs) are an emerging class of computer-aided programming systems. They ease the construction of programs by using satisfiability solvers to automate tasks such as verification, debugging, synthesis, and non-...
Comments