Thanassis Avgerinos
ABSTRACT
We present MergePoint, a new binary-only symbolic execution system for large-scale and fully unassisted testing of commodity off-the-shelf (COTS) software. MergePoint introduces veritesting, a new technique that employs static symbolic execution to amplify the effect of dynamic symbolic execution. Veritesting allows MergePoint to find twice as many bugs, explore orders of magnitude more paths, and achieve higher code coverage than previous dynamic symbolic execution systems. MergePoint is currently running daily on a 100 node cluster analyzing 33,248 Linux binaries; has generated more than 15 billion SMT queries, 200 million test cases, 2,347,420 crashes, and found 11,687 bugs in 4,379 distinct applications.
- Online Bibliography for Symbolic Execution. http:// sites.google.com/site/symexbib.Google Scholar
- A. V. Aho, R. Sethi, and J. D. Ullman. Compilers: Principles, Techniques, and Tools. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 1986. Google ScholarDigital Library
- J. R. Allen, K. Kennedy, C. Porterfield, and J. Warren. Conversion of Control Dependence to Data Dependence. In Proceedings of the 10th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages, pages 177–189, New York, NY, USA, 1983. ACM Press. Google ScholarDigital Library
- S. Anand, P. Godefroid, and N. Tillmann. Demand-Driven Compositional Symbolic Execution. In Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pages 367–381, Berlin, Heidelberg, 2008. Springer-Verlag. Google ScholarDigital Library
- D. Babic. Exploiting structure for scalable software verification. PhD thesis, University of British Columbia, Vancouver, Canada, 2008.Google Scholar
- D. Babic and A. J. Hu. Calysto: Scalable and Precise Extended Static Checking. In Proceedings of the 30th International Conference on Software Engineering, pages 211–220, New York, NY, USA, 2008. ACM. Google ScholarDigital Library
- S. Bardin, P. Herrmann, J. Leroux, O. Ly, R. Tabary, and A. Vincent. The BINCOA Framework for Binary Code Analysis. In Proceedings of the 23rd International Conference on Computer Aided Verification, pages 165– 170, Berlin, Heidelberg, 2011. Springer-Verlag. Google ScholarDigital Library
- D. Beyer, T. A. Henzinger, and G. Theoduloz. Configurable Software Verification: Concretizing the Convergence of Model Checking and Program Analysis. In Proceedings of the 19th International Conference on Computer Aided Verification, pages 504–518, Berlin, Heidelberg, 2007. Springer-Verlag. Google ScholarDigital Library
- P. Boonstoppel, C. Cadar, and D. Engler. RWset: Attacking Path Explosion in Constraint-Based Test Generation. In Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pages 351–366, Berlin, Heidelberg, 2008. Springer-Verlag. Google ScholarDigital Library
- E. Bounimova, P. Godefroid, and D. Molnar. Billions and Billions of Constraints: Whitebox Fuzz Testing in Production. In Proceedings of the 35th IEEE International Conference on Software Engineering, pages 122–131, Piscataway, NJ, USA, 2013. IEEE Press. Google ScholarDigital Library
- R. S. Boyer, B. Elspas, and K. N. Levitt. SELECT—a formal system for testing and debugging programs by symbolic execution. ACM SIGPLAN Notices, 10(6): 234–245, 1975. Google ScholarDigital Library
- D. Brumley, I. Jager, T. Avgerinos, and E. J. Schwartz. BAP: A Binary Analysis Platform. In Proceedings of the 23rd International Conference on Computer Aided Verification, pages 463–469. Springer-Verlag, 2011. Google ScholarDigital Library
- S. Bucur, V. Ureche, C. Zamfir, and G. Candea. Parallel symbolic execution for automated real-world software testing. In Proceedings of the 6th ACM SIGOPS European Conference on Computer Systems, pages 183–198. ACM Press, 2011. Google ScholarDigital Library
- C. Cadar and K. Sen. Symbolic execution for software testing: three decades later. Communications of the ACM, 56(2):82–90, 2013. Google ScholarDigital Library
- C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Engler. EXE : Automatically Generating Inputs of Death. In Proceedings of the 13th ACM Conference on Computer and Communications Security, New York, NY, USA, 2006. ACM. Google ScholarDigital Library
- C. Cadar, D. Dunbar, and D. Engler. KLEE: Unassisted and Automatic Generation of High-coverage Tests for Complex Systems Programs. In Proceedings of the 8th USENIX Symposium on Operating System Design and Implementation, pages 209–224, Berkeley, CA, USA, 2008. USENIX Association. Google ScholarDigital Library
- C. Cadar, D. Dunbar, and D. R. Engler. KLEE Coreutils Experiment. http://klee.github.io/klee/ CoreutilsExperiments.html, 2008.Google Scholar
- S. K. Cha, T. Avgerinos, A. Rebert, and D. Brumley. Unleashing Mayhem on Binary Code. In Proceedings of the 33rd IEEE Symposium on Security and Privacy, pages 380–394, Washington, DC, USA, 2012. IEEE Computer Society. Google ScholarDigital Library
- V. Chipounov, V. Kuznetsov, and G. Candea. S2E: A platform for in-vivo multi-path analysis of software systems. In Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems, pages 265–278, New York, NY, USA, 2011. ACM. Google ScholarDigital Library
- P. Collingbourne, C. Cadar, and P. H. Kelly. Symbolic crosschecking of floating-point and SIMD code. Proceedings of the 6th ACM SIGOPS European conference on Computer Systems, pages 315–328, 2011. Google ScholarDigital Library
- L. De Moura and N. Bjørner. Z3: An Efficient SMT Solver. In Proceedings of 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pages 337–340, Berlin, Heidelberg, 2008. Springer-Verlag.Google ScholarCross Ref
- I. Dillig, T. Dillig, and A. Aiken. Sound, Complete and Scalable Path-Sensitive Analysis. In Proceedings of the 29th ACM Conference on Programming Language Design and Implementation, pages 270–280, New York, NY, USA, 2008. ACM. Google ScholarDigital Library
- J. Filliˆ atre and S. Conchon. Type-safe modular hashconsing. In Proceedings of the Workshop on ML, pages 12–19, New York, NY, USA, 2006. ACM. Google ScholarDigital Library
- C. Flanagan and J. Saxe. Avoiding exponential explosion: Generating compact verification conditions. In Proceedings of the 28th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 193–205, New York, NY, USA, 2001. ACM. Google ScholarDigital Library
- P. Godefroid. Compositional Dynamic Test Generation. In Proceedings of the 34th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 47–54, New York, NY, USA, 2007. ACM. Google ScholarDigital Library
- P. Godefroid, N. Klarlund, and K. Sen. DART : Directed Automated Random Testing. In Proceedings of the 26th ACM Conference on Programming Language Design and Implementation, New York, NY, USA, 2005. ACM. Google ScholarDigital Library
- P. Godefroid, M. Y. Levin, and D. Molnar. Automated Whitebox Fuzz Testing. In Proceedings of the 15th Network and Distributed System Security Symposium. The Internet Society, 2008.Google Scholar
- P. Godefroid, M. Y. Levin, and D. Molnar. SAGE: Whitebox Fuzzing for Security Testing. Communications of the ACM, 55(3):40–44, 2012. Google ScholarDigital Library
- E. Goto. Monocopy and Associative Algorithms in Extended Lisp. Technical Report TR-74-03, University of Tokyo, 1974.Google Scholar
- T. Hansen, P. Schachte, and H. Søndergaard. State Joining and Splitting for the Symbolic Execution of Binaries. Runtime Verification, pages 76–92, 2009. Google ScholarDigital Library
- W. Howden. Methodology for the Generation of Program Test Data. IEEE Transactions on Computers, C-24(5):554–560, 1975. Google ScholarDigital Library
- J. Kinder and H. Veith. Jakstab: A Static Analysis Platform for Binaries. In Proceedings of the 20th International Conference on Computer Aided Verification, pages 423–427, Berlin, Heidelberg, 2008. Springer-Verlag. Google ScholarDigital Library
- J. C. King. Symbolic execution and program testing. Communications of the ACM, 19(7):385–394, 1976. Google ScholarDigital Library
- A. Koelbl and C. Pixley. Constructing Efficient Formal Models from High-Level Descriptions Using Symbolic Simulation. International Journal of Parallel Programming, 33(6):645–666, Dec. 2005. Google ScholarDigital Library
- V. Kuznetsov, J. Kinder, S. Bucur, and G. Candea. Efficient state merging in symbolic execution. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 193–204, New York, NY, USA, 2012. ACM. Google ScholarDigital Library
- C. Lattner and V. Adve. LLVM: A compilation framework for lifelong program analysis & transformation. In Proceedings of the International Symposium on Code Generation and Optimization: Feedback-directed and Runtime Optimization, pages 75–86, Washington, DC, USA, 2004. IEEE Computer Society. Google ScholarDigital Library
- K. R. M. Leino. Efficient weakest preconditions. Information Processing Letters, 93(6):281–288, 2005. Google ScholarDigital Library
- C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. J. Reddi, and K. Hazelwood. Pin: Building Customized Program Analysis Tols with Dynamic Instrumentation. In Proceedings of the 26th ACM Conference on Programming Language Design and Implementation, pages 190–200, New York, NY, USA, 2005. ACM. Google ScholarDigital Library
- P. D. Marinescu and C. Cadar. Make test-zesti: A symbolic execution solution for improving regression testing. In Proceedings of the 34th International Conference on Software Engineering, pages 716–726, Piscataway, NJ, USA, 2012. IEEE Press. Google ScholarDigital Library
- Mayhem. 1.2K Crashes in Debian, 2013. URL http://lists.debian.org/debian-devel/2013/06/ msg00720.html.Google Scholar
- Mayhem. Open Source Statistics & Analysis, 2013. URL http://www.forallsecure.com/summaries.Google Scholar
- D. Molnar, X. Li, and D. Wagner. Dynamic test generation to find integer bugs in x86 binary linux programs. In Proceedings of the USENIX Security Symposium, pages 67–82, 2009. Google ScholarDigital Library
- C. S. Păsăreanu and W. Visser. A survey of new trends in symbolic execution for software testing and analysis. International Journal on Software Tools for Technology Transfer, 11(4):339–353, Aug. 2009. Google ScholarDigital Library
- A. J. Romano. Linux Bug Release, July 2013. URL http://www.bugsdujour.com/release/.Google Scholar
- E. J. Schwartz, T. Avgerinos, and D. Brumley. All You Ever Wanted to Know about Dynamic Taint Analysis and Forward Symbolic Execution (but Might Have Been Afraid to Ask). In Proceedings of the 31st IEEE Symposium on Security and Privacy, pages 317–331, Washington, DC, USA, 2010. IEEE Computer Society. Google ScholarDigital Library
- K. Sen, D. Marinov, and G. Agha. CUTE: A Concolic Unit Testing Engine for C. In Proceedings of the 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 263–272, New York, NY, USA, 2005. ACM. Google ScholarDigital Library
- P. Tu and D. Padua. Efficient building and placing of gating functions. In Proceedings of the 16th ACM Conference on Programming Language Design and Implementation, pages 47–55, New York, NY, USA, 1995. ACM. Google ScholarDigital Library
- Y. Xie and A. Aiken. Scalable error detection using boolean satisfiability. In Proceedings of the 32nd ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 351–363, New York, NY, USA, 2005. ACM. Google ScholarDigital Library
- H. Zhu, P. A. V. Hall, and J. H. R. May. Software unit test coverage and adequacy. ACM Computing Surveys, 29(4):366–427, 1997. Google ScholarDigital Library
Index Terms
- Enhancing symbolic execution with veritesting
Recommendations
Veritesting Challenges in Symbolic Execution of Java
Scaling symbolic execution to industrial-sized programs is an important open research problem. Veritesting is a promising technique that improves scalability by combining the advantages of static symbolic execution with those of dynamic symbolic ...
Verifying systems rules using rule-directed symbolic execution
ASPLOS '13Systems code must obey many rules, such as "opened files must be closed." One approach to verifying rules is static analysis, but this technique cannot infer precise runtime effects of code, often emitting many false positives. An alternative is ...
Verifying systems rules using rule-directed symbolic execution
ASPLOS '13Systems code must obey many rules, such as "opened files must be closed." One approach to verifying rules is static analysis, but this technique cannot infer precise runtime effects of code, often emitting many false positives. An alternative is ...
Comments