Skip to main content
SpringerLink
Log in

Empirical Analysis and Modeling of Black-Box Mutational Fuzzing

  • Conference paper
Engineering Secure Software and Systems (ESSoS 2016)

Part of the book series: Lecture Notes in Computer Science ((LNSC,volume 9639))

Included in the following conference series:

Abstract

Black-box mutational fuzzing is a simple yet effective method for finding software vulnerabilities. In this work, we collect and analyze fuzzing campaign data of 60,000 fuzzing runs, 4,000 crashes and 363 unique bugs, from multiple Linux programs using CERT Basic Fuzzing Framework. Motivated by the results of empirical analysis, we propose a stochastic model that captures the long-tail distribution of bug discovery probability and exploitability. This model sheds light on practical questions such as what is the expected number of bugs discovered in a fuzzing campaign within a given time, why improving software security is hard, and why different parties (e.g., software vendors, white hats, and black hats) are likely to find different vulnerabilities. We also discuss potential generalization of this model to other vulnerability discovery approaches, such as recently emerged bug bounty programs.

figure afigure a

The stamp on the top of this paper refers to an approval process conducted by the ESSoS artifact evaluation committee chaired by Alessandra Gorla and Jacques Klein.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://github.com/movingname/fuzzingModel.

  2. 2.

    The method used to generate the hash is an extension of the fuzzy stack hash method proposed in the literature [24].

References

  1. Algarni, A., Malaiya, Y.: Software vulnerability markets: discoverers and buyers. Int. J. Comput. Appl. Technol. Inf. Sci. Eng. 8(3), 71–81 (2014)

    Google Scholar 

  2. Allodi, L.: The heavy tails of vulnerability exploitation. In: Piessens, F., Caballero, J., Bielova, N. (eds.) ESSoS 2015. LNCS, vol. 8978, pp. 133–148. Springer, Heidelberg (2015)

    Google Scholar 

  3. Alstott, J., Bullmore, E., Plenz, D.: Powerlaw: a python package for analysis of heavy-tailed distributions. PLoS ONE 9, e85777 (2014)

    Article  Google Scholar 

  4. Bishop, P., Bloomfield, R.: A conservative theory for long-term reliability-growth prediction [of software]. IEEE Trans. Reliab. 45(4), 550–560 (1996)

    Article  Google Scholar 

  5. Böhme, R., Schwartz, G.: Modeling cyber-insurance: towards a unifying framework. In: The Workshop on the Economics of Information Security (WEIS) (2010)

    Google Scholar 

  6. Brady, R.M., Anderson, R., Ball, R.C.: Murphy’s law, the fitness of evolving species, and the limits of software reliability. Number 471. University of Cambridge, Computer Laboratory (1999)

    Google Scholar 

  7. Cha, S.K., Woo, M., Brumley, D.: Program-adaptive mutational fuzzing. In: 36th IEEE Symposium on Security and Privacy (2015)

    Google Scholar 

  8. Edwards, B., Hofmeyr, S., Forrest, S.: Hype, heavy tails: a closer look at data breaches. In: The Workshop on the Economics of Information Security (WEIS) (2015)

    Google Scholar 

  9. Evans, C., Moore, M., Ormandy, T.: Fuzzing at scale. Google Online Security Blog

    Google Scholar 

  10. Fenton, N., Bieman, J.: Software metrics: a rigorous and practical approach. CRC Press, Boca Raton (2014)

    Book  MATH  Google Scholar 

  11. Forrester, J.E., Miller, B.P.: An empirical study of the robustness of windows nt applications using random testing. In: Proceedings of the 4th USENIX Windows System Symposium, Seattle, pp. 59–68 (2000)

    Google Scholar 

  12. Godefroid, P., Levin, M.Y., Molnar, D.A., et al.: Automated whitebox fuzz testing. In: The Network and Distributed System Security Symposium, vol. 8, pp. 151–166 (2008)

    Google Scholar 

  13. Hafiz, M., Fang, M.: Game of detections: how are security vulnerabilities discovered in the wild? Empirical Software Engineering, pp. 1–40 (2015)

    Google Scholar 

  14. Householder, A.D., Foote, J.M.: Probability-based parameter selection for black-box fuzz testing. In: CERT (2012)

    Google Scholar 

  15. W. Jackson. Has secure software development reached its limits? GCN

    Google Scholar 

  16. Johnson, B., Laszka, A., Grossklags, J.: Games of timing for security in dynamic environments. In: Khouzani, M.H.R., et al. (eds.) GameSec 2015. LNCS, vol. 9406, pp. 57–73. Springer, Heidelberg (2015). doi:10.1007/978-3-319-25594-1_4

    Chapter  Google Scholar 

  17. Jurczyk, M., Coldwind, G.: Ffmpeg and a thousand fixes. Google Online Security Blog

    Google Scholar 

  18. Laszka, A., Grossklags, J.: Should cyber-insurance providers invest in software security? In: Pernul, G., Ryan, P.Y.A., Weippl, E. (eds.) ESORICS 2015, Part I. LNCS, vol. 9326, pp. 483–502. Springer, Heidelberg (2015)

    Chapter  Google Scholar 

  19. Maillart, T., Sornette, D.: Heavy-tailed distribution of cyber-risks. Eur. Phys. J. B 75(3), 357–364 (2010)

    Article  MATH  Google Scholar 

  20. McNally, R., Yiu, K., Grove, D., Gerhardy, D.: Fuzzing: the state of the art. Technical report, DTIC Document (2012)

    Google Scholar 

  21. Miller, B.P., Fredriksen, L., So, B.: An empirical study of the reliability of unix utilities. Commun. ACM 33(12), 32–44 (1990)

    Article  Google Scholar 

  22. Miller, C.: Babysitting an army of monkeys. In: CanSecWest (2010)

    Google Scholar 

  23. Miller, D.R.: Exponential order statistic models of software reliability growth. IEEE Trans. Softw. Eng. 1, 12–24 (1986)

    Article  MATH  Google Scholar 

  24. Molnar, D., Li, X.C., Wagner, D.: Dynamic test generation to find integer bugs in x86 binary linux programs. In: USENIX Security Symposium, vol. 9 (2009)

    Google Scholar 

  25. Naraine, R.: Teenager hacks google chrome with three 0day vulnerabilities. ZDNet

    Google Scholar 

  26. Newman, M.: Power laws, Pareto distributions and Zipf’s law. Contemp. Phys. 46(5), 323–351 (2005)

    Article  Google Scholar 

  27. Radianti, J.: Eliciting information on the vulnerability black market from interviews. In: Proceedings of the SECURWARE, pp. 154–159 (2010)

    Google Scholar 

  28. Rebert, A., Cha, S.K., Avgerinos, T., Foote, J., Warren, D., Grieco, G., Brumley, D.: Optimizing seed selection for fuzzing. In: Proceedings of the USENIX Security Symposium, pp. 861–875 (2014)

    Google Scholar 

  29. Rue, R., Pfleeger, S.L.: Making the best use of cybersecurity economic models. IEEE Secur. Priv. 4, 52–60 (2009)

    Article  Google Scholar 

  30. Wang, T., Wei, T., Gu, G., Zou, W.: Taintscope: a checksum-aware directed fuzzing tool for automatic software vulnerability detection. In: IEEE Symposium on Security and Privacy (2010)

    Google Scholar 

  31. Woo, M., Cha, S.K., Gottlieb, S., Brumley, D.: Scheduling black-box mutational fuzzing. In: ACM Conference on Computer and Communications Security (2013)

    Google Scholar 

  32. Zhao, M., Grossklags, J., Liu, P.: An empirical study of web vulnerability discovery ecosystems. In: ACM Conference on Computer and Communications Security (2015)

    Google Scholar 

Download references

Acknowledgment

We sincerely thank our shepherd and the anonymous reviewers for their valuable comments and suggestions on early versions of this paper. This work was supported by ARO W911NF-13-1-0421 (MURI), NSF CCF-1320605, NSF CNS-1422594, NSF CNS-1505664, ARO W911NF-15-1-0576, and NIETP CAE Cybersecurity Grant.

Author information

Authors and Affiliations

  1. College of Information Sciences and Technology, Pennsylvania State University, State College, USA

    Mingyi Zhao & Peng Liu

Authors
  1. Mingyi Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingyi Zhao .

Editor information

Editors and Affiliations

  1. IMDEA Software Institute, Madrid, Spain

    Juan Caballero

  2. Paderborn University & Fraunhofer IEM, Paderborn, Germany

    Eric Bodden

  3. VU University, Amsterdam, The Netherlands

    Elias Athanasopoulos

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Zhao, M., Liu, P. (2016). Empirical Analysis and Modeling of Black-Box Mutational Fuzzing. In: Caballero, J., Bodden, E., Athanasopoulos, E. (eds) Engineering Secure Software and Systems. ESSoS 2016. Lecture Notes in Computer Science(), vol 9639. Springer, Cham. https://doi.org/10.1007/978-3-319-30806-7_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-30806-7_11

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-30805-0

  • Online ISBN: 978-3-319-30806-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation