Skip Abstract Section
Abstract
Search-based software testing (SBST) has been shown to be an effective technique to generate test cases automatically. Its effectiveness strongly depends on the guidance of the fitness function. Unfortunately, a common issue in SBST is the so-called flag problem, where the fitness landscape presents a plateau that provides no guidance to the search. In this article, we provide a series of novel testability transformations aimed at providing guidance in the context of commonly used API calls (e.g., strings that need to be converted into valid date/time objects). We also provide specific transformations aimed at helping the testing of REST Web Services. We implemented our novel techniques as an extension to EvoMaster, an SBST tool that generates system-level test cases. Experiments on nine open-source REST web services, as well as an industrial web service, show that our novel techniques improve performance significantly.
- [n.d.].OpenAPI/Swagger. Retrieved from https://swagger.io/.Google Scholar
- [n.d.]. RestAssured. Retrieved from https://github.com/rest-assured/rest-assured.Google Scholar
- [n.d.]. Spring Framework. Retrieved from https://spring.io.Google Scholar
- S. Ali, L. C. Briand, H. Hemmati, and R. K. Panesar-Walawege. 2010. A systematic review of the application and empirical investigation of search-based test-case generation. IEEE Trans. Softw. Eng. 36, 6 (2010), 742–762. Google ScholarDigital Library
- Nadia Alshahwan, Xinbo Gao, Mark Harman, Yue Jia, Ke Mao, Alexander Mols, Taijin Tei, and Ilya Zorin. 2018. Deploying search based software engineering with Sapienz at Facebook. In International Symposium on Search Based Software Engineering (SSBSE'18). Springer, 3–45.Google ScholarCross Ref
- Mohammad Alshraideh and Leonardo Bottaci. 2006. Search-based software test data generation for string data using program-specific search operators. Softw. Test., Verif. Reliab. 16, 3 (2006), 175–203. Google ScholarDigital Library
- Mohammad Alshraideh and Leonardo Bottaci. 2006. Search-based software test data generation for string data using program-specific search operators. Softw. Test., Verif. Reliab. 16, 3 (2006), 175–203. DOI:https://doi.org/10.1002/stvr.v16:3 Google ScholarDigital Library
- Andrea Arcuri. 2018. EvoMaster: Evolutionary multi-context automated system test generation. In IEEE International Conference on Software Testing, Verification and Validation (ICST'18). IEEE.Google ScholarCross Ref
- Andrea Arcuri. 2018. An experience report on applying software testing academic results in industry: We need usable automated test generation. Empir. Softw. Eng. 23, 4 (2018), 1959–1981. Google ScholarDigital Library
- A. Arcuri. 2018. Test suite generation with the many independent objective (MIO) algorithm. Inf. Softw. Technol. 104 (2018), 195–206.Google ScholarCross Ref
- Andrea Arcuri. 2019. RESTful API automated test case generation with evomaster. ACM Trans. Softw. Eng. Methodol. 28, 1 (2019), 3. Google ScholarDigital Library
- Andrea Arcuri. 2020. Automated blackbox and whitebox testing of RESTful APIs with evomaster. IEEE Softw. 38, 3 (2020), 72–78.Google ScholarCross Ref
- A. Arcuri and L. Briand. 2014. A hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering. Softw. Test., Verif. Reliab. 24, 3 (2014), 219–250. Google ScholarDigital Library
- Andrea Arcuri and Juan P Galeotti. 2020. Handling SQL databases in automated system test generation. ACM Trans. Softw. Eng. Methodol. 29, 4 (2020), 1–31. Google ScholarDigital Library
- Andrea Arcuri and Juan P. Galeotti. 2020. Handling SQL databases in automated system test generation. ACM Trans. Softw. Eng. Methodol. 29, 4 (2020), 1–31. Google ScholarDigital Library
- Andrea Arcuri and Juan P. Galeotti. 2020. Testability transformations for existing APIs. In IEEE 13th International Conference on Software Testing, Validation and Verification (ICST'20). IEEE, 153–163.Google Scholar
- Andrea Arcuri, Juan Pablo Galeotti, Bogdan Marculescu, and Man Zhang. 2021. EvoMaster: A search-based system test generation tool. J. Open Source Softw. 6, 57 (2021), 2153.Google ScholarCross Ref
- Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. RESTler: Stateful REST API fuzzing. In 41st International Conference on Software Engineering (ICSE'19). IEEE Press, 748–758. DOI:https://doi.org/10.1109/ICSE.2019.00083 Google ScholarDigital Library
- Roberto Baldoni, Emilio Coppa, Daniele Cono D'Elia, Camil Demetrescu, and Irene Finocchi. 2018. A survey of symbolic execution techniques. ACM Comput. Surv. 51, 3 (2018), 1–39. https://dl.acm.org/doi/pdf/10.1145/3182657? casa_token=b1T2K3NHAsIAAAAA:IIJUpB57vPm9Ld2_gSX9qTFbeR2ti87KuvCpp2fLVIA7x6LoQzcvqMBILwvwVRJF 4dyib3KCOmkd4A. Google ScholarDigital Library
- A. Baresel, D. Binkley, M. Harman, and B. Korel. 2004. Evolutionary testing in the presence of loop-assigned flags: A testability transformation approach. In ACM International Symposium on Software Testing and Analysis (ISSTA'04). 108–118. Google ScholarDigital Library
- A. Baresel and H. Sthamer. 2003. Evolutionary testing of flag conditions. In Genetic and Evolutionary Computation Conference (GECCO'03). 2442–2454. Google ScholarDigital Library
- D. W. Binkley, M. Harman, and K. Lakhotia. 2011. FlagRemover: A testability transformation for transforming loop-assigned flags. ACM Trans. Softw. Eng. Methodol. 20, 3 (2011), 12:1–12:33. DOI:https://doi.org/10.1145/2000791.2000796 Google ScholarDigital Library
- Cristian Cadar and Koushik Sen. 2013. Symbolic execution for software testing: three decades later. Commun. ACM 56, 2 (2013), 82–90. Google ScholarDigital Library
- James Clause, Wanchun Li, and Alessandro Orso. 2007. Dytan: A generic dynamic taint analysis framework. In International Symposium on Software Testing and Analysis. 196–206. Google ScholarDigital Library
- H. Converse, O. Olivo, and S. Khurshid. 2017. Non-semantics-preserving transformations for higher-coverage test generation using symbolic execution. In IEEE International Conference on Software Testing, Verification and Validation (ICST'17). 241–252. DOI:https://doi.org/10.1109/ICST.2017.29Google Scholar
- Hamza Ed-douibi, Javier Luis Cànovas Izquierdo, and Jordi Cabot. 2018. Automatic generation of test cases for REST APIs: A Specification-based approach. In IEEE 22nd International Enterprise Distributed Object Computing Conference (EDOC'18). 181–190.Google Scholar
- Roy Thomas Fielding. 2000. Architectural Styles and the Design of Network-based Software Architectures. Ph.D. Dissertation. University of California, Irvine.Google ScholarDigital Library
- Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: Automatic test suite generation for object-oriented software. In ACM Symposium on the Foundations of Software Engineering (FSE'11). 416–419. Google ScholarDigital Library
- Gordon Fraser and Andrea Arcuri. 2013. Whole test suite generation. IEEE Trans. Softw. Eng. 39, 2 (2013), 276–291. Google ScholarDigital Library
- Juan Pablo Galeotti, Gordon Fraser, and Andrea Arcuri. 2014. Extending a search-based test generator with adaptive dynamic symbolic execution. In ACM International Symposium on Software Testing and Analysis (ISSTA'14). ACM, 421–424. Google ScholarDigital Library
- Matthew J. Gallagher and V. Lakshmi Narasimhan. 1997. Adtest: A test data generation suite for ADA software systems. IEEE Trans. Softw. Eng. 23, 8 (1997), 473–484. Google ScholarDigital Library
- Patrice Godefroid, Bo-Yuan Huang, and Marina Polishchuk. 2020. Intelligent REST API data fuzzing. In ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE'20). Association for Computing Machinery, New York, NY, 725–736. DOI:https://doi.org/10.1145/3368089.3409719 Google ScholarDigital Library
- D. Gong and X. Yao. 2012. Testability transformation based on equivalence of target statements. Neural Comput. Applic. 21, 8 (2012), 1871–1882. DOI:https://doi.org/10.1007/s00521-011-0568-8Google ScholarCross Ref
- Mark Harman. 2018. We need a testability transformation semantics. In International Conference on Software Engineering and Formal Methods. Springer, 3–17.Google ScholarCross Ref
- M. Harman, A. Baresel, D. W. Binkley, R. M. Hierons, L. Hu, B. Korel, P. McMinn, and M. Roper. 2008. Testability transformation–-program transformation to improve testability. In Formal Methods and Testing, An Outcome of the FORTEST Network, Revised Selected Papers. 320–344. DOI:https://doi.org/10.1007/978-3-540-78917-8_11 Google ScholarDigital Library
- Mark Harman, Lin Hu, Rob Hierons, Joachim Wegener, Harmen Sthamer, André Baresel, and Marc Roper. 2004. Testability transformation. IEEE Trans. Softw. Eng. 30, 1 (2004), 3–16. Google ScholarDigital Library
- Mark Harman, S. Afshin Mansouri, and Yuanyuan Zhang. 2012. Search-based software engineering: Trends, techniques and applications. ACM Comput. Surv. 45, 1 (2012), 11. Google ScholarDigital Library
- Wei Huang, Yao Dong, Ana Milanova, and Julian Dolby. 2015. Scalable and precise taint analysis for Android. In International Symposium on Software Testing and Analysis. 106–117. Google ScholarDigital Library
- Stefan Karlsson, Adnan Causevic, and Daniel Sundmark. 2020. QuickREST: Property-based test generation of openAPI described RESTful APIs. In IEEE International Conference on Software Testing, Verification and Validation (ICST'20). IEEE.Google ScholarCross Ref
- J. C. King. 1976. Symbolic execution and program testing. Commun. ACM 19, 7 (1976), 385–394. Google ScholarDigital Library
- B. Korel. 1990. Automated software test data generation. IEEE Trans. Softw. Eng. 16, 8 (1990), 870–879. Google ScholarDigital Library
- Y. Li and G. Fraser. 2011. Bytecode testability transformation. In 3rd International Symposium on Search Based Software Engineering (SSBSE'11). 237–251. DOI:https://doi.org/10.1007/978-3-642-23716-4_21 Google ScholarDigital Library
- Yun Lin, Jun Sun, Gordon Fraser, Ziheng Xiu, Ting Liu, and Jin Song Dong. 2020. Recovering fitness gradients for interprocedural Boolean flags in search-based testing. In 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 440–451. Google ScholarDigital Library
- Linghui Luo, Eric Bodden, and Johannes Späth. 2019. A qualitative analysis of Android taint-analysis results. In 34th IEEE/ACM International Conference on Automated Software Engineering (ASE'19). IEEE, 102–114. Google ScholarDigital Library
- Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2020. RESTest: Black-box constraint-based testing of RESTful web APIs. In International Conference on Service-oriented Computing.Google ScholarDigital Library
- Phil McMinn. 2009. Search-based failure discovery using testability transformations to generate pseudo-oracles. In Genetic and Evolutionary Computation Conference (GECCO'09). 1689–1696. DOI:https://doi.org/10.1145/1569901.1570127 Google ScholarDigital Library
- P. McMinn, D. Binkley, and M. Harman. 2009. Empirical evaluation of a nesting testability transformation for evolutionary testing. ACM Trans. Softw. Eng. Methodol. 18, 3 (2009), 11:1–11:27. DOI:DOI:https://doi.org/10.1145/1525880.1525884 Google ScholarDigital Library
- Sam Newman. 2015. Building Microservices. O'Reilly Media, Inc. Google ScholarDigital Library
- Felix Pauck, Eric Bodden, and Heike Wehrheim. 2018. Do Android taint analysis tools keep their promises? In 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 331–341. Google ScholarDigital Library
- R. V. Rajesh. 2016. Spring Microservices. Packt Publishing Ltd. Google ScholarDigital Library
- José Miguel Rojas, José Campos, Mattia Vivanti, Gordon Fraser, and Andrea Arcuri. 2015. Combining multiple coverage criteria in search-based unit test generation. In International Symposium on Search Based Software Engineering. Springer, 93–108.Google ScholarCross Ref
- José Miguel Rojas, Gordon Fraser, and Andrea Arcuri. 2016. Seeding strategies in search-based unit test generation. Softw. Test., Verif. Reliab. 26, 5 (2016), 366–401. Google ScholarDigital Library
- Edward J. Schwartz, Thanassis Avgerinos, and David Brumley. 2010. All you ever wanted to know about dynamic taint analysis and forward symbolic execution (but might have been afraid to ask). In IEEE Symposium on Security and Privacy. IEEE, 317–331. Google ScholarDigital Library
- Omer Tripp, Marco Pistoia, Stephen J. Fink, Manu Sridharan, and Omri Weisman. 2009. TAJ: Effective taint analysis of web applications. ACM SIGPLAN Not. 44, 6 (2009), 87–97. Google ScholarDigital Library
- Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2020. RESTTESTGEN: Automated black-box testing of RESTful APIs. In IEEE International Conference on Software Testing, Verification and Validation (ICST'20). IEEE.Google ScholarCross Ref
- Stefan Wappler, Joachim Wegener, and André Baresel. 2009. Evolutionary testing of software with function-assigned flags. J. Syst. Softw. 82, 11 (2009), 1767–1779. DOI:https://doi.org/10.1016/j.jss.2009.06.037 Google ScholarDigital Library
- Man Zhang, Bogdan Marculescu, and Andrea Arcuri. 2019. Resource-based test case generation for RESTful web services. In Genetic and Evolutionary Computation Conference. 1426–1434. Google ScholarDigital Library
Index Terms
- Enhancing Search-based Testing with Testability Transformations for Existing APIs
Recommendations
Advanced White-Box Heuristics for Search-Based Fuzzing of REST APIs
Due to its importance and widespread use in industry, automated testing of REST APIs has attracted major interest from the research community in the last few years. However, most of the work in the literature has been focused on black-box fuzzing. ...
SQL data generation to enhance search-based system testing
GECCO '19: Proceedings of the Genetic and Evolutionary Computation ConferenceAutomated system test generation for web/enterprise systems requires either a sequence of actions on a GUI (e.g., clicking on HTML links), or direct HTTP calls when dealing with web services (e.g., REST and SOAP). However, web/enterprise systems do ...
Search-based failure discovery using testability transformations to generate pseudo-oracles
GECCO '09: Proceedings of the 11th Annual conference on Genetic and evolutionary computationTestability transformations are source-to-source program transformations that are designed to improve the testability of a program. This paper introduces a novel approach in which transformations are used to improve testability of a program by ...
Comments