Andrea Arcuri
Juan P. Galeotti
Skip Abstract Section
Abstract
Automated system test generation for web/enterprise systems requires either a sequence of actions on a GUI (e.g., clicking on HTML links and form buttons) or direct HTTP calls when dealing with web services (e.g., REST and SOAP). When doing white-box testing of such systems, their code can be analyzed, and the same type of heuristics (e.g., the branch distance) used in search-based unit testing can be employed to improve performance. However, web/enterprise systems do often interact with a database. To obtain higher coverage and find new faults, the state of the databases needs to be taken into account when generating white-box tests. In this work, we present a novel heuristic to enhance search-based software testing of web/enterprise systems, which takes into account the state of the accessed databases. Furthermore, we enable the generation of SQL data directly from the test cases. This is useful when it is too difficult or time consuming to generate the right sequence of events to put the database in the right state. Also, it is useful when dealing with databases that are “read-only” for the system under test, and the actual data are generated by other services. We implemented our technique as an extension of EVOMASTER, where system tests are generated in the JUnit format. Experiments on six RESTful APIs (five open-source and one industrial) show that our novel techniques improve coverage significantly (up to +16.5%), finding seven new faults in those systems.
- [n.d.]. Antlr. Retrieved from https://www.antlr.org/.Google Scholar
- [n.d.]. EclipseLink. Retrieved from http://www.eclipse.org/eclipselink/.Google Scholar
- [n.d.]. Hibernate. Retrieved from http://hibernate.org.Google Scholar
- [n.d.]. JDBI. Retrieved from http://jdbi.org/.Google Scholar
- [n.d.]. JOOQ. Retrieved from https://www.jooq.org/.Google Scholar
- [n.d.]. MongoDB. Retrieved from https://www.mongodb.com/.Google Scholar
- [n.d.]. P6Spy. Retrieved from https://github.com/p6spy/p6spy.Google Scholar
- [n.d.]. RestAssured. Retrieved from https://github.com/rest-assured/rest-assured.Google Scholar
- [n.d.]. Spring Framework. Retrieved from https://spring.io.Google Scholar
- [n.d.]. SQL. Retrieved from https://www.iso.org/standard/63555.html.Google Scholar
- S. Ali, L. C. Briand, H. Hemmati, and R. K. Panesar-Walawege. 2010. A systematic review of the application and empirical investigation of search-based test-case generation. IEEE Trans. Softw. Eng. 36, 6 (2010), 742--762.Google ScholarDigital Library
- S. Ali, M. Z. Iqbal, A. Arcuri, and L. C. Briand. 2013. Generating test data from OCL constraints with search techniques. IEEE Trans. Softw. Eng. 39, 10 (2013), 1376--1402.Google ScholarDigital Library
- Mohammad Alshraideh and Leonardo Bottaci. 2006. Search-based software test data generation for string data using program-specific search operators. Softw. Test. Verif. Reliabil. 16, 3 (2006), 175--203.Google ScholarDigital Library
- Andrea Arcuri. 2011. A theoretical and empirical analysis of the role of test sequence length in software testing for structural coverage. IEEE Trans. Softw. Eng. 38, 3 (2011), 497--519.Google ScholarDigital Library
- A. Arcuri. 2018. EvoMaster: Evolutionary multi-context automated system test generation. In Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST’18). IEEE, 394--397.Google ScholarCross Ref
- A. Arcuri. 2018. Test suite generation with the many independent objective (MIO) algorithm. Information and Software Technology (IST) 104 (2018), 195--206.Google ScholarCross Ref
- Andrea Arcuri. 2019. RESTful API automated test case generation with EvoMaster. ACM Trans. Softw. Eng. Methodol. 28, 1 (2019), 3.Google ScholarDigital Library
- A. Arcuri and L. Briand. 2014. A Hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Softw. Test. Verif. Reliabil. 24, 3 (2014), 219--250.Google ScholarDigital Library
- A. Arcuri and G. Fraser. 2016. Java enterprise edition support in search-based junit test generation. In Proceedings of the International Symposium on Search Based Software Engineering (SSBSE’16). Springer, 3--17.Google Scholar
- Andrea Arcuri and Juan P. Galeotti. 2019. SQL data generation to enhance search-based system testing. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’19). ACM, 1390--1398.Google Scholar
- A. Baresel and H. Sthamer. 2003. Evolutionary testing of flag conditions. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’03). 2442--2454.Google Scholar
- C. Binnig, D. Kossmann, E. Lo, and M. T. Özsu. 2007. QAGen: Generating query-aware test databases. In Proceedings of the ACM SIGMOD International Conference on Management of Data. ACM, 341--352.Google ScholarDigital Library
- C. Cadar and K. Sen. 2013. Symbolic execution for software testing: Three decades later. Commun. ACM 56, 2 (2013), 82--90.Google ScholarDigital Library
- J. Castelein, M. Aniche, M. Soltani, A. Panichella, and A. van Deursen. 2018. Search-based test data generation for SQL queries. In Proceedings of the ACM/IEEE International Conference on Software Engineering (ICSE’18). ACM, 1230--1230.Google ScholarDigital Library
- M. Emmi, R. Majumdar, and K. Sen. 2007. Dynamic test input generation for database applications. In Proceedings of the ACM International Symposium on Software Testing and Analysis (ISSTA’07). 151--162.Google Scholar
- G. Fraser and A. Arcuri. 2011. EvoSuite: Automatic test suite generation for object-oriented software. In Proceedings of the ACM Symposium on the Foundations of Software Engineering (FSE’11). 416--419.Google Scholar
- Gordon Fraser and Andrea Arcuri. 2013. Handling test length bloat. Softw. Test. Verif. Reliabil. 23, 7 (2013), 553--582.Google ScholarCross Ref
- G. Fraser and A. Arcuri. 2013. Whole test suite generation. IEEE Trans. Softw. Eng. 39, 2 (2013), 276--291.Google ScholarDigital Library
- A. Fuchs and H. Kuchen. 2017. Unit testing of database-driven Java enterprise edition applications. In Proceedings of the International Conference on Tests and Proofs. Springer, 59--76.Google Scholar
- Maryam Abdul Ghafoor, Muhammad Suleman Mahmood, and Junaid Haroon Siddiqui. 2020. Extending symbolic execution for automated testing of stored procedures. Softw. Qual. J. 28 (2020), 853--887. https://link.springer.com/article/10.1007/s11219-019-09453-6.Google ScholarCross Ref
- James Gosling, Bill Joy, Guy L. Steele, Gilad Bracha, and Alex Buckley. 2014. The Java Language Specification, Java SE 8 Edition (1st ed.). Addison-Wesley Professional.Google ScholarDigital Library
- Mark Harman, Lin Hu, Rob Hierons, Joachim Wegener, Harmen Sthamer, André Baresel, and Marc Roper. 2004. Testability transformation. IEEE Trans. Softw. Eng. 30, 1 (2004), 3--16.Google ScholarDigital Library
- M. Harman, S. A. Mansouri, and Y. Zhang. 2012. Search-based software engineering: Trends, techniques and applications. ACM Comput. Surv. 45, 1 (2012), 11.Google ScholarDigital Library
- S. Khalek, B. Elkarablieh, Y. Laleye, and S. Khurshid. 2008. Query-aware test generation using a relational constraint solver. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering (ASE’08). IEEE, 238--247.Google Scholar
- A. Kieyzun, P. J. Guo, K. Jayaraman, and M. D. Ernst. 2009. Automatic creation of SQL injection and cross-site scripting attacks. In Proceedings of the ACM/IEEE International Conference on Software Engineering (ICSE’09). IEEE Computer Society, 199--209.Google Scholar
- B. Korel. 1990. Automated software test data generation. IEEE Trans. Softw. Eng. 16, 8 (1990), 870--879.Google ScholarDigital Library
- William B. Langdon and Riccardo Poli. 1998. Fitness causes bloat. In Soft Computing in Engineering Design and Manufacturing. Springer, 13--22.Google Scholar
- Y. Li and G. Fraser. 2011. Bytecode testability transformation. In Proceedings of the 3rd International Symposium on Search Based Software Engineering (SSBSE’11), Lecture Notes in Computer Science, Myra B. Cohen and Mel Ó. Cinnéide (Eds.), Vol. 6956. Springer, 237--251. DOI:https://doi.org/10.1007/978-3-642-23716-4_21Google Scholar
- K. Mao, M. Harman, and Y. Jia. 2016. Sapienz: Multi-objective automated testing for android applications. In Proceedings of the ACM International Symposium on Software Testing and Analysis (ISSTA’16). ACM, 94--105.Google Scholar
- Phil McMinn. 2004. Search-based software test data generation: A survey. Softw. Test. Verif. Reliabil. 14, 2 (2004), 105--156. DOI:https://doi.org/10.1002/stvr.294Google ScholarDigital Library
- P. Mcminn, C. J. Wright, and G. M. Kapfhammer. 2015. The effectiveness of test coverage criteria for relational database schema integrity constraints. ACM Trans. Softw. Eng. Methodol. 25, 1 (2015), 8.Google ScholarDigital Library
- P. McMinn, C. J. Wright, C. Kinneer, C. J. McCurdy, M. Camara, and G. M. Kapfhammer. 2016. SchemaAnalyst: Search-based test data generation for relational database schemas. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME’16). IEEE, 586--590.Google Scholar
- A. Mesbah, A. Van Deursen, and S. Lenselink. 2012. Crawling Ajax-based web applications through dynamic analysis of user interface state changes. ACM Trans. Web 6, 1 (2012), 3.Google ScholarDigital Library
- S. Newman. 2015. Building Microservices. O’Reilly Media, Inc.Google ScholarDigital Library
- A. Panichella, F. Kifetew, and P. Tonella. 2018. Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Trans. Softw. Eng. 44, 2 (2018), 122--158.Google ScholarCross Ref
- J. M. Rojas, J. Campos, M. Vivanti, G. Fraser, and A. Arcuri. 2015. Combining multiple coverage criteria in search-based unit test generation. In Proceedings of the International Symposium on Search Based Software Engineering (SSBSE’15). Springer, 93--108.Google Scholar
- José Miguel Rojas, Gordon Fraser, and Andrea Arcuri. 2016. Seeding strategies in search-based unit test generation. Softw. Test. Verif. Reliabil. 26, 5 (2016), 366--401.Google ScholarDigital Library
- J. Thomé, A. Gorla, and A. Zeller. 2014. Search-based security testing of web applications. In Proceedings of the International Workshop on Search-Based Software Testing (SBST’14). ACM, 5--14.Google Scholar
- J. Tuya, M. J. Suárez-Cabal, and C. De La Riva. 2010. Full predicate coverage for testing SQL database queries. Softw. Test. Verif. Reliabil. 20, 3 (2010), 237--288.Google ScholarDigital Library
Index Terms
- Handling SQL Databases in Automated System Test Generation
Recommendations
Enhancing Search-based Testing with Testability Transformations for Existing APIs
Search-based software testing (SBST) has been shown to be an effective technique to generate test cases automatically. Its effectiveness strongly depends on the guidance of the fitness function. Unfortunately, a common issue in SBST is the so-called flag ...
SQL data generation to enhance search-based system testing
GECCO '19: Proceedings of the Genetic and Evolutionary Computation ConferenceAutomated system test generation for web/enterprise systems requires either a sequence of actions on a GUI (e.g., clicking on HTML links), or direct HTTP calls when dealing with web services (e.g., REST and SOAP). However, web/enterprise systems do ...
The Risks of Coverage-Directed Test Case Generation
A number of structural coverage criteria have been proposed to measure the adequacy of testing efforts. In the avionics and other critical systems domains, test suites satisfying structural coverage criteria are mandated by standards. With the advent of ...
Comments