Hong Zhu
Ian Bayley
ABSTRACT
Testing has been widely recognised as difficult for AI applications. This paper proposes a set of testing strategies for testing machine learning applications in the framework of the datamorphism testing methodology. In these strategies, testing aims at exploring the data space of a classification or clustering application to discover the boundaries between classes that the machine learning application defines. This enables the tester to understand precisely the behaviour and function of the software under test. In the paper, three variants of exploratory strategies are presented with the algorithms as implemented in the automated datamorphic testing tool Morphy. The correctness of these algorithms are formally proved. The paper also reports the results of some controlled experiments with Morphy that study the factors that affect the test effectiveness of the strategies.
- C. Aggarwal. 2015. Data Mining: The Textbook. Springer.Google Scholar
Digital Library
- X. Bai, J. Li, and A. Ulrich (Eds.). 2018. Proc. of IEEE/ACM 13th International Workshop on Automation of Software Test (AST 2018). IEEE Computer Society, Gothenburg, Sweden.Google ScholarDigital Library
- T. Y. Chen, F-C. Kuo, H. Liu, P-L. Poon, D. Towey, T. H. Tse, and Z. Q. Zhou. 2018. Metamorphic testing: A review of challenges and opportunities. ACM Comput. Surv. 51, 1, Article 4 (Jan 2018), 27 pages. https://doi.org/10.1145/3143561Google Scholar
- A. Gotlieb, M. Roper, and P Zhang (Eds.). 2019. Proc. of The First IEEE International Conference on Artificial Intelligence Testing (AITest 2019). IEEE Computer Society, Los Alamitos, CA, USA. https://doi.org/10.1109/AITest.2019Google Scholar
- M. Mohri, A. Rostamizadeh, and A. Talwalkar. 2012. Foundations of Machine Learning. The MIT Press.Google Scholar
- M. Roper and Z. Q. Zhou (Eds.). 2020. Proc. of The Second IEEE International Conference on Artificial Intelligence Testing (AITest 2020). IEEE Computer Society, Los Alamitos, CA, USA. (In Press) pages.Google Scholar
- S. Segura, D. Towey, Z. Q. Zhou, and T. Y. Chen. 2018. Metamorphic testing: testing the untestable. IEEE Software (2018), 1--1. https://doi.org/10.1109/MS.2018.2875968Google Scholar
- S. Shalev-Shwartz and S. Ben-David. 2014. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press.Google Scholar
- L. Shan and H. Zhu. 2009. Generating structurally complex test cases by data mutation: A case study of testing an automated modelling tool. Comput. J. 52, 5 (Aug 2009), 571--588.Google ScholarDigital Library
- M. Sutton, A. Greene, and P. Amini. 2007. Fuzzing: Brute Force Vulnerability Discovery. Addison-Wesley.Google ScholarDigital Library
- Y. Tian, K. Pei, S. Jana, and B Ray. 2018. DeepTest: Automated testing of deep-neural-network-driven autonomous cars. In Proc. of The 40th IEEE/ACM Int'l Conf. on Software Engineering (ICSE 2018). IEEE Computer Society, Gothenburg, Sweden, 303--314.Google ScholarDigital Library
- J. A. Whittaker. 2009. Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design. Pearson Education. https://books.google.co.uk/books?id=BsB0NpkcdgICGoogle Scholar
- X. Xie, J. W. K. Ho, C. Murphy, G. Kaiser, B. Xu, and T. Y. Chen. 2011. Testing and validating machine learning classifiers by metamorphic testing. Journal of Systems and Software 84 (2011), 544--558.Google ScholarDigital Library
- S. Yang, D. Towey, and Z. Zhou. 2019. Metamorphic exploration of an unsupervised clustering program. In Proc. of IEEE/ACM 4th International Workshop on Metamorphic Testing (MET 2019). IEEE Computer Society, 48--54.Google Scholar
- Z. Q. Zhou and L. Sun. 2019. Metamorphic testing of driverless cars. Commun. ACM 62, 3 (March 2019), 61--67.Google ScholarDigital Library
- H. Zhu. 2015. JFuzz: A tool for automated Java unit testing based on data mutation and metamorphic testing methods. In Proc. of The 2nd Int'l Conf. on Trustworthy Systems and Their Applications (TSA 2015). 8--15.Google ScholarDigital Library
- H. Zhu, I. Bayley, D. Liu, and X. Zheng. 2019. Morphy: A Datamorphic Software Test Automation Tool. Technical Report OBU-ECM-AFM-2019-01. School of Engineering, Computing and Mathematics, Oxford Brookes University, Oxford, UK. http://arxiv.org/abs/1912.09881Google Scholar
- H. Zhu, I. Bayley, D. Liu, and X. Zheng. 2020. Automation of Datamorphic Testing. In Proc. of 2nd IEEE International Conference on Artificial Intelligence Testing (AITest 2020). In Press.Google Scholar
- H. Zhu, P. Hall, and J. May. 1997. Software unit test coverage and adequacy. ACM Computing Survey 29, 4 (Dec. 1997), 366--427.Google ScholarDigital Library
- H. Zhu, D. Liu, I. Bayley, R. Harrison, and F. Cuzzolin. 2018. Datamorphic Testing: A Methodology for Testing AI Applications. Technical Report OBU-ECM-AFM-2018-02. School of Engineering, Computing and Mathematics, Oxford Brookes University, Oxford OX33 1HX, UK. http://arxiv.org/abs/1912.04900Google Scholar
- H. Zhu, D. Liu, I. Bayley, R. Harrison, and F. Cuzzolin. 2019. Datamorphic Testing: A Method for Testing Intelligent Applications. In Proc. of The First IEEE International Conference on Artificial Intelligence Testing (AITest 2019). IEEE Computer Society, Los Alamitos, CA, USA, 149--156. https://doi.org/10.1109/AITest.2019.00018Google ScholarCross Ref
Index Terms
- Exploratory Datamorphic Testing of Classification Applications
Recommendations
Discovering boundary values of feature-based machine learning classifiers through exploratory datamorphic testing
AbstractTesting has been widely recognised as difficult for AI applications. This paper proposes a set of testing strategies for testing machine learning applications in the framework of the datamorphism testing methodology. In these ...
Gamification of exploratory testing process
Gamify 2022: Proceedings of the 1st International Workshop on Gamification of Software Development, Verification, and ValidationExploratory testing is an experience-based testing technique often used in addition to formal ones. However, it can be used as a replacement for formal techniques when time is limited and/or documentation is poor. Conducting test charters and ...
The Role of the Tester's Knowledge in Exploratory Software Testing
We present a field study on how testers use knowledge while performing exploratory software testing (ET) in industrial settings. We video recorded 12 testing sessions in four industrial organizations, having our subjects think aloud while performing ...
Comments