Riccardo Coppola
Marco Torchiano
ABSTRACT
Context: Albeit different approaches exist for automated GUI testing of hybrid mobile applications, the practice appears to be not so commonly adopted by developers. A possible reason for such a low diffusion can be the fragility of the techniques, i.e. the frequent need for maintaining test cases when the GUI of the app is changed.
Goal: In this paper, we perform an assessment of the maintenance needed by test cases for a hybrid mobile app, and the related fragility causes.
Methods: We evaluated a small test suite with a Layout-based testing tool (Appium) and a Visual one (EyeAutomate) and observed the changes needed by tests during the co-evolution with the GUI of the app.
Results: We found that 20% Layout-based test methods and 30% Visual test methods had to be modified at least once, and that each release induced fragilities in 3-4% of the test methods.
Conclusion: Fragility of GUI tests can induce relevant maintenance efforts in test suites of large applications. Several principal causes for fragilities have been identified for the tested hybrid application, and guidelines for developers are deduced from them.
- Emil Alégroth, Zebao Gao, Rafael Oliveira, and Atif Memon. 2015. Conceptualization and evaluation of component-based testing unified with visual gui testing: an empirical study. In Software Testing, Verification and Validation (ICST), 2015 IEEE 8th International Conference on. IEEE, 1–10.Google Scholar
Cross Ref
- Emil Alégroth, Arvid Karlsson, and Alexander Radway. 2018. Continuous Integration and Visual GUI Testing: Benefits and Drawbacks in Industrial Practice. In Software Testing, Verification and Validation (ICST), 2018 IEEE 11th International Conference on. IEEE, 172–181.Google ScholarCross Ref
- Luca Ardito, Riccardo Coppola, Maurizio Morisio, and Marco Torchiano. 2019. Espresso vs. EyeAutomate: An Experiment for the Comparison of Two Generations of Android GUI Testing. In Proceedings of the Evaluation and Assessment on Software Engineering (EASE ’19). ACM, New York, NY, USA, 13–22. Google ScholarDigital Library
- L. Ardito, R. Coppola, M. Torchiano, and E. Alegroth. 2018. Towards Automated Translation between Generations of GUI-based Tests for Mobile Devices. In Proceedings of INTUITESTBEDS 2018, joint Workshop of the 4th International Workshop on User Interface Test Automation, and 8th Workshop on TESting Techniques for event BasED Software. ACM. Google ScholarDigital Library
- Stefan Bosnic, Ištvan Papp, and Sebastian Novak. 2016. The development of hybrid mobile applications with Apache Cordova. In 2016 24th Telecommunications Forum (TELFOR). IEEE, 1–4.Google Scholar
- Andreas Bruns, Andreas Kornstadt, and Dennis Wichmann. 2009. Web application tests with selenium. IEEE software 26, 5 (2009). Google ScholarDigital Library
- Victor R Basili-Gianluigi Caldiera and H Dieter Rombach. 1994. Goal question metric paradigm. Encyclopedia of software engineering 1 (1994), 528–532.Google Scholar
- Claes Claes, Wohlin, R Runeson, Per, H Höst, Martin, CO Ohlsson, Magnus, R Regnell, Björn, and Anders Wesslén. 2012. Experimentation in software engineering. Springer.Google Scholar
- Riccardo Coppola, Maurizio Morisio, and Marco Torchiano. 2017. Scripted GUI Testing of Android Apps: A Study on Diffusion, Evolution and Fragility. In Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering. ACM, 22–32. Google ScholarDigital Library
- R. Coppola, M. Morisio, and M. Torchiano. 2018. Maintenance of Android Widget-Based GUI Testing: A Taxonomy of Test Case Modification Causes. In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 151–158.Google Scholar
- R. Coppola, M. Morisio, and M. Torchiano. 2018. Mobile GUI Testing Fragility: A Study on Open-Source Android Applications. IEEE Transactions on Reliability (2018), 1–24.Google Scholar
- Riccardo Coppola, Maurizio Morisio, Marco Torchiano, and Luca Ardito. 2019. Scripted GUI testing of Android open-source apps: evolution of test code and fragility causes. Empirical Software Engineering (18 May 2019).Google Scholar
- Isabelle Dalmasso, Soumya Kanti Datta, Christian Bonnet, and Navid Nikaein. 2013. Survey, comparison and evaluation of cross platform mobile application development tools. In 2013 9th International Wireless Communications and Mobile Computing Conference (IWCMC). IEEE, 323–328.Google ScholarCross Ref
- Ronald Jabangwe, Henry Edison, and Anh Nguyen Duc. 2018. Software engineering process models for mobile app development: a systematic literature review. Journal of Systems and Software 145 (2018), 98–111.Google ScholarCross Ref
- Mona Erfani Joorabchi, Ali Mesbah, and Philippe Kruchten. 2013. Real challenges in mobile app development. In Empirical Software Engineering and Measurement, 2013 ACM/IEEE International Symposium on. IEEE, 15–24.Google ScholarCross Ref
- P. S. Kochhar, F. Thung, N. Nagappan, T. Zimmermann, and D. Lo. 2015. Understanding the Test Automation Culture of App Developers. In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST). 1–10.Google Scholar
- Maurizio Leotta, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2018. Pesto: Automated migration of DOM-based Web tests towards the visual approach. Software Testing, Verification And Reliability 28, 4 (2018), e1665.Google ScholarCross Ref
- Mario Linares-Vásquez, Kevin Moran, and Denys Poshyvanyk. 2017. Continuous, evolutionary and large-scale: A new perspective for automated mobile app testing. In Software Maintenance and Evolution (ICSME), 2017 IEEE International Conference on. IEEE, 399–410.Google ScholarCross Ref
- Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An empirical analysis of flaky tests. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 643–653. Google ScholarDigital Library
- I. Malavolta, S. Ruberto, T. Soru, and V. Terragni. 2015. End Users’ Perception of Hybrid Mobile Apps in the Google Play Store. In 2015 IEEE International Conference on Mobile Services. 25–32. Google ScholarDigital Library
- Gaurang Shah, Prayag Shah, and Rishikesh Muchhala. 2014. Software testing automation using appium. International Journal of Current Engineering and Technology 4, 5 (2014), 3528–3531.Google Scholar
Index Terms
- Fragility of layout-based and visual GUI test scripts: an assessment study on a hybrid mobile application
Recommendations
Scripted GUI Testing of Android Apps: A Study on Diffusion, Evolution and Fragility
PROMISE: Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software EngineeringBackground. Evidence suggests that mobile applications are not thoroughly tested as their desktop counterparts. In particular GUI testing is generally limited. Like web-based applications, mobile apps suffer from GUI test fragility, i.e. GUI test ...
Scripted GUI testing of Android open-source apps: evolution of test code and fragility causes
AbstractEvidence from empirical studies suggests that mobile applications are not thoroughly tested as their desktop counterparts. In particular, GUI testing is generally limited. Like web-based applications, mobile apps suffer from GUI testing fragility, ...
Multi-objective test case prioritization for GUI applications
SAC '13: Proceedings of the 28th Annual ACM Symposium on Applied ComputingTest case prioritization techniques are proposed to schedule execution of test cases in order to improve testing effectiveness. Various coverage criteria are used as surrogates for test case prioritization. They are expected to improve testing ...
Comments