Skip to main content
SpringerLink
Log in

Automated Requirements-Based Testing of Black-Box Reactive Systems

  • Conference paper
  • First Online:
NASA Formal Methods (NFM 2020)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 12229))

Included in the following conference series:

Abstract

We present a new approach to conformance testing of black-box reactive systems. We consider system specifications written as linear temporal logic formulas to generate tests as sequences of input/output pairs: inputs are extracted from the Büchi automata corresponding to the specifications, and outputs are obtained by feeding the inputs to the systems. Conformance is checked by comparing input/output sequences with automata traces to detect violations of the specifications. We consider several criteria for extracting tests and for stopping generation, and we compare them experimentally using both indicators of coverage and error-detection. The results show that our methodology can generate test suites with good system coverage and error-detection capability.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    http://www.syntcomp.org/.

  2. 2.

    Using the command line ltl2tgba -B -f “p0 \(-\) (X G p1 | ! F p1)”.

  3. 3.

    Fired with command line ltl2tgba -MD -f “p0 \(-\) (X G p1 | ! F p1)”.

  4. 4.

    All benchmarks are available at https://gitlab.sagelab.it/sage/benchmarks-tests.

  5. 5.

    http://www.syntcomp.org/.

  6. 6.

    http://cerbero-h2020.eu.

  7. 7.

    http://www.trossenrobotics.com/widowxrobotarm.

  8. 8.

    https://gitlab.sagelab.it/sage/SpecPro.

References

  1. Andrews, J.H., Briand, L.C., Labiche, Y., Namin, A.S.: Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Trans. Softw. Eng. 32(8), 608–624 (2006)

    Article  Google Scholar 

  2. Aniculaesei, A., Howar, F., Denecke, P., Rausch, A.: Automated generation of requirements-based test cases for an adaptive cruise control system. In: 2018 IEEE Workshop on Validation, Analysis and Evolution of Software Tests (VST), pp. 11–15. IEEE (2018)

    Google Scholar 

  3. Arcaini, P., Gargantini, A., Riccobene, E.: Online testing of LTL properties for java code. In: Bertacco, V., Legay, A. (eds.) HVC 2013. LNCS, vol. 8244, pp. 95–111. Springer, Cham (2013). https://doi.org/10.1007/978-3-319-03077-7_7

    Chapter  Google Scholar 

  4. Baier, C., Katoen, J.P.: Principles of Model Checking. MIT press, Cambridge (2008)

    MATH  Google Scholar 

  5. Barbot, B., Basset, N., Dang, T.: Generation of signals under temporal constraints for CPS testing. In: Badger, J.M., Rozier, K.Y. (eds.) NFM 2019. LNCS, vol. 11460, pp. 54–70. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20652-9_4

    Chapter  Google Scholar 

  6. Bartocci, E., Bloem, R., Nickovic, D., Roeck, F.: A counting semantics for monitoring LTL specifications over finite traces. In: Chockler, H., Weissenbacher, G. (eds.) CAV 2018. LNCS, vol. 10981, pp. 547–564. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-96145-3_29

    Chapter  Google Scholar 

  7. Bauer, A., Leucker, M., Schallhart, C.: Monitoring of real-time properties. In: Arun-Kumar, S., Garg, N. (eds.) FSTTCS 2006. LNCS, vol. 4337, pp. 260–272. Springer, Heidelberg (2006). https://doi.org/10.1007/11944836_25

    Chapter  MATH  Google Scholar 

  8. Bauer, A., Leucker, M., Schallhart, C.: Comparing LTL semantics for runtime verification. J. Logic Comput. 20(3), 651–674 (2010)

    Article  MathSciNet  Google Scholar 

  9. Bauer, A., Leucker, M., Schallhart, C.: Runtime verification for LTL and TLTL. ACM Trans. Softw. Eng. Methodol. (TOSEM) 20(4), 14 (2011)

    Article  Google Scholar 

  10. Bernot, G., Gaudel, M.C., Marre, B.: Software testing based on formal specifications: a theory and a tool. Softw. Eng. J. 6(6), 387–405 (1991)

    Article  Google Scholar 

  11. Burnstein, I.: Practical Software Testing: A Process-oriented Approach. Springer, Heidelberg (2006)

    MATH  Google Scholar 

  12. Cimatti, A., Clarke, E., Giunchiglia, E., Giunchiglia, F., Pistore, M., Roveri, M., Sebastiani, R., Tacchella, A.: NuSMV 2: an opensource tool for symbolic model checking. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, pp. 359–364. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45657-0_29

    Chapter  Google Scholar 

  13. Duret-Lutz, A., Lewkowicz, A., Fauchille, A., Michaud, T., Renault, É., Xu, L.: Spot 2.0—a framework for LTL and \(\omega \)-automata manipulation. In: Artho, C., Legay, A., Peled, D. (eds.) ATVA 2016. LNCS, vol. 9938, pp. 122–129. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46520-3_8

    Chapter  Google Scholar 

  14. Eisner, C., Fisman, D., Havlicek, J., Lustig, Y., McIsaac, A., Van Campenhout, D.: Reasoning with temporal logic on truncated paths. In: Hunt, W.A., Somenzi, F. (eds.) CAV 2003. LNCS, vol. 2725, pp. 27–39. Springer, Heidelberg (2003). https://doi.org/10.1007/978-3-540-45069-6_3

    Chapter  Google Scholar 

  15. Hsueh, M.C., Tsai, T.K., Iyer, R.K.: Fault injection techniques and tools. Computer 30(4), 75–82 (1997)

    Article  Google Scholar 

  16. Jacobs, S., Klein, F., Schirmer, S.: A high-level LTL synthesis format: Tlsf v1. 1. arXiv preprint arXiv:1604.02284 (2016)

  17. Jard, C., Jéron, T.: TGV: theory, principles and algorithms. Int. J. Softw. Tools Technol. Transfer 7(4), 297–315 (2005)

    Article  Google Scholar 

  18. Koch, B., Grabowski, J., Hogrefe, D., Schmitt, M.: Autolink-a tool for automatic test generation from SDL specifications. In: Proceedings. 2nd IEEE Workshop on Industrial Strength Formal Specification Techniques, pp. 114–125. IEEE (1998)

    Google Scholar 

  19. Krichen, M., Tripakis, S.: Black-box conformance testing for real-time systems. In: Graf, S., Mounier, L. (eds.) SPIN 2004. LNCS, vol. 2989, pp. 109–126. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24732-6_8

    Chapter  Google Scholar 

  20. Li, J., Conradi, R., Bunse, C., Torchiano, M., Slyngstad, O.P.N., Morisio, M.: Development with off-the-shelf components: 10 facts. IEEE Softw. 26(2), 80–87 (2009)

    Article  Google Scholar 

  21. Manna, Z., Pnueli, A.: Temporal Verification of Reactive Systems: Safety. Springer, Heidelberg (2012). https://doi.org/10.1007/978-1-4612-4222-2

    Book  MATH  Google Scholar 

  22. Masin, M., Palumbo, F., Myrhaug, H., de Oliveira Filho, J., Pastena, M., Pelcat, M., Raffo, L., Regazzoni, F., Sanchez, A., Toffetti, A., et al.: Cross-layer design of reconfigurable cyber-physical systems. In: Proceedings of the Conference on Design, Automation & Test in Europe, pp. 740–745. European Design and Automation Association (2017)

    Google Scholar 

  23. Luttenberger, M., Meyer, P.J., Sickert, S.: Strix (2018). https://strix.model.in.tum.de/. Accessed 27 June 2019

  24. Palumbo, F., et al.: CERBERO: cross-layer model-based framework for multi-objective design of reconfigurable systems in uncertain hybrid environments: Invited paper: CERBERO teams from UniSS, UniCA, IBM research, TASE, INSA-Rennes, UPM, USI, Abinsula, Ambiesense, TNO, S&T, CRF. In: Proceedings of the 16th ACM International Conference on Computing Frontiers, pp. 320–325. ACM (2019)

    Google Scholar 

  25. Pnueli, A.: The temporal logic of programs. In: 18th Annual Symposium on Foundations of Computer Science, 1977, pp. 46–57. IEEE (1977)

    Google Scholar 

  26. Schmitt, M., Ebner, M., Grabowski, J.: Test generation with autolink and test composer. In: Proceedings of 2nd Workshop of the SDL Forum Society on SDL and MSC-SAM, vol. 2000 (2000)

    Google Scholar 

  27. Tan, L., Sokolsky, O., Lee, I.: Specification-based testing with linear temporal logic. In: Conference on Information Reuse and Integration, pp. 483–498 (2004)

    Google Scholar 

  28. Utting, M., Legeard, B.: Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann Publishers Inc., San Francisco (2007)

    Google Scholar 

  29. Zeng, B., Tan, L.: Test reactive systems with Büchi automata: acceptance condition coverage criteria and performance evaluation. In: 2015 IEEE International Conference on Information Reuse and Integration, pp. 380–387. IEEE (2015)

    Google Scholar 

  30. Zeng, B., Tan, L.: Test reactive systems with Büchi-automaton-based temporal requirements. In: Bouabana-Tebibel, T., Rubin, S.H. (eds.) Theoretical Information Reuse and Integration. AISC, vol. 446, pp. 31–57. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-31311-5_2

    Chapter  Google Scholar 

Download references

Acknowledgments

The research of Luca Pulina and Simone Vuotto is part of the FitOptiVis project funded by the ECSEL Joint Undertaking under grant number H2020-ECSEL-2017-2-783162. The research of Luca Pulina has been also partially funded by the ECSEL JU Project COMP4DRONES and the Sardinian Regional Projects PROSSIMO (POR FESR Sardegna 2014/20-ASSE I).

Author information

Authors and Affiliations

  1. DIBRIS, University of Genoa, Viale Causa 13, 16145, Genoa, Italy

    Massimo Narizzano, Armando Tacchella & Simone Vuotto

  2. Chemistry and Pharmacy Department, University of Sassari, Via Vienna 2, Sassari, Italy

    Luca Pulina & Simone Vuotto

Authors
  1. Massimo Narizzano
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luca Pulina
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Armando Tacchella
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Simone Vuotto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Massimo Narizzano .

Editor information

Editors and Affiliations

  1. NASA Ames Research Center, Moffett Field, CA, USA

    Ritchie Lee

  2. SRI International, Menlo Park, CA, USA

    Susmit Jha

  3. KBR Inc., NASA Ames Research Center, Moffett Field, CA, USA

    Anastasia Mavridou

  4. NASA Ames Research Center, Moffett Field, CA, USA

    Dimitra Giannakopoulou

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Narizzano, M., Pulina, L., Tacchella, A., Vuotto, S. (2020). Automated Requirements-Based Testing of Black-Box Reactive Systems. In: Lee, R., Jha, S., Mavridou, A., Giannakopoulou, D. (eds) NASA Formal Methods. NFM 2020. Lecture Notes in Computer Science(), vol 12229. Springer, Cham. https://doi.org/10.1007/978-3-030-55754-6_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-55754-6_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-55753-9

  • Online ISBN: 978-3-030-55754-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation