Skip to main content
SpringerLink
Log in

Structural Specification-Based Testing: Automated Support and Experimental Evaluation

  • Conference paper
  • First Online:
Software Engineering — ESEC/FSE ’99 (ESEC 1999, SIGSOFT FSE 1999)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 1687))

Abstract

In this paper, we describe a testing technique, called structural specification-based testing (SST), which utilizes the formal specification of a program unit as the basis for test selection and test coverage measurement. We also describe an automated testing tool, called ADLscope, which supports SST for program units specified in Sun Microsystems’ Assertion Definition Language (ADL). ADLscope automatically generates coverage conditions from a program’s ADL specification. While the program is tested, ADLscope determines which of these conditions are covered by the tests. An uncovered condition exhibits aspects of the specification inadequately exercised during testing. The tester uses this information to develop new test data to exercise the uncovered conditions.

We provide an overview of SST’s specification-based test criteria and describe the design and implementation of ADLscope. Specification-based testing is guided by a specification, whereby the testing activity is directly related to what a component under test is supposed to do, rather than what it actually does. Specification-based testing is a significant advance in testing, because it is often more straightforward to accomplish and it can reveal failures that are often missed by traditional code-based testing techniques. As an initial evaluation of the capabilities of specification-based testing, we conducted an experiment to measure defect detection capabilities, code coverage and usability of SST/ADLscope; we report here on the results.

This research was sponsored in part by the Air Force Material Command, Rome Laboratory, and the Defense Advanced Research Projects Agency under agreement number #F30602-97-2-0033; and Sun Microsystems Inc. and the University of California under MICRO grants #95-124 and #96-002. The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official position or policy, either expressed or implied, of the U.S. Government, AFMC, Rome Laboratory, DARPA, or the University of California, and no official endorsement should be inferred.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. A. T. Acree, T. A. Budd, R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Mutation analysis. Technical Report TR GIT-ICS-79/08, Georgia Institute of Technology, September 1979.

    Google Scholar 

  2. Bender and Associates. http://www.softtest.com.

  3. T. A. Budd, R. J. Lipton, R. A. DeMillo, and F.G. Sayward. Mutation analysis. Department Comput. Sci. Res. Rep. 155, Yale University, New Haven, Connecticut, April 1979.

    Google Scholar 

  4. J. Chang. Automating specification-based test coverage measurement and test selection. PhD thesis, University of California, Irvine, March 1999.

    Google Scholar 

  5. J. Chang, D. J. Richardson. ADLscope: an automated specification-based unit testing tool. ICS Technical Report 98-26, Department of Information and Computer Science, University of California, Irvine, California, 1998.

    Google Scholar 

  6. J. Chang, D. J. Richardson, and S. Sankar. Structural specification-based testing with ADL. In Proceedings of the 1996 International Symposium on Software Testing and Analysis (ISSTA’96), pp. 62–70, San Diego, California, January 1996.

    Google Scholar 

  7. R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on test data selection: help for the practicing programmer. Computer, 4(11), April 1978.

    Google Scholar 

  8. R. K. Doong and P. G. Frankl. The ASTOOT approach to testing object-oriented programs. ACM Transactions on Software Engineering and Methodology, April 1994, pp. 101–130.

    Google Scholar 

  9. K. A. Foster. Error sensitive test case analysis (ESTCA). IEEE Transactions on Software Engineering, 6(3):258–264, May 1980.

    Article  Google Scholar 

  10. K. A. Foster. Sensitive test data for logic expressions. ACM SIGSOFT Software Engineering Notes, vol. 9, no. 2, pp. 120–126, April 1984.

    Article  Google Scholar 

  11. J. B. Goodenough and S. L. Gerhart. Toward a theory of test data selection. IEEE Transactions on Software Engineering, SE-1(2):156–173, June 1975.

    Article  MathSciNet  Google Scholar 

  12. James Gosling, Bill Joy, and Guy Steele. The Java Language Specification, Addison Wesley, 1996.

    Google Scholar 

  13. R. G. Hamlet. Testing programs with the aid of a compiler. IEEE Transactions on Software Engineering, SE-3(4):279–290, July 1977.

    Article  MathSciNet  Google Scholar 

  14. W. E. Howden. Weak mutation testing and completeness of test sets. IEEE Transactions on Software Engineering, SE-8(4):371–379, July 1982.

    Article  Google Scholar 

  15. M. Hughes and D. Stotts. Daistish: systematic algebraic testing for OO programs in the presence of side-effects. In Proceedings of the 1996 International Symposium on Software Testing and Analysis (ISSTA), pp.53–61, San Diego, California, January 1996.

    Google Scholar 

  16. P. Jalote. Specification and testing of abstract data types. Comput. Lang. Vol. 17, pp. 75–82, 1992.

    Article  Google Scholar 

  17. David C. Luckham and Friedrich W. von Henke. An overview of ANNA, a specification language for Ada. IEEE Software, 2(2):9–22, March 1985.

    Article  Google Scholar 

  18. Brian Marick. Generic Coverage Tool (GCT) User’s Guide, Testing Foundations, Champaign, Illinois, 1992.

    Google Scholar 

  19. G. J. Myers. The art of software testing. New York: John Wiley and Sons, 1979.

    Google Scholar 

  20. A. Jefferson Offutt. The coupling effect: fact or fiction. In Proceedings of the Third Workshop on Software Testing, Verification, and Analysis, December 1989.

    Google Scholar 

  21. T. J. Ostrand and M. J. Balcer. The category-partition method for specifying and generating functional tests. Communications of the ACM, 31(6):676–686, June 1988.

    Article  Google Scholar 

  22. Robert M. Poston. Automating Specification-based Software Testing, IEEE, 1996.

    Google Scholar 

  23. D. J. Richardson, S. L. Aha, and T. O. O’Malley. Specification-based test oracles for reactive systems. In Proceedings of the Fourteenth International Conference on Software Engineering, pages 105–118, Melbourne, Australia, May 1992.

    Google Scholar 

  24. D. J. Richardson and L. A. Clarke. Partition analysis: a method combining testing and verification. IEEE Transactions on Software Engineering, SE-11(12):1477–1490, December 1985.

    Article  Google Scholar 

  25. S. Sankar and R. Hayes. Specifying and testing software components using ADL. Technical Report SMLI TR-94-23, Sun Microsystems Laboratories, Inc., Mountain View, California, April 1994.

    Google Scholar 

  26. J. M. Spivey. The Z Notation: a Reference Manual. Prentice Hall, New York, 1989.

    MATH  Google Scholar 

  27. P. Stocks and D. Carrington. Test template framework: a specification-based testing case study. In Proceedings of the 1993 International Symposium on Software Testing and Analysis (ISSTA’93), pp. 11–18, Cambridge, Massachusetts, June 1993.

    Google Scholar 

  28. Sun Microsystems Inc., ADL Language Reference Manual, Release 1.1, Mountain View, California, December 1996.

    Google Scholar 

  29. Sun Microsystems Inc. ADLT FTP Download Site. http://ftp.uu.net/vendor/adl.

  30. K. C. Tai. Condition-based software testing strategies. In Proceedings of the 14th Annual International Computer Software and Applications Conference, 1990.

    Google Scholar 

  31. E. Weyuker, T. Goradia, and A. Singh. Automatically generating test data from a boolean specification. IEEE Transactions on Software Engineering, SE-20(5):353–363, May 1994.

    Article  Google Scholar 

  32. E. J. Weyuker and T. J. Ostrand. Theories of program testing and the application of revealing subdomains. IEEE Transactions on Software Engineering, SE-6(3):236–246, May 1980.

    Article  MathSciNet  Google Scholar 

  33. C. J. Van Wyk. Data structures and C programs. Addison-Wesley, 1988.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Sun Microsystems Inc., 901 San Antonio Road, MTV10-140, Palo Alto, CA, 94303-4900, USA

    Juei Chang

  2. Information and Computer Science, University of California, Irvine, CA, 92697-3425, USA

    Debra J. Richardson

Authors
  1. Juei Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Debra J. Richardson
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Institut für Informatik (IAM), Universität Bern, Neubrückstr. 10, CH-3012, Bern, Switzerland

    Oscar Nierstrasz

  2. ONERA Centre de Toulouse, DPRS/SAE, 2, Ave. E. Belin, F-31055, Toulouse Cedex, France

    Michel Lemoine

Rights and permissions

Reprints and permissions

Copyright information

© 1999 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Chang, J., Richardson, D.J. (1999). Structural Specification-Based Testing: Automated Support and Experimental Evaluation. In: Nierstrasz, O., Lemoine, M. (eds) Software Engineering — ESEC/FSE ’99. ESEC SIGSOFT FSE 1999 1999. Lecture Notes in Computer Science, vol 1687. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-48166-4_18

Download citation

  • DOI: https://doi.org/10.1007/3-540-48166-4_18

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-66538-0

  • Online ISBN: 978-3-540-48166-9

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation