Orthogonal Latin squares: an application of experiment design to compiler testing

Reviewer: Bruce Melvin Skwiersky

The author describes an approach to the generation of test cases which allows reasonable confidence in test results without necessitating exhaustive testing of all combinations of test factors and values. The paper is oriented to the type of tests which the author has been designing for the ADA Compiler Validation Capability (ACVC) test suite. The approach centers on the creation of square arrays (Latin squares and Greco-Latin squares) of test factors and values which can be expanded into source or source pseudocode test cases. A latin square is a square array of n letters each repeated n times and arranged so that each letter occurs just once in every row and column. Latin squares form the basis for an important type of experimental design. Two examples underlie much of the discussion in the paper. One, a crop treatment application experiment, illustrates experiment design using a Latin square. The other example is a stripped down version of the tests used in the ACVC test suite. It illustrates the combining of orthogonal Latin squares into a Greco-Latin square of 16 test cases combinations. In cases where Latin squares are appropriate, they produce results which are more reliable than would result from randomly chosen test cases. The trick is to find area of applicability. The technique requires that the “index-factors” (i.e., row and column labels) be independent of the “contents factors” (i.e., the contents of the matrix entries). Further, the number of index factors and contents factors must be such that they can be assigned to rows, columns, and contents so that the “squareness” can be preserved. For tests concerning compiler data types applied to index factors as neutral as character sets, the Latin square/Greco-Latin square technique is very powerful. It is a useful extension and partial formalization of the tricks or habits (e.g, end-point testing) which those involved in testing use to restrict the effort and cost of constructing and executing tests. The paper is written clearly at a level providing value to the student and expert alike. Ignorance of the origins of Latin and Greco-Latin squares in experiment design theory is no handicap to appreciating the suggestions made by the author. The only weakness of the paper is that, after the introductory comments, the author does not explicitly point out the limitations in the applicability of the technique.