On test case reduction for testing safety properties of manufacturing systems

Highlights

• Input-output conformance simulation relation is not adequate for testing safety properties of a system.

• Safe input-output conformance simulation relation is better suited to test safety properties of a system.

• Repetitive testing of same behaviors wastes time, this is addressed by using the bisimulation equivalence relation.

• Non-determinism increases uncertainty in the testing procedure.

• Subset construction method can help in removing the uncertainty associated with non-determinism.

Abstract

This paper presents an approach to reduce the number of test cases, and hence testing time for the safe input-output conformance simulation relation (safe-IOCOS). The safe-IOCOS relation requires the implementation to be trace equivalent with respect to the specification only for traces composed of safety behaviors, which makes safe-IOCOS a suitable relation to test safety properties in practical settings. However, in typical manufacturing systems, multiple safety behaviors are typically associated with each nominal operation in the implementation. Thus, if safe-IOCOS is used industrially then testing for safety related faults becomes time consuming as the traces composed of same safety behaviors gets tested multiple times. This is possible either if the target states reached after the execution of traces have the same past behavior or the same future behavior. To remedy this, two reduction methods are proposed in this paper, subset construction and bisimulation equivalence. Both reduction methods preserve the traces of the system. Using both subset construction and bisimulation, a given specification can be maximally reduced and then used to implement the manufacturing system. The implementation based on a maximally reduced bisimilar specification allows the test engineer to omit test cases if the same safety behavior has already been tested. Furthermore, faults related to missing safety behaviors that are associated with multiple traces can be uncovered more efficiently compared to if the non-reduced specification is used for testing. To summarize, testing is a laborious problem, which can benefit from methods that enable reduction in testing time and makes the testing procedure efficient in terms of uncovering errors.

Introduction

Currently, many machines in the industrial sector are controlled via programmable logic controllers (PLCs). These PLCs are usually programmed such that the installed machines are coordinated to carry out specified tasks. The controlled behavior of such machines is mainly discrete in nature, therefore these can be formally modeled as discrete event systems [1].

Discrete event systems evolve with respect to occurring events, while at each time instant occupying a specific state where certain conditions are valid. Formally, the interaction of such systems can be described by different variants of synchronous composition, see [2].

To ensure safety, a specialized controller called a safety PLC is typically used. The job of the safety PLC is to keep both machines and humans safe in critical situations.

When it comes to control code this can be generated either automatically or manually. For automatic generation of controllers the supervisory control theory [3] framework is often used in the formal domain. However, industrially, engineers mostly carry out this task manually. After the controller generation, the physical controller (the PLC) is coupled with the real plant in a closed-loop setting referred to as the implementation.

To increase the confidence that the implementation is fault free, testing of the implementation is carried out with respect to some specification. Frequently, this task is done using a formal approach like model-based testing (MBT) [4].

In model-based testing, the implementation, which is regarded as a black-box, is exposed to various inputs in accordance to the specification and the emitted outputs are observed. There are several variations of such an approach, and in this paper, we focus on two approaches i.e. input-output conformance simulation relation (IOCOS) [5] and safe input-output conformance simulation relation (safe-IOCOS) [6].

IOCOS is a more fine-grained variation of IOCO [7], [8], with an extra requirement on the inputs. According to the IOCOS testing framework presented in [5], [9], an implementation fails the IOCOS relation if the implementation is missing man- datory inputs or if it generates unexpected outputs. However, to uncover faults associated with safety in practical settings, IOCOS is found to be inadequate due to the subset requirement on outputs and the superset requirement on inputs [6].

Due to this, the safe input-output conformance simulation relation (safe-IOCOS) [6] is introduced. The safe-IOCOS relation requires equality for the inputs and outputs related to safety. And this requirement makes safe-IOCOS suitable to find faults related to safety. However, in practical settings, there are many safety behaviors implemented with each nominal control sequence of a production system that must be tested per the definition of safe-IOCOS. This obligation to test all repetitive safety combinations unnecessarily increases the time spent for testing.

For example, if a floor scanner is activated by a human entering a certain zone, then all machines in that zone must pause their activity until the human leaves. Hence, the safety of the floor scanner input is common for all the machines in that particular zone.

Though testing can only reveal the presence of faults, never their absence, it can raise the confidence in the system to be free of obvious and frequently occurring faults, if the system can be subjected to enough test cases. Thus, there is a need to reduce the testing time for this confidence to be raised. Reducing the testing time can be achieved if the specification, and hence the implementation, is “well-behaved” in the sense of not containing unnecessary branching and non-determinism.

There are various approaches that deal with reduction of test cases. For example [10], proposes an approach based on integer linear programming and on the properties of the control flow graph. Similarly, the approach proposed in [11] exploits neural networks to reduce test cases. However these approaches does not deal with the problem of unnecessary branching and non-determinism in relation to safety.

In an attempt to achieve this, a testing approach based on bisimulation [12] equivalence and the subset construction method [13] is studied in this paper.

Bisimulation is an equivalence relation that considers two states equivalent if they have the same future behavior. This property can be exploited to reduce the given specification such that traces with same future safety behaviors need not be tested multiple times.

The subset construction method, on the other hand, removes non-determinism by merging states with the same past behavior. This method can help to avoid test cases with non-deterministic behavior, which as a consequence allow the test engineer to avoid retesting traces that have already been tested.

In this paper we show how reduction methods called bisimulation and subset construction can be exploited to reduce the complexity of the specification before implementation to make testing efficient. It is shown that by employing bisimulation and subset construction, the number of test cases can be reduced for the safe-IOCOS relation, while the behavior of the system is preserved. Moreover, as non-determinism is removed from the system the confidence in the test procedure will increase. Furthermore, an example is presented to show how reducing the given specification helps in reducing test cases and consequently the time spent during testing of an implementation.

This paper is structured in seven sections. In Section 2, the formal definitions required to describe safe-IOCOS are detailed. Section 3 gives an overview of the safe-IOCOS testing relation. Section 4 introduces the bisimulation relation. In Section 5 the subset construction method and the problem associated with non-determinism are introduced. Section 6 introduces the proposed approach and formal proofs along with some examples modeled in the tool Supremica [14]. Finally, Section 7 concludes the paper and presents some future work directions.