Skip to main content
Log in

Abstract

Runtime enforcement (RE) is a technique to ensure that the (untrustworthy) output of a black-box system satisfies some desired properties. In RE, the output of the running system, modeled as a sequence of events, is fed into an enforcer. The enforcer ensures that the sequence complies with a certain property, by delaying or modifying events if necessary. This paper deals with predictive runtime enforcement, where the system is not entirely black-box, but we know something about its behavior. This a priori knowledge about the system allows to output some events immediately, instead of delaying them until more events are observed, or even blocking them permanently. This in turn results in better enforcement policies. We also show that if we have no knowledge about the system, then the proposed enforcement mechanism reduces to standard (non-predictive) runtime enforcement. All our results related to predictive RE of untimed properties are also formalized and proved in the Isabelle theorem prover. We also discuss how our predictive runtime enforcement framework can be extended to enforce timed properties.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Notes

  1. The Isabelle theories are available at: https://github.com/isabelle-theory/PredictiveRuntimeEnforcement.

  2. The Python implementation is available for download at: https://github.com/SrinivasPinisetty/PredictiveRE.

  3. Similarly to some monitoring frameworks [9, 19, 24], we consider safety and co-safety properties over finite words.

  4. We denote accepting locations using double circles.

  5. Co-safety properties are extension-closed languages by definition.

  6. The language accepted by the automaton \({\mathcal {C}}\) from state (pq) is empty if no accepting state is reachable from state (pq) in \({\mathcal {C}}\).

  7. In some earlier works [17, 19], constraint monotonicity is called a physical constraint.

  8. Remark 3 in [19] describes non-enforceable timed properties.

References

  1. Alur R, Dill DL (1994) A theory of timed automata. Theor Comput Sci 126:183–235. doi:10.1016/0304-3975(94)90010-8

    Article  MathSciNet  MATH  Google Scholar 

  2. Baier C, Bertrand N, Bouyer P, Brihaye T (2009) When are timed automata determinizable? In: Albers S, Marchetti-Spaccamela A, Matias Y, Nikoletseas SE, Thomas W (eds) 36th international colloquium on automata, languages and programming, ICALP 2009, Rhodes, Greece, July 5–12, 2009, proceedings, part II. Lecture notes in computer science, vol 5556. Springer, pp 43–54. doi:10.1007/978-3-642-02930-1_4

  3. Bloem R, Könighofer B, Könighofer R, Wang C (2015) Shield synthesis: runtime enforcement for reactive systems. In: TACAS. LNCS, vol 9035. Springer, Berlin

  4. Cassandras CG, Lafortune S (2006) Introduction to discrete event systems. Springer, Secaucus

    MATH  Google Scholar 

  5. Chabot H, Khoury R, Tawbi N (2011) Extending the enforcement power of truncation monitors using static analysis. Comput Secur 30(4):194–207

    Article  Google Scholar 

  6. Dolzhenko E, Ligatti J, Reddy S (2015) Modeling runtime enforcement with mandatory results automata. Int J Inf Secur 14(1):47–60. doi:10.1007/s10207-014-0239-8

    Article  Google Scholar 

  7. D’Silva V, Kroening D, Weissenbacher G (2008) A survey of automated techniques for formal software verification. IEEE Trans CAD Integr Circuits Syst 27(7):1165–1178. doi:10.1109/TCAD.2008.923410

    Article  Google Scholar 

  8. Evans D, Larochelle D (2002) Improving security using extensible lightweight static analysis. IEEE Softw 19(1):42–51. doi:10.1109/52.976940

    Article  Google Scholar 

  9. Falcone Y, Fernandez J, Mounier L (2012) What can you verify and enforce at runtime? STTT 14(3):349–382

    Article  Google Scholar 

  10. Falcone Y, Mounier L, Fernandez JC, Richier JL (2011) Runtime enforcement monitors: composition, synthesis, and enforcement abilities. Form Methods Syst Des 38(3):223–262

    Article  MATH  Google Scholar 

  11. Falcone Y, Jéron T, Marchand H, Pinisetty S (2016) Runtime enforcement of regular timed properties by suppressing and delaying events. Sci Comput Program 123:2–41

    Article  Google Scholar 

  12. Finkel O (2006) Undecidable problems about timed automata. In: Asarin E, Bouyer P (ed) Formal modeling and analysis of timed systems: 4th international conference, FORMATS 2006, Paris, France, September 25–27, 2006. Springer, Berlin, pp 187–199

  13. Ligatti J, Bauer L, Walker D (2009) Run-time enforcement of nonsafety policies. ACM Trans Inf Syst Secur 12(3):19:1–19:41

    Article  Google Scholar 

  14. Malan GR, Watson D, Jahanian F, Howell P (2000) Transport and application protocol scrubbing. In: Proceedings IEEE INFOCOM 2000, Israel, pp 1381–1390

  15. Nipkow T, Paulson LC, Wenzel M (2002) Isabelle/HOL—a proof assistant for higher-order logic. In: LNCS, vol 2283. Springer, Berlin

  16. Pinisetty S, Falcone Y, Jéron T, Marchand H (2014) Runtime enforcement of parametric timed properties with practical applications. In: Lesage J, Faure J, Cury JER, Lennartson B (eds) 12th international workshop on discrete event systems, WODES 2014. International federation of automatic control, pp. 420–427. http://www.ifac-papersonline.net/Discrete_Event_Systems/12th_International_Workshop_on_Discrete_Event_Systems__2014_/index.html

  17. Pinisetty S, Falcone Y, Jéron T, Marchand H (2014) Runtime enforcement of regular timed properties. In: Proceedings of the ACM symposium on applied computing (SAC-SVT). ACM, pp 1279–1286

  18. Pinisetty S, Falcone Y, Jéron T, Marchand H, Rollet A, Timo OLN (2012) Runtime enforcement of timed properties. In: Qadeer S, Tasiran S (eds) Proceedings of the third international conference on runtime verification (RV 2012). Lecture notes in computer science, vol 7687. Springer, Berlin, pp 229–244

  19. Pinisetty S, Falcone Y, Jéron T, Marchand H, Rollet A, Nguena Timo O (2014) Runtime enforcement of timed properties revisited. Form Methods Syst Des 45(3):381–422

    Article  MATH  Google Scholar 

  20. Pinisetty S, Preoteasa V, Tripakis S, Jéron T, Falcone Y, Marchand H (2016) Predictive runtime enforcement. In: Proceedings of the ACM symposium on applied computing (SAC-SVT). ACM (to appear)

  21. Pitt J, Mamdani EH (1999) A protocol-based semantics for an agent communication language. In: Dean T (ed) Proceedings of the sixteenth international joint conference on artificial intelligence, IJCAI 99, Stockholm, Sweden, July 31–August 6, 1999, 2 vol. Morgan Kaufmann, pp 486–491

  22. Raffelt H, Steffen B, Berg T, Margaria T (2009) Learnlib: a framework for extrapolating behavioral models. Int J Softw Tools Technol Transf 11(5):393–407. doi:10.1007/s10009-009-0111-8

    Article  Google Scholar 

  23. Renard M, Falcone Y, Rollet A, Pinisetty S, Jéron T, Marchand H (2015) Enforcement of (timed) properties with uncontrollable events. In: Leucker M, Rueda C, Valencia FD (eds) 12th international colloquium on theoretical aspects of computing–ICTAC 2015, Cali, Colombia, October 29–31, 2015, proceedings. Lecture notes in computer science, vol 9399. Springer, Berlin, pp 542–560

  24. Rosu G (2012) On safety properties and their monitoring. Sci Ann Comput Sci 22(2):327–365

    MathSciNet  Google Scholar 

  25. Schneider FB (2000) Enforceable security policies. ACM Trans Inf Syst Secur 3(1):30–50. doi:10.1145/353323.353382

    Article  MathSciNet  Google Scholar 

  26. Tripakis S (2006) Folk theorems on the determinization and minimization of timed automata. Inf Process Lett 99(6):222–226. doi:10.1016/j.ipl.2006.04.015

    Article  MathSciNet  MATH  Google Scholar 

  27. Tuglular T, Belli F (2009) Protocol-based testing of firewalls. In: 2009 fourth south-east European workshop on formal methods (SEEFM), pp 53–59

  28. Wooldridge M (2009) An introduction to multiagent systems, 2nd edn. Wiley Publishing, New York

    Google Scholar 

  29. Zhang X, Leucker M, Dong W (2012) Runtime verification with predictive semantics. In: 4th international symposium on NASA formal methods. LNCS, vol 7226. Springer, Berlin, pp 418–432

Download references

Acknowledgements

This work was supported in part by the Academy of Finland and the U.S. National Science Foundation (Awards #1329759 and #1139138). Thierry Jéron and Yliès Falcone acknowledge the support of the COST Action ARVI IC1402, which is supported by COST (European Cooperation in Science and Technology).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Srinivas Pinisetty.

Appendices

Appendix 1: Proofs

1.1 Proofs: untimed setting

In this section, we will discuss proofs of lemmas and theorems in Sect. 3. \(\psi \) and \(\varphi \) in this section are regular properties that are defined by automata \(\mathcal {A_\psi }\) and \(\mathcal {A_\varphi }\). To ease understanding, for some Lemmas, we provide manual proofs with some explanations. A document with all Isabelle proofs can be accessed from: https://github.com/isabelle-theory/PredictiveRuntimeEnforcement.

Proof (of Lemma 3)

We shall prove that the urgency constraint  (Ur’) is weaker than the urgency constraint (Ur), i.e.,

$$\begin{aligned} \begin{array}{ll} \forall \sigma \in \varSigma ^*: (\forall \sigma _{\mathrm{con}}\in \varSigma ^*: \sigma \cdot \sigma _{\mathrm{con}}\in \psi \implies \\ \qquad \exists \sigma '\in \varSigma ^*: \sigma ' \preccurlyeq \sigma _{\mathrm{con}}\wedge \sigma \cdot \sigma ' \in \varphi ) \implies E_{\psi ,\varphi }(\sigma ) = \sigma \end{array} \end{aligned}$$

\(\implies \)

$$\begin{aligned} \begin{array}{ll} \forall \sigma \in \varSigma ^*: (\forall \sigma _{\mathrm{con}}\in \varSigma ^*: \sigma \cdot \sigma _{\mathrm{con}}\in \psi \implies \sigma \cdot \sigma _{\mathrm{con}}\in \varphi )\\ \qquad \implies E_{\psi ,\varphi }(\sigma ) = \sigma . \end{array} \end{aligned}$$

Assume (Ur) and for \(\sigma \in \varSigma ^*\) assume that \((\forall \sigma _{\mathrm{con}}\in \varSigma ^*: \sigma \cdot \sigma _{\mathrm{con}}\in \psi \implies \sigma \cdot \sigma _{\mathrm{con}}\in \varphi )\) holds. We need to show that \(E_{\psi ,\varphi }(\sigma ) = \sigma \). We have

figure n

Therefore the hypothesis of (Ur) is true, so we obtain \(E_{\psi ,\varphi }(\sigma ) = \sigma \).\(\square \)

Proof (of Lemma 4)

We shall prove that given properties \(\psi , \varphi \subseteq \varSigma ^*\), when \(\psi = \varSigma ^*\), the constraint (Ur) is equivalent to the following:

$$\begin{aligned} \forall \sigma \in \varSigma ^*: \sigma \in \varphi \implies E_{\psi ,\varphi }(\sigma ) = \sigma . \end{aligned}$$
figure o

\(\square \)

Proof (of Lemma 5)

We shall prove that give properties \(\psi , \varphi \subseteq \varSigma ^*\), when \(\psi \subseteq \varphi \), for any word \(\sigma \in \varSigma ^*\), the output of the enforcement function is \(\sigma \) (\(E_{\psi ,\varphi }(\sigma ) = \sigma \)). Assume \(\psi \subseteq \varphi \), then we have:

figure p

From this, using urgency (Ur), we obtain \(E_{\psi ,\varphi }(\sigma ) = \sigma \).\(\square \)

Proof (of Lemma 10)

We shall prove the following properties of the enforcement function, and auxiliary functions \(\mathrm {store_{\psi ,\varphi }}\) and \(\kappa _{\psi ,\varphi }\).

For all \(\sigma ,\sigma '\in \varSigma ^*\) we have

  1. 1.

    \( \mathrm {store_{\psi ,\varphi }}(\sigma )=(\sigma _{s},\sigma _{c})\implies \sigma =\sigma _{s}\cdot \sigma _{c} \)

  2. 2.

    \(E_{\psi ,\varphi }(\sigma )\not =\epsilon \implies \kappa _{\psi ,\varphi }(E_{\psi ,\varphi }(\sigma ))\)

  3. 3.

    \(\kappa _{\psi ,\varphi }(\sigma ) \wedge \sigma \preccurlyeq \sigma ' \implies \sigma \preccurlyeq E_{\psi ,\varphi }(\sigma ')\)

  4. 4.

    \(\sigma \in \varphi \implies \kappa _{\psi ,\varphi }(\sigma )\)

Proof (of property 1 of Lemma 10)

Let us prove Property 1 using induction on the input sequence \(\sigma \).

Induction basis. If \(\sigma = \epsilon \), from the definition of the enforcement function (Definition 4), \(E_{\psi ,\varphi }(\sigma )= \epsilon \). From the definition of \(\mathrm {store_{\psi ,\varphi }}, \mathrm {store_{\psi ,\varphi }}(\epsilon ) = (\epsilon , \epsilon )\). Since \(\epsilon = \epsilon \cdot \epsilon \), this property holds for \(\sigma = \epsilon \).

Induction step. Assume that for every \(\sigma \in \varSigma ^*\) of some length \(n \in {\mathbb {N}}, \mathrm {store_{\psi ,\varphi }}(\sigma )=(\sigma _{s},\sigma _{c})\implies \sigma =\sigma _{s}\cdot \sigma _{c}\).

We now prove that for any \( a\in \varSigma \), property 1 holds for \(\sigma \cdot a\). We have the following two possible cases:

  • Case \(\kappa _{\psi ,\varphi }(\sigma \cdot a) = \mathsf {true}\). Since \(\kappa _{\psi ,\varphi }(\sigma \cdot a)\) is true, according to the definition of \(\mathrm {store_{\psi ,\varphi }}\), we will have \(\mathrm {store_{\psi ,\varphi }}(\sigma \cdot a) = (\sigma _{s}\cdot \sigma _{c}\cdot a, \epsilon )\). From the induction hypothesis, we have \(\sigma _{s}\cdot \sigma _{c}= \sigma \). So, we have \(\sigma _{s}\cdot \sigma _{c}\cdot a \cdot \epsilon = \sigma \cdot a\). Thus, the property holds.

  • Case \(\kappa _{\psi ,\varphi }(\sigma \cdot a) = \mathsf {false}\). Since \(\kappa _{\psi ,\varphi }(\sigma \cdot a)\) is false, according to the definition of \(\mathrm {store_{\psi ,\varphi }}\), we will have \(\mathrm {store_{\psi ,\varphi }}(\sigma \cdot a) = (\sigma _{s}, \sigma _{c}\cdot a)\). Using induction hypothesis, we will have \(\sigma _{s}\cdot \sigma _{c}\cdot a \cdot \epsilon = \sigma \cdot a\). Thus, the property holds.\(\square \)

Proof (of property 2 of Lemma 10)

Let us prove Property 2 using induction on the input sequence \(\sigma \).

Induction basis. If \(\sigma = \epsilon \), from the definition of the enforcement function (Definition 4), \(E_{\psi ,\varphi }(\epsilon )= \epsilon \). Since \(E_{\psi ,\varphi }(\epsilon ) = \epsilon \), property \(E_{\psi ,\varphi }(\sigma )\not =\epsilon \implies \kappa _{\psi ,\varphi }(E_{\psi ,\varphi }(\sigma ))\) trivially holds for \(\sigma = \epsilon \).

Induction step. Assume that for every \(\sigma \) of some length \(n \in {\mathbb {N}}, E_{\psi ,\varphi }(\sigma )\not =\epsilon \implies \kappa _{\psi ,\varphi }(E_{\psi ,\varphi }(\sigma ))\). Let \(\mathrm {store_{\psi ,\varphi }}(\sigma )= (\sigma _s, \sigma _c)\). According to Definition 4, we know that \(E_{\psi ,\varphi }(\sigma )= \varPi _1\Big (\mathrm {store_{\psi ,\varphi }}(\sigma )\Big ) = \sigma _s\).

We now prove that for any \( a\in \varSigma \), Property 2 also holds for \(\sigma \cdot a\). We have the following two possible cases:

  • Case \(E_{\psi ,\varphi }(\sigma \cdot a) = \epsilon \). The property trivially holds in this case.

  • Case \(E_{\psi ,\varphi }(\sigma \cdot a)\ne \epsilon \). We can also notice from Definition 4 that there are two possible cases based on whether \(\kappa _{\psi ,\varphi }(\sigma \cdot a)\) is either true or false.

    • Case \(\kappa _{\psi ,\varphi }(\sigma \cdot a) = true\). In this case, \(E_{\psi ,\varphi }(\sigma \cdot a) = \sigma _{s}\cdot \sigma _{c}\cdot a =\sigma \cdot a\) (from Property 1). Also, \(\kappa _{\psi ,\varphi }(\sigma \cdot a)\) is true in this case. Thus the property holds for \(\sigma \cdot a\).

    • Case \(\kappa _{\psi ,\varphi }(\sigma \cdot a) = false\). In this case, \(E_{\psi ,\varphi }(\sigma \cdot a) = E_{\psi ,\varphi }(\sigma )\), and using the induction hypothesis, we can conclude that the property holds also for \(\sigma \cdot a\).\(\square \)

Proof (of property 3 of Lemma 10)

Let us prove Property 3 using induction on the length of \(\sigma '\).

Induction basis. If \(\sigma ' = \epsilon \), then \(\sigma = \epsilon \). From the definition of the enforcement function (Definition 4), \(E_{\psi ,\varphi }(\sigma )= \epsilon \). Since \(\epsilon \preccurlyeq \epsilon \), property 3 trivially holds for \(\sigma = \epsilon \).

Induction step. Assume that for every \(\sigma ' \in \varSigma ^*\) of some length \(n \in {\mathbb {N}}, \kappa _{\psi ,\varphi }(\sigma ) \wedge \sigma \preccurlyeq \sigma ' \implies \sigma \preccurlyeq E_{\psi ,\varphi }(\sigma ')\) holds.

Let \(\mathrm {store_{\psi ,\varphi }}(\sigma ')= (\sigma '_s, \sigma '_c)\). According to Definition 4, we know that \(E_{\psi ,\varphi }(\sigma ')= \varPi _1\Big (\mathrm {store_{\psi ,\varphi }}(\sigma )\Big ) = \sigma '_s\). From property 1, we have \(\sigma ' = \sigma _{s}'\cdot \sigma _{c}'\).

We now prove that for any \( a\in \varSigma \), Property 3 holds for \(\sigma '\cdot a\). We have the following two possible cases:

  • Case \(\kappa _{\psi ,\varphi }(\sigma '\cdot a) = \mathsf {true}\). Since \(\kappa _{\psi ,\varphi }(\sigma \cdot a)\) is true, according to the definition of \(\mathrm {store_{\psi ,\varphi }}\), we will have \(\mathrm {store_{\psi ,\varphi }}(\sigma ' \cdot a) = (\sigma _{s}' \cdot \sigma _{c}' \cdot a, \epsilon )\). Consequently, \(E_{\psi ,\varphi }(\sigma '\cdot a)= \sigma _{s}' \cdot \sigma _{c}' \cdot a = \sigma '\cdot a\).

    We now have two subcases based on whether \(\sigma = \sigma ' \cdot a\) or not.

    • Case \(\sigma = \sigma ' \cdot a\). We already saw that \(\kappa _{\psi ,\varphi }(\sigma '\cdot a)\) is true, and \(E_{\psi ,\varphi }(\sigma '\cdot a)= \sigma '\cdot a = \sigma \). Consequently, we have \(\sigma \preccurlyeq E_{\psi ,\varphi }(\sigma '\cdot a)\). Thus the property \(\kappa _{\psi ,\varphi }(\sigma ) \wedge \sigma \preccurlyeq \sigma '\cdot a \implies \sigma \preccurlyeq E_{\psi ,\varphi }(\sigma ' \cdot a)\) holds in this case.

    • Case \(\sigma \preccurlyeq \sigma '\). From the induction hypothesis, we have \(\sigma \preccurlyeq E_{\psi ,\varphi }(\sigma ') = \sigma _{s}'\). In this case, we already showed that \(E_{\psi ,\varphi }(\sigma '\cdot a)= \sigma _{s}' \cdot \sigma _{c}' \cdot a\). Since \(\sigma \preccurlyeq \sigma _{s}'\), we also have \(\sigma \preccurlyeq \sigma _{s}' \cdot \sigma _{c}' \cdot a = E_{\psi ,\varphi }(\sigma '\cdot a)\). Thus the property \(\kappa _{\psi ,\varphi }(\sigma ) \wedge \sigma \preccurlyeq \sigma '\cdot a \implies \sigma \preccurlyeq E_{\psi ,\varphi }(\sigma ' \cdot a)\) holds in this case.

  • Case \(\kappa _{\psi ,\varphi }(\sigma '\cdot a) = \mathsf {false}\). Since \(\kappa _{\psi ,\varphi }(\sigma ' \cdot a)\) is false, according to the definition of \(\mathrm {store_{\psi ,\varphi }}\), we will have \(\mathrm {store_{\psi ,\varphi }}(\sigma ' \cdot a) = (\sigma _{s}', \sigma _{c}'\cdot a)\). In this case, \(E_{\psi ,\varphi }(\sigma ' \cdot a) = \sigma _{s}' = E_{\psi ,\varphi }(\sigma ')\).

    We again have two subcases here based on whether \(\sigma = \sigma ' \cdot a\) holds or not.

    • Case \(\sigma = \sigma ' \cdot a\). Since \(\kappa _{\psi ,\varphi }(\sigma '\cdot a)\) is false, the property trivially holds in this case.

    • Case \(\sigma \preccurlyeq \sigma '\). From the induction hypothesis, we have \(\sigma \preccurlyeq E_{\psi ,\varphi }(\sigma ') = \sigma _{s}'\). Since \(E_{\psi ,\varphi }(\sigma ' \cdot a) = E_{\psi ,\varphi }(\sigma ')\), consequently we have \( \sigma \preccurlyeq E_{\psi ,\varphi }(\sigma ' \cdot a)\). Thus the property \(\kappa _{\psi ,\varphi }(\sigma ) \wedge \sigma \preccurlyeq \sigma '\cdot a \implies \sigma \preccurlyeq E_{\psi ,\varphi }(\sigma ' \cdot a)\) holds in this case. \(\square \)

Proof (of property 4 of Lemma 10)

We shall prove that given properties \(\varphi , \psi \subseteq \varSigma ^*\), and for any word \(\sigma \in \varSigma ^*\), the following property holds:

$$\begin{aligned} \sigma \in \varphi \implies \kappa _{\psi ,\varphi }(\sigma ). \end{aligned}$$

Assume that \(\sigma \in \varphi \). Function \(\kappa _{\psi ,\varphi }(\sigma )\) is defined as follows:

$$\begin{aligned} \begin{array}{ll} \kappa _{\psi ,\varphi }(\sigma ) = (\forall \sigma _{\mathrm{con}}\in \varSigma ^*: \sigma \cdot \sigma _{\mathrm{con}}\in \psi \implies \\ \quad \quad \quad \exists \sigma '\in \varSigma ^*: \sigma ' \preccurlyeq \sigma _{\mathrm{con}}\wedge \sigma \cdot \sigma ' \in \varphi ) \end{array} \end{aligned}$$
figure q

Proof (of Lemma 12)

We shall prove that the Algorithm 1 implements the enforcement function \(E_{\psi ,\varphi }\) (Definition 4).

That is, we shall prove that if \(\sigma = a_1\cdots a_n\) is the sequence of events received so far by the enforcement algorithm, and if \(\sigma _1,\cdots ,\sigma _k\) are the sequences released by the algorithm for \(\sigma \), then

$$\begin{aligned} E_{\psi ,\varphi }(\sigma )=\sigma _1\cdot \ldots \cdot \sigma _k \text{ and } \sigma = E_{\psi ,\varphi }(\sigma )\cdot \sigma _c \end{aligned}$$

where \(\sigma _c\) corresponds to \(\sigma _c\) in the algorithm, equivalent to \(\sigma _c\) in the definition of \(E_{\psi ,\varphi }\).

Let us prove this lemma using induction on the length of the input sequence \(\sigma \).

Induction basis. If \(\sigma = \epsilon \), from the definition of the enforcement function (Definition 4), \(\mathrm {store_{\psi ,\varphi }}(\epsilon ) = (\sigma _s, \sigma _c)=(\epsilon , \epsilon )\), and \(E_{\psi ,\varphi }(\sigma )= \epsilon \).

Regarding the enforcement algorithm, \(\sigma _c\) in the algorithm is initialized to \(\epsilon \), since no event is received yet, this corresponds to the first iteration of the while loop, where the process is waiting for an event in line 5. Thus statement release in the algorithm is never executed yet. Concatenation of the output sequences released by the algorithm is thus \(\epsilon \). Consequently, Lemma 12 holds for \(\sigma = \epsilon \).

Induction step. Assume that for every \(\sigma \in \varSigma ^*\) of some length \(n \in {\mathbb {N}}\), Lemma 12 holds.

Regarding the enforcement function \(\mathrm {store_{\psi ,\varphi }}(\sigma )= (\sigma _s, \sigma _c)\), and \(E_{\psi ,\varphi }(\sigma )= \varPi _1\Big (\mathrm {store_{\psi ,\varphi }}(\sigma )\Big ) = \sigma _s\). After receiving \(\sigma , p=\delta _{\psi }(q_{\psi },\sigma )\), and \(q=\delta _{\varphi }(q_{\varphi },\sigma )\). Let \(\sigma _c\) be equal to \(\sigma _c\) in the algorithm, and let the concatenation of output sequences released by the algorithm be equal to \(E_{\psi ,\varphi }(\sigma ) = \sigma _s\).

We now prove that for any \( a\in \varSigma \), Lemma 12 holds for \(\sigma \cdot a\). We have the following two possible cases:

  • Case \(\kappa _{\psi ,\varphi }(\sigma \cdot a) = \mathsf {true}\). Regarding the enforcement function, since \(\kappa _{\psi ,\varphi }(\sigma \cdot a)\) is true, according to the definition of \(\mathrm {store_{\psi ,\varphi }}\), we will have \(\mathrm {store_{\psi ,\varphi }}(\sigma \cdot a) = (\sigma _{s}\cdot \sigma _{c}\cdot a, \epsilon )\). Consequently, \(E_{\psi ,\varphi }(\sigma \cdot a)= \sigma _{s}\cdot \sigma _{c}\cdot a = \sigma \cdot a\).

    Regarding the enforcement algorithm, upon receiving event a, after executing step 6 of the algorithm, \(p=\delta _{\psi }(q_{\psi },\sigma \cdot a)\), and \(q=\delta _{\varphi }(q_{\varphi },\sigma \cdot a)\). Since \(\kappa _{\psi ,\varphi }(\sigma \cdot a) = \mathsf {true}\), using Theorem 2, \({\mathcal {L}}(C,(p,q))=\emptyset \) and statement 8 (\(release(\sigma _c\cdot a)\)) will be executed. Statement 9 will also be executed in this case resetting \(\sigma _c\) in the algorithm to \(\epsilon \). Using the induction hypothesis, we know that the concatenation of output sequences released by the algorithm is equal to \(\sigma _s\). The new sequence released is \(\sigma _c\cdot a = E_{\psi ,\varphi }(\sigma \cdot a)\). Thus, Lemma 12 holds in this case.

  • Case \(\kappa _{\psi ,\varphi }(\sigma \cdot a) = \mathsf {false}\). Regarding the enforcement function, since \(\kappa _{\psi ,\varphi }(\sigma \cdot a)\) is false, according to the definition of \(\mathrm {store_{\psi ,\varphi }}\), we will have \(\mathrm {store_{\psi ,\varphi }}(\sigma \cdot a) = (\sigma _{s}, \sigma _{c}\cdot a)\). Consequently, \(E_{\psi ,\varphi }(\sigma \cdot a) = \sigma _{s}= E_{\psi ,\varphi }(\sigma )\).

    Regarding the enforcement algorithm, upon receiving event a, after executing step 6 of the algorithm, \(p=\delta _{\psi }(q_{\psi },\sigma \cdot a)\), and \(q=\delta _{\varphi }(q_{\varphi },\sigma \cdot a)\). Since \(\kappa _{\psi ,\varphi }(\sigma \cdot a)\) is false, using Theorem 2, \({\mathcal {L}}(C,(p,q))\ne \emptyset \). Consequently, statements 8, and 9 of the algorithm will not be executed, and statement 11 will be executed before going back to statement 3 (next iteration of the while loop). Using the induction hypothesis, Lemma 12 holds for \(\sigma \cdot a\) in this case. \(\square \)

1.2 Proofs: timed setting

In this section, we will discuss the proof of Theorem 3. \(\varphi \) and \(\psi \) in this section are regular timed properties that are defined by deterministic TA \(\mathcal {A_\varphi }\) and \(\mathcal {A_\psi }\).

Proof (of Theorem 3)

We shall prove that given two properties \(\psi \), and \(\varphi \), the enforcement function \(E_{\psi ,\varphi }\) as per Definition 8 is a timed predictive enforcer satisfying constraints (SndT), (TrT), (MoT), and (UrT).

Let us recall the definition of the enforcement function.

$$\begin{aligned} E_{\psi ,\varphi }(\sigma ) = \begin{array}{ll} \varPi _1 \left( {{\mathrm{\mathrm {store_{\psi ,\varphi }}}}}\left( \sigma \right) \right) , \end{array} \end{aligned}$$

where \({{\mathrm{\mathrm {store_{\psi ,\varphi }}}}}: {{\mathrm{tw}}}(\varSigma ) \rightarrow {{\mathrm{tw}}}(\varSigma ) \times {{\mathrm{tw}}}(\varSigma ) \) is defined as

where

$$\begin{aligned} \kappa _{\psi ,\varphi }(\sigma _n, \sigma _s, \sigma '_c) \mathop {=}\limits ^{{\scriptstyle \mathrm {def}}}{{\mathrm{CanD}}}(\sigma '_c) \cap {{\mathrm{Sure}}}_{\psi ,\varphi }(\sigma _n, \sigma _s) \end{aligned}$$

and

The proof of constraint (MoT) follows the same reasoning of proof of Property 3 of Lemma 10 in the untimed setting.

Regarding constraints (SndT), we prove that the enforcement function \(E_{\psi ,\varphi }\) satisfies the following condition:

$$\begin{aligned} \forall \sigma \in {{\mathrm{tw}}}(\sigma ): \sigma \in \psi \implies (E_{\psi ,\varphi }(\sigma ) = \epsilon \vee E_{\psi ,\varphi }(\sigma ) \models \varphi ) \end{aligned}$$

Regarding constraint (TrT), we prove a slightly stronger property:

$$\begin{aligned} \forall \sigma \in {{\mathrm{tw}}}(\varSigma ): E_{\psi ,\varphi }(\sigma ) \preccurlyeq \sigma \end{aligned}$$

Let us also recall the constraint (UrT):

$$\begin{aligned} \begin{array}{ll} \forall \sigma \in {{\mathrm{tw}}}(\varSigma ), \forall t\ge \mathrm {end}(\sigma ), \forall a\in \varSigma :\\ (\exists w\in {{\mathrm{tw}}}(\varSigma ): E_{\psi ,\varphi }(\sigma )\cdot w \ge _d\sigma \cdot (t,a) ~ \wedge ~ \mathrm {start}(w) \ge t ~ \wedge \\ ~~~ \forall \sigma _{\mathrm{con}}\in {{\mathrm{tw}}}(\varSigma ): \sigma \cdot (t,a)\cdot \sigma _{\mathrm{con}}\in \psi \implies \\ ~~~~~~ \exists \sigma '\in {{\mathrm{tw}}}(\varSigma ): \sigma ' \preccurlyeq _{d} \sigma _{\mathrm{con}}~ \wedge ~ E_{\psi ,\varphi }(\sigma ) \cdot w \cdot \sigma ' \in \varphi )\\ \implies \varPi _\varSigma (E_{\psi ,\varphi }(\sigma \cdot (t,a))) = \varPi _\varSigma (\sigma \cdot (t,a)). \end{array} \end{aligned}$$

We also prove that for any timed word \(\sigma \in {{\mathrm{tw}}}(\sigma )\), the function \({{\mathrm{\mathrm {store_{\psi ,\varphi }}}}}\) satisfies the following:

$$\begin{aligned} \varPi _\varSigma (\sigma _s\cdot \sigma _c)=\varPi _\varSigma (\sigma ) \end{aligned}$$

We shall prove it by an induction on the length of the input timed word \(\sigma \).

Induction basis. If \(\sigma = \epsilon \), from the definition of the enforcement function (Definition 8), \(E_{\psi ,\varphi }(\sigma )= \epsilon \). Since \(E_{\psi ,\varphi }(\sigma ) = \sigma = \epsilon \), constraints (SndT) and (UrT) trivially holds. Since \(\epsilon \preccurlyeq \epsilon , E_{\psi ,\varphi }(\sigma ) \preccurlyeq \sigma \) holds for \(\sigma = \epsilon \), and thus constraint (TrT) holds. Since \({{\mathrm{\mathrm {store_{\psi ,\varphi }}}}}(\epsilon ) = (\epsilon , \epsilon )\), for \(\sigma = \epsilon \) the condition \(\varPi _\varSigma (\sigma _s\cdot \sigma _c)=\varPi _\varSigma (\sigma )\) holds.

Induction step. Assume that for every \(\sigma \in {{\mathrm{tw}}}(\sigma )\) of some length \(n \in {\mathbb {N}}, E_{\psi ,\varphi }\) satisfies constraints (SndT), (TrT), and (UrT).

Let \(\mathrm {store_{\psi ,\varphi }}(\sigma )= (\sigma _s, \sigma _c)\). According to Definition 8, we know that \(E_{\psi ,\varphi }(\sigma )= \varPi _1\Big (\mathrm {store_{\psi ,\varphi }}(\sigma )\Big ) = \sigma _s\). Let \(\varPi _\varSigma (\sigma _s \cdot \sigma _c) = \varPi _\varSigma (\sigma )\).

We now prove that for any \(a\in \varSigma \), and \(t\ge \mathrm {end}(\sigma ), E_{\psi ,\varphi }\) satisfies constraints (SndT), (TrT), and (UrT) for \(\sigma \cdot (t,a)\). We have the following two possible cases:

  • Case \(\kappa _{\psi ,\varphi }(\sigma \cdot (t,a), \sigma _s, \sigma _c\cdot (t,a)) \ne \emptyset \).

    From the definition of \({{\mathrm{\mathrm {store_{\psi ,\varphi }}}}}\), we have \({{\mathrm{\mathrm {store_{\psi ,\varphi }}}}}(\sigma \cdot (t,a)) = (\sigma '_s, \epsilon )\) where \(\sigma '_s = E_{\psi ,\varphi }(\sigma \cdot (t,a)) = \sigma _s \cdot {{\mathrm{min}}}_{\preceq _{\scriptscriptstyle \mathrm lex},\mathrm {end}}\kappa _{\psi ,\varphi }(\sigma \cdot (t,a), \sigma _s, \sigma _c\cdot (t,a))\). We also know that \(\varPi _\varSigma (\sigma _s \cdot \sigma _c) = \varPi _\varSigma (\sigma )\), and thus we have \(\varPi _\varSigma (E_{\psi ,\varphi }(\sigma \cdot (t,a))) = \varPi _\varSigma (\sigma '_s \cdot \epsilon ) = \varPi _\varSigma (\sigma \cdot (t,a))\).

    Note that \({{\mathrm{CanD}}}(\sigma _c\cdot (t,a))\) computes all the delayed timed words of \(\sigma _c\cdot (t,a)\) that start at or after t. Thus we have \(E_{\psi ,\varphi }(\sigma \cdot (t,a)) \preccurlyeq _d\sigma \cdot (t,a)\), and consequently we have \(E_{\psi ,\varphi }(\sigma \cdot (t,a)) \preccurlyeq _d \sigma \cdot (t,a)\). Thus (TrT) holds in this case.

    According to the definition of \({{\mathrm{Sure}}}_{\psi ,\varphi }()\), the set \(\kappa _{\psi ,\varphi }(\sigma \cdot (t,a), \sigma _s, \sigma _c\cdot (t,a))\) which is \({{\mathrm{CanD}}}(\sigma _c\cdot (t,a))\cap {{\mathrm{Sure}}}_{\psi ,\varphi }(\sigma \cdot (t,a), \sigma _s)\) is non-empty if and only if the hypothesis of the (UrT) constraint is satisfied. We already showed that \(\varPi _\varSigma (E_{\psi ,\varphi }(\sigma \cdot (t,a))) = \varPi _\varSigma (\sigma \cdot (t,a))\) in this case. Thus the constraint (UrT) holds.

    Regarding constraint (SndT), we have the following two cases based on whether \(\sigma \cdot (t,a)\) belongs to \(\psi \) or not.

    • Case \(\sigma \cdot (t,a) \not \in \psi \). Constraint (SndT) trivially holds in this case.

    • Case \(\sigma \cdot (t,a) \in \psi \). Since \(\kappa _{\psi ,\varphi }(\sigma \cdot (t,a), \sigma _s, \sigma _c\cdot (t,a))\) is non-empty, we know that \({{\mathrm{Sure}}}_{\psi ,\varphi }(\sigma \cdot (t,a), \sigma _s)\) is non-empty. In this case, since \(\sigma \cdot (t,a) \in \psi \), one possible continuation \(\sigma _{\mathrm{con}}\) is \(\epsilon \), and thus all the timed words w that belong to the set \({{\mathrm{Sure}}}_{\psi ,\varphi }(\sigma \cdot (t,a), \sigma _s)\) are the timed words satisfying the condition \(\sigma _s\cdot w \in \varphi \). Since \(\kappa _{\psi ,\varphi }(\sigma \cdot (t,a), \sigma _s, \sigma _c\cdot (t,a))\) is \({{\mathrm{CanD}}}(\sigma _c\cdot (t,a))\cap {{\mathrm{Sure}}}_{\psi ,\varphi }(\sigma \cdot (t,a), \sigma _s)\), and \(E_{\psi ,\varphi }(\sigma \cdot (t,a)) = \sigma _s \cdot {{\mathrm{min}}}_{\preceq _{\scriptscriptstyle \mathrm lex},\mathrm {end}}\kappa _{\psi ,\varphi }(\sigma \cdot (t,a), \sigma _s, \sigma _c\cdot (t,a))\), we can conclude that \(E_{\psi ,\varphi }(\sigma \cdot (t,a)) \in \varphi \) satisfying constraint (SndT).

  • Case \(\kappa _{\psi ,\varphi }(\sigma \cdot (t,a), \sigma _s, \sigma _c\cdot (t,a)) =\emptyset \). From the definition of \({{\mathrm{\mathrm {store_{\psi ,\varphi }}}}}\), we have \({{\mathrm{\mathrm {store_{\psi ,\varphi }}}}}(\sigma \cdot (t,a)) = (\sigma _s, \sigma _c\cdot (t,a))\), and consequently we have \(E_{\psi ,\varphi }(\sigma \cdot (t,a)) = \sigma _s = E_{\psi ,\varphi }(\sigma )\). Using the induction hypothesis, we can conclude that the constraints (SndT) and (TrT) also hold for \(\sigma \cdot (t,a)\). Also \(\varPi _\varSigma (\sigma _s\cdot \sigma _c\cdot (t,a) ) = \varPi _\varSigma (\sigma \cdot (t,a))\) holds. Regarding constraint (UrT), from the definition of \(\kappa _{\psi ,\varphi }( )\) and \({{\mathrm{Sure}}}_{\psi ,\varphi }()\) since \(\kappa _{\psi ,\varphi }(\sigma \cdot (t,a), \sigma _s, \sigma _c\cdot (t,a)) =\emptyset \), we can conclude that the hypothesis of the constraint (UrT) does not hold in this case. Thus, the constraint (UrT) trivially holds in this case.

\(\square \)

Appendix 2: Implementation details

The predictive enforcement monitoring algorithm described in Sect. 3.5 is implemented in Python. Source files and examples can be downloaded from: https://github.com/SrinivasPinisetty/PredictiveRE.

Module Automata.py contains all the functionality related to defining automata and operations on automata. Module Enforcer.py contains an implementation of the predictive enforcement algorithm described in Sect. 3.5. Using these modules is simple, and only requires Python. Let us now see an example illustrating how to describe the automata \(\psi , \varphi \), and how to invoke the enforcer method.

In addition to the Python system module, import both the modules Automata and Enforcer.

figure r

Describe automata \(\psi \), and \(\varphi \). In this example, \(\psi \) is the automaton in Fig. 4, and \(\varphi \) is the automaton in Fig. 3.

figure s
figure t

The enforcer method with automata \(\psi , \varphi \), and some test input sequence can be invoked as follows:

figure u

We provide other examples in the directories examplesUsage and examples Additional. All the examples used for evaluation (see Sect. 3.7) are available in the directory examplesAdditional. To run the examples in the directories examplesUsage and examplesAdditional via command prompt, navigate to the directory, and invoke each script in the directory. For example if test1.py is under the examplesUsage directory, you can invoke it using the command python test1.py.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pinisetty, S., Preoteasa, V., Tripakis, S. et al. Predictive runtime enforcement. Form Methods Syst Des 51, 154–199 (2017). https://doi.org/10.1007/s10703-017-0271-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10703-017-0271-1

Keywords

Navigation