Explaining the Effectiveness of Small Refinement Heuristics in Program Verification with CEGAR

Abstract

Safety property (i.e., reachability) verification is undecidable for Turing-complete programming languages. Nonetheless, recent progress has lead to heuristic verification methods, such as the ones based on predicate abstraction with counterexample-guided refinement (CEGAR), that work surprisingly well in practice. In this paper, we investigate the effectiveness of the small refinement heuristic which, for abstraction refinement in CEGAR, uses (the predicates in) a small proof of the given counterexample’s spuriousness [3, 12, 17, 22]. Such a method has shown to be quite effective in practice but thus far lacked a theoretical backing. Our core result is that the heuristic guarantees certain bounds on the number of CEGAR iterations, relative to the size of a proof for the input program.

A Proofs of the Main Results

1.1 A.1 Proof of Theorem 1

The following lemma states that each run of the refinement process returns a new proof.

Lemma 1

(Progress). Suppose $\mathsf {Abs}(P,F)$ returns $\pi$ and $\mathsf {Ref}(\pi )$ returns $F'$. Then, $F' \not \subseteq F$.

Proof

Because $\mathsf {Ref}(\pi )$ returns $F'$, we have $F' \vdash \pi$. Also, because $\mathsf {Abs}(P,F)$ returns $\pi$, we have $\mathsf {Abs}(\pi ,F) \ne \mathsf {safe}$, and so $F \not \vdash \pi$. Therefore, $F \ne F'$, and by the monotonicity of $\vdash$, it follows that $F' \not \subseteq F$.     $\square$

Theorem 1

Let the proof size metric be generic. Then, $\textsc {CegVerif}_{\textsc {SR}}{}$ converges in at most $\textit{exp}( mpfsize (P))$ many CEGAR iterations given a program P.

Proof

Let $F_P$ be a proof of P such that $mpfsize (P) = { size}(F_P)$. Then, for any counterexample $\pi$ of P (i.e., $\mathsf {Abs}(P,F')$ returns $\pi$ for some $F'$), $F_P \vdash \pi$, and therefore $mpfsize (\pi ) \le { size}(F_P)$. There are at most $c^{f( mpfsize (P))}$ proofs of size at most $f({ size}(F_P))$. Therefore, by Lemma 1, $\textsc {CegVerif}_{\textsc {SR}}$ converges in at most $\textit{exp}( mpfsize (P))$ many CEGAR iterations.     $\square$

1.2 A.2 Proof of Theorem 10

Theorem

10. Let the proof size metric be syntactic, $\vdash _{ cfg}$ be the proof relation, and $\mathsf {Abs}^{ crt}$ be the abstraction process with $\mathsf {cex}^{ syn}$ as the counterexample generator. Then, $\textsc {CegVerif}_{\textsc {SR}}$ converges in $\textit{poly}( mpfsize (P))^{{ maxhds}(P)}$ many CEGAR iterations given a CFG program P.

Proof

Let $F_P$ be a proof of $P = (G,T,v_{ ini},v_{ err})$ such that $mpfsize (P) = { size}(F_P)$. Then, for any counterexample $\pi$ of P (i.e., $\mathsf {Abs}^{ crt}(P,F)$ returns $\pi$ for some F), $F_P \vdash \pi$, and so $mpfsize (\pi ) \le { size}(F_P)$. Let ${ lim} = (f({ size}(F_P))+2)^{{ maxhds}(P)}+1$ (“$+2$” accounts for $\{{\bot ,\top }\}$). Let $\pi _{ lim}$ be a counterexample in $\mathsf {cex}^{ syn}(P)$ obtained by unfolding the loops at least ${ lim}$ many times. We show that for any F such that $F \vdash _{ cfg}\pi _{ lim}$ and ${ size}(F) \le f({ size}(F_P))$, $F \vdash _{ crt}P$. (We call such a counterexample $\pi _{ lim}$ sufficiently large.) Then, the result follows from the fact that there are only ${ lim}$ many counterexamples in $\mathsf {cex}^{ syn}(P)$ that are obtained by unfolding the loops at most ${ lim}$ many times, and the fact that no counterexample is returned more than once by the refinement process in a run of $\textsc {CegVerif}_{\textsc {SR}}$.

Let $\pi _{ lim}= (G_{ lim},T_{ lim},v_{ ini},v_{ err})$. Let $\sigma _{ lim}: v (G_{ lim})\rightarrow F\cup \{{\bot ,\top }\}$ be such that ${ size}(F) \le f({ size}(F_P))$ and $\sigma _{ lim}\vdash _{ cfg}\pi _{ lim}$. Let $k \ge { lim}$ be the number of times the loops are unfolded in $\pi _{ lim}$. We construct $\sigma : v (G)\rightarrow F^{\wedge \vee }$ such that $\sigma \vdash _{ crt}^F P$ by “folding” the unfolded loops in a bottom up manner. We initialize $\sigma = \sigma _{ lim}$, and $\gamma = \pi _{ lim}$. We iteratively fold $\gamma$ from leaf loops, while maintaining the property that $\sigma \vdash _{ crt}^F \gamma$. Then, the result follows from the fact that $\gamma$ becomes P at the root of the folding process.

Let $\gamma$ and $\sigma$ be the current CFG and its Cartesian predicate abstraction node-wise inductive invariant (i.e., $\sigma \vdash _{ crt}^F \gamma$). Let $\gamma = C[t_1' \cup t_2' \cup \dots t_k' \cup {t_k}\mathord \setminus \{{\varvec{h}_k}\})]$ where $\varvec{h}_0 = \varvec{h}$, $t_0 = t$, $\varvec{h}_{i+1} = `(\varvec{h}_i)$, ${t_i}' = t_i\langle { v (t_i)\mathord \setminus \{{\varvec{h}_i}\}, sc (t_i)}\rangle$, and $t_{i+1} = t_i\langle {\varvec{h}_i,\varvec{h}_i}\rangle$ for each $i \in \{{0,\dots ,k-1}\}$. Let $\gamma ' = C[{ loop}\;({\varvec{h}})\;{t}]$. That is, $\gamma '$ is obtained by folding the unfolded loop of $\gamma$. We construct $\sigma ': v (\gamma ')\rightarrow F^{\wedge \vee }$ such that $\sigma ' \vdash _{ crt}^F \gamma '$ as follows. Because $\mathop {\textit{ran}}\nolimits (\sigma _{lim}) = F\cup \{{\bot ,\top }\}$, for some $1 \le i_1 < i_2 \le (f({ size}(F_P))+2)^{|\varvec{h}|}+1 \le k$, $\sigma _{ lim}(h^{i_1}) = \sigma _{ lim}(h^{i_2})$ for each $h \in \{{\varvec{h}}\}$. By construction, $\sigma _{ lim}(h^i) = \sigma (h^i)$ for each $i \in \{{1,\dots ,k}\}$ and $h \in \{{\varvec{h}}\}$. We set $\sigma '(v) = \bigvee _{i=1}^{i_2} \sigma (v^i)$ for each $v \in sc (t)$, and $\sigma '(v) = \sigma (v)$ for each $v \in v (\gamma )\setminus sc (t)$. Then, it can be seen that $\sigma ' \vdash _{ crt}^F \gamma '$.     $\square$

1.3 A.3 Proof of Theorem 11

Theorem

11. Let the proof size metric be generic, $\vdash _{ cfg}$ be the proof relation, and $\mathsf {Abs}^{ crt}$ be the abstraction process with $\mathsf {cex}^{ syn}$ as the counterexample generator. Then, there exists a CFG program P with a proof $F \vdash _{ cfg}P$ on which $\textsc {CegVerif}_{\textsc {SR}}$ may take $\textit{exp}({ size}(F))$ many CEGAR iterations to converge.

Proof

We show that $P_{ ex}$ from Fig. 4 is such a program. As remarked before, we have $F_{ inv} \vdash _{ cfg}P_{ ex}$ where $F_{ inv} = \{a = b \Rightarrow y = x + z\}$. For each $k > 0$, let $\pi _k \in \mathsf {cex}^{ syn}(P_{ ex})$ be the counterexample obtained by unfolding each loop k times, and let $F_k = \{{\bigvee F \mid F \subseteq \{{\phi _i \mid 0 \le i \le k}\}}\}$ where $\phi _i \equiv x = a \wedge y=b+i \wedge z=i$. Then, $F_k \vdash _{ cfg}\pi _k$ for each $k > 0$, and $\bigcup _{i=1}^j F_i \not \vdash _{ crt}\pi _k$ for each $k > j > 0$.

Let $\ell > 0$. Define ${ size}$ as follows: ${ size}(F_{ inv}) = \ell$, ${ size}(F_k) = \lfloor \log k\rfloor$ for $k \in \{{1,\dots ,2^\ell -1}\}$, and ${ size}(F') = \ell +{ syntactic}(F')$ for $F' \in \mathcal {P}(\mathcal {T})\setminus (\{{F_{ inv}}\} \cup \{{F_k \mid 1 \le k \le 2^\ell -1}\})$ where ${ syntactic}(F')$ is the syntactic size of $F'$. Note that ${ size}$ is a generic proof size metric. Then, $\textsc {CegVerif}_{\textsc {SR}}$ takes $2^{{ size}(F_{ inv})}$ iterations to converge when $\mathsf {Abs}^{ crt}$ returns the counterexamples $\pi _1,...\pi _{2^\ell -1}$ in the first $2^\ell -1$ iterations and $\mathsf {Ref}$ returns the proofs $F_1,\dots ,F_{2^\ell -1}$ before returning $F_{ inv}$.     $\square$