Counterexample-Guided Quantifier Instantiation for Synthesis in SMT

Abstract

We introduce the first program synthesis engine implemented inside an SMT solver. We present an approach that extracts solution functions from unsatisfiability proofs of the negated form of synthesis conjectures. We also discuss novel counterexample-guided techniques for quantifier instantiation that we use to make finding such proofs practically feasible. A particularly important class of specifications are single-invocation properties, for which we present a dedicated algorithm. To support syntax restrictions on generated solutions, our approach can transform a solution found without restrictions into the desired syntactic form. As an alternative, we show how to use evaluation function axioms to embed syntactic restrictions into constraints over algebraic datatypes, and then use an algebraic datatype decision procedure to drive synthesis. Our experimental evaluation on syntax-guided synthesis benchmarks shows that our implementation in the CVC4 SMT solver is competitive with state-of-the-art tools for synthesis.

This work is supported in part by the European Research Council (ERC) Project Implicit Programming and Swiss National Science Foundation Grant Constraint Solving Infrastructure for Program Analysis.

This paper is dedicated to the memory of Morgan Deters who died unexpectedly in Jan 2015.

1 Introduction

The synthesis of functions that meet a given specification is a long-standing fundamental goal that has received great attention recently. This functionality directly applies to the synthesis of functional programs [17, 18] but also translates to imperative programs through techniques that include bounding input space, verification condition generation, and invariant discovery [28–30]. Function synthesis is also an important subtask in the synthesis of protocols and reactive systems, especially when these systems are infinite-state [3, 27]. The SyGuS format and competition [1, 2, 22] inspired by the success of the SMT-LIB and SMT-COMP efforts [5], has significantly improved and simplified the process of rigorously comparing different solvers on synthesis problems.

Connection between synthesis and theorem proving was established already in early work on the subject [12, 20]. It is notable that early research [20] found that the capabilities of theorem provers were the main bottleneck for synthesis. Taking lessons from automated software verification, recent work on synthesis has made use of advances in theorem proving, particularly in SAT and SMT solvers. However, that work avoids formulating the overall synthesis task as a theorem proving problem directly. Instead, existing work typically builds custom loops outside of an SMT or SAT solver, often using numerous variants of counterexample-guided synthesis. A typical role of the SMT solver has been to validate candidate solutions and provide counterexamples that guide subsequent search, although approaches such as symbolic term exploration [15] also use an SMT solver to explore a representation of the space of solutions. In existing approaches, SMT solvers thus receive a large number of separate queries, with limited communication between these different steps.

Contributions. In this paper, we revisit the formulation of the overall synthesis task as a theorem proving problem. We observe that SMT solvers already have some of the key functionality for synthesis; we show how to improve existing algorithms and introduce new ones to make SMT-based synthesis competitive. Specifically, we do the following.

• We show how to formulate an important class of synthesis problems as the problem of disproving universally quantified formulas, and how to synthesize functions automatically from selected instances of these formulas.

• We present counterexample-guided techniques for quantifier instantiation, which are crucial to obtain competitive performance on synthesis tasks.

• We discuss techniques to simplify the synthesized functions, to help ensure that they are small and adhere to specified syntactic requirements.

• We show how to encode syntactic restrictions using theories of algebraic datatypes and axiomatizable evaluation functions.

• We show that for an important class of single-invocation properties, the synthesis of functions from relations, the implementation of our approach in CVC4 significantly outperforms leading tools from the SyGuS competition.

Preliminaries. Since synthesis involves finding (and so proving the existence) of functions, we use notions from many-sorted second-order logic to define the general problem. We fix a set $\mathbf {S}$ of sort symbols and an (infix) equality predicate $\approx$ of type $\sigma \times \sigma$ for each $\sigma \in \mathbf {S}$. For every non-empty sort sequence $\mathbf {\sigma }\in \mathbf {S}^+$ with $\mathbf {\sigma }= \sigma _1 \cdots \sigma _n\sigma$, we fix an infinite set $\mathbf {X}_{\mathbf {\sigma }}$ of variables $x^{\sigma _1 \cdots \sigma _n \sigma }$ of type $\sigma _1 \times \cdots \times \sigma _n \rightarrow \sigma$. For each sort $\sigma$ we identity the type $() \rightarrow \sigma$ with $\sigma$ and call it a first-order type. We assume the sets $\mathbf {X}_{\mathbf {\sigma }}$ are pairwise disjoint and let $\mathbf {X}$ be their union. A signature $\varSigma$ consists of a set $\varSigma ^\mathrm {s} \subseteq \mathbf {S}$ of sort symbols and a set $\varSigma ^\mathrm {f}$ of function symbols $f^{\sigma _1 \cdots \sigma _n \sigma }$ of type $\sigma _1 \times \cdots \times \sigma _n \rightarrow \sigma$, where $n \ge 0$ and $\sigma _1, \ldots , \sigma _n, \sigma \in \varSigma ^\mathrm {s}$. We drop the sort superscript from variables or function symbols when it is clear from context or unimportant. We assume that signatures always include a Boolean sort $\mathsf {Bool}$ and constants $\top$ and $\bot$ of type $\mathsf {Bool}$ (respectively, for true and false). Given a many-sorted signature $\varSigma$ together with quantifiers and lambda abstraction, the notion of well-sorted ($\varSigma$-)term, atom, literal, clause, and formula with variables in $\mathbf {X}$ are defined as usual in second-order logic. All atoms have the form $s \approx t$. Having $\approx$ as the only predicate symbol causes no loss of generality since we can model other predicate symbols as function symbols with return sort $\mathsf {Bool}$. We will, however, write just t in place of the atom $t \approx \top$, to simplify the notation. A $\varSigma$-term/formula is ground if it has no variables, it is first-order if it has only first-order variables, that is, variables of first-order type. When ${\varvec{x}} = (x_1,\ldots ,x_n)$ is a tuple of variables and Q is either $\forall$ or $\exists$, we write $Q \varvec{x}\, \varphi$ as an abbreviation of $Q x_1 \cdots Q x_n\, \varphi$. If e is a $\varSigma$-term or formula and ${\varvec{x}} = (x_1,\ldots ,x_n)$ has no repeated variables, we write $e[\varvec{x}]$ to denote that all of e’s free variables are from $\varvec{x}$; if ${\varvec{t}} = (t_1,\ldots ,t_n)$ is a term tuple, we write $e[\varvec{t}]$ for the term or formula obtained from e by simultaneously replacing, for all $i=1,\ldots ,n$, every occurrence of $x_i$ in e by $t_i$. A $\varSigma$ -interpretation $\mathcal {I}$ maps: each $\sigma \in \varSigma ^\mathrm {s}$ to a non-empty set $\sigma ^\mathcal {I}$, the domain of $\sigma$ in $\mathcal {I}$, with $\mathsf {Bool}^\mathcal {I}= \{\top , \bot \}$; each $u^{\sigma _1 \cdots \sigma _n\sigma } \in \mathbf {X}\cup \varSigma ^\mathrm {f}$ to a total function $u^\mathcal {I}: \sigma _1^\mathcal {I}\times \cdots \times \sigma _n^\mathcal {I}\rightarrow \sigma ^\mathcal {I}$ when $n > 0$ and to an element of $\sigma ^\mathcal {I}$ when $n = 0$. The interpretation $\mathcal {I}$ induces as usual a mapping from terms t of sort $\sigma$ to elements $t^\mathcal {I}$ of $\sigma ^\mathcal {I}$. If $x_1, \ldots ,x_n$ are variables and $v_1,\ldots ,v_n$ are well-typed values for them, we denote by $\mathcal {I}[x_1 \mapsto v_1, \ldots , x_n \mapsto v_n]$ the $\varSigma$-interpretation that maps each $x_i$ to $v_i$ and is otherwise identical to $\mathcal {I}$. A satisfiability relation between $\varSigma$-interpretations and $\varSigma$-formulas is defined inductively as usual.

A theory is a pair $T = (\varSigma , \mathbf {I})$ where $\varSigma$ is a signature and $\mathbf {I}$ is a non-empty class of $\varSigma$-interpretations, the models of T, that is closed under variable reassignment (i.e., every $\varSigma$-interpretation that differs from one in $\mathbf {I}$ only in how it interprets the variables is also in $\mathbf {I}$) and isomorphism. A $\varSigma$-formula $\varphi [\varvec{x}]$ is T -satisfiable (resp., T -unsatisfiable) if it is satisfied by some (resp., no) interpretation in $\mathbf {I}$. A satisfying interpretation for $\varphi$ models (or is a model of) $\varphi$. A formula $\varphi$ is T -valid, written $\models _{T}\varphi$, if every model of T is a model of $\varphi$. Given a fragment $\mathbf {L}$ of the language of $\varSigma$-formulas, a $\varSigma$-theory T is satisfaction complete with respect to $\mathbf {L}$ if every T-satisfiable formula of $\mathbf {L}$ is T-valid. In this paper we will consider only theories that are satisfaction complete wrt the formulas we are interested in. Most theories used in SMT (in particular, all theories of a specific structure such various theories of the integers, reals, strings, algebraic datatypes, bit vectors, and so on) are satisfaction complete with respect to the class of closed first-order $\varSigma$-formulas. Other theories, such as the theory of arrays, are satisfaction complete only with respect to considerably more restricted classes of formulas.

2 Synthesis Inside an SMT Solver

We are interested in synthesizing computable functions automatically from formal logical specifications stating properties of these functions. As we show later, under the right conditions, we can formulate a version of the synthesis problem in first-order logic alone, which allows us to tackle the problem using SMT solvers.

We consider the synthesis problem in the context of some theory T of signature $\varSigma$ that allows us to provide the function’s specification as a $\varSigma$-formula. Specifically, we consider synthesis conjectures expressed as (well-sorted) formulas of the form

$$\begin{aligned} \exists f^{\sigma _1 \cdots \sigma _n\sigma } \, \forall x_1^{\sigma _1} \, \cdots \, \forall x_n^{\sigma _n} \, P[f, x_1, \ldots , x_n] \end{aligned}$$

(1)

or $\exists f\, \forall \varvec{x}\, P[f, \varvec{x}]$, for short, where the second-order variable f represents the function to be synthesized and P is a $\varSigma$-formula encoding properties that f must satisfy for all possible values of the input tuple $\varvec{x} = (x_1,\ldots ,x_n)$. In this setting, finding a witness for this satisfiability problem amounts to finding a function of type $\sigma _1 \times \cdots \times \sigma _n \rightarrow \sigma$ in some model of T that satisfies $\forall \varvec{x}\, P[f, \varvec{x}]$. Since we are interested in automatic synthesis, we the restrict ourselves here to methods that search over a subspace S of solutions representable syntactically as $\varSigma$-terms. We will say then that a synthesis conjecture is solvable if it has a syntactic solution in S.

In this paper we present two approaches that work with classes $\mathbf {L}$ of synthesis conjectures and $\varSigma$-theories T that are satisfaction complete wrt $\mathbf {L}$. In both approaches, we solve a synthesis conjecture $\exists f\, \forall \varvec{x}\, P[f, \varvec{x}]$ by relying on quantifier-instantiation techniques to produce a first-order $\varSigma$-term $t[\varvec{x}]$ of sort $\sigma$ such that $\forall \varvec{x}\, P[t, \varvec{x}]$ is T-satisfiable. When this t is found, the synthesized function is denoted by $\lambda \varvec{x}.\, t$ .

In principle, to determine the satisfiability of $\exists f\, \forall \varvec{x}\, P[f, \varvec{x}]$ an SMT solver supporting the theory T can consider the satisfiability of the (open) formula $\forall \varvec{x}\, P[f, \varvec{x}]$ by treating f as an uninterpreted function symbol. This sort of Skolemization is not usually a problem for SMT solvers as many of them can process formulas with uninterpreted symbols. The real challenge is the universal quantification over $\varvec{x}$ because it requires the solver to construct internally (a finite representation of) an interpretation of f that is guaranteed to satisfy $P[f, \varvec{x}]$ for every possible value of $\varvec{x}$ [11, 24].

More traditional SMT solver designs to handle universally quantified formulas have focused on instantiation-based methods to show unsatisfiability. They generate ground instances of those formulas until a refutation is found at the ground level [10]. While these techniques are incomplete in general, they have been shown to be quite effective in practice [9, 25]. For this reason, we advocate approaches to synthesis geared toward establishing the unsatisfiability of the negation of the synthesis conjecture:

$$\begin{aligned} \forall f\,\exists \varvec{x}\, \lnot P[f, \varvec{x}] \end{aligned}$$

(2)

Thanks to our restriction to satisfaction complete theories, (2) is T-unsatisfiable exactly when the original synthesis conjecture (1) is T-satisfiable.¹ Moreover, as we explain in this paper, a syntactic solution $\lambda x.\,t$ for (1) can be constructed from a refutation of (2), as opposed to being extracted from the valuation of f in a model of $\forall \varvec{x}\, P[f, \varvec{x}]$.

Two Synthesis Methods. Proving (2) unsatisfiable poses its own challenge to current SMT solvers, namely, dealing with the second-order universal quantification of f. To our knowledge, no SMT solvers so far had direct support for higher-order quantification. In the following, however, we describe two specialized methods to refute negated synthesis conjectures like (2) that build on existing capabilities of these solvers.

The first method applies to a restricted, but fairly common, case of synthesis problems $\exists f\, \forall \varvec{x}\, P[f, \varvec{x}]$ where every occurrence of f in P is in terms of the form $f(\varvec{x})$. In this case, we can express the problem in the first-order form $\forall \varvec{x}. \exists y. Q[\varvec{x},y]$ and then tackle its negation using appropriate quantifier instantiation techniques.

The second method follows the syntax-guided synthesis paradigm  [1, 2] where the synthesis conjecture is accompanied by an explicit syntactic restriction on the space of possible solutions. Our syntax-guided synthesis method is based on encoding the syntax of terms as first-order values. We use a deep embedding into an extension of the background theory T with a theory of algebraic data types, encoding the restrictions of a syntax-guided synthesis problem.

For the rest of the paper, we fix a $\varSigma$ -theory T and a class $\mathbf {P}$ of quantifier-free $\varSigma$ -formulas $P[f,\varvec{x}]$ such that T is satisfaction complete with respect to the class of synthesis conjectures $\mathbf {L}:= \{\exists f\, \forall \varvec{x}\, P[f, \varvec{x}] \mid P \in \mathbf {P}\}$ .

3 Refutation-Based Synthesis

When axiomatizing properties of a desired function f of type $\sigma _1 \times \cdots \times \sigma _n \rightarrow \sigma$, a particularly well-behaved class are single-invocation properties (see, e.g., [13]). These properties include, in particular, standard function contracts, so they can be used to synthesize a function implementation given its postcondition as a relation between the arguments and the result of the function. This is also the form of the specification for synthesis problems considered in complete functional synthesis [16–18]. Note that, in our case, we aim to prove that the output exists for all inputs, as opposed to, more generally, computing the set of inputs for which the output exists.

A single-invocation property is any formula of the form $Q[\varvec{x}, f(\varvec{x})]$ obtained as an instance of a quantifier-free formula $Q[\varvec{x}, y]$ not containing f. Note that the only occurrences of f in $Q[\varvec{x}, f(\varvec{x})]$ are in subterms of the form $f(\varvec{x})$ with the same tuple $\varvec{x}$ of pairwise distinct variables.² The conjecture $\exists f\, \forall {\varvec{x}}\, Q[\varvec{x}, f(\varvec{x})]$ is logically equivalent to the first-order formula

$$\begin{aligned} \forall {\varvec{x}}\, \exists y\, Q[{\varvec{x}}, y] \end{aligned}$$

(3)

By the semantics of $\forall$ and $\exists$, finding a model $\mathcal {I}$ for it amounts (under the axioms of choice) to finding a function $h:\sigma _1^\mathcal {I}\times \cdots \times \sigma _n^\mathcal {I}\rightarrow \sigma ^\mathcal {I}$ such that for all ${\varvec{s}} \in \sigma _1^\mathcal {I}\times \cdots \times \sigma _n^\mathcal {I}$, the interpretation $\mathcal {I}[{\varvec{x}} \mapsto {\varvec{s}}, y \mapsto h({\varvec{s}})]$ satisfies $Q[{\varvec{x}}, y]$. This section considers the case when $\mathbf {P}$ consists of single-invocation properties and describes a general approach for determining the satisfiability of formulas like (3) while computing a syntactic representation of a function like h in the process. For the latter, it will be convenient to assume that the language of functions contains an if-then-else operator $\mathsf {ite}$ of type $\mathsf {Bool}\times \sigma \times \sigma \rightarrow \sigma$ for each sort $\sigma$, with the usual semantics.

If (3) belongs to a fragment that admits quantifier elimination in T, such as the linear fragment of integer arithmetic, determining its satisfiability can be achieved using an efficient method for quantifier elimination [7, 21]. Such cases have been examined in the context of software synthesis [17]. Here we propose instead an alternative instantiation-based approach aimed at establishing the unsatisfiability of the negated form of (3):

$$\begin{aligned} \exists {\varvec{x}}\, \forall y\, \lnot Q[{\varvec{x}}, y] \end{aligned}$$

(4)

or, equivalently, of a Skolemized version $\forall y\, \lnot Q[\mathbf {\mathsf {k}}, y]$ of  (4) for some tuple $\mathbf {\mathsf {k}}$ of fresh uninterpreted constants of the right sort. Finding a T-unsatisfiable finite set $\varGamma$ of ground instances of $\lnot Q[{\varvec{k}}, y]$, which is what an SMT solver would do to prove the unsatisfiability of (4), suffices to solve the original synthesis problem. The reason is that, then, a solution for f can be constructed directly from $\varGamma$, as indicated by the following result.

Proposition 1

Suppose some set $\varGamma = \{\lnot Q[\mathbf {\mathsf {k}}, t_1[\mathbf {\mathsf {k}}]], \ldots , \lnot Q[\mathbf {\mathsf {k}}, t_p[\mathbf {\mathsf {k}}]]\}$ where $t_1[{\varvec{x}}]$, $\ldots$, $t_p[{\varvec{x}}]$ are $\varSigma$-terms of sort $\sigma$ is T-unsatisfiable. One solution for $\exists f\, \forall {\varvec{x}}\, Q[{\varvec{x}}, f({\varvec{x}})]$ is $\lambda {\varvec{x}}.\, \mathsf {ite}( Q[{\varvec{x}}, t_p], t_p, (\,\cdots \, \mathsf {ite}( Q[{\varvec{x}}, t_2], t_2, t_1 ) \,\cdots \, ))$.

Example 1

Let T be the theory of linear integer arithmetic with the usual signature and integer sort $\mathsf {Int}$. Let ${\varvec{x}} = (x_1, x_2)$. Now consider the property

$$\begin{aligned} P[f, {\varvec{x}}] {:}{=} f( {\varvec{x}} ) \ge x_1 \wedge f( {\varvec{x}} ) \ge x_2 \wedge ( f( {\varvec{x}} ) \approx x_1 \vee f( {\varvec{x}} ) \approx x_2 ) \end{aligned}$$

(5)

with f of type $\mathsf {Int}\times \mathsf {Int}\rightarrow \mathsf {Int}$ and $x_1, x_2$ of type $\mathsf {Int}$. The synthesis problem $\exists f\, \forall {\varvec{x}}\, P[f, {\varvec{x}}]$ is solved exactly by the function that returns the maximum of its two inputs. Since P is a single-invocation property, we can solve that problem by proving the T-unsatisfiability of the conjecture $\exists {\varvec{x}}\, \forall y\, \lnot Q[{\varvec{x}}, y]$ where

$$\begin{aligned} Q[{\varvec{x}}, y] \; {:}{=} \; y \ge x_1 \wedge y \ge x_2 \wedge ( y \approx x_1 \vee y \approx x_2 ) \end{aligned}$$

(6)

After Skolemization the conjecture becomes $\forall y\, \lnot Q[\mathbf {\mathsf {a}}, y]$ for fresh constants $\mathbf {\mathsf {a}} = (\mathsf {a}_1, \mathsf {a}_2)$. When asked to determine the satisfiability of that conjecture an SMT solver may, for instance, instantiate it with $\mathsf {a}_1$ and then $\mathsf {a}_2$ for y, producing the T-unsatisfiable set $\{\lnot Q[\mathbf {\mathsf {a}}, \mathsf {a}_1], \lnot Q[\mathbf {\mathsf {a}}, \mathsf {a}_2]\}$. By Proposition 1, one solution for $\forall {\varvec{x}}\, P[f, {\varvec{x}}]$ is $f = \lambda {\varvec{x}}.\, \mathsf {ite}( Q[{\varvec{x}}, x_2], x_2, x_1 )$, which simplifies to $\lambda {\varvec{x}}.\, \mathsf {ite}( x_2 \ge x_1, x_2, x_1 )$, representing the desired maximum function. $\blacksquare$

Fig. 1.

A refutation-based synthesis procedure for single-invocation property $\exists f\, \forall {\varvec{x}}\, Q[{\varvec{x}}, f({\varvec{x}})]$.

Synthesis by Counterexample-Guided Quantifier Instantiation. Given Proposition 1, the main question is how to get the SMT solver to generate the necessary ground instances from $\forall y\, \lnot Q[\mathbf {\mathsf {k}}, y]$. Typically, SMT solvers that reason about quantified formulas use heuristic quantifier instantiation techniques based on E-matching [9], which instantiates universal quantifiers with terms occurring in some current set of ground terms built incrementally from the input formula. Using E-matching-based heuristic instantiation alone is unlikely to be effective in synthesis, where required terms need to be synthesized based on the semantics of the input specification. This is confirmed by our preliminary experiments, even for simple conjectures. We have developed instead a specialized new technique, which we refer to as counterexample-guided quantifier instantiation, that allows the SMT solver to quickly converge in many cases to the instantiations that refute the negated synthesis conjecture (4).

The new technique is similar to a popular scheme for synthesis known as counterexample-guided inductive synthesis, implemented in various synthesis approaches (e.g., [14, 29]), but with the major difference of being built-in directly into the SMT solver. The technique is illustrated by the procedure in Fig. 1, which grows a set $\varGamma$ of ground instances of $\lnot Q[\mathbf {\mathsf {k}}, y]$ starting with the formula $\mathsf {G} \Rightarrow Q[\mathbf {\mathsf {k}}, \mathsf {e}]$ where $\mathsf {G}$ and $\mathsf {e}$ are fresh constants of sort $\mathsf {Bool}$ and $\sigma$, respectively. Intuitively, $\mathsf {e}$ represents a current, partial solution for the original synthesis conjecture $\exists f\, \forall {\varvec{x}}\, Q[{\varvec{x}}, f({\varvec{x}})]$, while $\mathsf {G}$ represents the possibility that the conjecture has a (syntactic) solution in the first place.

The procedure, which may not terminate in general, terminates either when $\varGamma$ becomes unsatisfiable, in which case it has found a solution, or when $\varGamma$ is still satisfiable but all of its models falsify $\mathsf {G}$, in which case the search for a solution was inconclusive. The procedure is not solution-complete, that is, it is not guaranteed to return a solution whenever there is one. However, thanks to Proposition 1, it is solution-sound: every $\lambda$-term it returns is indeed a solution of the original synthesis problem.

Finding Instantiations. The choice of the term t in Step 2 of the procedure is intentionally left under Specified because it can be done in a number of ways. Having a good heuristic for such instantiations is, however, critical to the effectiveness of the procedure in practice. In a $\varSigma$-theory T, like integer arithmetic, with a fixed interpretation for symbols in $\varSigma$ and a distinguished set of ground $\varSigma$-terms denoting the elements of a sort, a simple, if naive, choice for t in Fig. 1 is the distinguished term denoting the element ${\mathsf {e}}^\mathcal {I}$. For instance, if $\sigma$ is $\mathsf {Int}$ in integer arithmetic, t could be a concrete integer constant ($0,\pm 1, \pm 2, \ldots$). This choice amounts to testing whether points in the codomain of the sought function f satisfy the original specification P.

More sophisticated choices for t, in particular where t contains the variables ${\varvec{x}}$, may increase the generalization power of this procedure and hence its ability to find a solution. For instance, our present implementation in the cvc4 solver relies on the fact that the model $\mathcal {I}$ in Step 2 is constructed from a set of equivalence classes over terms computed by the solver during its search. The procedure selects the term t among those in the equivalence class of e, other than e itself. For instance, consider formula (6) from the previous example that encodes the single-invocation form of the specification for the max function. The DPLL(T) architecture, on which cvc4 is based, finds a model for $Q[\mathbf {\mathsf {a}}, \mathsf {e} ]$ with $\mathbf {\mathsf {a}} = (\mathsf {a}_1, \mathsf {a}_2)$ only if it can first find a subset M of that formula’s literals that collectively entail $Q[ \mathbf {\mathsf {a}}, \mathsf {e} ]$ at the propositional level. Due to the last conjunct of (6), M must include either $\mathsf {e} \approx \mathsf {a}_1$ or $\mathsf {e} \approx \mathsf {a}_2$. Hence, whenever a model can be constructed for $Q[ \mathbf {\mathsf {a}}, e ]$, the equivalence class containing e must contain either $\mathsf {a}_1$ or $\mathsf {a}_2$. Thus using the above selection heuristic, the procedure in Fig. 1 will, after at most two iterations of the loop in Step 2, add the instances $\lnot Q[ \mathbf {\mathsf {a}}, \mathsf {a}_1 ]$ and $\lnot Q[ \mathbf {\mathsf {a}}, \mathsf {a}_2 ]$ to $\varGamma$. As noted in Example 1, these two instances are jointly T-unsatisfiable. We expect that more sophisticated instantiation techniques can be incorporated. In particular, both quantifier elimination techniques [7, 21] and approaches currently used to infer invariants from templates [8, 19] are likely to be beneficial for certain classes of synthesis problems. The advantage of developing these techniques within an SMT solver is that they directly benefit both synthesis and verification in the presence of quantified conjectures, thus fostering cross-fertilization between different fields.

4 Refutation-Based Syntax-Guided Synthesis

In syntax-guided synthesis, the functional specification is strengthened by an accompanying set of syntactic restrictions on the form of the expected solutions. In a recent line of work [1, 2, 22] these restrictions are expressed by a grammar R (augmented with a kind of let binder) defining the language of solution terms, or programs, for the synthesis problem. In this section, we present a variant of the approach in the previous section that incorporates the syntactic restriction directly into the SMT solver via a deep embedding of the syntactic restriction R into the solver’s logic. The main idea is to represent R as a set of algebraic datatypes and build into the solver an interpretation of these datatypes in terms of the original theory T.

While our approach is parametric in the background theory T and the restriction R, it is best explained here with a concrete example.

Fig. 2.

Axiomatization of the evaluation operators in grammar R from Example 2.

Example 2

Consider again the synthesis conjecture (6) from Example 1 but now with a syntactic restriction R for the solution space expressed by these algebraic datatypes:

$$\begin{aligned} \begin{array}{l@{\quad }l@{\quad }l} \mathsf {S} &{} := &{} \mathsf {x}_1 \mid \mathsf {x}_2 \mid \mathsf {zero} \mid \mathsf {one} \mid \mathsf {plus}(\mathsf {S}, \mathsf {S}) \mid \mathsf {minus}(\mathsf {S}, \mathsf {S}) \mid \mathsf {if}( \mathsf {C}, \mathsf {S}, \mathsf {S} ) \\ \mathsf {C} &{} := &{} \mathsf {leq}(\mathsf {S}, \mathsf {S}) \mid \mathsf {eq}(\mathsf {S}, \mathsf {S}) \mid \mathsf {and}(\mathsf {C}, \mathsf {C}) \mid \mathsf {not}(\mathsf {C}) \end{array} \end{aligned}$$

The datatypes are meant to encode a term signature that includes nullary constructors for the variables $x_1$ and $x_2$ of (6), and constructors for the symbols of the arithmetic theory T. Terms of sort $\mathsf {S}$ (resp., $\mathsf {C}$) refer to theory terms of sort $\mathsf {Int}$ (resp., $\mathsf {Bool}$).

Instead of the theory of linear integer arithmetic, we now consider its combination $T_\mathrm {D}$ with the theory of the datatypes above extended with two evaluation operators, that is, two function symbols $\mathsf {ev}^{\mathsf {S} \times \mathsf {Int}\times \mathsf {Int}\rightarrow \mathsf {Int}}$ and $\mathsf {ev}^{\mathsf {C} \times \mathsf {Int}\times \mathsf {Int}\rightarrow \mathsf {Bool}}$ respectively embedding $\mathsf {S}$ in $\mathsf {Int}$ and $\mathsf {C}$ in $\mathsf {Bool}$. We define $T_\mathrm {D}$ so that all of its models satisfy the formulas in Fig. 2. The evaluation operators effectively define an interpreter for programs (i.e., terms of sort $\mathsf {S}$ and $\mathsf {C}$) with input parameters $x_1$ and $x_2$.

It is possible to instrument an SMT solver that supports user-defined datatypes, quantifiers and linear arithmetic so that it constructs automatically from the syntactic restriction R both the datatypes $\mathsf {S}$ and $\mathsf {C}$ and the two evaluation operators. Reasoning about $\mathsf {S}$ and $\mathsf {C}$ is done by the built-in subsolver for datatypes. Reasoning about the evaluation operators is achieved by reducing ground terms of the form $\mathsf {ev}(d, t_1, t_2)$ to smaller terms by means of selected instantiations of the axioms from Fig. 2, with a number of instances proportional to the size of term d. It is also possible to show that $T_\mathrm {D}$ is satisfaction complete with respect to the class

$$\begin{aligned} \mathbf {L}_2:= & {} \{ \exists g\, \forall {\varvec{z}}\, P[\lambda {\varvec{z}}.\, \mathsf {ev}(g, {\varvec{z}}),\, {\varvec{x}}] \mid P[f, {\varvec{x}}] \in \mathbf {P}\} \end{aligned}$$

where instead of terms of the form $f(t_1, t_2)$ in P we have, modulo $\beta$-reductions, terms of the form $\mathsf {ev}(g, t_1, t_2)$.³ For instance, the formula $P[f, {\varvec{x}}]$ in Eq. (5) from Example 1 can be restated in $T_\mathrm {D}$ as the formula below where g is a variable of type $\mathsf {S}$:

$$\begin{aligned} P_\mathsf {ev}[ g, {\varvec{x}} ]:= & {} \mathsf {ev}( g, {\varvec{x}} ) \ge x_1 \wedge \mathsf {ev}( g,{\varvec{x}} ) \ge x_2 \wedge ( \mathsf {ev}( g, {\varvec{x}} ) \approx x_1 \vee \mathsf {ev}( g,{\varvec{x}} ) \approx x_2 ) \end{aligned}$$

In contrast to $P[f, {\varvec{x}}]$, the new formula $P_\mathsf {ev}[ g, {\varvec{x}} ]$ is first-order, with the role of the second-order variable f now played by the first-order variable g.

When asked for a solution for (5) under the restriction R, the instrumented SMT solver will try to determine instead the $T_\mathrm {D}$-unsatisfiability of $\forall g\, \exists {\varvec{x}}\, \lnot P_\mathsf {ev}[g, {\varvec{x}}]$. Instantiating g in the latter formula with $s := \mathsf {if}( \mathsf {leq}(\mathsf {x}_1, \mathsf {x}_2), \mathsf {x}_2, \mathsf {x}_1 )$, say, produces a formula that the solver can prove to be $T_\mathrm {D}$-unsatisfiable. This suffices to show that the program $\mathsf {ite}(x_1 \le x_2, x_2, x_1)$, the analogue of s in the language of T, is a solution of the synthesis conjecture (5) under the syntactic restriction R.    $\blacksquare$

Fig. 3.

A refutation-based syntax-guided synthesis procedure for $\exists f\, \forall {\varvec{x}}\, P_\mathsf {ev}[f,{\varvec{x}}]$.

To prove the unsatisfiability of formulas like $\forall g\, \exists {\varvec{x}}\, \lnot P_\mathsf {ev}[g, {\varvec{x}}]$ in the example above we use a procedure similar to that in Sect. 3, but specialized to the extended theory $T_\mathrm {D}$. The procedure is described in Fig. 3. Like the one in Fig. 1, it uses an uninterpreted constant $\mathsf {e}$ representing a solution candidate, and a Boolean variable $\mathsf {G}$ representing the existence of a solution. The main difference, of course, is that now $\mathsf {e}$ ranges over the datatype representing the restricted solution space. In any model of $T_\mathrm {D}$, a term of datatype sort evaluates to a term built exclusively with constructor symbols. This is why the procedure returns in Step 2b the value of $\mathsf {e}$ in the model $\mathcal {I}$ found in Step 2a. As we showed in the previous example, a program that solves the original problem can then be reconstructed from the returned datatype term.

Fig. 4.

A run of the procedure from Fig. 3.

Implementation. We implemented the procedure in the cvc4 solver. Figure 4 shows a run of that implementation over the conjecture from Example 2. In this run, note that each model found for $\mathsf {e}$ satisfies all values of counterexamples found for previous candidates. After the sixth iteration of Step 2a, the procedure finds the candidate $\mathsf {if}( \mathsf {leq}(\mathsf {x}_1, \mathsf {x}_2), \mathsf {x}_2, \mathsf {x}_1 )$, for which no counterexample exists, indicating that the procedure has found a solution for the synthesis conjecture. Currently, this problem can be solved in about 0.5 s in the latest development version of cvc4.

To make the procedure practical it is necessary to look for small solutions to synthesis conjectures. A simple way to limit the size of the candidate solutions is to consider smaller programs before larger ones. Adapting techniques for finding finite models of minimal size [26], we use a strategy that starting, from $n = 0$, searches for programs of size $n+1$ only after its has exhausted the search for programs of size n. In solvers based on the DPLL(T) architecture, like cvc4, this can be accomplished by introducing a splitting lemma of the form $( \mathsf {size}( \mathsf {e} ) \le 0 \vee \lnot \mathsf {size}( \mathsf {e} ) \le 0 )$ and asserting $\mathsf {size}( \mathsf {e} ) \le 0$ as the first decision literal, where $\mathsf {size}$ is a function symbol of type $\sigma \rightarrow \mathsf {Int}$ for every datatype sort $\sigma$ and stands for the function that maps each datatype value to its term size (i.e., the number of non-nullary constructor applications in the term). We do the same for $\mathsf {size}( \mathsf {e} ) \le 1$ if and when $\lnot \mathsf {size}( \mathsf {e} ) \le 0$ becomes asserted. We extended the procedure for algebraic datatypes in cvc4  [6] to handle constraints involving $\mathsf {size}$. The extended procedure remains a decision procedure for input problems with a concrete upper bound on terms of the form $\mathsf {size}(u)$, for each variable or uninterpreted constant u of datatype sort in the problem. This is enough for our purposes since the only term u like that in our synthesis procedure is $\mathsf {e}$.

Proposition 2

With the search strategy above, the procedure in Fig. 3 has the following properties:

• 1. (Solution Soundness) Every term it returns can be mapped to a solution of the original synthesis conjecture $\exists f\,\forall {\varvec{x}}\, P[f, {\varvec{x}}]$ under the restriction R.

• 2. (Refutation Soundness) If it answers “no solution found”, the original conjecture has no solutions under the restriction R.

• 3. (Solution Completeness) If the original conjecture has a solution under R, the procedure will find one.

Note that by this proposition the procedure can diverge only if the input synthesis conjecture has no solution. We refer the reader to a longer version of this paper for a proof of Proposition 2 [23]. For a general idea, the proof of solution soundness is based on the observation that when the procedure terminates at Step 2b, $\varGamma$ has an unsatisfiable core with just one instance of $\lnot P[g, {\varvec{x}}]$. The procedure is refutation sound since when no model of $\varGamma$ in Step 2a satisfies $\mathsf {G}$, we have that even an arbitrary $\mathsf {e}$ cannot satisfy the current set of instances added to $\varGamma$ in Step 2b. Finally, the procedure is solution complete first of all because Step 2a and 2b are effective thanks to the decidability of the background theory $T_\mathrm {D}$. Each execution of Step 2a is guaranteed to produce a new candidate since $T_\mathrm {D}$ is also satisfaction complete. Thus, in the worst case, the procedure amounts to an enumeration of all possible programs until a solution is found.

5 Single Invocation Techniques for Syntax-Guided Problems

In this section, we considered the combined case of single-invocation synthesis conjectures with syntactic restrictions. Given a set R of syntactic restrictions expressed by a datatype $\mathsf {S}$ for programs and a datatype $\mathsf {C}$ for Boolean expressions, consider the case where (i) $\mathsf {S}$ contains the constructor $\mathsf {if} : \mathsf {C} \times \mathsf {S} \times \mathsf {S} \rightarrow \mathsf {S}$ (with the expected meaning) and (ii) the function to be synthesized is specified by a single-invocation property that can be expressed as a term of sort $\mathsf {C}$. This is the case for the conjecture from Example 2 where the property $P_\mathsf {ev}[g, {\varvec{x}}]$ can be rephrased as:

$$\begin{aligned} P_{\mathsf {C}}[g, {\varvec{x}}]:= & {} \mathsf {ev}( \mathsf {and}( \mathsf {leq}( \mathsf {x}_1, g ), \mathsf {and}( \mathsf {leq}( \mathsf {x}_2, g ), \mathsf {or}( \mathsf {eq}( g, \mathsf {x}_1 ), \mathsf {eq}( g, \mathsf {x}_2 ) ) ) ), {\varvec{x}}) \end{aligned}$$

(7)

where again g has type $\mathsf {S}$, ${\varvec{x}} = (x_1, x_2)$, and $x_1$ and $x_2$ have type $\mathsf {Int}$. The procedure in Fig. 1 can be readily modified to apply to this formula, with $P_{\mathsf {C}}[g, \mathbf {\mathsf {k}}]$ and g taking the role respectively of $Q[\mathbf {\mathsf {k}}, y]$ and y in that figure, since it generates solutions meeting our syntactic requirements. Running this modified procedure instead the one in Fig. 3 has the advantage that only the outputs of a solution need to be synthesized, not conditions in $\mathsf {ite}$-terms. However, in our experimental evaluation found that the overhead of using an embedding into datatypes for syntax-guided problems is significant with respect to the performance of the solver on problems with no syntactic restrictions. For this reason, we advocate an approach for single-invocation synthesis conjectures with syntactic restrictions that runs the procedure from Fig. 1 as is, ignoring the syntactic restrictions R, and subsequently reconstructs from its returned solution one satisfying the restrictions. For that it is useful to assume that terms t in T can be effectively reduced to some (T-equivalent and unique) normal form, which we denote by $t\!\downarrow$.

Say the procedure from Fig. 1 returns a solution $\lambda {\varvec{x}}.\, t$ for a function f. To construct from that a solution that meets the syntactic restrictions specified by datatype $\mathsf {S}$, we run the iterative procedure described in Fig. 5. This procedure maintains an evolving set A of triples of the form ( t, s, D ), where D is a datatype, t is a term in normal form, s is a term satisfying the restrictions specified by D. The procedure incrementally makes calls to the subprocedure $\mathrm {rcon}$, which takes a normal form term t, a datatype D and the set A above, and returns a pair ( s, U ) where s is a term equivalent to t in T, and U is a set of pairs $(s', D')$ where $s'$ is a subterm of s that fails to satisfy the syntactic restriction expressed by datatype $D'$. Overall, the procedure alternates between calling $\mathrm {rcon}$ and adding triples to A until $\mathrm {rcon}( t, D, A )$ returns a pair of the form $( s, \emptyset )$, in which case s is a solution satisfying the syntactic restrictions specified by $\mathsf {S}$.

Fig. 5.

A procedure for finding a term equivalent to t that meets the syntactic restrictions specified by datatype $\mathsf {S}$.

Example 3

Say we wish to construct a solution equivalent to $\lambda x_1\,x_2.\, x_1+(2*x_2)$ that meets restrictions specified by datatype $\mathsf {S}$ from Example 2. To do so, we let $A = \emptyset$, and call $\mathrm {rcon}((x_1+(2*x_2))\!\downarrow , \mathsf {S}, A )$. Since A is empty and $+$ is the analogue of constructor $\mathsf {plus}^{ \mathsf {S} \mathsf {S} \mathsf {S}}$ of $\mathsf {S}$, assuming $(x_1+(2*x_2))\!\downarrow \ = x_1+(2*x_2)$, we may choose to return a pair based on the result of calling $\mathrm {rcon}$ on $x_1\!\downarrow$ and $(2*x_2)\!\downarrow$. Since $\mathsf {x}_1^{\mathsf {S}}$ is a constructor of $\mathsf {S}$ and $x_1\!\downarrow \ = x_1$, $\mathrm {rcon}( x_1, \mathsf {S}, A )$ returns $( x_1, \emptyset )$. Since $\mathsf {S}$ does not have a constructor for $*$, we must either choose a term t such that $t\!\downarrow \ = (2*x_2)\!\downarrow$ where the topmost symbol of t is the analogue of a constructor in $\mathsf {S}$, or otherwise return the pair $( 2*x_2, \{ (2*x_2, \mathsf {S} ) \} )$. Suppose we do the latter, and thus $\mathrm {rcon}( x_1+(2*x_2), \mathsf {S}, A )$ returns $( x_1+(2*x_2), \{ (2*x_2, \mathsf {S} ) \} )$. Since the second component of this pair is not empty, we pick in Step 2b the first element of $\mathsf {S}$, $\mathsf {x}_1$ say, and add $( x_1, x_1, \mathsf {S} )$ to A. We then call $\mathrm {rcon}( (x_1+(2*x_2))\!\downarrow , \mathsf {S}, A )$ which by the same strategy above returns $( x_1+(2*x_2), \{ (2*x_2, \mathsf {S} ) \} )$. This process continues until we pick, the term $\mathsf {plus}( \mathsf {x_2}, \mathsf {x_2} )$ say, whose analogue is $x_2+x_2$. Assuming $(x_2+x_2)\!\downarrow \ = (2*x_2)\!\downarrow$, after adding the pair $( 2*x_2, x_2+x_2, \mathsf {S} )$ to A, $\mathrm {rcon}( (x_1+(2*x_2))\!\downarrow , \mathsf {S}, A )$ returns the pair $( x_1+(x_2+x_2), \emptyset )$, indicating that $\lambda x_1\,x_2.\, x_1+(x_2+x_2)$ is equivalent to $\lambda x_1\,x_2.\, x_1+(2*x_2)$, and meets the restrictions specified by $\mathsf {S}$.    $\blacksquare$

This procedure depends upon the use of normal forms for terms. It should be noted that, since the top symbol of t is generally $\mathsf {ite}$, this normalization includes both low-level rewriting of literals within t, but also includes high-level rewriting techniques such as $\mathsf {ite}$ simplification, redundant subterm elimination and destructive equality resolution. Also, notice that we are not assuming that $t\!\downarrow \ = s\!\downarrow$ if and only if t is equivalent to s, and thus normal forms only underapproximate an equivalence relation between terms. Having a (more) consistent normal form for terms allows us to compute a (tighter) underapproximation, thus improving the performance of the reconstruction. In this procedure, we use the same normal form for terms that is used by the individual decision procedures of cvc4. This is unproblematic for theories such as linear arithmetic whose normal form for terms is a sorted list of monomials, but it can be problematic for theories such as bitvectors. As a consequence, we use several optimizations, omitted in the description of the procedure in Fig. 5, to increase the likelihood that the procedure terminates in a reasonable amount of time. For instance, in our implementation the return value of $\mathrm {rcon}$ is not recomputed every time A is updated. Instead, we maintain an evolving directed acyclic graph (dag), whose nodes are pairs ( t, S ) for term t and datatype S (the terms we have yet to reconstruct), and whose edges are the direct subchildren of that term. Datatype terms are enumerated for all datatypes in this dag, which is incrementally pruned as pairs are added to A until it becomes empty. Another optimization is that the procedure $\mathrm {rcon}$ may choose to try simultaneously to reconstruct multiple terms of the form $f( t_1, \ldots , t_n )$ when matching a term t to a syntactic specification S, reconstructing t when any such term can be reconstructed.

Although the overhead of this procedure can be significant when large subterms do not meet the syntactic restrictions, we found that in practice it quickly terminates successfully for a majority of the solutions we considered where reconstruction was possible, as we discuss in the next section. Furthermore, it makes our implementation more robust, since it effectively treats in the same way different properties that are equal modulo normalization (which is parametric in the built-in theories we consider).

6 Experimental Evaluation

We implemented the techniques from the previous sections in the SMT solver cvc4  [4], which has support for quantified formulas and a wide range of theories including arithmetic, bitvectors, and algebraic datatypes. We evaluated our implementation on 243 benchmarks used in the SyGuS 2014 competition [1] that were publicly available on the StarExec execution service [31]. The benchmarks are in a new format for specifying syntax-guided synthesis problems [22]. We added parsing support to cvc4 for most features of this format. All SyGuS benchmarks considered contain synthesis conjectures whose background theory is either linear integer arithmetic or bitvectors. We made some minor modifications to benchmarks to avoid naming conflicts, and to explicitly define several bitvector operators that are not supported natively by cvc4.

We considered multiple configurations of cvc4 corresponding to the techniques mentioned in this paper. Configuration cvc4+sg executes the syntax-guided procedure from Sect. 4, even in cases where the synthesis conjecture is single-invocation. Configuration cvc4+si-r executes the procedure from Sect. 3 on all benchmarks having conjectures that it can deduce are single-invocation. In total, it discovered that 176 of the 243 benchmarks could be rewritten into a form that was single-invocation. This configuration simply ignores any syntax restrictions on the expected solution. Finally, configuration cvc4+si uses the same procedure used by cvc4+si-r but then attempts to reconstruct any found solution as a term in required syntax, as described in Sect. 5.

We ran all configurations on all benchmarks on the StarExec cluster.⁴ We provide comparative results here primarily against the enumerative CEGIS solver ESolver  [32], the winner of the SyGuS 2014 competition. In our tests, we found that ESolver performed significantly better than the other entrants of that competition.

Fig. 6.

Results for single-invocation synthesis conjectures, showing times (in seconds) and number of benchmarks solved by each solver and configuration over 8 benchmark classes with a 3600 s timeout. The number of benchmarks solved by configuration cvc4+si-r are in parentheses because its solutions do not necessarily satisfy the given syntactic restrictions.

Benchmarks with Single-Invocation Synthesis Conjectures. The results for benchmarks with single-invocation properties are shown in Fig. 6. Configuration cvc4+si-r found a solution (although not necessarily in the required language) very quickly for a majority of benchmarks. It terminated successfully for 168 of 176 benchmarks, and in less than a second for 159 of those. Not all solutions found using this method met the syntactic restrictions. Nevertheless, our methods for reconstructing these solutions into the required grammar, implemented in configuration cvc4+si, succeeded in 102 cases, or 61 % of the total. This is 32 more benchmarks than the 70 solved by ESolver, the best known solver for these benchmarks so far. In total, cvc4+si solved 34 benchmarks that ESolver did not, while ESolver solved 2 that cvc4+si did not.

The solutions returned by cvc4+si-r were often large, having an order of 10 K subterms for harder benchmarks. However, after exhaustively applying simplification techniques during reconstruction with configuration cvc4+si, we found that the size of those solutions is comparable to other solvers, and in some cases even smaller. For instance, among the 68 benchmarks solved by both ESolver and cvc4+si, the former produced a smaller solution in 15 cases and the latter in 9. Only in 2 cases did cvc4+si produce a solution that had 10 more subterms than the solution produced by ESolver. This indicates that in addition to having a high precision, the techniques from Sect. 5 used for solution reconstruction are effective also at producing succinct solutions for this benchmark library.

Configuration cvc4+sg does not take advantage of the fact that a synthesis conjecture is single-invocation. However, it was able to solve 48 of these benchmarks, including a small number not solved by any other configuration, like one from the icfp class whose solution was a single argument function over bitvectors that shifted its input right by four bits. In addition to being solution complete, cvc4+sg always produces solutions of minimal term size, something not guaranteed by the other solvers and cvc4 configurations. Of the 47 benchmarks solved by both cvc4+sg and ESolver, the solution returned by cvc4+sg was smaller than the one returned by ESolver in 6 cases, and had the same size in the others. This provides an experimental confirmation that the fairness techniques for term size described in Sect. 4 ensure minimal size solutions.

Fig. 7.

Results for synthesis conjectures that are not single-invocation, showing times (in seconds) and numbers of benchmarks solved by cvc4 and ESolver over 4 benchmark classes with a 3600 s timeout.

Benchmarks with Non-single-invocation Synthesis Conjectures. Configuration cvc4+sg is the only cvc4 configuration that can process benchmarks with synthesis conjectures that are not single-invocation. The results for ESolver and cvc4+sg on such benchmarks from SyGuS 2014 are shown in Fig. 7. Configuration cvc4+sg solved 53 of them over a total of 67. ESolver solved 58 and additionally reported that 6 had no solution. In more detail, ESolver solved 7 benchmarks that cvc4+sg did not, while cvc4+sg solved 2 benchmarks (from the vctrl class) that ESolver could not solve. In terms of precision, cvc4+sg is quite competitive with the state of the art on these benchmarks. To give other points of comparison, at the SyGuS 2014 competition [1] the second best solver (the Stochastic solver) solved 40 of these benchmarks within a one hour limit and Sketch solved 23.

Overall Results. In total, over the entire SyGuS 2014 benchmark set, 155 benchmarks can be solved by a configuration of cvc4 that, whenever possible, runs the methods for single-invocation properties described in Sect. 3, and otherwise runs the method described in Sect. 4. This number is 27 higher than the 128 benchmarks solved in total by ESolver. Running both configuration cvc4+sg and cvc4+si in parallel⁵ solves 156 benchmarks, indicating that cvc4 is highly competitive with state-of-the-art tools for syntax guided synthesis. cvc4 ’s performance is noticeably better than ESolver on single-invocation properties, where our new quantifier instantiation techniques give it a distinct advantage.

Competitive Advantage on Single-Invocation Properties in the Presence of Ite. We conclude by observing that for certain classes of benchmarks, configuration cvc4+si scales significantly better than state-of-the-art synthesis tools. Figure 8 shows this in comparison with ESolver for the problem of synthesizing a function that computes the maximum of n integer inputs. As reported by Alur et al. [1], no solver in the SyGuS 2014 competition was able to synthesize such a function for $n = 5$ within one hour.

For benchmarks from the array class, whose solutions are loop-free programs that compute the first instance of an element in a sorted array, the best reported solver for these in [1] was Sketch, which solved a problem for an array of length 7 in approximately 30 minutes.⁶ In contrast, cvc4+si was able to reconstruct solutions for arrays of size 15 (the largest benchmark in the class) in 0.3 s, and solved each of the benchmarks in the class but 8 within 1 s.

Fig. 8.

Results for parametric benchmarks class encoding the maximum of n integers. The columns show the run time for ESolver and cvc4 with a 3600 s timeout.

7 Conclusion

We have shown that SMT solvers, instead of just acting as subroutines for automated software synthesis tasks, can be instrumented to perform synthesis themselves. We have presented a few approaches for enabling SMT solvers to construct solutions for the broad class of syntax-guided synthesis problems and discussed their implementation in cvc4. This is, to the best of our knowledge, the first implementation of synthesis inside an SMT solver and it already shows considerable promise. Using a novel quantifier instantiation technique and a solution enumeration technique for the theory of algebraic datatypes, our implementation is competitive with the state of the art represented by the systems that participated in the 2014 syntax-guided synthesis competition. Moreover, for the important class of single-invocation problems when syntax restrictions permit the if-then-else operator, our implementation significantly outperforms those systems.