A Practical Approach to Testing Calibration Strategies

Abstract

A calibration strategy tries to match target moments using a model’s parameters. We propose tests for determining whether this is possible. The tests use moments at random parameter draws to assess whether the target moments are similar to the computed ones (evidence of existence) or appear to be outliers (evidence of non-existence). Our experiments show the tests are effective at detecting both existence and non-existence in a non-linear model. Multiple calibration strategies can be quickly tested using just one set of simulated data. Applying our approach to indirect inference allows for the testing of many auxiliary model specifications simultaneously. Code is provided.

Appendices

A Lack of Existence in Indirect Inference with Misspecification

In this appendix, we show indirect inference does not guarantee existence for misspecified models.

Consider the following indirect inference setup. Let the auxiliary model be $y = \gamma _1 + \gamma _2 \epsilon$ with $\epsilon \sim N(0,1)$. Then the maximum likelihood estimates (MLE) for $\gamma _1$ and $\gamma _2$ are the mean and square root of the sample variance. Let $\gamma ^*_1$ and $\gamma ^*_2$ denote the MLE estimates from the actual data and $\gamma _1(\theta )$ and $\gamma _2(\theta )$ the MLE estimates using simulated model data. For clarity, suppose there is an infinite amount of both simulated and actual data. Then $\gamma ^*_1,\gamma _1(\theta )$ are the population mean from the data and model, respectively, and $\gamma ^*_2,\gamma _2(\theta )$ are the population standard deviation from the data and model.

If indirect inference is done by trying to find a $\theta ^*$ such that $\gamma _i(\theta ^*) = \gamma ^*_i$, then evidently the deep model must be able to simultaneously match the data’s mean and variance. Under misspecification, this is not always possible. E.g., suppose the data generating process (DGP) is $\tilde{y} = 1 + 2\epsilon$ with $\epsilon \sim N(0,1)$—so the auxiliary model in fact nests the DGP. Then if the deep model is an exponential with parameter $\theta$, existence will not hold because the mean is $\theta ^{-1}$ (which requires $\theta ^*=1$) and the variance is $\theta ^{-2}$ (which does not equal 4 at $\theta ^*=1$). Hence, indirect inference does not guarantee existence when the model is misspecified.

Unsurprisingly then, this is also true with the Gallant and Tauchen (1996) approach. For instance, letting $\phi (y;\gamma _1,\gamma _2)$ denote the normal density with mean $\gamma _1$ and standard deviation $\gamma _2$, the expected score of the auxiliary model is

$$\begin{aligned} \mathbb {E} \begin{bmatrix} \frac{\partial \log (\phi (y;\gamma _1,\gamma _2))}{\partial \gamma _1}\\ \frac{\partial \log (\phi (y;\gamma _1,\gamma _2))}{\partial \gamma _2} \end{bmatrix} = \mathbb {E} \begin{bmatrix} \frac{y-\gamma _1}{\gamma _2} \\ -\frac{1}{\gamma _2}+\frac{(y-\gamma _1)^2}{\gamma _2^3} \end{bmatrix} = \begin{bmatrix} \frac{\mathbb {E}(y)-\gamma _1}{\gamma _2} \\ -\frac{1}{\gamma _2}+\frac{1}{\gamma _2^3}\mathbb {E}(y-\gamma _1)^2 \end{bmatrix}. \end{aligned}$$

(3)

For existence, there must be a $\theta$ such that this expectation—taken with respect to model data—equals zero when evaluated at $\gamma _1^*,\gamma _2^*$. From the first component of the vector, this requires $\mathbb {E}(y) = \gamma _1$, i.e., the deep model must be able to reproduce the mean in the data. Supposing $\theta$ is chosen to make this hold, the second component then requires that the variance of deep-model-generated data at $\theta$ equals $\gamma ^{*2}_2$.²³ So if the deep model cannot simultaneously reproduce the mean and variance in the data, existence will not hold. The example DGP and deep model discussed above provide an example where this is the case.

B The Chatterjee and Eyigungor (2012) Model

In this appendix, we more thoroughly describe the Chatterjee and Eyigungor (2012) model.

A sovereign seeks to maximize $\mathbb {E}_0 \sum ^\infty _{t=0} \beta ^t u(c_t)$ where u is the period utility function and $\beta$ the time discount factor. The sovereign’s “output” is a stochastic endowment consisting of a persistent component y that evolves according to a Markov chain plus an i.i.d. shock $m\sim U[-\overline{m},\overline{m}]$. As Chatterjee and Eyigungor (2012) discuss, the m shock’s role is simply to aid convergence.²⁴ A default triggers an entry to autarky, which entails (1) a direct loss in output $\phi (y) = \max \{0,d_0 y+ d_1 y^2\}$; (2) exclusion from borrowing; and (3) m being replaced with $-\overline{m}$. The sovereign returns from autarky with probability $\xi$.

Every unit of debt matures probabilistically at rate $\lambda \in (0,1]$ so that $\lambda =1$ is short-term debt and $\lambda \in (0,1)$ is long-term debt. In keeping with the literature, let b denote assets so that $-b$ is debt. The total stock of assets evolves according to $b' = x + (1-\lambda ) b$ with $-x$ being new debt issuance. The price paid on debt issuance is $q(b',y)$, which depends only on $b'$ and y as they are sufficient statistics for determining future repayment rates. For each unit of debt that does not mature, the sovereign must pay a coupon z. The sovereign’s budget constraint when not in autarky is $c = y+m+(\lambda + (1-\lambda )z)b + q(b',y) (b'-(1-\lambda )b)$. In autarky, the sovereign’s budget constraint is $c = y - \phi (y)+( -\overline{m}).$

When the sovereign is not in autarky, he compares the value of repaying debt with the value of default and optimally chooses to default, $d=1$, or not, $d=0$. Let the default policy function be denoted d(b, y, m) and the next-period asset policy be denoted a(b, y, m). With $r_f$ the risk-free world interest rate and risk-neutral (foreign) purchasers of sovereign debt, bond prices must satisfy

$$\begin{aligned} q(b',y) = \mathbb {E}_{y',m'|y} \left[ (1-d(b',y',m')) \frac{\lambda + (1-\lambda ) (z+q(a(b',y',m'),y'))}{1+r_f}\right] . \end{aligned}$$

(4)

Equilibrium is a fixed point in which (1) q satisfies the above functional equation given d and a; and (2) d and a are optimal given q.

The mapping of the model to data for consumption and output is the obvious one. Less obvious is the definition of spreads and net exports. Following Chatterjee and Eyigungor (2012), we define spreads for a given $(b',y)$ pair as the difference between “internal rate of return” r and the risk-free rate r. Specifically, r is defined as the solution to²⁵

$$\begin{aligned} q(b',y) =\frac{\lambda + (1-\lambda ) (z+q(b',y))}{1+r(b',y)} \end{aligned}$$

(5)

and spreads are $(1+r)^4 - (1+r^f)^4$ since the model is calibrated to quarterly time periods. Net exports is more simply defined as output less consumption, which is $y+m - c$ when not in autarky and $y-\phi (y) + (-\overline{m}) - c$ in autarky.