A limit result for the prior predictive applied to checking for prior-data conflict

Abstract

We consider checking for prior-data conflict in a Bayesian analysis via a tail probability based on the prior predictive distribution. We establish the appropriateness of this measure in the sense that the limiting value of the tail probability measures the extent to which the true value of the parameter is a surprising value from the prior.

Introduction

The relevance of the results of a statistical analysis depends upon the inputs chosen by the analyst. If the inputs are deemed not to be appropriate, then one has reason to doubt any conclusions drawn. For a Bayesian statistical analysis, these inputs comprise the sampling model $\{P_{\theta }:\theta \in \Theta \}$ for the data considered here as a collection of probability measures, one of which is supposed to have generated the observed data, the prior $\Pi$ on the model parameter and perhaps a loss function. One way to assess the relevance of these inputs is to see whether or not these make sense in light of the data collected. From this point of view, we have a model failure when the data observed is surprising for every probability distribution in the model. In this paper, we are concerned with assessing whether or not there is a prior-data conflict.

Intuitively, prior-data conflict arises when the likelihood is relatively high, where the prior is relatively low. While this seems easy to assess via a graph when dealing with a one-dimensional parameter, more formal methods seem necessary in general. Various approaches have been proposed for assessing prior-data conflict; see for example, Young and Pettit (1996), Evans and Moshonov (2006), and Marshall and Spiegelhalter (2007). Some Bayesian model checking methods, such as those discussed in Box (1980) and Gelman et al. (1996) could also be considered as assessments of the prior although this is confounded with checking the model. Separating out the assessment of the prior from the model gives greater information concerning which of these choices might be in conflict with the data. Box (1980) proposed the tail probability $M(m\left(X\right)\le m\left(x\right))$, where $x$ is the observed data, and $m\left(x\right)={\int }_{\Theta }{f}_{\theta }\left(x\right)\Pi \left(d\theta \right)$ is the density of the data associated with the prior predictive measure $M$ on the sample space $\mathcal{X}$, where $P_{\theta }$ has density $f_{\theta }$. We show in Example 1 that this is not appropriate for checking the prior.

In this paper, we prove a consistency result for the check for prior-data conflict discussed in Evans and Moshonov, 2006, Evans and Moshonov, 2007. Suppose that $T:\mathcal{X}\to \mathcal{T}$ is a minimal sufficient statistic for $\{P_{\theta }:\theta \in \Theta \}$ with density $f_{\theta T}$ on $\mathcal{T}$. The tail probability

$$M_T(m_T\left(T\right)\le m_T\left(T\left(x\right)\right))\text{,}$$

was proposed for checking for prior-data conflict where $M_T$ is the prior predictive distribution of $T$. The following example motivates why (1) is suitable for checking for prior-data conflict.

Example 1 Location Normal

Suppose that $x=(x_1,\dots ,x_n)$ is a sample from a $N(\mu ,1)$ distribution where $\mu \in R^1$ is unknown. Then a minimal sufficient statistic is given by $T_n\left(x\right)=\bar{x}$ and $T_n\left(x\right)$ converges almost surely to the true value ${\mu }_{\mathrm{true}}$ as $n\to \infty$. Suppose we put a $N({\mu }_0,{\sigma }_0^2)$ prior on $\mu$. Then $M_{T_n}$ is the $N({\mu }_0,{\sigma }_0^2+1/n)$ distribution and this converges in distribution to the $N({\mu }_0,{\sigma }_0^2)$ distribution. Also $m_{T_n}\left(t\right)$ converges almost surely to the prior density ${\left(2\pi \right)}^{-1/2}{\sigma }_0^{-1}exp\{-{(t-{\mu }_0)}^2/2{\sigma }_0^2\}$ uniformly for $t$ in a compact set. A simple computation then shows that (1) converges almost surely to $2(1-\Phi (|{\mu }_{\mathrm{true}}-{\mu }_0|/{\sigma }_0))$ which assesses how far out in the tails of the prior ${\mu }_{\mathrm{true}}$ lies.

Now consider the Box (1980) tail probability for this problem. We have that $X\sim N_n(0,I_n+\tau 1_n{1}_n^{\prime })$ where $I_n$ is the $n\times n$ identity matrix, $1_n$ is a vector of $n$ ones, and

$$M(m\left(X\right)\le m\left(x\right))=1-G_n\left(x^{\prime }(I_n-\frac{\tau }{1+n\tau }{1}_n{1}_n^{\prime })x\right)$$

where $G_n$ is the chi-squared$\left(n\right)$ distribution function. The quadratic form can be decomposed as

$$x^{\prime }(I_n-\frac{\tau }{1+n\tau }{1}_n{1}_n^{\prime })x=\sum _{i=1}^n{(x_i-{\bar{x}}_n)}^2+\frac{n}{1+n\tau }{\bar{x}}_n^2=V_n+W_n\text{,}$$

where, conditionally given $\theta ,V_n\sim {\chi }^2(n-1),W_n=O_p\left(1\right)$ and $V_n$ and $W_n$ are independent. Now $({\chi }^2\left(n\right)-n)/\sqrt{2n}\stackrel{d}{\to }N(0,1)$ and $G_n(n+x\sqrt{2n})-\Phi \left(x\right)=O\left(n^{-1/2}\right)$ uniformly in $x$, by Theorem XVI.4.1 in Feller (1971). Hence

$$M(m\left(X\right)\le m\left(x\right))=1-\Phi ((V_n-(n-1))/\sqrt{2n})+O_p\left(n^{-1/2}\right)$$

where we have used the uniform continuity of $\Phi$. Since $(V_n-(n-1))/\sqrt{2n}=(V_n-(n-1))/\sqrt{2(n-1)}\times \sqrt{1-1/n}\stackrel{d}{\to }N(0,1)$ we have that $M(m\left(X\right)\le m\left(x\right))\stackrel{d}{\to }\mathrm{Uniform}(0,1)$. This limit is independent of the prior and whether or not ${\mu }_{\mathrm{true}}$ is in the tails of the prior. Therefore, this tail probability is not useful for checking for prior-data conflict.

While the potential ill effects of a prior-data conflict have long been recognized, it is not clear what one should do when we conclude that a conflict exists. One can note, however, that we have learned something of relevance and it seems only fair that an analyst report this. Also, we note that the situation is similar with model checking as it is not clear what we should do when we have a failure and this does excuse us from these checks. The typical response to model failure is that we must modify the model in some way, perhaps by enlarging the family of distributions. Similarly, when a prior-data conflict exists, our response can be to use a new prior that is less informative in the sense that we can expect fewer prior-data conflicts. We discuss this in Section 4.

A criticism of (1) is that, in the case of continuous models, (1) is not invariant under smooth transformations. For suppose that $W:\mathcal{T}\to \mathcal{W}$ is 1-1 and smooth and let $J_W\left(t\right)$ be the reciprocal of the Jacobian determinant of $W$ evaluated at $t$. Then $W$ is also minimal sufficient and (1) applied to $W$ gives the tail probability $M_T(m_T\left(T\right)J_W\left(T\right)\le m_T\left(T\left(x\right)\right)J_W\left(T\left(x\right)\right))$ which is generally different than (1). This issue is avoided if we use the approach discussed in Evans and Jang (2010a) to get the invariant tail probability $M_T(m_T^{\ast }\left(T\right)\le m_T^{\ast }\left(T\left(x\right)\right))$ where $m_T^{\ast }\left(t\right)=m_T\left(t\right)E(J_T^{-1}\left(X\right)\mid T\left(X\right)=t),J_T\left(x\right)={|det(dT\left(x\right)\circ d{T}^{\prime }\left(x\right))|}^{-1/2}$ and $dT$ is the differential of $T$. The factor $E(J_T^{-1}\left(X\right)\mid T\left(X\right)=t)$ corrects for volume distortions due to the transformation $T$. Note that whenever $T$ is linear, then $E(J_T^{-1}\left(X\right)\mid T\left(X\right)=t)$ is constant and the invariant tail probability is the same as (1). This is the case for all but one of our examples. We state a relevant convergence result for this tail probability in Section 2.

In Section 2, we provide theorems, with proofs in the Appendix, for the convergence of (1) to the tail probability $\Pi (\pi \left(\theta \right)\le \pi \left({\theta }_{\mathrm{true}}\right))$ where ${\theta }_{\mathrm{true}}$ is the true value of the parameter, i.e., (1) is a consistent assessment of whether or not the true value of the parameter is in the tails of the prior. In Section 3, we provide some applications. In Section 4, we discuss what one can do when a prior-data conflict is encountered.