On Covariate Importance for Regression Models with Multivariate Response

Abstract

We address the question of identifying the relative importance of covariates for model response, a form of sensitivity analysis. Relative importance is typically implemented as part of the model building procedure, e.g., forward variable selection or backward elimination. Here, we take a different perspective. We assume a model with multiple covariates and multivariate response has been selected and formulate criteria to assess covariate importance. Hence, with regard to covariates, our approach is joint, post model fitting, rather than conditional or sequential model creation. The noteworthy challenge we accommodate is the handling of multivariate response where individual regressions may give differing, perhaps conflicting, relative importances. In addition, we recognize that, according to the model specification, importance/sensitivity to covariates may be a global or a local issue. For models with multivariate response, we provide a criterion that (i) produces one sensitivity coefficient for each covariate, (ii) takes into account the model specification of uncertainty, and (iii) is based only on the model parameters but does not require a distribution on the covariates. However, with a prior on the covariates, in special cases, we show that comparison of covariates using this criterion gives the same results as comparison of marginal variances of the inverse predictive distributions of the covariates. We illustrate with an application examining sensitivity of tree abundance to climate.

Supplementary materials accompanying this paper appear on-line.

Appendices

Appendix 1: Proof of Theorem 2.2

The model in (3) and (4) implies that the joint distribution of $\mathbf {Y}$ and $\mathbf {X}$ is Gaussian and therefore the conditional distribution $[ \mathbf {X}|\mathbf {Y},\theta ]$ is also Gaussian with covariance matrix

$$\begin{aligned} \Sigma _{X|Y}&= \left( \Sigma _{XX}^{-1} + B^\mathrm{T} \Sigma _{Y|X}^{-1} B \right) ^{-1} \\&= \begin{pmatrix} \frac{\sigma ^2}{\sigma ^4 - \sigma _{12}^2} + \mathbf {b}_1^\mathrm{T} \Sigma _{Y|X}^{-1} \mathbf {b}_1 &{} \frac{-\sigma _{12}}{\sigma ^4 - \sigma _{12}^2} + \mathbf {b}_1^\mathrm{T} \Sigma _{Y|X}^{-1} \mathbf {b}_2\\ \frac{-\sigma _{12}}{\sigma ^4 - \sigma _{12}^2} + \mathbf {b}_2^\mathrm{T} \Sigma _{Y|X}^{-1} \mathbf {b}_1&{} \frac{\sigma ^2}{\sigma ^4 - \sigma _{12}^2} + \mathbf {b}_2^\mathrm{T} \Sigma _{Y|X}^{-1} \mathbf {b}_2 \end{pmatrix}^{-1} \end{aligned}$$

If follows that

$$\begin{aligned} \mathrm {Var}(X_1|\mathbf {Y},\theta )&\le \mathrm {Var}(X_2|\mathbf {Y},\theta ) \\ \Leftrightarrow \qquad \frac{\sigma ^2}{\sigma ^4 - \sigma _{12}^2} + \mathbf {b}_2^\mathrm{T} \Sigma _{Y|X}^{-1} \mathbf {b}_2&\le \frac{\sigma ^2}{\sigma ^4 - \sigma _{12}^2} + \mathbf {b}_1^\mathrm{T} \Sigma _{Y|X}^{-1} \mathbf {b}_1 \\ \Leftrightarrow \qquad \mathbf {b}_2^\mathrm{T} \Sigma _{Y|X}^{-1} \mathbf {b}_2&\le \mathbf {b}_1^\mathrm{T} \Sigma _{Y|X}^{-1} \mathbf {b}_1 . \end{aligned}$$

Appendix 2: Proof of Remark 2.3

Theorem 6.1

Let

$$\begin{aligned} \mathbf {Y} | \mathbf {X}, B, \Sigma _{Y|X}&\sim \mathcal {N}_r ( B \mathbf {X} , \Sigma _{Y|X}) \quad \text {and} \quad \mathbf {X} | \Sigma _{XX} \sim \mathcal {N}_p ( {\varvec{\mu }}_X, \Sigma _{XX} ). \end{aligned}$$

(21)

Let $\tilde{\mathbf {X}}$ be the standardized $\mathbf {X}$, i.e., $\tilde{\mathbf {X}} = D^{-1} \mathbf {X}$ where $D = \mathrm{diag}(\sigma _{X_1}, \ldots , \sigma _{X_p})$ and $\sigma _{X_j}$ is the marginal standard deviation of $X_j$. Let

$$\begin{aligned} \mathbf {Y} | \tilde{\mathbf {X}}, \tilde{B}, \tilde{\Sigma }_{Y|X}&\sim \mathcal {N}_r ( \tilde{B} \tilde{\mathbf {X}} , \tilde{\Sigma }_{Y|X}) \quad \mathrm{and} \quad \tilde{\mathbf {X}} | R \sim \mathcal {N}_p ( \tilde{{\varvec{\mu }}}_X, R ) \end{aligned}$$

(22)

be the corresponding relationship between $\mathbf {Y}$ and $\tilde{\mathbf {X}}$ where $R$ is a correlation matrix. Then

$$\begin{aligned} \tilde{B}^\intercal \tilde{\Sigma }_{Y|X}^{-1} \tilde{B} = D B^\intercal \Sigma _{Y|X}^{-1} B D . \end{aligned}$$

(23)

Proof

It follows from (21) that $\mathbf {Y}$ and $\mathbf {X}$ are jointly normal and we denote the joint covariance matrix as $\Sigma = \left( \begin{array}{ll} \Sigma _{YY} &{} \Sigma _{YX} \\ \Sigma _{XY} &{} \Sigma _{XX} \end{array} \right)$ where $\Sigma _{YY}$ and $\Sigma _{YX}$ can be found from $B$, $\Sigma _{Y|X}$ and $\Sigma _{XX}$. Also, the joint distribution of $\mathbf {Y}$ and $\tilde{\mathbf {X}}$ is normal with covariance matrix $\tilde{\Sigma } = \left( \begin{array}{ll} \Sigma _{YY} &{} \Sigma _{YX} D^{-1} \\ D^{-1}\Sigma _{XY} &{} D^{-1} \Sigma _{XX} D^{-1} \end{array} \right)$. Furthermore, $\tilde{B}$ and $B$ can be written as $B = \Sigma _{YX} \Sigma _{XX}^{-1}$ and $\tilde{B} = \tilde{\Sigma }_{YX} R^{-1}$, where $R=D^{-1} \Sigma _{XX} D^{-1}$. Finally, note that $\tilde{\Sigma }_{Y|X} = \Sigma _{Y|X}$. Putting all this together we get

$$\begin{aligned} \tilde{B}^\intercal \tilde{\Sigma }_{Y|X}^{-1} \tilde{B}&= (D \Sigma _{XX}^{-1} D ) D^{-1}\Sigma _{XY} \Sigma _{Y|X}^{-1} \Sigma _{YX} D^{-1} ( D \Sigma _{XX}^{-1} D)\\&= D B^\intercal \Sigma _{Y|X}^{-1} B D . \end{aligned}$$

Proof of Remark 2.3

$$\begin{aligned} \mathrm {Var}(X_1/\sigma _{X_1}|\mathbf {Y},\theta ) = \mathrm {Var}(\tilde{X}_1|\mathbf {Y},\theta )&\le \mathrm {Var}(\tilde{X}_2|\mathbf {Y},\theta ) = \mathrm {Var}(X_2/\sigma _{X_2}|\mathbf {Y},\theta ) \\ \qquad \qquad \qquad \text {if and only if } \qquad \tilde{\mathbf {b}}_1^\mathrm{T} \tilde{\Sigma }_{Y|X}^{-1} \tilde{\mathbf {b}}_1&\ge \tilde{\mathbf {b}}_2^\mathrm{T} \tilde{\Sigma }_{Y|X}^{-1} \tilde{\mathbf {b}}_2 \\ \qquad \qquad \qquad \text {if and only if } \qquad \mathcal {I}_1 / \sigma ^2_{X_1} = \mathbf {b}_1^\mathrm{T} \Sigma _{Y|X}^{-1} \mathbf {b}_1 / \sigma ^2_{X_1}&\ge \mathbf {b}_2^\mathrm{T} \Sigma _{Y|X}^{-1} \mathbf {b}_2 / \sigma ^2_{X_2} =\mathcal {I}_2 / \sigma ^2_{X_2}. \end{aligned}$$

The first line follows from Theorem 2.1 and the second line follows from Theorem 6.1.

Appendix 3: Theorem for $p>2$

Theorem 7.1

Let $\mathbf {Y}$ and $\mathbf {X}$ be $r$ and $p$ dimensional random vectors where $p>2$. Let

$$\begin{aligned} \mathbf {Y} | \mathbf {X} \sim \mathcal {N}_r ({\varvec{\mu }}_{Y|X} + B\mathbf {X}, \Sigma _{Y|X} ) \quad \mathrm{and} \quad \mathbf {X} \sim \mathcal {N}_p ({\varvec{\mu }}_X, \Sigma _{XX}). \end{aligned}$$

(24)

For integers $k$, $k^\prime$, let $\mathbf {X}_s = (X_k,X_{k^\prime })^\mathrm{T}$ and let $\mathbf {X}_{-s}$ be a vector of the remaining elements of $\mathbf {X}$. Also, assume that the marginal variances of $X_k$ and $X_{k^\prime }$ are equal, i.e., $\Sigma _{X_sX_s} = \left( \begin{array}{ll} \sigma ^2 &{} \sigma _{12} \\ \sigma _{12} &{} \sigma ^2 \end{array} \right)$ and that the parameters $\theta = ({\varvec{\mu }}_{Y|X}, {\varvec{\mu }}_X, B, \Sigma _{Y|X}, \Sigma _{XX})$ are known. Then

$$\begin{aligned} \mathrm {Var}(X_k|\mathbf {Y}-B_{-s}\mathbf {X}_{-s},\theta )&\le \mathrm {Var}(X_{k^\prime }|\mathbf {Y}-B_{-s}\mathbf {X}_{-s},\theta )\\ \mathrm{if\,and\,only\,if\,} \qquad \mathcal {I}_{k} = \mathbf {b}_{k} \Sigma _{Y|X}^{-1} \mathbf {b}_{k}&\ge \mathbf {b}_{k^\prime } \Sigma _{Y|X}^{-1} \mathbf {b}_{k^\prime } = \mathcal {I}_{k^\prime }, \end{aligned}$$

where $\mathbf {b}_k$ is the $k$th column of $B$ and $B_{-s}$ is $B$ without the $k$th and $k^\prime$th columns.

Proof

Let $\mathbf {U} = \mathbf {Y} - B_{-s} \mathbf {X}_{-s}$. Then

$$\begin{aligned} \mathbf {U} | \mathbf {X}_s \sim \mathcal {N}_r ({\varvec{\mu }}_{Y|X} + B_s\mathbf {X}_s, \Sigma _{Y|X} ) \quad \text {and} \quad \mathbf {X}_s \sim \mathcal {N}_2 ({\varvec{\mu }}_{X_s}, \Sigma _{X_s X_s} ), \end{aligned}$$

(25)

where $B_s = \begin{bmatrix} \mathbf {b}_k&\mathbf {b}_{k^\prime } \end{bmatrix}$. By Theorem 2.1, it then follows that

$$\begin{aligned} \mathrm {Var}(X_k|\mathbf {U},\theta ) \le \mathrm {Var}(X_{k^\prime }|\mathbf {U},\theta ) \quad \Leftrightarrow \quad \mathbf {b}_{k^\prime }^\mathrm{T} \Sigma _{Y|X}^{-1} \mathbf {b}_{k^\prime } \le \mathbf {b}_{k}^\mathrm{T} \Sigma _{Y|X}^{-1} \mathbf {b}_k . \end{aligned}$$

(26)

To see why (25) holds we note that (24) implies that the joint distribution of $\mathbf {Y}$, $\mathbf {X}_s$, and $\mathbf {X}_{-s}$ is normal. Hence, the joint distribution of $\mathbf {U}$ and $\mathbf {X}_s$ is also normal since $(\mathbf {U}^\mathrm{T},\mathbf {X}^\mathrm{T}_s)^\mathrm{T}$ is a linear transformation of $(\mathbf {Y}^\mathrm{T},\mathbf {X}^\mathrm{T}_s, \mathbf {X}^\mathrm{T}_{-s})^\mathrm{T}$. Therefore, the conditional distribution of $\mathbf {U}|\mathbf {X}_s$ is normal. The mean vector and covariance matrix can be found via iterative expectations:

$$\begin{aligned} E(\mathbf {U} | \mathbf {X}_s)&= E\left( \left. E(\mathbf {Y} - B_{-s} \mathbf {X}_{-s} | \mathbf {X}_s, \mathbf {X}_{-s} ) \right| \mathbf {X}_s \right) \\&= E\left( \left. {\varvec{\mu }}_{Y|X} + B_{-s} \mathbf {X}_{-s}+ B_s \mathbf {X}_s - B_{-s} \mathbf {X}_{-s} \right| \mathbf {X}_s \right) \\&= E\left( \left. {\varvec{\mu }}_{Y|X} + B_s \mathbf {X}_s \right| \mathbf {X}_s \right) = {\varvec{\mu }}_{Y|X} + B_s \mathbf {X}_s \\ \text {and} \quad \mathrm {Var}(\mathbf {U} | \mathbf {X}_s)&= \mathrm {Var}\left( \left. E(\mathbf {Y} - B_{-s} \mathbf {X}_{-s} | \mathbf {X}_{-s}, \mathbf {X}_{s}) \right| \mathbf {X}_s\right) \\&\quad + E\left( \mathrm {Var}( \left. \mathbf {Y} - B_{-s} \mathbf {X}_{-s} | \mathbf {X}_{-s}, \mathbf {X}_{s} ) \right| \mathbf {X}_s \right) \\&= \mathrm {Var}\left( \left. {\varvec{\mu }}_{Y|X} + B_s \mathbf {X}_s \right| \mathbf {X}_s\right) + E\left( \mathrm {Var}( \left. \mathbf {Y} | \mathbf {X}_{-s}, \mathbf {X}_{s} ) \right| \mathbf {X}_s \right) \\&= 0 + E\left( \left. \Sigma _{Y|X} \right| \mathbf {X}_s \right) = \Sigma _{Y|X}. \end{aligned}$$