How to implement the bootstrap in static or stable dynamic regression models: test statistic versus confidence region approach
Introduction
There exists a well-known correspondence between confidence sets and hypothesis testing. A confidence set with confidence coefficient 1−α entails the set of parameter values which are not rejected by a hypothesis test performed at significance level α. This duality is exploited in this paper to define and examine several bootstrap testing procedures in static and stable dynamic regression models. See Hansen (1999) for an analysis of bootstrap procedures which exploit this duality in unstable autoregressive models. For comprehensive overviews on the bootstrap see the books by Davison and Hinkley (1997), Efron and Tibshirani (1993) and Hall (1992), while the papers of Li and Maddala (1996), Horowitz (1997) and Berkowitz and Kilian (2000) provide recent reviews from an econometric (time-series) perspective. The bootstrap procedures to be examined here differ in their choice of the type of estimator employed in the resampling scheme and in the form of the test statistic used. More specifically, a resampling scheme can use either a restricted or an unrestricted estimator in defining bootstrap observables. In addition, a bootstrap test statistic can be centred around either a restricted or an unrestricted estimator. The use of a restricted estimator in both the resampling scheme and the test statistic is typical for the test statistic approach (in which one tries to assess the null distribution), while the use of an unrestricted estimator is characteristic for the confidence region approach (which does not adhere to just one specific null hypothesis); see Beran (1986) for more details about the two approaches in the context of independent and identically distributed (i.i.d.) random vectors. We shall examine these two approaches and their cross-fertilizations in a regression context.
To illustrate the basic idea, consider the simple regression modelwhere y and x are n×1 vectors and ε is a n×1 random vector whose components are i.i.d. with unknown distribution Fε with mean 0 and finite variance σ2, i.e. εi∼Fε(0,σ2) for i=1,…,n. Suppose that we are interested in testing the null hypothesis against the one-sided alternative for some given value of δ0. In general, the finite-sample distribution of the test statistic used, say T, is unknown since it depends on some nuisance parameters. The bootstrap tackles this problem by replacing the nuisance parameters by their empirical analogues. For example, the non-parametric bootstrap replaces the unknown distribution function Fε by the empirical distribution function (EDF) of the residuals. In general, however, the bootstrap distribution of the test statistic does not possess a closed-form expression. In practice, fortunately, any desired characteristic of the bootstrap distribution can be approximated by a bootstrap simulation. If denotes the bth bootstrap realization of the test statistic T, then the bootstrap uses the empirical distribution of to approximate the actual distribution of T. Below, we shall describe in some detail how to test the hypothesis using the bootstrap based on either the test statistic or on the confidence region approach.
Letdenote the well-known t-statistic, which would have a Student's t-distribution if the disturbances εi were normally distributed and δ=δ0. In addition, let denote the bootstrap observables defined by the ‘restricted’ resampling scheme where denotes the estimator of μ under the null hypothesis and is drawn randomly with replacement from the (un)restricted residuals. The bootstrap analogue of the t-statistic τ(δ0) based on the bootstrap sample is given bywhere the bootstrap estimators and will be defined explicitly in the next section. Let denote the α-quantile of the bootstrap distribution of the test statistic , i.e. , where the super-index denotes that the resampling scheme is based on restricted estimators while the sub-index r indicates that restricted (r) estimators have been used in the test statistic. The notation stresses the fact that the probability is defined conditional on the data and with respect to resampling scheme (3). The bootstrap test procedure rejects the restriction δ=δ0 if the observed t-statistic τ(δ0) given in (2) is smaller than the α-quantile of the bootstrap distribution of , i.e. the hypothesis is rejected against the alternative if . From the analysis in Beran (1986), it follows that, under the null hypothesis, the rejection probability of the bootstrap test based on the quantiles of the theoretical bootstrap distribution (i.e. for B=∞) converges to α, i.e. as n→∞.1
An asymptotically pivotal root for the regression parameter δ is given byA root is a function of both the parameter of interest and its estimator, and can be used to construct a confidence interval; see Beran (1987) for more details on roots. In the confidence region approach, the bootstrap sample is based on the ‘unrestricted’ resampling scheme where is drawn randomly with replacement from the residuals. Letdenote the bootstrap analogue of the root R(δ) based on the bootstrap sample . Note that in the bootstrapped root , the estimator takes over the pivoting role of the unknown parameter δ. Let denote the α-quantile of the bootstrap distribution of the root , i.e. , where the super-index denotes that the bootstrap data is based on the ‘unrestricted’ resampling scheme while the sub-index u indicates that the root is based on the unrestricted (u) estimators. A one-sided (1−α)-level percentile-t confidence interval for δ is given bywhich equals the set of parameters δ defined by . Consider the situation that the parameter value under test is δ0. The parameter value δ0 lies outside the bootstrap confidence interval if the condition holds. Since the root R(δ0) equals the t-statistic τ(δ0), the corresponding bootstrap test procedure rejects the restriction δ=δ0 if the t-statistic τ(δ0) is smaller than the α-quantile of the bootstrap distribution of , i.e. the hypothesis is rejected against the alternative if . Under certain regularity conditions, Freedman (1981) has shown that the bootstrap test also has the correct asymptotic rejection probability under the null hypothesis.
To summarize, we have distinguished two different bootstrap testing procedures. For testing the null hypothesis against the alternative using the t-statistic τ(δ0), the bootstrap test rejects H0 if whereas the bootstrap test rejects H0 if . In Section 3, it will be shown that in static regression models the tests and are equivalent in finite samples since the α-quantile of the bootstrap distribution based on the test statistic approach coincides with the α-quantile of the bootstrap distribution based on the confidence region approach, i.e. . In the case where the model contains only an intercept, this equivalence has been noticed before by, for instance, Beran (1986, Example 2) and Tibshirani (1992).
Instead of combining (un)restricted estimators in both the resampling scheme and the bootstrap test statistic, one could also opt for a crosswise combination. However, Hall and Wilson (1991) warn against the use of the procedure based on the bootstrap distribution of the t-statistic and the ‘unrestricted’ resampling scheme , because this latter implementation is said to lead to low test power. Yet another implementation results by combining the root with the ‘restricted’ resampling scheme . The latter two procedures will be referred to as hybrid implementations since they mix up the test statistic with the confidence region approach. So, by combining two formulations of a test statistic, viz., and , with two ways to construct the bootstrap observables, four different bootstrap test procedures are obtained. The aim of this paper is to investigate the differences and similarities between these various implementations in both static and dynamic multiple regression models. The papers by Carpenter (1999), DiCiccio and Romano (1990) and Hansen (1999) also exploit the test statistic approach to construct confidence intervals, although their focus is quite distinct from ours. Moreover, none of these papers considers the hybrid implementations.
The paper is organized as follows. Section 2 takes a closer look at bootstrap hypothesis testing and defines the various test statistics in linear regression models for both single and joint hypotheses. In Section 3, we investigate the various bootstrap implementations and examine whether the one-to-one correspondence between the test statistic and confidence region approach already found in models with just an intercept continues to hold in finite samples of multiple regressions when testing one or several parameters jointly. Furthermore, we derive the asymptotic rejection probabilities for each testing procedure and we demonstrate that bootstrap tests based on any of the two hybrid implementations will produce zero rejections asymptotically, irrespective whether the null hypothesis is true or false. The only exception is the case where restricted estimators are used in resampling and the statistic is centred around the unrestricted estimator and the alternative hypothesis is one-sided. Then there is a serious overrejection problem. We provide an intuitive explanation for this failure of the two hybrid implementations in the case where a single coefficient is under test. In Section 3, we also take a closer look at the issue of using restricted or unrestricted residuals in the resampling schemes. In Section 4, the finite-sample performance of the two asymptotically valid implementations is investigated in stable dynamic models. We observe that in dynamic regression models, the test statistic and confidence region approach lead to different findings in finite samples. In a small-scale Monte Carlo study, finite-sample inference based on the two approaches is examined in an ARMA(1,1) model. In Section 5, the two approaches are compared on the basis of an empirical example. The final section discusses the major findings.
Section snippets
Various bootstrap regression test procedures
In the standard—not necessarily Gaussian—linear multiple regression modelwhere y is a n×1 vector, X is a fixed n×k matrix, β is a k×1 vector of unknown parameters and ε is a n×1 random vector with i.i.d. components εi∼Fε(0,σ2). In addition, we assume that the number of coefficients k is fixed, only is observable and the matrix X has rank k. For the asymptotic analysis we assume that X′X/n→Q as n→∞, where Q has finite elements and is positive definite. Of course, these conditions
Properties of the various bootstrap test procedures
We begin this section by showing in Proposition 1 that the F-statistic obtained by the confidence region approach is algebraically equivalent to the F-statistic obtained by the test statistic approach, i.e. . Self-evidently, the same result holds for the t-statistics and . Proposition 1 In the linear fixed-regressors model (8), the F-statistics and defined in (12) and (13) are equivalent. Proof Consider the OLS estimator based on the resampling scheme
Bootstrap hypothesis testing in dynamic models
By the theorems given in Freedman (1984), the asymptotic results of Proposition 2, Proposition 3 immediately carry over to the stable autoregressive model of order one with fixed exogenous variableswhere y−1=(y0,…,yn−1) and εi∼Fε. However, the one-to-one relation between the test statistic and confidence region approach, as established in Proposition 1 for fixed-regressors models, no longer holds here. In dynamic models with a lagged-dependent variable, we have to resort to a
Empirical illustration
To compare the different (bootstrap) approximations in practice, we consider annual data on unemployment for the US from 1890 through 1988 as used by Nelson and Plosser (1982) and extended by Schotman and van Dijk (1991). Note that the unemployment rate is the only time series of the 14 considered by Nelson and Plosser which appears to be stationary. We focus on the construction of confidence intervals. Instead of fitting a AR(p) model, we chose to model the data as an ARMA(1,1) model. ML
Concluding remarks
We have contrasted two approaches for testing a hypothesis using the bootstrap. The test statistic approach is based on restricted estimators in both the test statistic and resampling scheme, while the confidence region approach employs unrestricted estimators in the test statistic and resampling scheme. Two hybrid approaches emerge if the test statistic approach is combined with the confidence region approach. The hybrid approaches are shown to be cursed with serious size (and power) problems
Acknowledgements
This research was sponsored by the Economics Research Foundation (ECOZOEK), which is part of the Netherlands Organization for Scientific Research (NWO). This paper is a completely revised version of the discussion paper “How to Implement Bootstrap Hypothesis Testing in Static and Dynamic Regression Models”, which was first presented at the EC2 meeting in Oxford (16–17 December 1993, UK). We are grateful to the editor Richard Blundell, an associate editor, and two anonymous referees for their
References (34)
Bootstrap-based critical values for the information-matrix test
Journal of Econometrics
(1994)- et al.
Trends and random walks in macroeconomic time series
Journal of Monetary Economics
(1982) - et al.
Bootstrapping unstable first order autoregressive processes
The Annals of Statistics
(1991) Simulated power functions
The Annals of Statistics
(1986)Prepivoting to reduce level error of confidence sets
Biometrika
(1987)- et al.
Recent developments in bootstrapping time series
Econometric Reviews
(2000) - et al.
Bootstrapping regression models with many parameters
Edgeworth correction by bootstrap in autoregressions
The Annals of Statistics
(1988)Test inversion bootstrap confidence intervals
Journal of the Royal Statistical Society, Series B
(1999)Comment on ‘Recent developments in bootstrapping time series’
Econometric Reviews
(1999)
Bootstrap Methods and their Applications
The size distortion of bootstrap tests
Econometric Theory
Nonparametric confidence limits by resampling methods and least favorable families
International Statistical Review
An Introduction to the Bootstrap
Unit root bootstrap tests for AR(1) models
Biometrika
Bootstrapping regression models
The Annals of Statistics
On bootstrapping two-stage least-squares estimates in stationary linear models
The Annals of Statistics
Cited by (18)
Cross-section bootstrap for CCE regressions
2024, Journal of EconometricsLM tests of spatial dependence based on bootstrap critical values
2015, Journal of EconometricsCitation Excerpt :However, as pointed out by Davidson (2007) and reiterated in Godfrey (2009, p. 82), it is not always the case that the asymptotic analysis seems to provide a good explanation of what is observed in finite samples. For the residual-based bootstrap method which is followed in this paper, Godfrey (2009, Ch. 3), based on the work of van Giersbergen and Kiviet (2002) and MacKinnon (2002), give a detailed discussion on the type of residuals (restricted under the null hypothesis or unrestricted) to be resampled and the type of estimates (restricted or unrestricted) of the nuisance parameters to be used as parameters in the bootstrap world. However, the debate on the choices of parameter estimates and residuals does not seem to have been settled.
Alternative approaches to implementing Lagrange multiplier tests for serial correlation in dynamic regression models
2007, Computational Statistics and Data AnalysisBootstrapping heteroskedastic regression models: Wild bootstrap vs. pairs bootstrap
2005, Computational Statistics and Data AnalysisCitation Excerpt :Thus, they recommend the use of restricted residuals to compute the HCCME based test. On the other hand, van Giersbergen and Kiviet (2002) argue that if the null is not true, the use of unrestricted residuals could improve the power and should be preferred. However, this last argument is not supported in simulation experiments, see MacKinnon (2002).
24-h electrical load data - A sequential or partitioned time series?
2003, NeurocomputingCitation Excerpt :The overall training, validation and novelty sets, used for performance determination, topology and training cessation point in the parallel and sequential models, is shown in Table 1. The techniques used for input selection (Section 4.2) utilise different training and validation sets where the dates vary due to the use of a bootstrapping technique [11], which allows a statistical evaluation of the different input selections. In this case, eight bootstraps are constructed, where the validation set occupies a different range for each set (Fig. 3 and Table 2).
HOW RELIABLE ARE BOOTSTRAP-BASED HETEROSKEDASTICITY ROBUST TESTS?
2023, Econometric Theory