Wavelet-based gradient boosting

Abstract

A new data science tool named wavelet-based gradient boosting is proposed and tested. The approach is special case of componentwise linear least squares gradient boosting, and involves wavelet functions of the original predictors. Wavelet-based gradient boosting takes advantages of the approximate $\ell _1$ penalization induced by gradient boosting to give appropriate penalized additive fits. The method is readily implemented in R and produces parsimonious and interpretable regression fits and classifiers.

Appendix: Highest-density region grids

We now provide details of the highest density region (HDR) grids used in Figures 3 and 5.

Let $\varvec{x}=(x_1,\ldots ,x_n)$ be a generic univariate sample and ${\widehat{p}}$ be a probability density estimate based on $\varvec{x}$. Then a $100(1-\tau )\%$ highest-density region estimate is

$$\begin{aligned} {\widehat{R}}_{\tau }=\{x\in {\mathbb R}:{\widehat{p}}(x)\ge {\widehat{p}}_{\tau }\} \end{aligned}$$

where ${\widehat{p}}_{\tau }$ is chosen so that the probability mass of ${\widehat{p}}$ over the set ${\widehat{R}}_{\tau }$ does not exceed $1-\tau$. See, for example, Samworth and Wand (2010) for a precise mathematical definition of ${\widehat{p}}_{\tau }$.

The most commonly used estimator ${\widehat{p}}$ for HDR estimation is the kernel density estimator

$$\begin{aligned} {\widehat{p}}(x)=\frac{1}{nh}\sum _{i=1}^n K\left( \frac{x-x_i}{h}\right) \end{aligned}$$

where $K$ is a kernel function and $h>0$ is a bandwidth (see e.g. Wand and Jones 1995). Recently, Samworth and Wand (2010) devised an automatic rule for selection of $h$ in the HDR estimation context. The R package hdrcde (Hyndman 2010) implements both HDR estimation and the Samworth-Wand bandwidth selector. Figure 8 shows 80% HDR estimate for the variable distance to coastline variable in the Sydney residential property prices data. The corresponding HDR grid of size 50 is shown at the base of the plot.

Fig. 8

A HDR grid for the predictor variable distance to coastline in the Sydney real estate data example. The data are shown as red dots with vertical jittering to enhance visualization. The green curve is a kernel density estimate and the blue bars are the estimated 80% highest density region. The rug at the base of the plot is the corresponding HDR grid of size 50