Bayesian partition modelling

Abstract

This paper reviews recent ideas in Bayesian classification modelling via partitioning. These methods provide predictive estimates for class assignments using averages of a sample of models generated from the posterior distribution of the model parameters. We discuss modifications to the basic approach more suitable for problems when there are many predictor variables and/or a large training smple.

Introduction

The general classification problem involves predicting the class conditional probabilities at any location $\text{x}$ within some domain of interest $\text{X}$. This is nearly always achieved by “training” a model with a set of historic data pairings, $\text{D}\text{={y}{}_{\mathrm{i}}\text{,}\text{x}{}_{\mathrm{i}}\text{}}{}_1{}^{\mathrm{n}}$. Adopting a Bayesian approach involves determining the posterior predictive distribution, $\text{p(y=k|}\text{x}\text{,}\text{D}\text{)}$, for $\text{x}\text{∈}\text{X}$, where k∈{1,…,K} and denotes the class label. A model, indexed by parameters $\text{θ}$, is often introduced to capture our beliefs about the underlying probability surface. This allows us to write the (predictive) distribution of interest as

$$\text{p(y|}\text{x}\text{,}\text{D}\text{)=}\text{∫}\text{Θ}\text{p(y|}\text{x}\text{,}\text{D}\text{,}\text{θ}\text{)p(}\text{θ}\text{|}\text{D}\text{)}\text{d}\text{θ}\text{∝}\text{∫}\text{Θ}\text{p(y|}\text{x}\text{,}\text{θ}\text{)p(}\text{D}\text{|}\text{θ}\text{)p(}\text{θ}\text{)}\text{d}\text{θ}\text{,}$$

where Θ is the space defined by all the allowable models.

In essence, predictive Bayesian modelling is straightforward. Once a set of models to perform the classification is identified, all that is required is a prior for $\text{θ}$ and the definition of a likelihood $\text{p(}\text{D}\text{|}\text{θ}\text{)}$. Further, the likelihood is usually defined naturally from the context of the problem so the model prior is very important. Hence, the definition of the model space, Θ, is critical to the performance of the data analysis. Even though (1) compensates to some degree for model misspecification, through integration out of the model parameters, if the model space has no support over models close to the truth poor out-of-sample predictive results will ensue. Thus, taking Θ to cover a wide variety of possible models is desirable but this often means that the integral in (1) cannot be performed analytically. Approximations to the integral are readily available by simulation from $\text{p(}\text{θ}\text{|}\text{D}\text{)}$, most commonly with Markov chain Monte Carlo methods (e.g. Gelfand and Smith, 1990). However, depending on the form of Θ, the model space can be difficult to sample from, often due to correlation in the parameters of $\text{θ}$. As efficient estimation of $\text{p(y|}\text{x}\text{,}\text{D}\text{)}$ relies on an approximation to the posterior $\text{p(}\text{θ}\text{|}\text{D}\text{)}$, forms for $\text{θ}$ which allow efficient sampling should be preferred. One such type of model is the Bayesian partition model of Holmes et al. (1999).

In this paper, we think of partition models as those which explain the data by fitting local distributions to local areas of the predictor space $\text{X}$. Further, we make the extra restriction that the parameters of the local distributions are mutually independent. Partition models of this type are well established in the literature dating back at least to the work on recursive partitioning by Morgan and Sonquist (1963), which was further developed by Friedman (1979), leading to the well-known book on classification and regression trees by Breiman et al. (1984). Classification trees involve determining partitions of the data which lead to groupings of the datapoints into terminal nodes which, ideally, are dominated by one class. Classification trees are highly interpretable models but suffer from relatively poor out-of-sample predictive performance. This is because the partitions of the predictor space are forced to be made on a single predictor, so are necessarily axis-parallel, and the partitioning proceeds greedily, so successive “best” splits to the data are made at each step. The machine learning community, in particular, has continued to research generalisations of (recursive) partitioning models, most notably improvements in the space of possible splits (e.g. Murthy et al., 1994).

The Bayesian partition model (BPM) is motivated by the same basic principles of earlier models of the same type, i.e. points close by in predictor space come from the same local distribution. However, a more flexible partitioning scheme is proposed. Further, Bayesian model averaging (Raftery et al., 1996) ideas allow for prediction using averages (or ensembles) of models, rather than single ones which have a somewhat unappealing discontinuous nature. As well as describing the BPM of Holmes et al. (1999) in detail, we comment on both its strengths and weaknesses. In particular, we introduce a new model which is based on the partition model but is more suited to data-mining applications when there are a large number of training data points and/or a large number of predictor variables (or features). This new model, known as the product partition model (PPM), utilises a modified form of standard naive Bayes formula (e.g. Domingos and Pazzini, 1997) which assumes

$$\text{p(y|}\text{x}\text{)∝p(}\text{x}\text{|y)×p(y)=}\text{∏}\text{i=1}\text{P}\text{p(x}{}_{\mathrm{i}}\text{|y)×p(y),}$$

where $\text{x}\text{=(x}{}_1\text{,…,x}{}_{\mathrm{P}}\text{)}$ is a general feature vector assuming P predictors. The main advantage of this model is the speed with which the MCMC algorithm can be run, allowing fast computation of the posterior. This greater speed comes at the cost of slightly poorer predictive power but this degradation appears acceptable for the large datasets that are typically encountered in modern data-mining. Further, the method allows the user to identify important variables and gives an idea as to how the class assignment is affected by each predictor.

The paper is arranged as follows. In the next section, we review the BPM of Holmes et al. (1999) and Section 3 then introduces the PPM. Section 4 contains a comparison of the two approaches in both classification error and computational time and lastly Section 5 contains a brief discussion.