Information divergence estimation based on data-dependent partitions

Abstract

This work studies the problem of information divergence estimation based on data-dependent partitions. A histogram-based data-dependent estimate is proposed adopting a version of Barron-type histogram-based estimate. The main result is the stipulation of sufficient conditions on the partition scheme to make the estimate strongly consistent. Furthermore, when the distributions are equipped with density functions in $({\mathbb{R}}^d,\mathcal{B}({\mathbb{R}}^d))$, we obtain sufficient conditions that guarantee a density-free strongly consistent information divergence estimate. In this context, the result is presented for two emblematic partition schemes: the statistically equivalent blocks (Gessaman's data-driven partition) and data-dependent tree-structured vector quantization (TSVQ).

Introduction

Let P and Q be probability measures defined on $({\mathbb{R}}^d,\mathcal{B}({\mathbb{R}}^d))$, the finite dimensional Euclidean space equipped with the Borel sigma field, then the information divergence of P with respect to Q is expressed by (see, e.g., Kullback, 1958, Gray, 1990),

$$D(P\parallel Q)=\underset{\pi \in \mathcal{Q}}{\sup }\sum _{A\in \pi }P(A)·\log \frac{P(A)}{Q(A)}\text{,}$$

where $\mathcal{Q}$ denotes the collection of finite measurable partitions of ${\mathbb{R}}^d$. For this quantity to be finite, it is necessary that $P⪡Q$ (Kullback, 1958), which makes $\partial P/\partial Q(x)$ the Radon–Nicodym (RN) derivative of P with respect to Q well defined. Considering the important case when P and Q are absolutely continuous with respect to the Lebesgue measure $\lambda$, i.e., $P⪡\lambda$ and $Q⪡\lambda$, it is sometime convenient to use the following expression (see, Gray, 1990):

$$D(P\parallel Q)={\int }_{{\mathbb{R}}^d}p(x)·\log \frac{p(x)}{q(x)}\lambda (\partial x)\text{,}$$

where $p(x)=\partial P/\partial \lambda (x)$ and $q(x)=\partial Q/\partial \lambda (x)$ are the density functions of P and Q, respectively. The information divergence, also known Kullback–Leibler (KL) divergence or relative entropy, is a well known fundamental quantity in statistics and information theory (Kullback, 1958, Cover and Thomas, 1991, Gray, 1990). In statistics, KL divergence expresses the average information per observation to discriminate between two probabilistic models (Kullback, 1958). In large deviations, it characterizes the rate function, which reflects the exponential decay of convergence of empirical measures to their probabilities, Sanov’s Theorem (see, e.g., den Hollander, 2000), and the rate of decay of the probability of error in a binary hypothesis testing problem, Stein’s Lemma (see Cover and Thomas, 1991).

On the application side, mainly because of its role as a discriminative measure (Kullback, 1958), the information divergence has found wide use in statistical learning-decision problems. It has been adopted as an optimality criterion for parameter re-estimation (Singer and Warmuth, 1996, Juang and Rabiner, 1985), as a similarity measure for modeling clustering and indexing (Vasconcelos, 2004b, Vasconcelos, 2000, Do and Vetterli, 2002), as an indicator to quantify the effect of estimation error in a Bayes decision approach (Vasconcelos, 2004a, Silva and Narayanan, 2009), to quantify the approximation error of vector quantization in statistical hypothesis testing (Jain et al., 2002, Poor and Tomas, 1977) and as fidelity indicator for feature selection and feature extraction (Saito and Coifman, 1994, Novovicova et al., 1996). These learning scenarios do not have access to the distributions and consequently they rely on empirical data to estimate this quantity. A standard setting considers X₁,‥,X_n and Y₁,‥,Y_n to be independent and identically distributed (i.i.d.) realizations of P and Q, respectively. Then the problem becomes finding a distribution-free function or estimator $\widehat{D}(·)$ from ${\mathbb{R}}^{d·n}\times {\mathbb{R}}^{d·n}$ to $\mathbb{R}$ such that $\widehat{D}(X_1,‥,X_n,Y_1,‥,Y_n)$ converges to $D(P\parallel Q)$ almost surely as n tends to infinity (strong consistency).

In this regard, the closely related problem of differential entropy estimation has been systematically studied for distributions equipped with densities, adopting for instance non-parametric histogram-based, kernel-based and nearest-neighbor techniques. In these settings, the conditions for density-free strong consistency are well understood. An excellent review can be found in Beirlant et al. (1997) and some recent contributions in Darbellay and Vajda (1999) and Paninski, 2003, Paninski, 2008. Another closely related problem is the non-parametric density estimation, as the KL divergence is a functional of two probability measures. In this context the classical problem of strong consistency in L₁ sense is well understood (Lugosi and Nobel, 1996, Devroye and Györfi, 1985). More recent work on non-parametric distribution estimation considers consistency under stronger notions (Györfi and van der Meulen, 1994, Barron et al., 1992, Györfi et al., 1998, Berlinet et al., 1998). In particular the seminal work of Barron et al. (1992) proposed variations of classical histogram-based density estimates to achieve consistency in two types of information divergences, motivated by the learning problem on universal lossy compression. This approach has been extended by Györfi et al. (1998) and Berlinet et al. (1998) for the problem of consistency in ${\chi }^2$- divergence and in Csiszár's $\varphi$- divergence (where the information divergence is a particular case), respectively. Although the two aforementioned research lines have been systematically explored, to the best of our knowledge, their estimates and results do not extend directly to the consistent estimation of information divergence. The main reason is that the learning setting here is different. On the one hand, we need to consider finite samples from the two distributions, P and Q, while on the other, we need to infer the distributions from the data in a way that is appropriate to the particular nature of the divergence information functional. However, because of their inherent connections, the extensions of techniques and results from distribution and differential entropy estimation to KL divergence estimation are important directions to explore.

In that spirit, there have been some recent contributions, in particular for P and Q defined in $({\mathbb{R}}^d,\mathcal{B}({\mathbb{R}}^d))$ and both absolutely continuous with respect to the Lebesgue measure $\lambda$. The first important reference in this regard is from Wang et al. (2005), who proposed a histogram-based divergence estimation based on partitioning the space in statistically equivalent intervals. Sufficient conditions on the proposed data-driven partition were stipulated to guarantee strong consistency. Silva and Narayanan (2007) took this direction a step further finding consistency conditions for a general family of data-driven partition schemes. The main limitation of these two works is that they are only valid when the sample points of P and Q are taken to infinity in a specific order, one after the other, which limits their applicability. Alternatively, Nguyen et al. (2007) proposed a variational approach to estimate the divergence (see, Gray, 1990, den Hollander, 2000). Under certain approximation assumptions and smoothness condition on the likelihood-ratio, strong consistency and asymptotic rate of convergence for the proposed estimate were obtained. More recently, Wang et al. (2009) proposed nearest-neighbor techniques, where mean-square consistency was the main focus of analysis.

In this work we present contributions in the area of histogram-based information divergence estimation, in particular studying data-driven partitions schemes (Lugosi and Nobel, 1996, Nobel, 1996, Devroye et al., 1996, Darbellay and Vajda, 1999). We have significantly improved the initial findings in Wang et al. (2005) and Silva and Narayanan (2007). We reformulate the problem, propose new estimates and results to address properly the case when the samples of P and Q jointly tend to infinity, and furthermore, report new practical implications by getting concrete density-free KL divergence estimates from previously unexplored multivariate data-driven partition schemes.

Specifically in Section 3, we present the general histogram-based estimation scheme. This scheme quantizes the space function of the data and constructs a version of the Barron-type of histogram-based density estimate (Barron et al., 1992) as a way to approximate $\delta P/\delta Q(x)$, which can be considered the sufficient statistics for the problem. Then assuming that $D(P\parallel Q)<\infty$, Theorem 4 in Section 5 characterizes sufficient conditions on the partitions scheme to make the estimate strongly consistent. This result does not require P and Q to be absolutely continuous with respect to $\lambda$, and furthermore, it is valid for distributions defined on a general measurable space $(\mathcal{X},\mathcal{S})$. Concerning the approximation error presented in Section 4, we adopt Csiszár's notion of asymptotically sufficient partitions (Csiszár, 1967, Csiszár, 1973), and when $P⪡\lambda$ and $Q⪡\lambda$, Theorem 2 presents a condition for this error to vanish based on a shrinking cell property for data-driven partitions (Lugosi and Nobel, 1996, Breiman et al., 1984, Devroye et al., 1996). For the estimation error, in Section 5, we use the Vapnik–Chervonenkis (1971) (VC) inequality (Vapnik, 1998; see also Devroye et al., 1996, Lugosi and Nobel, 1996) and characterize a concentration result on the empirical distributions, Lemma 3, that makes this error tend to zero as n tend to infinity with probability one.

In the second part of this work, we explore applications of our main result. In Section 7 consistency is demonstrated for multivariate statistically equivalent blocks—Gessaman's (1970) data-dependent partition, while Section 8 shows equivalent results for tree-structured vector quantizations (TSVQ) (Devroye et al., 1996, Breiman et al., 1984, Nobel, 2002). Importantly in both settings, a range of parametric values are characterized to obtain a family of density-free consistent estimates. The main challenge faced in deriving these results, is to prove the adopted shrinking cell condition, which is achieved from the adaptive nature of the data-driven partition schemes. Finally, some of the proofs and derivations are organized in the appendix.