On the Estimation of Entropy for Non-negative Data

Abstract

The traditional nonparametric entropy estimators based on the popular Rosenblatt–Parzen kernel density estimator using symmetric kernels may not be appropriate in the context of non-negative data. We, thus consider the Poisson weights-based density estimator instead, which is specifically tailored for non-negative data. The asymptotic properties of the new estimators are established and a simulation study is carried out to compare the performance of the new estimators, especially concerning those using the Rosenblatt–Parzen kernel density estimator, demonstrating a better performance of the new estimators.

Appendix 1: Proofs

1.1 Proof of Theorem 1

Writing $\hat{H}_f^{{\mathrm{Meanlog-Pois}}}$ as an integral with respect to the empirical distribution function $F_n(x)$, we have

$$\begin{aligned} -\frac{1}{n}\sum _{i=1}^{n}\log \hat{f}^{{\mathrm{Pois}}}_n(X_i)&=-\int _{0}^{\infty }\log \hat{f}^{{\mathrm{Pois}}}_n(x){\mathrm{d}}F_n(x)\\&=-\int _{0}^{\infty }\log f(x){\mathrm{d}}F_n(x) \\&\quad -\int _{0}^{\infty }\bigg (\log \hat{f}^{{\mathrm{Pois}}}_n(x)-\log f(x)\bigg ){\mathrm{d}}F_n(x)\\&=-\frac{1}{n}\sum _{i=1}^{n}\log f(X_i)-I_{n}, \end{aligned}$$

(A.1)

where

$$\begin{aligned} I_n:=\int _{0}^{\infty }\bigg (\log \hat{f}^{{\mathrm{Pois}}}_n(x)-\log f(x)\bigg ){\mathrm{d}}F_n(x). \end{aligned}$$

On the other hand, we know that $-\frac{1}{n}\sum _{i=1}^{n}\log f(X_i)$ is an unbiased, strongly consistent estimator for $H_f$ by the strong law of large numbers, and is root-n asymptotic normal by the central limit theorem if ${\mathbb {E}}[\log f(X)^2]<\infty$. That is

$$\begin{aligned} \left| - \frac{1}{n}\sum _{i=1}^{n}\log f(X_i)-H_f\right| {\mathop {\rightarrow }\limits ^{{\mathrm{a.s.}}}}0, \end{aligned}$$

and

$$\begin{aligned} \sqrt{n}\bigg ( -\frac{1}{n}\sum _{i=1}^{n}\log f(X_i)-H_f \bigg ){\mathop {\rightarrow }\limits ^{D}}{\mathcal {N}}(0,{\text {Var}}[\log f(X)]), \end{aligned}$$

as $n\rightarrow \infty .$ Therefore, it is sufficient to prove that

$$\begin{aligned} \sqrt{n}I_n {\mathop {\rightarrow }\limits ^{{\mathrm{a.s.}}}}0 \,\,\, {\text { as }}\,\, n\rightarrow \infty . \end{aligned}$$

In order to establish this, we decompose $I_n,$ into two parts as,

$$\begin{aligned} I_n&=\int _{0}^{\infty }\big (\log \hat{f}^{{\mathrm{Pois}}}_n(x)-\log f(x)\big )d(F_n(x)-F(x))\\&\quad +\int _{0}^{\infty }\big (\log \hat{f}^{{\mathrm{Pois}}}_n(x)-\log f(x)\big ){\mathrm{d}}F(x)\\&=I_{n,1}+I_{n,2}, \end{aligned}$$

(A.2)

where

$$\begin{aligned} I_{n,1}=\int _{0}^{\infty }\bigg (\log \hat{f}^{{\mathrm{Pois}}}_n(x)-\log f(x)\bigg ) d(F_n(x)-F(x)), \end{aligned}$$

(A.3)

$$\begin{aligned} {\mathrm{and}}~I_{n,2}=\int _{0}^{\infty }\bigg (\log \hat{f}^{{\mathrm{Pois}}}_n(x)-\log f(x)\bigg ) {\mathrm{d}}F(x) \end{aligned}$$

(A.4)

and analyze them separately.

• Analysis of $I_{n,2}$ Since the function $\log z$ is continuous and differentiable for all $z>0$, we can use the Taylor expansion centering at a to get

  $$\begin{aligned} \log z = \log a +\frac{z-a}{tz+(1-t)a}, \end{aligned}$$

  where $t\in (0,1)$. By letting $z=\hat{f}_n^{{\mathrm{Pois}}}(x)$ and $a=f(x)$, we obtain

  $$\begin{aligned} \log \hat{f}_n^{{\mathrm{Pois}}}(x)-\log f(x) = \frac{\hat{f}_n^{{\mathrm{Pois}}}(x)-f(x)}{t\hat{f}_n^{{\mathrm{Pois}}}(x)+(1-t)f(x)}. \end{aligned}$$

  .

  As $\Vert \hat{f}_n^{{\mathrm{Pois}}}-f\Vert _{\infty }{\mathop {\rightarrow }\limits ^{{\mathrm{a.s.}}}}0$ uniformly if $k=O(n^h)$ for $0<h<3/4$, it implies that $t\hat{f}_n^{{\mathrm{Pois}}}(x)+(1-t)f(x){\mathop {\rightarrow }\limits ^{{\mathrm{a.s.}}}}f(x)$. Meanwhile, Recall from Chaubey et al. [11] that if $f'(x)$ satisfies the Lipschitz of order $\alpha >0$, i.e., there exists a finite positive C such that

  $$\begin{aligned} |f'(s)-f'(t)|\le C|s-t|^{\alpha } \quad \forall s,t\in {\mathbb {R}}^+, \end{aligned}$$

  then for fixed $x\in {\mathbb {R}}$ we have

  $$\begin{aligned} \hat{f}_n^{{\mathrm{Pois}}}(x)-f(x)=\frac{1}{2k}f'(x)+O(k^{-1-\alpha }). \end{aligned}$$

  (A.5)

  Thus, by dominant convergence theorem (DCT), from (A.4) we have

  $$\begin{aligned} \sqrt{n}|I_{n,2}|&=\left| \int _{0}^{\infty }\sqrt{n}\bigg (\frac{\hat{f}_n^{{\mathrm{Pois}}} (x)-f(x)}{t\hat{f}_n^{{\mathrm{Pois}}}(x)+(1-t)f(x)}\bigg )f(x){\mathrm{d}}x\right| \\&=\left| \int _{0}^{\infty }\sqrt{n}\bigg (\frac{\frac{1}{2k}f'(x)+O(k^{-1-\alpha })}{t\hat{f}_n^{{\mathrm{Pois}}}(x)+(1-t)f(x)}\bigg )f(x){\mathrm{d}}x\right| \\&\approx \left| \int _{0}^{\infty }\sqrt{n}\bigg (\frac{\frac{1}{2k}f'(x)+O(k^{-1-\alpha })}{f(x)}\bigg )f(x){\mathrm{d}}x\right| \\&=\left| \int _{0}^{\infty }\sqrt{n}\left( \frac{1}{2k}f'(x)+O(k^{-1-\alpha })\right) {\mathrm{d}}x\right| \\&{\mathop {\rightarrow }\limits ^{{\mathrm{a.s.}}}} 0, \end{aligned}$$

  (A.6)

  under the condition that $k=O(n^h)$ for $1/2<h<3/4$ and $\int _{0}^{\infty }f'(x){\mathrm{d}}x<\infty$

• Analysis of $I_{n,1}$ By the law of the iterated logarithm, we have

  $$\begin{aligned} \Vert F_n-F\Vert _{\infty }=O\big (n^{-1/2}(\log \log n)^{1/2}\big )\,\, {\text { a.s.}} \end{aligned}$$

  (A.7)

  Also, from (A.5), given that $f'(x)/f(x)<\infty$ $\forall x\in {\mathbb {R}}^+$, we get

  $$\begin{aligned} \frac{\hat{f}_n^{{\mathrm{Pois}}}(x)}{f(x)}=1+\frac{1}{2k}\frac{f'(x)}{f(x)} +O(k^{-k-\alpha }) =1+O(k^{-1}). \end{aligned}$$

  (A.8)

  Thus, for $k=O(n^h), 1/2<h<3/4,$ using integration by parts, we have

  $$\begin{aligned} \sqrt{n}|I_{n,1}|&=\sqrt{n}\left| \bigg [\big (F_n(x)-F(x)\big )\log \bigg (\frac{\hat{f}_n^{{\mathrm{Pois}}}(x)}{f(x)}\bigg )\bigg ]^{\infty }_0\right. \\&\quad \left. -\int _{0}^{\infty }\frac{{\mathrm{d}}}{{\mathrm{d}}x}\bigg (\log \frac{\hat{f}_n^{{\mathrm{Pois}}}(x)}{f(x)} \bigg )\big ( F_n(x)-F(x) \big ){\mathrm{d}}x\right| \\&\le \sqrt{n}\Vert F_n-F\Vert _{\infty }\left| \log \bigg (\frac{\hat{f}_n^{{\mathrm{Pois}}}(x)}{f(x)} \bigg )\right| ^{\infty }_0\\&\quad +\sqrt{n}||F_n-F||_{\infty }\left| \int _{0}^{\infty }\frac{{\mathrm{d}}}{{\mathrm{d}}x} \bigg (\log \frac{\hat{f}_n^{{\mathrm{Pois}}}(x)}{f(x)} \bigg ){\mathrm{d}}x\right| \\&=2\sqrt{n}||F_n-F||_{\infty }\left| \log \bigg (\frac{\hat{f}_n^{{\mathrm{Pois}}}(x)}{f(x)}\bigg ) \right| ^{\infty }_0\\&{\mathop {\rightarrow }\limits ^{{\mathrm{a.s.}}}} 0, \end{aligned}$$

  (A.9)

  that follows from (A.7) and (A.8).

Now from Eqs. (A.2), (A.6), and (A.9) we get,

$$\begin{aligned} \sqrt{n}I_n{\mathop {\rightarrow }\limits ^{{\mathrm{a.s.}}}}0, \end{aligned}$$

which implies

$$\begin{aligned} \hat{H}^{{\mathrm{Meanlog-Pois}}}_f=-\frac{1}{n}\sum _{i=1}^{n}\log f(X_i)+o(n^{-1/2}) \,\,{\text {almost}}\,{\text {surely}}. \end{aligned}$$

This completes the proof of the theorem. □

1.2 Proof of Theorem 2

Let ${\tilde{F}}_n(x)=\int _{0}^{x}\hat{f}_n^{{\mathrm{Pois}}}(t){\mathrm{d}}t$ that is a smooth version of the empirical distribution function $F_n(x).$ Similarly, let ${{\tilde{S}}_n}(x)=1-{\tilde{F}}_n(x),$ be the corresponding smooth version of the empirical survival function $S_n(x)=1-F_n(x),$ and denote the survival function $1-F(x)$ by S(x). We can now decompose $\hat{H}_f^{{\mathrm{Plugin-Pois}}}$ as

$$\begin{aligned} \hat{H}_f^{{\mathrm{Plugin-Pois}}}&=-\int _{0}^{\infty }\hat{f}_n^{{\mathrm{Pois}}}(x)\log \hat{f}_n^{{\mathrm{Pois}}}(x){\mathrm{d}}x\\&=-\int _{0}^{\infty }\log \hat{f}_n^{{\mathrm{Pois}}}(x)d({\tilde{F}}_n(x)-F_n(x)+F_n(x))\\&=-\int _{0}^{\infty }\log \hat{f}_n^{{\mathrm{Pois}}}(x)dF_n(x)-\int _{0}^{\infty }\log \hat{f}_n^{{\mathrm{Pois}}}(x)d({\tilde{F}}_n(x)-F_n(x))\\&=-\frac{1}{n}\sum _{i=1}^{n}\log \hat{f}^{{\mathrm{Pois}}}_n(X_i)-\int _{0}^{\infty }\log \hat{f}_n^{{\mathrm{Pois}}}(x)d(S_n(x)-{\tilde{S}}_n(x))\\&=\hat{H}_f^{{\mathrm{Meanlog-Pois}}}-\int _{0}^{\infty }\bigg (\log \hat{f}_n^{{\mathrm{Pois}}}(x) -\log f(x)\bigg )d(S_n(x)-{\tilde{S}}_n(x))\\&\quad -\int _{0}^{\infty }\log f(x)d(S_n(x)-{\tilde{S}}_n(x))\\&=\hat{H}_f^{{\mathrm{Meanlog-Pois}}}-U_{n,1}-U_{n,2}, \end{aligned}$$

(A.10)

where

$$\begin{aligned} U_{n,1}&:= \int _{0}^{\infty }\bigg (\log \hat{f}_n^{{\mathrm{Pois}}}(x)-\log f(x)\bigg )d(S_n(x) -{\tilde{S}}_n(x)),\\ U_{n,2}&:= \int _{0}^{\infty }\log f(x)d(S_n(x)-{\tilde{S}}_n(x)). \end{aligned}$$

Therefore, with the asymptotic properties of $H_f^{{\mathrm{Meanlog-Pois}}}$ in the Theorem 1, it is sufficient to prove

$$\begin{aligned} \sqrt{n}U_{n,1}&{\mathop {\rightarrow }\limits ^{{\mathrm{a.s.}}}}&0,\\ \sqrt{n}U_{n,2}&{\mathop {\rightarrow }\limits ^{{\mathrm{a.s.}}}}&0. \end{aligned}$$

• Analysis of $U_{n,1}$ Recall that under the conditions $k\rightarrow \infty$, $n^{-1}k\rightarrow 0$, and f(x) being absolutely continuous having a bounded derivative $f'(\cdot )$ a.e. on ${\mathbb {R}}^+$, [8] showed that (see their Theorem 3.2)

  $$\begin{aligned} \Vert {\tilde{S}}_n-S_n\Vert _{\infty }=\sup _{x\in {\mathbb {R}}^+}\{ {\tilde{S}}_n(x)-S_n(x) \}=O(n^{-3/4}(\log n)^{1+\delta }) \ \ {\text{a.s.}} \, {\text { as }}\, n\rightarrow \infty , \end{aligned}$$

  (A.11)

  where $\delta >0$ is arbitrary. Also, from (A.8), given that $f'(x)/f(x)<\infty$ $\forall x\in {\mathbb {R}}^+$, we get

  $$\begin{aligned} \frac{\hat{f}_n^{{\mathrm{Pois}}}(x)}{f(x)}=1+\frac{1}{2k}\frac{f'(x)}{f(x)}+O(k^{-k-\alpha }) =1+O(k^{-1}). \end{aligned}$$

  As a result, using integration by parts, we get

  $$\begin{aligned} \sqrt{n}|U_{n,1}|&=\sqrt{n}\left| \int _{0}^{\infty }\log \bigg (\frac{\hat{f}_n^{{\mathrm{Pois}}} (x)}{f(x)}\bigg ) d\big (S_n(x)-{\tilde{S}}_n(x)\big )\right| \\&=\sqrt{n}\left| \left[ \log \bigg (\frac{\hat{f}_n^{{\mathrm{Pois}}}(x)}{f(x)}\bigg )\big (S_n(x) -{\tilde{S}}_n(x)\big )\right] ^{\infty }_0\right. \\&\quad \left. -\int _{0}^{\infty }\frac{{\mathrm{d}}}{{\mathrm{d}}x}\bigg (\log \frac{\hat{f}_n^{{\mathrm{Pois}}}(x)}{f(x)}\bigg ) \big (S_n(x)-{\tilde{S}}_n(x)\big ){\mathrm{d}}x\right| \\&\le \sqrt{n}\Vert {\tilde{S}}_n-S_n\Vert _{\infty }\left| \log \bigg (\frac{\hat{f}_n^{{\mathrm{Pois}}}(x)}{f(x)}\bigg )\right| ^{\infty }_0 \\&\quad +\sqrt{n}||{\tilde{S}}_n-S_n||_{\infty }\left| \log \bigg (\frac{\hat{f}_n^{{\mathrm{Pois}}} (x)}{f(x)}\bigg )\right| ^{\infty }_0\\&=2\sqrt{n}\Vert {\tilde{S}}_n-S_n\Vert _{\infty }\left| \log \bigg (\frac{\hat{f}_n^{{\mathrm{Pois}}}(x)}{f(x)}\bigg )\right| ^{\infty }_0\\&{\mathop {\rightarrow }\limits ^{{\mathrm{a.s.}}}} 0. \end{aligned}$$

  (A.12)

• Analysis of $U_{n,2}$ Similarly, we can use integration by part on $U_{n,2}$ to obtain

  $$\begin{aligned} \sqrt{n}|U_{n,2}|&=\sqrt{n}\left| \int _{0}^{\infty }\log f(x)d\big (S_n(x) -{\tilde{S}}_n(x)\big )\right| \\&=\sqrt{n}\left| \left[ \log f(x)\big (S_n(x)-{\tilde{S}}_n(x)\big ) \right] ^{\infty }_0\right. \\&\quad \left. -\int _{0}^{\infty }\frac{{\mathrm{d}}}{{\mathrm{d}}x}(\log f(x)) \big (S_n(x) -{\tilde{S}}_n(x)\big ){\mathrm{d}}x\right| \\&\le \sqrt{n}||{\tilde{S}}_n-S_n||_{\infty }\left| \log f(x)\right| ^{\infty }_0 +\sqrt{n}||{\tilde{S}}_n-S_n||_{\infty }\left| \log f(x)\right| ^{\infty }_0\\&=2\sqrt{n}||{\tilde{S}}_n-S_n||_{\infty }\left| \log f(x)\right| ^{\infty }_0\\&{\mathop {\rightarrow }\limits ^{{\mathrm{a.s.}}}} 0, \end{aligned}$$

  (A.13)

  where the last equation above follows by the assumption that f(x) is bounded away from zero, which implies that $\left| \log f(x)\right| <\infty$.

Therefore, using Eqs. (A.10), (A.12) and (A.13), we obtain

$$\begin{aligned} \hat{H}_f^{{\mathrm{Plugin-Pois}}}=-\frac{1}{n}\sum _{i=1}^{n}\log f(X_i)+o(n^{-1/2}) \,{\text{a.s.}}\,{\mathrm{as}}~n\rightarrow \infty , \end{aligned}$$

which completes the proof of the theorem. □