Figure 1

Bias and average mean absolute error of various entropy estimators. (a) Average ${\log }_2$ transformed bias across the $[0,\log (M\left)\right]$ nats interval ($y$ axis), as a function of the number of observations $n$ ($x$ axis), with the number of states $M=16$. Black: our new combined ${\widehat{H}}_{\text{comb}}\left(\mathit{n}\right)$ entropy estimator [Eq. (26)]. Dashed green: our new polynomial estimator $\widehat{T}\left(\mathit{n}\right)$ [Eq. (10)]. Dashed red: NSB estimator [21]. Red: Bayesian estimator with uniform prior on $\mathit{p}$ [23]. Dashed blue: Grassberger estimator [13]. Dashed-dotted orange: Panzeri-Treves estimator [27]. Magenta: classic Miller-Madow estimator [12]. Cyan: plugin estimator [Eq. (1)]. Dashed black: 10$\%$ of the average entropy across the $[0,\ln (M\left)\right]$ nats interval. (b and c) As in (a), but now for $M=81,225$. (d) Average mean absolute error of the entropy estimates, averaged across the $[0,\log (M\left)\right]$ nats interval, with $M=16$. Legends are similar to (a–c). (e and f): Similar to (d), but now for $M=81,225$.Reuse & Permissions

Figure 2

(a–c) Entropy estimator bias for various levels of the entropy. Vectors of probabilities were generated by varying the concentration parameter $\beta$ of a Dirichlet distribution with steps of 1/10 and drawing one vector of probabilities $\mathit{p}$ from each Dirichlet distribution. The $x$ axis corresponds to the entropy of $\mathit{p}$. The $y$ axis corresponds to the estimator bias in nats. Black: our new combined ${\widehat{H}}_{\text{comb}}\left(\mathit{n}\right)$ entropy estimator [Eq. (26)]. Dashed green: our new polynomial estimator $\widehat{T}\left(\mathit{n}\right)$ [Eq. (10)]. Dashed red: NSB estimator [21]. Red: Bayesian estimator with uniform prior on $\mathit{p}$ [23]. Dashed blue: Grassberger estimator [13]. Dashed-dotted orange: Panzeri-Treves estimator [27]. Magenta: classic Miller-Madow estimator [12]. Dashed cyan: plugin estimator [Eq. (1)]. Dashed black: to be estimated entropy of $\mathit{p}$ (true entropy). (a–c) correspond to number of states $M=16,81,225$ and number of observations $n=\sqrt{M}=4,9,15$, respectively. (d–f) Expected value of various entropy estimators ($y$ axis) as a function of the number of observations $n$ with the expected entropy of $\mathit{p}$ equal to $\ln \left(M\right)-1/1000$, that is, close to the maximum entropy. (d–f) correspond to number of states $M=16,81,225$, respectively. (g–i) Similar to (d–f), but now for the expected entropy of $\mathit{p}$ equal to 1/10000, that is, close to the minimum entropy. The Bayesian estimator with uniform prior on $\mathit{p}$ is not shown because of scaling.Reuse & Permissions