Entropy estimation of very short symbolic sequences

Annick Lesne, Jean-Luc Blanc, and Laurent Pezard

Phys. Rev. E 79, 046208 – Published 7 April 2009

Abstract

While entropy per unit time is a meaningful index to quantify the dynamic features of experimental time series, its estimation is often hampered in practice by the finite length of the data. We here investigate the performance of entropy estimation procedures, relying either on block entropies or Lempel-Ziv complexity, when only very short symbolic sequences are available. Heuristic analytical arguments point at the influence of temporal correlations on the bias and statistical fluctuations, and put forward a reduced effective sequence length suitable for error estimation. Numerical studies are conducted using, as benchmarks, the wealth of different dynamic regimes generated by the family of logistic maps and stochastic evolutions generated by a Markov chain of tunable correlation time. Practical guidelines and validity criteria are proposed. For instance, block entropy leads to a dramatic overestimation for sequences of low entropy, whereas it outperforms Lempel-Ziv complexity at high entropy. As a general result, the quality of entropy estimation is sensitive to the sequence temporal correlation hence self-consistently depends on the entropy value itself, thus promoting a two-step procedure. Lempel-Ziv complexity is to be preferred in the first step and remains the best estimator for highly correlated sequences.

Received 19 December 2008

DOI:https://doi.org/10.1103/PhysRevE.79.046208

Authors & Affiliations

Annick Lesne^1,*, Jean-Luc Blanc², and Laurent Pezard²

¹Institut des Hautes Études Scientifiques, Le Bois-Marie, 35 route de Chartres, F-91440 Bures-sur-Yvette, France
²Laboratoire de Neurosciences Intégratives et Adaptatives, UMR 6149 CNRS, Université d'Aix-Marseille , 3 Place Victor Hugo, F-13331 Marseille Cedex 03, France

^*Permanent address: LPTMC, Université Pierre et Marie Curie-Paris 6, 4 place Jussieu, F-75252 Paris Cedex 05, France

Article Text (Subscription Required)

Click to Expand

References (Subscription Required)

Click to Expand

Issue

Vol. 79, Iss. 4 — April 2009

Reuse & Permissions

Access Options

Author publication services for translation and copyediting assistance advertisement

Images

Figure 1
(Color online) Behavior of the block-entropy estimator ${\hat{H}}_{n}$ when increasing the word length $n$ in the deterministic case (left) and the stochastic case (right) for binary sequences of length $N = 1000$ . In the deterministic case, sequences were obtained using the logistic map with different parameter values: $a = a_{c} \approx 3.569 (h = 0)$ for the critical behavior with long-range correlations, $a = 3.83$ for the periodic regime $(h = 0)$ , and $a = 4$ for the fully chaotic regime (where $H_{n} = n ln 2$ and $h = ln 2$ ). In the stochastic case, fully chaotic behavior was obtained using $a = 1 / 2$ , periodic dynamics using $a = 0.1$ , and critical behavior using $a = 10^{- 3} ⪡ 1$ . The vertical dashed line indicates the theoretical location $n^{*} = ln N / h$ of the crossover between good and bad statistics when $h > 0$ (chaotic regime) above which ${\hat{H}}_{n}$ saturates to a value $ln N$ .Reuse & Permissions
Figure 2
(Color online) Entropy estimation quality ( $\hat{h} - h$ ; first row) and standard deviation of the estimation ( $S D \hat{h}$ ; second row) as a function of the true entropy value $h$ in the case of a deterministic evolution (left column) and in the stochastic case (right column). In the deterministic case, sequences are generated from a logistic map with control parameter $a ∊ [3.5, 4]$ . In the stochastic case, sequences are generated from a Markov chain with a transition matrix parameter $a ∊ [0, 1 / 2]$ . The exact value $h$ of the entropy is computed as the Lyapunov exponent from a very long typical trajectory in the logistic case; it is known analytically in the Markov case. The estimators are calculated for 100 symbolic sequences of length $N = 1000$ generated with random initial conditions for 500 values of the control parameter $a$ . Values represented on the figures are the mean value (first row) and standard deviation (second row) of these 100 estimations.Reuse & Permissions
Figure 3
(Color online) Entropy estimation quality ( $\hat{h} - h$ ; first row) and standard deviation of the estimation ( $S D \hat{h}$ ; second row) as a function of the average integrated correlation time $τ_{n}$ in the deterministic case (left column) and stochastic case (right column). The numerical procedure and graphical representations are similar to those described in Fig. 2.Reuse & Permissions
Figure 4
(Color online) Entropy estimation quality $\hat{h} - h$ as a function of the sequence length $N$ in the deterministic case (left column) and stochastic case (right column) for full randomness (top row) and infinite-range correlations (bottom row). The insets depict the absolute value of the error $(| \hat{h} - h |)$ as a function of the sequence length $(N)$ in a semilogarithmic scale for longer sequences $(200 \leq N \leq 10^{4})$ for each case. Fully random sequences are generated using the chaotic logistic map with parameter $a = 4$ (deterministic case) and an uncorrelated Markov chain with parameter $a = 1 / 2$ (stochastic case). Sequences with infinite-range correlations are generated using the critical logistic map with parameter $a = a_{c} \approx 3.569$ (deterministic case) and Markov chain with parameter $a = 10^{- 3} ⪡ 1$ , i.e., with correlation time $τ (a) \approx 1 / a ⪢ 1$ (stochastic case). The entropy of 200 symbolic sequences with random initial conditions and varying length $N ∊ [200, 2000]$ was computed using each estimator. Values represented on the figures correspond to the mean value over these 200 sequences. Standard deviation is depicted using vertical error bars. For the deterministic case and critical behavior, the captions are the same as for the other plots but for clarity, they are not depicted on the figure.Reuse & Permissions

Physical Review E

covering statistical, nonlinear, biological, and soft matter physics