Some properties of Rényi entropy and Rényi entropy rate

Abstract

In this paper, we define the conditional Rényi entropy and show that the so-called chain rule holds for the Rényi entropy. Then, we introduce a relation for the rate of Rényi entropy and use it to derive the rate of the Rényi entropy for an irreducible-aperiodic Markov chain. We also show that the bound for the Rényi entropy rate is simply the Shannon entropy rate.

Introduction

In 1948, Shannon [30] used some axioms to introduce Shannon entropy. Subsequently, a number of sets of axioms were proposed by Khinchin [16] and others [10]. In 1961, Rényi [27] generalized Shannon entropy to a one-parameter family of entropies by defining an entropy of order α which is called the Rényi entropy. Some authors have modified the axioms associated with Rényi entropy [10]. Recently, Jizba and Arimitsu [14] used the extant axioms for Shannon and Rényi entropies to introduce a set of new axioms on the basis of which one can get both Shannon and Rényi entropies.

The concept of Rényi entropy has a number of applications in coding theory [5], [9], statistical mechanics [7], [17], statistics and related fields [1], [13], [22], [34] and other areas (see for instance [3] and the references therein).

We know that in the case of Shannon entropy, conditional entropy can be derived for random variables. Furthermore, there is a relation between conditional Shannon entropy and the joint Shannon entropy of random variables. This relation is called chain rule [6]. In the case of conditional Rényi entropy of random variables, however, there is no established definition of this quantity yet. Cachin [4] has given a definition on the basis of conditional Shannon entropy. For this definition, however, the chain rule does not hold. But one can use the axioms introduced by Jizba and Arimitsu [14] to derive conditional Rényi entropy and thereby prove the validity of the chain rule.

The application of the conditional Rényi entropy can be found in many areas such as quantum systems [32], biomedical engineering [20], cryptography [4], fields related to statistics [15], economics [2] and other areas [8], [23].

Through the introduction of entropy in the probability theory, entropy and stochastic processes became linked and the entropy rate was defined for stochastic processes. For a discrete-time process X = (X_n)_n⩾1, the entropy at time n is defined as the entropy of the n-dimensional random vector of X, and the entropy rate is defined by the limit of the entropy at time n divided by n, when the limit exists. For a stationary stochastic process with finite state space, Shannon proved [30] that the Shannon entropy rate exists. He also obtained the entropy rate for an ergodic Markov chain in the form

$${\overline{H}}_1(X)=-\sum _i{\pi }_i\sum _j{p}_{\mathit{ij}}\log p_{\mathit{ij}}$$

where p_ij, i,j = 1,2, … ,n are transition probabilities and Π = (π_i), i = 1,2,...,n, is the stationary distribution of the chain. Here, Π = ΠP, where P = (p_ij) and ${\sum }_{i=1}^n{\pi }_i=1$.

The existence of the Shannon entropy rate for an irreducible Markov chain with infinite state space was proved by Klimko and Sucheston [18]. It can be shown that (1.1) is valid for the rate of Shannon entropy of an irreducible Markov chain with infinite state space.

The Rényi entropy rate was first defined by Rached et al. [25] for an ergodic Markov chain with a finite state space. The entropy rate of this process is expressed as

$${\overline{H}}_{\alpha }(X)=\frac{1}{1-\alpha }\log \lambda \alpha >0\text{,}\alpha \ne 1$$

where λ is the largest positive eigenvalue of the matrix $(p_{\mathit{ij}}^{\alpha })i\text{,}j=1\text{,}2\text{,}\dots \text{,}n$ and p_ij are transition probabilities of the chain. Recently, it has been shown in Ref. [24] that the Rényi entropy rate for an irreducible-aperiodic Markov chain with infinite state space is

$${\overline{H}}_{\alpha }(X)=\frac{1}{1-\alpha }\log R^{-1}\alpha >0\text{,}\alpha \ne 1$$

where R is the convergence radius of the matrix $(p_{\mathit{ij}}^{\alpha })$, i, j = 1, 2, …, and p_ij are transition probabilities of the chain.

The Rényi entropy rate has revealed several operational characteristics in coding theory and error exponent [5], [25], [26], [31]. Rényi entropy rate for stochastic processes is given by [12], and for fields related to statistics refer to [11].

This paper is organized as follows. In Section 2, the conditional Rényi entropy is obtained and some properties of the Rényi entropy are presented. In Section 3, we show that the chain rule holds for the Rényi entropy, and introduce a relation for obtaining the rate of Rényi entropy. Then, using this relation, the Rényi entropy rate for an irreducible-aperiodic Markov chain, with finite and infinite state spaces, is derived. Finally, in Section 4, we show that for an irreducible-aperiodic Markov chain the bound for the Rényi entropy rate is simply the Shannon entropy rate.