Mathematical inequalities for some divergences

doi:10.1016/j.physa.2011.07.052

Physica A: Statistical Mechanics and its Applications

Volume 391, Issues 1–2, 1 January 2012, Pages 388-400

https://doi.org/10.1016/j.physa.2011.07.052 Get rights and content

Abstract

Some natural phenomena are deviating from standard statistical behavior and their study has increased interest in obtaining new definitions of information measures. But the steps for deriving the best definition of the entropy of a given dynamical system remain unknown. In this paper, we introduce some parametric extended divergences combining Jeffreys divergence and Tsallis entropy defined by generalized logarithmic functions, which lead to new inequalities. In addition, we give lower bounds for one-parameter extended Fermi–Dirac and Bose–Einstein divergences. Finally, we establish some inequalities for the Tsallis entropy, the Tsallis relative entropy and some divergences by the use of the Young’s inequality.

Highlights

► We introduce some extended divergences combining Jeffreys and Tsallis divergence. ► Then we give new some inequalities for them. ► We also give lower bounds for extended Fermi–Dirac and Bose–Einstein divergences. ► Finally, we establish some inequalities for some divergences by Young inequality.

Introduction

For the study of multifractals, in 1988, Tsallis [1] introduced one-parameter extended entropy of Shannon entropy by $H_{q} (p) \equiv - \sum_{j = 1}^{n} p_{j}^{q} {ln}_{q} p_{j} = \sum_{j = 1}^{n} p_{j} {ln}_{q} \frac{1}{p_{j}}, (q \geq 0, q \neq 1)$ where $p = {p_{1}, p_{2}, \dots, p_{n}}$ is a probability distribution with $p_{j} > 0$ for all $j = 1, 2, \dots, n$ and the $q -$ logarithmic function for $x > 0$ is defined by ${ln}_{q} (x) \equiv \frac{x^{1 - q} - 1}{1 - q}$ which uniformly converges to the usual logarithmic function $log (x)$ in the limit $q \to 1$ . Therefore Tsallis entropy converges to Shannon entropy in the limit $q \to 1$ : $lim_{q \to 1} H_{q} (p) = H_{1} (p) \equiv - \sum_{j = 1}^{n} p_{j} log p_{j} .$ It is also known that Rényi entropy [2] $R_{q} (p) \equiv \frac{1}{1 - q} log (\sum_{j = 1}^{n} p_{j}^{q})$ is a one-parameter extension of Shannon entropy.

For two probability distributions $p = {p_{1}, p_{2}, \dots, p_{n}}$ and $r = {r_{1}, r_{2}, \dots, r_{n}}$ we have divergences based on these quantities (1), (3). We denote by $D_{q} (p ∥ r) \equiv \sum_{j = 1}^{n} p_{j}^{q} ({ln}_{q} p_{j} - {ln}_{q} r_{j}) = - \sum_{j = 1}^{n} p_{j} {ln}_{q} \frac{r_{j}}{p_{j}} .$ Tsallis relative entropy. Tsallis relative entropy converges to the usual relative entropy (divergence, Kullback–Leibler information) in the limit $q \to 1$ : $lim_{q \to 1} D_{q} (p ∥ r) = D_{1} (p ∥ r) \equiv \sum_{j = 1}^{n} p_{j} (log p_{j} - log r_{j}) .$ We also denote by $R_{q} (p ∥ r)$ the Rényi relative entropy [2] defined by $R_{q} (p ∥ r) \equiv \frac{1}{q - 1} log (\sum_{j = 1}^{n} p_{j}^{q} r_{j}^{1 - q}) .$ Obviously ${lim}_{q \to 1} R_{q} (p ∥ r) = D_{1} (p ∥ r)$ .

The divergences can be considered to be a generalization of entropies in the sense that Shannon entropy can be reproduced by the divergence $log n - D_{1} (p ∥ u)$ for the uniform distribution $u = {1 / n, 1 / n, \dots, 1 / n}$ . Therefore the study of divergences is important for the development of information science. In this paper, we study several mathematical inequalities related to some generalized divergences.

Section snippets

Two parameter entropies and divergences

In this section and throughout the rest of the paper we consider $p = {p_{1}, p_{2}, \dots, p_{n}}$ and $r = {r_{1}, r_{2}, \dots, r_{n}}$ with $p_{j} > 0, r_{j} > 0$ for all $j = 1, 2, \dots, n$ to be probability distributions.

We start by introducing the Tsallis quasilinear entropies, that is we extend the quasilinear entropies using the Tsallis recipe.

Definition 2.1

For a continuous and strictly monotonic function $ψ$ on $(0, \infty)$ and $r \geq 0$ with $r \neq 1$ (the nonextensivity parameter), Tsallis quasilinear entropy ( $r$ -quasilinear entropy) is defined by $I_{r}^{ψ} (p) \equiv {ln}_{r} ψ^{- 1} (\sum_{j = 1}^{n} p_{j} ψ (\frac{1}{p_{j}})) .$

Tsallis type divergences

In what follows Jeffreys and Jensen–Shannon divergences are extended in the context of Tsallis theory. We firstly review their definitions.

Definition 3.1

[18], [19]

The Jeffreys divergence is defined by $J_{1} (p ∥ r) \equiv D_{1} (p ∥ r) + D_{1} (r ∥ p)$ and the Jensen–Shannon divergence is defined as $J S_{1} (p ∥ r) \equiv \frac{1}{2} D_{1} (p ∥ \frac{p + r}{2}) + \frac{1}{2} D_{1} (r ∥ \frac{p + r}{2}) .$

Analogously we may define the following divergences.

Definition 3.2

The Jeffreys–Tsallis divergence is $J_{r} (p ∥ r) \equiv D_{r} (p ∥ r) + D_{r} (r ∥ p)$ and the Jensen–Shannon–Tsallis divergence is $J S_{r} (p ∥ r) \equiv \frac{1}{2} D_{r} (p ∥ \frac{p + r}{2}) + \frac{1}{2} D_{r} (r ∥ \frac{p + r}{2}) .$

We find that $J_{r} (p$

Fermi–Dirac and Bose–Einstein type divergences

Fermi–Dirac and Bose–Einstein divergences are very often relevent in physics problems related to elementary particles which obey such statistics, called fermions, respectively bosons. As one-parameter extension of Fermi–Dirac entropy and Bose–Einstein entropy (see also Refs. [22], [23], [24], [25]), that is of $I_{1}^{F D} (p) \equiv - \sum_{j = 1}^{n} p_{j} log p_{j} - \sum_{j = 1}^{n} (1 - p_{j}) log (1 - p_{j})$ and $I_{1}^{B E} (p) \equiv - \sum_{j = 1}^{n} p_{j} log p_{j} + \sum_{j = 1}^{n} (1 + p_{j}) log (1 + p_{j}),$ the Fermi–Dirac–Tsallis entropy was introduced in Ref. [23] and physical phenomena for power-law

Young’s inequality and Tsallis entropies with finite sum

We establish more inequalities involving Tsallis entropy and Tsallis relative entropy applying Young’s inequality.

Lemma 5.1 Young’s inequality

Let $m, n \geq 0$ and $p, q \in R$ such that $\frac{1}{p} + \frac{1}{q} = 1$ . If $p < 0$ (then $0 < q < 1$ ) or $0 < p < 1$ (then $q < 0$ ), then one has $\frac{m^{p}}{p} + \frac{n^{q}}{q} \leq m n$ .

Lemma 5.2

(i)
Let $p, q \in R$ satisfying $\frac{1}{1 - p} + \frac{1}{1 - q} = 1$ . If $p > 1$ and $0 < q < 1$ , or if $0 < p < 1$ and $q > 1$ , then ${ln}_{p} x + {ln}_{q} y \leq x y - 1 .$
(ii)
Let $p, q \in R$ satisfying $\frac{1}{p - 1} + \frac{1}{q - 1} = 1$ . If $p < 1$ and $1 < q < 2$ , or if $1 < p < 2$ and $q < 1$ , then ${ln}_{p} \frac{1}{x} + {ln}_{q} \frac{1}{y} \geq - x y + 1 .$

Proof

(i)
Using Lemma 5.1, we obtain ${ln}_{p} x + {ln}_{q} y = \frac{x^{1 - p} - 1}{1 - p} + \frac{y^{1 - q} - 1}{1 - q} \leq x y - 1 .$
(ii)
Lemma 5.1 leads to ${ln}_{p} \frac{1}{x} + {ln}_{q}$

Concluding remarks

As we have seen, we established a generalized inequality for a new quantity defined by Jeffreys divergence and Tsallis entropy. In addition, we gave lower bounds for one-parameter extended Fermi–Dirac and Bose–Einstein divergences. These lower bounds give the sharper inequalities than their nonnegativity. The nonnegativity of Tsallis divergence was applied to obtain the maximum entropy principle in Ref. [29]. Therefore one could check if also the nonnegativity of one-parameter extended

Acknowledgments

The author (S.F.) was supported in part by the Japanese Ministry of Education, Science, Sports and Culture, Grant-in-Aid for Encouragement of Young Scientists (B), 20740067. The author (F.-C. M.) was supported by CNCSIS Grant 420/2008.

References (30)

M. Masi
A step beyond Tsallis and Rényi entropies
Phys. Lett. A
(2005)
H. Suyari et al.
Multiplicative duality, $q$ -triplet and $(μ, ν, q)$ -relation derived from the one-to-one correspondence between the $(μ, ν)$ -multinomial coefficient and Tsallis entropy $S_{q}$
Physica A
(2008)
H. Suyari
Mathematical structures derived from the $q$ -multinomial coefficient in Tsallis statistics
Physica A
(2006)
A.M. Teweldeberhan et al.
On the cut-off prescriptions associated with power-law generalized thermostatistics
Phys. Lett. A
(2005)
F. Büyükkilic et al.
A fractal approach to entropy and distribution functions
Phys. Lett. A
(1993)
J.M. Conroy et al.
Thermodynamic consistency of the $q$ -deformed Fermi–Dirac distribution in nonextensive thermostatics
Phys. Lett. A
(2010)
C. Tsallis
Possible generalization of Bolzmann–Gibbs statistics
J. Stat. Phys.
(1988)
A. Rényi
On measures of entropy and information
B.D. Sharma et al.
New nonadditive measures of inaccuracy
J. Math. Sci.
(1975)
B.D. Sharma et al.
New nonadditive measures of relative information
J. Comb. Inform. Syst. Sci.
(1977)

V. Schwämmle et al.

Two-parameter generalization of the logarithm and exponential functions and Boltzmann–Gibbs–Shannon entropy

J. Math. Phys.

(2007)

S. Arimoto

Information-theoretic considerations on estimation problems

Inform. Control

(1971)

E. Boekee et al.

The $R$ -norm information measure

Inform. Control

(1980)

E. Aktürk, G.B. Bağci, R. Sever, Is Sharma–Mittal entropy really a step beyond Tsallis and Rényi entropies,...

M. Masi, Generalized information-entropy measures and Fisher information,...

Cited by (33)

On Jensen–Rényi and Jeffreys–Rényi type f-divergences induced by convex functions
2020, Physica A: Statistical Mechanics and its Applications
In this paper, Rényi type $f$ -divergences like Jensen–Rényi and Jeffreys–Rényi are studied for convex functions. Some comparison theorems for two divergences are provided. The obtained results imply for introduced divergences some new inequalities corresponding to nonnegative convex functions. Some examples of known entropies are introduced in order to show a few applications of new divergences.
Inequalities related to some types of entropies and divergences
2019, Physica A: Statistical Mechanics and its Applications
The aim of this paper is to discuss new results concerning some kinds of parametric extended entropies and divergences. As a result of our studies for mathematical properties on entropy and divergence, we give new bounds for the Tsallis quasilinear entropy and divergence by applying the Hermite-Hadamard inequality. We also give bounds for biparametric extended entropies and divergences which have been given in Furuichi (2010). In addition, we study $(r, q)$ -quasilinear entropies and divergences as alternative biparametric extended entropy and divergence, and then we give bounds for them. Finally we obtain inequalities for an extended Lin’s divergence and some characterizations of Fermi-Dirac entropy and Bose-Einstein entropy.
The residual extropy of order statistics
2018, Statistics and Probability Letters
Generalizations of Crooks and Lin's results on Jeffreys–Csiszár and Jensen–Csiszár f-divergences
2016, Physica A: Statistical Mechanics and its Applications
Citation Excerpt :
Therefore we now introduce the following notions. Bounding a given divergence by other ones is an important research problem [6,9,5,1,7,2,17,8]. Each refinement of such bounds provides a deeper insight into physical interpretations and applications.
Tsallis and Rényi divergences of generalized Jacobi polynomials
2016, Physica A: Statistical Mechanics and its Applications
Citation Excerpt :
Tsallis and Rényi entropies generalize Shannon entropy. Tsallis entropy was primarily introduced by C. Tsallis in his paper [1] and, in the last couple of years, it has been intensively studied (see e.g., Refs. [2–6] and many other papers). Rényi entropy appeared for the first time in Ref. [7].
Bounds for Jeffreys-Tsallis and Jensen-Shannon-Tsallis divergences
2014, Physica A: Statistical Mechanics and its Applications

View all citing articles on Scopus

View full text

Mathematical inequalities for some divergences

Abstract

Highlights

Introduction

Section snippets

Two parameter entropies and divergences

Tsallis type divergences

[18], [19]

Fermi–Dirac and Bose–Einstein type divergences

Young’s inequality and Tsallis entropies with finite sum

Concluding remarks

Acknowledgments

Phys. Lett. A

Physica A

Physica A

Phys. Lett. A

Phys. Lett. A

Phys. Lett. A

Possible generalization of Bolzmann–Gibbs statistics

J. Stat. Phys.

On measures of entropy and information

New nonadditive measures of inaccuracy

J. Math. Sci.

New nonadditive measures of relative information

J. Comb. Inform. Syst. Sci.

Two-parameter generalization of the logarithm and exponential functions and Boltzmann–Gibbs–Shannon entropy

J. Math. Phys.

Information-theoretic considerations on estimation problems

Inform. Control

The R-norm information measure

Inform. Control

The $R$ -norm information measure