Abstract
This chapter briefly summarizes some basic numerical measures of uncertainty, from interval bounds to information-theoretic quantities such as (Shannon) information and entropy. This discussion then naturally leads to consideration of distances (and distance-like functions) between probability measures.
As we know, there are known knowns. There are things we know we know. We also know there are known unknowns. That is to say we know there are some things we do not know. But there are also unknown unknowns, the ones we don’t know we don’t know.
Donald Rumsfeld
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
A. D. Aleksandrov. Additive set functions in abstract spaces. Mat. Sb., 8: 307–348, 1940.
A. D. Aleksandrov. Additive set functions in abstract spaces. Mat. Sb., 9: 536–628, 1941.
A. D. Aleksandrov. Additive set functions in abstract spaces. Mat. Sb., 13: 169–238, 1943.
S. M. Ali and S. D. Silvey. A general class of coefficients of divergence of one distribution from another. J. Roy. Statist. Soc. Ser. B, 28:131–142, 1966.
L. Ambrosio, N. Gigli, and G. Savaré. Gradient Flows in Metric Spaces and in the Space of Probability Measures. Lectures in Mathematics ETH Zürich. Birkhäuser Verlag, Basel, second edition, 2008.
B. M. Ayyub and G. J. Klir. Uncertainty Modeling and Analysis in Engineering and the Sciences. Chapman & Hall/CRC, Boca Raton, FL, 2006. doi: 10.1201/9781420011456.
P. Billingsley. Probability and Measure. Wiley Series in Probability and Mathematical Statistics. John Wiley & Sons Inc., New York, third edition, 1995.
K. Chaloner and I. Verdinelli. Bayesian experimental design: a review. Statist. Sci., 10(3):273–304, 1995.
T. M. Cover and J. A. Thomas. Elements of Information Theory. Wiley-Interscience [John Wiley & Sons], Hoboken, NJ, second edition, 2006.
I. Csiszár. Eine informationstheoretische Ungleichung und ihre Anwendung auf den Beweis der Ergodizität von Markoffschen Ketten. Magyar Tud. Akad. Mat. Kutató Int. Közl., 8:85–108, 1963.
M. M. Deza and E. Deza. Encyclopedia of Distances. Springer, Heidelberg, third edition, 2014. doi: 10.1007/978-3-662-44342-2.
A. L. Gibbs and F. E. Su. On choosing and bounding probability metrics. Int. Stat. Rev., 70(3):419–435, 2002. doi: 10.1111/j.1751-5823.2002. tb00178.x.
E. Hansen and G. W. Walster. Global Optimization using Interval Analysis, volume 264 of Monographs and Textbooks in Pure and Applied Mathematics. Marcel Dekker, Inc., New York, second edition, 2004.
L. Jaulin, M. Kieffer, O. Didrit, and É. Walter. Applied Interval Analysis: With Examples in Parameter and State Estimation, Robust Control and Robotics. Springer-Verlag London Ltd., London, 2001. doi: 10.1007/ 978-1-4471-0249-6.
E. T. Jaynes. Probability Theory: The Logic of Science. Cambridge University Press, Cambridge, 2003. doi: 10.1017/CBO9780511790423. Edited and with a foreword by G. Larry Bretthorst.
C. Kraft. Some conditions for consistency and uniform consistency of statistical procedures. Univ. California Publ. Statist., 2:125–141, 1955.
S. Kullback and R. A. Leibler. On information and sufficiency. Ann. Math. Statistics, 22:79–86, 1951.
D. V. Lindley. On a measure of the information provided by an experiment. Ann. Math. Statist., 27:986–1005, 1956.
D. J. C. MacKay. Information Theory, Inference and Learning Algorithms. Cambridge University Press, New York, 2003.
R. E. Moore. Interval Analysis. Prentice-Hall Inc., Englewood Cliffs, N.J., 1966.
T. Morimoto. Markov processes and the H-theorem. J. Phys. Soc. Japan, 18:328–331, 1963.
M. S. Pinsker. Information and Information Stability of Random Variables and Processes. Translated and edited by Amiel Feinstein. Holden-Day, Inc., San Francisco, Calif.-London-Amsterdam, 1964.
C. E. Shannon. A mathematical theory of communication. Bell System Tech. J., 27:379–423, 623–656, 1948.
T. Steerneman. On the total variation and Hellinger distance between signed measures; an application to product measures. Proc. Amer. Math. Soc., 88(4):684–688, 1983. doi: 10.2307/2045462.
C. Villani. Topics in Optimal Transportation, volume 58 of Graduate Studies in Mathematics. American Mathematical Society, Providence, RI, 2003. doi: 10.1007/b12016.
C. Villani. Optimal Transport: Old and New, volume 338 of Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]. Springer-Verlag, Berlin, 2009. doi: 10.1007/978-3-540-71050-9.
L. Wasserman. Bayesian model selection and model averaging. J. Math. Psych., 44(1):92–107, 2000. doi: 10.1006/jmps.1999.1278.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Sullivan, T.J. (2015). Measures of Information and Uncertainty. In: Introduction to Uncertainty Quantification. Texts in Applied Mathematics, vol 63. Springer, Cham. https://doi.org/10.1007/978-3-319-23395-6_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-23395-6_5
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-23394-9
Online ISBN: 978-3-319-23395-6
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)