Abstract
The standard loss functions used in the literature on probabilistic prediction are the log loss function, the Brier loss function, and the spherical loss function; however, any computable proper loss function can be used for comparison of prediction algorithms. This note shows that the log loss function is most selective in that any prediction algorithm that is optimal for a given data sequence (in the sense of the algorithmic theory of randomness) under the log loss function will be optimal under any computable proper mixable loss function; on the other hand, there is a data sequence and a prediction algorithm that is optimal for that sequence under either of the two other standard loss functions but not under the log loss function.
An erratum to this chapter can be found at http://dx.doi.org/10.1007/978-3-319-23534-9_21
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bickel, J.E.: Some comparisons among quadratic, spherical, and logarithmic scoring rules. Decis. Anal. 4, 49–65 (2007)
Caruana, R., Niculescu-Mizil, A.: An empirical comparison of supervised learning algorithms. In: Proceedings of the Twenty Third International Conference on Machine Learning, pp. 161–168. ACM, New York (2006)
Dowe, D.L., Wallace, C.S.: Foreword re C. Comput. J. 51, 523–560 (2008)
Dowe, D.L.: Introduction to Ray Solomonoff 85th memorial conference. In: Dowe, D.L. (ed.) Algorithmic Probability and Friends. LNCS, vol. 7070, pp. 1–36. Springer, Heidelberg (2013)
Gurevich, Y., Passmore, G.O.: Impugning randomness, convincingly. Stud. Logica. 82, 1–31 (2012)
Haussler, D., Kivinen, J., Warmuth, M.K.: Sequential prediction of individual sequences under general loss functions. IEEE Trans. Inf. Theory 44, 1906–1925 (1998)
Kalnishkan, Y.: Predictive complexity for games with finite outcome spaces. In: Vovk, V., Papadopoulos, H., Gammerman, A. (eds.) Measures of Complexity: Festschrift for Alexey Chervonenkis, pp. 119–141. Springer, Berlin (2015)
Kalnishkan, Y., Vovk, V.: The existence of predictive complexity and the Legendre transformation.Technical report. CLRC-TR-00-04, Computer Learning Research Centre, Royal Holloway, University of London (March 2000)
Kalnishkan, Y., Vovk, V., Vyugin, M.V.: A criterion for the existence of predictive complexity for binary games. In: Ben-David, S., Case, J., Maruoka, A. (eds.) ALT 2004. LNCS (LNAI), vol. 3244, pp. 249–263. Springer, Heidelberg (2004)
Reid, M.D., Frongillo, R.M., Williamson, R.C., Mehta, N.: Generalized mixability via entropic duality. In: Proceedings of the 28th Conference on Learning Theory, pp. 1501–1522 (2015)
Vovk, V.: Aggregating strategies. In: Fulk, M., Case, J. (eds.) Proceedings of the Third Annual Workshop on Computational Learning Theory, pp. 371–383. Morgan Kaufmann, San Mateo (1990)
Vovk, V.: A game of prediction with expert advice. J. Comput. Syst. Sci. 56, 153–173 (1998)
Vovk, V.: Probability theory for the Brier game. Theoret. Comput. Sci. 261, 57–79 (2001)
Acknowledgments
I am grateful to Mitya Adamskiy, Yuri Kalnishkan, Ilia Nouretdinov, Ivan Petej, and Vladimir V’yugin for useful discussions. Thanks to an anonymous reviewer whose remarks prompted me to add Sect. 6 (and were used in both questions and answers). This work has been supported by EPSRC (grant EP/K033344/1) and the Air Force Office of Scientific Research (grant “Semantic Completions”).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Vovk, V. (2015). The Fundamental Nature of the Log Loss Function. In: Beklemishev, L., Blass, A., Dershowitz, N., Finkbeiner, B., Schulte, W. (eds) Fields of Logic and Computation II. Lecture Notes in Computer Science(), vol 9300. Springer, Cham. https://doi.org/10.1007/978-3-319-23534-9_20
Download citation
DOI: https://doi.org/10.1007/978-3-319-23534-9_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-23533-2
Online ISBN: 978-3-319-23534-9
eBook Packages: Computer ScienceComputer Science (R0)