Skill Scores and modified Lorenz domination in default forecasts

doi:10.1016/j.econlet.2019.05.006

Economics Letters

Volume 181, August 2019, Pages 61-64

https://doi.org/10.1016/j.econlet.2019.05.006 Get rights and content

Highlights

•
We show how probability forecasts can be ranked for different sets of events.
•
This ranking generalizes the refinement ordering which is only applicable to identical sets of events.
•
We show which skill scores comply with Lorenz curves, extending Krämer and Neumärker (2016).

Abstract

We characterize the class of skill scores which comply with a novel partial ordering of probability forecasts based on Lorenz curves and overall default rates.

Introduction

Recent decades have witnessed a substantial surge in both empirical and theoretical interest in comparing the accuracy of default predictions in the rating industry (Hauck and Neyer, 2014, Boumparis et al., 2015, among many others). To the extent that default predictions can be viewed as probability forecasts, this is in part a straightforward application of the general theory of probability forecasting motivated mostly from meteorology. However, given the huge importance that default predictions now command in the banking industry, and the ensuing need to evaluate the quality of competing forecasters, a lot of current research takes its motivation from the credit rating business.

One strand of research investigates scalar measures of goodness such as the Brier score and related scoring rules (Winkler, 1996, Lahiri and Yang, 2013) or measures derived from ROC or Gini curves. Another one considers partial orderings such as the DeGroot and Fienberg (1983) refinement ordering which allow unanimous comparisons in cases where individual scores contradict each other (see Krämer and Güttler, 2008 for an example). Krämer (2006) combines these strands and shows that some probability forecaster $A$ dominates $B$ for all strictly proper scoring rules if and only if $A$ is more empirically refined than $B$ .

The present paper extends previous results to the case where the debtors under consideration are no longer identical. It is well known that conventional scoring rules are misleading in this case. Even the trivial forecaster which assigns the overall default rate to all obligors produces excellent results for small overall default probabilities. So skill scores must be used instead. Below we characterize the class of skill scores which complies with modified Lorenz domination of default predictions, extending (Krämer and Neumärker, 2016).

Section snippets

Skill scores

Let $0 = a_{1} < a_{2} < \dots < a_{k} = 1$ be a finite set of possible forecasts of default probabilities, which contains the overall default rate $p$ . This notation follows Krämer and Neumärker (2016). Let $q^{A} (a_{j})$ be the relative frequency with which default probability $a_{j}$ is predicted by forecaster $A$ (similarly for $B$ ). We only consider forecasts which are well calibrated, i.e. where $P (default | a_{j}) = a_{j} (j = 1, \dots, k)$ In addition, we confine ourselves to theoretical distributions, i.e. we do not distinguish between relative

Expected score functions and modified Lorenz dominance

Krämer and Neumärker (2016) show that the Brier skill score and the logarithmic skill score respect the modified Lorenz ordering. However, it is easy to construct skill scores from strictly proper scoring rules which do not. Take $R (a, q) = \sum_{j = i}^{k} q (a_{i}) (0 . 5^{4} - {(a_{i} - 0.5)}^{4})$ and consider two calibrated forecasters $D$ and $E$ with $q^{D} = {[0, 0.7, 0, 0.3]}^{'}$ , $q^{E} = {[0.7, 0, 0.3, 0]}^{'}$ , $a = {[0.1, 0.2, 0.4, 0.8]}^{'}$ and $p^{D} = 0.38$ , $p^{E} = 0.19$ . As the Lorenz curve is invariant under positive scaling, $D$ has the same Lorenz curve as $E$ . Therefore $D$

An application

As an illustration, Table 1 shows three year default rates obtained from the web pages of Moody’s and S&P (Moody’s, 2017, Standard & Poor’s, 2017).

Equalizing realized relative default frequencies and predicted default probabilities, both agencies are well calibrated by construction. Fig. 2 presents the resulting Lorenz curves; it shows that S&P predicted default probabilities are more spread out.

Since in addition $p^{M} = \sum_{i = 1}^{5} a_{i}^{M} q^{M} (a_{i}) = 1.60 % < p^{S} = \sum_{i = 1}^{5} a_{i}^{S} q^{S} (a_{i}) = 4.89 %,$ S&P dominates Moody’s in the

References (13)

BoumparisP. et al.
Has the crisis affected the behavior of the rating agencies? Panel evidence from the Eurozone
Econom. Lett.
(2015)
HauckA. et al.
Disagreement between rating agencies and bond opacity: A theoretical perspective
Econom. Lett.
(2014)
KrämerW. et al.
Comparing the accuracy of default predictions in the rating industry for different sets of obligors
Econom. Lett.
(2016)
LahiriK. et al.
Forecasting binary outcomes
DeGrootM. et al.
Probability forecasting, stochastic dominance, and the lorenz curve
DeGrootM. et al.
The comparison and evaluation of forecasters
Statistician
(1983)

There are more references available in the full text version of this article.

Cited by (2)

Unaccounted model risk for Basel IRB models deemed acceptable by conventional validation criteria
2023, Risk Management
A Technical Note on New Applications of Lorenz Curves in Business Based on Pareto Principles
2021, International Journal of Applied Economics, Finance and Accounting

^☆: Research supported by DFG-Sonderforschungsbereich 823.

View full text

Skill Scores and modified Lorenz domination in default forecasts☆

Highlights

Abstract

Introduction

Section snippets

Skill scores

Expected score functions and modified Lorenz dominance

An application

Econom. Lett.

Econom. Lett.

Econom. Lett.

Probability forecasting, stochastic dominance, and the lorenz curve

The comparison and evaluation of forecasters

Statistician