The skill of multi-model seasonal forecasts of the wintertime North Atlantic Oscillation

Abstract.

The skill assessment of a set of wintertime North Atlantic Oscillation (NAO) seasonal predictions in a multi-model ensemble framework has been carried out. The multi-model approach consists in merging the ensemble hindcasts of four atmospheric general circulation models forced with observed sea surface temperatures to create a multi-model ensemble. Deterministic (ensemble-mean based) and probabilistic (categorical) NAO hindcasts have been considered. Two different sets of NAO indices have been used to create the hindcasts. A first set is defined as the projection of model anomalies onto the NAO spatial pattern obtained from atmospheric analyses. The second set obtains the NAO indices by standardizing the leading principal component of each single-model ensemble. Positive skill is found with both sets of indices, especially in the case of the multi-model ensemble. In addition, the NAO definition based upon the single-model leading principal component shows a higher skill than the hindcasts obtained using the projection method. Using the former definition, the multi-model ensemble shows statistically significant (at 5% level) positive skill in a variety of probabilistic scoring measures. This is interpreted as a consequence of the projection method being less suitable because of the presence of errors in the spatial NAO patterns of the models. The positive skill of the seasonal NAO found here seems to be due not to the persistence of the long-term (decadal) variability specified in the initial conditions, but rather to a good simulation of the year-to-year variability. Nevertheless, most of the NAO seasonal predictability seems to be due to the correct prediction of particular cases such as the winter of 1989. The higher skill of the multi-model has been explained on the basis of a more reliable description of large-scale tropospheric wave features by the multi-model ensemble, illustrating the potential of multi-model experiments to better identify mechanisms that explain seasonal variability in the atmosphere.

Appendix 1: scoring rules

A tool commonly used to evaluate the association between ensemble-mean hindcasts and verification is the time correlation coefficient. This measure is independent of the mean and variance of both variables. As in the rest of the study, different climatologies for hindcasts and verification were computed using the cross-validation technique, making the correlation estimator unbiased (Déqué 1997).

A set of verification measures has been used to assess the quality of the probabilistic hindcasts: the ranked probability skill score (RPSS), the receiver operating characteristic (ROC) area under the curve, the Peirce skill score (PSS), and the odds ratio skill score (ORSS). Most of them, along with estimates of the associated error, are described in Stephenson (2000), Zhang and Casey (2000), and Thornes and Stephenson (2001), where the reader is referred to for more specific definitions and properties.

The accuracy measure for RPSS is the ranked probability score (RPS). RPS was first proposed by Epstein (1969b) and simplified by Murphy (1971). This score for categorical probabilistic forecasts is a generalization of the Brier score for ranked categories. For J ranked categories, the RPS can be written:

$$ RPS({\mathbf{r}},{\mathbf{d}}) = {{1}\over{J - 1}} \sum\limits_{i = 1}^J {\left(\sum\limits_{k = 1}^i {r_k} - \sum\limits_{k = 1}^i {d_k} \right)} ^2 $$

(1)

where the vector $\mathbf{r} = (r_1, \ldots , r_J) (\sum\nolimits_{k = 1}^J {r_k = 1})$ represents an estimate of the forecast PDF and d = (d ₁, …, d _J) corresponds to the verification PDF where d _k is a delta function which equals to 1 if category k occurs and 0 otherwise. By using cumulative probabilities, it takes into account the ordering of the categories, though for finite ensemble sizes, the estimated probabilities for the event to be in different categories strongly depend on the estimate of the category thresholds. RPS can be accumulated for several time steps or grid points over a region, or both. The RPSS expresses the relative improvement of the forecast against a reference score. The reference score used here has been the climatological probability hindcast, which, under the assumption of a Gaussian distribution of the observations, is the forecast without any skill that minimises the RPS (Déqué et al. 1994). The RPSS is defined as:

$$ RPSS = 100 \left(1 - \frac{RPS_{\text{forecast}}} {RPS_{\text{climatol}}} \right) $$

(2)

Such skill score is 100 for a perfect forecast, 0 for a probabilistic forecast which is no more accurate than a trivial forecast using long-term climatology, and negative for even worse forecasts, as random or biased values. To provide an estimate of the skill score significance, the calculations were repeated 100 times for a given time series (either a grid point or the NAO index). Each time, the order of the individual hindcasts was scrambled (this preserves the PDF of the variable), then computing the skill score, and finally taking the 5% upper threshold of the resulting skill distribution.

RPSS can be a too stringent measure of skill by requiring a correct estimate of a simplified PDF. Then, a set of simple accuracy measures for binary events is made based upon the hit rate H, or the relative number of times an event was forecast when it occurred, and the false alarm rate F, or the relative number of times the event was forecast when it did not occur (Jolliffe and Stephenson 2003). They are based on the likelihood-base rate factorization of the joint probability distribution of forecasts and verifications (Murphy and Winkler 1987). To derive them, a contingency table is computed, wherein the cells are occupied by the number of hits (a, number of cases when an event is forecast and is also observed), false alarms (b, number of cases the event is not observed but is forecast), misses (c, number of cases the event is observed but not forecast), and correct rejections (d, number of no-events correctly forecast) for every ensemble member. Then, the hit rate and the false alarm rate take the form:

$$ H = \frac{a} {a + c}\quad F = \frac{b} {b + d} $$

(3)

The previous scheme allows for the definition of a reliability measure, the bias B. Reliability is another attribute of forecast quality and corresponds to the ability of the forecast system to average probabilities equal to the frequency of the observed event. The bias indicates whether the forecasts of an event are being issued at a higher rate than the frequency of observed events. It reads:

$$ B = \frac{a + b} {a + c} $$

(4)

A bias greater than 1 indicates over-forecasting, i.e., the model forecasts the event more often than it is observed. Consequently, a bias lower than 1 indicates under-forecasting.

The Peirce skill score (PSS) is a simple measure of skill that equals to the difference between the hit rate and the false alarm rate:

$$ PSS = H - F $$

(5)

When the score is greater than zero, the hit rate exceeds the false alarm rate so that the closer the value of PSS to 1, the better. The standard error formula for this score assumes independence of hit and false alarm rates and, for large enough samples, it is computed as:

$$ \sigma _{PSS} = \sqrt{\frac{H(1 - H)} {a + c} + \frac{F(1 - F)} {b + d}} $$

(6)

The odds ratio (OR) is an accuracy measure that compares the odds of making a good forecast (a hit) to the odds of making a bad forecast (a false alarm):

$$ OR = \frac{H} {1 - H} \frac{1 - F} {F} $$

(7)

The ratio is greater than one when the hit rate exceeds the false alarm rate, and is unity when forecast and reference values are independent. It presents the advantage of being independent of the forecast bias. Furthermore, it has the property that the natural logarithm of the odds ratio is asymptotically normally distributed with a standard error of 1/(n _h )^1/2 where

$$ \frac{1} {n_h} = \frac{1} {a} + \frac{1} {b} + \frac{1} {c} + \frac{1} {d} $$

(8)

To test whether there is any skill, one can test against the null hypothesis that the forecasts and verifications are independent with log odds of zero. A simple skill score, the odds ratio skill score (ORSS), ranging from –1 to +1, where a score of zero represents no skill, may be obtained from the odds ratio through the expression:

$$ ORRS = \frac{OR - 1} {OR + 1} = \frac{H - F} {H + F - 2HF} $$

(9)

Thornes and Stephenson (2001) provide a useful table with the minimum values of ORSS needed to have significant skill at different levels of confidence depending on the value of n _h .

The ROC (Swets 1973) is a signal-detection curve plotting the hit rate against the false alarm rate for a specific event over a range of probability decision thresholds (Evans et al. 2000; Graham et al. 2000; Zhang and Casey 2000). Basically, it indicates the performance in terms of hit and false alarm rate stratified by the verification. The probability of detection is a probability decision threshold that converts probabilistic binary forecasts into deterministic binary forecasts. For each probability threshold, a contingency table is obtained from which the hit and false alarm rates are computed. For instance, consider a probability threshold of 10%. The event is forecast in those cases where the probability is equal to or greater than 10%. This calculation is repeated for thresholds of 20%, 30%, up to 100% (or whatever other selection of intervals, depending mainly on the ensemble size). Then, the hit rate is plotted against the false alarm rate to produce a ROC curve. Ideally, the hit rate will always exceed the false alarm rate and the curve will lie in the upper-left-hand portion of the diagram. The hit rate increases by reducing the probability threshold, but at the same time the false alarm rate is also increased. The standardized area enclosed beneath the curve is a simple accuracy measure associated with the ROC, with a range from 0 to 1. A system with no skill (made by either random or constant forecasts) will achieve hits at the same rate as false alarms and so its curve will lie along the 45° line and enclose a standardized area of 0.5. As the ROC is based upon a stratification by the verification it provides no information about reliability of the forecasts, and hence the curves cannot be improved by improving the climatology of the system. The skill score significance was assessed, as in the case of RPSS, by Monte Carlo methods.