Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-848d4c4894-p2v8j Total loading time: 0.001 Render date: 2024-05-22T17:29:59.890Z Has data issue: false hasContentIssue false

Chapter 6 - Ensemble-based atmospheric data assimilation

Published online by Cambridge University Press:  03 December 2009

By
Thomas M. Hamill
Edited by
Tim Palmer  and
Renate Hagedorn
Thomas M. Hamill
Affiliation:
Physical Sciences Division, NOAA Earth System Research Laboratory, Boulder
Tim Palmer
Affiliation:
European Centre for Medium-Range Weather Forecasts
Renate Hagedorn
Affiliation:
European Centre for Medium-Range Weather Forecasts
Get access

Summary

Ensemble-based data assimilation techniques are being explored as possible alternatives to current operational analysis techniques such as three- or four-dimensional variational assimilation. Ensemble-based assimilation techniques utilise an ensemble of parallel data assimilation and forecast cycles. The background-error covariances are estimated using the forecast ensemble and are used to produce an ensemble of analyses. The background-error covariances are flow dependent and often have very complicated structure, providing a very different adjustment to the observations than are seen from methods such as three-dimensional variational assimilation. Though computationally expensive, ensemble-based techniques are relatively easy to code, since no adjoint nor tangent linear models are required, and previous tests in simple models suggest that dramatic improvements over existing operational methods may be possible.

A review of the ensemble-based assimilation is provided here, starting from the basic concepts of Bayesian assimilation. Without some simplification, full Bayesian assimilation is computationally impossible for model states of large dimension. Assuming normality of error statistics and linearity of error growth, the state and its error covariance may be predicted optimally using Kalman filter (KF) techniques. The ensemble Kalman filter (EnKF) is then described. The EnKF is an approximation to the KF in that background-error covariances are estimated from a finite ensemble of forecasts. However, no assumptions about linearity of error growth are made. Recent algorithmic variants on the standard EnKF are also described, as well as methods for simplifying the computations and increasing the accuracy. Examples of ensemble-based assimilations are provided in simple and more realistic dynamical systems.

Type
Chapter
Information
Predictability of Weather and Climate , pp. 124 - 156
Publisher: Cambridge University Press
Print publication year: 2006

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Anderson, J. L. (2001). An ensemble adjustment filter for data assimilation. Mon. Weather Rev., 129, 2884–9032.0.CO;2>CrossRefGoogle Scholar
Anderson, J. L. (2003). A local least squares framework for ensemble filtering. Mon. Weather Rev., 131, 634–422.0.CO;2>CrossRefGoogle Scholar
Anderson, J. L. and Anderson, S. L. (1999). A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Mon. Weather Rev., 127, 2741–582.0.CO;2>CrossRefGoogle Scholar
Andrews, A. (1968). A square-root formulation of the Kalman covariance equations. AIAA J., 6, 1165–8CrossRefGoogle Scholar
Bennett, A. F., Chua, B. S. and Leslie, L. M. (1996). Generalized inversion of a global numerical weather prediction model. Met. Atmos. Phys., 60, 165–78CrossRefGoogle Scholar
Bishop, C. H., Etherton, B. J. and Majumdar, S. J. (2001). Adaptive sampling with the ensemble transform Kalman filter. 1: Theoretical aspects. Mon. Weather Rev., 129, 420–362.0.CO;2>CrossRefGoogle Scholar
Blanchet, I., Frankignoul, C. and Cane, M. A. (1997). A comparison of adaptive Kalman filters for a tropical Pacific ocean model. Mon. Weather Rev., 125, 40–582.0.CO;2>CrossRefGoogle Scholar
Bouttier, F. (1994). A dynamical estimation of forecast error covariances in an assimilation system. Mon. Weather Rev., 122, 2376–902.0.CO;2>CrossRefGoogle Scholar
Buizza, R., Miller, M. and Palmer, T. N. (1999). Stochastic representation of model uncertainties in the ECMWF ensemble prediction system. Quart. J. Roy. Meteor. Soc., 125, 2887–908CrossRefGoogle Scholar
Burgers, G., Leeuwen, P. J. and Evensen, G. (1998). Analysis scheme in the ensemble Kalman filter. Mon. Weather Rev., 126, 1719–242.0.CO;2>CrossRefGoogle Scholar
Casella, G. and , R. L. Berger (1990). Statistical Inference. Duxbury Press
Cohn, S. E. (1997). An introduction to estimation theory. J. Meteorol. Soc. Jpn., 75(1B), 257–88CrossRefGoogle Scholar
Cohn, S. E. and Parrish, D. F. (1991). The behavior of forecast error covariances for a Kalman filter in two dimensions. Mon. Weather Rev., 119, 1757–852.0.CO;2>CrossRefGoogle Scholar
Courtier, P., Thépaut, J.-N. and Hollingsworth, A. (1994). A strategy for operational implementation of 4D-Var, using an incremental approach. Quart. J. Roy. Meteor. Soc., 120, 1367–87CrossRefGoogle Scholar
Daley, R. (1991). Atmospheric Data Analysis. Cambridge University Press
Daley, R. (1992). Estimating model-error covariances for applications to atmospheric data assimilation. Mon. Weather Rev., 120, 1735–462.0.CO;2>CrossRefGoogle Scholar
Daley, R. (1993). Estimating observation error statistics for atmospheric data assimilation. Ann. Geophys., 11, 634–47Google Scholar
Daley, R. (1997). Atmospheric data assimilation. J. Meteorol. Soc. Jpn., 75(1B), 319–29CrossRefGoogle Scholar
Dee, D. P. (1995). On-line estimation of error covariance parameters for atmospheric data assimilation. Mon. Weather Rev., 123, 1128–452.0.CO;2>CrossRefGoogle Scholar
Dee, D. P. and Todling, R. (2000). Data assimilation in the presence of forecast bias: the GEOS moisture analysis. Mon. Weather Rev., 128, 3268–822.0.CO;2>CrossRefGoogle Scholar
Dowell, D. C., Zhang, F., Wicker, L. J., Snyder, C. and Crook, N. A. (2004). Wind and thermodynamic retrievals in the 17 May 1981 Arcadia, Oklahoma supercell: ensemble Kalman filter experiments. Mon. Weather Rev., 132, 1982–20052.0.CO;2>CrossRefGoogle Scholar
Ehrendorfer, M. (1994a). The Liouville equation and its potential usefulness for the prediction of forecast skill. I: Theory. Mon. Weather Rev., 122, 703–132.0.CO;2>CrossRefGoogle Scholar
Ehrendorfer, M. (1994b). The Liouville equation and its potential usefulness for the prediction of forecast skill. II: Applications. Mon. Weather Rev., 122, 714–282.0.CO;2>CrossRefGoogle Scholar
Etherton, B. J. and Bishop, C. H. (2004). Resilience of hybrid ensemble / 3DVAR analysis schemes to model error and ensemble covariance error. Mon. Weather Rev., 132, 1065–802.0.CO;2>CrossRefGoogle Scholar
Evans, R. E., Harrison, M. S. J. and Graham, R. J. (2000). Joint medium-range ensembles from the Met. Office and ECMWF systems. Mon. Weather Rev., 128, 3104–272.0.CO;2>CrossRefGoogle Scholar
Evensen, G. (1992). Using the extended Kalman filter with a multilayer quasi-geostrophic ocean model. J. Geophys. Res., 97, 17905–24CrossRefGoogle Scholar
Evensen, G. (1994). Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res., 99(C5), 10143–62CrossRefGoogle Scholar
Evensen, G. (2003). The ensemble Kalman filter: theoretical formulation and practical implementation. Ocean Dynam., 53, 343–67CrossRefGoogle Scholar
Evensen, G. and Leeuwen, P. J. (1996). Assimilation of Geosat altimeter data for the Agulhas current using the ensemble Kalman filter with a quasigeostrophic model. Mon. Weather Rev., 124, 85–962.0.CO;2>CrossRefGoogle Scholar
Farrell, B. F. and Ioannou, P. J. (2001). State estimation using a reduced-order Kalman filter. J. Atmos. Sci., 58, 3666–802.0.CO;2>CrossRefGoogle Scholar
Fisher, M. (1998). Development of a Simplified Kalman Filter. ECMWF Research Department Technical Memorandum 260. European Centre for Medium-Range Weather Forecasts. 16 pp. Available from Library, ECMWF, Shinfield Park, Reading, Berkshire, RG2 9AX, England
Gardiner, C. W. (1985). Handbook of Stochastic Methods, 2nd edn. SpringerGoogle Scholar
Gaspari, G. and Cohn, S. E. (1999). Construction of correlation functions in two and three dimensions. Quart. J. Roy. Meteor. Soc., 125, 723–57CrossRefGoogle Scholar
Gauthier, P., Courtier, P. and Moll, P. (1993). Assimilation of simulated lidar data with a Kalman filter. Mon. Weather Rev., 121, 1803–202.0.CO;2>CrossRefGoogle Scholar
Gelb, A. (ed.) (1974). Applied Optimal Estimation. MIT PressGoogle Scholar
Ghil, M. (1989). Meteorological data assimilation for oceanography. Part 1: Description and theoretical framework. Dyn. Atmos. Oceans, 13, 171–218CrossRefGoogle Scholar
Ghil, M. and Malanotte-Rizzoli, P. (1991). Data assimilation in meteorology and oceanography. Adv. Geophys., 33, 141–266CrossRefGoogle Scholar
Gordon, N. J., Salmond, D. J. and Smith, A. F. M. (1993). Novel approach to nonlinear/non-Gaussian Bayesian state estimation. IEEE Proceedings – F, 140, 107–13Google Scholar
Hamill, T. M. and Snyder, C. (2000). A hybrid ensemble Kalman filter / 3d-variational analysis scheme. Mon. Weather Rev., 128, 2905–192.0.CO;2>CrossRefGoogle Scholar
Hamill, T. M. and Snyder, C. (2002). Using improved background-error covariances from an ensemble Kalman filter for adaptive observations. Mon. Weather Rev., 130, 1552–722.0.CO;2>CrossRefGoogle Scholar
Hamill, T. M. and Whitaker, J. S. (2005). Accounting for the error due to unresolved scales in ensemble data assimilation: a comparison of different approaches. Mon. Weather Rev., 133, 3132–47CrossRefGoogle Scholar
Hamill, T. M., Whitaker, J. S. and Snyder, C. (2001). Distance-dependent filtering of background-error covariance estimates in an ensemble Kalman filter. Mon. Weather Rev., 129, 2776–902.0.CO;2>CrossRefGoogle Scholar
Hamill, T. M., Snyder, C. and Whitaker, J. S. (2003). Ensemble forecasts and the properties of flow-dependent analysis-error covariance singular vectors. Mon. Weather Rev., 131, 1741–58CrossRefGoogle Scholar
Hansen, J. A. (2002). Accounting for model error in ensemble-based state estimation and forecasting. Mon. Weather Rev., 130, 2373–912.0.CO;2>CrossRefGoogle Scholar
Harrison, M. S. J., Palmer, T. N., Richardson, D. S. and Buizza, R. (1999). Analysis and model dependencies in medium-range ensembles: two transplant case studies. Quart. J. Roy. Meteor. Soc., 125, 2487–515CrossRefGoogle Scholar
Hastie, T. R., Tibshirani and Friedman, J. (2001). The Elements of Statistical Learning. Springer
Heemink, A. W., Verlaan, M. and Segers, A. J. (2001). Variance-reduced ensemble Kalman filtering. Mon. Weather Rev., 129, 1718–282.0.CO;2>CrossRefGoogle Scholar
Hou, D., Kalnay, E. and Droegemeier, K. K. (2001). Objective verification of the SAMEX-98 ensemble forecasts. Mon. Weather Rev., 129, 73–912.0.CO;2>CrossRefGoogle Scholar
Houtekamer, P. L. and Mitchell, H. L. (1998). Data assimilation using an ensemble Kalman filter technique. Mon. Weather Rev., 126, 796–8112.0.CO;2>CrossRefGoogle Scholar
Houtekamer, P. L. and Mitchell, H. L. (1999). Reply to comment on “Data assimilation using an ensemble Kalman filter technique.” Mon. Weather Rev., 127, 1378–92.0.CO;2>CrossRefGoogle Scholar
Houtekamer, P. L. and Mitchell, H. L. (2001). A sequential ensemble Kalman filter for atmospheric data assimilation. Mon. Weather Rev., 129, 123–372.0.CO;2>CrossRefGoogle Scholar
Houtekamer, P. L., Lefaivre, L. and Derome, J. (1996a). The RPN ensemble prediction system. In Proceedings ECMWF Seminar on Predictability, Vol. II, Reading, pp. 121–146. [Available from ECMWF, Shinfield Park, Reading, Berkshire RG2 9AX, United Kingdom]Google Scholar
Houtekamer, P. L. J., Derome, H. Ritchie and Mitchell, H. L. (1996b). A system simulation approach to ensemble prediction. Mon. Weather Rev., 124, 1225–422.0.CO;2>CrossRefGoogle Scholar
Houtekamer, P. L., Mitchell, H. L., Pellerin, G., et al. (2005). Atmospheric data assimilation with the ensemble Kalman filter: results with real observations. Mon. Weather Rev., 133, 604–20CrossRefGoogle Scholar
Hunt, B. R., Kalnay, E., Kostelich, E. J., et al. (2004). Four-dimensional ensemble Kalman filtering. Tellus, 56A, 273–7CrossRefGoogle Scholar
Jazwinski, A. H. (1970). Stochastic Processes and Filtering Theory. Academic PressGoogle Scholar
Kalman, R. E. (1960). A new approach to linear filtering and prediction problems. T. ASME – J. Basic Eng., 82D, 35–45CrossRefGoogle Scholar
Kalman, R. E. and Bucy, R. S. (1961). New results in linear filtering and prediction theory. T. ASME – J. Basic Eng., 83D, 95–108CrossRefGoogle Scholar
Kaminski, P. G., Bryson, A. E. Jr and Schmidt, S. F. (1971). Discrete square root filtering: a survey of current techniques. IEEE T. Automat. Control, AC-16, 727–36CrossRefGoogle Scholar
Keppenne, C. L. (2000). Data assimilation into a primitive equation model with a parallel ensemble Kalman filter. Mon. Weather Rev., 128, 1971–812.0.CO;2>CrossRefGoogle Scholar
Keppenne, C. L. and Rienecker, M. M. (2002). Initial testing of a massively parallel ensemble Kalman filter with the Poseidon isopycnal ocean general circulation model. Mon. Weather Rev., 130, 2951–652.0.CO;2>CrossRefGoogle Scholar
Lacarra, J. F. and Talagrand, O. (1988). Short-range evolution of small perturbations in a barotropic model. Tellus, 40A, 81–95CrossRefGoogle Scholar
Lawson, G. and Hansen, J. A. (2003). Implications of stochastic and deterministic filters as ensemble-based data assimilation methods in varying regimes of error growth. Mon. Weather Rev., 132, 1966–812.0.CO;2>CrossRefGoogle Scholar
Dimet, F.-X. and Talagrand, O. (1986). Variational algorithms for analysis and assimilation of meteorological observations: theoretical aspects. Tellus, 38A, 97–110CrossRefGoogle Scholar
Leith, C. E. (1983). Predictability in theory and practice. In Large-scale Dynamical Processes in the Atmosphere, ed.Hoskins, B. J. and Pearce, R. P., chapter 13. Academic PressGoogle Scholar
Lermusiaux, P. F. J. (2002). On the mapping of multivariate geophysical fields: sensitivities to size, scales, and dynamics. J. Atmos. Ocean. Tech., 19, 1602–372.0.CO;2>CrossRefGoogle Scholar
Lermusiaux, P. F. J. and Robinson, A. R. (1999). Data assimilation via error subspace statistical estimation. 1: Theory and schemes. Mon. Weather Rev., 127, 1385–4072.0.CO;2>CrossRefGoogle Scholar
Li, Z. and Navon, I. M. (2001). Optimality of variational data assimilation and its relationship with the Kalman filter and smoother. Quart. J. Roy. Meteor. Soc., 127, 661–83CrossRefGoogle Scholar
Liu, Z.-Q. and Rabier, F. (2003). The potential of high-density observations for numerical weather prediction: a study with simulated observations. Quart. J. Roy. Meteor. Soc., 129, 3013–35CrossRefGoogle Scholar
Lorenc, A. C. (1981). A global three-dimensional multivariate statistical interpolation scheme. Mon. Weather Rev., 109, 701–212.0.CO;2>CrossRefGoogle Scholar
Lorenc, A. C. (1986). Analysis methods for numerical weather prediction. Quart. J. Roy. Meteor. Soc., 112, 1177–94CrossRefGoogle Scholar
Lorenc, A. C. (2003). The potential of the ensemble Kalman filter for NWP – a comparison with 4D-Var. Quart. J. Roy. Meteor. Soc., 129, 3183–203CrossRefGoogle Scholar
Maybeck, P. S. (1979). Stochastic Models, Estimation, and Control. Vol. 1 Academic PressGoogle Scholar
Mitchell, H. L. and Houtekamer, P. L. (2000). An adaptive ensemble Kalman filter. Mon. Weather Rev., 128, 416–332.0.CO;2>CrossRefGoogle Scholar
Mitchell, H. L., Houtekamer, P. L. and Pellerin, G. (2002). Ensemble size, balance, and model-error representation in an ensemble Kalman filter. Mon. Weather Rev., 130, 2791–8082.0.CO;2>CrossRefGoogle Scholar
Molteni, F., Buizza, R., Palmer, T. N. and Petroliagis, T. (1996). The ECMWF ensemble prediction system: methodology and validation. Quart. J. Roy. Meteor. Soc., 122, 73–119CrossRefGoogle Scholar
Palmer, T. N. (2001). A nonlinear dynamical perspective on model error: a proposal for non-local stochastic-dynamic parametrization in weather and climate prediction models. Quart. J. Roy. Meteor. Soc., 127, 279–304Google Scholar
Parrish, D. F. and Derber, J. C. (1992). The National Meteorological Center's Spectral Statistical Interpolation Analysis System. Mon. Weather Rev., 120, 1747–632.0.CO;2>CrossRefGoogle Scholar
Penland, C. (2003). A stochastic approach to nonlinear dynamics: a review. (Electronic supplement to ‘Noise out of chaos and why it won't go away’), Bull. Am. Meteorol. Soc., 84, 921–5CrossRefGoogle Scholar
Pham, D. T. (2001). Stochastic methods for sequential data assimilation in strongly nonlinear systems. Mon. Weather Rev., 129, 1194–2072.0.CO;2>CrossRefGoogle Scholar
Potter, J. (1964). W matrix Augmentation. M.I.T. Instrumentation Laboratory Memo SGA 5–64, Massachusetts Institute of Technology, Cambridge, MassachusettsGoogle Scholar
Rabier, F., Thepaut, J.-N. and Courtier, P. (1998). Extended assimilation and forecast experiments with a four-dimensional variational assimilation system. Quart. J. Roy. Meteor. Soc., 124, 1861–87CrossRefGoogle Scholar
Rabier, F., Järvinen, H., Klinker, E., Mahfouf, J.-F. and Simmons, A. (2000). The ECMWF operational implementation of four-dimensional variational assimilation. I: Experimental results with simplified physics. Quart. J. Roy. Meteor. Soc., 126, 1143–70CrossRefGoogle Scholar
Reichle, R. H. and Koster, R. D. (2004). Assessing the impact of horizontal error correlations in background fields on soil moisture estimation. J. Hydrometeorol., 4, 1229–422.0.CO;2>CrossRefGoogle Scholar
Reichle, R. H., McLaughlin, D. B. and Entekhabi, D. (2002). Hydrologic data assimilation with the ensemble Kalman filter. Mon. Weather Rev., 130, 103–142.0.CO;2>CrossRefGoogle Scholar
Reichle, R. H., Walker, J. P., Koster, R. D. and Houser, P. R. (2003). Extended versus ensemble Kalman filtering for land data assimilation. J. Hydrometeorol., 3, 728–402.0.CO;2>CrossRefGoogle Scholar
Richardson, D. S. (2000). Ensembles using multiple models and analyses. Quart. J. Roy. Meteor. Soc., 127, 1847–64CrossRefGoogle Scholar
Sardeshmukh, P. D., Penland, C. and Newman, M. (2001). Rossby waves in a stochastically fluctuating medium. In Stochastic Climate Models, ed. Imkeller, P. and Storch, J.-S., pp. 369–84, Progress in Probability 49. Birkhaueser, BaselGoogle Scholar
Snyder, C. and Zhang, F. (2003). Assimilation of simulated Doppler radar observations with an ensemble Kalman filter. Mon. Weather Rev., 131, 1663–77CrossRefGoogle Scholar
Shutts, G. (2004). A Stochastic Kinetic-energy Backscatter Algorithm for Use in Ensemble Prediction Systems. ECMWF Technical Memorandum 449. Available from ECMWF, Shinfield Park, Reading, Berkshire RG2 9AX, United KingdomGoogle Scholar
Talagrand, O. (1997). Assimilation of observations, an introduction. J. Meteorol. Soc. Jpn., 75(1B), 191–209CrossRefGoogle Scholar
Tippett, M. K., Anderson, J. L., Bishop, C. H., Hamill, T. M. and Whitaker, J. S. (2003). Ensemble square root filters. Mon. Weather Rev., 131, 1485–902.0.CO;2>CrossRefGoogle Scholar
Toth, Z. and Kalnay, E. (1993). Ensemble forecasting at NMC: the generation of perturbations. Bull. Am. Meteorol. Soc., 74, 2317–302.0.CO;2>CrossRefGoogle Scholar
Toth, Z. and Kalnay, E. (1997). Ensemble forecasting at NCEP and the breeding method. Mon. Weather Rev., 12, 3297–3192.0.CO;2>CrossRefGoogle Scholar
Leeuwen, P. J. (1999). Comment on “Data assimilation using an ensemble Kalman filter technique”. Mon. Weather Rev., 127, 1374–72.0.CO;2>CrossRefGoogle Scholar
Wang, X. and Bishop, C. H. (2003). A comparison of breeding and ensemble transform Kalman filter ensemble forecast schemes. J. Atmos. Sci., 60, 1140–582.0.CO;2>CrossRefGoogle Scholar
Whitaker, J. S. and Hamill, T. M. (2002). Ensemble data assimilation without perturbed observations. Mon. Weather Rev., 130, 1913–242.0.CO;2>CrossRefGoogle Scholar
Whitaker, J. S., Compo, G. P., Wei, X. and Hamill, T. M. (2004). Reanalysis without radiosondes using ensemble data assimilation. Mon. Weather Rev., 132, 1190–2002.0.CO;2>CrossRefGoogle Scholar
Zhang, F., Snyder, C. and Sun, J. (2004). Impacts of initial estimate and observation availability on convective-scale data assimilation with an ensemble Kalman filter. Mon. Weather Rev., 132, 1238–532.0.CO;2>CrossRefGoogle Scholar
Ziehmann, C. (2000). Comparison of a single-model EPS with a multi-model ensemble consisting of a few operational models. Tellus, 52a, 280–99CrossRefGoogle Scholar
Zupanski, D. (1997). A general weak constraint applicable to operational 4DVAR data assimilation systems. Mon. Weather Rev., 125, 2274–922.0.CO;2>CrossRefGoogle Scholar

Save book to Kindle

To save this book to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×