Skip to main content
SpringerLink
Log in

Acceptance criteria for urban dispersion model evaluation

  • Original Paper
  • Published:
Meteorology and Atmospheric Physics Aims and scope Submit manuscript

Abstract

The authors suggested acceptance criteria for rural dispersion models’ performance measures in this journal in 2004. The current paper suggests modified values of acceptance criteria for urban applications and tests them with tracer data from four urban field experiments. For the arc-maximum concentrations, the fractional bias should have a magnitude <0.67 (i.e., the relative mean bias is less than a factor of 2); the normalized mean-square error should be <6 (i.e., the random scatter is less than about 2.4 times the mean); and the fraction of predictions that are within a factor of two of the observations (FAC2) should be >0.3. For all data paired in space, for which a threshold concentration must always be defined, the normalized absolute difference should be <0.50, when the threshold is three times the instrument’s limit of quantification (LOQ). An overall criterion is then applied that the total set of acceptance criteria should be satisfied in at least half of the field experiments. These acceptance criteria are applied to evaluations of the US Department of Defense’s Joint Effects Model (JEM) with tracer data from US urban field experiments in Salt Lake City (U2000), Oklahoma City (JU2003), and Manhattan (MSG05 and MID05). JEM includes the SCIPUFF dispersion model with the urban canopy option and the urban dispersion model (UDM) option. In each set of evaluations, three or four likely options are tested for meteorological inputs (e.g., a local building top wind speed, the closest National Weather Service airport observations, or outputs from numerical weather prediction models). It is found that, due to large natural variability in the urban data, there is not a large difference between the performance measures for the two model options and the three or four meteorological input options. The more detailed UDM and the state-of-the-art numerical weather models do provide a slight improvement over the other options. The proposed urban dispersion model acceptance criteria are satisfied at over half of the field experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  • Allwine KJ (2007) Field studies for validation of urban dispersion models: current status and research needs. Report No. PNNL-SA-58392 Presented at AGU 2007 Fall Meeting, San Francisco, CA

  • Allwine KJ, Flaherty JE (2006a) Joint urban 2003: study overview and instrument locations, PNNL-15967. Pacific Northwest National Laboratory, Richland, WA

  • Allwine KJ, Flaherty JE (2006b) Urban dispersion program MSG05 field study: summary of tracer and meteorological measurements, PNNL-15969. Pacific Northwest National Laboratory, Richland, WA (The complete data set is available on a separate CD)

  • Allwine KJ, Flaherty JE (2007) Urban dispersion program overview and MID05 field study summary, PNNL-16696. Pacific Northwest National Laboratory, Richland, WA (The complete meteorological data set is available on a separate CD)

  • Allwine KJ, Shinn JH, Streit GE, Clawson KL, Brown M (2002) Overview of urban 2000. A multiscale field study of dispersion through an urban environment. Bull Am Meteorol Soc 83:521–536

    Article  Google Scholar 

  • Allwine KJ, Leach M, Stockham L, Shinn J, Hosker R, Bowers J, Pace J (2004) Overview of joint urban 2003—an atmospheric dispersion study in Oklahoma City, symposium on planning, nowcasting and forecasting in the urban zone. American Meteorological Society, January 11–15, 2004, Seattle, Washington. Available at http://www.ametsoc.org

  • Chang JC, Hanna SR (2004) Air quality model performance evaluation. Meteorol Atmos Phys 87:167–196

    Article  Google Scholar 

  • Chang JC, Hanna SR (2010) Independent results validation of JEM for urban applications. Final report prepared for JEM IV&V program office, Available from Chang Consultants, 8927 Colesbury Place, Fairfax, VA 22031

  • Chang JC, Hanna SR, Boybeyi Z, Franzese P (2005) Use of Salt Lake City urban 2000 data to evaluate the Urban-HPAC model. J Appl Meteorol 44(5):485–501

    Article  Google Scholar 

  • Chang JC, Boybeyi Z, Tang H, Cervone G, Bakosi J, Hanna S, Hendrick E, Santos L, Bowers J (2007) Independent verification and validation of joint effects model. Prepared for Naval Surface Warfare Center, Dahlgren Division. Available from Chang Consultants, 8927 Colesbury Place, Fairfax, VA 22031

  • Clawson KL, Carter RG, Lacroix D, Biltoft C, Hukari N, Johnson R, Rich J, Beard S, Strong T (2005) Joint urban 2003 (JU03) SF6 atmospheric tracer field tests. NOAA Tech Memo OAR ARL-254, Air Resources Lab, Silver Spring, MD

  • DTRA (2008) HPAC Version 5.0 SP1, DTRA, 8725 John J. Kingman Road, MSC 6201, Ft. Belvoir, VA 22060-6201

  • Girardi F, Graziani G, van Velzen D, Galmarini S, Mosca S, Bianconi R, Bellasio R, Klug W, Fraser G (1998) The European Tracer Experiment, EUR 18143 EN, ISBN 92-828-5007-2, Joint Research Centre, European Commission

  • Hall DJ, Spanton AM, Hargrave M, Walker S, Griffiths IH, Brooks DR (2004) The Urban Dispersion Model (UDM): Version 2.9 Technical Documentation. Defence Science and Technology Laboratory, Physical Sciences Department, Porton Down, Salisbury, UK

  • Hanna SR (1989) Confidence limits for air quality model evaluations as estimated by bootstrap and jackknife resampling methods. Atmos Environ 23:1385–1398

    Article  Google Scholar 

  • Hanna SR, Baja E (2009) A simple urban dispersion model tested with tracer data from Oklahoma City and New York City. Atmos Environ 43:778–786

    Article  Google Scholar 

  • Hanna SR, Britter RE, Franzese P (2003) A baseline urban dispersion model evaluated with Salt Lake City and Los Angeles Tracer data. Atmos Environ 37:5069–5082

    Article  Google Scholar 

  • Hanna S, Fabian P, Chang J, Venkatram A, Britter R, Neophytou M, Brook D (2004) Use of Urban 2000 field data to determine whether there are significant differences between the performance measures of several urban dispersion models. Paper 7.3, AMS 5th Conference on Urban Environment. Available at http://www.ametsoc.org

  • Hanna SR, Brown MJ, Camelli FE, Chan S, Coirier WJ, Hansen OR, Huber AH, Kim S, Reynolds RM (2006) Detailed simulations of atmospheric flow and dispersion in urban downtown areas by Computational Fluid Dynamics (CFD) models—An application of five CFD models to Manhattan. Bull Am Meteorol Soc 87:1713–1726

    Article  Google Scholar 

  • Hanna SR, Chang JC, White JM, Bowers JF (2007a) Evaluation of the HPAC dispersion model with the JU2003 tracer observations. Paper 10.1, AMS 7th Conference on the Urban Environment. Available at http://www.ametsoc.org

  • Hanna SR, White JM, Zhou Y (2007b) Observed winds, turbulence, and dispersion in built-up downtown areas in Oklahoma City and Manhattan. Bound Lay Meteorol 125:441–468

    Article  Google Scholar 

  • Hanna SR, White JM, Troiler J, Vernot R, Brown M, Kaplan H, Alexander Y, Moussafir J, Wang Y, Williamson C, Hannan J, Hendrick E (2011) Comparisons of JU2003 observations with four diagnostic urban wind flow and Lagrangian particle dispersion models. Atmos Environ 45:4073–4081

    Google Scholar 

  • Sykes RI, Parker S, Henn D, Chowdhury B (2007) SCIPUFF Version 2.3 Technical Documentation. L-3 Titan Corp, POB 2229, Princeton 08543-2229, NJ 08543-2229

  • Warner S, Platt N, Heagy JF (2004) User-oriented two-dimensional measure of effectiveness for the evaluation of transport and dispersion models. J Appl Meteorol 43:58–73

    Article  Google Scholar 

  • Watson TB, Heiser J, Kalb P, Dietz RN, Wilke R, Weiser R, Vignato G (2006) The New York City urban dispersion program March 2005 field study: tracer methods and results. BNL-75592-2006, Brookhaven National Laboratory, POB 5000, Upton, NY 11973–5000

  • White JM, Bowers JF, Hanna SR, Lundquist JK (2009) Importance of using observations of mixing depths in order to avoid large prediction errors by a transport and dispersion model. J Atmos Oceanic Tech 26:22–32

    Article  Google Scholar 

Download references

Acknowledgments

This research has been sponsored by the Defense Threat Reduction Agency (DTRA), the JEM Program Office, and the National Science Foundation (under award 0750878). The DTRA program manager is Rick Fry and the JEM program manager is Thomas Smith. The authors appreciate the discussions of the PFT data with Jerry Allwine and Julia Flaherty of the Pacific Northwest National Laboratory, and guidance concerning the SF6 data provided by Kirk Clawson of NOAA’s Field Research Division.

Author information

Authors and Affiliations

  1. Hanna Consultants, Kennebunkport, ME, 04046, USA

    Steven Hanna

  2. Homeland Security Studies and Analysis Institute, Arlington, VA, 22206, USA

    Joseph Chang

Authors
  1. Steven Hanna
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joseph Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Steven Hanna.

Additional information

Responsible editor: S. Trini Castelli.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hanna, S., Chang, J. Acceptance criteria for urban dispersion model evaluation. Meteorol Atmos Phys 116, 133–146 (2012). https://doi.org/10.1007/s00703-011-0177-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00703-011-0177-1

Keywords

Search

Navigation