Elsevier

Journal of Computational Physics

Volume 256, 1 January 2014, Pages 118-134
Journal of Computational Physics

A conservative multi-tracer transport scheme for spectral-element spherical grids

https://doi.org/10.1016/j.jcp.2013.08.050Get rights and content

Abstract

Atmospheric models used for practical climate simulation must be capable handling the transport of hundreds of tracers. For computational efficiency conservative multi-tracer semi-Lagrangian type transport schemes are appropriate. Global models based on high-order Galerkin approach employ highly non-uniform spectral-element grids, and semi-Lagrangian transport is a challenge on those grids. A conservative semi-Lagrangian scheme (SPELT — SPectral-Element Lagrangian Transport) employing a multi-moment compact reconstruction procedure is developed for non-uniform quadrilateral grids. The scheme is based on a characteristic semi-Lagrangian method that avoids complex and expensive upstream area computations. The SPELT scheme has been implemented in the High-Order Method Modeling Environment (HOMME), which is based on a cubed-sphere grid with spectral-element spatial discretization. Additionally, we show the (strong) scalability and multi-tracer efficiency using several benchmark tests. The SPELT solution can be made monotonic (positivity preserving) by combining the flux-corrected transport algorithm, which is demonstrated on a uniform resolution grid. In particular, SPELT can be efficiently used for non-uniform grids and provides accurate and stable results for high-resolution meshes.

Introduction

In the past two decades, global spectral methods have dominated in atmospheric modeling. However, the main disadvantage of these methods is that they require expensive non-local communication operations, which prevents them from using the full potential of available massive parallel petascale computers. In recent years, high-order accuracy is being achieved by using finite-element type models, including the spectral-element (SE) or continuous Galerkin and the discontinuous Galerkin (DG) methods (see [31], [32], [14], [26]). These element-based and high-order Galerkin methods have the nice feature that the data for parallel communication is reduced to a minimum due to localized computations. Additionally, element-based methods allow a more flexible choice of domain and local mesh refinement as, for example, spectral methods.

The spectral-element dynamical core [31] based on the High-Order Method Modeling Environment (HOMME) [8] framework is the default dynamical core for the Community Atmosphere Model (CAM, version 5.2 and higher) — CAM-SE. The grid system in HOMME is based on the cubed-sphere geometry resulting from a gnomonic equiangular projection of the sphere. It has been shown in [7] that this approach is highly scalable, up to 170 000 cores on the Oak Ridge LCF Cray XT5 JaguarPF system. However, with a growing number of tracers in todayʼs atmospheric modeling applications transport becomes a dominating factor of the total computational costs; for example, more than O(100) tracers are used in the current chemistry version of CAM, and this number is likely to increase in future applications. A native SE transport scheme combined with non-oscillatory features is available in HOMME [15]. Unfortunately, it is computationally prohibitive for a large number of tracers due to three communications per time step and a relatively small time step of the explicit Runge–Kutta based approach [7]. In addition, SE advection is coupled with a hyper-diffusion operator that also increases the computational cost.

Thus the multi-tracer transport is a major computational bottleneck of the HOMME dynamical core for climate simulations. Furthermore, HOMME supports unstructured grids. Therefore it is desirable to have multi-tracer efficient schemes capable of handling such grid system. Conservative semi-Lagrangian methods with moderate Courant numbers can be employed for efficiently handling the multi-tracer problems [11]. Using a moderate Courant number for the semi-Lagrangian advection does not adversely affect the parallel efficiency of the Eulerian host model. But this Courant number can still be chosen several times larger than that for a high-order advection scheme. Recently, a third-order Conservative Semi-LAgrangian Multi-tracer transport scheme (CSLAM [20]) has been integrated in HOMME [11]. This is achieved by overlaying the HOMME SE grids with uniform finite-volume cells. However, the CSLAM scheme is designed for uniform grids and requires a 5×5 reconstruction stencil (as implemented in HOMME), for which extra interpolations are required at the cube corner (ghost cells). In addition, CSLAM relies on computing the upstream (Lagrangian) areas through an intricate process. For that an expensive search algorithm is needed to locate and compute the overlap areas [20]. This implies that for an arbitrary unstructured cubed-sphere grid, implementation of a finite-volume semi-Lagrangian scheme such as CSLAM may be prohibitively complicated.

In order to preserve the parallel efficiency of the host model HOMME, the multi-tracer transport algorithm should depend only on a compact computational stencil and should also be suitable for unstructured grid systems. The multi-moment based finite-volume approach introduced in [34], [3] employs a single-cell reconstruction procedure and is well-suited for our application. The upstream overlap-area computation required for the conservative semi-Lagrangian scheme can be avoided by using the characteristic semi-Lagrangian approach, where the fluxes are integrated along the characteristics as shown in [4]. We further extend this approach for arbitrary quadrilateral grids on the cubed sphere by designing a conservative semi-Lagrangian scheme on the spectral-element grid, hereinafter referred to as SPELT — SPectral-Element Lagrangian Transport. For the SPELT scheme the upstream search is reduced to a point search on the cubed sphere and the reconstruction is local, which makes SPELT extremely attractive for arbitrary grids. SPELT is a conservative flux-based scheme (with a monotonic option) and is accurate on arbitrary quadrilateral grids. Moreover, a new algorithm in HOMME shows that SPELT is highly scalable and multi-tracer efficient. The performance of SPELT in climate simulations is beyond the scope of this manuscript and will be discussed in a future work [13].

The remainder of the paper is organized as follows: in Section 2 we briefly describe semi-Lagrangian schemes for the transport equation. Section 3 describes our SPELT algorithm, in particular the scheme for arbitrary quadrilateral grids. Section 4 shows the multi-tracer efficiency, scalability and some high-resolution experiments comparing the relative merits of both schemes in HOMME.

Section snippets

Tracer transport schemes

The flux-form transport equation for a scalar ρ(x,y,t) in 2D Cartesian (x,y)-plane, without a source or sink, can be written as follows:ρt+(uρ)x+(vρ)yρt+F=0,t(0,T], where (u,v) is the wind velocity vector, F=(ρu,ρv) is the flux, and the initial condition is prescribed as ρ(x,y,t=0)=ρ0(x,y). Equivalently, (1) can be cast in the following Lagrangian form:DDtA(t)ρdA=0, where D/Dt=t+ux+vy is the Lagrangian (material) derivative, A is the area (volume) in which the fluid density ρ

SPELT — SPectral-Element Lagrangian Transport

This section focuses on the design of our SPELT algorithm on non-uniform quadrilateral grids.

Implementation of finite-volume schemes in HOMME

The domain decomposition (horizontal) in HOMME follows along the cubed-sphere grid. In particular, the parallel strategy relies on the spectral elements, e.g., all cube faces are divided in equi-distant meshes. A standard configuration is based on 4×4 or 5×5 Gaussian–Legendre–Lobatto (GLL) points. That means the number of unknowns for each core is at least 16 or 25 if we run the model with a minimum of one spectral element per core, e.g., Fig. 4(a) shows one spectral element with 5×5 GLL

Numerical experiments

We first discuss the numerical results with the SPELT scheme on a non-uniform cubed-sphere grid and compare them against a uniform resolution equiangular cubed-sphere grid. Monotone results with FCT will be discussed only for the uniform resolution grid. Parallel performance of SPELT is tested in the HOMME framework.

Conclusions

We have presented a new conservative semi-Lagrangian scheme (SPELT) on arbitrary quadrilateral spherical grids. SPELT is a flux-based characteristic semi-Lagrangian method where the fluxes are integrated along the trajectory. This avoids computationally expensive and cumbersome upstream overlap-area searching, as required in regular conservative semi-Lagrangian methods. Therefore, the implementation on complex grids and for super computers is relatively easy. The reconstruction procedure for

Acknowledgements

The authors gratefully acknowledge the careful internal review by Dr. Robert Klöfkorn (National Center for Atmospheric Research, USA). The authors were partially supported by the DOE BER Program under award DE-SC0006959. Parts of this work were done when both authors participated the Multiscale Numerics for the Atmosphere and Ocean program at the Isaac Newton Institute (INI) for Mathematical Sciences, Cambridge, UK. We thank Dr. Hilary Weller (University of Reading, UK) and one anonymous

References (36)

  • M.R. Norman et al.

    A low communication and large time step explicit finite-volume solver for non-hydrostatic atmospheric dynamics

    J. Comput. Phys.

    (2011)
  • C. Ronchi et al.

    The “cubed sphere”: A new method for the solution of partial differential equations in spherical geometry

    J. Comput. Phys.

    (1996)
  • M. Taylor et al.

    The spectral element method for the shallow water equations on the sphere

    J. Comput. Phys.

    (1997)
  • D. Williamson et al.

    A standard test set for numerical approximations to the shallow water equations in spherical geometry

    J. Comput. Phys.

    (1992)
  • F. Xiao et al.

    Unified formulation for compressible and incompressible flows by using multi-integrated moments II: Multi-dimensional version for compressible and incompressible flows

    J. Comput. Phys.

    (2006)
  • S.T. Zalesak

    Fully multidimensional flux-corrected transport algorithms for fluids

    J. Comput. Phys.

    (1979)
  • R. Bermejo et al.

    The conversion of the semi-Lagrangian advection schemes to quasi-monotone schemes

    Mon. Weather Rev.

    (1992)
  • C.G. Chen et al.

    A multi-moment transport model on cubed-sphere grid

    Int. J. Numer. Methods Fluids

    (2010)
  • Cited by (15)

    • A high order characteristic discontinuous Galerkin scheme for advection on unstructured meshes

      2016, Journal of Computational Physics
      Citation Excerpt :

      This scheme may in theory be extended to arbitrarily high order and is currently in use within the High-Order Methods Modelling Environment (HOMME) atmospheric model [6]. Another scheme implemented within the HOMME model, Spectral Element Lagrangian Transport (SPELT) [7] uses a quadratic polynomial reconstruction of the tracer within each element to integrate the flux across the Lagrangian pre-image of the element at intermediate time levels to third order spatial accuracy. Like the CSLAM algorithm, this scheme uses line integrals instead of swept regions for greater computational efficiency, however the nature of the tracer field reconstructions limits the scheme to quadrilateral grids.

    • 3D conservative cascade semi-Lagrangian transport scheme using reduced latitude-longitude grid (CCS-RG)

      2016, Journal of Computational Physics
      Citation Excerpt :

      In recent comparison [5], FVSL methods (including 2D CCS-RG [32]) proved to be competitive with all other types of schemes in terms of accuracy and other requirements for the transport schemes. These results, along with the ability of using large time steps, the outstanding multi-tracer efficiency [33] of FVSL schemes and the recent progress in the development of the massively-parallel SL algorithms [33,30,34] rise our believe that FVSL would be reasonable method for treating the transport in the future atmospheric models. One more question about numerical methods for the future high-resolution global atmospheric models is their ability to use quasi-uniform grids on the sphere.

    • High-order central ENO finite-volume scheme for hyperbolic conservation laws on three-dimensional cubed-sphere grids

      2015, Journal of Computational Physics
      Citation Excerpt :

      In particular, the angularly equidistant mapping described in [2] is used to generate the initial six blocks of the grids. Significant advances have been made for second-order and high-order discretizations on cubed-sphere grids in recent years [6,7,53,8–16]. Most of these works target 2D cubed-sphere grids with curvilinear coordinates, and only few address high-order discretizations on 3D cubed-sphere grids.

    View all citing articles on Scopus
    1

    The National Center for Atmospheric Research is sponsored by the National Science Foundation.

    View full text