Efficient 3D DNS of gas–solid flows on Fermi GPGPU

Abstract

Three-dimensional (3D) gas–solid Direct Numerical Simulation (DNS) requires huge computational resources which imposes a great challenge to both current hardware and software conditions. In this article, an efficient implementation of 3D gas–solid DNS with Lattice Boltzmann Method and Discrete Element Method is developed on a Fermi GPGPU. An Immersed Moving Boundary approach is utilized to impose the no-slip condition at particle–fluid interfaces. Optimization strategies such as changing the sequence of collision and propagation in grid evolution and making multiple kernel executing concurrently are discussed in detail. This algorithm is demonstrated to be competitive both in terms of accuracy and performance. Approximately 131 Millions of Lattices Update Per Second have been achieved, indicating that this GPGPU implementation is very suitable for 3D gas–solid DNS.

Introduction

Gas–solid flows, which are frequently encountered in process engineering, have received extensive attention during the last century. However, theoretical and experimental work still face fundamental challenges due to the inherent non-linear non-equilibrium multi-scale characteristics [1] in such complex multi-phase systems. At the same time, computational modeling has emerged as the third pillar in scientific studies for its controllable, non-intrusive and easy-to-implement features. Thus, computational modeling has been employed in a wide range of fields, including gas–solid flows [2], [3].

In gas–solid simulations, Direct Numerical Simulation (DNS) plays an indispensable role for its ability to investigate micro-scale details and provide constitutive laws for higher level methods such as Two-Fluid Modeling (TFM) [4] and Discrete Particle Modeling (DPM) [5]. These capabilities make DNS a hot spot in gas–solid flows simulation community in recent years [6]. Despite these advantages, DNS of a typical industry-scale even laboratory-scale system seems out of reach for conventional hard- and software. This is essentially caused by the requirement to resolve the local no-slip condition at gas–particle interfaces which requires substantially more resolution and thus computational power than for single-phase flows. Moreover, DNS of gas–solid flows is intrinsically computationally demanding as gas phase needs to be resolved below particle scale at least one order of magnitude to obtain accurate surface integrals. This means for a three-dimensional (3D) case, the number of gas grid nodes exceeds the number of solid particles at least by three orders of magnitude. For a system containing thousands to millions of solid particles, the requirement of gas resolution inevitably implies a huge gap to conventional computer capability.

For the aspect of posing no-slip condition at gas–solid interface in DNS, several solutions have been proposed in recent years in both grid-based and particle-based formulations. In the grid-based approach, the simulation domain is discretized into structured or unstructured grids where special skills such as applying semi-analytical expression based on a Stokes formulation to correct gas velocity adjacent to solid boundary [7], [8] or adaptive grids to make grids adhere to solid boundary [9] are needed to treat gas–solid interface. Although grid-based techniques have a solid theoretical foundation and relatively higher accuracy, the implicit solution of the Poisson equation and the dynamic adaptation of body-fitted grids consume a substantial part of the overall computational effort and may cause poor scalability. In particle-based methods such as Smoothed Particle Hydrodynamics [10] and Macro-scale Pseudo-particle Method [11], the simulation domain is discretized by Lagrange particles both for gas and solid phases (for solid phase these particles are termed “frozen” particles) and a mirror velocity is associated to both gas and “frozen” particles to prescribe consistent velocities at the respective interface boundaries [12]. Although particle-based methods are much easier to implement and can achieve good scalability, much more gas particles are needed to discretize the modeling domain to obtain comparable numerical accuracy as compared to grid-based schemes. Thus in most cases, the required number of gas particles is undoubtedly beyond present reach. In the past two decades, a popular numerical scheme named Lattice Boltzmann Method (LBM) [13] which is based on a simplified Boltzmann equation has attracted much attention for modeling of particle–fluid flows [14]. In combination with appropriate kinetic boundary conditions, its data locality and second-order accuracy provides at least comparable efficiency as compared to traditional grid-based methods [15]. At the same time, the explicit nature of this scheme allows faster modeling speed and improved parallel efficiency. In addition, Nobel and Torczynski [16] proposed the so-called Immersed Moving Boundary (IMB) condition with sub-grid resolution to confirm no-slip boundary condition at gas–solid interface. Such interface treatment was applied by Feng et al. [17] and Wang et al. [18] where stable and reasonable results are obtained.

For the computing power, driven by the insatiable demand of high-resolution real-time display in graphic rending, General Purpose Graphics Processing Units (GPGPU) have evolved into a highly parallel platform with tremendous computational performance and memory bandwidth for SIMD applications. Peak performance of the latest generation of GPGPU card can provide about 1.5 Teraflops in single precision which is significantly higher than conventional CPU chips [19]. Besides, the introduction of Compute Unified Device Architecture (CUDA) by Nvidia corporation in 2007 provides a rather convenient programming interface for developers who are familiar with C/C++ programming language with very low learning curve. Due to the characteristics that both collision and propagation are achieved locally for standard stencils, LBM is proven to be very suitable to GPGPU computing in fine-grained parallelism with attractive speedup up to about two orders of magnitude compared to mainstream CPUs [20]. Thus, LBM in conjunction with IMB implemented on GPGPU may offer an alternative possibility for 3D gas–solid DNS.

In this paper, we describe an efficient implementation of 3D gas–solid DNS with LBM on a single Fermi GPGPU card since it is the starting point of high performance parallel computing on multiple GPGPUs. The LBM scheme for gas flow and Discrete Element Modeling (DEM) for solid particle collision will be briefly described first, together with the IMB treatment of no-slip gas–particle interface. Next, the detailed implementation of each portion on GPGPU will be discussed with emphasis on optimization issues. Validation and performance are discussed quantitatively. A 3D doubly-periodic gas–solid suspension with solid/gas density ratio over 1000 shows the potential of applying this efficient algorithm to both scientific and engineering investigations.