Nicolas Offermans
ABSTRACT
The present work is targeted at performing a strong scaling study of the high-order spectral element fluid dynamics solver Nek5000. Prior studies such as [5] indicated a recommendable metric for strong scalability from a theoretical viewpoint, which we test here extensively on three parallel machines with different performance characteristics and interconnect networks, namely Mira (IBM Blue Gene/Q), Beskow (Cray XC40) and Titan (Cray XK7). The test cases considered for the simulations correspond to a turbulent flow in a straight pipe at four different friction Reynolds numbers Reτ = 180, 360, 550 and 1000. Considering the linear model for parallel communication we quantify the machine characteristics in order to better assess the scaling behaviors of the code. Subsequently sampling and profiling tools are used to measure the computation and communication times over a large range of compute cores. We also study the effect of the two coarse grid solvers XXT and AMG on the computational time. Super-linear scaling due to a reduction in cache misses is observed on each computer. The strong scaling limit is attained for roughly 5000-10,000 degrees of freedom per core on Mira, 30,000 - 50,0000 on Beskow, with only a small impact of the problem size for both machines, and ranges between 10,000 and 220,000 depending on the problem size on Titan. This work aims at being a reference for Nek5000 users and also serves as a basis for potential issues to address as the community heads towards exascale supercomputers.
- M. O. Deville, P. F. Fischer, and E. H. Mund. High-Order Methods for Incompressible Fluid Flow. Cambridge University Press, 2002. Cambridge Books Online.Google Scholar
Cross Ref
- P. Fischer, J. Lottes, S. Kerkemeier, O. Marin, K. Heisey, A. Obabko, E. Merzari, and Y. Peet. Nek5000: User's manual. Technical Report ANL/MCS-TM-351, Argonne National Laboratory, 2015.Google Scholar
- P. F. Fischer. An overlapping schwarz method for spectral element solution of the incompressible Navier-Stokes equations. Journal of Computational Physics, 133(1):84--101, 1997. Google ScholarDigital Library
- P. F. Fischer. Projection techniques for iterative solution of Ax = b with successive right-hand sides. Computer Methods in Applied Mechanics and Engineering, 163(1-4):193--204, 1998.Google ScholarCross Ref
- P. F. Fischer, K. Heisey, and M. Min. Scaling Limits for PDE-Based Simulation (Invited). AIAA Aviation. American Institute of Aeronautics and Astronautics, jun 2015. doi:10.2514/6.2015-3049.Google Scholar
- P. F. Fischer and J. W. Lottes. Domain Decomposition Methods in Science and Engineering, chapter Hybrid Schwarz-Multigrid Methods for the Spectral Element Method: Extensions to Navier-Stokes, pages 35--49. Springer Berlin Heidelberg, Berlin, Heidelberg, 2005.Google Scholar
- T. Hoefler, T. Schneider, and A. Lumsdaine. Characterizing the influence of system noise on large-scale applications by simulation. In Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, SC '10, pages 1--11, Washington, DC, USA, 2010. IEEE Computer Society. Google ScholarDigital Library
- M. Hutchinson, A. Heinecke, H. Pabst, G. Henry, M. Parsani, and D. Keyes. Efficiency of high order spectral element methods on petascale architectures. ISC High Performance, 2016.Google ScholarCross Ref
- G. K. Khoury, P. Schlatter, A. Noorani, P. F. Fischer, G. Brethouwer, and A. V. Johansson. Direct numerical simulation of turbulent pipe flow at moderately high Reynolds numbers. Flow, Turbulence and Combustion, 91(3):475--495, 2013.Google ScholarCross Ref
- J. Lottes. Independent quality measures for symmetric algebraic multigrid components. Argonne National Laboratory, Mathematics & Computer Science Division, 2005.Google Scholar
- M. Otten, J. Gong, A. Mametjanov, A. Vose, J. Levesque, P. Fischer, and M. Min. An mpi/openacc implementation of a high-order electromagnetics solver with gpudirect communication. International Journal of High Performance Computing Applications, 2016.Google ScholarCross Ref
- A. G. Tomboulides, J. C. Y. Lee, and S. A. Orszag. Numerical simulation of low Mach number reactive flows. Journal of Scientific Computing, 12(2):139--167, 1997. Google ScholarDigital Library
- H. Tufo and P. Fischer. Fast parallel direct solvers for coarse grid problems. Journal of Parallel and Distributed Computing, 61(2):151--177, 2001. Google ScholarDigital Library
- H. M. Tufo and P. F. Fischer. Terascale spectral element algorithms and implementations. In Proceedings of the 1999 ACM/IEEE Conference on Supercomputing, SC '99, New York, NY, USA, 1999. ACM. Google ScholarDigital Library
- On the Strong Scaling of the Spectral Element Solver Nek5000 on Petascale Systems
Recommendations
Strong Scaling of OpenACC enabled Nek5000 on several GPU based HPC systems
HPCAsia '22: International Conference on High Performance Computing in Asia-Pacific RegionWe present new results on the strong parallel scaling for the OpenACC-accelerated implementation of the high-order spectral element fluid dynamics solver Nek5000. The test case considered consists of a direct numerical simulation of fully-developed ...
OpenACC acceleration of the Nek5000 spectral element code
We present a case study of porting NekBone, a skeleton version of the Nek5000 code, to a parallel GPU-accelerated system. Nek5000 is a computational fluid dynamics code based on the spectral element method used for the simulation of incompressible flow. ...
NekRS, a GPU-accelerated spectral element Navier–Stokes solver
AbstractThe development of NekRS, a GPU-oriented thermal-fluids simulation code based on the spectral element method (SEM) is described. For performance portability, the code is based on the open concurrent compute abstraction and leverages ...
Highlights- We describe the development of NekRS, a GPU-oriented spectral element code for CFD.
Comments