Abstract
Porting a complex scientific code, such as the E3SM land model (ELM), onto a new computing architecture is challenging. The paper presents design strategies and technical approaches to develop an ELM ecosystem dynamics model with compiler directives (OpenACC) on NVIDIA GPUs. The code has been refactored with advanced OpenACC features (such as deepcopy and routine directives) to reduce memory consumption and to increase the levels of parallelism through parallel loop reconstruction and new data structures. As a result, the optimized parallel implementation achieved more than a 140-time speedup (50 ms vs 7600 ms), compared to a naive implementation that uses OpenACC routine directive and parallelizes the code across existing loops on a single NVIDIA V100. On a fully loaded computing node with 44 CPUs and 6 GPUs, the code achieved over a 3.0-times speedup, compared to the original code on the CPU. Furthermore, the memory footprint of the optimized parallel implementation is 300 MB, which is around 15% of the 2.15 GB of memory consumed by a naive implementation. This study is the first effort to develop the ELM component on GPUs efficiently to support ultra-high-resolution land simulations at continental scales.
This research was supported as part of the Energy Exascale Earth System Model (E3SM) project, funded by the U.S. Department of Energy, Office of Science, Office of Biological and Environmental Research. This research used resources of the Oak Ridge Leadership Computing Facility and Experimental Computing Laboratory at the Oak Ridge National Laboratory, which are supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The spin-up simulation is used for the ELM to reach a state of statistical equilibrium under the applied climatic forcing.
- 2.
The transit simulation is referred to the ELM simulation of post industry revolution period (1850-present) with rising CO2, greenhouse emission, dynamical vegetation, and land use and land change, including urbanization.
- 3.
AnNVIDIAinteractivekernelprofiler, https://developer.nvidia.com/nsight-compute.
- 4.
An NVIDIA performance analysis tool, https://developer.nvidia.com/nsight-systems.
References
Bertagna, L., et al.: A performance-portable nonhydrostatic atmospheric dycore for the energy exascale earth system model running at cloud-resolving resolutions. In: SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–14. IEEE (2020)
Bourzac, K.: Supercomputing poised for a massive speed boost. Nature 551(7680), 554–557 (2017)
Burrows, S., et al.: The doe e3sm v1. 1 biogeochemistry configuration: Description and simulated ecosystem-climate responses to historical changes in forcing. J. Adv. Model. Earth Syst. 12(9), e2019MS001766 (2020)
Ghane, M., Chandrasekaran, S., Cheung, M.S.: Assessing performance implications of deep copy operations via microbenchmarking. CoRR abs/1906.01128 (2019). http://arxiv.org/abs/1906.01128
Golaz, J.C., et al.: The doe e3sm coupled model version 1: overview and evaluation at standard resolution. J. Adv. Model. Earth Syst. 11(7), 2089–2129 (2019)
Kim, J.Y., Kang, J.S., Joh, M.: Gpu acceleration of mpas microphysics wsm6 using openacc directives: performance and verification. Comput. Geosci. 146, 104627 (2021)
Thornton, P.E., Shrestha, R., Thornton, M., Kao, S.C., Wei, Y., Wilson, B.E.: Gridded daily weather data for North America with comprehensive uncertainty quantification. Sci. Data 8(1), 1–17, 104627 (2021)
Wang, D., et al.: Scientific functional testing platform for environmental models: an application to community land model. In: International Workshop on Software Engineering for High Performance Computing in Science, 37th International Conference on Software Engineering (2015)
Wang, D.: A functional test platform for the community land model. Environ. Model. Softw. 55, 25–31 (2014)
Zhang, S.: Optimizing high-resolution community earth system model on a heterogeneous many-core supercomputing platform. Geosci. Model Dev. 13(10), 4809–4829 (2020)
Zheng, W., Wang, D., Song, F.: XScan: an integrated tool for understanding open source community-based scientific code. In: Rodrigues, J.M.F., et al. (eds.) ICCS 2019. LNCS, vol. 11536, pp. 226–237. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22734-0_17
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Schwartz, P., Wang, D., Yuan, F., Thornton, P. (2022). Developing an ELM Ecosystem Dynamics Model on GPU with OpenACC. In: Groen, D., de Mulatier, C., Paszynski, M., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M.A. (eds) Computational Science – ICCS 2022. ICCS 2022. Lecture Notes in Computer Science, vol 13351. Springer, Cham. https://doi.org/10.1007/978-3-031-08754-7_38
Download citation
DOI: https://doi.org/10.1007/978-3-031-08754-7_38
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-08753-0
Online ISBN: 978-3-031-08754-7
eBook Packages: Computer ScienceComputer Science (R0)