Victor A. Mateevitsi
Michael E. Papka
ABSTRACT
In the realm of Computational Fluid Dynamics (CFD), the demand for memory and computation resources is extreme, necessitating the use of leadership-scale computing platforms for practical domain sizes. This intensive requirement renders traditional checkpointing methods ineffective due to the significant slowdown in simulations while saving state data to disk. As we progress towards exascale and GPU-driven High-Performance Computing (HPC) and confront larger problem sizes, the choice becomes increasingly stark: to compromise data fidelity or to reduce resolution. To navigate this challenge, this study advocates for the use of in situ analysis and visualization techniques. These allow more frequent data "snapshots" to be taken directly from memory, thus avoiding the need for disruptive checkpointing. We detail our approach of instrumenting NekRS, a GPU-focused thermal-fluid simulation code employing the spectral element method (SEM), and describe varied in situ and in transit strategies for data rendering. Additionally, we provide concrete scientific use-cases and report on runs performed on Polaris, Argonne Leadership Computing Facility’s (ALCF) 44 Petaflop supercomputer and Jülich Wizard for European Leadership Science (JUWELS) Booster, Jülich Supercomputing Centre’s (JSC) 71 Petaflop High Performance Computing (HPC) system, offering practical insight into the implications of our methodology.
- John David Anderson and John Wendt. 1995. Computational fluid dynamics. Vol. 206. Springer.Google Scholar
- Marco Atzori, Wiebke Köpp, Steven W. D. Chien, Daniele Massaro, Fermín Mallor, Adam Peplinski, Mohamad Rezaei, Niclas Jansson, Stefano Markidis, Ricardo Vinuesa, Erwin Laure, Philipp Schlatter, and Tino Weinkauf. 2022. In situ visualization of large-scale turbulence simulations in Nek5000 with ParaView Catalyst. The Journal of Supercomputing 78, 3 (Feb. 2022), 3605–3620. https://doi.org/10.1007/s11227-021-03990-3Google ScholarDigital Library
- Utkarsh Ayachit, Andrew Bauer, Berk Geveci, Patrick O’Leary, Kenneth Moreland, Nathan Fabian, and Jeffrey Mauldin. 2015. ParaView Catalyst: Enabling In Situ Data Analysis and Visualization. In Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization(ISAV2015). Association for Computing Machinery, New York, NY, USA, 25–29. https://doi.org/10.1145/2828612.2828624Google ScholarDigital Library
- Utkarsh Ayachit, Brad Whitlock, Matthew Wolf, Burlen Loring, Berk Geveci, David Lonie, and E. Wes Bethel. 2016. The SENSEI Generic In Situ Interface. In 2016 Second Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV). 40–44. https://doi.org/10.1109/ISAV.2016.013Google ScholarCross Ref
- Bennett Bernardoni, Nicola Ferrier, Joseph Insley, Michael E Papka, Saumil Patel, and Silvio Rizzi. 2018. In situ visualization and analysis to design large scale experiments in computational fluid dynamics. In 2018 IEEE 8th Symposium on Large Data Analysis and Visualization (LDAV). IEEE, 94–95.Google ScholarCross Ref
- Hank Childs. 2012. In Situ Processing. (Nov. 2012). https://escholarship.org/uc/item/3st8x19dGoogle Scholar
- Hank Childs. 2012. VisIt: An End-User Tool for Visualizing and Analyzing Very Large Data. (Nov. 2012). https://escholarship.org/uc/item/69r5m58vGoogle Scholar
- Paul Fischer, Stefan Kerkemeier, Misun Min, Yu-Hsiang Lan, Malachi Phillips, Thilina Rathnayake, Elia Merzari, Ananias Tomboulides, Ali Karakus, Noel Chalmers, and Tim Warburton. 2022. NekRS, a GPU-accelerated spectral element Navier–Stokes solver. Parallel Comput. 114 (Dec. 2022), 102982. https://doi.org/10.1016/j.parco.2022.102982Google ScholarDigital Library
- Paul Fischer, Einar M. Ronquist, Daniel Dewey, and Anthony T. Patera. 1988. Spectral element methods: Algorithms and architectures. Technical Report NAS 1.26:182701. https://ntrs.nasa.gov/citations/19880011494 NTRS Author Affiliations: Massachusetts Inst. of Tech. NTRS Document ID: 19880011494 NTRS Research Center: Legacy CDMS (CDMS).Google Scholar
- Paul Frederick Fischer. 1989. Spectral element solution of the Navier-Stokes equations on high performance distributed-memory parallel processors. PhD Thesis. Massachusetts Institute of Technology.Google Scholar
- A. Karakus, N. Chalmers, K. Świrydowicz, and T. Warburton. 2019. A GPU accelerated discontinuous Galerkin incompressible flow solver. J. Comput. Phys. 390 (Aug. 2019), 380–404. https://doi.org/10.1016/j.jcp.2019.04.010Google ScholarDigital Library
- Essam E. Khalil. 2021. CFD History and Applications. ARCHIVES OF AKADEMIA BARU ARTICLES 4, 2 (July 2021), 43–46. https://www.akademiabaru.com/index.php/archives/article/view/278 Number: 2.Google Scholar
- T Kuhlen, R Pajarola, and K Zhou. 2011. Parallel in situ coupling of simulation with a fully featured visualization system. In Proceedings of the 11th Eurographics Conference on Parallel Graphics and Visualization (EGPGV), Vol. 10. Eurographics Association Aire-la-Ville, Switzerland, 101–109.Google Scholar
- Matthew Larsen, Eric Brugger, Hank Childs, and Cyrus Harrison. 2022. Ascent: A Flyweight In Situ Library for Exascale Simulations. In In Situ Visualization for Computational Science(Mathematics and Visualization), Hank Childs, Janine C. Bennett, and Christoph Garth (Eds.). Springer International Publishing, Cham, 255–279. https://doi.org/10.1007/978-3-030-81627-8_12Google ScholarCross Ref
- Kwan-Liu Ma, Chaoli Wang, Hongfeng Yu, and Anna Tikhonova. 2007. In-situ processing and visualization for ultrascale simulations. Journal of Physics: Conference Series 78, 1 (July 2007), 012043. https://doi.org/10.1088/1742-6596/78/1/012043Google ScholarCross Ref
- Victor A. Mateevitsi, Mathis Bode, Nicola Ferrier, Paul Fischer, Jens Henrik Göbbert, Joseph A. Insley, Yu-Hsiang Lan, Misun Min, Michael E. Papka, Saumil Patel, Silvio Rizzi, and Jonathan Windgassen. 2023. Software and Analysis for paper: Scaling Computational Fluid Dynamics: In Situ Visualization of NekRS using SENSEI. https://doi.org/10.5281/zenodo.8377974Google ScholarCross Ref
- David Medina. 2015. OKL: A Unified Language for Parallel Architectures. (June 2015). https://scholarship.rice.edu/handle/1911/102233 Accepted: 2018-06-19T17:49:54Z.Google Scholar
- David S. Medina, Amik St-Cyr, and T. Warburton. 2014. OCCA: A unified approach to multi-threading languages. https://doi.org/10.48550/arXiv.1403.0968 arXiv:1403.0968 [cs].Google ScholarCross Ref
- Paul Messina. 2017. The Exascale Computing Project. Computing in Science & Engineering 19, 3 (May 2017), 63–67. https://doi.org/10.1109/MCSE.2017.57 Conference Name: Computing in Science & Engineering.Google ScholarCross Ref
- Misun Min, Yu-Hsiang Lan, Paul Fischer, Elia Merzari, Stefan Kerkemeier, Malachi Phillips, Thilina Rathnayake, April Novak, Derek Gaston, Noel Chalmers, and Tim Warburton. 2022. Optimization of full-core reactor simulations on summit. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis(SC ’22). IEEE Press, Dallas, Texas, 1–11.Google ScholarDigital Library
- Kenneth Moreland, Christopher Sewell, William Usher, Li-ta Lo, Jeremy Meredith, David Pugmire, James Kress, Hendrik Schroots, Kwan-Liu Ma, Hank Childs, Matthew Larsen, Chun-Ming Chen, Robert Maynard, and Berk Geveci. 2016. VTK-m: Accelerating the Visualization Toolkit for Massively Threaded Architectures. IEEE Computer Graphics and Applications 36, 3 (May 2016), 48–58. https://doi.org/10.1109/MCG.2016.48 Conference Name: IEEE Computer Graphics and Applications.Google ScholarDigital Library
- Ambrish Pandey, Dmitry Krasnov, Katepalli R Sreenivasan, and Jörg Schumacher. 2022. Convective mesoscale turbulence at very low Prandtl numbers. Journal of Fluid Mechanics 948 (2022), A23. Publisher: Cambridge University Press.Google ScholarCross Ref
- Rick Stevens, Jini Ramprakash, Paul Messina, Michael Papka, and Katherine Riley. 2019. Aurora: Argonne’s Next-Generation Exascale Supercomputer. Technical Report. Argonne National Lab. (ANL), Argonne, IL (United States). https://www.osti.gov/sciencecinema/biblio/1562918Google Scholar
- I Wald, GP Johnson, J Amstutz, C Brownlee, A Knoll, J Jeffers, J Günther, and P Navratil. 2017. OSPRay - A CPU Ray Tracing Framework for Scientific Visualization. IEEE Transactions on Visualization and Computer Graphics 23, 1 (Jan. 2017), 931–940. https://doi.org/10.1109/TVCG.2016.2599041 Conference Name: IEEE Transactions on Visualization and Computer Graphics.Google ScholarDigital Library
- E. E. Zajac. 1964. Computer-made perspective movies as a scientific and communication tool. Commun. ACM 7, 3 (March 1964), 169–170. https://doi.org/10.1145/363958.363993Google ScholarDigital Library
- Kasia Świrydowicz, Noel Chalmers, Ali Karakus, and Tim Warburton. 2019. Acceleration of tensor-product operations for high-order finite element methods. The International Journal of High Performance Computing Applications 33, 4 (July 2019), 735–757. https://doi.org/10.1177/1094342018816368 Publisher: SAGE Publications Ltd STM.Google ScholarDigital Library
Index Terms
- Scaling Computational Fluid Dynamics: In Situ Visualization of NekRS using SENSEI
Index terms have been assigned to the content through auto-classification.
Recommendations
Large-Scale Distributed Computational Fluid Dynamics on the Information Power Grid using Globus
FRONTIERS '99: Proceedings of the The 7th Symposium on the Frontiers of Massively Parallel ComputationThis paper describes an experiment in which a large-scale scientific application developed for tightly-coupled parallel machines is adapted to the distributed execution environment of the Information Power Grid (IPG). A brief overview of the IPG and a ...
Flow Visualization in Computational Fluid Dynamics
Several flow visualization techniques using the Lagran gian approach are proposed for analyzing the numerical solutions of unsteady flow fields computed by the Eu lerian approach. We show how these methods can be used to assess the validity of solutions ...
Comments