Kenneth Moreland
Ron Oldfield
Michael E. Papka
ABSTRACT
One of the most pressing issues with petascale analysis is the transport of simulation results data to a meaningful analysis. Traditional workflow prescribes storing the simulation results to disk and later retrieving them for analysis and visualization. However, at petascale this storage of the full results is prohibitive. A solution to this problem is to run the analysis and visualization concurrently with the simulation and bypass the storage of the full results. One mechanism for doing so is in transit visualization in which analysis and visualization is run on I/O nodes that receive the full simulation results but write information from analysis or provide run-time visualization. This paper describes the work in progress for three in transit visualization solutions, each using a different transport mechanism.
- H. Abbasi, M. Wolf, G. Eisenhauer, S. Klasky, K. Schwan, and F. Zheng. DataStager: Scalable data staging services for petascale applications. In Proceedings of the 18th ACM International Symposium on High Performance Distributed Computing (HPDC'09), 2009. DOI=10.1145/1551609.1551618. Google Scholar
Digital Library
- S. Ahern, A. Shoshani, K.-L. Ma, et al. Scientific discovery at the exascale. Report from the DOE ASCR 2011 Workshop on Exascale Data Management, Analysis, and Visualization, February 2011.Google Scholar
- J. Biddiscombe, J. Soumagne, G. Oger, D. Guibert, and J.-G. Piccinali. Parallel computational steering and analysis for hpc applications using a paraview interface and the hdf5 dsm virtual file driver. In Eurographics Symposium on Parallel Graphics and Visualization, pages 91--100, 2011. DOI=10.2312/EGPGV/EGPGV11/091--100. Google ScholarDigital Library
- K. Chand, B. Fix, T. Dahlgren, L. F. Diachin, X. Li, C. Ollivier-Gooch, E. S. Seol, M. S. Shephard, T. Tautges, and H. Trease. The ITAPS iMesh interface. Technical Report Version 0.7, U. S. Department of Energy: Science Discovery through Advanced Computing (SciDAC), 2007.Google Scholar
- H. Childs. Architectural challenges and solutions for petascale postprocessing. Journal of Physics: Conference Series, 78(012012), 2007. DOI=10.1088/1742--6596/78/1/012012.Google Scholar
- C. Docan, M. Parashar, and S. Klasky. DataSpaces: An interaction and coordination framework for coupled simulation workflows. In 19th ACM International Symposium on High Performance and Distributed Computing (HPDC'10), Chicago, IL, June 2010. Google ScholarDigital Library
- C. Docan, F. Zhang, M. Parashar, J. Cummings, N. Podhorszki, and S. Klasky. Experiments with memory-to-memory coupling for end-to-end fusion simulation workflows. In 10th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid'10), pages 293--301, Melbourne, Australia, May 2010. Google ScholarDigital Library
- S. Doi, T. Takei, and H. Matsumoto. Experiences in large-scale volume data visualization with RVSLIB. Computer Graphics, 35(2), May 2001.Google Scholar
- A. Esnard, N. Richart, and O. Coulaud. A steering environment for online parallel visualization of legacy parallel simulations. In Proceedings of the 10th International Symposium on Distributed Simulation and Real-Time Applications (DS-RT 2006), pages 7--14, October 2006. DOI=10.1109/DS-RT.2006.7. Google ScholarDigital Library
- N. Fabian, K. Moreland, D. Thompson, A. C. Bauer, P. Marion, B. Geveci, M. Rasquin, and K. E. Jansen. The ParaView coprocessing library: A scalable, general purpose in situ visualization library. In Proceedings of the IEEE Symposium on Large-Scale Data Analysis and Visualization, October 2011.Google ScholarCross Ref
- R. Haimes and D. E. Edwards. Visualization in a parallel processing environment. In Proceedings of the 35th AIAA Aerospace Sciences Meeting, number AIAA Paper 97-0348, January 1997.Google ScholarCross Ref
- C. Johnson, S. G. Parker, C. Hansen, G. L. Kindlmann, and Y. Livnat. Interactive simulation and visualization. IEEE Computer, 32(12):59--65, December 1999. DOI=10.1109/2.809252. Google ScholarDigital Library
- C. Johnson, R. Ross, et al. Visualization and knowledge discovery. Report from the DOE/ASCR Workshop on Visual Analysis and Data Exploration at Extreme Scale, October 2007.Google Scholar
- E. S. H. Jr., R. L. Bell, M. G. Elrick, A. V. Farnsworth, G. I. Kerley, J. M. McGlaun, S. V. Petney, S. A. Silling, P. A. Taylor, and L. Yarrington. CTH: A software family for multi-dimensional shock physics analysis. In R. Brun and L. Dumitrescu, editors, Proceedings of the 19th International Symposium on Shock Physics, volume 1, pages 377--382, Marseille, France, July 1993.Google Scholar
- D. Kotz. Disk-directed I/O for MIMD multiprocessors. In H. Jin, T. Cortes, and R. Buyya, editors, High Performance Mass Storage and Parallel I/O: Technologies and Applications, chapter 35, pages 513--535. IEEE Computer Society Press and John Wiley & Sons, 2001.Google Scholar
- J. Lofstead, F. Zheng, S. Klasky, and K. Schwan. Adaptable, metadata rich IO methods for portable high performance IO. In IEEE International Symposium on Parallel & Distributed Processing, IPDPS'09, May 2009. DOI=10.1109/IPDPS.2009.5161052. Google ScholarDigital Library
- J. Lofstead, F. Zheng, Q. Liu, S. Klasky, R. Oldfield, T. Kordenbrock, K. Schwan, and M. Wolf. Managing variability in the IO performance of petascale storage systems. In Proceedings of the Conference on High Performance Computing, Networking, Storage and Analysis, SC'10, New Orleans, LA, November 2010. Google ScholarDigital Library
- B. H. McCormick, T. A. DeFanti, and M. D. Brown, editors. Visualization in Scientific Computing (special issue of Computer Graphics), volume 21. ACM, 1987.Google Scholar
- S. Microsystems. RPC: remote procedure call protocol specification, version 2. Technical Report RFC 1057, Sun Microsystems, Inc., June 1988.Google ScholarDigital Library
- A. Nisar, W. keng Liao, and A. Choudhary. Scaling parallel I/O performance through I/O delegate and caching system. In Proceedings of the 2008 ACM/IEEE Conference on Supercomputing, November 2008. Google ScholarDigital Library
- R. A. Oldfield, S. Arunagiri, P. J. Teller, S. Seelam, R. Riesen, M. R. Varela, and P. C. Roth. Modeling the impact of checkpoints on next-generation systems. In Proceedings of the 24th IEEE Conference on Mass Storage Systems and Technologies, San Diego, CA, September 2007. Google ScholarDigital Library
- R. A. Oldfield, A. B. Maccabe, S. Arunagiri, T. Kordenbrock, R. Riesen, L. Ward, and P. Widener. Lightweight I/O for scientific applications. In Proceedings of the IEEE International Conference on Cluster Computing, Barcelona, Spain, Sept. 2006.Google ScholarCross Ref
- R. A. Oldfield, P. Widener, A. B. Maccabe, L. Ward, and T. Kordenbrock. Efficient data-movement for lightweight I/O. In Proceedings of the 2006 International Workshop on High Performance I/O Techniques and Deployment of Very Large I/O Systems, Barcelona, Spain, Sept. 2006.Google ScholarCross Ref
- R. A. Oldfield, A. Wilson, G. Davidson, and C. Ulmer. Access to external resources using service-node proxies. In Proceedings of the Cray User Group Meeting, Atlanta, GA, May 2009.Google Scholar
- R. A. Oldfield, D. E. Womble, and C. C. Ober. Efficient parallel I/O in seismic imaging. International Journal of High Performance Computing Applications, 12(3):333--344, Fall 1998.Google ScholarDigital Library
- T. Peterka, H. Yu, R. Ross, and K.-L. Ma. Parallel volume rendering on the IBM Blue Gene/P. In Proceedings of Eurographics Parallel Graphics and Visualization Symposium 2008, 2008. Google ScholarDigital Library
- T. Peterka, H. Yu, R. Ross, K.-L. Ma, and R. Latham. End-to-end study of parallel volume rendering on the IBM Blue Gene/P. In Proceedings of ICPP'09, pages 566--573, September 2009. DOI=10.1109/ICPP.2009.27. Google ScholarDigital Library
- M. Polte, J. Lofstead, J. Bent, G. Gibson, S. Klasky, Q. Liu, M. Parashar, N. Podhorszki, K. Schwan, M. Wingate, and M. Wolf. ...and eat it too: High read performance in write-optimized HPC I/O middleware file formats. In Proceedings of Petascale Data Storage Workshop 2009 at Supercomputing 2009, November 2009. Google ScholarDigital Library
- C. Reiss, G. Lofstead, and R. Oldfield. Implementation and evaluation of a staging proxy for checkpoint I/O. Technical report, Sandia National Laboratories, Albuquerque, NM, August 2008.Google Scholar
- R. B. Ross, T. Peterka, H.-W. Shen, Y. Hong, K.-L. Ma, H. Yu, and K. Moreland. Visualization and parallel I/O at extreme scale. Journal of Physics: Conference Series, 125(012099), 2008. DOI=10.1088/1742--6596/125/1/012099.Google Scholar
- K. E. Seamons and M. Winslett. Multidimensional array I/O in Panda 1.0. Journal of Supercomputing, 10(2):191--211, 1996. Google ScholarDigital Library
- A. H. Squillacote. The ParaView Guide: A Parallel Visualization Application. Kitware Inc., 2007. ISBN 1--930934--21--1.Google Scholar
- R. Tchoua, S. Klasky, N. Podhorszki, B. Grimm, A. Khan, E. Santos, C. Silva, P. Mouallem, and M. Vouk. Collaborative monitoring and analysis for simulation scientists. In 2010 International Symposium on Collaborative Technologies and Systems, (CTS 2010), pages 235--244, Chicago, IL, USA, May 2010.Google ScholarCross Ref
- Y. Tian, S. Klasky, H. Abbasi, J. Lofstead, R. Grout, N. Podhorszki, Q. Liu, Y. Wang, and W. Yu. Edo: Improving read performance for scientific applications through elastic data organization. In IEEE Cluster 2011, Austin, TX, 2011. Google ScholarDigital Library
- V. Vishwanath, M. Hereld, V. Morozov, and M. E. Papka. Topology-aware data movement and staging for I/O acceleration on BlueGene/P supercomputing systems. In IEEE/ACM International Conference for High Performance Computing, Networking, Storage and Analysis (SC 2011), November 2011. Google ScholarDigital Library
- C. H. Whiting and K. E. Jansen. A stabilized finite element method for the incompressible Navier-Stokes equations using a hierarchical basis. International Journal for Numerical Methods in Fluids, 35(1):93--116, January 2001.Google ScholarCross Ref
- B. Whitlock. Getting data into VisIt. Technical Report LLNL-SM-446033, Lawrence Livermore National Laboratory, July 2010.Google Scholar
- F. Zhang, C. Docan, M. Parashar, and S. Klasky. Enabling multi-physics coupled simulations within the PGAS programming framework. In IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), pages 84--93, 2011. Google ScholarDigital Library
Index Terms
- Examples of in transit visualization
Recommendations
Cinema image-based in situ analysis and visualization of MPAS-ocean simulations
We created an in situ exploration visualization of an MPAS-Ocean simulation.We leveraged compositing in Cinema to provide interactive exploration.We decreased the storage footprint of the analysis and visualization results. Due to power and I/O ...
Catalyst-ADIOS2: In Transit Analysis for Numerical Simulations Using Catalyst 2 API
High Performance ComputingAbstractIn this article, we present a novel approach to bring in transit capabilities to numerical simulations which are already able to do in situ analysis with Catalyst 2. This approach combines the stable ABI of Catalyst 2, to replace the in situ ...
Including in Situ Visualization and Analysis in PDI
High Performance ComputingAbstractThe goal of this work was to integrate in situ possibilities into the general-purpose code-coupling library PDI [1]. This is done using the simulation code Alya as an example. Here, an open design is taken into account to later create ...
Comments