Skip to main content
SpringerLink
Log in

Hybrid hierarchy storage system in MilkyWay-2 supercomputer

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

With the rapid improvement of computation capability in high performance supercomputer system, the imbalance of performance between computation subsystem and storage subsystem has become more and more serious, especially when various big data are produced ranging from tens of gigabytes up to terabytes. To reduce this gap, large-scale storage systems need to be designed and implemented with high performance and scalability.MilkyWay-2 (TH-2) supercomputer system with peak performance 54.9 Pflops, definitely has this kind of requirement for storage system. This paper mainly introduces the storage system in MilkyWay-2 supercomputer, including the hardware architecture and the parallel file system. The storage system in MilkyWay-2 supercomputer exploits a novel hybrid hierarchy storage architecture to enable high scalability of I/O clients, I/O bandwidth and storage capacity. To fit this architecture, a user level virtualized file system, named H2FS, is designed and implemented which can cooperate local storage and shared storage together into a dynamic single namespace to optimize I/O performance in IO-intensive applications. The evaluation results show that the storage system in MilkyWay-2 supercomputer can satisfy the critical requirements in large scale supercomputer, such as performance and scalability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Franks B. Taming the Big Data Tidal Wave: Finding Opportunities in Huge Data Streams with Advanced Analytics. www.wiley.com, 2012

    Google Scholar 

  2. Verta O, Mastroianni C, Talia D. A super-peer model for resource discovery services in large-scale grids. Future Generation Computer Systems, 2005, 21(8): 1235–1248

    Article  Google Scholar 

  3. Bent J, Grider G, Kettering Br, Manzanares A, McClelland M, Torres A, Torrez A. Storage challenges at Los Alamos National Lab. In: Proceedings of the 2012 Symposium on Massive Storage Systems and Technologies. 2012: 1–5

    Google Scholar 

  4. Watson R W, Coyne R A. The parallel I/O architecture of the high-performance storage system. In: Proceedings of the 14th IEEE Symposium on Mass Storage Systems. 1995, 27–44

    Chapter  Google Scholar 

  5. Lofstead J, Zheng F, Liu Q, Klasky S, Oldfield R, Kordenbrock T, Schwan K, Wolf M. Managing variability in the IO performance of petascale storage system. IEEE Computer Society, 2010: 1–12

    Google Scholar 

  6. Zhuge H. The Knowledge Grid. Singapore: World Scientific, 2004

    Book  Google Scholar 

  7. Oldfield R A, Maccabe A B, Arunagiri S, Kordenbrock T, Riesen R, Ward L, Widener P. Lightweight I/O for scientific applications. Technical Report of Sandia National Laboratories, 2006, 1–11

    Google Scholar 

  8. Liu N, Cope J, Carns P H, Carothers C D, Ross R B, Grider G, Crume A, Maltzahn C. On the role of burst buffers in leadership-class storage systems. In: Proceedings of the 2012 Symposium on Massive Storage Systems and Technologies. 2012: 1–11

    Google Scholar 

  9. Zhou E Q, Lu Y T, Zhang W, Dong Y. H2FS: a hybrid hierarchy file-system for scalable data-intensive computing for HPC systems. Poster paper in International Supercomputing Conference. 2013

    Google Scholar 

  10. Lustre: A scalable, high-performance file system. Cluster File Systems Inc. Whitepaper, Version 1.0, November 2002. http://www.lustre.org/docs/whitepaper.pdf

  11. Xie M, Lu Y T, Liu L, Cao H J, Yang X J. Implementation and evaluation of network interface and message passing services for TianHe-1A supercomputer. In: Proceedings of the 19th Annual IEEE Symposium on High Performance Interconnects. 2011, 78–86

    Google Scholar 

  12. Welch B, Unangst M, Abbasi Z, Gibson G, Mueller B, Small J, Zelenka J, Zhou B. Scalable performance of the panasas parallel file system. FAST, 2008, 8: 1–17

    Google Scholar 

  13. Top500 Lists, http://www.top500.org/lists/

  14. Ryu K D, Inglett T A, Bellofatto R, Blocksome, M. A, Gooding T, Kumar S, Mamidala A R, Megerian, M G, Miller S, Nelson M T, Rosenburg B, Smith B, Van O J, Wang A, Wisniewski R W. IBM Blue Gene/Q system software stack. IBM Journal of Research and Development, 2013, 57(1/2): 1–12

    Article  Google Scholar 

  15. Rogers J. Power efficiency and performance with ORNL’s cray XK7 Titan. Star Craft Companion, 2012: 1040–1050

    Google Scholar 

  16. Yu W, Vetter J S, Oral H S. Performance characterization and optimization of parallel i/o on the cray XT. In: Proceedings of the IEEE International Symposium on Parallel and Distributed Processing. 2008, 1–11

    Google Scholar 

  17. Yu W, Oral S, Vetter J, Barrett R. Efficiency evaluation of Ccray XT parallel IO stack. Cray User Group Meeting, 2007, 1–9

    Google Scholar 

  18. Miyazaki H, Kusano Y, Shinjou N, et al. Overview of the K computer system. Fujitsu Scientific and Technical Journal, 2012, 48

  19. Xing J, Xiong J, Sun N, Jie M. Adaptive and scalable metadata management to support a trillion files. In: Proceedings of the ACM Conference on High Performance Computing Networking, Storage and Analysis. 2009, 26: 1–11

    Article  Google Scholar 

  20. Surendra B, Chou J, Rübel O, Prabhat, Karimabadi H, Daughton W S, Roytershteyn V, Bethel E W, Howison M, Hsu K J, Lin KW, Shoshani A, Uselton A, Wu K. Parallel I/O, analysis, and visualization of a trillion particle simulation. Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society Press, 2012, 59: 1–12

    Google Scholar 

  21. Payne M, Widener P, Wolf M, Abbasi H, McManus S, Bridges P G, Schwan K. Exploiting latent I/O asynchrony in petascale science applications. In: Proceedings of the 4th IEEE International Conference on EScience. 2008: 410–411

    Google Scholar 

  22. Ali N, Carns P, Iskra K, Kimpe D, Lang S, Latham R, Ross R, Ward L, Sadayappan P. Scalable I/O forwarding framework for high-performance computing systems. In: Proceedings of the 2009 IEEE International Conference on Cluster Computing and Workshops. 2009, 1–10

    Chapter  Google Scholar 

  23. Lu Y Y, Shu J W, Li S, Yi L T. Accelerating distributed updates with asynchronous ordered writes in a parallel file system. In: Proceedings of the 2012 IEEE International Conference on Cluster Computing. 2012, 302–310

    Chapter  Google Scholar 

  24. Shepler S, Callaghan B, Robinson D, Thurlow R, Sun Microsystems Inc., Beame C, Hummingbird Ltd., Eisler M, Doveck D, Network Appliance Inc. Network file system version 4 protocol. Network, 2003, 3530

    Google Scholar 

  25. Goodson G, Welch B, Halevy B, Black D, Adamson A. NFSv4 pNFS extensions. Technical Report, 2005

    Google Scholar 

  26. Carns P H, Settlemyer B W, Ligon W B III. Using server-to-server communication in parallel file systems to simplify consistency and improve performance. In: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing. 2008, 6

    Google Scholar 

  27. Devulapalli A, Ohio P W. File creation strategies in a distributed metadata file system. In: Proceedings of the 2007 IEEE International on Parallel and Distributed Processing Symposium. 2007, 1–10

    Chapter  Google Scholar 

  28. Carns P, Lang S, Ross R, Vilayannur M, Kunkel J, Ludwig T. Small-file access in parallel file systems. In: Proceedings of the 2009 IEEE International Symposium on Parallel and Distributed Processing. 2009, 1–11

    Chapter  Google Scholar 

  29. Sakai K, Sumimoto S, Kurokawa M. High-performance and highly reliable file system for the K computer. FUJITSU Science Technology, 2012, 48(3): 302–209

    Google Scholar 

  30. Liu Q, Klasky S, Podhorszki N, Lofstead H, Abbasi C S, Chang J, Cummings D, Dinakar C, Docan S, Ethier R, Grout T, Kordenbrock Z, Lin X, Ma R, Oldfield M, Parashar A, Romosan N, Samatova K, Schwan A, Shoshani Y, Tian M, Wolf W, Yu F, Zhang F, Zheng F. ADIOS: powering I/O to extreme scale computing. 1–6

  31. Lofstead J, Zheng F, Klasky S, Schwan K. Adaptable, metadata rich IO methods for portable high performance IO. In: Proceedings of the 2009 International Parallel and Distributed Processing. 2009, 1–10

    Google Scholar 

  32. Li J, Liao W, Choudhary A, Ross R, Thakur R, Gropp W, Latham R, Siegel A, Gallagher B, Zingale M. Parallel netCDF: a high-performance scientific I/O interface. In: Proceedings of the 2003 ACM/IEEE Conference on Supercomputing. 2003, 39

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory of High Performance Computing, Changsha, 410073, China

    Weixia Xu, Yutong Lu, Enqiang Zhou, Yong Dong, Wei Zhang & Haitao Chen

  2. College of Computer, National University of Defense Technology, Changsha, 410073, China

    Weixia Xu, Yutong Lu, Qiong Li, Enqiang Zhou, Zhenlong Song, Yong Dong, Wei Zhang, Dengping Wei, Xiaoming Zhang, Haitao Chen, Jianying Xing & Yuan Yuan

Authors
  1. Weixia Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yutong Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qiong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Enqiang Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhenlong Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yong Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Wei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Dengping Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Xiaoming Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Haitao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Jianying Xing
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Yuan Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yutong Lu.

Additional information

Weixia Xu received his MS in computer science from National University of Defense Technology (NUDT), China in 1992. Currently he is a professor at this university. His research interests include system structure of parallel computing and system structure of multi-core microprocessor. He has been working in College of Computer in NUDT since 1984. He is the deputy chief designer of TianHe-1 supercomputer and MilkyWay-2 supercomputer.

Yutong Lu received her MS and PhD in computer science from National University of Defense Technology (NUDT), China. Currently she is a professor at the university. Her research interests include parallel system management, high speed communication, distributed file systems, and advanced programming environments with MPI. She is a director designer of MilkyWay-2 supercomputer.

Qiong Li is currently a professor in College of Computer, National University of Defense Technology (NUDT), China. She received her MS and PhD in computer science from NUDT, China. Her current research interests include high performance computing and storage technology.

Enqiang Zhou is currently an associate professor in College of Computer, National University of Defense Technology (NUDT), China. His current research interests include high performance computing, storage technology and distributed file system.

Zhenglong Song is currently an assistant professor in College of Computer, National University of Defense Technology (NUDT), China. His current research interests include high performance computing and storage technology.

Yong Dong is currently an assistant professor in College of Computer, National University of Defense Technology (NUDT), China. His current research interests include high performance computing and distributed file system.

Wei Zhang received his BS, MS and PhD in computer science from College of Computer, National University of Defense Technology (NUDT), China. He is currently an assistant professor in College of Computer, NUDT, China. His current research interests include high performance computing and distributed simulation.

Dengping Wei received her MS and PhD in computer science from National University of Defense Technology (NUDT), China, in 2005 and 2011, respectively. Currently she is an assistant professor in College of Computer, NUDT, China. Her research interests mainly involve high performance computing, storage technology and big data.

Xiaoming Zhang is currently an associate professor in computer science in National University of Defense Technology (NUDT), China. His research interests mainly involve high performance computing and interconnection network.

Haitao Chen is an associate professor in computer science in National University of Defense Technology (NUDT), China. His research interests mainly involve high performance computing, performance evaluation.

Jianying Xing received his MS and PhD in computer science from National University of Defense Technology (NUDT), China in 2007 and 2011, respectively. Currently he is an assistant professor in College of Computer, NUDT, China. His research interests mainly involve high performance computing, performance evaluation.

Yuan Yuan received his MS and PhD in computer science from National University of Defense Technology (NUDT), China in 2007 and 2011, respectively. Currently he is an assistant professor in College of Computer, NUDT, China. His research interests mainly involve high performance computing, performance evaluation.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, W., Lu, Y., Li, Q. et al. Hybrid hierarchy storage system in MilkyWay-2 supercomputer. Front. Comput. Sci. 8, 367–377 (2014). https://doi.org/10.1007/s11704-014-3499-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11704-014-3499-6

Keywords

Search

Navigation