Skip to main content
SpringerLink
Log in

Smart Clustering of HPC Applications Using Similar Job Detection Methods

  • Conference paper
  • First Online:
Parallel Processing and Applied Mathematics (PPAM 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13826))

Abstract

In order for supercomputer resources to be effectively used, it is necessary to constantly analyze various aspects of the operation of modern HPC systems. One of the most significant aspects is the efficiency of execution of parallel applications running on a supercomputer. To study this, system administrators need to constantly monitor and analyze the entire flow of running jobs. This is a very difficult task, and there are several reasons for this - a large number and a significant variety of executed applications; the extreme complexity of the structure of modern HPC systems, and, as a result, a huge number of characteristics that need to be evaluated for each job. One way to make this analysis easier is to cluster similar jobs. Such clustering allows you to infer the behavior and performance issues of all jobs in the cluster by examining only one of these jobs, and it also helps to better understand the structure of the supercomputer job flow as a whole. In this paper, we propose a new method that allows solving this clustering task with high accuracy. This smart clustering method analyzes both static information on the executable files and dynamic data about the behavior of applications during their execution. Using the Lomonosov-2 supercomputer as an example, we demonstrate how this method can help in practice to facilitate the analysis of the execution efficiency of supercomputing applications.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. High performance computing market size to surpass USD 64.65. https://www.globenewswire.com/news-release/2022/04/04/2415844/0/en/High-Performance-Computing-Market-Size-to-Surpass-USD-64-65-Bn-by-2030.html

  2. Agrawal, K., Fahey, M., Mclay, R., James, D.: User environment tracking and problem detection with xalt, pp. 32–40, November 2014. https://doi.org/10.1109/HUST.2014.6

  3. Ates, E., et al.: Taxonomist: application detection through rich monitoring data. In: Aldinucci, M., Padovani, L., Torquati, M. (eds.) Euro-Par 2018. LNCS, vol. 11014, pp. 92–105. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-96983-1_7

    Chapter  Google Scholar 

  4. Berndt, D.J., Clifford, J.: Using dynamic time warping to find patterns in time series. In: Proceedings of the 3rd International Conference on Knowledge Discovery and Data Mining, AAAIWS 1994, pp. 359–370. AAAI Press (1994). http://dl.acm.org/citation.cfm?id=3000850.3000887

  5. Ester, M., Kriegel, H.P., Sander, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD 1996, pp. 226–231. AAAI Press (1996)

    Google Scholar 

  6. Gurrutxaga, I., Arbelaitz, O., Martín, J., Muguerza, J., Pérez, J., Perona, I.: Sihc: a stable incremental hierarchical clustering algorithm, pp. 300–304, January 2009

    Google Scholar 

  7. Halawa, M., Díaz Redondo, R., Vilas, A.: Unsupervised kpis-based clustering of jobs in HPC data centers. Sensors 20, 4111 (2020). https://doi.org/10.3390/s20154111

  8. Hubert, L.J., Arabie, P.: Comparing partitions. J. Classif. 2, 193–218 (1985)

    Article  MATH  Google Scholar 

  9. Joseph, E., Conway, S.: Major trends in the worldwide HPC market. Technical Report (2017). https://hpcuserforum.com/presentations/stuttgart2017/IDC-update-HLRS.pdf

  10. Kuhn, A., Ducasse, S., Gîrba, T.: Semantic clustering: identifying topics in source code. Inf. Softw. Technol. 49(3), 230–243 (2007). https://doi.org/10.1016/j.infsof.2006.10.017, https://www.sciencedirect.com/science/article/pii/S0950584906001820, 12th Working Conference on Reverse Engineering

  11. Le, Q.V., Mikolov, T.: Distributed representations of sentences and documents. CoRR abs/1405.4053 (2014)

    Google Scholar 

  12. Nikitenko, D.A., Shvets, P.A., Voevodin, V.V.: Why do users need to take care of their HPC applications efficiency? Lobachevskii J. Math. 41(8), 1521–1532 (2020). https://doi.org/10.1134/s1995080220080132

    Article  Google Scholar 

  13. Pedregosa, F., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    MathSciNet  MATH  Google Scholar 

  14. Rousseeuw, P.J.: Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20, 53–65 (1987). https://doi.org/10.1016/0377-0427(87)90125-7, https://www.sciencedirect.com/science/article/pii/0377042787901257

  15. Shaikhislamov, D., Voevodin, V.: Solving the problem of detecting similar supercomputer applications using machine learning methods. In: Sokolinsky, L., Zymbler, M. (eds.) PCT 2020. CCIS, vol. 1263, pp. 46–57. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-55326-5_4

    Chapter  Google Scholar 

  16. Shin, M., Park, G., Park, C.Y., Lee, J., Kim, M.: Application-specific feature selection and clustering approach with HPC system profiling data. J. Supercomput. 77(7), 6817–6831 (2021). https://doi.org/10.1007/s11227-020-03533-2

    Article  Google Scholar 

  17. Stefanov, K., Voevodin, V., Zhumatiy, S., Voevodin, V.: Dynamically reconfigurable distributed modular monitoring system for supercomputers (dimmon). In: 4th International Young Scientist Conference on Computational Science. Procedia Computer Science, vol. 66, pp. 625–634. Elsevier B.V Netherlands (2015). https://doi.org/10.1016/j.procs.2015.11.071

  18. Steinbach, M., Karypis, G., Kumar, V.: A comparison of document clustering techniques. In: KDD Workshop on Text Mining (2000)

    Google Scholar 

  19. Tuncer, O., et al.: Diagnosing performance variations in HPC applications using machine learning. In: Kunkel, J.M., Yokota, R., Balaji, P., Keyes, D. (eds.) ISC High Performance 2017. LNCS, vol. 10266, pp. 355–373. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-58667-0_19

    Chapter  Google Scholar 

  20. Voevodin, V.V., et al.: supercomputer lomonosov-2: large scale, deep monitoring and fine analytics for the user community. Supercomput. Front. Innov. 6(2), 4–11 (2019). https://doi.org/10.14529/jsfi190201

  21. Duračík, M., Krsak, E., Hrkút, P.: Scalable source code plagiarism detection using source code vectors clustering, pp. 499–502, November 2018. https://doi.org/10.1109/ICSESS.2018.8663708

Download references

Acknowledgements

The results described in this paper were achieved at Lomonosov Moscow State University with the financial support of the Russian Science Foundation, agreement No. 21-71-30003.

Author information

Authors and Affiliations

  1. Lomonosov Moscow State University, Moscow, Russia

    Denis Shaikhislamov & Vadim Voevodin

Authors
  1. Denis Shaikhislamov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vadim Voevodin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Denis Shaikhislamov .

Editor information

Editors and Affiliations

  1. Czestochowa University of Technology, Czestochowa, Poland

    Roman Wyrzykowski

  2. University of Tennessee, Knoxville, TN, USA

    Jack Dongarra

  3. University of Southern California, Marina del Rey, CA, USA

    Ewa Deelman

  4. Czestochowa University of Technology, Czestochowa, Poland

    Konrad Karczewski

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shaikhislamov, D., Voevodin, V. (2023). Smart Clustering of HPC Applications Using Similar Job Detection Methods. In: Wyrzykowski, R., Dongarra, J., Deelman, E., Karczewski, K. (eds) Parallel Processing and Applied Mathematics. PPAM 2022. Lecture Notes in Computer Science, vol 13826. Springer, Cham. https://doi.org/10.1007/978-3-031-30442-2_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-30442-2_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-30441-5

  • Online ISBN: 978-3-031-30442-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation