Todd Mytkowicz
Matthias Hauswirth
Peter F. Sweeney
ABSTRACT
Performance analysts profile their programs to find methods that are worth optimizing: the "hot" methods. This paper shows that four commonly-used Java profilers (xprof , hprof , jprofile, and yourkit) often disagree on the identity of the hot methods. If two profilers disagree, at least one must be incorrect. Thus, there is a good chance that a profiler will mislead a performance analyst into wasting time optimizing a cold method with little or no performance improvement.
This paper uses causality analysis to evaluate profilers and to gain insight into the source of their incorrectness. It shows that these profilers all violate a fundamental requirement for sampling based profilers: to be correct, a sampling-based profilermust collect samples randomly.
We show that a proof-of-concept profiler, which collects samples randomly, does not suffer from the above problems. Specifically, we show, using a number of case studies, that our profiler correctly identifies methods that are important to optimize; in some cases other profilers report that these methods are cold and thus not worth optimizing.
- B. Alpern, C. R. Attanasio, J. J. Barton, M. G. Burke, P. Cheng, J.-D. Choi, A. Cocchi, S. J. Fink, D. Grove, M. Hind, S. F. Hummel, D. Lieber, V. Litvinov, M. F. Mergen, T. Ngo, J. R. Russell, V. Sarkar, M. J. Serrano, J. C. Shepherd, S. E. Smith, V. C. Sreedhar, H. Srinivasan, and J. Whaley. The Jalapeño virtual machine. IBM Systems Journal, 39(1):211--238, February 2000. Google Scholar
Digital Library
- M. Arnold and D. Grove. Collecting and exploiting high-accuracy call graph profiles in virtual machines. In Proc. of Int'l Symposium on Code Generation and Optimization, pages 51--62, Los Alamos, CA, March 2005. IEEE Computer Society. Google ScholarDigital Library
- S. M. Blackburn, R. Garner, C. Hoffman, A. M. Khan, K. S. McKinley, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J. E. B. Moss, A. Phansalkar, D. Stefanović, T. VanDrunen, D. von Dincklage, and B. Wiedermann. The DaCapo benchmarks: Java benchmarking development and analysis. In Proc. of ACM SIGPLAN Conf. on Object-Oriented Programing, Systems, Languages, and Applications, pages 169--190, Portland, OR, Oct. 2006. ACM. Google ScholarDigital Library
- S. M. Blackburn, P. Cheng, and K. S. Mckinley. Myths and realities: The performance impact of garbage collection. In Proc. of ACM SIMETRICS Conf. onMeasurement andModeling Computer Systems, pages 25--36, New York, NY, Jan. 2004. ACM. Google ScholarDigital Library
- D. Buytaert, A. Georges, M. Hind, M. Arnold, L. Eeckhout, and K. De Bosschere. Using HPM-sampling to drive dynamic compilation. In Proc. of ACM SIGPLAN Conf. on Object-Oriented Programing, Systems, Languages, and Applications, pages 553--568, Montreal, Canada, Oct. 2007. ACM. Google ScholarDigital Library
- Dehao Chen, Neil Vachharajani, and Robert Hundt. Taming hardware event samples for fdo compilation. International Symposium on Code Generation and Optimization (CGO), 2010. Google ScholarDigital Library
- A. Diwan, E. Moss, and R. Hudson. Compiler support for garbage collection in a statically typed language. SIGPLAN Not., 27(7):273--282, 1992. Google ScholarDigital Library
- M. Dmitriev. Selective profiling of Java applications using dynamic bytecode instrumentation. In Proc. of IEEE Int'l Symposium on Performance Analysis of Systems and Software, pages 141--150, Washington, DC, March 2004. IEEE. Google ScholarDigital Library
- E. Duesterwald and V. Bala. Software profiling for hot path prediction: less is more. SIGPLAN Not., 35(11):202--211, 2000. Google ScholarDigital Library
- Eclipse: Open source java profiler v4.6.1. http://www.eclipse.org/tptp/.Google Scholar
- Ej technologies: Commercial java profiler. http://www.ejtechnologies.com/products/jprofiler/overview.html.Google Scholar
- A. Georges, D. Buytaert, and L. Eeckhout. Statistically rigorous Java performance evaluation. In Proc. of ACM SIGPLAN Conf. on Objectoriented Programming, Systems, Languages and Applications, pages 57--76, Montreal, Canada, Oct. 2007. ACM. Google ScholarDigital Library
- S. L. Graham, P. B. Kessler, and M. K. Mckusick. Gprof: A call graph execution profiler. In Proc. of ACM SIGPLAN Symposium on Compiler Construction, pages 120--126, Boston, Mass., 1982. ACM. Google ScholarDigital Library
- D. Gu, C. Verbrugge, and E. Gagnon. Code layout as a source of noise in JVM performance. Studia Informatica Universalis, pages 83--99, 2004.Google Scholar
- R. Hegger, H. Kantz, and T. Schreiber. Practical implementation of nonlinear time series methods: The TISEAN package. Chaos, 9(2):413--435, 1999.Google ScholarCross Ref
- Sam Kash Kachigan. Statistical Analysis: An Interdisciplinary Introduction to Univariate & Multivariate Methods. Radius Press, 1986.Google Scholar
- S. Mccanne and C. Torek. A randomized sampling clock for CPU utilization estimation and code profiling. In Proc. of the Winter USENIX Conf., pages 387--394, San Diego, CA, Jan. 1993.Google Scholar
- T. Moseley, A. Shye, V.J. Reddi, D. Grunwald, and R. Peri. Shadow profiling: Hiding instrumentation costs with parallelism. In Proc. of Int'l Symposium on Code Generation and Optimization, pages 198--208, Washington, DC, March 2007. IEEE Computer Society. Google ScholarDigital Library
- T. Mytkowicz, A. Diwan, M. Hauswirth, and P. F. Sweeney. Producing wrong data without doing anything obviously wrong! In Proc. of Int'l Conf. on Architectural Support for Programming Languages and Operating Systems, pages 265--276, Washington, DC, March 2009. ACM. Google ScholarDigital Library
- Netbeans: Open source java profiler. v6.7. http://profiler.netbeans.org/.Google Scholar
- J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, 1st edition, 2000. Google ScholarDigital Library
- S. Rubin, R. Bodík, and T. Chilimbi. An efficient profile-analysis framework for data-layout optimizations. SIGPLAN Not., 37(1):140--153, 2002. Google ScholarDigital Library
- hprof: an open source java profiler. http://java.sun.com/developer/technicalArticles/Programming/HPROF.htmlGoogle Scholar
- xprof: Internal profiler for hotspot. http://java.sun.com/docs/books/performance/1st edition/html/JPAppHotspot.fm.html.Google Scholar
- J. Whaley. A portable sampling-based profiler for java virtual machines. In Proc. of Conf. on Java Grande, pages 78--87, New York, NY, 2000. ACM. Google ScholarDigital Library
- Yourkit, llc: Commercial java profiler. http://www.yourkit.comGoogle Scholar
Index Terms
- Evaluating the accuracy of Java profilers
Recommendations
Evaluating the accuracy of Java profilers
PLDI '10Performance analysts profile their programs to find methods that are worth optimizing: the "hot" methods. This paper shows that four commonly-used Java profilers (xprof , hprof , jprofile, and yourkit) often disagree on the identity of the hot methods. ...
Don’t Trust Your Profiler: An Empirical Study on the Precision and Accuracy of Java Profilers
MPLR 2023: Proceedings of the 20th ACM SIGPLAN International Conference on Managed Programming Languages and RuntimesTo identify optimisation opportunities, Java developers often use sampling profilers that attribute a percentage of run time to the methods of a program. Even so these profilers use sampling, are probabilistic in nature, and may suffer for ...
Don’t Trust Your Profiler: An Empirical Study on the Precision and Accuracy of Java Profilers (Poster Abstract)
MPLR 2023: Proceedings of the 20th ACM SIGPLAN International Conference on Managed Programming Languages and RuntimesTo identify optimisation opportunities, Java developers often use sampling profilers that attribute a percentage of run time to the methods of a program. Even so these profilers use sampling, are probabilistic in nature, and may suffer for ...
Comments