Abstract
The term GreenAI refers to a novel approach to Deep Learning, that is more aware of the ecological impact and the computational efficiency of its methods. The promoters of GreenAI suggested the use of Floating Point Operations (FLOPs) as a measure of the computational cost of Neural Networks; however, that measure does not correlate well with the energy consumption of hardware equipped with massively parallel processing units like GPUs or TPUs. In this article, we propose a simple refinement of the formula used to compute floating point operations for convolutional layers, called \(\alpha \)-FLOPs, explaining and correcting the traditional discrepancy with respect to different layers, and closer to reality. The notion of \(\alpha \)-FLOPs relies on the crucial insight that, in case of inputs with multiple dimensions, there is no reason to believe that the speedup offered by parallelism will be uniform along all different axes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amdahl, G.M.: Validity of the single processor approach to achieving large-scale computing capabilities. In: AFIPS Conference Proceedings, vol. 30, pp. 483–485 (1967)
Anthony, L.F.W., Kanding, B., Selvan, R.: Carbontracker: tracking and predicting the carbon footprint of training deep learning models. CoRR, abs/2007.03051 (2020)
Arm cortex-r8 mpcore processor (2018). https://developer.arm.com/documentation/100400/0002/floating-point-unit-programmers-model/instruction-throughput-and-latency?lang=en. Accessed 26 Apr 2021
Asperti, A.: Computational complexity via finite types. ACM Trans. Comput. Log. 16(3), 26:1–26:25 (2015)
Asperti, A., Evangelista, D., Piccolomini, E.L.: A survey on variational autoencoders from a green AI perspective. SN Comput. Sci. 2(4), 301 (2021)
Blackford, L.S., et al.: An updated set of basic linear algebra subprograms (BLAS). ACM Trans. Math. Softw. 28(2), 135–151 (2002)
Cao, Q., Balasubramanian, A., Balasubramanian, N.: Towards accurate and reliable energy measurement of NLP models. CoRR, abs/2010.05248 (2020)
clBLAS. http://clmathlibraries.github.io/clBLAS/. Accessed 26 Apr 2021
AMD Corporation. Software optimization guide for AMD family 19h processors (pub), November 2020. https://www.amd.com/system/files/TechDocs/56665.zip. Accessed 25 Apr 2021
Intel Corporation. Intel® Xeon scalable processor® instruction throughput and latency, August 2017. https://software.intel.com/content/dam/develop/public/us/en/documents/intel-xeon-scalable-processor-throughput-latency.pdf. Accessed 25 Apr 2021
cuBLAS. https://docs.nvidia.com/cuda/cublas/index.html. Accessed 26 Apr 2021
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, 2–7 June 2019, vol. 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics (2019)
Gadou, M., Banerjee, T., Arunachalam, M., Ranka, S.: Multiobjective evaluation and optimization of CMT-bone on multiple CPU/GPU systems. Sustain. Comput.: Inform. Syst. 22, 259–271 (2019)
Gordon, A., et al.: Morphnet: fast & simple resource-constrained structure learning of deep networks. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018, pp. 1586–1595. IEEE Computer Society (2018)
Gupta, U., et al.: Chasing carbon: the elusive environmental footprint of computing. CoRR, abs/2011.02839 (2020)
Gustafson, J.L.: Reevaluating Amdahl’s law. Commun. ACM 31(5), 532–533 (1988)
Jeon, Y., Kim, J.: Constructing fast network through deconstruction of convolution. In: Bengio, S., Wallach, H.M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3–8 December 2018, Montréal, Canada, pp. 5955–5965 (2018)
Jhurani, C., Mullowney, P.: A GEMM interface and implementation on NVIDIA GPUs for multiple small matrices. J. Parallel Distrib. Comput. 75, 133–140 (2015)
Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, 30 April–3 May 2018, Conference Track Proceedings. OpenReview.net (2018)
Lacoste, A., Luccioni, A., Schmidt, V., Dandres, T.: Quantifying the carbon emissions of machine learning. CoRR, abs/1910.09700 (2019)
MacAvaney, S., Nardini, F.M., Perego, R., Tonellotto, N., Goharian, N., Frieder, O.: Efficient document re-ranking for transformers by precomputing term representations. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2020, Virtual Event, China, 25–30 July 2020, pp. 49–58. ACM (2020)
Molchanov, P., Tyree, S., Karras, T., Aila, T., Kautz, J.: Pruning convolutional neural networks for resource efficient inference. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, 24–26 April 2017, Conference Track Proceedings. OpenReview.net (2017)
Patterson, D.A., Hennessy, J.L.: Computer Architecture: A Quantitative Approach. Morgan Kaufmann, Burlington (2017)
Rodrigues, R., Annamalai, A., Koren, I., Kundu, S.: A study on the use of performance counters to estimate power in microprocessors. IEEE Trans. Circuits Syst. II Express Briefs 60(12), 882–886 (2013)
Schwartz, R., Dodge, J., Smith, N.A., Etzioni, O.: Green AI. Commun. ACM 63(12), 54–63 (2020)
Strubell, E., Ganesh, A., McCallum, A.: Energy and policy considerations for modern deep learning research. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, 7–12 February 2020, pp. 13693–13696. AAAI Press (2020)
Tan, M., Le, Q.V.: Efficientnet: rethinking model scaling for convolutional neural networks. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9–15 June 2019, Long Beach, California, USA, Proceedings of Machine Learning Research, vol. 97, pp. 6105–6114. PMLR (2019)
van Wynsberghe, A.: Sustainable AI: AI for sustainability and the sustainability of AI. AI Ethics 1, 213–218 (2021). https://doi.org/10.1007/s43681-021-00043-6
Veniat, T., Denoyer, L.: Learning time/memory-efficient deep architectures with budgeted super networks. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018, pp. 3492–3500. IEEE Computer Society (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Asperti, A., Evangelista, D., Marzolla, M. (2022). Dissecting FLOPs Along Input Dimensions for GreenAI Cost Estimations. In: Nicosia, G., et al. Machine Learning, Optimization, and Data Science. LOD 2021. Lecture Notes in Computer Science(), vol 13164. Springer, Cham. https://doi.org/10.1007/978-3-030-95470-3_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-95470-3_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-95469-7
Online ISBN: 978-3-030-95470-3
eBook Packages: Computer ScienceComputer Science (R0)