Summary
This paper discusses a new automatic differentiation (AD) system that correctly and automatically accepts nested and dynamic use of the AD operators, without any manual intervention. The system is based on a new formulation of AD as highly generalized first-class citizens in a λ-calculus, which is briefly described. Because the λ-calculus is the basis for modern programming-language implementation techniques, integration of AD into the λ-calculus allows AD to be integrated into an aggressive compiler. We exhibit a research compiler which does this integration. Using novel analysis techniques, it accepts source code involving free use of a first-class forward AD operator and generates object code which attains numerical performance comparable to, or better than, the most aggressive existing AD systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Church, A.: The Calculi of Lambda Conversion. Princeton University Press, Princeton, NJ (1941)
Hascoët, L., Pascual, V.: TAPENADE 2.1 user’s guide. Rapport technique 300, INRIA, Sophia Antipolis (2004). URL http://www.inria.fr/rrrt/rt-0300.html
Hughes, J.: Why functional programming matters. The Computer Journal 32(2), 98–107 (1989). URL http://www.md.chalmers.se/~rjmh/Papers/whyfp.html
Johnsson, T.: Lambda lifting: Transforming programs to recursive equations. In: Functional Programming Languages and Computer Architecture. Springer-Verlag, Nancy, France (1985)
Nielson, F., Nielson, H.R., Hankin, C.: Principles of Program Analysis. Springer-Verlag, New York (1999)
Pearlmutter, B.A., Siskind, J.M.: Reverse-mode AD in a functional framework: Lambda the ultimate backpropagator. ACM Trans. on Programming Languages and Systems (2008). In press
Siskind, J.M., Pearlmutter, B.A.: First-class nonstandard interpretations by opening closures. In: Proceedings of the 2007 Symposium on Principles of Programming Languages, pp. 71–6. Nice, France (2007)
Siskind, J.M., Pearlmutter, B.A.: Nesting forward-mode AD in a functional framework. Higher-Order and Symbolic Computation (2008). To appear
Siskind, J.M., Pearlmutter, B.A.: Putting the automatic back into AD: Part I, What’s wrong. Tech. Rep. TR-ECE-08-02, School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA (2008). URL ftp://ftp.ecn.purdue.edu/qobi/TR-ECE-08-02.pdf
Siskind, J.M., Pearlmutter, B.A.: Using polyvariant union-free flow analysis to compile a higher-order functional-programming language with a first-class derivative operator to efficient Fortran-like code. Tech. Rep. TR-ECE-08-01, School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA (2008). URL http://docs.lib.purdue.edu/ecetr/367/
Wadler, P.L.: Comprehending monads. In: Proceedings of the 1990 ACM Conference on Lisp and Functional Programming, pp. 61–78. Nice, France (1990)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Pearlmutter, B.A., Siskind, J.M. (2008). Using Programming Language Theory to Make Automatic Differentiation Sound and Efficient. In: Bischof, C.H., Bücker, H.M., Hovland, P., Naumann, U., Utke, J. (eds) Advances in Automatic Differentiation. Lecture Notes in Computational Science and Engineering, vol 64. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-68942-3_8
Download citation
DOI: https://doi.org/10.1007/978-3-540-68942-3_8
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-68935-5
Online ISBN: 978-3-540-68942-3
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)