Abstract
The problem of parametrizing single hidden layer scalar neural networks with continuous activation functions is investigated. A connection is drawn between realization theory for linear dynamical systems, rational functions, and neural networks that appears to be new. A result of this connection is a general parametrization of such neural networks in terms of strictly proper rational functions. Some existence and uniqueness results are derived. Jordan decompositions are developed, which show how the general form can be expressed in terms of a sum of canonical second order sections. The parametrization may be useful for studying learning algorithms.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
F. Albertini, E. D. Sontag, and V. Maillot, Uniqueness of weights for neural networks,Artificial Neural Networks for Speech and Vision (R. Mammone, ed.), Chapman and Hall, London, 1993, 115–125.
A. C. Antoulas and B. D. O. Anderson, On the scalar rational interpolation problem,IMA Journal of Mathematical Control and Information,3 (1986), 61–88.
R. B. Barrar and H. L. Loeb, On extended varisolvent families,Journal D'Analyse Mathématique,26 (1973), 243–254.
K. L. Blackmore, R. C. Williamson, and I. M. Y. Mareels, Local minima and attractors at infinity of gradient descent learning algorithms,Journal of Mathematical Systems, Estimation and Control (1995), to appear.
G. Cybenko, Approximation by superpositions of a sigmoidal function,Mathematics of Control, Signals,and Systems,2 (1989), 303–314.
P. J. Davis,Interpolation and Approximation, Dover, New York, 1975.
W. F. Donoghue, Jr,Monotone Matrix Functions and Analytic Continuation, Springer-Verlag, Berlin, 1974.
K.-I. Funahashi, On the approximate realization of continuous mappings by neural networks,Neural Networks,2 (1989), 183–192.
W. Gautschi, A survey of Gauss-Christoffel quadrature formulae,E. B. Christoffel, The Influence of His Work on Mathematics and the Physical Sciences (P. Butzer and F. Fehér, eds.), Birkhäuser, Basel, 1981, 72–147.
G. M. Georgiou and C. Koutsougeras, Complex domain backpropagation,IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing,39 (1992), 330–334.
W. B. Gragg and A. Lindquist, On the partial realization problem,Linear Algebra and its Applications,50 (1983), 277–319.
G. H. Hardy and E. M. Wright,An Introduction to the Theory of Numbers, Oxford University Press, Oxford, 1979.
U. Helmke, Waring's problem for binary forms,Journal of Pure and Applied Algebra,80 (1992), 29–45.
K. Hornik, M. Stinchcombe, and H. White, Multilayer feedforward networks are universal approximators,Neural Networks,2 (1989), 359–366.
T. Kailath,Linear Systems, Prentice-Hall, Englewood Cliffs, 1980.
R. E. Kalman, On partial realizations, transfer functions, and canonical forms,Acta Polytechnica Scandinavica,31 (1979), 9–32.
R. E. Kalman, P. L. Falb, and M. A. Arbib,Topics in Mathematical System Theory, McGraw-Hill, New York, 1969.
T. Kato,Perturbation Theory for Linear Operators, Springer-Verlag, Berlin, 1966.
H. Leung and S. Haykin, The complex backpropagation algorithm,IEEE Transactions on Signal Processing,39 (1991), 2101–2104.
C. Martin and M. Stamp, A note on the error in Gaussian quadrature, Preprint, 1992.
R. A. Silverman,Introductory Complex Analysis, Dover, New York, 1972.
E. D. Sontag,Mathematical Control Theory: Deterministic Finite Dimensional Systems, Springer-Verlag, New York, 1990.
E. D. Sontag and H. J. Sussmann, Backpropagation can give rise to spurious local minima even for networks without hidden layers,Complex Systems,3 (1989), 91–106.
H. J. Sussmann, Uniqueness of weights for minimal feedforward nets with a given inputoutput map,Neural Networks,5 (1992), 589–593.
J. J. Sylvester, An essay on canonical forms, supplement to a sketch of a memoir on elimination,Collected Mathematical Papers I, paper 34, 1851.
R. C. Williamson and U. Helmke, Existence and uniqueness results for neural network approximations,IEEE Transactions on Neural Networks,6 (1995), 2–13.
Author information
Authors and Affiliations
Additional information
This work was supported by the Australian Research Council, the Australian Telecommunications and Electronics Research Board, and the Boeing Commencai Aircraft Company (thanks to John Moore).
Rights and permissions
About this article
Cite this article
Helmke, U., Williamson, R.C. Neural networks, rational functions, and realization theory. Math. Control Signal Systems 8, 27–49 (1995). https://doi.org/10.1007/BF01212365
Received:
Revised:
Issue Date:
DOI: https://doi.org/10.1007/BF01212365