Copyright © 2000 Elsevier Science Ltd. All rights reserved.
Contributed article
Information complexity of neural networks
Received 9 February 1999;
Abstract
This paper studies the question of lower bounds on the number of neurons and examples necessary to program a given task into feedforward neural networks. We introduce the notion of information complexity of a network to complement that of neural complexity. Neural complexity deals with lower bounds for neural resources (numbers of neurons) needed by a network to perform a given task within a given tolerance. Information complexity measures lower bounds for the information (i.e. number of examples) needed about the desired input–output function. We study the interaction of the two complexities, and so lower bounds for the complexity of building and then programming feed-forward nets for given tasks. We show something unexpected a priori—the interaction of the two can be simply bounded, so that they can be studied essentially independently. We construct radial basis function (RBF) algorithms of order n3 that are information-optimal, and give example applications.
Author Keywords: Feedforward neural network; Complexity; Information complexity; Neural complexity; Radial basis functions; RBF networks; Learning
Article Outline
- 1. Introduction
- 2. Definitions and main result
- 3. Examples
- 4. Neural complexity (k=∞)
- Information complexity (n=∞)Question 2. Given an unknown function f in some class, what is the smallest number k of examples
for which it is (theoretically) possible to estimate f within error ε (assuming unlimited access to hidden units)?
- Interaction of information and neural complexitiesQuestion 3. Given information about f with k examples, what is the best network approximating f which uses at most n neurons in the hidden layer?
- Acknowledgements
- References
Corresponding author. Tel.: +1-617-353-2560; fax: +1-617-353-8100; email: mkon@bu.edu






E-mail Article
Add to my Quick Links

Cited By in Scopus (7)






