Original contribution
Nonlinear neural networks: Principles, mechanisms, and architectures,☆☆

https://doi.org/10.1016/0893-6080(88)90021-4Get rights and content

Abstract

An historical discussion is provided of the intellectual trends that caused nineteenth century interdisciplinary studies of physics and psychobiology by leading scientists such as Helmholtz, Maxwell, and Mach to splinter into separate twentieth-century scientific movements. The nonlinear, nonstationary, and nonlocal nature of behavioral and brain data are emphasized. Three sources of contemporary neural network research—the binary, linear, and continuous-nonlinear models—are noted. The remainder of the article describes results about continuous-nonlinear models: Many models of content-addressable memory are shown to be special cases of the Cohen-Grossberg model and global Liapunov function, including the additive, brain-state-in-a-box, McCulloch-Pitts, Boltzmann machine, Hartline-Ratliff-Miller, shunting, masking field, bidirectional associative memory, Volterra-Lotka, Gilpin-Ayala, and Eigen-Schuster models. A Liapunov functional method is described for proving global limit or oscillation theorems for nonlinear competitive systems when their decision schemes are globally consistent or inconsistent, respectively. The former case is illustrated by a model of a globally stable economic market, and the latter case is illustrated by a model of the voting paradox. Key properties of shunting competitive feedback networks are summarized, including the role of sigmoid signalling, automatic gain control, competitive choice and quantization, tunable filtering, total activity normalization, and noise suppression in pattern transformation and memory storage applications. Connections to models of competitive learning, vector quantization, and categorical perception are noted. Adaptive resonance theory (ART) models for self-stabilizing adaptive pattern recognition in response to complex real-time nonstationary input environments are compared with off-line models such as autoassociators, the Boltzmann machine, and back propagation. Special attention is paid to the stability and capacity of these models, and to the role of top-down expectations and attentional processing in the active regulation of both learning and fast information processing. Models whose performance and learning are regulated by internal gating and matching signals, or by external environmentally generated error signals, are contrasted with models whose learning is regulated by external teacher signals that have no analog in natural real-time environments. Examples from sensory-motor control of adaptive vector encoders, adaptive coordinate transformations, adaptive gain control by visual error signals, and automatic generation of synchronous multijoint movement trajectories illustrate the former model types. Internal matching processes are shown capable of discovering several different types of invariant environmental properties. These include ART mechanisms which discover recognition invariants, adaptive vector encoder mechanisms which discover movement invariants, and autoreceptive associative mechanisms which discover invariants of self-regulating target position maps.

References (207)

  • D.H. Ackley et al.

    A learning algorithm for Boltzmann machines

    Cognitive Science

    (1985)
  • J.S. Albus

    A theory of cerebellar function

    Mathematical Biosciences

    (1971)
  • S. Amari et al.

    Mathematical theory on formation of category detecting nerve cells

    Biological Cybernetics

    (1978)
  • D.J. Amit et al.

    Information storage in neural networks with low levels of activity

    Physical Review A

    (1987)
  • J.A. Anderson

    A memory model utilizing spatial correlation functions

    Kybernetik

    (1968)
  • J.A. Anderson

    Cognitive and psychological computation with neural models

    IEEE Transactions

    (1983)
  • J.A. Anderson et al.

    Distinctive features, categorical perception, and probability learning: Some applications of a neural model

    Psychological Review

    (1977)
  • L.E. Arend et al.

    Difference information in brightness perception

    Perception and Psychophysics

    (1971)
  • Banquet, J.-P., & Grossberg, S. (in press). Probing cognitive processes through the structure of event-related...
  • J. Beck

    Surface color perception

    (1972)
  • E.L. Bienenstock et al.

    Theory for the development of neuron selectivity: Orientation specificity and binocular interaction in visual cortex

    Journal of Neuroscience

    (1982)
  • E.G. Boring

    A history of experimental psychology

    (1950)
  • G.S. Brindley

    The use made by the cerebellum of the information that it receives from sense organs

    International Brain Research Organization Bulletin

    (1964)
  • Bullock, D., & Grossberg, S. (in press-a). Neural dynamics of planned arm movements: Emergent invariants and...
  • Bullock, D., & Grossberg, S. (in press-b). Neuromuscular realization of planned trajectories: Adaptive and automatic...
  • E.R. Caianiello

    Outline of a theory of thought and thinking machines

    Journal of Theoretical Biology

    (1961)
  • L. Campbell et al.

    The life of James Clerk Maxwell

    (1882)
  • G.A. Carpenter

    A geometric approach to singular perturbation problems with applications to nerve impulse equations

    Journal of Differential Equations

    (1977)
  • G.A. Carpenter

    Periodic solutions of nerve impulse equations

    Journal of Mathematical Analysis and Applications

    (1977)
  • G.A. Carpenter

    Bursting phenomena in excitable membranes

    SIAM Journal on Applied Mathematics

    (1979)
  • G.A. Carpenter

    Normal and abnormal signal patterns in nerve cells

  • G.A. Carpenter et al.

    Category learning and adaptive pattern recognition: A neural network model

  • G.A. Carpenter et al.

    A massively parallel architecture for a self-organizing neural pattern recognition machine

    Computer Vision, Graphics, and Image Processing

    (1987)
  • G.A. Carpenter et al.

    Associative learning, adaptive pattern recognition, and cooperative-competitive decision making by neural networks

  • Carpenter, G. A., & Grossberg, S. (in press). ART 2: Self-organization of stable category recognition codes for analog...
  • M.A. Cohen et al.

    Absolute stability of global pattern formation and parallel memory storage by competitive neural networks

    IEEE Transactions

    (1983)
  • M.A. Cohen et al.

    Neural dynamics of brightness perception: Features, boundaries, diffusion, and resonance

    Perception and Psychophysics

    (1984)
  • M. Cohen et al.

    Neural dynamics of speech and language coding: Developmental programs, perceptual grouping, and competition for short term memory

    Human Neurobiology

    (1986)
  • M. Cohen et al.

    Masking fields: A massively parallel neural architecture for learning, recognizing, and predicting multiple groupings of patterned data

    Applied Optics

    (1987)
  • T.N. Cornsweet

    Visual perception

    (1970)
  • L.X. Cubeddu et al.

    Frequency-dependent effects of neuronal uptake inhibitors on the autoreceptor-mediated modulation of dopamine and acetylcholine release from the rabbit striatum

    Journal of Pharmacology and Experimental Therapeutics

    (1983)
  • M.L. Dubocovich et al.

    Modulation of the stimulation-evoked release of 3H-dopamine through activation of dopamine autoreceptors of the D-2 subtype in the isolated rabbit retina

  • R.O. Duda et al.

    Pattern classification and scene analysis

    (1973)
  • M. Eigen et al.

    The hypercycle: A principle of natural self-organization, B: The abstract hypercycle

    Naturwissenschaften

    (1978)
  • S.A. Ellias et al.

    Pattern formation, contrast control, and oscillations in the short term memory of shunting on-center off-surround networks

    Biological Cybernetics

    (1975)
  • J.H. Elman et al.

    Perceptual switching in bilinguals

    Journal of the Acoustical Society of America

    (1977)
  • G.B. Ermentrout et al.

    Temporal oscillations in neuronal nets

    Journal of Mathematical Biology

    (1979)
  • G.B. Ermentrout et al.

    Large scale spatially organized activity in neural nets

    SIAM Journal on Applied Mathematics

    (1980)
  • J.A. Feldman et al.

    Connectionist models and their properties

    Cognitive Science

    (1982)
  • W.J. Freeman

    Mass action in the nervous system

    (1975)
  • W.J. Freeman

    EEG analysis gives model of neuronal template-matching mechanism for sensory search with olfactory bulb

    Biological Cybernetics

    (1979)
  • M. Fujita

    Simulation of adaptive modification of the vestibulo-ocular reflex with an adaptive filter model of the cerebellum

    Biological Cybernetics

    (1982)
  • M. Fujita

    Adaptive filter model of the cerebellum

    Biological Cybernetics

    (1982)
  • S. Geman

    Stochastic relaxation methods for image restoration and expert systems

  • S. Geman

    Stochastic relaxation methods for image restoration and expert systems

  • S. Geman et al.

    Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (1984)
  • M.E. Gilpin et al.

    Global models of growth and competition

  • R.T. Glazebrook

    James Clerk Maxwell and modern physics

    (1905)
  • Cited by (1341)

    View all citing articles on Scopus

    Based upon a lecture given on October 7, 1986 at the NSF meeting on Neural Networks and Neuromorphic System, Woburn, Massachusetts.

    ☆☆

    Supported in part by the Air Force Office of Scientific Research (AFOSR F49620-86-C-0037 and AFOSR F49620-87-C-0018) and the National Science Foundation (NSF IRI-84-17756).

    View full text