Skip to main content
SpringerLink
Log in

The dynamics of sparse random networks

  • Published:
Biological Cybernetics Aims and scope Submit manuscript

Abstract

Recurrent neural networks with full symmetric connectivity have been extensively studied as associative memories and pattern recognition devices. However, there is considerable evidence that sparse, asymmetrically connected, mainly excitatory networks with broadly directed inhibition are more consistent with biological reality. In this paper, we use the technique of return maps to study the dynamics of random networks with sparse, asymmetric connectivity and nonspecific inhibition. These networks show three qualitatively different kinds of behavior: fixed points, cycles of low period, and extremely long cycles verging on aperiodicity. Using statistical arguments, we relate these behaviors to network parameters and present empirical evidence for the accuracy of this statistical model. The model, in turn, leads to methods for controlling the level of activity in networks. Studying random, untrained networks provides an understanding of the intrinsic dynamics of these systems. Such dynamics could provide a substrate for the much more complex behavior shown when synaptic modification is allowed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Amaral DG, Ishizuka N, Claiborne B (1990) Neurons, numbers and the hippocampal networks. Prog Brain Res, 83:1–11

    Google Scholar 

  • Amari S (1971) Characteristics of randomly connected threshold-element networks and network systems. Proc IEEE 59:35–47

    Google Scholar 

  • Amari S (1972) Learning patterns and pattern sequences by self-organizing nets of threshold elements. IEEE Trans Comput C-21:1197–1206

    Google Scholar 

  • Amari S (1974) A method of statistical neurodynamics. Kybernetik 14:201–215

    Google Scholar 

  • Amari S (1989) Characteristics of sparsely encoded associative memory. Neural Networks 2:451–457

    Google Scholar 

  • Amari S, Maginu K (1988) Statistical neurodynamics of associative memory. Neural Networks 1:63–73

    Google Scholar 

  • Amari S, Yoshida K, Kanatani K (1977) A mathematical foundation for statistical neurodynamics. SIAM J Appl Math 33:95–126

    Google Scholar 

  • Ambrose-Ingerson J, Granger, R, Lynch G (1990) Simulation of paleocortex performs hierarchical clustering. Science 247:1344–1348

    Google Scholar 

  • Bauer K, Krey U (1990) On learning and recognition of temporal sequences of correlated patterns. Z Phys B Condensed Matter 79:461–475

    Google Scholar 

  • Buhmann J, Schulten K (1988) Storing sequences of biased patterns in neural networks with stochastic dynamics. In: Eckmiller R, Malsburg C von der (eds) Neural computers. (NATO ASI series, vol F41) Springer, Berlin Heidelberg New York, pp 231–242

    Google Scholar 

  • Cohen MA, Grossberg S (1983) Absolute stability of global pattern formation and parallel memory storage by competitive neural networks. IEEE Trans Syst Man Cybern 13:815–826

    Google Scholar 

  • Crisanti A, Sompolinsky H (1987) Dynamics of spin systems with randomly asymmetric bonds: Langevin dynamics and a spherical model. Phys Rev A36:4922–4939

    Google Scholar 

  • Derrida B, Pomeau Y (1986) Random networks of automata: a simple annealed approximation. Europhys Lett 1:45–49

    Google Scholar 

  • Derrida B, Gardner E, Zippelius A (1987) An exactly solvable asymmetric neural network model. Europhys Lett 4:167–173

    Google Scholar 

  • Eichenbaum H, Otto T (1992) The hippocampus — what does it do? Behav Neural Biol 57:2–36

    Google Scholar 

  • Fukushima K (1973) A model of associative memory in the brain. Kybernetik 12:58–63

    Google Scholar 

  • Furman GG (1965) Comparison of models for subtractive and shunting lateral-inhibition in receptor neurons. Kybernetik 2:257–274

    Google Scholar 

  • Gardner-Medwin AR (1976) The recall of events through the learning of associations between their parts. Proc R Soc Lond [Biol] 194:375–402

    Google Scholar 

  • Gardner-Medwin AR (1989) Doubly modifiable synapses: a model of short and long term auto-associative memory. Proc R Soc Lond [Biol] 238:137–154

    Google Scholar 

  • Gibson WG, Robinson J (1992) Statistical analysis of the dynamics of a sparse associative memory. Neural Networks 5:645–661

    Google Scholar 

  • Grossberg S (1988) Nonlinear neural networks: principles, mechanisms, and architectures. Neural Networks 1:17–61

    Google Scholar 

  • Gutfreund H, Mèzard M (1988) Processing of temporal sequences in neural networks. Phys Rev Lett 61:235–238

    Google Scholar 

  • Gutfreund H, Reger JD, Young AP (1988) The nature of attractors in an asymmetric spin glass with deterministic dynamics. J Phys A Math Nucl Gen 21:2775–2797

    Google Scholar 

  • Hemmen JL van, Kühn R (1991) Collective phenomena in neural networks. In: Domany E, Hemmen JL van, Schulten K (eds) Models of neural networks. Springer, Berlin Heidelberg New York

    Google Scholar 

  • Hertz A, Grinstein G, Solla SA (1987) Irreversible spin glasses and neural networks. In: Hemmen JL van, Morgenstern I (eds) Heidelberg colloquium on glassy dynamics. Springer, Berlin Heidelberg New York, pp 538–546

    Google Scholar 

  • Hertz J, Krogh A, Palmer RG (1991) Introduction to the theory of neural computation. Addison-Wesley, Redwood City, Calif

    Google Scholar 

  • Herz A, Sulzer B, Kühn R, Hemmen JL van (1989) Hebbian learning reconsidered: representation of static and dynamic objects in associative neural nets. Biol Cybern 60:457–467

    Google Scholar 

  • Heskes TM, Gielen S (1992) Retrieval of pattern sequences at variable speeds in a neural network model. Neural Networks 5:145–152

    Google Scholar 

  • Hopfield JJ (1982) Neural networks and physical systems with emergent collective computational abilities. Proc Natl Acad Sci USA 79:2554–2558

    Google Scholar 

  • Ishizuka N, Weber J, Amaral DG (1990) Organization of intrahippocampal projections originating from CA3 pyramidal cells in the rat. J Comp Neurol 295:580–623

    Google Scholar 

  • Kleinfeld D (1986) Sequential state generation by model neural networks. Proc Natl Acad Sci USA 83:9469–9473

    Google Scholar 

  • Kree R, Zippelius A (1991) Asymmetrically diluted neural networks. In: Domany E, Hemmen JL van, Schulten K (eds) Models of neural networks. Springer, Berlin Heidelberg New York, pp 193–212

    Google Scholar 

  • Kürten KE (1988) Critical phenomena in model neural networks. Phys Lett A129:157–160

    Google Scholar 

  • Levy WB (1989) A computational approach to hippocampal function. In: Hawkins RD, Bower GH (eds) Computational models of learning in simple neural systems. (The psychology of learning and motivation, vol 23). Academic Press, San Diego, pp 243–305

    Google Scholar 

  • Marr D (1971) Simple memory: a theory for archicortex. Philos Trans R Soc Lond [Biol] 262:23–81

    Google Scholar 

  • McNaughton BL, Nadel L (1989) Hebb-Marr networks and the neurobiological representation of action in space. In: Gluck MA, Rumelhart D (eds) Neuroscience and connectionist theory. Erlbaum, Hillsdale, NJ, pp 1–63

    Google Scholar 

  • Nakao M, Watanabe K, Takahashi T, Mizutani Y, Yamamoto M (1992) Structural properties of network attractor associated with neuronal dynamics transition. Proceedings of International Joint Conference on Neural Networks, Baltimore, vol 3, pp 529–534

    Google Scholar 

  • Nützel K (1991) The length of attractors in asymmetric random neural networks with deterministic dynamics. J Phys A Math Nucl Gen 24:L151–157

    Google Scholar 

  • O'Keefe J, Nadel L (1978) The hippocampus as a cognitive map. Oxford University Press, Oxford

    Google Scholar 

  • Parisi G (1986) Asymmetric neural networks and the process of learning. J Phys A Math Nucl Gen 19:L675-L680

    Google Scholar 

  • Reiss M, Taylor JG (1991) Storing temporal sequences, Neural Networks 4:773–787

    Google Scholar 

  • Riedel U, Kühn R, van Hemmen JL (1988) Temporal sequences and chaos in neural nets. Phys Rev A 28: 1105–1108

    Google Scholar 

  • Rieger H, Schreckenberg M, Zittartz J (1989) Glauber dynamics of the asymmetric SK-model. Z Phys B Condensed Matter 74:527–538

    Google Scholar 

  • Rolls ET (1989) Functions of neuronal networks in the hippocampus and neocortex in memory. In: Byrne JH, Berry WO (eds) Neural models of plasticity. Academic Press, New York, pp 240–265

    Google Scholar 

  • Rolls ET, Treves A (1990) The relative advantages of sparse versus distributed encoding for associative neuronal networks in the brain. Network 1:407–421

    Google Scholar 

  • Segal M (1990) Serotonin attenuates a slow inhibitory postsynaptic potential in rat hippocampal neurons. Neuroscience 36:631–641

    Google Scholar 

  • Sompolinsky H, Kanter I (1986) Temporal association in asymmetric neural networks. Phys Rev Lett 57:2861–2864

    Google Scholar 

  • Sompolinsky H, Crisanti A, Sommers HJ (1988) Chaos in random neural networks. Phys Rev Lett 61:259–262

    Google Scholar 

  • Spitzner P, Kinzel W (1989a) Freezing transition in asymmetric random neural networks with deterministic dynamics. Z Phys B Condensed Matter 77:511–517

    Google Scholar 

  • Spitzner P, Kinzel W (1989b) Hopfield network with directed bonds. Z Phys B Condensed Matter 74:539–545

    Google Scholar 

  • Squire LR, Zola-Morgan S (1991) The medial temporal lobe memory system. Science 253:1380–1386

    Google Scholar 

  • Swanson LW, Köhler C, Björklund A (1987) The limbic region. I. The septohippocampal system. In: Björklund A, Hökfelt T, Swanson LW (eds) Integrated Systems of the CNS, part 1. (Handbook of chemical neuroanatomy, vol 5) Elsevier, Amsterdam, pp 125–277

    Google Scholar 

  • Treves A, Rolls ET (1992) Computational constraints suggest the need for two distinct input systems to the hippocampal CA3. Hippocampus 2:189–200

    Google Scholar 

  • Tsuda I (1992) Dynamic link of memory — chaotic memory map in nonequilibrium neural networks. Neural Networks 5:313–326

    Google Scholar 

  • Willshaw DJ, Buckingham JT (1990) An assessment of Marr's theory of the hippocampus as a temporary memory store. Philos Trans R Soc Lond [Biol] 329:205–215

    Google Scholar 

  • Yao Y, Freeman WJ (1990) Model of biological pattern recognition with spatially chaotic dynamics. Neural Networks 3:153–170

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Neurosurgery, Health Sciences Center, University of Virginia, Box 420, 22908, Charlottesville, VA, USA

    Ali A. Minai & William B. Levy

Authors
  1. Ali A. Minai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. William B. Levy
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Minai, A.A., Levy, W.B. The dynamics of sparse random networks. Biol. Cybern. 70, 177–187 (1993). https://doi.org/10.1007/BF00200831

Download citation

  • Received:

  • Accepted:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF00200831

Keywords

Search

Navigation