Elsevier

Neural Networks

Volume 18, Issues 5–6, July–August 2005, Pages 532-540
Neural Networks

2005 Special Issue
Non-homogenous neural networks with chaotic recursive nodes: Connectivity and multi-assemblies structures in recursive processing elements architectures

https://doi.org/10.1016/j.neunet.2005.06.035Get rights and content

Abstract

This paper addresses recurrent neural architectures based on bifurcating nodes that exhibit chaotic dynamics, with local dynamics defined by first order parametric recursions. In the studied architectures, logistic recursive nodes interact through parametric coupling, they self organize, and the network evolves to global spatio-temporal period-2 attractors that encode stored patterns. The performance of associative memories arrangements is measured through the average error in pattern recovery, under several levels of prompting noise. The impact of the synaptic connections magnitude on architecture performance is analyzed in detail, through pattern recovery performance measures and basin of attraction characterization. The importance of a planned choice of the synaptic connections scale in RPEs architectures is shown. A strategy for minimizing pattern recovery degradation when the number of stored patterns increases is developed. Experimental results show the success of such strategy. Mechanisms for allowing the studied associative networks to deal with asynchronous changes in input patterns, and tools for the interconnection between different associative assemblies are developed. Finally, coupling in heterogeneous assemblies with diverse recursive maps is analyzed, and the associated synaptic connections are equated.

Introduction

Artificial Neural Networks employing nodes with complex dynamics and the representation and storage of information through spatio-temporal patterns, as well as the use of such elements for the modeling of biological phenomena in the nervous system, are growing subjects in recent years (Farhat et al., 1994, Kaneko and Tsuda, 2001, Kozma and Freeman, 2000, Wang, 2004). Neural models with bifurcation and chaotic dynamics play a fundamental role in this scenario (Del-Moral-Hernandez, 1998, Del-Moral-Hernandez, 2001, Del-Moral-Hernandez, 2005, Freeman, 1992, Hopfield, 1982, Kaneko and Tsuda, 2001, Principe et al., 2001). In this context, we have developed architectures with Recursive Processing Elements nodes (RPEs) which explore their cyclic final states to represent information (Del-Moral-Hernandez, 1998, Del-Moral-Hernandez, 2001, Del-Moral-Hernandez, 2003, Del-Moral-Hernandez et al., 2003, Del-Moral-Hernandez and Silva, 2004, Del-Moral-Hernandez, 2005). The prototypical node of our studies is the logistic recursion, and the coupling of several of these nodes, as represented in Fig. 1, allows for the implementation of assemblies with auto and hetero associative functionalities. Among other advantages, these associative networks present superior pattern recovery performance as compared to traditional neural networks and have the ability to explore chaotic dynamics for enhanced pattern search.

A reasonable number of results have been generated with these chaotic nodes architectures, showing their effectiveness in implementing associative structures. One of the most relevant of these results is the improved pattern recovery ability of the RPEs associative networks with respect to traditional Hopfield networks: the Hopfield network has larger average recovery errors than the RPEs network does, by a factor that ranges from 1.5 to 2, which is clearly significant—a comparative plot presents this result later in the paper, in Section 3 (Del-Moral-Hernandez, 2003). Perhaps more important than this performance result is the fact that such networks, and their associated methodology, allow for information representation and computation with periodic and chaotic attractors, thus offering a richer scenario than traditional computing with fixed point attractors.

An important current challenge in expanding the use of RPEs regards the possibility of using these nodes to build more general architectures and the possibility of designing multi-assemblies structures, with a number of interacting memory units, each one dealing with a different volume of stored memories, different pattern lengths, and possibly employing heterogeneous nodes. When this is the case, personalized treatment of the chaotic node features, of the storage load of each assembly, as well as of the size (number of nodes) of each assembly, has to be provided in order to ensure good levels of performance for each one of the modules involved in the whole structure. Such a goal is not guaranteed if an homogeneous design approach is taken, as is shown later in this paper.

The following two sections allow the understanding of the main issues involved in the design and operation of RPEs architectures. Section 4 analyzes in detail the importance of choosing the proper scale for the synaptic connections in RPEs networks. It studies how this choice impacts on network performance, and warns us of the associated need of controlling such a quantitative aspect of the RPEs architectures. Section 5 discusses how associative assemblies dealing with different loads of stored patterns can keep such control of the synaptic connections scale. Section 6 addresses the mechanisms that allow for the RPEs assemblies to monitor input patterns and equates related aspects on the use of diverse RPE nodes in heterogeneous structures.

Section snippets

Architectures of bifurcating recursive processing elements—network structure and operation

The basic element of the associative architectures studied in this work is the logistic recursion node, which in our context sometimes will be named RPE node. Eq. (1) describes the local dynamics of the logistic RPE nodes (Del-Moral-Hernandez, 2001, Del-Moral-Hernandez, 2003, Del-Moral-Hernandez, 2005, Hilborn, 1994).xi,n+1=pi,nxi,n(1xi,n)

xi,n+1=Rpi,n(xi,n)

In Eq. (1), n is the discrete time, xi,n and xi,n+1 represent consecutive values of the state variable for node i, and the pi,n is the

Performance evaluation in associative networks

The central functionality of the associative networks described in the previous section is the recovery of noisy patterns. Therefore, a practical way to evaluate the association performance of a given architecture is to measure how much the Hamming distance between the noisy prompting pattern and the ‘clean stored pattern’ is reduced (and ideally taken to zero), through the action of the associative memory.

Since in the general case the prompting noise is random and can have different

Synaptic connections magnitude and its impact on network stability and performance

Differently from traditional Hopfield networks composed of threshold units nodes, for which the choice of scale for the connection matrix W is irrelevant1, the order of magnitudes of the wij connections plays a critical role in the RPEs network performance, as the experiments presented in this section demonstrate. The proper operation of the RPEs network depends on the

Increase of W scale with the load of associative modules and strategies of normalization

An associative assembly feature which is closely related to the average magnitude <|wij|> addressed in the previous section is the number of stored patterns M. This happens because in the Hebbian learning methods employed in RPEs networks, the larger the number of stored patterns M, the larger the strength of the wij connections (Haykin, 1999, Hopfield, 1982; ). This relation appears explicitly in Eq. (8), which shows the way the connection matrix W is evaluated from the set of stored patterns ξ

Asynchronous input changes, modular design and heterogeneous multi associative assemblies

The presence of the incoming paths represented in Fig. 1 indicates the following important features of the represented architecture: the possibility of time-varying inputs, the ability of connecting directly the outputs of a given RPEs assembly to the inputs of another, and a mechanism for hetero-association (available through the SWik connections described below) (Del-Moral-Hernandez, 2005).

Eqs. (9), (10) define the way to compose the input information and the internal states xi. This is done

Conclusions

The discussions of concepts and tools on chaotic RPEs networks presented in the initial sections give comprehensive material on how these networks can be designed and applied. Thus, they offer a good amount of information for pursuing additional developments. The several experiments on magnitude scaling in synaptic connections allow the understanding of performance degradation processes that take place in RPEs architectures when the control of wij magnitudes is not appropriate. The developments

References (18)

  • E. Del-Moral-Hernandez

    Neural networks with chaotic recursive nodes: techniques for the design of associative memories, contrast with Hopfield architectures, and extensions for time-dependent inputs

    Neural Networks

    (2003)
  • E. Del-Moral-Hernandez

    Artificial neural networks based on bifurcating recursive processing elements

    (1998)
  • E. Del-Moral-Hernandez

    Studying neural networks of bifurcating recursive processing elements—quantitative methods for architecture design (2084)

    (2001)
  • Del-Moral-Hernandez, E. (2005). Non-homogenous structures in neural networks with chaotic recursive nodes: Dealing with...
  • E. Del-Moral-Hernandez et al.

    Analog realization of arbitrary one dimensional maps

    IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications

    (2003)
  • E. Del-Moral-Hernandez et al.

    A new hybrid neural architecture (MLP+RPE) for hetero association

    International Joint Conference on Neural Networks

    (2004)
  • N. Farhat et al.

    Complexity and chaotic dynamics in spiking neuron embodiment

    SPIE Critical Review

    (1994)
  • W.J. Freeman

    Tutorial on neurobiology: From single neuron to brain chaos

    International Journal of Bifurcation and Chaos

    (1992)
  • S. Haykin

    Neural networks: A comprehensive foundation

    (1999)
There are more references available in the full text version of this article.

Cited by (9)

View all citing articles on Scopus

The author would like to thank the University of São Paulo, FAPESP and CNPq, for supporting this work. An abbreviated version of some portions of this article appeared in (Del-Moral-Hernandez, 2005), published under the IEEE copyright.

View full text