2005 Special IssueNon-homogenous neural networks with chaotic recursive nodes: Connectivity and multi-assemblies structures in recursive processing elements architectures☆
Introduction
Artificial Neural Networks employing nodes with complex dynamics and the representation and storage of information through spatio-temporal patterns, as well as the use of such elements for the modeling of biological phenomena in the nervous system, are growing subjects in recent years (Farhat et al., 1994, Kaneko and Tsuda, 2001, Kozma and Freeman, 2000, Wang, 2004). Neural models with bifurcation and chaotic dynamics play a fundamental role in this scenario (Del-Moral-Hernandez, 1998, Del-Moral-Hernandez, 2001, Del-Moral-Hernandez, 2005, Freeman, 1992, Hopfield, 1982, Kaneko and Tsuda, 2001, Principe et al., 2001). In this context, we have developed architectures with Recursive Processing Elements nodes (RPEs) which explore their cyclic final states to represent information (Del-Moral-Hernandez, 1998, Del-Moral-Hernandez, 2001, Del-Moral-Hernandez, 2003, Del-Moral-Hernandez et al., 2003, Del-Moral-Hernandez and Silva, 2004, Del-Moral-Hernandez, 2005). The prototypical node of our studies is the logistic recursion, and the coupling of several of these nodes, as represented in Fig. 1, allows for the implementation of assemblies with auto and hetero associative functionalities. Among other advantages, these associative networks present superior pattern recovery performance as compared to traditional neural networks and have the ability to explore chaotic dynamics for enhanced pattern search.
A reasonable number of results have been generated with these chaotic nodes architectures, showing their effectiveness in implementing associative structures. One of the most relevant of these results is the improved pattern recovery ability of the RPEs associative networks with respect to traditional Hopfield networks: the Hopfield network has larger average recovery errors than the RPEs network does, by a factor that ranges from 1.5 to 2, which is clearly significant—a comparative plot presents this result later in the paper, in Section 3 (Del-Moral-Hernandez, 2003). Perhaps more important than this performance result is the fact that such networks, and their associated methodology, allow for information representation and computation with periodic and chaotic attractors, thus offering a richer scenario than traditional computing with fixed point attractors.
An important current challenge in expanding the use of RPEs regards the possibility of using these nodes to build more general architectures and the possibility of designing multi-assemblies structures, with a number of interacting memory units, each one dealing with a different volume of stored memories, different pattern lengths, and possibly employing heterogeneous nodes. When this is the case, personalized treatment of the chaotic node features, of the storage load of each assembly, as well as of the size (number of nodes) of each assembly, has to be provided in order to ensure good levels of performance for each one of the modules involved in the whole structure. Such a goal is not guaranteed if an homogeneous design approach is taken, as is shown later in this paper.
The following two sections allow the understanding of the main issues involved in the design and operation of RPEs architectures. Section 4 analyzes in detail the importance of choosing the proper scale for the synaptic connections in RPEs networks. It studies how this choice impacts on network performance, and warns us of the associated need of controlling such a quantitative aspect of the RPEs architectures. Section 5 discusses how associative assemblies dealing with different loads of stored patterns can keep such control of the synaptic connections scale. Section 6 addresses the mechanisms that allow for the RPEs assemblies to monitor input patterns and equates related aspects on the use of diverse RPE nodes in heterogeneous structures.
Section snippets
Architectures of bifurcating recursive processing elements—network structure and operation
The basic element of the associative architectures studied in this work is the logistic recursion node, which in our context sometimes will be named RPE node. Eq. (1) describes the local dynamics of the logistic RPE nodes (Del-Moral-Hernandez, 2001, Del-Moral-Hernandez, 2003, Del-Moral-Hernandez, 2005, Hilborn, 1994).
In Eq. (1), n is the discrete time, xi,n and xi,n+1 represent consecutive values of the state variable for node i, and the pi,n is the
Performance evaluation in associative networks
The central functionality of the associative networks described in the previous section is the recovery of noisy patterns. Therefore, a practical way to evaluate the association performance of a given architecture is to measure how much the Hamming distance between the noisy prompting pattern and the ‘clean stored pattern’ is reduced (and ideally taken to zero), through the action of the associative memory.
Since in the general case the prompting noise is random and can have different
Synaptic connections magnitude and its impact on network stability and performance
Differently from traditional Hopfield networks composed of threshold units nodes, for which the choice of scale for the connection matrix W is irrelevant1, the order of magnitudes of the wij connections plays a critical role in the RPEs network performance, as the experiments presented in this section demonstrate. The proper operation of the RPEs network depends on the
Increase of W scale with the load of associative modules and strategies of normalization
An associative assembly feature which is closely related to the average magnitude <|wij|> addressed in the previous section is the number of stored patterns M. This happens because in the Hebbian learning methods employed in RPEs networks, the larger the number of stored patterns M, the larger the strength of the wij connections (Haykin, 1999, Hopfield, 1982; ). This relation appears explicitly in Eq. (8), which shows the way the connection matrix W is evaluated from the set of stored patterns ξ
Asynchronous input changes, modular design and heterogeneous multi associative assemblies
The presence of the incoming paths represented in Fig. 1 indicates the following important features of the represented architecture: the possibility of time-varying inputs, the ability of connecting directly the outputs of a given RPEs assembly to the inputs of another, and a mechanism for hetero-association (available through the SWik connections described below) (Del-Moral-Hernandez, 2005).
Eqs. (9), (10) define the way to compose the input information and the internal states xi. This is done
Conclusions
The discussions of concepts and tools on chaotic RPEs networks presented in the initial sections give comprehensive material on how these networks can be designed and applied. Thus, they offer a good amount of information for pursuing additional developments. The several experiments on magnitude scaling in synaptic connections allow the understanding of performance degradation processes that take place in RPEs architectures when the control of wij magnitudes is not appropriate. The developments
References (18)
Neural networks with chaotic recursive nodes: techniques for the design of associative memories, contrast with Hopfield architectures, and extensions for time-dependent inputs
Neural Networks
(2003)Artificial neural networks based on bifurcating recursive processing elements
(1998)Studying neural networks of bifurcating recursive processing elements—quantitative methods for architecture design (2084)
(2001)- Del-Moral-Hernandez, E. (2005). Non-homogenous structures in neural networks with chaotic recursive nodes: Dealing with...
- et al.
Analog realization of arbitrary one dimensional maps
IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications
(2003) - et al.
A new hybrid neural architecture (MLP+RPE) for hetero association
International Joint Conference on Neural Networks
(2004) - et al.
Complexity and chaotic dynamics in spiking neuron embodiment
SPIE Critical Review
(1994) Tutorial on neurobiology: From single neuron to brain chaos
International Journal of Bifurcation and Chaos
(1992)Neural networks: A comprehensive foundation
(1999)
Cited by (9)
Robust pattern recognition using chaotic dynamics in Attractor Recurrent Neural Network
2010, Proceedings of the International Joint Conference on Neural NetworksContext dependent pattern recognition mdash; A framework for hybrid architectures bridging chaotic neural networks based on recursive Processing Elements and symbolic information
2009, Proceedings of the International Joint Conference on Neural NetworksDesign of chaotic neural network for robust phoneme recognition
2008, Proceedings of the 8th IEEE International Symposium on Signal Processing and Information Technology, ISSPIT 2008RPE-BAM networks: Bidirectional heteroassociation in neural networks of recursive nodes with rich dynamics
2008, Proceedings of the International Joint Conference on Neural NetworksDesign of a chaotic neural network by using chaotic nodes and NDRAM network
2008, Proceedings of the International Joint Conference on Neural Networks
- ☆
The author would like to thank the University of São Paulo, FAPESP and CNPq, for supporting this work. An abbreviated version of some portions of this article appeared in (Del-Moral-Hernandez, 2005), published under the IEEE copyright.