Common nature of learning between BP-type and Hopfield-type neural networks

Abstract

Being two famous neural networks, the error back-propagation (BP) algorithm based neural networks (i.e., BP-type neural networks, BPNNs) and Hopfield-type neural networks (HNNs) have been proposed, developed, and investigated extensively for scientific research and engineering applications. They are different from each other in a great deal, in terms of network architecture, physical meaning and training pattern. In this paper of literature-review type, we present in a relatively complete and creative manner the common natures of learning between BP-type and Hopfield-type neural networks for solving various (mathematical) problems. Specifically, comparing the BPNN with the HNN for the same problem-solving task, e.g., matrix inversion as well as function approximation, we show that the BPNN weight-updating formula and the HNN state-transition equation turn out to be essentially the same. Such interesting phenomena promise that, given a neural-network model for a specific problem solving, its potential dual neural-network model can thus be developed.

Introduction

As a branch of artificial intelligence, artificial neural networks (ANNs) have attracted considerable attention as candidates for novel computational systems [1], [2], [3], [4], [5]. Benefiting from parallel processing capability, inherent nonlinearity, distributed storage and adaptive learning ability, a rich repertoire of ANNs (such as those shown in Fig. 1) have been developed and investigated [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17]. They have been applied widely in many scientific and engineering fields, such as data mining, classification and diagnosis, image and signal processing, control system design, and equations solving. Generally speaking, an ANN is an interconnected group of artificial neurons that uses a mathematical or computational model for information processing based on a connectionistic approach to computation [4], [5]. In view of this point, ANNs can also be termed as simulated neural networks or simply neural networks. Note that, according to the definition of ANN, the combination of inputs, outputs, neurons, and connection weights constitute the architecture of a neural network. Therefore, ANNs can be classified in different categories in terms of architecture. Specifically, according to the nature of connectivity, neural networks can be classified into two categories: feedforward neural networks and feedback neural networks [4], [5]. In general, the working-signal is not fed back between neurons of feedforward neural networks; while the working-signal is fed back between neurons in recurrent neural networks. That is, feedforward neural networks have no loops; while feedback neural networks have loops because of feedback connections [3], [5]. More specifically, in a feedback neural network, signals propagate in only one direction, from an input stage through intermediate neurons to an output stage. Therefore, data from neurons of a lower layer are sent to neurons of an upper layer by feedforward connection networks. By contrast, in a feedback neural network, signals propagate from the output of any neuron to the input of any neuron, and thus they bring data from neurons of an upper layer back to neurons of a lower layer [4], [5].

Being one classical feedforward neural network, the one based on the error back-propagation (BP) algorithm, i.e., BP neural network, was developed through the work of Werbos [18], Rumelhart, McClelland and others [8] in the mid 1970s and 1980s. By following such inspiring thoughts, more and more neural networks based on the BP algorithm or its variants (termed, BP-type neural networks, BPNNs) have been developed and involved in many theoretical analyses and real-world applications [13], [14], [15], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29]. In general, with a number of artificial neurons connected, a feedforward neural network can be constructed, e.g., the one shown in the left of Fig. 1. Then, by means of the iterative BP training procedure, such a neural network would have the marvelous approximation, generalization and prediction abilities. Note that, theoretically speaking, a 3-layer feedforward neural network can approximate any nonlinear continuous function with an arbitrary accuracy. Besides, for these BPNNs, the classical error back-propagation algorithm can be summarized simply as

$$W(k+1)=W\left(k\right)+\mathrm{\Delta }W=W\left(k\right)-\eta {\frac{\partial e\left(W\right)}{\partial W}|}_{W=W\left(k\right)}$$

where W denotes a vector or matrix of neural weights, and $k=0,1,2,\dots$ denotes the iteration index during the training procedure. In addition, $\mathrm{\Delta }W$ denotes the weight-updating value at the kth iteration of the training procedure, with η denoting the learning rate (or termed, learning step-size) which should be small enough. Furthermore, e(W) denotes the error function that is used to monitor and control such a BP-training procedure. By iteratively adjusting the weights and biases of the neural network for the sake of minimizing the network-output error e(W), such a BPNN can be trained for mathematical problems solving, function approximation, system identification, or pattern classification. Nowadays, the BPNN is one of the most widely used neural-network models in the computational-intelligence research and engineering fields [13], [14], [15], [21], [22], [23], [24], [25], [26], [27], [28], [29]. For example, Xu et al. developed a BPNN model to map the complex non-linear relationship between microstructural parameters and elastic modulus of the composite [25]. In [26], a BPNN improved by genetic algorithm (GA) was established by Zhang et al. to model the relation between welding appearance and the characteristics of the molten-pool-shadows. Besides, for integrating the BPNN with GA, an iteration optimization approach was presented and investigated by Huang et al. [27].

Being another classical neural network but with feedback, Hopfield neural network was originally proposed by Hopfield in the early 1980s [9]. This seminal work has inspired many researchers to investigate alternative neural networks for solving a wide variety of mathematical problems arising in numerous fields of science and engineering (e.g., matrix inversion in robotic redundancy resolution). Thus, lots of Hopfield-type neural networks (HNNs) have been proposed, developed and investigated [4], [5], [10], [15], [16], [17], [30], [31], [32], [33], e.g., the one shown in the right of Fig. 1. Note that such HNNs are, sometimes, called gradient neural networks, in the sense that the gradient-descent method is generally exploited in the HNN design process. Traditionally speaking, to obtain an HNN for a specific problem solving, a norm-based scalar-valued lower-bounded energy function is firstly constructed such that its minimal point is the solution to the problem; and, secondly, an HNN is developed to evolve along the typical negative-gradient direction of the energy function until the minimum of the energy function is reached. In mathematics, the dynamics-model description of such an HNN for solving a specific problem $f\left(X\right)=0$ [with $f(·)$ being a smooth linear or nonlinear mapping] can be summarized simply and readily as

$$\dot{X}=-\gamma \frac{\partial \epsilon \left(X\right)}{\partial X},$$

where $\gamma >0\in R$ is used to scale the convergence rate of the HNN model, and $\dot{X}$ denotes the time derivative of state X of the HNN. In addition, the energy function $\epsilon \left(X\right)=\Vert f\left(X\right){\Vert }^2/2$ with $\Vert ·\Vert$ denoting the two-norm of a vector or the Frobenius norm of a matrix accordingly. Evidently, derived from (2), the resultant HNN models are generally depicted in explicit dynamics. In other words, such HNNs would exhibit dynamical behavior; i.e., given an initial state, the state of an HNN evolves as time elapses [4], [5]. If the HNN is stable without oscillation, an equilibrium state can eventually be reached, which is exactly the solution to the problem $f\left(X\right)=0$. Recently, due to the in-depth research on neural networks, the artificial neural-dynamic approach based on HNNs has been viewed as a powerful alternative to online computation and optimization [4], [5], [15], [16], [17], [30], [31], [32], [33]. Especially, being a class of HNNs, fractional/fractional-order neural networks have been designed, analyzed, and applied to different applications [34], [35], [36], [37], [38], [39].

Comparing BPNNs and HNNs (with their general architectures shown in Fig. 1), we find that they are different from each other in a great deal, e.g., with different origins, concepts, definitions, physical meanings, structures, and training patterns. However, as we may realize, the presented BP algorithm (1) is essentially a gradient-descent based error-minimization method, which adjusts the weights to bring the neural-network input/output behavior into a desired mapping as of some specific application tasks or environments. In addition, as mentioned previously, the gradient-descent method is exploited to construct HNNs for problems solving. Thus, we have asked ourselves a special question: does there exist a relationship between BP-type and Hopfield-type neural networks for a specific problem solving? The answer appears to be YES, which is illustrated in this paper of literature-review type through six positive and different examples together with an application to function approximation.

In detail, this paper presents the common natures of learning between BPNNs and HNNs for solving various mathematical problems encountered in science and engineering fields. More specifically, comparing the BPNN and the HNN for the same problem-solving task (e.g., matrix inversion or function approximation), we show that the BPNN weight-updating formula and the HNN state-transition equation turn out to be essentially the same (or say, mathematically the same). In other words, such two neural networks can essentially possess the same mathematical expressions and computational results, though they are deemed completely different. The interesting phenomenon makes us believe that, given a neural-network model for a specific problem solving, its potential dual neural-network model can be developed correspondingly. Note that HNNs and BPNNs have stood in the fields of neural networks for decades (specifically, since the work of Hopfield [9] in 1982, the work of Werbos [18] in 1974 and the work of Rumelhart and McClelland et al. [8] in 1986). Differing from the separate researches on such two different-type neural networks, this paper, in a relatively complete and creative manner, reveals the connections/links (or termed, common natures of learning) between BPNNs and HNNs for various (mathematical) problems solving which are enriched with an application to function approximation. It is an evident state-of-the-art merit, as it establishes the significant and unique research bridge between BPNNs and HNNs.