Coexistence and local stability of multiple equilibria in neural networks with piecewise linear nondecreasing activation functions

doi:10.1016/j.neunet.2009.11.010

Neural Networks

Volume 23, Issue 2, March 2010, Pages 189-200

https://doi.org/10.1016/j.neunet.2009.11.010 Get rights and content

Abstract

In this paper, we investigate the neural networks with a class of nondecreasing piecewise linear activation functions with $2 r$ corner points. It is proposed that the $n$ -neuron dynamical systems can have and only have ${(2 r + 1)}^{n}$ equilibria under some conditions, of which ${(r + 1)}^{n}$ are locally exponentially stable and others are unstable. Furthermore, the attraction basins of these stationary equilibria are estimated. In the case of $n = 2$ , the precise attraction basin of each stable equilibrium point can be figured out, and their boundaries are composed of the stable manifolds of unstable equilibrium points. Simulations are also provided to illustrate the effectiveness of our results.

Introduction

In past decades, neural networks have been extensively studied due to their applications in image processing, pattern recognition, associative memories and many other fields. Wilson and Cowan (1972) studied the dynamics of spatially localized neural populations, and introduced two functions $E (t)$ and $I (t)$ to characterize the states of excitatory neurons and inhibitory neurons, respectively. And in Wilson and Cowan (1973), authors derived the following differential equations ${\begin{cases} μ \frac{\partial}{\partial t} 〈 E (x, t) 〉 = - 〈 E (x, t) 〉 + [1 - r_{e} 〈 E (x, t) 〉] S_{e} [α μ [ϱ_{e} 〈 E (x, t) 〉 \otimes β_{e e} (x) - ϱ_{i} 〈 I (x, t) 〉 \otimes β_{i e} (x) \pm 〈 P (x, t) 〉]] \\ μ \frac{\partial}{\partial t} 〈 I (x, t) 〉 = - 〈 I (x, t) 〉 + [1 - r_{i} 〈 I (x, t) 〉] S_{i} [α μ [ϱ_{e} 〈 E (x, t) 〉 \otimes β_{e i} (x) - ϱ_{i} 〈 I (x, t) 〉 \otimes β_{i i} (x) \pm 〈 Q (x, t) 〉]] \end{cases}$ where $〈 E (x, t) 〉, 〈 I (x, t) 〉$ represent time coarse-grained excitatory and inhibitory activities, respectively; $S_{e} [\cdot], S_{i} [\cdot]$ are the expected proportions of excitatory neurons and inhibitory neurons receiving at least threshold excitation per unit time; $β_{j j^{'}} (x)$ stands for the probability that cells of class $j^{'}$ be connected with cells of class $j$ a distance $x$ away; $\otimes$ denotes spatial convolution, $P (x, t), Q (x, t)$ are the afferent stimuli to excitatory neurons and inhibitory neurons, respectively. The results obtained are closely related to the biological systems and succeeded in providing qualitative descriptions of several neural processes. Grossberg (1973) introduced another class of recurrent on-center off-surround networks, which were shown to be capable of contrast enhancing significant input information; sustaining this information in short term memory; producing multistable equilibrium points that normalize, or adapt, the field’s total activity; suppressing noise; and preventing saturation of population response even to input patterns whose intensities are high (Ellias & Grossberg, 1975). In such an on-center off-surround anatomy, a given population excites itself (and possibly near populations) and inhibits populations that are further away (and possibly itself and nearby populations also). And in Cohen and Grossberg (1983), Cohen–Grossberg neural networks were proposed, which can be described by the following differential equations $\frac{d u_{i}}{d t} = a_{i} (u_{i}) [b_{i} (u_{i}) - \sum_{j = 1}^{n} c_{i j} h_{j} (u_{j})], i = 1, \dots, n .$

In particular, let $a_{i} (\cdot) \equiv 1, b_{i} (x) = - d_{i} x$ , then, the Cohen–Grossberg neural networks reduce to the following Hopfield neural networks $\frac{d u_{i} (t)}{d t} = - d_{i} u_{i} (t) + \sum_{j = 1}^{n} w_{i j} f_{j} (u_{j} (t)) + I_{i}, i = 1, \dots, n,$ where $u_{i} (t)$ represents the state of the $i$ -th unit at time $t$ ; $d_{i} > 0$ denotes the rate with which the $i$ -th unit will reset its potential to the resting state in isolation when disconnected from the network and external inputs; $w_{i j}$ corresponds to the connection weight of the $j$ -th unit on the $i$ -th unit; $f_{j} (\cdot)$ is the activation function; and $I_{i}$ stands for the external input.

There has been a large number of works on the dynamics of neural networks in the literature. Note that in many existing works, the authors mainly focused on the existence of a unique equilibrium and its stability, see Chen (2001), Chen and Amari (2001) and other papers. However, in practice, it is desired that the network has several equilibria, of which each represents an individual pattern. For example, in the Cellular Neural Networks (CNNs) with saturated activation function $f_{j} (x) = \frac{| x + 1 | - | x - 1 |}{2}, j = 1, \dots, n$ see (Fig. 1), a pattern or an associative memory is usually stored as a binary vector in ${- 1, 1}^{n}$ , and the process of the pattern recognition or memory attaining is that the system converges to certain stable equilibrium with all components located in $(- \infty, - 1)$ or $(1, + \infty)$ . Also in some neuromorphic analog circuits, multistable dynamics even play an essential role, as revealed in Douglas, Koch, Mahowald, Martin, and Suarez (1995), Hahnloser, Sarpeshkar, Mahowald, Douglas, and Seung (2000), and Wersing, Beyn, and Ritter (2001). Therefore, the study of coexistence and stability of multiple equilibrium points, in particular, the attraction basin, is of great interest in both theory and applications.

In an earlier paper (Chen & Amari, 2001), the authors pointed out that the 1-neuron neural network model $\frac{d u (t)}{d t} = - u (t) + (1 + ϵ) g (u (t))$ , where $ϵ$ is a small positive number and $g (u) = tanh (u)$ , has three equilibrium points and two of them are locally stable, one is unstable. Recently, for the $n$ -neuron neural networks, many results have been reported in the literature, see Ma and Wu, 2007, Remy et al., 2008, Shayer and Campbell, 2000, Zeng and Wang, 2006, Zhang and Tan, 2004 and Zhang, Yi, and Yu (2008). In Zeng and Wang (2006), by decomposing phase space $R^{n}$ into $3^{n}$ subsets, the authors investigated the multiperiodicity of delayed cellular neural networks and showed that the $n$ -neural networks can have $2^{n}$ stable periodic orbits located in $2^{n}$ subsets of $R^{n}$ . The multistability of Cohen–Grossberg neural networks with a general class of piecewise activation functions was also discussed in Cao, Feng, and Wang (2008). It was shown in Cao et al. (2008), Cheng, Lin, and Shih (2007), Zeng and Wang (2006), and other papers that under some conditions, the $n$ -neuron networks can have $2^{n}$ locally exponentially stable equilibrium points located in $2^{n}$ saturation regions. But it is still unknown what happens in the remaining $3^{n} - 2^{n}$ subsets. In Cheng, Lin, and Shih (2006), the authors indicated that there can be $3^{n}$ equilibrium points for the $n$ -neuron neural networks. However, they only gave emphasis on $2^{n}$ equilibrium points which are stable in a class of subsets with positive invariance, never mentioned the stability nor the dynamical behaviors of solutions in other $3^{n} - 2^{n}$ subsets. To the best of our knowledge, there are few papers addressing the dynamics in the remaining $3^{n} - 2^{n}$ subsets of $R^{n}$ , nor the attraction basins of all stable equilibrium points.

In this paper, we investigate the neural networks (1) and deal with these issues. To be more general, we present a class of nondecreasing piecewise linear activation functions with $2 r$ corner points, which can be described by: $f_{j} (x) = {\begin{cases} m_{j}^{1} & - \infty < x < p_{j}^{1}, \\ \frac{m_{j}^{2} - m_{j}^{1}}{q_{j}^{1} - p_{j}^{1}} (x - p_{j}^{1}) + m_{j}^{1} & p_{j}^{1} \leq x \leq q_{j}^{1}, \\ m_{j}^{2} & q_{j}^{1} < x < p_{j}^{2}, \\ \frac{m_{j}^{3} - m_{j}^{2}}{q_{j}^{2} - p_{j}^{2}} (x - p_{j}^{2}) + m_{j}^{2} & p_{j}^{2} \leq x \leq q_{j}^{2}, \\ m_{j}^{3} & q_{j}^{2} < x < p_{j}^{3}, \\ \dots \dots & \dots \dots \\ \dots \dots & \dots \dots \\ \frac{m_{j}^{r + 1} - m_{j}^{r}}{q_{j}^{r} - p_{j}^{r}} (x - p_{j}^{r}) + m_{j}^{r} & p_{j}^{r} \leq x \leq q_{j}^{r}, \\ m_{j}^{r + 1} & q_{j}^{r} < x < + \infty, \end{cases}$ where $r \geq 1, {m_{j}^{k}}_{k = 1}^{r + 1}$ is an increasing constant series, $p_{j}^{k}, q_{j}^{k}, k = 1, 2, \dots, r$ are constants with $- \infty < p_{j}^{1} < q_{j}^{1} < p_{j}^{2} < q_{j}^{2} < \dots \dots < p_{j}^{r} < q_{j}^{r} < + \infty, j = 1, 2, \dots, n$ .

The neural networks with activation function (2) can store many more patterns or associative memories than those with saturated function, It is meaningful in applications Fig. 2.

In the following, we will precisely figure out all equilibria for the system (1), and investigate the stability and attraction basin for each equilibrium. Discussions and Simulations are also provided to illustrate and verify theoretical results.

We begin with the multistability for $r = 1$ .

Section snippets

Case I: $r = 1$

In this case, the activation function $f_{j}$ reduces to $f_{j} (x) = {\begin{cases} m_{j} & - \infty < x < p_{j}, \\ \frac{M_{j} - m_{j}}{q_{j} - p_{j}} (x - p_{j}) + m_{j} & p_{j} \leq x \leq q_{j}, \\ M_{j} & q_{j} < x < + \infty, \end{cases}$ where $m_{j}, M_{j}, p_{j}, q_{j}$ are constants with $m_{j} < M_{j}, p_{j} < q_{j}, j = 1, 2, \dots, n$ . And we first investigate 2-neuron neural networks. The $n$ -neuron neural networks can be dealt with similarly.

Case II: $r \geq 1$

Inspired by the discussions above, in this section, we discuss the dynamical system (1) with activation $f_{j}$ described by (2).

Theorem 3

Suppose that ${\begin{cases} - d_{i} p_{i}^{k} + w_{i i} m_{i}^{k} + \sum_{j \neq i} max {w_{i j} m_{j}^{1}, w_{i j} m_{j}^{r + 1}} + I_{i} < 0, \\ - d_{i} q_{i}^{k} + w_{i i} m_{i}^{k + 1} + \sum_{j \neq i} min {w_{i j} m_{j}^{1}, w_{i j} m_{j}^{r + 1}} + I_{i} > 0, \end{cases}$ for $i, j = 1, 2, \dots, n, k = 1, 2, \dots, r$ . Then, the dynamical system(1)has ${(2 r + 1)}^{n}$ equilibrium points. Among them, ${(r + 1)}^{n}$ are locally stable and others are unstable.

Proof

Obviously, $R$ can be divided into $(2 r + 1)$ subsets, so that $R^{n}$ can be divided into ${(2 r + 1)}^{n}$ subsets. For example, when $r = 2$ , $R^{2}$

Attraction basins of equilibria

In this section, we investigate attraction basins of equilibria for the system (1) with activation functions (2).

We begin with the case $n = 2$ and $r = 1$ . Under conditions (5), Theorem 1 tells us that there are 4 locally stable equilibrium points $u^{S_{1}}, u^{S_{2}}, u^{S_{3}}, u^{S_{4}}$ , in subsets $S_{1}, S_{2}, S_{3}, S_{4}$ , respectively, and 5 unstable equilibrium points $u^{Ξ_{1}}, u^{Ξ_{2}}, u^{Ξ_{3}}, u^{Ξ_{4}}, u^{Λ}$ in $Ξ_{1}, Ξ_{2}, Ξ_{3}, Ξ_{4}, Λ$ , respectively.

Take a further look at the dynamics in subsets $Ξ_{1}, Ξ_{2}, Ξ_{3}, Ξ_{4}$ , for example, in $Ξ_{1} = [p_{1}, q_{1}] \times (q_{2}, \infty)$ , system (1) takes the

Discussions

In Sections 2 Case I:, 3 Case II:, we investigate the multistability of neural networks (1) with the activation function given in (2). The proposed method is also applicable when different neurons have different activation functions.

In fact, let $r_{1}, \dots, r_{n}$ be constants such that the activation $f_{j}$ of the $j$ -th neuron has $2 r_{j}$ corner points. Then, we have

Corollary 1

Suppose that ${\begin{cases} - d_{i} p_{i}^{k} + w_{i i} f_{i} (p_{i}^{k}) + \sum_{j \neq i} max {w_{i j} m_{j}^{1}, w_{i j} m_{j}^{r_{j} + 1}} + I_{i} < 0, \\ - d_{i} q_{i}^{k} + w_{i i} f_{i} (q_{i}^{k}) + \sum_{j \neq i} min {w_{i j} m_{j}^{1}, w_{i j} m_{j}^{r_{j} + 1}} + I_{i} > 0, \end{cases}$ for $k = 1, \dots, r_{i}, i, j = 1, 2, \dots, n$ . Then, the

Simulations

In the following, we present three more examples to illustrate the effectiveness of the theoretical results.

Example 1

Consider the following neural network with 2-neurons: ${\begin{cases} \frac{d u_{1} (t)}{d t} = - u_{1} (t) + 4 f_{1} (u_{1} (t)) + f_{2} (u_{2} (t)) - 1, \\ \frac{d u_{2} (t)}{d t} = - 2 u_{2} (t) + f_{1} (u_{1} (t)) + 5 f_{2} (u_{2} (t)) - 1, \end{cases}$ where the activation functions are $f_{i} (x) = \frac{| x + 1 | - | x - 1 |}{2}, i = 1, 2$ .

It is easy to see that the conditions (5) are satisfied. Therefore, by Theorem 1, there exist 9 equilibria, and 4 of them are locally stable while others are unstable. In fact, the equilibrium

Conclusions

In this paper, we study the neural networks with a class of activation functions, which are nondecreasing piecewise linear with $2 r (r \geq 1)$ corner points. We prove that such neural networks have multiple equilibria. Some of them are locally stable and others are unstable. More precisely, such neural networks have ${(2 r + 1)}^{n}$ equilibria in all, ${(r + 1)}^{n}$ of which are locally exponentially stable and others are unstable. We also give the attraction region for each locally stable equilibrium. For the

Acknowledgements

Wang Lili is supported by Graduate Innovation Foundation of Fudan University under Grant EYH1411028. Lu Wenlian is supported by the National Natural Sciences Foundation of China under Grant No. 60804044, and sponsored by Shanghai Pujiang Program No. 08PJ14019. Chen Tianping is supported by the National Natural Sciences Foundation of China under Grant No. 60774074, 60974015 and SGST 09DZ2272900.

References (19)

J. Cao et al.
Multistability and multiperiodicity of delayed Cohen–Grossberg neural networks with a general class of activation functions
Physica D: Nonlinear Phenomena
(2008)
T. Chen
Global exponential stability of delayed hopfield neural networks
Neural Networks
(2001)
T. Chen et al.
New theorems on global convergence of some dynamical systems
Neural Networks
(2001)
C. Cheng et al.
Multistability and convergence in delayed neural networks
Physica D: Nonlinear Phenomena
(2007)
É Remy et al.
Graphic requirements for multistability and attractive cycles in a Boolean dynamical framework
Advances in Applied Mathematics
(2008)
H.R. Wilson et al.
Excitatory and inhibitory interactions in localized populations of model neurons
Biophysical Journal
(1972)
C. Cheng et al.
Multistability in recurrent neural networks
SIAM Journal on Applied Mathematics
(2006)
M.A. Cohen et al.
Absolute stability of global pattern formation and parallel memory storage by competitive neural networks
IEEE Transactions on Systems, Man and Cybernetics
(1983)
R. Douglas et al.
Recurrent excitation in neocortical circuits
Science
(1995)

There are more references available in the full text version of this article.

Cited by (102)

Multistability of switched complex-valued neural networks with state-dependent switching rules
2023, Neurocomputing
The paper focuses on the multistability problems of the switched complex-valued neural networks with state-dependent switching rules. Based on the differential inclusions theory and fixed point theorem, several sufficient conditions are derived to ascertain that there exist $25^{n}$ equilibria, $9^{n}$ of which are locally ex for n-neuron switched complex-valued neural networks. The number of stable equilibria of an n-neuron switched complex-valued neural network increases significantly from $4^{n}$ to $9^{n}$ compared with the conventional complex-valued neural networks. Finally, four numerical examples are presented to substantiate the theoretical results.
Multistability of Cohen–Grossberg neural networks based on activation functions with multiple discontinuous points
2023, Chaos, Solitons and Fractals
This paper studies multistability of Cohen-Grossberg neural networks (CGNNs) with a kind of discontinuous activation function (AF), which has multiple discontinuous points. Under some criteria, CGNNs with the discontinuous AF designed in this paper can produce ${(4 k + 1)}^{n}$ equilibrium points (EPs), therein ${(3 k + 1)}^{n}$ EPs are located at the continuous points in the AF and ${(2 k + 1)}^{n}$ EPs are locally exponentially stable. CGNNs with the designed AF can produce even larger quantity of stable/total EPs compared with the AF in existing literature. Therefore, when CGNNs with the designed discontinuous AF are applied to associative memory, they could store more prototype patterns. Moreover, the attraction basins of the stable EPs in CGNNs are estimated and enlarged. A numerical example is illustrated to testify the correctness of the obtained results.
Multiple asymptotical ω-periodicity of fractional-order delayed neural networks under state-dependent switching
2023, Neural Networks
This paper presents theoretical results on multiple asymptotical $ω$ -periodicity of a state-dependent switching fractional-order neural network with time delays and sigmoidal activation functions. Firstly, by combining the geometrical properties of activation functions with the range of switching threshold, a partition of state space is given. Then, the conditions guaranteeing that the solutions can approach each other infinitely in each positive invariant set are derived. Furthermore, the $S$ -asymptotical $ω$ -periodicity and the convergence of solutions in positive invariant sets are discussed. It is worth noting that the number of attractors increases to $3^{n}$ from $2^{n}$ in a neural network without switching. Finally, three numerical examples are given to substantiate the theoretical results.
Ramp approximations of Michaelis–Menten functions in a model of plant metabolism
2022, Physica D: Nonlinear Phenomena
In 2019, Adams, Ehlting, and Edwards showed that in a model of plant phenylalanine metabolism following Michaelis–Menten kinetics, there are two mechanisms by which primary metabolism is prioritized over secondary metabolism when synthesis rates of shikimate, a precursor to phenylalanine, are low: the Precursor Shutoff Valve (PSV), a form of metabolic regulation effected by a series of reactions called the Shikimate Ester Loop (SEL), and threshold separation. They found that the SEL is completely effective in prioritizing primary metabolism when shikimate production is low, but when the SEL is absent, this prioritization is only seen when the threshold constant associated with the secondary pathway is sufficiently larger than that of the primary pathway. Since nonlinear terms can make analysis difficult, here we replace the Michaelis–Menten terms in Adams and colleagues’ model with piecewise approximations we call ramp functions. We show that the ramp function model behaves the same as the original model under low shikimate conditions; the SEL effects PSV type regulation regardless of threshold constants, while without the SEL, primary metabolism is only prioritized when the threshold constant of the secondary pathway is sufficiently large. This is in contrast to a step function version of the model studied by Edwards and Wood in 2021 where the SEL did not effect PSV type regulation on its own.
Multistability of state-dependent switching neural networks with discontinuous nonmonotonic piecewise linear activation functions
2021, Neurocomputing
Citation Excerpt :
In multistability analysis, the activation functions are mainly concentrated on nondecreasing saturated activation functions, sigmoid activation functions, Maxican–Hat-type activation functions and so on. In the existing works, it should be remarkable that the research on multistability basically concentrates on neural networks with continuous activation functions [20–26]. When handling dynamical systems with very high-slope nonlinear elements, we often to model them using discontinuous right sides, rather than using high slopes with limited values [27–29].
This paper presents the theoretical results on the multistability of state-dependent switching neural networks with discontinuous nonmonotonic piecewise linear activation functions. For n-neurons switching model, this paper shows that neural networks have 7ⁿ equilibrium points, 6ⁿ of which are located at the continuous points of activation functions and others are located at the discontinuous points of activation functions. Among these equilibrium points, 4ⁿ or 5ⁿ are stable and others are unstable, which depend on the relationship between the switching threshold and the discontinuous points of the activation functions. Compared with existing results, this paper reveals that switching threshold and discontinuous character are crucial in increasing the number of equilibrium points. Two examples are presented to verify the theoretical results.
Ramp approximations of sigmoid control functions in gene networks
2021, Physica D: Nonlinear Phenomena
In models for networks of regulatory interactions of biological molecules, the sigmoid relationship between concentration of regulating bodies and the production rates they control has led to the use of continuous-time ‘switching’ systems, sometimes referred to as Glass networks, which result from a simplifying assumption that the switching behaviour occurs instantaneously at particular threshold values. Though this assumption produces highly tractable models, it also causes analytic difficulties, such as non-uniqueness, in certain cases, due to the discontinuities of the system. Here, the use of ramp functions is explored as an alternative approximation to the sigmoid, which restores continuity to the vector field and removes the assumption of infinitely steep switching by linearly interpolating the focal point values used in a corresponding Glass network. A general framework for describing a ramp system using the ‘focal points’ of the corresponding Glass network is given. Solutions of two-dimensional networks are explored, and then higher-dimensional networks under certain restrictions. Periodic behaviour is explored using mappings between threshold boundaries. Limitations in these methods are explored, and a general proof of the existence of periodic solutions in negative feedback loops with ramp interactions is given.

View all citing articles on Scopus

View full text

Coexistence and local stability of multiple equilibria in neural networks with piecewise linear nondecreasing activation functions

Abstract

Introduction

Section snippets

Case I: r=1

Case II: r≥1

Attraction basins of equilibria

Discussions

Simulations

Conclusions

Acknowledgements

Physica D: Nonlinear Phenomena

Neural Networks

Neural Networks

Physica D: Nonlinear Phenomena

Advances in Applied Mathematics

Biophysical Journal

Multistability in recurrent neural networks

SIAM Journal on Applied Mathematics

Absolute stability of global pattern formation and parallel memory storage by competitive neural networks

IEEE Transactions on Systems, Man and Cybernetics

Recurrent excitation in neocortical circuits

Science

Case I: $r = 1$

Case II: $r \geq 1$