On the number of equilibrium states in weakly coupled random networks
Introduction
The number of equilibrium states (fixed points) is an important issue in the study of the dynamics of neural networks and statistical physics. The equilibrium states correspond to stored patterns (memory). The point of the dynamics is to recover a stored pattern when given a distorted pattern as an initial condition. In other words, the initial network state follows a neural dynamical trajectory and arrives at an equilibrium state. Roughly speaking, the number of equilibrium states corresponds to the size of the memory (see, e.g. Amit, 1989; Hopfield, 1982).
In this paper we study the asymptotic behavior of the expected number of equilibrium states for some Hopfield-like model which will be described in details in the sequel. In this model the connection weights (connectivities) consist of symmetric and antisymmetric parts. Note that the methods used in Amari (1990) Tanaka and Edwards (1980) for the symmetric case cannot be generalized to asymmetric connectivities with a larger antisymmetric part, i.e. c>1 in (1.1) below. Asymmetry is essential when learning is taken into consideration (see, e.g. Amit, 1989). However, the asymptotic behaviors become quite difficult to analyze. Intuitively, we show that a small antisymmetric part results in an exponentially large expected number of equilibrium states and a large antisymmetric parts has an opposite result.
Related works may be found in Amari 1974, Amari 1990, Bray and Moore (1980), Date et al. (1995), Date (1996), Derrida and Gardner (1986), Kurata (1990), Tanaka and Edwards (1980) and references therein.
Now consider a fully interconnected network consisting of n elements (neurons) in which the ith output , takes values 1 or −1. , the connection weight from the jth element to the ith element is a composition of two components, namely,where are i.i.d. , and c is a constant.
wijs and wija represent the symmetric weight and the antisymmetric weight respectively. c denotes the relative weight of these two components, hence, it suffices to consider c⩾0. One may regard c=∞ as a completely antisymmetric case, i.e. wij=wija.
Let T denote the (random) nonlinear transformation from {−1,1}n to {−1,1}n defined bywhere sgn(u)=1 when u>0 and is −1 otherwise. And the dynamics of the network is described by xk+1=Txk, where xk denotes the output at time k. This dynamics is widely used as a first approximation of neural dynamics.
A point x in {−1,1}n is called an equilibrium state (fixed point) if Tx=x. We are interested in the asymptotics of the expected number of equilibrium states when n→∞.
The approach here is applicable for all c. The basic idea consists of using specific representations of Gaussian random vectors, conditioning on the minimum of i.i.d. Gaussian random variables, and large deviations.
Specifically let us consider x=(1,…,1), the probability of having this particular x as an equilibrium state iswhere
It is easy to see that the probability of having other state in {−1,1}n as an equilibrium state is the same as (1.3). Hence the expected number of equilibrium states, denoted by E(c), of (1.1) is
The problem is boiled down to study the asymptotic probability of the Gaussian random vector Vn in the first quadrant, where Vn has mean zero and covariance matrix {rij}:with the interpretation rij=−1/n, for i≠j, if c=∞. For brevity, let For c>1,Vn is negatively correlated.
Moreover Vn can be represented aswhere ξ1,…,ξn are i.i.d. and .
For c<∞ (d>−1), the probability (1.3) can be rewritten asThen by conditioning on and Cramér theorem, the asymptotics of the expected number E(c) of the equilibrium states is obtained in Theorem 1. Roughly speaking, there exists a constant k(c) such thatwhere k(c)<0 for for c>1, and E(1)=1. The formula for k(c) can be found in Theorem 1.
For c=∞ (d=−1), the completely antisymmetric cases, Vn is represented byNote that n+1 i.i.d. r.v.'s are used instead of just n r.v.'s, and probability (1.3) is of the formA direct asymptotic analysis yields Theorem 2. Roughly,k(∞)>0.
Note that , is asymptotically Gaussian for quite general wij's. However, the analysis here is quite delicate. It does depend on the Gaussian assumption on wij. Further generalization needs careful study. And a possible connection to extremal value theory deserves some investigation.
Section snippets
The case with the presence of symmetric component, 0⩽c<∞
Recall that d=(1−c2)/(1+c2). Choose . The limit of αn, denoted by β, as n goes to ∞ is . For . And for . And let cn=(αn/β)((n−1)/n)/(1−αn/n). cn goes to 1 as n goes to ∞. Note that for .
Now the calculation of (1.5),where F(x) is the distribution of .
For a
The completely antisymmetric case
For the completely antisymmetric case, c=∞, the probability from formula (1.6) iswhere . Note that and ξk−ηn⩾0.
The last term in the previous inequality is bounded bywhere Mn may be chosen as .if we only consider the exponential component, and can be ignored compared with the second term in (3.1) after going through the following calculation.
Acknowledgements
Professor Shun-ichi Amari first introduced the problem to us. The authors would like to thank him as well as Professors Stuart Geman and Richard Vitale for fruitful discussions. They would like to thank the Division of Applied Mathematics, Brown University for the hospitality.
References (11)
- Amari, S., 1974. A method of statistical neurodynamics. Kybernetik 14,...
- Amari, S., 1990. Mathematical foundations of neurocomputing. Proc. IEEE 78,...
- et al.
Metastable states in spin glasses
J. Phys. C
(1980) - Date, A., Kurata, K., Amari, Shun-ichi, 1995. Number of equilibrium states in a randomly and symmetrically connected...
- Date, A., 1996. Neuronal model dependences for the randomly and symmetrically connected networks. Technical Report of...
Cited by (0)
- 1
Supported by JSPS Research Fellowships for Young Scientists and partially supported by Grant-in-Aid for Scientific Research from the Ministry of Education, Science and Culture of Japan.
- 2
Partially supported by NSC Grant 86-2115-M-001-008 and NSC Grant 34081F.