Elsevier

Statistics & Probability Letters

Volume 49, Issue 3, 15 September 2000, Pages 291-297
Statistics & Probability Letters

On the number of equilibrium states in weakly coupled random networks

https://doi.org/10.1016/S0167-7152(00)00060-2Get rights and content

Abstract

A fully interconnected network consisting of n elements with outputs x={xi,xi=+1,−1,1⩽i⩽n}, connection weights wij=wijs+cwija composed of symmetric and antisymmetric parts, and dynamics described by x↦{sign(∑wijxj)} is considered. Here the {wijs,wija,wkk,i<j} are i.i.d. N(0,1) and wijs=wjis,−wija=wjia,c is a constant in [0,∞].c=∞ is interpreted as wij=wija. The asymptotic behavior of the expected number of the equilibrium states of the network is studied as n→∞.

Introduction

The number of equilibrium states (fixed points) is an important issue in the study of the dynamics of neural networks and statistical physics. The equilibrium states correspond to stored patterns (memory). The point of the dynamics is to recover a stored pattern when given a distorted pattern as an initial condition. In other words, the initial network state follows a neural dynamical trajectory and arrives at an equilibrium state. Roughly speaking, the number of equilibrium states corresponds to the size of the memory (see, e.g. Amit, 1989; Hopfield, 1982).

In this paper we study the asymptotic behavior of the expected number of equilibrium states for some Hopfield-like model which will be described in details in the sequel. In this model the connection weights (connectivities) consist of symmetric and antisymmetric parts. Note that the methods used in Amari (1990) Tanaka and Edwards (1980) for the symmetric case cannot be generalized to asymmetric connectivities with a larger antisymmetric part, i.e. c>1 in (1.1) below. Asymmetry is essential when learning is taken into consideration (see, e.g. Amit, 1989). However, the asymptotic behaviors become quite difficult to analyze. Intuitively, we show that a small antisymmetric part results in an exponentially large expected number of equilibrium states and a large antisymmetric parts has an opposite result.

Related works may be found in Amari 1974, Amari 1990, Bray and Moore (1980), Date et al. (1995), Date (1996), Derrida and Gardner (1986), Kurata (1990), Tanaka and Edwards (1980) and references therein.

Now consider a fully interconnected network consisting of n elements (neurons) in which the ith output xi,i=1,…,n, takes values 1 or −1. wij,1⩽i,j⩽n, the connection weight from the jth element to the ith element is a composition of two components, namely,wij=wijs+cwija,where {wijs,wija,wiis:i<j} are i.i.d. N(0,1),wijs=wjis,wija=−wjia, and c is a constant.

wijs and wija represent the symmetric weight and the antisymmetric weight respectively. c denotes the relative weight of these two components, hence, it suffices to consider c⩾0. One may regard c=∞ as a completely antisymmetric case, i.e. wij=wija.

Let T denote the (random) nonlinear transformation from {−1,1}n to {−1,1}n defined byTx=sgnj=1nwijxji=1,…,n,where sgn(u)=1 when u>0 and is −1 otherwise. And the dynamics of the network is described by xk+1=Txk, where xk denotes the output at time k. This dynamics is widely used as a first approximation of neural dynamics.

A point x in {−1,1}n is called an equilibrium state (fixed point) if Tx=x. We are interested in the asymptotics of the expected number of equilibrium states when n→∞.

The approach here is applicable for all c. The basic idea consists of using specific representations of Gaussian random vectors, conditioning on the minimum of i.i.d. Gaussian random variables, and large deviations.

Specifically let us consider x=(1,…,1), the probability of having this particular x as an equilibrium state isPmin1⩽i⩽njwij>0=Pmin1⩽i⩽nVin>0,where Vn={(1/n(1+c2))∑jwij}1⩽i⩽n.

It is easy to see that the probability of having other state in {−1,1}n as an equilibrium state is the same as (1.3). Hence the expected number of equilibrium states, denoted by E(c), of (1.1) is2n×Pmin1⩽i⩽nVin>0.

The problem is boiled down to study the asymptotic probability of the Gaussian random vector Vn in the first quadrant, where Vn has mean zero and covariance matrix {rij}:rii=1,rij=1−c2n(1+c2)ifi≠jwith the interpretation rij=−1/n, for ij, if c=∞. For brevity, let d=(1−c2)/(1+c2),−1⩽d⩽1. For c>1,Vn is negatively correlated.

Moreover Vn can be represented asVn=n−dnξiαnn1nξki=1,…,n,where ξ1,…,ξn are i.i.d. N(0,1) and αn=1±1+(nd/(n−d)).

For c<∞ (d>−1), the probability (1.3) can be rewritten asPmin1⩽k⩽nξk>αnn1nξk.Then by conditioning on min1⩽i⩽nξi1 and Cramér theorem, the asymptotics of the expected number E(c) of the equilibrium states is obtained in Theorem 1. Roughly speaking, there exists a constant k(c) such thatlnE(c)∼−k(c)n,where k(c)<0 for 0⩽c⩽1,k(c)>0 for c>1, and E(1)=1. The formula for k(c) can be found in Theorem 1.

For c=∞ (d=−1), the completely antisymmetric cases, Vn is represented byVn=n+1nξi1n+11n+1ξki=1,…,n.Note that n+1 i.i.d. r.v.'s are used instead of just n r.v.'s, and probability (1.3) is of the formPmin1⩽k⩽nξk>1n+11n+1ξk.A direct asymptotic analysis yields Theorem 2. Roughly,E(∞)=O(e−k(∞)nlnn),k(∞)>0.

Note that ((1/n)∑jwij,(1/n)∑jwkj),i≠k, is asymptotically Gaussian for quite general wij's. However, the analysis here is quite delicate. It does depend on the Gaussian assumption on wij. Further generalization needs careful study. And a possible connection to extremal value theory deserves some investigation.

Section snippets

The case with the presence of symmetric component, 0⩽c<∞

Recall that d=(1−c2)/(1+c2). Choose αn=1−1+nd/(n−d). The limit of αn, denoted by β, as n goes to ∞ is 1−1+d. For −1<d<0,0<β<1. And for 0<d⩽1,1−2⩽β<0. And let cn=(αn/β)((n−1)/n)/(1−αn/n). cn goes to 1 as n goes to ∞. Note that for d=0(c=1),E(1)=1.

Now the calculation of (1.5),Pmin1⩽k⩽nξk>αnn1nξk=nPmin1⩽k⩽nξk11>αnn1nξk.Pmin1⩽k⩽nξk11>αnn1nξk=−∞12πexpx22Pξ2>x,…,ξn>x,x>βcnn−12nξkdx=−∞12πexpx22Px>βcnn−12nξk|ξ2>x,…,ξn>x(1−F(x))n−1dx,where F(x) is the distribution of N(0,1).

For a

The completely antisymmetric case

For the completely antisymmetric case, c=∞, the probability from formula (1.6) isPmin1⩽k⩽nξk>1n+11n+1ξk⩽P1nk−ηn)<Yn,where ηn=min1⩽k⩽nξk,Yn=∑1nk−(1/(n+1))∑1n+1ξk). Note that YnN(0,(1+1/n2)(n/(n+1))2) and ξkηn⩾0.

The last term in the previous inequality is bounded byP{Yn>Mnn}+P1nk−ηn)<Mnn,where Mn may be chosen as lnn.P{Yn>Mnn}∼expMnn2,if we only consider the exponential component, and can be ignored compared with the second term in (3.1) after going through the following calculation.

Acknowledgements

Professor Shun-ichi Amari first introduced the problem to us. The authors would like to thank him as well as Professors Stuart Geman and Richard Vitale for fruitful discussions. They would like to thank the Division of Applied Mathematics, Brown University for the hospitality.

References (11)

  • Amari, S., 1974. A method of statistical neurodynamics. Kybernetik 14,...
  • Amari, S., 1990. Mathematical foundations of neurocomputing. Proc. IEEE 78,...
  • A.J. Bray et al.

    Metastable states in spin glasses

    J. Phys. C

    (1980)
  • Date, A., Kurata, K., Amari, Shun-ichi, 1995. Number of equilibrium states in a randomly and symmetrically connected...
  • Date, A., 1996. Neuronal model dependences for the randomly and symmetrically connected networks. Technical Report of...
There are more references available in the full text version of this article.

Cited by (0)

1

Supported by JSPS Research Fellowships for Young Scientists and partially supported by Grant-in-Aid for Scientific Research from the Ministry of Education, Science and Culture of Japan.

2

Partially supported by NSC Grant 86-2115-M-001-008 and NSC Grant 34081F.

View full text