Non-convex Multi-species Hopfield Models

Abstract

In this work we introduce a multi-species generalization of the Hopfield model for associative memory, where neurons are divided into groups and both inter-groups and intra-groups pair-wise interactions are considered, with different intensities. Thus, this system contains two of the main ingredients of modern deep neural-network architectures: Hebbian interactions to store patterns of information and multiple layers coding different levels of correlations. The model is completely solvable in the low-load regime with a suitable generalization of the Hamilton–Jacobi technique, despite the Hamiltonian can be a non-definite quadratic form of the Mattis magnetizations. The family of multi-species Hopfield model includes, as special cases, the 3-layers Restricted Boltzmann Machine with Gaussian hidden layer and the Bidirectional Associative Memory model.

Appendix A. Proof of Lemma 1

Before proving Lemma 1 we define the matrix $\tilde{\mathbf{J }}^c$ as

$$\begin{aligned} \tilde{\mathbf{J }}^c= c \ \mathbf{diag }(k^*\alpha _1^2, .. . . , k^*\alpha _\nu ^2) -\mathbf{J }, \end{aligned}$$

(A.1)

where $k^*$ is defined as in Lemma (1), then it holds the following

Lemma 2

The quadratic form stemming from (A.1) is positive definite if $c>1+\displaystyle \frac{\nu -1}{k^*}$. Moreover, we get $\tilde{\mathbf{J }}^c = \tilde{\mathbf{P }}^T\tilde{\mathbf{P }}$, where the rows of $\tilde{\mathbf{P }}$ are the linearly independent vectors given by

$$\begin{aligned}&\mathbf{v }^1= \frac{\sqrt{k^*(c-1)-(\nu -1)}}{\sqrt{\nu }} (\alpha _1, . . . \ , \alpha _\nu )\\&\mathbf{v }^2= \frac{\sqrt{k^*(c-1)+1}}{\sqrt{2}} (\alpha _1, -\alpha _2, 0, . . . \ , 0)\\&\vdots \\&{\mathbf{v }}^a= \frac{\sqrt{k^*(c-1)+1}}{\sqrt{a(a-1)}} (\alpha _1, . . .. \ , \alpha _{a-1}, \alpha _{a}-a\alpha _{a}, 0, . . . ,0)\\&\vdots \\&\mathbf{v }^\nu = \frac{\sqrt{k^*(c-1)+1}}{\sqrt{\nu (\nu -1)}} (\alpha _1, . . . \ , \alpha _{\nu -1}, \alpha _{\nu }-\nu \alpha _{\nu }). \end{aligned}$$

Proof

Let $\mathbf{A }=\mathbf{diag }(\alpha _1, \dots , \alpha _{\nu })$ and define

$$\begin{aligned} \tilde{\mathbf{T }}_{ab}^c :={\left\{ \begin{array}{ll} (c-1)k^* \; \; \; \text {se } \; \; a=b\\ \\ \ -1 \; \; \; \ \ \ \ \ \text {se } \; \; a\ne b, \end{array}\right. } \end{aligned}$$

(A.2)

such that we can write $\tilde{\mathbf{J }}^c=\mathbf{A }\tilde{\mathbf{T }}\mathbf{A }$. We now derive the eigenvectors of $\tilde{\mathbf{T }}^c$. In order for w to be eigenvector with eigenvalue $\lambda$, the following homogeneous system must be fulfilled

$$\begin{aligned} (\tilde{\mathbf{T }}^c-\lambda \mathbb {I}_\nu )\mathbf{w }=0. \end{aligned}$$

(A.3)

Focusing on the ith component

$$\begin{aligned} ((\tilde{\mathbf{T }}^c-\lambda \mathbb {I}_\nu )\mathbf{w })_{i}= -\displaystyle \sum _{a\ne i} w_{a} + ((c-1)k^*-\lambda )w_{i}= - \displaystyle \sum _{a=1}^{\nu } w_{a} + ((c-1)k^*-\lambda +1) w_{i}. \end{aligned}$$

If $\displaystyle \sum _{a=1}^{\nu } w_{a}\ne 0$, then $\mathbf{w }^1={\nu }^{-1/2}(1, \dots , 1)$ is eigenvector with eigenvalue $\lambda = (c-1)k^*+1-\nu$, being positive if $c>1+(\nu -1)/k^*$.

The remaining eigenvectors live in the subspace of dimension $\nu -1$, characterized by $\displaystyle \sum _{a=1}^{\nu } w_{a}=0$, and are related to the same eigenvalue $\lambda =(c-1)k^*+1$. Normalizing, we get that the base of eigenvectors of the subspace orthogonal to the subspace $\mathbb {R} \mathbf{w }^1$ is

$$\begin{aligned}&\mathbf{w }^2=\frac{\sqrt{k^*(c-1)+1}}{\sqrt{2}}(1,-1, 0, \dots , 0)\\&\vdots \\&\mathbf{w }^a=\frac{\sqrt{k^*(c-1)+1}}{\sqrt{a(a-1)}}(1,\dots , 1, 1-a, 0 , \dots , 0)\\&\vdots \\&\mathbf{w }^a=\frac{\sqrt{k^*(c-1)+1}}{\sqrt{\nu (\nu -1)}}(1,\dots , 1, 1-\nu ). \end{aligned}$$

Let ${\mathbf{P }}^\prime$ the matrix of dimension $\nu \times \nu$, whose rows are the vectors $\mathbf{w }_a$, $a=1, \dots , \nu$. The matrix $\mathbf{P }^\prime$ is such that ${\mathbf{P }^\prime }^T\mathbf{P }^\prime ={\tilde{\mathbf{T }}}^c$, therefore if we pose $\tilde{\mathbf{P }}=\mathbf{P }^\prime \mathbf{A }$ we get

$$\begin{aligned} \tilde{\mathbf{P }}^T\tilde{\mathbf{P }}= (\mathbf{P }^\prime \mathbf{A })^T(\mathbf{P }^\prime \mathbf{A })=\mathbf{A }{\mathbf{P }^\prime }^T\mathbf{P }^\prime \mathbf{A }= \mathbf{A }{\tilde{\mathbf{T }}}^c \mathbf{A }=\tilde{\mathbf{J }}^c. \end{aligned}$$

Finally, we notice that the rows of $\tilde{\mathbf{P }}$ are

$$\begin{aligned}&\mathbf{v }^1= \frac{\sqrt{(c-1)k^*-\nu +1}}{\sqrt{\nu }} (\alpha _1, . . . \ , \alpha _\nu )\\&\mathbf{v }^2= \frac{\sqrt{k^*(c-1)+1}}{\sqrt{2}} (\alpha _1, -\alpha _2, 0, . . . \ , 0)\\&\vdots \\&{\mathbf{v }}^a= \frac{\sqrt{k^*(c-1)+1}}{\sqrt{a(a-1)}} (\alpha _1, . . . \ , \alpha _{a-1}, \alpha _{a}-a\alpha _{a}, 0, . . . ,0)\\&\vdots \\&\mathbf{v }^\nu = \frac{\sqrt{k^*(c-1)+1}}{\sqrt{\nu (\nu -1)}} (\alpha _1, . . . \ , \alpha _{\nu -1}, \alpha _{\nu }-\nu \alpha _{\nu }). \end{aligned}$$

$\square$

We are now ready to prove Lemma (1).

Proof

We want to prove that if the condition $c>1+\displaystyle \frac{\nu -1}{k^*}$ holds, then (5.1) is positive definite. Along the proof we will exploit the fact that $\tilde{\mathbf{J }}^c$ defined in (A.1) is positive definite and we shall consider the diagonal matrix given by the difference between (5.1) and (A.1):

$$\begin{aligned} \mathbf{J }^c-\tilde{\mathbf{J }}^c ={\left\{ \begin{array}{ll} &{} (c-1)(k_a-k^*)\alpha _a^2 \; \; \; \text {se } \; \; a=b \\ \\ &{} 0\; \; \;\;\;\; \ \ \ \ \ \text {se } \; \; a\ne b. \end{array}\right. } \end{aligned}$$

(A.4)

Therefore, if $c>1+\displaystyle \frac{\nu -1}{k^*}$, then $\mathbf{J }^c-\tilde{\mathbf{J }}^c >0$. If $\mathbf{u }\in \mathbb {R}^\nu$ is an arbitrary vector

$$\begin{aligned} \mathbf{u }^T \bigl (\mathbf{J }^c-\tilde{\mathbf{J }}^c\bigr ) \mathbf{u }\ge 0 \end{aligned}$$

(A.5)

from which we get

$$\begin{aligned} \mathbf{u }^T \mathbf{J }^c\mathbf{u }> \mathbf{u }^T\tilde{\mathbf{J }}^c\mathbf{u }> 0. \end{aligned}$$

(A.6)

The last inequality is ensured by Lemma (2) and this concludes the proof. $\square$