Learning node labels with multi-category Hopfield networks

Abstract

In several real-world node label prediction problems on graphs, in fields ranging from computational biology to World Wide Web analysis, nodes can be partitioned into categories different from the classes to be predicted, on the basis of their characteristics or their common properties. Such partitions may provide further information about node classification that classical machine learning algorithms do not take into account. We introduce a novel family of parametric Hopfield networks (m-category Hopfield networks) and a novel algorithm (Hopfield multi-category—HoMCat), designed to appropriately exploit the presence of property-based partitions of nodes into multiple categories. Moreover, the proposed model adopts a cost-sensitive learning strategy to prevent the remarkable decay in performance usually observed when instance labels are unbalanced, that is, when one class of labels is highly underrepresented than the other one. We validate the proposed model on both synthetic and real-world data, in the context of multi-species function prediction, where the classes to be predicted are the Gene Ontology terms and the categories the different species in the multi-species protein network. We carried out an intensive experimental validation, which on the one hand compares HoMCat with several state-of-the-art graph-based algorithms, and on the other hand reveals that exploiting meaningful prior partitions of input data can substantially improve classification performances.

Appendices

Appendix 1: Convergence proof

The following fact adapts the convergence proof for Hopfield networks with neuron activation values $\{\sin \alpha , -\cos \alpha \}$ [21] to the case with $m\,\ge\,2$ categories of neurons $V_1, V_2,\ldots , V_m$, each with a different couple of activation values $\{\sin \alpha _k, -\cos \alpha _k\}$, for neurons belonging to category $V_k$.

Fact 5

A m-Category Hopfield Network ${\fancyscript{H}} = \langle \,\varvec{W}, \varvec{b}, \alpha _1, \ldots , \alpha _m, \varvec{\lambda }\rangle$ with neurons $V=\{ 1, 2,\ldots , n\}$ and asynchronous dynamics (1), starting from any given network state, eventually reaches a stable state at a local minimum of the energy function.

Proof

We observe that the energy (2) is equivalent to the following

$$\begin{aligned} E(\varvec{x}) = -\frac{1}{2} \sum _{i,j=1 }^n{ W_{ij}x_i x_j } + \sum _{i = 1}^{n}{x_i \lambda _i} \end{aligned}$$

(11)

During the iteration $t+1$, each unit i is selected and updated according to the update rule (1). To simplify the notation we set $x'_i = x_i(t+1)$ and $x_i = x_i(t)$. If the unit i does not change its state, then the energy of the system does not change as well, and we set $x'_i = x_i$ . Otherwise, the network reaches a new global state $\varvec{x}' = (x_1, \ldots , x'_i, \ldots , x_n)$ having energy $E(\varvec{x}'$). Since by definition $W_{ii} = 0$, the difference between $E(\varvec{x}'$) and $E(\varvec{x}$) is given by all terms in the summation (11) which contain $x'_i$ and $x_i$, that is,

$$\begin{aligned} E(\varvec{x}')-E(\varvec{x})= & - \sum _{x'_i \ne x_i} \left( (x'_i - x_i)\left( \sum _{j=1}^n{ W_{ij}x_j } - \lambda _i\right) \right) \nonumber \\= & - \sum _{x'_i \ne x_i} \left( (x'_i - x_i)B_i \right) \end{aligned}$$

(12)

where $B_i = \sum _{j=1}^n{ W_{ij}x_j } - \lambda _i$. The factor $\frac{1}{2}$ disappears from the computation because the terms $W_{ij} x_i x_j$ appear twice in the double summation in (11). If $B_i > 0$, it means that $x'_i = \sin \alpha _{b_i}$ and $x_i = -\cos \alpha _{b_i}$ (see 1); since $0\le \alpha _{b_i}< \frac{\pi }{2}$, $(x'_i - x_i) > 0$ and $E(\varvec{x}') - E(\varvec{x}) < 0$. If $B_i < 0$, it means that $x'_i = -\cos \alpha _{b_i}$ and $x_i = \sin \alpha _{b_i}$ and again $E(\varvec{x}') - E(\varvec{x}) < 0$. If $B_i$ = 0, the energy value does not change after the update. Hence, the energy (11) is a monotonically decreasing function. Moreover, since the connections $W_{ij}$ are nonnegative, the energy E is lower bounded by the value

$$\begin{aligned} E_{\mathrm {LB}}=-\sum _{i = 1}^n\sum _{j=1}^n \left( \sin (\alpha _{b_i}) \sin (\alpha _{b_j}) W_{ij} - \lambda _i\right) \end{aligned}$$

(13)

and the dynamics is guaranteed to converge to a fixed point, corresponding to a local minimum of the energy.$\square$

Appendix 2: Generative process for synthetic data

We provide here detailed information about the generative process of the two families of experiments on synthetic data described in Sect. 5.1. The generator used in the experiments is available from the authors upon request.

The main prerequisite to perform the first family of experiments is the generation of separable patterns. For this purpose we decided to work in the $\Delta$ space, where each node $i\in V$ is mapped in a 2m-dimensional point $\Delta (i)\equiv \{\Delta ^+_1(i),\Delta ^-_1(i),\ldots ,\Delta ^+_m(i),\Delta ^-_m(i)\}$ where

$$\Delta ^+_k(i)= \sum _{j\in V_k\cap S^+}\!\!\!\!\!\!\!\!w_{ij},\quad \Delta ^-_k(i)=\sum _{j\in V_k\cap S^-} w_{ij}, \quad k=1,\ldots ,m$$

(14)

and, according to the notation used throughout the paper, $V_k$ is the k-th category, $S^+$ (resp. $S^-$) the set of positive (resp. negative) labeled instances, and $W_{ij}$ the weight connecting nodes i and j.

Namely, after having fixed a labeling imbalance $\epsilon$, we randomly generated a set of n points $\Delta (i)$ in the unitary hypercube, partitioned them into m categories according to their Euclidean similarity in the $\Delta$ space. Then, we drew $\alpha _k$ uniformly for each category k, and computed the threshold $\lambda$ guaranteeing the correct labeling imbalance, i.e., satisfying the following equation:

$$\begin{aligned} \sum _{i=1}^n{\rm {HS}}\left( \sum _{k=1}^m\left( \sin {\alpha _k}\Delta ^+_k(i)-\cos {\alpha _k}\Delta ^-_k(i) \right) -\lambda \right) =\frac{n \epsilon }{\epsilon +1} \end{aligned}$$

(15)

where ${\rm {HS}}$ is the Heaviside step function, i.e., ${\rm {HS}}(x)=1$ if $x\ge 0$, 0 otherwise.

We run the proposed learning algorithm starting with m categories, and subsequently evaluated its performance on a lower number $\widetilde{m}< m$ of them. Although other strategies could be adopted, the decision of working directly in the $\Delta$ space was motivated by the need of exerting more control on the construction of separable instances. We select the number $\widetilde{m}$ of categories to be investigated as a divisor of m. For instance, in case $m=4$, by choosing $\widetilde{m}=2$ we may easily consider the reduced $\Delta$ space obtained by associating with node i the point $\{\Delta _1^+(i)+\Delta _2^+(i),\ \Delta _1^-(i)+\Delta _2^-(i),\ \Delta _3^+(i)+\Delta _4^+(i),\ \Delta _3^-(i)+\Delta _4^-(i)\}$ which, in turn, can be coupled with two parameters ($\alpha _1$ and $\alpha _2)$ out of four. Of course, in order to preserve the concept of category, this aggregation should be performed by merging together the most similar categories, where similarity is again defined in terms of Euclidean metric in the $\Delta$ space.

Given the set of nodes V, partitioned into positives $V_+$ and negatives $V_-$, the vector $\varvec{b}$ partitioning V in m disjoint categories and the corresponding parameters $(\alpha _1, \ldots , \alpha _m, \lambda )$, we compute the initial labeling ${\tilde{\varvec{x}}}$ as follows:

$$\begin{aligned} {\tilde{x}}_{i} = \left\{ \begin{array}{rl} \sin \alpha _{b_{i}} \,& \text{ if } i \in V_+\\ -\cos \alpha _{b_{i}} & \text{ if } i \in V_-\\ \end{array} \right. \end{aligned}$$

(16)

Then, by exploiting the associative memory functionality of the Hopfield network, we generated the weight matrix $\varvec{W}$ through the naive Hebbian rule [25] in order to store the pattern ${\tilde{\varvec{x}}}$, i.e., ${\tilde{\varvec{x}}}$ becomes a fixed point of the dynamics [26]. Moreover, to make the convergence to ${\tilde{\varvec{x}}}$ as much complex as possible and hence to amplify the fixed point perturbations, we learned the weights $\varvec{W}$ in order to store also other $\mu$ randomly generated patterns with the same label imbalance $\epsilon = \frac{|V_+|}{|V_-|}$, where $\mu \simeq 0.13|V|$ is the capacity of the network when trained with the naive Hebbian rule. Then, after having hidden the labels for a subset $U \subset V$ chosen according to the fixed $\frac{|S|}{|V|}$ ratio, for each value of the standard deviation $\sigma$ modulating the injected noise, we run the network restricted to U till convergence and finally we measure the perturbation of the attained stable state with respect to ${\tilde{\varvec{x}}}_u$.