Adaptively local consistent concept factorization for multi-view clustering

Abstract

Many real-world datasets consist of multiple views of data items. The rough method of combining multiple views directly through feature concatenation cannot uncover the optimal latent structure shared by multiple views, which would benefit many data analysis applications. Recently, multi-view clustering methods have emerged and been applied to solving many machine learning problems. However, most multi-view clustering methods ignore the joint information of multi-view data or neglect the quality difference between different views of data, resulting in decreased learning performance. In this paper, we discuss a multi-view clustering algorithm based on concept factorization that effectively fuses useful information to derive a better representation for more effective clustering. We incorporate two regularizers into the concept factorization framework. Specifically, one regularizer is adopted to force the coefficient matrix to move smoothly on the underlying manifold. The other regularizer is used to learn the latent clustering structure from different views. Both of these regularizers are incorporated into the concept factorization framework to learn the latent representation matrix. Optimization problems are solved efficiently via an iterative algorithm. The experimental results on seven real-world datasets demonstrate that our approach outperforms the state-of-the-art multi-view clustering algorithms.

(Proofs of theorem1)

The objective function O of ALCCF in Eq. (10) is bounded from below by zero obviously. In order to prove Theorem 1, we need show that the function O is nonincreasing under the update formulae Eq. (13), (14). Since the last two terms of O are only related to $\mathbf{{V}}$, the update formula for $\mathbf{{U}}$ in ALCCF is same as CF. Therefore, the convergence proof of CF can be used to show that O is nonincreasing under the update formula in Eq. (13). Please see (Lee and Seung 2001; Brunet et al. 2004) for details.

Next, we will prove that O is nonincreasing under the update formula Eq. 14. To prove it, we will use auxiliary function similar to that used in the expectation–maximization algorithm (Dempster et al. 1977). The auxiliary function is defined as follows; $G(v,v')$ is an auxiliary function for F(v) if the conditions

$$\begin{aligned} G(v,v') \ge F(v),\quad G(v,v) = F(v) \end{aligned}$$

are satisfied.

Now, we show a very useful lemma as follows.

Lemma 1

If G is an auxiliary function of F, then F is nonincreasing under the update

$$\begin{aligned} {v^{(K + 1)}} = \mathop {\arg \min }\limits _v G(v,{v^{(K)}}) \end{aligned}$$

(20)

Proof

$$\begin{aligned} F({v^{(K + 1)}}) \le G({v^{(K + 1)}},{v^{(K)}}) \le G({v^{(K)}},{v^{(K)}}) = F({v^{(K)}}) \end{aligned}$$

$\square$

Next we will show that with a proper auxiliary function the update step for $\mathbf{{V}}$ in Eq.(14) is exactly the update in Eq. (20).

We use ${F_{ab}}$ to denote the part of O which is only relevant to ${v_{ab}}$. And we can easily check that

$$\begin{aligned} F_{ab}'= & {} {\left( {\frac{{\partial O}}{{\partial {\mathbf{{V}}^{(v)}}}}} \right) _{ab}} = \left( - 2{\mathbf{{K}}^{(v)}}{\mathbf{{U}}^{(v)}} + 2{\mathbf{{V}}^{(v)}}{({\mathbf{{U}}^{(v)}})^T}{\mathbf{{K}}^{(v)}}{\mathbf{{U}}^{(v)}} \right. \nonumber \\&\left. + \sum \limits _{s = 1}^{{n_v}} {{\theta _{vs}}(\mathrm{{2}}{\mathbf{{V}}^{(s)}} - 2{({\mathbf{{V}}^{(v)}})^T})} \mathrm{{ + }}{2\alpha \mathbf{{LV}}} \right) _{ab} \nonumber \\\end{aligned}$$

(21)

$$\begin{aligned} F_{ab}^{''}= & {} {\left( {\frac{{\partial O}}{{\partial {\mathbf{{V}}^{(v)}}}}} \right) _{ab}} = 2({({\mathbf{{U}}^{(v)}})^T}{\mathbf{{K}}^{(v)}}{\mathbf{{U}}^{(v)}})_{bb}\nonumber \\&+ 2\sum \limits _{s = 1}^{{n_v}} {{\theta _{vs}}} + 2\alpha \mathbf{{L}}_{aa}^{(v)} \end{aligned}$$

(22)

Since the update is essentially element-wise, it is sufficient to show that each ${F_{ab}}$ is nonincreasing under the update step of Eq. (14).

Lemma 2

Function

$$\begin{aligned}&G(v,v_{ab}^{(K)}) = {F_{ab}}(v_{ab}^{(K)}) + F_{ab}'(v_{ab}^{(K)})(v -v_{ab}^{(K)})\nonumber \\&\quad + \frac{{{{\left( {{\mathbf{{V}}^{(v)}}{{({\mathbf{{U}}^{(v)}})}^T}{\mathbf{{K}}^{(v)}}{\mathbf{{U}}^{(v)}} \quad \quad + \sum \limits _{s = 1}^{{n_v}} {{\theta _{vs}}{\mathbf{{V}}^{(v)}}} + \alpha {\mathbf{{D}}^{(v)}}{\mathbf{{V}}^{(v)}}} \right) }_{ab}}}}{{v_{ab}^{(K)}}}{(v - v_{ab}^{(K)})^2} \quad \end{aligned}$$

(23)

is an auxiliary function for ${F_{ab}}$, the part of O which is only relevant to ${v_{ab}}$.

Proof

Firstly, it is obvious that $G(v,v) = {F_{ab}}(v)$.

Secondly, we compare the Taylor series expansion of ${F_{ab}}(v)$

$$\begin{aligned}&{F_{ab}}(v) = {F_{ab}}(v_{ab}^{(K)}) + F_{ab}'(v_{ab}^{(K)})(v - v_{ab}^{(K)})\nonumber \\&\quad + {\left( ({{{({\mathbf{{U}}^{(v)}})^T}{\mathbf{{K}}^{(v)}}{\mathbf{{U}}^{(v)}})_{bb} + \sum \nolimits _{s = 1}^{{n_v}} {{\theta _{vs}}{\mathbf{{V}}^{(s)}}} + \alpha \mathbf{{L}}_{aa}^{(v)})}} \right) }{(v - v_{ab}^{(K)})^2} \nonumber \\ \end{aligned}$$

(24)

with Eq.(23) to find that $G(v,v_{ab}^{(K)}) \ge {F_{ab}}(v)$ is equivalent to

$$\begin{aligned} \begin{aligned}&\frac{{{\mathbf{{V}}^{(v)}}{({\mathbf{{U}}^{(v)}})^T}{\mathbf{{K}}^{(v)}}{\mathbf{{U}}^{(v)}} + \sum \nolimits _{s = 1}^{{n_v}} {{\theta _{vs}}{\mathbf{{V}}^{(v)}}} + \alpha {\mathbf{{D}}^{(v)}}{\mathbf{{V}}^{(v)}}}}{{v_{ab}^{(K)}}}\\&\ge {\mathbf{{K}}^{(v)}}{\mathbf{{U}}^{(v)}} + \sum \limits _{s = 1}^{{n_v}} {{\theta _{vs}}{\mathbf{{V}}^{(s)}}} + \alpha \mathbf{{L}}_{aa}^{(v)}. \end{aligned} \end{aligned}$$

(25)

We have

$$\begin{aligned} \begin{aligned}&{\left( {{\mathbf{{V}}^{(v)}}{({\mathbf{{U}}^{(v)}})^T}{\mathbf{{K}}^{(v)}}{\mathbf{{U}}^{(v)}}} \right) _{ab}} \\&\quad = \sum \limits _{l = 1}^k {v_{al}^{(K)}{{({({\mathbf{{U}}^{(v)}})^T}{\mathbf{{K}}^{(v)}}{\mathbf{{U}}^{(v)}})}_{lb}}} \\&\quad \ge v_{ab}^{(K)}{({({\mathbf{{U}}^{(v)}})^T}{\mathbf{{K}}^{(v)}}{\mathbf{{U}}^{(v)}})_{bb}} \end{aligned} \end{aligned}$$

(26)

and

$$\begin{aligned} \begin{aligned} \alpha {({\mathbf{{D}}^{(v)}}{\mathbf{{V}}^{(v)}})_{ab}}&= \alpha \sum \nolimits _{j = 1}^M {\mathbf{{D}}_{aj}^{(v)}v_{jb}^{(K)} \ge } \alpha (\mathbf{{D}}_{aa}^{(v)}v_{ab}^{(K)})\\&\ge \alpha {({\mathbf{{D}}^{(v)}} - {\mathbf{{S}}^{(v)}})_{aa}}v_{ab}^{(K)} = \alpha \mathbf{{L}}_{aa}^{(v)}v_{ab}^{(K)}. \end{aligned} \end{aligned}$$

(27)

Therefore, Eq. (25) holds and we get $G(v,v_{ab}^{(K)}) \ge {F_{ab}}(v)$. $\square$

Now, we can show the convergence of Theorem 1:

Proof

of Theorem 1 Replacing $G(v,v_{ab}^{(K)})$ in Eq. (20) by Eq.(23) results in the update rule:

$$\begin{aligned}&v_{ab}^{(K + 1)} = v_{ab}^{(K)} \nonumber \\&\quad - v_{ab}^{(K)}\frac{{F_{ab}'(v_{ab}^{(K)})}}{{2{{\left( {{\mathbf{{V}}^{(v)}}{{({\mathbf{{U}}^{(v)}})}^T}{\mathbf{{K}}^{(v)}}{\mathbf{{U}}^{(v)}}} \right) }_{ab}} + 2{{\left( {\sum \nolimits _{s = 1}^{{n_v}} {{\theta _{vs}}{\mathbf{{V}}^{(v)}}} } \right) }_{ab}} + 2\alpha {{\left( {{\mathbf{{D}}^{(v)}}{\mathbf{{V}}^{(v)}}} \right) }_{ab}}}}\nonumber \\&\quad = v_{ab}^{(K)}\frac{{{{\left( {{\mathbf{{K}}^{(v)}}{\mathbf{{U}}^{(v)}} + \sum \nolimits _{s = 1}^{{n_v}} {{\theta _{vs}}{\mathbf{{V}}^{(s)}}} + \alpha {\mathbf{{S}}^{(v)}}{\mathbf{{V}}^{(v)}}} \right) }_{ab}}}}{{{{\left( {{\mathbf{{V}}^{(v)}}{{({\mathbf{{U}}^{(v)}})}^T}{\mathbf{{K}}^{(v)}}{\mathbf{{U}}^{(v)}} + \sum \nolimits _{s = 1}^{{n_v}} {{\theta _{vs}}{\mathbf{{V}}^{(v)}}} + \alpha {\mathbf{{D}}^{(v)}}{\mathbf{{V}}^{(v)}}} \right) }_{ab}}}} \end{aligned}$$

(28)

Since Eq.(20) is an auxiliary function, ${F_{ab}}$ is nonincreasing under this updating rule. $\square$