MAP approximation to the variational Bayes Gaussian mixture model and application

Abstract

The learning of variational inference can be widely seen as first estimating the class assignment variable and then using it to estimate parameters of the mixture model. The estimate is mainly performed by computing the expectations of the prior models. However, learning is not exclusive to expectation. Several authors report other possible configurations that use different combinations of maximization or expectation for the estimation. For instance, variational inference is generalized under the expectation–expectation (EE) algorithm. Inspired by this, another variant known as the maximization–maximization (MM) algorithm has been recently exploited on various models such as Gaussian mixture, Field-of-Gaussians mixture, and sparse-coding-based Fisher vector. Despite the recent success, MM is not without issue. Firstly, it is very rare to find any theoretical study comparing MM to EE. Secondly, the computational efficiency and accuracy of MM is seldom compared to EE. Hence, it is difficult to convince the use of MM over a mainstream learner such as EE or even Gibbs sampling. In this work, we revisit the learning of EE and MM on a simple Bayesian GMM case. We also made theoretical comparison of MM with EE and found that they in fact obtain near identical solutions. In the experiments, we performed unsupervised classification, comparing the computational efficiency and accuracy of MM and EE on two datasets. We also performed unsupervised feature learning, comparing Bayesian approach such as MM with other maximum likelihood approaches on two datasets.

Appendix

In our proposed approach for solving Bayesian GMM using the MM algorithm, we emphasis on its computational efficiency. For completeness, we now discuss how we can obtain convergence by checking the lower bound for each iteration.

$$\begin{aligned} \mathcal {L}\ge & {} {\int }{\int }{\int }{\int } q(z,\mu ,\tau ,\pi )\ln \left[ \frac{p(x,z,\mu ,\tau ,\pi )}{q(z,\mu ,\tau ,\pi )}\right] {\hbox {d}}z\,{\hbox {d}}\mu \,{\hbox {d}}\tau \,{\hbox {d}}\pi \nonumber \\= & {} E\left[ \ln p(x,z,\mu ,\tau ,\pi )\right] -E\left[ \ln q(z,\mu ,\tau ,\pi )\right] \nonumber \\= & {} E\left[ \ln p(x\mid z,\mu ,\tau )\right] +E\left[ \ln p(z\mid \pi )\right] +E\left[ \ln p(\pi )\right] \nonumber \\&+\,E\left[ \ln p(\mu ,\tau )\right] -E\left[ \ln q(z)\right] \nonumber \\&-\,E\left[ \ln q(\pi )\right] -E\left[ \ln q(\mu ,\tau )\right] \end{aligned}$$

(52)

In the lower bound expression above, each expectation function is taken with respect to all of the hidden variables and is simplified using Jensen’s inequality as follows

$$\begin{aligned}&E\left[ \ln p(x\mid z,\mu ,\tau )\right] \nonumber \\&\quad =E\left[ \mathop {\sum }\limits _{k=1}^{K}\mathop {\sum }\limits _{n=1}^{N}\ln \mathcal {N} \left( x_{n}\mid \mu _{k},\tau _{k}^{-1}\right) ^{z_{nk}}\right] \nonumber \\&\quad = \mathop {\sum }\limits _{k=1}^{K} \mathop {\sum }\limits _{n=1}^{N}\left\{ \frac{1}{2} E\left[ \ln \tau _{k}\right] -\frac{E\left[ \tau _{k}\right] }{2} (x_{n}{-}E\left[ \mu _{k}\right] )^{2}\right\} E\left[ z_{nk}\right] \end{aligned}$$

(53)

$$\begin{aligned} E\left[ \ln p(z\mid \pi )\right]= & {} E\left[ \mathop {\sum }\limits _{k=1}^{K} \mathop {\sum }\limits _{n=1}^{N}z_{nk}\ln \pi _{k}\right] \nonumber \\= & {} \mathop {\sum }\limits _{k=1}^{K}\mathop {\sum }\limits _{n=1}^{N}E\left[ z_{nk}\right] E\left[ \ln \pi _{k}\right] \end{aligned}$$

(54)

$$\begin{aligned} E\left[ \ln p(\pi )\right]= & {} E\left[ \mathop {\sum }\limits _{k=1}^{K}\ln {\hbox {Dir}}\left( \pi _{k}\mid \alpha _{0}\right) \right] \nonumber \\= & {} (\alpha _{0}-1) \mathop {\sum }\limits _{k=1}^{K}E\left[ \ln \pi _{k}\right] \end{aligned}$$

(55)

$$\begin{aligned}&E\left[ \ln p(\mu ,\tau )\right] \nonumber \\&\quad =E\left[ \mathop {\sum }\limits _{k=1}^{K}\mathcal {\ln N}\left( \mu _{k}\mid m_{0},(\lambda _{0}\tau _{k})^{-1}\right) {\hbox {Gam}}\left( \tau _{k}\mid a_{0},b_{0}\right) \right] \nonumber \\&\quad =\mathop {\sum }\limits _{k=1}^{K}\left( -\frac{(E\left[ \mu _{k}\right] - m_{0})^{2}}{2\left( \lambda _{0}E\left[ \tau _{k}\right] \right) ^{-1}} +(a_{0}-1)E\left[ \ln \tau _{k}\right] -b_{0}E\left[ \tau _{k}\right] \right) \nonumber \\ \end{aligned}$$

(56)

$$\begin{aligned} E\left[ \ln q(z)\right]= & {} \mathop {\sum }\limits _{n=1}^{N}\mathop {\sum }\limits _{k=1}^{K}\left\{ \ln E\left[ \pi _{k}\right] +\frac{1}{2}\ln E\left[ \tau _{k}\right] \right. \nonumber \\&\quad -\frac{E\left[ \tau _{k}\right] }{2}(x_{n}-E\left[ \mu _{k}\right] )^{2}\Biggr \} E\left[ z_{nk}\right] \end{aligned}$$

(57)

$$\begin{aligned} E\left[ \ln q(\pi )\right]= & {} E\left[ \mathop {\sum }\limits _{n=1}^{N} \mathop {\sum }\limits _{k=1}^{K}E\left[ z_{nk}\right] \ln \pi _{k}\right. \nonumber \\&\left. +(\alpha _{0}-1)\mathop {\sum }\limits _{k=1}^{K}\ln \pi _{k}\right] \nonumber \\= & {} \mathop {\sum }\limits _{n=1}^{N}\mathop {\sum }\limits _{k=1}^{K}E\left[ z_{nk}\right] \ln E\left[ \pi _{k}\right] \nonumber \\&+(\alpha _{0}-1)\mathop {\sum }\limits _{k=1}^{K}\ln E\left[ \pi _{k}\right] \end{aligned}$$

(58)

$$\begin{aligned}&E\left[ \ln q(\mu ,\tau )\right] \nonumber \\&\quad =E\left[ \mathop {\sum }\limits _{k=1}^{K}\left( \mathop {\sum }\limits _{n=1}^{N}\left\{ \frac{\ln \tau _{k}}{2}-\frac{\tau _{k}}{2}(x_{n}-E\left[ \mu _{k}\right] )^{2}\right\} E\left[ z_{nk}\right] \right. \right. \nonumber \\&\qquad \left. \left. -\frac{(E\left[ \mu _{k}\right] -m_{0})^{2}}{2(\lambda _{0}\tau _{k})^{-1}}+(a_{0}-1)\ln \tau _{k}-b_{0}\tau _{k}) \right) \right] \nonumber \\&\quad =\mathop {\sum }\limits _{k=1}^{K}\left( \mathop {\sum }\limits _{n=1}^{N}\left\{ \frac{E\left[ \ln \tau _{k}\right] }{2}-\frac{E\left[ \tau _{k}\right] }{2}(x_{n}-E\left[ \mu _{k}\right] )^{2}\right\} E\left[ z_{nk}\right] \right. \nonumber \\&\qquad \left. -\frac{(E\left[ \mu _{k}\right] -m_{0})^{2}}{2(\lambda _{0}E\left[ \tau _{k}\right] )^{-1}}+(a_{0}-1)E\left[ \ln \tau _{k}\right] -b_{0}\left[ E\tau _{k}\right] \right) \end{aligned}$$

(59)

In practice, it is not necessary to compute lower bound as we can visualize the convergence by checking the changes in weight of each cluster using the equation below, as seen in the experiments later on.

$$\begin{aligned} \pi _{k}=\frac{{\sum }_{n=1}^{N}E\left[ z_{nk}\right] }{{\sum }_{j=1}^{K}{\sum }_{n=1}^{N}E\left[ z_{nj}\right] } \end{aligned}$$

(60)