Improvement of deep cross-modal retrieval by generating real-valued representation

Background and literature survey

The biggest challenge in natural language processing (NLP) is to design algorithms, which allows computers to understand natural language to perform different tasks. It is recommended to represent each word in form of a vector as most of the machine learning algorithms are not capable of processing text directly in its raw form. The process of converting a word into a vector is called vectorization, which represents each word into vector space. Broadly the vectorization methods are categorized into (a) local representation method and (b) distributional representation method. The most common local representation method is called bag-of-words (BoW), where each word is represented as ${\mathbb{R}}^{|V|\times 1}$ vector with all 0’s and one 1 at the index of the word in the corpus. However, the generated matrix is sparse in nature, which is inefficient for computation, and the similarity between different words is not preserved, as the inner product between two different one-hot vectors is zero. On the other hand, in distributional representation, each word w_i in the corpus is represented by featurized representation, which is denoted as w_i ∈ ${\mathcal{R}}^d$, where each word is represented in d dimensions.

The distributional representation generates distributional word vectors, which follows the concept of the distributional hypothesis (Mikolov et al., 2013b), which states that words that occur in the same contexts tend to have similar meanings. The distributional word vectors are generated from count-based models or prediction based models. The count-based models generate implicit distributional vectors using dimensionality-reduction techniques, which map data in the high-dimensional space to a space of fewer latent dimensions. The most popular method is singular value decomposition (SVD) (Van Loan, 1976), which generates embedding of each word in the vocabulary using matrix factorization, but fails when the dimensionality of matrices is very large as the computational cost for m × n matrix is O (mn²). The most popular count-based method is Glove (Mikolov et al., 2013b), which generates implicit vector and achieve better performance in comparison with other matrix-based methods. Another broader classification for the generation of distributional word vector is prediction based models, which are neural network based algorithms. Such models directly create low-dimensional implicit distributional representations. An example of such a model is word2vec. The below section covers a detailed description of the generation of word vectors using Glove and word2vec.

word2vec

The word2vec is a feed-forward based neural network, which has two algorithms: continuous bag-of-words (CBOW) and skip-gram (SG) (Mikolov et al., 2013a, 2013b). Figure 2 (Mikolov et al., 2013a) shows a description of CBOW and SG where CBOW predicts the probability of center word w(t) and SG predicts the probability of surrounding words w(t + j).

Working of SG model

SG predicts the probability of surrounding given a center word. For training of the network, there is an objective function that maximizes the probability of surrounding words given center word for each position of text t in the window size of m.

(1) $${\mathrm{J}}^{\mathrm{\prime }}\left(\theta \right)=\prod _{\mathrm{t}=1}^{\mathrm{T}}\prod _{\begin{array}{c}-m\le j\le m\\ j\ne 0\end{array}}\mathrm{P}({\mathrm{w}}_{\mathrm{t}+\mathrm{j}}|{\mathrm{w}}_{\mathrm{t}};\theta )$$Here $\mathrm{P}({\mathrm{w}}_{\mathrm{t}+\mathrm{j}}|{\mathrm{w}}_{\mathrm{t}})$ is probability of surrounding words ${\mathrm{w}}_{\mathrm{t}+\mathrm{j}}$ given center word w_t. Equation (1) can be rewritten as Eq. (2)

${\mathrm{w}}_{\mathrm{t}}$can be rewritten as

(2) $$\mathrm{J}\left(\theta \right)=-{\displaystyle \frac{1}{\mathrm{T}}\sum _{\mathrm{t}=1}^{\mathrm{T}}\sum _{\begin{array}{c}-m\le j\le m\\ j\ne 0\end{array}}\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{P}({\mathrm{w}}_{\mathrm{t}+\mathrm{j}}|{\mathrm{w}}_{\mathrm{t}};\theta )}$$ $\mathrm{P}({\mathrm{w}}_{\mathrm{t}+\mathrm{j}}|{\mathrm{w}}_{\mathrm{t}})$ can be rewritten as $\mathrm{P}\left(\mathrm{o}|\mathrm{c}\right)$, which specifies the probability of surrounding words o given center word c, and softmax function is used to generate probability.

(3) $$\mathrm{P}\left(\mathrm{o}|\mathrm{c}\right)={\displaystyle \frac{{\mathrm{e}}^{{\mathrm{u}}_0^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}}{{\sum }_{\mathrm{w}=1}^{\mathrm{V}}{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{w}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}}}$$where u₀ specifies vector representation of the surrounding word at index 0 and v_c specifies the vector representation of center word. Equation (3) can be applied in Eq. (2),

(4) $$\mathrm{J}\left(\theta \right)=\log {\displaystyle \frac{{\mathrm{e}}^{{\mathrm{u}}_0^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}}{{\sum }_{\mathrm{w}=1}^{\mathrm{V}}{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{w}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}}}$$Now, the objective is to optimize v_c and u_w. So need to take the derivative with respect to ${\mathrm{v}}_{\mathrm{c}}$ and ${\mathrm{u}}_{\mathrm{w}}$.

(5) $$\mathrm{J}\left(\theta \right)={\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\mathrm{v}}_{\mathrm{c}}}\log {\mathrm{e}}^{{\mathrm{u}}_0^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}-{\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\mathrm{v}}_{\mathrm{c}}}\log \sum _{\mathrm{w}=1}^{\mathrm{V}}{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{w}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}}}$$where,

(5a) $${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\mathrm{v}}_{\mathrm{c}}}\log {\mathrm{e}}^{{\mathrm{u}}_0^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}=u_0}$$

(5b) $${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\mathrm{v}}_{\mathrm{c}}}\log \sum _{\mathrm{w}=1}^{\mathrm{V}}{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{w}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}={\displaystyle \frac{1}{{\sum }_{\mathrm{w}=1}^{\mathrm{V}}{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{w}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}}\times {\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\mathrm{v}}_{\mathrm{c}}}\sum _{\mathrm{w}=1}^{\mathrm{V}}{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{w}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}}}}$$

$$={\displaystyle \frac{1}{{\sum }_{\mathrm{w}=1}^{\mathrm{V}}{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{w}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}}\times {\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\mathrm{v}}_{\mathrm{c}}}\sum _{\mathrm{x}=1}^V{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{x}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}}}$$

$$={\displaystyle \frac{1}{{\sum }_{\mathrm{w}=1}^{\mathrm{V}}{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{w}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}}\times \sum _{\mathrm{x}=1}^{\mathrm{V}}{\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\mathrm{v}}_{\mathrm{c}}}{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{x}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}}}$$

$$={\displaystyle \frac{1}{{\sum }_{\mathrm{w}=1}^{\mathrm{V}}{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{w}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}}\times \sum _{\mathrm{x}=1}^V{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{x}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}{\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\mathrm{v}}_{\mathrm{c}}}{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{x}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}}}$$

$$={\displaystyle \frac{1}{{\sum }_{\mathrm{w}=1}^{\mathrm{V}}{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{w}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}}\times \sum _{x=1}^V{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{x}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}{u}_x}$$

Combine (5.a) and (5.b),

$$\mathrm{J}\left(\theta \right)={\mathrm{u}}_0-{\displaystyle \frac{1}{{\sum }_{\mathrm{w}=1}^{\mathrm{V}}{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{w}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}}\times \sum _{\mathrm{x}=1}^{\mathrm{V}}{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{x}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{c}}}{\mathrm{u}}_{\mathrm{x}}}$$

Above equation can be rewritten as

(6) $$\mathrm{J}\left(\theta \right)={\mathrm{u}}_0-\sum _{\mathrm{x}=1}^{\mathrm{V}}\mathrm{P}\left(\mathrm{x}|\mathrm{c}\right).{\mathrm{u}}_{\mathrm{x}}$$Where, u₀ is the actual ground truth and P(x|c) is the probability of each surrounding word x given the center word c, and u_x is the average of all possible surrounding words. So cost function of SG guarantees that the probability of occurring surrounding words maximizes given a center word.

Working of continuous bag-of-words model

CBOW predicts the probability of a center word given surrounding words. Input to CBOW is d dimensional one-hot vector representation of a center word. The representation of a center word is generated by multiplying d dimensional vector with the weight matrix W of size p × d where p is the featurized representation of a word.

(7) $${\mathrm{h}}_{\mathrm{p}\times 1}={\mathrm{W}}_{\mathrm{p}\times \mathrm{d}}^{\mathrm{T}}{\mathrm{x}}_{\mathrm{d}}={\mathrm{V}}_{\mathrm{c}}$$

The above representation is a vector representation of the center word V_c. The representation of outside words is generated by multiplying center representation with the weight matrix W′.

(8) $${\mathrm{u}}_{\mathrm{d}\times 1}={{\mathrm{W}}^{\prime }}_{\mathrm{d}\times \mathrm{p}}^{\mathrm{T}}{\mathrm{h}}_{\mathrm{p}\times 1}={\mathrm{V}}_{\mathrm{w}}^{\mathrm{T}}{\mathrm{V}}_{\mathrm{c}}$$Where, V_c is a vector representation of the center word and ${\mathrm{V}}_{\mathrm{w}}$ is a vector representation of surrounding words. It is a prediction-based model so need to find the probability of a word given the center word P(w|c).

(9) $${\mathrm{y}}_{\mathrm{i}}=\mathrm{P}\left(\mathrm{w}|\mathrm{c}\right)=\phi \left({\mathrm{u}}_{\mathrm{i}}\right)={\displaystyle \frac{{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{i}}}}{{\sum }_{{\mathrm{i}}^{\prime }}{\mathrm{e}}^{{\mathrm{u}}_{\mathrm{i}}}}={\displaystyle \frac{{\mathrm{e}}^{{\mathrm{V}}_{\mathrm{w}}^{\mathrm{T}}{\mathrm{V}}_{\mathrm{c}}}}{{\sum }_{{\mathrm{w}}^{\prime }\in \mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}}{\mathrm{e}}^{{\mathrm{V}}_{{\mathrm{w}}^{\prime }}^{\mathrm{T}}{\mathrm{V}}_{\mathrm{c}}}}}}$$

There is an objective function, which maximizes P(w|c) by adjusting the hyper parameters i.e., ${\mathrm{v}}_{\mathrm{c}}$ and v_w.

(10) $$\mathrm{l}\left(\theta \right)=\sum _{\mathrm{w}\in \mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}}\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{P}(\mathrm{w}|\mathrm{c};\theta )$$Put value of Eq. (9) in Eq. (10),

$$\mathrm{l}\left(\theta \right)=\sum _{\mathrm{w}\in \mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}}\mathrm{l}\mathrm{o}\mathrm{g}{\displaystyle \frac{{\mathrm{e}}^{{\mathrm{V}}_{\mathrm{w}}^{\mathrm{T}}{\mathrm{V}}_{\mathrm{c}}}}{{\sum }_{{\mathrm{w}}^{\prime }\in \mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}}{\mathrm{e}}^{{\mathrm{V}}_{{\mathrm{w}}^{\prime }}^{\mathrm{T}}{\mathrm{V}}_{\mathrm{c}}}}}$$

$$=\sum _{\mathrm{w}\in \mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}}\mathrm{l}\mathrm{o}\mathrm{g}{\mathrm{e}}^{{\mathrm{V}}_{\mathrm{w}}^{\mathrm{T}}{\mathrm{V}}_{\mathrm{c}}}-\sum _{\mathrm{w}\in \mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}}\mathrm{l}\mathrm{o}\mathrm{g}{\displaystyle \frac{1}{{\sum }_{{\mathrm{w}}^{\prime }\in \mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}}{\mathrm{e}}^{{\mathrm{V}}_{{\mathrm{w}}^{\prime }}^{\mathrm{T}}{\mathrm{V}}_{\mathrm{c}}}}}$$

To optimize the hyper parameter, need to take derivation with respect to v_c and v_w.

$${\displaystyle \frac{\mathrm{\partial }l}{\mathrm{\partial }{\mathrm{v}}_{\mathrm{w}}}=\sum _{\mathrm{w}\in \mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}}{\mathrm{v}}_{\mathrm{c}}-{\displaystyle \frac{1}{{\sum }_{{\mathrm{w}}^{\mathrm{\prime }}\in \mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}}{\mathrm{e}}^{{\mathrm{V}}_{{\mathrm{w}}^{\mathrm{\prime }}}^{\mathrm{T}}{\mathrm{V}}_{\mathrm{c}}}}\times {\displaystyle \frac{\mathrm{\partial }l}{\mathrm{\partial }w}{\mathrm{e}}^{{\mathrm{V}}_{{\mathrm{w}}^{\prime }}^{\mathrm{T}}{\mathrm{V}}_{\mathrm{c}}}}}}$$

$$=\sum _{\mathrm{w}\in \mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}}{\mathrm{v}}_{\mathrm{c}}-{\displaystyle \frac{1}{{\sum }_{{\mathrm{w}}^{\mathrm{\prime }}\in \mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}}{\mathrm{e}}^{{\mathrm{V}}_{{\mathrm{w}}^{\mathrm{\prime }}}^{\mathrm{T}}{\mathrm{V}}_{\mathrm{c}}}}\times {\mathrm{e}}^{{\mathrm{V}}_{{\mathrm{w}}^{\mathrm{\prime }}}^{\mathrm{T}}{\mathrm{V}}_{\mathrm{c}}}\times {\mathrm{v}}_{\mathrm{c}}}$$

$$=\sum _{\mathrm{w}\in \mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}}{\mathrm{v}}_{\mathrm{c}}-\mathrm{P}(\mathrm{w}|\mathrm{c}){\mathrm{v}}_{\mathrm{c}}$$

$$=\sum _{\mathrm{w}\in \mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}}{\mathrm{v}}_{\mathrm{c}}[1-\mathrm{P}(\mathrm{w}|\mathrm{c})]$$

For optimization, gradient descent algorithm is applied and hyper parameter is optimized.

(11) $${\mathrm{V}}_{\mathrm{w}}={\mathrm{V}}_{\mathrm{w}}-\eta {\mathrm{V}}_{\mathrm{c}}\left[1-\mathrm{P}\left(\mathrm{w}|\mathrm{c}\right)\right]$$

Similar steps are followed for hyperparameter V_c.

(12) $${\mathrm{V}}_{\mathrm{c}}={\mathrm{V}}_{\mathrm{c}}-\eta {\mathrm{V}}_{\mathrm{w}}\left[1-\mathrm{P}\left(\mathrm{w}|\mathrm{c}\right)\right]$$

So CBOW and SG preserve the semantic similarity by following the distributional hypothesis in comparison with BoW model.

Glove (Count based method)

In contrast to word2vec, Glove captures the co-occurrence of a word from the entire corpus (Pennington, Socher & Manning, 2014). Glove first constructs the global co-occurrence matrix ${\mathrm{X}}_{\mathrm{i}\mathrm{j}}$, which gives information about how often words i and j appear in the entire corpus. The size of the matrix can be minimized by the factorization process, which generates a lower-dimensional matrix such that reconstruction loss is minimized. The objective of the Glove model is to learn the vectors ${\mathrm{v}}_{\mathrm{i}}$ (vector representation of i word) and ${\mathrm{v}}_{\mathrm{j}}$ (vector representation of j word), which are fruitful to information which is in the form of ${\mathrm{X}}_{\mathrm{i}\mathrm{j}}$. The similarity between words is captured by finding the inner product ${\mathrm{v}}_{\mathrm{i}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{j}}$, which gives similarity between words i and j. This similarity is proportional to P(j|i) or P(i|j), where P(j|i) gives the probability of word j given the word i.

$${\mathrm{v}}_{\mathrm{i}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{j}}=\log \mathrm{P}(\mathrm{j}|\mathrm{i})$$where, $\log \mathrm{P}(\mathrm{j}|\mathrm{i})={\displaystyle \frac{{\mathrm{X}}_{\mathrm{i}\mathrm{j}}}{\sum {\mathrm{X}}_{\mathrm{i}\mathrm{j}}}={\displaystyle \frac{{\mathrm{X}}_{\mathrm{i}\mathrm{j}}}{{\mathrm{X}}_{\mathrm{i}}}}}$

(13) $${\mathrm{v}}_{\mathrm{i}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{j}}=\log {\mathrm{X}}_{\mathrm{i}\mathrm{j}}-\log {\mathrm{X}}_{\mathrm{i}}$$Similarly,

(14) $${\mathrm{v}}_{\mathrm{j}}^{\mathrm{T}}{\mathrm{v}}_i=\log {\mathrm{X}}_{\mathrm{i}\mathrm{j}}-\log {\mathrm{X}}_{\mathrm{j}}$$

Equations (15) and (16) are added,

$$2{\mathrm{v}}_{\mathrm{j}}^{\mathrm{T}}{\mathrm{v}}_i=2\log {\mathrm{X}}_{\mathrm{i}\mathrm{j}}-\log {\mathrm{X}}_i-\log {\mathrm{X}}_{\mathrm{j}}$$

(15) $${\mathrm{v}}_{\mathrm{j}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{i}}=\log {\mathrm{X}}_{\mathrm{i}\mathrm{j}}-{\displaystyle \frac{1}{2}\log {\mathrm{X}}_{\mathrm{i}}-{\displaystyle \frac{1}{2}\log {\mathrm{X}}_{\mathrm{j}}}}$$

Here v_i and v_j are learnable parameters and ${\mathrm{X}}_{\mathrm{i}}$, ${\mathrm{X}}_{\mathrm{j}}$ is word specific biases, which will be learned as well. The above equation can be rewritten as

(16) $${\mathrm{v}}_{\mathrm{j}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{i}}+{\mathrm{b}}_{\mathrm{i}}+{\mathrm{b}}_{\mathrm{j}}=\log {\mathrm{X}}_{\mathrm{i}\mathrm{j}}$$where ${\mathrm{b}}_{\mathrm{i}}$ is word specific bias for word i and ${\mathrm{b}}_{\mathrm{j}}$ is word specific bias for word j. All these parameters are learnable parameters, whereas X_ij is the actual ground truth that can be known from the global co-occurrence matrix. Equation (16) can be formulated as an optimization problem, which gives the difference between predicted value using model parameters and the actual value computed from the given corpus.

(17) $${}_{{\mathrm{v}}_{\mathrm{i}},{\mathrm{v}}_{\mathrm{j}},{\mathrm{b}}_{\mathrm{i}},{\mathrm{b}}_{\mathrm{j}}}^{\mathrm{m}\mathrm{i}\mathrm{n}}\sum _{\mathrm{i},\mathrm{j}}{\left({\mathrm{v}}_{\mathrm{j}}^{\mathrm{T}}{\mathrm{v}}_{\mathrm{i}}+{\mathrm{b}}_{\mathrm{i}}+{\mathrm{b}}_{\mathrm{j}}-\log {\mathrm{X}}_{\mathrm{i}\mathrm{j}}\right)}^2$$In comparison with word2vec, Glove maintains both the local and global context of a word from the entire corpus. To select an appropriate vectorization method, which maintains intra-modal semantic coherence, the below section covers experiments performed using different vectorization methods on Multi-Modal datasets. The Convolutional Neural Network (CNN) is adopted for image modality in the proposed framework, as it has shown promising performance in many computer vision applications (Bhandare et al., 2016).

Proposed framework for cross-modal retrieval

In this section, we present our proposed framework, which generates real-valued common sub-space. It also covers the learning algorithm outlined in Algorithm 1.

Problem formulation

The proposed framework has image and text modality, which is denoted by $\mathrm{\Psi }={\left\{\left({\mathrm{X}}_{\mathrm{i}},{\mathrm{Y}}_{\mathrm{i}}\right)\right\}}_{\mathrm{i}=1}^{\mathrm{n}}$ where ${\mathrm{X}}_{\mathrm{i}}$ and ${\mathrm{Y}}_{\mathrm{i}}$ is image and text sample respectively. Each instance of $\left({\mathrm{X}}_{\mathrm{i}},{\mathrm{Y}}_{\mathrm{i}}\right)$ has a semantic label vector ${\mathrm{Z}}_{\mathrm{i}}=\left[{\mathrm{z}}_{1\mathrm{i}},{\mathrm{z}}_{2\mathrm{i}},\dots .,{\mathrm{z}}_{\mathrm{C}\mathrm{i}}\right]\in {\mathrm{R}}^{\mathrm{C}}$, where C is the number of categories. The similarity matrix S_ij = 1, if i^th instance of image and text modality matches to the j^th category, otherwise ${\mathrm{S}}_{\mathrm{i}\mathrm{j}}=0$. The feature vectors of image and text modality lie in different representation space, so direct composition is not possible for retrieval. The objective is to learn two functions, ${\mathrm{u}}_{\mathrm{i}}=\mathrm{f}\left({\mathrm{x}}_{\mathrm{i}},{\theta }_{\mathrm{x}}\right)\in {\mathbb{R}}^d$ and ${\mathrm{v}}_{\mathrm{i}}=\mathrm{g}\left({\mathrm{y}}_{\mathrm{i}},{\theta }_{\mathrm{y}}\right)\in {\mathbb{R}}^d$ for image and text modality respectively, where d is the dimension of a common sub-space. The ${\theta }_{\mathrm{x}}$ and ${\theta }_{\mathrm{y}}$ are hyper parameters of image and text modality, respectively. The generated common sub-space allows direct comparison for retrieval even though samples come from different statistical properties.

Proposed framework: Improvement of deep cross-modal retrieval (IDCMR)

Figure 3 shows the proposed framework for image and text modality. The convolutional layers of Convolutional Neural Network (CNN) for image modality are pretrained on ImageNet, which generates high-level representation for each image. CNN has five convolutional layers and three fully connected layers. Detailed configuration of the convolutional layer is given in the proposed framework. Each convolutional layer contains “f: num × size × size”, which specifies the number of the filter with specific size, “s” indicates stride, “pad” indicates padding, and “pool” indicates downsampling factor. The common representation for each image is generated by fully connected layers. The number in the last fully connected layer (fc8) indicates the number of neurons or dimensionality of the output layer. Similarly, the Glove model for text modality is pretrained on Google News, which represents each word in form of feature vector. The text matrix is given to fully connected layers to learn the common representation for text. To learn a common representation from image and text modality, the two sub-networks share the weights of the last layers, which generate the same representation for semantic similar image and text modality. In this work, real-valued coordinated representation is generated, which preserves intra-modal and inter-modal semantic similarity. The inter-modal similarity is preserved by minimizing the (i) discrimination loss in the label space ${\mathrm{J}}_1$. The prediction of label from feature spaces is possible, by connecting a linear classifier on top of each network. (ii) discrimination loss in text and image representation J₂, and (iii) modality-invariant loss J₃ in the common sub-space. Further, the intra-modal similarity is preserved by selecting an appropriate training model for each modality. The biggest challenge in text modality is to preserve semantic similarities between words. There are many distributional representation methods available and the challenge is to select an appropriate method, which preserves intra-modal similarity between different words of text modality. The below section covers the learning algorithm, experiments of different distributional models, and performance comparison of the proposed framework with state-of-the-art methods.

[a] Calculate the discrimination loss in the label space

Once the features are learned from image and text modality, a linear classifier C is connected to image and text sub networks, which predicts the semantic labels of the projected features. This predicted label should preserve the semantic similarity with label space. The discrimination loss in the label space is calculated by J₁ using the following equation:

(18) $${\mathrm{J}}_1={\displaystyle \frac{1}{\mathrm{n}}{\mathrm{C}}^{\mathrm{T}}\mathrm{U}-{\mathrm{S}}_{\mathrm{F}}+{\displaystyle \frac{1}{\mathrm{n}}{\mathrm{C}}^{\mathrm{T}}\mathrm{V}-{\mathrm{S}}_{\mathrm{F}}}}$$where, ||·||_F is Frobenius norm and n is the number of instances.

[b] Calculate the discrimination loss of both text and image modality in the common sub-space

The inter-modal similarity is further preserved by minimizing discrimination loss from image and text representation in the common sub-space, as denoted by Eq. (19).

(19) $${\mathrm{J}}_2={\displaystyle \frac{1}{\mathrm{n}}\left[-\sum _{\mathrm{i},\mathrm{j}=1}^{\mathrm{n}}[{\mathrm{S}}_{\mathrm{i}\mathrm{j}}{\theta }_{\mathrm{i}\mathrm{j}}-\log \left(1+{\theta }_{\mathrm{i}\mathrm{j}}\right)]\right]+{\displaystyle \frac{1}{\mathrm{n}}\left[-\sum _{\mathrm{i},\mathrm{j}=1}^{\mathrm{n}}[{\mathrm{S}}_{\mathrm{i}\mathrm{j}}{\phi }_{\mathrm{i}\mathrm{j}}-\log \left(1+{\phi }_{\mathrm{i}\mathrm{j}}\right)]\right]+{\displaystyle \frac{1}{\mathrm{n}}\left[-\sum _{\mathrm{i},\mathrm{j}=1}^{\mathrm{n}}[{\mathrm{S}}_{\mathrm{i}\mathrm{j}}{\varphi }_{\mathrm{i}\mathrm{j}}-\log \left(1+{\varphi }_{\mathrm{i}\mathrm{j}}\right)]\right]}}}$$

The first part of Eq. (19), preserves the semantic similarity between image representation U and text representation V with similarity matrix S, which is denoted as,

$${\theta }_{\mathrm{i}\mathrm{j}}={\mathrm{U}}_{\ast \mathrm{i}}^{\mathrm{T}}{\mathrm{V}}_{\ast \mathrm{j}}$$

The above equation should maximize the likelihood

$$\mathrm{P}({\mathrm{S}}_{\mathrm{i}\mathrm{j}}|{\mathrm{U}}_{\ast \mathrm{i}},{\mathrm{V}}_{\ast \mathrm{j}})={\mathrm{S}}_{\mathrm{i}\mathrm{j}}=1\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{n}{\mathrm{S}}_{\mathrm{i}\mathrm{j}}=1$$

(20) $$1-\sigma \left({\theta }_{\mathrm{i}\mathrm{j}}\right)\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{n}{\mathrm{S}}_{\mathrm{i}\mathrm{j}}=0$$where $\sigma ({\theta }_{\mathrm{i}\mathrm{j}})={\displaystyle \frac{1}{1+{\mathrm{e}}^{-{\theta }_{\mathrm{i}\mathrm{j}}}}}$ is a sigmoid function that exists between 0 to 1, and it is preferable when there is a need to predict the probability as an output. Since the probability of anything exists between a range of 0 to 1, sigmoid is the right choice.

It is represented as,

$$\mathrm{P}({\mathrm{S}}_{\mathrm{i}\mathrm{j}}|{\mathrm{U}}_{\ast \mathrm{i}},{\mathrm{V}}_{\ast \mathrm{j}})=\pi (\sigma \left({\theta }_{\mathrm{i}\mathrm{j}}\right){)}^{{\mathrm{S}}_{\mathrm{i}\mathrm{j}}}{\left(1-\sigma \left({\theta }_{\mathrm{i}\mathrm{j}}\right)\right)}^{1-{\mathrm{S}}_{\mathrm{i}\mathrm{j}}}$$

(21) $$=\sum _{\mathrm{i},\mathrm{j}=1}^{\mathrm{n}}[{\mathrm{S}}_{\mathrm{i}\mathrm{j}}{\theta }_{\mathrm{i}\mathrm{j}}+\log \left(1-{\theta }_{\mathrm{i}\mathrm{j}}\right)]$$

Equation (21) can be rewritten as below cost function which forces representation ${\theta }_{\mathrm{i}\mathrm{j}}$ to be larger when S_ij = 1 and vice versa.

(22) $$\mathrm{J}=-\sum _{\mathrm{i},\mathrm{j}=1}^{\mathrm{n}}[{\mathrm{S}}_{\mathrm{i}\mathrm{j}}{\theta }_{\mathrm{i}\mathrm{j}}-\log \left(1+{\theta }_{\mathrm{i}\mathrm{j}}\right)]$$So, here cost function forces ${\theta }_{\mathrm{i}\mathrm{j}}$ to be larger when and vice versa.

The second part and third part of the equation measures the similarities with image representation and text representations.

$${\phi }_{\mathrm{i}\mathrm{j}}={\mathrm{U}}_{\ast \mathrm{i}}^{\mathrm{T}}{\mathrm{U}}_{\ast \mathrm{j}}$$ ${\phi }_{\mathrm{i}\mathrm{j}}$ is image representation, for instance, i and j whereas

$${\varphi }_{\mathrm{i}\mathrm{j}}={\mathrm{V}}_{\ast \mathrm{i}}^{\mathrm{T}}{\mathrm{V}}_{\ast \mathrm{j}}$$ ${\varphi }_{\mathrm{i}\mathrm{j}}$ is text representation, for instance, i and j.

[c] Calculate the modality wise invariance loss

(23) $${\mathrm{J}}_3={\displaystyle \frac{1}{\mathrm{n}}\mathrm{U}-{\mathrm{V}}_{\mathrm{F}}}$$The final objective function is,

(24) $$\mathrm{J}={\mathrm{J}}_1+\lambda {\mathrm{J}}_2+\eta {\mathrm{J}}_3$$

The final objective function of IDMR in Eq. (24) can be optimized during the stochastic gradient descent algorithm. The $\lambda$ and $\eta$ are hyper parameters. The J₁, J₂, and J₃ are the loss functions, used to preserve inter-modal similarity between image and text modality. The proposed framework has used the sigmoid activation function, which is a nonlinear function used to learn complex structures in the data. However, sometimes it suffers from vanishing gradient descent, which prevents deep networks to learn from learning effectively. The problem of vanishing gradient can be solved by using another activation function, like rectified linear activation unit (ReLU).