Robust latent semantic exploration for image retrieval in social media

Abstract

With the speedy development of social media, more and more multimedia data are generated by users with tags associated. The tag information provides the extra cue to link multimedia data in addition to the multimedia content itself. However, the manually added tags are always with noise and not correct enough. Moreover, the semantically similar tags exist massively but cannot be accounted for well. This paper proposes a new algorithm to robustly combine multimedia content and associated tags by mining the latent semantic which takes into account the semantically similar tags. The $l_{2,1}$ norm is proposed to employ in latent semantic indexing for a more robust latent space, and a word-to-vector based clustering method is proposed to address the massive tags with similar meaning. The experiments on extensive data demonstrate the proposed method. Compared to the existing latent semantic based methods, the algorithm proposed a more robust model to deal with noise.

Introduction

Recent years have witnessed the boom of social media with the rapid development of smart phones and convenient internet access. People can easily acquire and share information and communicate with others on these platforms. One distinctive feature of social media is that it is more easier for users to create and share their own contents. Among the massive social media platforms, many image and video sharing web sites have become very popular. On these platforms, users not only produce and store their own contents but also can view and comment on others׳ contents. These platforms provide the social network characteristics which make users get connected and communicate with one another in multiple forms. Users can view the contents from one another and are provided the ability to comment on these contents by using tags. These tags can be viewed as annotation multimedia contents.

However, since users do not have the obligation to add tags accurately and thoroughly, directly using tags as annotation will bring problems due to being noisy and incomplete. Users sometimes give tags not from the same angle. There are many ways to describe one multimedia content, so it is impossible to add all possible tags completely for users. For example, for the landscape images taken at the same place, different users would pay attention to different aspects and hence prefer to add different tags to describe this place. Consequently, although these two images are visually similar, the difference introduced by tags will cause that these two images are not “similar” during searching. On the other hand, there is also the possibility that the visually different images are declared “similar” based on the associated tags.

As a reference, in the dataset proposed in [1], it has been shown that the average precision of user tags is about 0.5 and the average recall (completeness rate) of the user tags is 0.5 as well. Hence it means half of the tags created by users are noise and half of the true labels are missing.

To address the noisy and missing tags, a lot of works have been proposed recently. Tag refinement is one potential solution to improve the quality of user-generated tags associated to the multimedia data [2], [3], [4], [5]. The relevance between tag and image is explored and further refined in these works. In [2] tag refinement is formulated as a tag ranking problem and a probabilistic framework was proposed to rank the associated tags. In [3], content consistency, tag consistency and low rank properties are studied in the mean time to generate a refined tag set. Tang et al. [4] used a robust graph and employed semi-supervised learning technique to learn a tag ranking model to do tag refinement.

In this paper, we propose an automatic image annotation algorithm by introducing a new latent semantic space to discover the semantic structure hidden in image and its tags. During the latent space construction, we use $l_{2,1}$ norm as the regularizor which has been demonstrated more robust to noise. To address those tags in similar meanings but regarded as different words, a word to vector based clustering method is proposed to build connection with similar tags. In addition, visual feature learnt from a deep convolutional neural network is invoked to compute the visual similarity between images. Based on these improvements, the proposed image retrieval method is demonstrated superior with using extensive dataset.

The rest of this paper is organized as follows. Section 2 reviews the foundation of the proposed algorithm, i.e., the latent semantic indexing method and the low rank based latent semantic discovery means. Section 3 is devoted to presenting the proposed robust latent semantic mining method. The experimental results are provided in Section 4. We conclude this paper in Section 5.