Mining user-contributed photos for personalized product recommendation

doi:10.1016/j.neucom.2013.09.018

Neurocomputing

Volume 129, 10 April 2014, Pages 409-420

https://doi.org/10.1016/j.neucom.2013.09.018 Get rights and content

Abstract

With the advent and popularity of social media, users are willing to share their experiences by photos, reviews, blogs, and so on. The social media contents shared by these users reveal potential shopping needs. Product recommender is not limited to just e-commerce sites, it can also be expanded to social media sites. In this paper, we propose a novel hierarchical user interest mining (Huim) approach for personalized products recommendation. The input of our approach consists of user-contributed photos and user generated content (UGC), which include user-annotated photo tags and the comments from others in a social site. The proposed approach consists of four steps. First, we make full use of the visual information and UGC of its photos to mine user's interest. Second, we represent user interest by a topic distribution vector, and apply our proposed Huim to enhance interest-related topics. Third, we also represent each product by a topic distribution vector. Then, we measure the relevance of user and product in the topic space and determine the rank of each product for the user. We conduct a series of experiments on Flickr users and the products from Bing Shopping. Experimental results show the effectiveness of the proposed approach.

Introduction

With the rapid development of e-commerce, products recommender system has been exploited to suggest attractive and useful products' information to facilitate user's decision-making process. The intelligent of products recommendation can help users to deal with information overload and provide them personalized services [39]. Product recommendation is popular in e-commerce sites. Some e-commerce sites such as Amazon and Bingshopping recommend products to users based on previous buys as well as what others have been bought when they bought the same product. They keep tracks of users spending and analyze their interests by collaborative filtering [3]. However, in collaborative filtering based products recommendation approaches, only the relevance of users is considered. Thus, they are not personalized to user's interest.

With the booming of social networks, more and more people are will to share their personal affairs, new things and their favorite photos with their friends. For example, Facebook has about one billion users. Flickr is photo sharing website, it also have a very large amount of users. The total number of photos shared by users in Flickr had reached 6 billion by August 2011. There are about 3 million photos uploaded by users each day. The user contributed photos and user generated content can reveal the user's interests very well [44], [45], [46], [50], [51]. Thus the social media websites are the ideal platforms to facilitate the personalized products recommendation.

To improve user experience by making ads relevant to the webpage content, Broder et al. proposed a system for contextual ad matching based on a combination of semantic and syntactic features [40]. They used the semantic phrase to classify the webpage and the ads into taxonomy, and then ranked ads by the proximity of the ads and webpage categories. Although they classified both ads and page content within a large taxonomy, they ignored that some ads are relevant to several topics. For instance, the tag canon is relevant to digital camera and also relevant to the bags of camera. Thus, it is better to represent user interests by topic distribution vector rather than taxonomy.

Taking above mentions into consideration, we propose a novel hierarchical user interest mining (Huim) method to explore user's potential shopping needs based on user-contributed photos in her/his social media sites. We recommend personalized products according to the mined user interests. There are three main problems needed to be solved: (1) the gap between appearances of user-contributed photos and their textual descriptions (i.e. UGC). For example, when a user uploaded some images of her new iphone, she may label images by the words “the amazing apple”. In this circumstance, the tag of apple has two meanings: fruit and electronic product. So we need to mine more content related information from both the visual information and the UGC of user contributed photos. (2) The noise and ambiguous tags in UGC have negative effect to mine user's interests. The textual descriptions contain informal expressions with noise tags (such as the preposition and other non-topic tags) and some ambiguous tags which generated by users. Thus, to recommend personalized products, we need to suppress the noise and ambiguous tags, and to enhanced user interest topics for the user contributed photos. (3) How to measure the relevance of user and product. If a user shared some photos of basketball game, then it is reasonable to suggest some products about basketball like knee pad, and it is also acceptable to recommend basketball video games of 360 Xbox. Among the above three problems measuring the relevance is the core problem in product recommendation.

To fulfill personalized product recommendation, we propose a hierarchical topic vector representation approach to represent user interest and product. Our approach is carried out as follows: (1) tag enrichment for the user contributed photos, (2) introduce a public topic space and map user interest and product descriptions to it and get their topic distribution vectors, (3) enhance user interested topics by a hierarchical approach is proposed to suppress noise and ambiguous textual descriptions, (4) measure the relevance of user and product in the hierarchical public topic space and rank the products for the user.

The main contributions of this paper are summarized as follows: (1) propose a personalized product recommendation system which mining users' interests from their contributed photos; (2) propose an effective user and product relevant measurement approach by introducing a hierarchical public topic space; (3) propose an effective hierarchical user interest representation approach which is robust to suppress noise and ambiguous textual description and enhance user interested topics.

Compared to our preliminary version [50], several improvements are made: (1) the detailed steps of the proposed hierarchical user interest mining approach are provided; (2) we extend the approach from brand recommendation to more general product recommendation, and (3) more experimental results and discussions are provided.

The remainder of this paper is organized as follows. In Section 2, we present the related works on products recommendation, user interest mining and multimedia advertising. In Section 3, our personalized products recommendation based on hierachical user interest mining approach is introduced in detail. Experiments and discussions are given in Section 4 and conclusions are drawn in Section 5.

Section snippets

Related work

In this section, we briefly review the related works on products recommendation, user interest mining from social media, and multimedia advertising.

Overview

This section details our proposed solutions on personalized products recommendation based on user-contributed photos from social media sites. The input of our approach is user shared photos of the same webpage and their corresponding textual descriptions. The system is shown in Fig. 1. Our approach consists of three parts: (1) hierarchical user interest representation. We map user's information (UGC and enriched tags of the photos) to a hierarchical public topic space. And we represent user

Experiments and discussions

To show the effectiveness of the proposed personalized product recommendation approach using Huim, we compare it with Argo [20]. Experiments are conducted on real Flickr users and products of Bingshopping [43].

Conclusion

In this paper, a personalized product recommendation approach is proposed by mining user interest from user-contributed photos in social media sites. User's interests can be well disclosed from their shared photos in a webpage. The topic space can be utilized to measure the bridge the gaps in measuring the relevance of a user and products. The hierarchical structures of topic spaces are valuable for enhancing user's interested topics and suppress noise and ambiguous textual descriptions. The

He Feng was a member of SMILES LAB of Xi'an Jiaotong Univerisity from Sept 2011 to July 2013. He received the M.S. degree in school of electronic and information engineering from Xi'an Jiaotong University, Xi'an, China, in 2013. His interest is related to social recommendation.

References (51)

S.W. Changchien et al.
Mining association rules procedure to support on-line recommendation by customers and products fragmentation
Expert Syst. Appl.
(2001)
D.R. Liu et al.
Integrating AHP and data mining for product recommendation based on customer lifetime value
Inf. Manage.
(2005)
Y.K. Cao et al.
An intelligent fuzzy-based recommendation system for consumer electronic products
Expert Syst. Appl.
(2007)
X. Qian et al.
Tagging photos using users' vocabularies
Neurocomputing
(2013)
A.Evans, M.Fernandez, et al., Adaptive multimedia access: from user needs semantic personalization, in: IEEE...
C.-H. Chang et al.
ViSA: virtual spotlighted advertising
ACM Multimedia
(2008)
F Fouss et al.
Random-walk computation of similar-ities between nodes of a graph, with application to collaborative recommendation
IEEE Trans. Knowl. Data Eng.
(2007)
H.H.Dai, L.Z.Zhao, Z.Q.Nie, J.-R.Wen, L.Wang, Y.Li, Detecting online commercial intention, ACM International WWW...
H.Y Liu et al.
A generic virtual content insertion system based on visual attention analysis
ACM Multimedia
(2008)
H.N.Zhou, Z.Y.Liu, Page frame segmentation for contextural advertising in print on demand books, in: CVPR Workshops,...

J.Hu, H.-J.Zeng, H.Li, C.Niu, Z.Chen, Demographic prediction based on user's browsing behavior, ACM International WWW...

J. Trajkova et al.

Improving ontology-based user profiles

RIAO

(2004)

K.W Wan et al.

Robust goal-mouth detection for virtual content insertion

Argo: intelligent advertising by mining a user's interest from his photo collections

ACM Data Min. Audience Intell. Advertising

(2009)

X.-J. Wang et al.

Annotating images by mining image search results, pattern analysis and machine intelligence

IEEE Trans. Pattern Anal. Mach. Intell.

(2008)

Cited by (40)

Impact of user-generated travel posts on travel decisions: A comparative study on Weibo and Xiaohongshu
2022, Annals of Tourism Research Empirical Insights
With the growing popularity and versatility of mobile applications, this study examined consumers' usage of multi-functional social media content apps and the influence of user-generated travel posts on their travel decision-making. Using the case of Chinese millennial travelers and Weibo and Xiaohongshu as representative apps, semi-structured interviews and surveys were conducted. Results showed that while respondents used Weibo more often in daily life, Xiaohongshu was more likely to influence their destination selection. Key attributes of travel posts and preferences of different user groups were identified. The relative importance of each attribute in guiding dining/accommodation/transportation/shopping/attraction decisions was examined. The most active app users placed lower importance on the number of images and higher importance on length of text and language style.
An empirical study of personalized advertising recommendation based on DBSCAN clustering of sina weibo user-generated content
2021, Procedia Computer Science
Today, with the dividend of mobile Internet traffic disappearing, mobile advertising is focusing on powerful algorithms of personalized recommendation. Sina Weibo has abundant advertising spaces and the accuracy of its recommendation directly affects user experience. UGC (user-generated content) is the selected or even original edited content for interaction through users’ subjective interests, which reflects users’ inner interests within a certain period of time. Adaptive DBSCAN clustering is performed on Weibo UGC, and the clustering center represents the user’s interest to a certain extent. By refining the text theme of advertisements to be recommended, the similarity between clustering center and the recommended theme is obtained, and the advertising category with high similarity is recommended to user. The experiment proves that the method is effective.
Social media monitoring: What can marketers learn from Facebook brand photos?
2020, Journal of Business Research
Citation Excerpt :
In the field of computer vision, scholars have developed systems to generate recommendations from the photos posted to social media platforms, but they have not evaluated these systems in terms of whether their recommendations successfully match users' interests. Feng and Qian (2014) implement a system that derives personalized product recommendations from social media photos. Wang, Yu, Zhang, Cai, and Ma (2009) present a system that proposes advertisements based on a user's collection of photos.
Users upload >350 million photos per day to Facebook. While considerable research has explored text-based user-generated content on social media, research on photos is still in its early stages. This paper uses a sample of 44,765 Facebook photos from 503 Facebook users in the United States and Germany to determine the degree to which photos play an integral role in people's social media communications. The analysis shows that uploading brand photos (i.e., photos containing a brand name or logo) is related to brand love, brand loyalty, and word-of-mouth (WOM) endorsement of the brand in question. We then code a subsample of these photos for content and train a powerful hybrid machine learning algorithm combining genetic search and artificial neural networks. The resulting algorithm is able to predict users' brand love, brand loyalty, and WOM endorsement from the content of their brand photos posted on Facebook. Finally, we discuss the implications for social media marketing, in particular social media monitoring.
Salience-aware adaptive resonance theory for large-scale sparse data clustering
2019, Neural Networks
Sparse data is known to pose challenges to cluster analysis, as the similarity between data tends to be ill-posed in the high-dimensional Hilbert space. Solutions in the literature typically extend either k-means or spectral clustering with additional steps on representation learning and/or feature weighting. However, adding these usually introduces new parameters and increases computational cost, thus inevitably lowering the robustness of these algorithms when handling massive ill-represented data. To alleviate these issues, this paper presents a class of self-organizing neural networks, called the salience-aware adaptive resonance theory (SA-ART) model. SA-ART extends Fuzzy ART with measures for cluster-wise salient feature modeling. Specifically, two strategies, i.e. cluster space matching and salience feature weighting, are incorporated to alleviate the side-effect of noisy features incurred by high dimensionality. Additionally, cluster weights are bounded by the statistical means and minimums of the samples therein, making the learning rate also self-adaptable. Notably, SA-ART allows clusters to have their own sets of self-adaptable parameters. It has the same time complexity of Fuzzy ART and does not introduce additional hyperparameters that profile cluster properties. Comparative experiments have been conducted on the ImageNet and BlogCatalog datasets, which are large-scale and include sparsely-represented data. The results show that, SA-ART achieves 51.8% and 18.2% improvement over Fuzzy ART, respectively. While both have a similar time cost, SA-ART converges faster and can reach a better local minimum. In addition, SA-ART consistently outperforms six other state-of-the-art algorithms in terms of precision and F1 score. More importantly, it is much faster and exhibits stronger robustness to large and complex data.
Recommender systems based on multiple social networks correlation
2018, Future Generation Computer Systems
Citation Excerpt :
The interpersonal relationship, especially the circles of friends, of social networks makes it possible to solve the cold start and sparsity problem. The rich of social media give us some valuable clues to recommend user favorite items [7], such as music [8], video [9], preferred brand/products [10], user’s preferred tags when sharing a photo to social media networks [11], and user interested travel places by exploring social community contributed photos [12]. Many social network based models [13] have been proposed to improve the performance of the RS.
As development of social networks, social recommendation method, an effective information filtering technology, based on sociology rule and network theory, has improved performance of recommendation system and cold start problem. For data deep fusion and diverse development of social platform, the social relationship between users becomes more and more complex. The complexity of multiple social networks challenges social recommendation. However, most existing social recommendation methods focus on single social network, and multi-layer recommendation methods ignore nonlinearity and coupling between different social relationships. To tackle these problems, we propose a probabilistic matrix factorization model for multiply social networks joint recommendation framework based on joint probability distribution. This model analyzes different types of classic social networks and distribution function of user preferences similarity. Then we present unified model of recommendation based on social networks, as well as extensible multiply social networks joint recommendation method. The experimental results demonstrate comparing with relevant social recommendation algorithms; our method performs better on some evaluation indexes such as accuracy and errors.
An intelligent movie recommendation system through group-level sentiment analysis in microblogs
2016, Neurocomputing
Citation Excerpt :
In the final section, the conclusion is made and more discussions are put together for future efforts.2 Our work is related to two research fields, recommender systems [20] and user generated content mining [21,22]. Recommender system: There exist two types of collaborative filtering approaches, namely, neighborhood-based and model-based.
The emergence of the online media sharing sites (e.g. Youtube, Youku, and Hulu) have introduced new challenges in program recommendation in online networks. However, there is a bottleneck that the amount of available viewing logs and user friendship networks are too limited to design effective recommendation algorithms. Thus, carrying out an intelligent program recommendation system is important for these sites. In this work, we propose a novel model which turns to the social networks and mine user preferences information expressed in microblogs for evaluating the similarity between online movies and TV episodes. To the best of our knowledge, it is the first effort to bridge the gap between movie and TV watchers domain with social network activities. Moreover, it is the first approach that can solve the “cold-start” problem in movie and TV recommendation system. Series of data mining approaches and social computing models have been adopted in this work. Similar programs found from the social network are further used to suggest programs in other media devices. This work can be easily applied in online media streaming sites in order that intelligent recommendations of programs can be made to the customers through mining microblogs.

View all citing articles on Scopus

Xueming Qian (M'10) received the B.S. and M.S. degrees in Xi'an University of Technology, Xi'an, China, in 1999 and 2004, respectively, and the Ph.D. degree in the School of Electronics and Information Engineering, Xi'an Jiaotong University, Xi'an, China, in 2008. He was awarded Microsoft fellowship in 2006. From 1999 to 2001, he was an Assistant Engineer at Shannxi Daily. From 2008 till now, he is a faculty member of the School of Electronics and Information Engineering, Xi'an Jiaotong University. Now he is an associate professor of the School of Electronics and Information Engineering, Xi'an Jiaotong University. He is the director of SMILES LAB. He was a visit scholar at Microsoft research Asia from Aug. 2010 to March 2011. His research interests include video/image analysis, indexing, and retrieval.

^☆: This work was supported in part by National Natural Science Foundation of China (NSFC) Project No. 60903121 and No. 61173109, and Microsoft Research Asia.

View full text

Mining user-contributed photos for personalized product recommendation☆

Abstract

Introduction

Section snippets

Related work

Overview

Experiments and discussions

Conclusion

Expert Syst. Appl.

Inf. Manage.

Expert Syst. Appl.

Neurocomputing

ViSA: virtual spotlighted advertising

ACM Multimedia

Random-walk computation of similar-ities between nodes of a graph, with application to collaborative recommendation

IEEE Trans. Knowl. Data Eng.

A generic virtual content insertion system based on visual attention analysis

ACM Multimedia

Improving ontology-based user profiles

RIAO

Robust goal-mouth detection for virtual content insertion

ACM Multimedia

GameSense: game-like in-image advertising

Multimedia Tools Appl.

Modeling the clickstream: implications for web-based advertising efforts

Marketing Sci.

Contextual in-image advertising

ACM Multimedia

Contextual internet multimedia advertising

IEEE Spec. Issues Internet Vision

VideoSense: a contextual in-video advertising system

IEEE Circuits Syst. Soc.

VideoSense: towards effective online video advertising

ACM Multimedia

AdOn: toward contextual overlay in-video advertising

ACM Multimedia

AdImage: video advertising by image matching and Ad scheduling optimization

ACM SIGIR

Advertising based on users' photos

ICME

Argo: intelligent advertising by mining a user's interest from his photo collections

ACM Data Min. Audience Intell. Advertising

Annotating images by mining image search results, pattern analysis and machine intelligence

IEEE Trans. Pattern Anal. Mach. Intell.