Dmitri Goldenberg
ABSTRACT
Personalization is one of the key applications in machine learning with widespread usage across e-commerce, entertainment, production, healthcare and many other industries. While various machine learning techniques present novel state-of-the-art advances and super-human performance year-over-year, personalization and recommender-systems applications are often late-adopters of novel solutions due to problem hardness and implementation complexity. This tutorial presents recent advances across the personalization industry and demonstrates their practical applications in real case-studies of world-leading online platforms. Key trends such as deep learning, causality and active exploration with bandits are depicted with real examples and demonstrated alongside their business considerations and implementation challenges.Rising topics like explainability, fairness, natural interfaces and content generation are covered, touching on aspects of both technology and user experience. Our tutorial relies on recent advances in the field and on work conducted at Booking.com, where we implement personalization models on one of the world's leading online travel platform.
Supplemental Material
- Elizabeth Aguirre, Dominik Mahr, Dhruv Grewal, Ko de Ruyter, and Martin Wetzels. 2015. Unraveling the personalization paradox: The effect of information collection and trust-building strategies on online advertisement effectiveness. Journal of retailing, Vol. 91, 1 (2015), 34--49.Google Scholar
Cross Ref
- Xavier Amatriain and Deepak Agarwal. 2016. Tutorial: Lessons learned from building real-life recommender systems. In Proceedings of the 10th ACM Conference on Recommender Systems. 433--433.Google ScholarDigital Library
- Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. 2019. Guidelines for human-AI interaction. In Proceedings of the 2019 chi conference on human factors in computing systems. 1--13.Google ScholarDigital Library
- Andrea Barraza-Urbina and Dorota Glowacka. 2020. Introduction to Bandits in Recommender Systems. In Proceedings of the 14th ACM Conference on Recommender Systems (RecSys '20). Association for Computing Machinery.Google ScholarDigital Library
- Justin Basilico. 2019. Recent Trends in Personalization: A Netflix Perspective. ICML.Google Scholar
- Robert M Bell and Yehuda Koren. 2007. Lessons from the Netflix prize challenge. Acm Sigkdd Explorations Newsletter, Vol. 9, 2 (2007), 75--79.Google ScholarDigital Library
- Shlomo Berkovsky and Jill Freyne. 2015. Web personalization and recommender systems. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2307--2308.Google ScholarDigital Library
- Lucas Bernardi, Themistoklis Mavridis, and Pablo Estevez. 2019. 150 Successful Machine Learning Models: 6 Lessons Learned at Booking. com. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1743--1751.Google ScholarDigital Library
- Alex Beutel, Paul Covington, Sagar Jain, Can Xu, Jia Li, Vince Gatto, and Ed H Chi. 2018. Latent cross: Making use of context in recurrent recommender systems. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. 46--54.Google ScholarDigital Library
- Sarah Bird, Krishnaram Kenthapadi, Emre Kiciman, and Margaret Mitchell. 2019. Fairness-aware machine learning: Practical challenges and lessons learned. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. 834--835.Google ScholarDigital Library
- Zahra Dasht Bozorgi, Irene Teinemaa, Marlon Dumas, Marcello La Rosa, and Artem Polyvyanyy. 2020. Process Mining Meets Causal Machine Learning: Discovering Causal Rules from Event Logs. In 2020 2nd International Conference on Process Mining (ICPM). IEEE, 129--136.Google ScholarCross Ref
- Olivier Chapelle and Lihong Li. 2011. An Empirical Evaluation of Thompson Sampling. In Proceedings of the 24th International Conference on Neural Information Processing Systems (NIPS'11). Curran Associates Inc., USA, 2249--2257.Google ScholarDigital Library
- Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, and Wenwu Ou. 2019. Behavior sequence transformer for e-commerce recommendation in alibaba. In Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data. 1--4.Google ScholarDigital Library
- Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems. 7--10.Google ScholarDigital Library
- Peng Cui, Zheyan Shen, Sheng Li, Liuyi Yao, Yaliang Li, Zhixuan Chu, and Jing Gao. 2020. Causal Inference Meets Machine Learning. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 3527--3528.Google ScholarDigital Library
- Floris Devriendt, Darie Moldovan, and Wouter Verbeke. 2018. A literature survey and experimental evaluation of the state-of-the-art in uplift modeling: A stepping stone toward the development of prescriptive analytics. Big data, Vol. 6, 1 (2018), 13--41.Google Scholar
- Paolo Dragone, Rishabh Mehrotra, and Mounia Lalmas. 2019. Deriving User-and Content-specific Rewards for Contextual Bandits. In The World Wide Web Conference. 2680--2686.Google Scholar
- Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. arxiv: stat.ML/1506.02142Google Scholar
- Ruoyuan Gao and Chirag Shah. 2020. Counteracting Bias and Increasing Fairness in Search and Recommender Systems. In Proceedings of the 14th ACM Conference on Recommender Systems (RecSys '20). Association for Computing Machinery.Google ScholarDigital Library
- Dmitri Goldenberg, Javier Albert, Lucas Bernardi, and Pablo Estevez. 2020. Free Lunch! Retrospective Uplift Modeling for Dynamic Promotions Recommendation within ROI Constraints.. In Proceedings of the 14th ACM Conference on Recommender Systems .Google ScholarDigital Library
- Dmitri Goldenberg, Kostia Kofman, Pavel Levin, Sarai Mizrachi, Maayan Kafry, and Guy Nadav. 2021. Booking.com WebTour 2021 Data Challenge. http://www.bookingchallenge.com. In ACM WSDM Workshop on Web Tourism (WSDM Webtour'21) .Google Scholar
- Thore Graepel, Joaquin Candela, Thomas Borchert, and Ralf Herbrich. 2010. Web-Scale Bayesian Click-Through Rate Prediction for Sponsored Search Advertising in Microsoft's Bing Search Engine. 13--20.Google Scholar
- Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: A Factorization-Machine based Neural Network for CTR Prediction. 1725--1731. https://doi.org/10.24963/ijcai.2017/239Google Scholar
- Benjamin Gutelman and Pavel Levin. 2020. Efficient Image Gallery Representations at Scale Through Multi-Task Learning. In Proceedings of the 43th International ACM SIGIR Conference on Research and Development in Information Retrieval .Google ScholarDigital Library
- Balázs Hidasi, Massimo Quadrana, Alexandros Karatzoglou, and Domonkos Tikk. 2016. Parallel Recurrent Neural Network Architectures for Feature-Rich Session-Based Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys '16). Association for Computing Machinery, New York, NY, USA, 241--248. https://doi.org/10.1145/2959100.2959167Google ScholarDigital Library
- Emre Kiciman and Amit Sharma. 2019. Causal Inference and Counterfactual Reasoning (3hr Tutorial). In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. 828--829.Google ScholarDigital Library
- Bart P Knijnenburg, Niels JM Reijmer, and Martijn C Willemsen. 2011. Each to his own: how different users call for different interaction methods in recommender systems. In Proceedings of the fifth ACM conference on Recommender systems. 141--148.Google ScholarDigital Library
- Bart P Knijnenburg and Martijn C Willemsen. 2015. Evaluating recommender systems with user experiments. In Recommender Systems Handbook. Springer, 309--352.Google Scholar
- Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix factorization techniques for recommender systems. Computer, Vol. 42, 8 (2009), 30--37.Google ScholarDigital Library
- Mandy Korpusik, Shigeyuki Sakaki, Francine Chen, and Chen Yan-Ying. 2016. Recurrent Neural Networks for Customer Purchase Prediction on Twitter. In CBRecSys 2016 .Google Scholar
- Sören R Künzel, Jasjeet S Sekhon, Peter J Bickel, and Bin Yu. 2019. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the national academy of sciences, Vol. 116, 10 (2019), 4156--4165.Google Scholar
- Tobias Lang and Matthias Rettenmeier. 2017. Understanding consumer behavior with recurrent neural networks. In Workshop on Machine Learning Methods for Recommender Systems .Google Scholar
- Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. 2010. A contextual-bandit approach to personalized news article recommendation. In WWW .Google Scholar
- Dawen Liang, Rahul G. Krishnan, Matthew D. Hoffman, and Tony Jebara. 2018. Variational Autoencoders for Collaborative Filtering. arxiv: stat.ML/1802.05814Google Scholar
- James McInerney, Benjamin Lacker, Samantha Hansen, Karl Higley, Hugues Bouchard, Alois Gruson, and Rishabh Mehrotra. 2018. Explore, exploit, and explain: personalizing explainable recommendations with bandits. In Proceedings of the 12th ACM Conference on Recommender Systems. 31--39.Google ScholarDigital Library
- Rishabh Mehrotra, Ben Carterette, Yong Li, Quanming Yao, Chen Gao, James Kwok, Qiang Yang, and Isabelle Guyon. 2020. Advances in Recommender Systems: From Multi-stakeholder Marketplaces to Automated RecSys. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 3533--3534.Google ScholarDigital Library
- Sarai Mizrachi and Pavel Levin. 2019. Combining Context Features in Sequence-Aware Recommender Systems.. In RecSys (Late-Breaking Results). 11--15.Google Scholar
- Michael J Pazzani and Daniel Billsus. 2007. Content-based recommendation systems. In The adaptive web. Springer, 325--341.Google ScholarDigital Library
- Massimo Quadrana and Paolo Cremonesi. 2018. Sequence-Aware Recommendation. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys '18). Association for Computing Machinery, New York, NY, USA, 539--540. https://doi.org/10.1145/3240323.3241621Google Scholar
- Massimo Quadrana, Dietmar Jannach, and Paolo Cremonesi. 2019. Tutorial: Sequence-Aware Recommender Systems. In Companion Proceedings of The 2019 World Wide Web Conference. 1316--1316.Google Scholar
- Nicholas J Radcliffe. 2007. Using control groups to target on predicted lift: Building and assessing uplift models. Direct Marketing Analytics Journal, Vol. 1 (2007), 1421.Google Scholar
- Steffen Rendle. 2010. Factorization machines. In 2010 IEEE International Conference on Data Mining. IEEE, 995--1000.Google ScholarDigital Library
- Eren Sezener, Marcus Hutter, David Budden, Jianan Wang, and Joel Veness. 2020. Online Learning in Contextual Bandits using Gated Linear Networks. arxiv: cs.LG/2002.11611Google Scholar
- Amit Sharma and Emre Kiciman. 2020. Causal Inference and Counterfactual Reasoning. In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD. 369--370.Google ScholarDigital Library
- Aleksandrs Slivkins. 2019. Introduction to Multi-Armed Bandits. CoRR, Vol. abs/1904.07272 (2019). arxiv: 1904.07272 http://arxiv.org/abs/1904.07272Google ScholarCross Ref
- Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 1441--1450.Google ScholarDigital Library
- Niek Tax, Irene Teinemaa, and Sebastiaan J van Zelst. 2020. An interdisciplinary comparison of sequence modeling methods for next-element prediction. Software and Systems Modeling, Vol. 19, 6 (2020), 1345--1365.Google ScholarDigital Library
- Jun Xu, Xiangnan He, and Hang Li. 2018. Deep learning for matching in search and recommendation. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 1365--1368.Google ScholarDigital Library
- Xiaotian Yu, Michael R. Lyu, and Irwin King. 2017. CBRAP: Contextual Bandits with RAndom Projection. In AAAI.Google Scholar
- Yongfeng Zhang, Zuohui Fu, Yikun Xian, and Yi Zhang. 2020. Tutorial on Conversational Recommender Systems. In Proceedings of the 14th ACM Conference on Recommender Systems (RecSys '20). Association for Computing Machinery.Google Scholar
- Yongfeng Zhang, Jiaxin Mao, and Qingyao Ai. 2019. WWW'19 Tutorial on Explainable Recommendation and Search. In Companion Proceedings of The 2019 World Wide Web Conference. 1330--1331.Google Scholar
- Zhenyu Zhao and Totte Harinen. 2019. Uplift modeling for multiple treatments with cost optimization. In 2019 IEEE International Conference on Data Science and Advanced Analytics (DSAA). IEEE, 422--431.Google ScholarCross Ref
- Guorui Zhou, Chengru Song, Xiaoqiang Zhu, et al. 2018. Deep Interest Network for Click-Through Rate Prediction. In KDD 2018.Google Scholar
Index Terms
- Personalization in Practice: Methods and Applications
Recommendations
New Recommendation Techniques for Multicriteria Rating Systems
Traditional single-rating recommender systems have been successful in a number of personalization applications, but the research area of multicriteria recommender systems has been largely untouched. Taking full advantage of multicriteria ratings in ...
Interactive collaborative filtering
CIKM '13: Proceedings of the 22nd ACM international conference on Information & Knowledge ManagementIn this paper, we study collaborative filtering (CF) in an interactive setting, in which a recommender system continuously recommends items to individual users and receives interactive feedback. Whilst users enjoy sequential recommendations, the ...
Effects of Personalized and Aggregate Top-N Recommendation Lists on User Preference Ratings
Prior research has shown a robust effect of personalized product recommendations on user preference judgments for items. Specifically, the display of system-predicted preference ratings as item recommendations has been shown in multiple studies to bias ...
Comments