Soon-Gyo Jung
Joni Salminen
Bernard J. Jansen
ABSTRACT
We address a recommendation task for next likely flight destination to customers of a major international airline company. We compare performance using historical flight data and an actual user evaluation. Using two years of historical flight data consisting of tens of millions of flights, an ensemble and a collaborative filtering approach obtained an accuracy of 47% and 20% using a test set of 100,000 customers, respectively, highlighting the challenge of the domain. We then evaluated our recommendations on 10,000 actual customers, with a 45-45-10 split among ensemble, collaborative filtering, and control group. The overall predictive power employed with real users was 23%, with the ensemble method having a predictive power of 19% and 30% for collaborative filtering. Results indicate that, in complex and shifting domains such as this one, one cannot rely solely on historical data for evaluating the impact of user recommendations. We discuss implications for recommendation systems and future research in this and related domains.
- J. Beel, M. Genzmehr, S. Langer, A. Nürnberger, and B. Gipp, "A comparative analysis of offline and online evaluations and discussion of research paper recommender system evaluation," in Proceedings of the international workshop on reproducibility and replication in recommender systems evaluation, 2013, pp. 7--14.Google Scholar
- S. Chen, W. Huang, M. Chen, J. Zhong, and J. Cheng, "Airlines Content Recommendations Based on Passengers' Choice Using Bayesian Belief Networks," in Bayesian Inference, J. P. Tejedor, Ed., ed: IntechOpen, 2017.Google Scholar
- H. Fani, E. Jiang, E. Bagheri, F. Al-Obeidat, W. Du, and M. Kargar, "User community detection via embedding of social network structure and temporal content," Information Processing & Management, vol. 57, p. 102056, 2020.Google ScholarDigital Library
- S. Fitchett and A. Cockburn, "AccessRank: predicting what users will do next," presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Austin, Texas, USA, 2012.Google ScholarDigital Library
- J. L. Herlocker, J. A. Konstan, L. G. Terveen, and J. T. Riedl, "Evaluating collaborative filtering recommender systems," ACM Transactions on Information Systems, vol. 22, pp. 5--53, 2004.Google ScholarDigital Library
- C. Hueglin and F. Vannotti, "Data mining techniques to improve forecast accuracy in airline business," presented at the Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, San Francisco, CA, 2001.Google Scholar
- B. P. Knijnenburg and M. C. Willemsen, "Evaluating recommender systems with user experiments," in Recommender Systems Handbook, ed Boston, MA: Springer, 2015, pp. 309--352.Google Scholar
- R. D. Lawrence, S. J. Hong, and J. Cherrier, "Passenger-based predictive modeling of airline no-show rates," presented at the Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, Washington, D.C., 2003.Google Scholar
- D. Lian, V. W. Zheng, and X. Xie, "Collaborative filtering meets next check-in location prediction," presented at the Proceedings of the 22nd International Conference on World Wide Web, Rio de Janeiro, Brazil, 2013.Google Scholar
- X. Ling, W. Deng, C. Gu, H. Zhou, C. Li, and F. Sun, "Model Ensemble for Click Prediction in Bing Search Ads," presented at the Proceedings of the 26th International Conference on World Wide Web Companion, Perth, Australia, 2017.Google Scholar
- P. Pu, L. Chen, and R. Hu, "A user-centric evaluation framework for recommender systems," in Proceedings of the fifth ACM conference on Recommender Systems, 2011, pp. 157--164.Google Scholar
- S. J. Racine and J. P. Curtin, "Developing an airline freight management system: meeting airline and end-user challenges," presented at the CHI '03 Extended Abstracts on Human Factors in Computing Systems, Ft. Lauderdale, Florida, USA, 2003.Google Scholar
- S. Renjith, A. Sreekumar, and M. Jathavedan, "An extensive study on the evolution of context-aware personalized travel recommender systems," Information Processing & Management, vol. 57, p. 102078, 2020.Google ScholarDigital Library
- P. Sánchez and A. Bellogíns, "Building user profiles based on sequences for content and collaborative filtering," Information Processing & Management, vol. 56, pp. 192--211, 2019.Google ScholarCross Ref
- O. S. Shalom, N. Koenigstein, U. Paquet, and H. P. Vanchinathan, "Beyond Collaborative Filtering: The List Recommendation Problem," presented at the Proceedings of the 25th International Conference on World Wide Web, Montreal, Quebec, Canada, 2016.Google Scholar
- Y. Wang, C. Breitinger, B. Sommer, F. Schreiber, and H. Reiterer, "Comparing Sequential and Temporal Patterns from Human Mobility Data for Next-Place Prediction," presented at the Adjunct Publication of the 26th Conference on User Modeling, Adaptation and Personalization, Singapore, Singapore, 2018.Google Scholar
Index Terms
- Things Change: Comparing Results Using Historical Data and User Testing for Evaluating a Recommendation Task
Recommendations
Transparent, Scrutable and Explainable User Models for Personalized Recommendation
SIGIR'19: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information RetrievalMost recommender systems base their recommendations on implicit or explicit item-level feedback provided by users. These item ratings are combined into a complex user model, which then predicts the suitability of other items. While effective, such ...
User Experience and The Role of Personalization in Critiquing-Based Conversational Recommendation
Critiquing — where users propose directional preferences to attribute values — has historically been a highly popular method for conversational recommendation. However, with the growing size of catalogs and item attributes, it becomes increasingly ...
Implicit Recommendation with Interest Change and User Influence
ICSCA '19: Proceedings of the 2019 8th International Conference on Software and Computer ApplicationsAiming at the problem of rich websites in campus without targeted recommendation, which makes it difficult for users to find the information resources of high interest and high quality, this paper proposes an implicit feedback recommendation algorithm ...
Comments