Gediminas Adomavicius
ABSTRACT
The paper introduces stability as a new measure of the recommender systems performance. In general, we define a recommendation algorithm to be "stable" if its predictions for the same items are consistent over a period of time, assuming that any new ratings that have been submitted to the recommender system over the same period of time are in complete agreement with system's prior predictions. In this paper, we advocate that stability should be a desired property of recommendation algorithms, because unstable recommendations can lead to user confusion and, therefore, reduce trust in recommender systems. Furthermore, we empirically evaluate stability of several popular recommendation algorithms. Our results suggest that model-based recommendation techniques demonstrate higher stability than memory-based collaborative filtering heuristics. We also find that the stability measure for recommendation techniques is influenced by many factors, including the sparsity of the initial rating data, the number of new incoming ratings (representing the length of the time period over which the stability is being measured), the distribution of the newly added rating values, and the rating normalization procedures employed by the recommendation algorithms.
Supplemental Material
- }}Adomavicius, G. and A. Tuzhilin. Toward the Next Generation of Recommendation System: A Survey of the State-of-the-Art and Possible Extensions. IEEE Transactions on Knowledge and Data Engineering, 17(6): 734--749, 2005. Google Scholar
Digital Library
- }}Bell, R. M. and Y. Koren. Scalable Collaborative Filtering with Jointly Derived Neighborhood Interpolation Weights. In Proceeding of the Seventh IEEE International Conference on Data Mining. Omaha, NE, USA, 2007. Google ScholarDigital Library
- }}Bell, R. M. and Y. Koren. Improved Neighborhood-based Collaborative Filtering. In Proceeding of the KDD Cup'07. San Jose, CA, USA, 2007.Google Scholar
- }}Bell, R. M. and Y. Koren. Lessons from the Netflix prize challenge. ACM SIGKDD Explorations Newsletter, 9(2): 75--79, 2007. Google ScholarDigital Library
- }}Bennet, J. and S. Lanning. The Netflix Prize. In Proceeding of the KDD Cup and Workshop, 2007.Google Scholar
- }}Funk, S. Netflix Update: Try This at Home. Netflix Update: Try This at Home, last update 2006, cited 2010. Available from: http://sifter.org/~simon/journal/20061211.html.Google Scholar
- }}Grouplens. Movielens Data Sets. 2006.Google Scholar
- }}Herlocker, J., J. Kostan, A. Borchers, and J. Riedl. An Algorithmic Framework for Performing Collaborative Filtering. In Proceeding of the 22nd ACM SIGIR Conference on Information Retrieval, 1999. Google ScholarDigital Library
- }}Herlocker, J., J. Konstan, K. Terveen, and J. Riedl. Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems, 22(1): 5--53, 2004. Google ScholarDigital Library
- }}Komiak, S. and I. Benbasat. The effects of personalization and familiarity on trust and adoption of recommendation agents. MIS Quarterly, 30(4): 941--960, 2006. Google ScholarCross Ref
- }}Koren, Y., R. Bell, and C. Volinsky. Matrix Factorization Techniques For Recommender Systems. IEEE Computer, 42: 30--37, 2009. Google ScholarDigital Library
- }}Kostan, J., B. Miller, D. Maltz, J. Herlocker, L. Gordon, and J. Riedl. GroupLens: Applying Collaborative Filtering to Usenet news. Communications of the ACM, 40: 77--87, 1997. Google ScholarDigital Library
- }}Lam, S. and J. Riedl. Shilling Recommender Systems for Fun and Profit. In Proceeding of the the 13th international conference on World Wide Web New York City, NY, 2004. Google ScholarDigital Library
- }}Massa, P. and B. Bhattacharjee. Using trust in recommender systems: An experimental analysis, in Trust Management, Proceeding, 221--235. Springer-Verlag Berlin: Berlin, 2004.Google Scholar
- }}Mobasher, B., R. Burke, and J. J. Sandvig. Model-Based Collaborative Filtering as a Defense against Profile Injection Attacks. In Proceeding of the 21st Conference on Artificial Intelligence (AAAI'06). Boston, MA, 2006. Google ScholarDigital Library
- }}Mobasher, B., R. Burke, C. Williams, and R. Bhaumik. Analysis and detection of segment-focused attacks against collaborative recommendation, in Advances in Web Mining and Web Usage Analysis, 96--118. Springer-Verlag Berlin: Berlin, 2006. Google ScholarDigital Library
- }}Mobasher, B., R. Burke, R. Bhaumik, and C. Williams. Toward Trustworthy Recommender Systems: An Analysis of Attack Models and Algorithm Robustness. ACM Transactions on Internet Technology, 7(4): 23:1--23:38, 2007. Google ScholarDigital Library
- }}O'Donovan, J. and B. Smyth. Mining trust values from recommendation errors. International Journal on Artificial Intelligence Tools, 15(6): 945--962, 2006.Google ScholarCross Ref
- }}O'Donovan, J. and B. Smyth. Trust in recommender systems. In Proceeding of the 10th international conference on Intelligent user interfaces. San Diego, California, USA, 2005. Google ScholarDigital Library
- }}Turney, P. Technical Note: Bias and the Quantification of Stability. Machine Learning, 20: 23--33, 1995. Google ScholarDigital Library
- }}Wang, W. and I. Benbasat. Trust in and adoption of online recommendation agents. Journal of the Association for Information Systems, 6(3): 72--101, 2005.Google ScholarCross Ref
Index Terms
- On the stability of recommendation algorithms
Recommendations
Stability of Recommendation Algorithms
The article explores stability as a new measure of recommender systems performance. Stability is defined to measure the extent to which a recommendation algorithm provides predictions that are consistent with each other. Specifically, for a stable ...
Classification, Ranking, and Top-K Stability of Recommendation Algorithms
Recommendation stability measures the extent to which a recommendation algorithm provides predictions that are consistent with each other. Several approaches have been proposed in prior work to defining, measuring, and improving the stability of ...
Typicality-Based Collaborative Filtering Recommendation
Collaborative filtering (CF) is an important and popular technology for recommender systems. However, current CF methods suffer from such problems as data sparsity, recommendation inaccuracy, and big-error in predictions. In this paper, we borrow ideas ...
Comments