Skip to main content
SpringerLink
Log in
Book cover

Recommender Systems Handbook pp 385–419Cite as

Recommender Systems in Industry: A Netflix Case Study

  • Chapter

Abstract

The Netflix Prize put a spotlight on the importance and use of recommender systems in real-world applications. Many the competition provided many lessons about how to approach recommendation and many more have been learned since the Grand Prize was awarded in 2009. The evolution of industrial applications of recommender systems has been driven by the availability of different kinds of user data and the level of interest for the area within the research community. The goal of this chapter is to give an up-to-date overview of recommender systems techniques used in an industrial setting. We will give a high-level description the practical use of recommendation and personalization techniques. We will highlight some of the main lessons learned from the Netflix Prize. We will then use Netflix personalization as a case study to describe several approaches and techniques used in a real-world recommendation system. Finally, we will pinpoint what we see as some promising current research avenues and unsolved problems that deserve attention in this domain from an industry perspective.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   299.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   379.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    The application of Matrix Factorization to the task of rating prediction closely resembles the technique known as Singular Value Decomposition used, for example, to identify latent factors in Information Retrieval. Therefore, it is common to see people referring to this MF solution as SVD.

  2. 2.

    For practical purposes we consider responses below a few hundred milliseconds (e.g. 200) to be real-time.

  3. 3.

    Intermediate recommendations usually represent lists of items that have been pre-selected and even ranked in advanced but need to undergo further processing such as filtering or re-ranking before being presented to the user.

  4. 4.

    Chukwa is a Hadoop subproject devoted to large-scale log collection and analysis.

  5. 5.

    Hadoop is an open-source software framework for storage and large-scale processing of data-sets on clusters of commodity hardware.

  6. 6.

    Apache Hive is a data warehouse infrastructure built on top of Hadoop for providing data summarization, query, and analysis.

  7. 7.

    Pig is a high-level platform for creating MapReduce programs used with Hadoop using a language called Pig Latin.

  8. 8.

    The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware.

  9. 9.

    Amazon S3 (Simple Storage Service) is an online file storage web service offered by Amazon Web Services.

  10. 10.

    Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

  11. 11.

    Apache Cassandra is an open source distributed database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure.

  12. 12.

    EVCache is a distributed in-memory data store for the cloud.

  13. 13.

    MySQL is one of the most popular open source relational databases.

  14. 14.

    It is important to note that the term “social recommendation” was originally used to describe collaborative filtering approaches [7, 74].

References

  1. Agarwal, D., Chen, B.C., Elango, P., Ramakrishnan, R.: Content recommendation on web portals. Commun. ACM 56(6), 92–101 (2013). DOI 10.1145/2461256.2461277. URL http://doi.acm.org/10.1145/2461256.2461277

  2. Agarwal, D., Chen, B.C., Pang, B.: Personalized recommendation of user comments via factor models. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ‘11, pp. 571–582. Association for Computational Linguistics, Stroudsburg, PA, USA (2011). URL http://dl.acm.org/citation.cfm?id=2145432.2145499

  3. Ahmed, A., Teo, C.H., Vishwanathan, S., Smola, A.: Fair and balanced: Learning to present news stories. In: Proceedings of the Fifth ACM International Conference on Web Search and Data Mining, WSDM ‘12, pp. 333–342. ACM, New York, NY, USA (2012). DOI 10.1145/2124295.2124337. URL http://doi.acm.org/10.1145/2124295.2124337

  4. Amatriain, X., Lathia, N., Pujol, J.M., Kwak, H., Oliver, N.: The wisdom of the few: a collaborative filtering approach based on expert opinions from the web. In: Proc. of 32nd ACM SIGIR, SIGIR ‘09, pp. 532–539. ACM, New York, NY, USA (2009). DOI 10.1145/1571941.1572033. URL http://dx.doi.org/10.1145/1571941.1572033

  5. Amatriain, X., Pujol, J.M., Oliver, N.: I Like It…I Like It Not: Evaluating User Ratings Noise in Recommender Systems. In: G.J. Houben, G. McCalla, F. Pianesi, M. Zancanaro (eds.) User Modeling, Adaptation, and Personalization, vol. 5535, chap. 24, pp. 247–258. Springer Berlin (2009). DOI 10.1007/978-3-642-02247-0_24. URL http://dx.doi.org/10.1007/978-3-642-02247-0_24

  6. Andersen, R., Borgs, C., Chayes, J., Feige, U., Flaxman, A., Kalai, A., Mirrokni, V., Tennenholtz, M.: Trust-based recommendation systems: an axiomatic approach. In: Proc. of the 17th WWW, WWW ‘08, pp. 199–208. ACM, New York, NY, USA (2008). DOI 10.1145/1367497.1367525. URL http://doi.acm.org/10.1145/1367497.1367525

  7. Basu, C., Hirsh, H., Cohen, W.: Recommendation as classification: using social and content-based information in recommendation. In: Proc. of AAAI ‘98, AAAI ‘98/IAAI ‘98, pp. 714–720. American Association for Artificial Intelligence, Menlo Park, CA, USA (1998). URL http://dl.acm.org/citation.cfm?id=295240.295795

  8. Bell, R.M., Koren, Y.: Lessons from the Netflix Prize Challenge. SIGKDD Explor. Newsl. 9(2), 75–79 (2007). DOI 10.1145/1345448.1345465. URL http://dx.doi.org/10.1145/1345448.1345465

  9. Berndhardsson, E.: Music recommendations at spotify (2013)

    Google Scholar 

  10. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003). URL http://dl.acm.org/citation.cfm?id=944919.944937

  11. Bourke, S., McCarthy, K., Smyth, B.: Power to the people: exploring neighbourhood formations in social recommender system. In: Proc. of Recsys ‘11, RecSys ‘11, pp. 337–340. ACM, New York, NY, USA (2011). DOI 10.1145/2043932.2043997. URL http://doi.acm.org/10.1145/2043932.2043997

  12. Breiman, L.: Random forests. Machine learning 45(1), 5–32 (2001)

    Article  MATH  Google Scholar 

  13. Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., Hullender, G.: Learning to rank using gradient descent. In: Proceedings of the 22nd ICML, ICML ‘05, pp. 89–96. ACM, New York, NY, USA (2005). DOI 10.1145/1102351.1102363. URL http://dx.doi.org/10.1145/1102351.1102363

  14. Burke, R.: The adaptive web. chap. Hybrid Web Recommender Systems, pp. 377–408 (2007). DOI 10.1007/978-3-540-72079-9_12. URL http://dx.doi.org/10.1007/978-3-540-72079-9_12

  15. Cao, Z., Liu, T.: Learning to rank: From pairwise approach to listwise approach. In: In Proceedings of the 24th ICML, pp. 129–136 (2007). URL http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.64.1518

  16. Celma, O.: Music Recommendation and Discovery: The Long Tail, Long Fail, and Long Play in the Digital Music Space. Springer (2010)

    Google Scholar 

  17. Chapelle, O., Keerthi, S.S.: Efficient algorithms for ranking with SVMs. Information Retrieval 13, 201–215 (2010). DOI 10.1007/s10791-009-9109-9. URL http://dx.doi.org/10.1007/s10791-009-9109-9

  18. Chen, W.Y., Chu, J.C., Luan, J., Bai, H., Wang, Y., Chang, E.Y.: Collaborative filtering for orkut communities: Discovery of user latent behavior. In: Proceedings of the 18th International Conference on World Wide Web, WWW ‘09, pp. 681–690. ACM, New York, NY, USA (2009). DOI 10.1145/1526709.1526801. URL http://doi.acm.org/10.1145/1526709.1526801

  19. Das, A.S., Datar, M., Garg, A., Rajaram, S.: Google news personalization: Scalable online collaborative filtering. In: Proceedings of the 16th International Conference on World Wide Web, WWW ‘07, pp. 271–280. ACM, New York, NY, USA (2007). DOI 10.1145/1242572.1242610. URL http://doi.acm.org/10.1145/1242572.1242610

  20. Davidson, J., Liebald, B., Liu, J., Nandy, P., Van Vleet, T., Gargi, U., Gupta, S., He, Y., Lambert, M., Livingston, B., Sampath, D.: The youtube video recommendation system. In: Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys ‘10, pp. 293–296. ACM, New York, NY, USA (2010). DOI 10.1145/1864708.1864770. URL http://doi.acm.org/10.1145/1864708.1864770

  21. Diaz-Aviles, E., Georgescu, M., Nejdl, W.: Swarming to rank for recommender systems. In: Proc. of Recsys ‘12, RecSys ‘12, pp. 229–232. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2366001. URL http://doi.acm.org/10.1145/2365952.2366001

  22. Elkan, C., Noto, K.: Learning classifiers from only positive and unlabeled data. In: Proc. of the 14th ACM SIGKDD, KDD ‘08, pp. 213–220. ACM, New York, NY, USA (2008). DOI 10.1145/1401890.1401920. URL http://dx.doi.org/10.1145/1401890.1401920

  23. Freund, Y., Iyer, R., Schapire, R.E., Singer, Y.: An efficient boosting algorithm for combining preferences. J. Mach. Learn. Res. 4, 933–969 (2003). URL http://portal.acm.org/citation.cfm?id=964285

  24. Frey, B.J., Dueck, D.: Clustering by passing messages between data points. Science 315, 2007 (2007)

    Article  MathSciNet  Google Scholar 

  25. Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Annals of Statistics pp. 1189–1232 (2001)

    Google Scholar 

  26. Fujiwara, Y., Nakatsuji, M., Yamamuro, T., Shiokawa, H., Onizuka, M.: Efficient personalized pagerank with accuracy assurance. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ‘12, pp. 15–23. ACM, New York, NY, USA (2012). DOI 10.1145/2339530.2339538. URL http://doi.acm.org/10.1145/2339530.2339538

  27. Funk, S.: Netflix update: Try this at home. http://sifter.org/ simon/journal/20061211.html (2006). URL http://sifter.org/~simon/journal/20061211.html

  28. Gorgoglione, M., Panniello, U., Tuzhilin, A.: The effect of context-aware recommendations on customer purchasing behavior and trust. In: Proc. of Recsys ‘11, RecSys ‘11, pp. 85–92. ACM, New York, NY, USA (2011). DOI 10.1145/2043932.2043951. URL http://doi.acm.org/10.1145/2043932.2043951

  29. Gupta, P., Goel, A., Lin, J., Sharma, A., Wang, D., Zadeh, R.: Wtf: The who to follow service at twitter. In: Proceedings of the 22Nd International Conference on World Wide Web, WWW ‘13, pp. 505–514. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland (2013). URL http://dl.acm.org/citation.cfm?id=2488388.2488433

  30. Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004). DOI http://doi.acm.org/10.1145/963770.963772

    Google Scholar 

  31. Hu, Y., Koren, Y., Volinsky, C.: Collaborative Filtering for Implicit Feedback Datasets. In: Proc. of the 2008 Eighth ICDM, ICDM ‘08, vol. 0, pp. 263–272. IEEE Computer Society, Washington, DC, USA (2008). DOI 10.1109/ICDM.2008.22. URL http://dx.doi.org/10.1109/ICDM.2008.22

  32. in the Industry, R.S.: Recommendation systems in the industry. Tutorial at Recsys 2009 (2009)

    Google Scholar 

  33. Jamali, M., Ester, M.: Trustwalker: a random walk model for combining trust-based and item-based recommendation. In: Proc. of KDD ‘09, KDD ‘09, pp. 397–406. ACM, New York, NY, USA (2009). DOI 10.1145/1557019.1557067. URL http://doi.acm.org/10.1145/1557019.1557067

  34. Jeh, G., Widom, J.: Simrank: A measure of structural-context similarity. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ‘02, pp. 538–543. ACM, New York, NY, USA (2002). DOI 10.1145/775047.775126. URL http://doi.acm.org/10.1145/775047.775126

  35. Karatzoglou, A., Amatriain, X., Baltrunas, L., Oliver, N.: Multiverse recommendation: n-dimensional tensor factorization for context-aware collaborative filtering. In: Proc. of the fourth ACM Recsys, RecSys ‘10, pp. 79–86. ACM, New York, NY, USA (2010). DOI 10.1145/1864708.1864727. URL http://dx.doi.org/10.1145/1864708.1864727

  36. Karimzadehgan, M., Li, W., Zhang, R., Mao, J.: A stochastic learning-to-rank algorithm and its application to contextual advertising. In: Proceedings of the 20th WWW, WWW ‘11, pp. 377–386. ACM, New York, NY, USA (2011). DOI 10.1145/1963405.1963460. URL http://doi.acm.org/10.1145/1963405.1963460

  37. Karypis, G.: Evaluation of item-based top-n recommendation algorithms. In: CIKM ‘01: Proceedings of the tenth international conference on Information and knowledge management, pp. 247–254. ACM, New York, NY, USA (2001). DOI http://doi.acm.org/10.1145/ 502585.502627

    Google Scholar 

  38. Knijnenburg, B.P.: Conducting user experiments in recommender systems. In: Proceedings of the sixth ACM conference on Recommender systems, RecSys ‘12, pp. 3–4. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2365956. URL http://doi.acm.org/10.1145/2365952.2365956

  39. Koenigstein, N., Nice, N., Paquet, U., Schleyen, N.: The xbox recommender system. In: Proceedings of the Sixth ACM Conference on Recommender Systems, RecSys ‘12, pp. 281–284. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2366015. URL http://doi.acm.org/10.1145/2365952.2366015

  40. Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T., Xu, Y.: Trustworthy online controlled experiments: five puzzling outcomes explained. In: Proceedings of KDD ‘12, pp. 786–794. ACM, New York, NY, USA (2012). DOI 10.1145/2339530.2339653. URL http://doi.acm.org/10.1145/2339530.2339653

  41. Kohavi, R., Henne, R.M., Sommerfield, D.: Practical guide to controlled experiments on the web: Listen to your customers not to the hippo. In: Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ‘07, pp. 959–967. ACM, New York, NY, USA (2007). DOI 10.1145/1281192.1281295. URL http://doi.acm.org/10.1145/1281192.1281295

  42. Koren, Y.: Factorization meets the neighborhood: a multifaceted collaborative filtering model. In: Proceedings of the 14th ACM SIGKDD, KDD ‘08, pp. 426–434. ACM, New York, NY, USA (2008). DOI 10.1145/1401890.1401944. URL http://dx.doi.org/10.1145/1401890.1401944

  43. Koren, Y.: Collaborative filtering with temporal dynamics. In: Proceedings of the 15th ACM SIGKDD, KDD ‘09, pp. 447–456. ACM, New York, NY, USA (2009). DOI 10.1145/1557019.1557072. URL http://dx.doi.org/10.1145/1557019.1557072

  44. Koren, Y., Bell, R., Volinsky, C.: Matrix Factorization Techniques for Recommender Systems. Computer 42(8), 30–37 (2009). DOI 10.1109/MC.2009.263. URL http://dx.doi.org/10.1109/MC.2009.263

  45. Lagun, D., Hsieh, C.H., Webster, D., Navalpakkam, V.: Towards better measurement of attention and satisfaction in mobile search. In: Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ‘14, pp. 113–122. ACM, New York, NY, USA (2014). DOI 10.1145/2600428.2609631. URL http://doi.acm.org/10.1145/2600428.2609631

  46. Lamere, P.B.: I’ve got 10 million songs in my pocket: Now what? In: Proceedings of the Sixth ACM Conference on Recommender Systems, RecSys ‘12, pp. 207–208. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2365994. URL http://doi.acm.org/10.1145/2365952.2365994

  47. Li, L., Chu, W., Langford, J., Schapire, R.E.: A contextual-bandit approach to personalized news article recommendation. In: Proceedings of the 19th International Conference on World Wide Web, WWW ‘10, pp. 661–670. ACM, New York, NY, USA (2010). DOI 10.1145/1772690.1772758. URL http://doi.acm.org/10.1145/1772690.1772758

  48. Linden, G., Smith, B., York, J.: Amazon.com recommendations: Item-to-item collaborative filtering. IEEE Internet Computing 7(1), 76–80 (2003). DOI 10.1109/MIC.2003.1167344. URL http://dx.doi.org/10.1109/MIC.2003.1167344

  49. Liu, J., Pedersen, E., Dolan, P.: Personalized news recommendation based on click behavior. In: 2010 International Conference on Intelligent User Interfaces (2010)

    Google Scholar 

  50. Liu, N.N., Meng, X., Liu, C., Yang, Q.: Wisdom of the better few: cold start recommendation via representative based rating elicitation. In: Proc. of RecSys ‘11, RecSys ‘11. ACM, New York, NY, USA (2011). DOI 10.1145/2043932.2043943. URL http://doi.acm.org/10.1145/2043932.2043943

  51. McLaughlin, M.R., Herlocker, J.L.: A collaborative filtering algorithm and evaluation metric that accurately model the user experience. In: Proc. of SIGIR ‘04 (2004)

    Google Scholar 

  52. Mcnee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: how accuracy metrics have hurt recommender systems. In: CHI ‘06: CHI ‘06 extended abstracts on Human factors in computing systems, pp. 1097–1101. ACM Press, New York, NY, USA (2006). DOI 10.1145/ 1125451.1125659

    Google Scholar 

  53. Navalpakkam, V., Jentzsch, L., Sayres, R., Ravi, S., Ahmed, A., Smola, A.: Measurement and modeling of eye-mouse behavior in the presence of nonlinear page layouts. In: Proceedings of the 22Nd International Conference on World Wide Web, WWW ‘13, pp. 953–964. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland (2013). URL http://dl.acm.org/citation.cfm?id=2488388.2488471

  54. Navalpakkam, V., Jentzsch, L., Sayres, R., Ravi, S., Ahmed, A., Smola, A.: Measurement and modeling of eye-mouse behavior in the presence of nonlinear page layouts. In: Proceedings of the 22Nd International Conference on World Wide Web, WWW ‘13, pp. 953–964. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland (2013). URL http://dl.acm.org/citation.cfm?id=2488388.2488471

  55. Ning, X., Karypis, G.: Sparse linear methods with side information for top-n recommendations. In: Proc. of the 21st WWW, WWW ‘12 Companion, pp. 581–582. ACM, New York, NY, USA (2012). DOI 10.1145/2187980.2188137. URL http://doi.acm.org/10.1145/2187980.2188137

  56. Noel, J., Sanner, S., Tran, K., Christen, P., Xie, L., Bonilla, E.V., Abbasnejad, E., Della Penna, N.: New objective functions for social collaborative filtering. In: Proc. of WWW ‘12, WWW ‘12, pp. 859–868. ACM, New York, NY, USA (2012). DOI 10.1145/2187836.2187952. URL http://doi.acm.org/10.1145/2187836.2187952

  57. O’Donovan, J., Smyth, B.: Trust in recommender systems. In: Proc. of IUI ‘05, IUI ‘05, pp. 167–174. ACM, New York, NY, USA (2005). DOI 10.1145/1040830.1040870. URL http://doi.acm.org/10.1145/1040830.1040870

  58. Oku, K., Nakajima, S., Miyazaki, J., Uemura, S.: Context-aware SVM for context-dependent information recommendation. In: Proc. of the 7th Conference on Mobile Data Management (2006)

    Google Scholar 

  59. Parra, D., Amatriain, X.: Walk the Talk: Analyzing the relation between implicit and explicit feedback for preference elicitation. In: J.A. Konstan, R. Conejo, J.L. Marzo, N. Oliver (eds.) User Modeling, Adaption and Personalization, Lecture Notes in Computer Science, vol. 6787, chap. 22, pp. 255–268. Springer, Berlin, Heidelberg (2011). DOI 10.1007/978-3-642-22362-4_22. URL http://dx.doi.org/10.1007/978-3-642-22362-4\_22

  60. Parra, D., Karatzoglou, A., Amatriain, X., Yavuz, I.: Implicit feedback recommendation via implicit-to-explicit ordinal logistic regression mapping. In: Proc. of the 2011 CARS Workshop (2011)

    Google Scholar 

  61. Pizzato, L., Rej, T., Chung, T., Koprinska, I., Kay, J.: Recon: A reciprocal recommender for online dating. In: Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys ‘10, pp. 207–214. ACM, New York, NY, USA (2010). DOI 10.1145/1864708.1864747. URL http://doi.acm.org/10.1145/1864708.1864747

  62. Rabkin, A., Katz, R.: Chukwa: A system for reliable large-scale log collection. In: Proceedings of the 24th International Conference on Large Installation System Administration, LISA’10, pp. 1–15. USENIX Association, Berkeley, CA, USA (2010). URL http://dl.acm.org/citation.cfm?id=1924976.1924994

  63. Radlinski, F., Kurup, M., Joachims, T.: How does clickthrough data reflect retrieval quality? In: Proc. of the 17th CIKM, CIKM ‘08, pp. 43–52. ACM, New York, NY, USA (2008). DOI 10.1145/1458082.1458092. URL http://dx.doi.org/10.1145/1458082.1458092

  64. Reda, A., Park, Y., Tiwari, M., Posse, C., Shah, S.: Metaphor: A system for related search recommendations. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management, CIKM ‘12, pp. 664–673. ACM, New York, NY, USA (2012). DOI 10.1145/2396761.2396847. URL http://doi.acm.org/10.1145/2396761.2396847

  65. Rendle, S.: Factorization Machines. In: Proc. of 2010 IEEE ICDM, pp. 995–1000. IEEE (2010). DOI 10.1109/ICDM.2010.127. URL http://dx.doi.org/10.1109/ICDM.2010.127

  66. Rendle, S., Freudenthaler, C., Gantner, Z., Thieme, L.S.: BPR: Bayesian personalized ranking from implicit feedback. In: Proceedings of the 25th UAI, UAI ‘09, pp. 452–461. AUAI Press, Arlington, Virginia, United States (2009). URL http://portal.acm.org/citation.cfm?id=1795167

  67. Rendle, S., Freudenthaler, C., Thieme, L.S.: Factorizing personalized Markov chains for next-basket recommendation. In: Proc. of the 19th WWW, WWW ‘10, pp. 811–820. ACM, New York, NY, USA (2010). DOI 10.1145/1772690.1772773. URL http://dx.doi.org/10.1145/1772690.1772773

  68. Rendle, S., Gantner, Z., Freudenthaler, C., Schmidt-Thieme, L.: Fast context-aware recommendations with factorization machines. In: Proc. of the 34th ACM SIGIR, SIGIR ‘11, pp. 635–644. ACM, New York, NY, USA (2011). DOI 10.1145/2009916.2010002. URL http://doi.acm.org/10.1145/2009916.2010002

  69. Ribeiro, M.T., Lacerda, A., Veloso, A., Ziviani, N.: Pareto-efficient hybridization for multi-objective recommender systems. In: Proceedings of the sixth ACM conference on Recommender systems, RecSys ‘12, pp. 19–26. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2365962. URL http://doi.acm.org/10.1145/2365952.2365962

  70. Rodriguez, M., Posse, C., Zhang, E.: Multiple objective optimization in recommender systems. In: Proceedings of the sixth ACM conference on Recommender systems, RecSys ‘12, pp. 11–18. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2365961. URL http://doi.acm.org/10.1145/2365952.2365961

  71. Salakhutdinov, R., Mnih, A., Hinton, G.E.: Restricted Boltzmann machines for collaborative filtering. In: Proc of ICML ‘07. ACM, New York, NY, USA (2007)

    Google Scholar 

  72. Science: Rockin’ to the Music Genome. Science 311(5765), 1223d– (2006). DOI 10.1126/science.311.5765.1223d. URL http://www.sciencemag.org

  73. Sha, X., Quercia, D., Michiardi, P., Dell’Amico, M.: Spotting trends: the wisdom of the few. In: Proc. of the Recsys ‘12, RecSys ‘12, pp. 51–58. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2365967. URL http://doi.acm.org/10.1145/2365952.2365967

  74. Shardanand, U., Maes, P.: Social information filtering: algorithms for automating word of mouth. In: Proc. of SIGCHI ‘95, CHI ‘95, pp. 210–217. ACM Press/Addison-Wesley Publishing Co., New York, NY, USA (1995). DOI 10.1145/223904.223931. URL http://dx.doi.org/10.1145/223904.223931

  75. Shi, Y., Karatzoglou, A., Baltrunas, L., Larson, M., Hanjalic, A., Oliver, N.: TFMAP: optimizing MAP for top-n context-aware recommendation. In: Proc. of the 35th SIGIR, SIGIR ‘12, pp. 155–164. ACM, New York, NY, USA (2012). DOI 10.1145/2348283.2348308. URL http://doi.acm.org/10.1145/2348283.2348308

  76. Shi, Y., Karatzoglou, A., Baltrunas, L., Larson, M., Oliver, N., Hanjalic, A.: CLiMF: learning to maximize reciprocal rank with collaborative less-is-more filtering. In: Proc. of the sixth Recsys, RecSys ‘12, pp. 139–146. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2365981. URL http://dx.doi.org/10.1145/2365952.2365981

  77. Sigurbjörnsson, B., van Zwol, R.: Flickr tag recommendation based on collective knowledge. In: Proceedings of the 17th International Conference on World Wide Web, WWW ‘08, pp. 327–336. ACM, New York, NY, USA (2008). DOI 10.1145/1367497.1367542. URL http://doi.acm.org/10.1145/1367497.1367542

  78. Steck, H.: Training and testing of recommender systems on data missing not at random. In: Proc. of the 16th ACM SIGKDD, KDD ‘10, pp. 713–722. ACM, New York, NY, USA (2010). DOI 10.1145/1835804.1835895. URL http://dx.doi.org/10.1145/1835804.1835895

  79. Steck, H.: Item popularity and recommendation accuracy. In: Proceedings of the fifth ACM conference on Recommender systems, RecSys ‘11, pp. 125–132. ACM, New York, NY, USA (2011). DOI 10.1145/2043932.2043957. URL http://doi.acm.org/10.1145/2043932.2043957

  80. Steck, H.: Evaluation of recommendations: Rating-prediction and ranking. In: Proceedings of the 7th ACM Conference on Recommender Systems, RecSys ‘13, pp. 213–220. ACM, New York, NY, USA (2013). DOI 10.1145/2507157.2507160. URL http://doi.acm.org/10.1145/2507157.2507160

  81. Stern, D.H., Herbrich, R., Graepel, T.: Matchbox: large scale online bayesian recommendations. In: Proc. of the 18th WWW, WWW ‘09, pp. 111–120. ACM, New York, NY, USA (2009). DOI 10.1145/1526709.1526725. URL http://dx.doi.org/10.1145/1526709.1526725

  82. Takács, G., Pilászy, I., Németh, B., Tikk, D.: Major components of the gravity recommendation system. SIGKDD Explor. Newsl. 9(2), 80–83 (2007). DOI 10.1145/1345448.1345466. URL http://doi.acm.org/10.1145/1345448.1345466

  83. Takács, G., Tikk, D.: Alternating least squares for personalized ranking. In: Proc. of Recsys ‘12, RecSys ‘12, pp. 83–90. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2365972. URL http://doi.acm.org/10.1145/2365952.2365972

  84. Tan, M., Xia, T., Guo, L., Wang, S.: Direct optimization of ranking measures for learning to rank models. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ‘13, pp. 856–864. ACM, New York, NY, USA (2013). DOI 10.1145/2487575.2487630. URL http://doi.acm.org/10.1145/2487575.2487630

  85. Teh, Y.W., Jordan, M.I., Beal, M.J., Blei, D.M.: Hierarchical dirichlet processes. Journal of the American Statistical Association 101 (2004)

    Google Scholar 

  86. Valizadegan, H., Jin, R., Zhang, R., Mao, J.: Learning to Rank by Optimizing NDCG Measure. In: Proc. of SIGIR ‘00, pp. 41–48 (2000). URL http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.154.8402

  87. Vargas, S., Castells, P.: Rank and relevance in novelty and diversity metrics for recommender systems. In: Proceedings of the fifth ACM conference on Recommender systems, RecSys ‘11, pp. 109–116. ACM, New York, NY, USA (2011). DOI 10.1145/2043932.2043955. URL http://doi.acm.org/10.1145/2043932.2043955

  88. Wang, J., Sarwar, B., Sundaresan, N.: Utilizing related products for post-purchase recommendation in e-commerce. In: Proceedings of the Fifth ACM Conference on Recommender Systems, RecSys ‘11, pp. 329–332. ACM, New York, NY, USA (2011). DOI 10.1145/2043932.2043995. URL http://doi.acm.org/10.1145/2043932.2043995

  89. Wang, J., Zhang, Y., Posse, C., Bhasin, A.: Is it time for a career switch? In: Proceedings of the 22Nd International Conference on World Wide Web, WWW ‘13, pp. 1377–1388. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland (2013). URL http://dl.acm.org/citation.cfm?id=2488388.2488509

  90. Weston, J., Yee, H., Weiss, R.: Learning to rank recommendations with the k-order statistic loss. In: ACM International Conference on Recommender Systems (RecSys) (2013). URL http://dl.acm.org/citation.cfm?id=2507210

  91. Xia, F., Liu, T.Y., Wang J.and Zhang, W., Li, H.: Listwise approach to learning to rank: theory and algorithm. In: Proc. of the 25th ICML, ICML ‘08, pp. 1192–1199. ACM, New York, NY, USA (2008). DOI 10.1145/1390156.1390306. URL http://dx.doi.org/10.1145/1390156.1390306

  92. Xiong, L., Chen, X., Huang, T., J. Schneider, J.G.C.: Temporal collaborative filtering with bayesian probabilistic tensor factorization. In: Proceedings of SIAM Data Mining (2010)

    Google Scholar 

  93. Xu, J., Li, H.: AdaRank: a boosting algorithm for information retrieval. In: Proc. of SIGIR ‘07, SIGIR ‘07, pp. 391–398. ACM, New York, NY, USA (2007). DOI 10.1145/1277741.1277809. URL http://dx.doi.org/10.1145/1277741.1277809

  94. Xu, J., Liu, T.Y., Lu, M., Li, H., Ma, W.Y.: Directly optimizing evaluation measures in learning to rank. In: Proc. of SIGIR ‘08, pp. 107–114. ACM, New York, NY, USA (2008). DOI 10.1145/1390334.1390355. URL http://dx.doi.org/10.1145/1390334.1390355

  95. Y, K., Sill, J.: OrdRec: an ordinal model for predicting personalized item rating distributions. In: RecSys ‘11, pp. 117–124 (2011)

    Google Scholar 

  96. Yang, S., Long, B., Smola, A., Zha, H., Zheng, Z.: Collaborative competitive filtering: learning recommender using context of user choice. In: Proc. of the 34th ACM SIGIR, SIGIR ‘11, pp. 295–304. ACM, New York, NY, USA (2011). DOI 10.1145/2009916.2009959. URL http://dx.doi.org/10.1145/2009916.2009959

  97. Yang, X., Steck, H., Guo, Y., Liu, Y.: On top-k recommendation using social networks. In: Proc. of RecSys ‘12, RecSys ‘12, pp. 67–74. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2365969. URL http://doi.acm.org/10.1145/2365952.2365969

  98. Yi, J., Chen, Y., Li, J., Sett, S., Yan, T.W.: Predictive model performance: Offline and online evaluations. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ‘13, pp. 1294–1302. ACM, New York, NY, USA (2013). DOI 10.1145/2487575.2488215. URL http://doi.acm.org/10.1145/2487575.2488215

Download references

Author information

Authors and Affiliations

  1. Netflix, 100 Winchester Cr., Los Gatos, CA, 95032, USA

    Xavier Amatriain & Justin Basilico

  2. Quora, 150 Castro St., Mountain View, USA

    Xavier Amatriain

Authors
  1. Xavier Amatriain
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Justin Basilico
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xavier Amatriain .

Editor information

Editors and Affiliations

  1. Faculty of Computer Science, Free University of Bozen-Bolzano, Bolzano - Bozen, Italy

    Francesco Ricci

  2. Information Systems Engineering, Ben-Gurion University of the Negev, Beer-Sheva, Israel

    Lior Rokach

  3. Ben-Gurion University of the Negev, Beer-Sheva, Israel

    Bracha Shapira

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer Science+Business Media New York

About this chapter

Cite this chapter

Amatriain, X., Basilico, J. (2015). Recommender Systems in Industry: A Netflix Case Study. In: Ricci, F., Rokach, L., Shapira, B. (eds) Recommender Systems Handbook. Springer, Boston, MA. https://doi.org/10.1007/978-1-4899-7637-6_11

Download citation

  • DOI: https://doi.org/10.1007/978-1-4899-7637-6_11

  • Publisher Name: Springer, Boston, MA

  • Print ISBN: 978-1-4899-7636-9

  • Online ISBN: 978-1-4899-7637-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation