Skip to main content
SpringerLink
Log in
SpringerLink
Log in

Online Evaluations for Everyone: Mr. DLib’s Living Lab for Scholarly Recommendations

  • Conference paper
  • First Online:
Advances in Information Retrieval (ECIR 2019)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11438))

Included in the following conference series:

Abstract

We introduce the first ‘living lab’ for scholarly recommender systems. This lab allows recommender-system researchers to conduct online evaluations of their novel algorithms for scholarly recommendations, i.e., recommendations for research papers, citations, conferences, research grants, etc. Recommendations are delivered through the living lab’s API to platforms such as reference management software and digital libraries. The living lab is built on top of the recommender-system as-a-service Mr. DLib. Current partners are the reference management software JabRef and the CORE research team. We present the architecture of Mr. DLib’s living lab as well as usage statistics on the first sixteen months of operating it. During this time, 1,826,643 recommendations were delivered with an average click-through rate of 0.21%.

This publication has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under Grant Number 13/RC/2106. We are further grateful for the support received by Samuel Pearce and Siddharth Dinesh.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Hanbury, A., et al.: Evaluation-as-a-service: overview and outlook. arXiv preprint arXiv:1512.07454 (2015)

  2. Hopfgartner, F., et al.: Report on the evaluation-as-a-service (EaaS) expert workshop. In: ACM SIGIR Forum, pp. 57–65. ACM (2015)

    Google Scholar 

  3. Hopfgartner, F., et al.: Evaluation-as-a-service for the computational sciences: overview and outlook. J. Data Inf. Qual. (JDIQ) 10, 15 (2018)

    Google Scholar 

  4. Brodt, T., Hopfgartner, F.: Shedding light on a living lab: the CLEF NEWSREEL open recommendation platform. In: Proceedings of the 5th Information Interaction in Context Symposium, pp. 223–226. ACM (2014)

    Google Scholar 

  5. Hopfgartner, F., et al.: Benchmarking news recommendations: the clef newsreel use case. In: ACM SIGIR Forum, pp. 129–136. ACM (2016)

    Google Scholar 

  6. Kille, B., et al.: Overview of NewsREEL’16: multi-dimensional evaluation of real-time stream-recommendation algorithms. In: Fuhr, N., et al. (eds.) CLEF 2016. LNCS, vol. 9822, pp. 311–331. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-44564-9_27

    Chapter  Google Scholar 

  7. Balog, K., Elsweiler, D., Kanoulas, E., Kelly, L., Smucker, M.D.: Report on the CIKM workshop on living labs for information retrieval evaluation. In: ACM SIGIR Forum, pp. 21–28. ACM (2014)

    Google Scholar 

  8. Carevic, Z., Schüller, S., Mayr, P., Fuhr, N.: Contextualised browsing in a digital library’s living lab. In: Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries, Fort Worth, Texas, USA, pp. 89–98. ACM (2018)

    Google Scholar 

  9. Li, S., Brusilovsky, P., Su, S., Cheng, X.: Conference paper recommendation for academic conferences. IEEE Access 6, 17153–17164 (2018)

    Article  Google Scholar 

  10. Vargas, S., Hristakeva, M., Jack, K.: Mendeley: recommendations for researchers. In: Proceedings of the 10th ACM Conference on Recommender Systems, Boston, Massachusetts, USA, pp. 365–365. ACM (2016)

    Google Scholar 

  11. Färber, M., Thiemann, A., Jatowt, A.: CITEWERTs: a system combining cite-worthiness with citation recommendation. In: Pasi, G., Piwowarski, B., Azzopardi, L., Hanbury, A. (eds.) ECIR 2018. LNCS, vol. 10772, pp. 815–819. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-76941-7_82

    Chapter  Google Scholar 

  12. Jia, H., Saule, E.: Graph Embedding for Citation Recommendation, arXiv preprint arXiv:1812.03835 (2018)

  13. Beierle, F., Tan, J., Grunert, K.: Analyzing social relations for recommending academic conferences. In: Proceedings of the 8th ACM International Workshop on Hot Topics in Planet-scale mObile computing and online Social neTworking, pp. 37–42. ACM (2016)

    Google Scholar 

  14. Yu, S., Liu, J., Yang, Z., Chen, Z., Jiang, H., Tolba, A., Xia, F.: PAVE: personalized academic venue recommendation exploiting co-publication networks. J. Netw. Comput. Appl. 104, 38–47 (2018)

    Article  Google Scholar 

  15. Kou, N.M., Mamoulis, N., Li, Y., Li, Y., Gong, Z., et al.: A topic-based reviewer assignment system. Proc. VLDB Endowment 8, 1852–1855 (2015)

    Article  Google Scholar 

  16. Lian, J.W., Mattei, N., Noble, R., Walsh, T.: The conference paper assignment problem: using order weighted averages to assign indivisible goods. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)

    Google Scholar 

  17. Tan, J., Wan, X., Liu, H., Xiao, J.: QuoteRec: toward quote recommendation for writing. ACM Trans. Inf. Syst. (TOIS) 36, 34 (2018)

    Article  Google Scholar 

  18. Kong, X., Jiang, H., Wang, W., Bekele, T.M., Xu, Z., Wang, M.: Exploring dynamic research interest and academic influence for scientific collaborator recommendation. Scientometrics 113, 369–385 (2017)

    Article  Google Scholar 

  19. Moreira, G.S.P., de Souza, G.A., da Cunha, A.M.: Comparing offline and online recommender system evaluations on long-tail distributions. In: Proceedings of the ACM Recommender Systems Conference RecSys (2015)

    Google Scholar 

  20. Rossetti, M., Stella, F., Zanker, M.: Contrasting offline and online results when evaluating recommendation algorithms. In: Proceedings of the 10th ACM Conference on Recommender Systems, Boston, Massachusetts, USA, pp. 31–34. ACM (2016)

    Google Scholar 

  21. Beel, J., Aizawa, A., Breitinger, C., Gipp, B.: Mr. DLib: recommendations-as-a-service (RaaS) for academia. In: Proceedings of the 17th ACM/IEEE Joint Conference on Digital Libraries, Toronto, Ontario, Canada, pp. 313–314. IEEE Press (2017)

    Google Scholar 

  22. Beel, J., Collins, A., Aizawa, A.: The architecture of Mr. DLib’s scientific recommender-system API. In: Proceedings of the 26th Irish Conference on Artificial Intelligence and Cognitive Science (AICS), CEUR-WS, pp. 78–89 (2018)

    Google Scholar 

  23. Feyer, S., Siebert, S., Gipp, B., Aizawa, A., Beel, J.: Integration of the scientific recommender system Mr. DLib into the reference manager JabRef. In: Jose, Joemon M., et al. (eds.) ECIR 2017. LNCS, vol. 10193, pp. 770–774. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-56608-5_80

    Chapter  Google Scholar 

  24. Kopp, O., Breitenbuecher, U., Mueller, T.: CloudRef - towards collaborative reference management in the cloud. In: Proceedings of the 10th Central European Workshop on Services and their Composition (2018)

    Google Scholar 

  25. Hristakeva, M., et al.: Building recommender systems for scholarly information. In: Proceedings of the 1st Workshop on Scholarly Web Mining, pp. 25–32. ACM (2017)

    Google Scholar 

  26. Knoth, P., et al.: Towards effective research recommender systems for repositories. In: Proceedings of the Open Repositories Conference (2017)

    Google Scholar 

  27. Pontika, N., Anastasiou, L., Charalampous, A., Cancellieri, M., Pearce, S., Knoth, P.: CORE recommender: a plug in suggesting open access content (2017). http://hdl.handle.net/1842/23359

  28. Beel, J., Dinesh, S., Mayr, P., Carevic, Z., Raghvendra, J.: Stereotype and most-popular recommendations in the digital library sowiport. In: Proceedings of the 15th International Symposium of Information Science (ISI), pp. 96–108 (2017)

    Google Scholar 

  29. Beel, J., Smyth, B., Collins, A.: RARD II: The 2nd Related-Article Recommendation Dataset. arXiv:1807.06918 [cs.IR] (2018)

  30. Beel, J., Langer, S., Gipp, B., Nuernberger, A.: The architecture and datasets of Docear’s research paper recommender system. D-Lib Mag. 20 (2014)

    Google Scholar 

  31. Hienert, D., Sawitzki, F., Mayr, P.: Digital library research in action-supporting information retrieval in sowiport. D-Lib Mag. 21 (2015)

    Google Scholar 

  32. Mayr, P.: Sowiport User Search Sessions Data Set (SUSS). GESIS Datorium (2016)

    Google Scholar 

  33. Stempfhuber, M., Schaer, P., Shen, W.: Enhancing visibility: integrating grey literature in the SOWIPORT information cycle. In: International Conference on Grey Literature, pp. 23–29 (2008)

    Google Scholar 

  34. Beel, J., Breitinger, C., Langer, S., Lommatzsch, A., Gipp, B.: Towards reproducibility in recommender-systems research. User Model. User-Adap. Inter. (UMUAI) 26, 69–101 (2016)

    Article  Google Scholar 

  35. Ferro, N., Fuhr, N., Rauber, A.: Introduction to the special issue on reproducibility in information retrieval: tools and infrastructures. J. Data Inf. Qual. (JDIQ) 10, 14 (2018)

    Google Scholar 

  36. Collins, A., Tkaczyk, D., Beel, J.: A novel approach to recommendation algorithm selection using meta-learning. In: Proceedings of the 26th Irish Conference on Artificial Intelligence and Cognitive Science (AICS). CEUR-WS, pp. 210–219 (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Statistics, ADAPT Centre, Trinity College Dublin, Dublin, Ireland

    Joeran Beel & Andrew Collins

  2. Digital Content and Media Sciences Division, National Institute of Informatics, Tokyo, Japan

    Joeran Beel

  3. IPVS, University of Stuttgart, Stuttgart, Germany

    Oliver Kopp

  4. Department of Informatics, Technical University of Munich, Garching, Germany

    Linus W. Dietz

  5. Knowledge Media Institute, The Open University, London, UK

    Petr Knoth

Authors
  1. Joeran Beel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrew Collins
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Oliver Kopp
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Linus W. Dietz
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Petr Knoth
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joeran Beel .

Editor information

Editors and Affiliations

  1. University of Strathclyde, Glasgow, UK

    Leif Azzopardi

  2. Bauhaus Universität Weimar, Weimar, Germany

    Benno Stein

  3. Universität Duisburg-Essen, Duisburg, Germany

    Norbert Fuhr

  4. GESIS - Leibniz Institute for the Social Sciences, Cologne, Germany

    Philipp Mayr

  5. Delft University of Technology, Delft, The Netherlands

    Claudia Hauff

  6. University of Twente, Enschede, The Netherlands

    Djoerd Hiemstra

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Beel, J., Collins, A., Kopp, O., Dietz, L.W., Knoth, P. (2019). Online Evaluations for Everyone: Mr. DLib’s Living Lab for Scholarly Recommendations. In: Azzopardi, L., Stein, B., Fuhr, N., Mayr, P., Hauff, C., Hiemstra, D. (eds) Advances in Information Retrieval. ECIR 2019. Lecture Notes in Computer Science(), vol 11438. Springer, Cham. https://doi.org/10.1007/978-3-030-15719-7_27

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-15719-7_27

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-15718-0

  • Online ISBN: 978-3-030-15719-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation