Skip to main content
SpringerLink
Log in

Characterizing Relevance on Mobile and Desktop

  • Conference paper
Book cover Advances in Information Retrieval (ECIR 2016)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 9626))

Included in the following conference series:

Abstract

Relevance judgments are central to Information retrieval evaluation. With increasing number of hand held devices at users disposal today, and continuous improvement in web standards and browsers, it has become essential to evaluate whether such devices and dynamic page layouts affect users notion of relevance. Given dynamic web pages and content rendering, we know little about what kind of pages are relevant on devices other than desktop. With this work, we take the first step in characterizing relevance on mobiles and desktop. We collect crowd sourced judgments on mobile and desktop to systematically determine whether screen size of a device and page layouts impact judgments. Our study shows that there are certain difference between mobile and desktop judgments. We also observe different judging times, despite similar inter-rater agreement on both devices. Finally, we also propose and evaluate display and viewport specific features to predict relevance. Our results indicate that viewport based features can be used to reliably predict mobile relevance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://trec.nist.gov/.

  2. 2.

    http://research.nii.ac.jp/ntcir/.

  3. 3.

    Viewport is the framed area on a display screen of mobile or desktop for viewing information.

  4. 4.

    AMT (https://requester.mturk.com/) is a crowd sourcing marketplace to conduct experiments by recruiting multiple participants in exchange for compensation.

  5. 5.

    rel = high-rel+rel, non-rel=some-rel+non-rel.

  6. 6.

    http://www.seleniumhq.org/.

References

  1. Borlund, P.: The concept of relevance in IR. J. Am. Soc. Inf. Sci. Technol. 54(10), 913–925 (2003)

    Article  Google Scholar 

  2. Buchanan, G., Farrant, S., Jones, M., Thimbleby, H., Marsden, G., Pazzani, M.: Improving mobile internet usability. In: Proceedings of WWW. ACM (2001)

    Google Scholar 

  3. Church, K., Oliver, N.: Understanding mobile web and mobile search usein today’s dynamic mobile landscape. In: Proceedings of MobileHCI. ACM (2011)

    Google Scholar 

  4. Church, K. Smyth, B.: Understanding the intent behind mobile information needs. In: Proceedings of IUI. ACM (2009)

    Google Scholar 

  5. Church, K., Smyth, B., Bradley, K., Cotter, P.: A large scale studyof european mobile search behaviour. In: Proceedings MobileHCI. ACM (2008)

    Google Scholar 

  6. Church, K., Smyth, B., Cotter, P., Bradley, K.: Mobile informationaccess: a study of emerging search behavior on the mobile internet. ACM Trans. Web 1(1), 4 (2007)

    Article  Google Scholar 

  7. Guo, Q., Jin, H., Lagun, D., Yuan, S., Agichtein, E.: Miningtouch interaction data on mobile devices to predict web search result relevance. In: Proceedings of SIGIR. ACM (2013)

    Google Scholar 

  8. Kamvar, M., Baluja, S.: A large scale study of wireless searchbehavior: Google mobile search. In: Proceedings SIGCHI. ACM (2006)

    Google Scholar 

  9. Kamvar, M., Baluja, S.: Deciphering trends in mobile search. Computer 40(8), 58–62 (2007)

    Article  Google Scholar 

  10. Kamvar, M., Kellar, M., Patel, R., Xu, Y.: Computers and iphonesand mobile phones, oh my!: a logs-based comparison of search users on differentdevices. In: Proceedings of WWW. ACM (2009)

    Google Scholar 

  11. Kazai, G., Kamps, J., Milic-Frayling, N.: An analysis of humanfactors and label accuracy in crowdsourcing relevance judgments. Inf. Retr. (2013)

    Google Scholar 

  12. Li, J., Huffman, S., Tokuda, A.: Good abandonment in mobile andpc internet search. In: Proceedings of SIGIR. ACM (2009)

    Google Scholar 

  13. Schamber, L., Eisenberg, M.: Relevance: The search for a definition (1988)

    Google Scholar 

  14. Shokouhi, M., Jones, R., Ozertem, U., Raghunathan, K., Diaz, F.: Mobile query reformulations. In: Proceedings SIGIR. ACM (2014)

    Google Scholar 

  15. Song, Y., Ma, H., Wang, H., Wang, K.: Exploring and exploiting user search behavior on mobile and tablet devices to improve search relevance. In: Proceedings of WWW (2013)

    Google Scholar 

  16. Tombros, A., Ruthven, I., Jose, J.M.: How users assess web pages for information seeking. J. Am. Soc. Inf. Sci. Technol. 56(4), 327–344 (2005)

    Article  Google Scholar 

  17. Tossell, C., Kortum, P., Rahmati, A., Shepard, C., Zhong, L.: Characterizing web use on smartphones. In: Proceedings of the SIGCHI. ACM (2012)

    Google Scholar 

  18. Xu, Y.C., Chen, Z.: Relevance judgment: What do informationusers consider beyond topicality? JASIST 57(7), 961–973 (2006)

    Article  Google Scholar 

  19. Yi, J., Maghoul, F., Pedersen, J.: Deciphering mobile search pat-terns: a study of yahoo! mobile search queries. In: Proceedings of the WWW. ACM (2008)

    Google Scholar 

  20. Zhang, Y., Zhang, J., Lease, M., Gwizdka, J.: Multidimensional relevance modeling via psychometrics and crowdsourcing. In: Proceedings of the SIGIR. ACM (2014)

    Google Scholar 

  21. Zhu, J., Zou, H., Rosset, S., Hastie, T.: Multi-class adaboost. Stat. Interface 2(3), 349–360 (2009)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University College London, London, UK

    Manisha Verma & Emine Yilmaz

Authors
  1. Manisha Verma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Emine Yilmaz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Manisha Verma .

Editor information

Editors and Affiliations

  1. Department of Information Engineering, University of Padua, Padova, Italy

    Nicola Ferro

  2. Faculty of Informatics, University of Lugano (USI), Lugano, Switzerland

    Fabio Crestani

  3. Department of Computer Science, Katholieke Universiteit Leuven, Heverlee, Belgium

    Marie-Francine Moens

  4. Systèmes d’informations, Big Data et Recherche d’Information, Institut de Recherche en Informatique de Toulouse IRIT/équipe SIG, Toulouse Cedex 04, France

    Josiane Mothe

  5. Yahoo! Labs London, London, UK

    Fabrizio Silvestri

  6. Department of Information Engineering, University of Padua, Padova, Italy

    Giorgio Maria Di Nunzio

  7. TU Delft - EWI/ST/WIS, Delft, The Netherlands

    Claudia Hauff

  8. Department of Information Engineering, University of Padua, Padova, Italy

    Gianmaria Silvello

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Verma, M., Yilmaz, E. (2016). Characterizing Relevance on Mobile and Desktop. In: Ferro, N., et al. Advances in Information Retrieval. ECIR 2016. Lecture Notes in Computer Science(), vol 9626. Springer, Cham. https://doi.org/10.1007/978-3-319-30671-1_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-30671-1_16

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-30670-4

  • Online ISBN: 978-3-319-30671-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation