Gaurav Baruah
ABSTRACT
We propose a utility-based framework for the evaluation of push notification systems that monitor document streams for users' topics of interest. Our starting point is that users derive either positive utility (i.e., "gain") or negative utility (i.e., "pain") from consuming system updates. By separately keeping track of these quantities, we can measure system effectiveness in a gain vs. pain tradeoff space. The Pareto Frontier of evaluated systems represents the state of the art: for each system on the frontier, no other system can offer more gain without more pain. Our framework has several advantages: it unifies three previous TREC evaluations, subsumes existing metrics, and provides more insightful analyses. Furthermore, our approach can easily accommodate more refined user models and is extensible to different information-seeking modalities.
- James Allan. 2002. Topic Detection and Tracking: Event-Based Information Organization. Kluwer Academic Publishers, Dordrecht, The Netherlands. Google Scholar
Digital Library
- Javed Aslam, Matthew Ekstrand-Abueg, Virgil Pavlu, Richard McCreadie, Fernando Diaz, and Tetsuya Sakai 2014. TREC 2014 Temporal Summarization Track Overview TREC.Google Scholar
- Javed Aslam, Matthew Ekstrand-Abueg, Virgil Pavlu, Fernando Diaz, and Tetsuya Sakai. 2013. TREC 2013 Temporal Summarization. In TREC.Google Scholar
- Leif Azzopardi. 2016. Simulation of Interaction: A Tutorial on Modelling and Simulating User Interaction and Search Behaviour. In SIGIR. 1227--1230. Google ScholarDigital Library
- Leif Azzopardi, Kalervo Jarvelin, Jaap Kamps, and Mark D. Smucker 2011. Report on the SIGIR 2010 Workshop on the Simulation of Interaction. SIGIR Forum, Vol. 44, 2 (2011), 35--47. Google ScholarDigital Library
- Gaurav Baruah, Mark D. Smucker, and Charles L. A. Clarke. 2015. Evaluating Streams of Evolving News Events. In SIGIR. 675--684. Google ScholarDigital Library
- Charles L. A. Clarke and Mark D. Smucker 2014. Time Well Spent IIiX '14. 205--214. Google ScholarDigital Library
- Qi Guo, Fernando Diaz, and Elad Yom-Tov 2013. Updating Users about Time Critical Events. In ECIR. 483--494. Google ScholarDigital Library
- Heikki Keskustalo, Kalervo Jarvelin, Ari Pirkola, and Jaana Kekalainen 2008. Intuition-Supporting Visualization of User's Performance Based on Explicit Negative Higher-Order Relevance. In SIGIR. 675--682. Google ScholarDigital Library
- David D. Lewis. 1995. The TREC-4 Filtering Track. In TREC. 165--180.Google Scholar
- Jimmy Lin, Miles Efron, Yulu Wang, and Garrick Sherman. 2015. Overview of the TREC-2015 Microblog Track. TREC.Google Scholar
- Jimmy Lin, Adam Roegiest, Luchen Tan, Richard McCreadie, Ellen Voorhees, and Fernando Diaz. 2016. Overview of the TREC 2016 Real-Time Summarization Track TREC.Google Scholar
- David Maxwell, Leif Azzopardi, Kalervo Jarvelin, and Heikki Keskustalo 2015. Searching and Stopping: An Analysis of Stopping Rules and Strategies CIKM. 313--322. Google ScholarDigital Library
- Alistair Moffat and Justin Zobel 2008. Rank-Biased Precision for Measurement of Retrieval Effectiveness. ACM TOIS, Vol. 27, 1, Article 2 (Dec. 2008), 27 pages. Google ScholarDigital Library
Index Terms
- The Pareto Frontier of Utility Models as a Framework for Evaluating Push Notification Systems
Recommendations
A Study of Realtime Summarization Metrics
CIKM '16: Proceedings of the 25th ACM International on Conference on Information and Knowledge ManagementUnexpected news events, such as natural disasters or other human tragedies, create a large volume of dynamic text data from official news media as well as less formal social media. Automatic real-time text summarization has become an important tool for ...
Approximating Pareto frontier using a hybrid line search approach
The aggregation of objectives in multiple criteria programming is one of the simplest and widely used approach. But it is well known that this technique sometimes fail in different aspects for determining the Pareto frontier. This paper proposes a new ...
A Comparison of Nuggets and Clusters for Evaluating Timeline Summaries
CIKM '17: Proceedings of the 2017 ACM on Conference on Information and Knowledge ManagementThere is growing interest in systems that generate timeline summaries by filtering high-volume streams of documents to retain only those that are relevant to a particular event or topic. Continued advances in algorithms and techniques for this task ...
Comments