Laurence A. F. Park
ABSTRACT
Rank aggregation is the process of aggregating multiple rankings provided by multiple assessors, of a given set of items, into a single ranking. Each assessor, whether it be human or computer based, is a resource that we use to obtain the multiple rankings. The accuracy of the aggregated ranking depends on the accuracy of the assessor ranking and the assessor coverage of the items. Our question is, given limited assessment resources, should each assessor rank many items to obtain item coverage, spending little time on each item, or should each assessor rank only a few items, but spend more time on each item to obtain a high accuracy ranking? In this article, we take a first step towards answering this question, by developing a model, based on simulation, showing the effect of the number of items assigned to an assessor and the accuracy of the assessment on the precision of the aggregated ranking. We find that when using Binomial allocation of items to assessors, increasing the assessor accuracy provides a greater increase in aggregated rank accuracy.
- N. Ailon. Aggregation of partial rankings, p-ratings and top-m lists. Algorithmica, 57(2):284--300, 2010. Google Scholar
Digital Library
- N. Ailon, M. Charikar, and A. Newman. Aggregating inconsistent information: ranking and clustering. Journal of the ACM (JACM), 55(5):23, 2008. Google ScholarDigital Library
- R. Arora and M. Meila. Consensus ranking with signed permutations. In Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics, pages 117--125, 2013.Google Scholar
- C. Dwork, R. Kumar, M. Naor, and D. Sivakumar. Rank aggregation methods for the web. In Proceedings of the 10th international conference on World Wide Web, pages 613--622. ACM, 2001. Google ScholarDigital Library
- J. Friedman, T. Hastie, R. Tibshirani, et al. Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors). The annals of statistics, 28(2):337--407, 2000.Google Scholar
- D. F. Gleich and L.-h. Lim. Rank aggregation via nuclear norm minimization. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 60--68. ACM, 2011. Google ScholarDigital Library
- J. Guiver and E. Snelson. Bayesian inference for plackett-luce ranking models. In proceedings of the 26th annual international conference on machine learning, pages 377--384. ACM, 2009. Google ScholarDigital Library
- D. S. Hochbaum. Ranking sports teams and the inverse equal paths problem. In Internet and Network Economics, pages 307--318. Springer, 2006. Google ScholarDigital Library
- A. Klementiev, D. Roth, and K. Small. Unsupervised rank aggregation with distance-based models. In Proceedings of the 25th international conference on Machine learning, pages 472--479. ACM, 2008. Google ScholarDigital Library
- S. Negahban, S. Oh, and D. Shah. Iterative ranking from pair-wise comparisons. In Advances in Neural Information Processing Systems, pages 2474--2482, 2012.Google Scholar
- L. A. F. Park and G. Stone. Inducing controlled error over variable length ranked lists. In Advances in Knowledge Discovery and Data Mining, volume 8444 of Lecture Notes in Computer Science, pages 259--270. Springer International Publishing, 2014.Google ScholarCross Ref
Index Terms
- The effect of assessor coverage and assessor accuracy on rank aggregation precision
Recommendations
Effective rank aggregation for metasearching
Nowadays, mashup services and especially metasearch engines play an increasingly important role on the Web. Most of users use them directly or indirectly to access and aggregate information from more than one data sources. Similarly to the rest of the ...
Group recommendations with rank aggregation and collaborative filtering
RecSys '10: Proceedings of the fourth ACM conference on Recommender systemsThe majority of recommender systems are designed to make recommendations for individual users. However, in some circumstances the items to be selected are not intended for personal usage but for a group; e.g., a DVD could be watched by a group of ...
Is Rank Aggregation Effective in Recommender Systems? An Experimental Analysis
Survey Paper and Regular PaperRecommender Systems are tools designed to help users find relevant information from the myriad of content available online. They work by actively suggesting items that are relevant to users according to their historical preferences or observed actions. ...
Comments