Pavel Kucherbaev
Florian Daniel
ABSTRACT
Task execution timeliness, i.e., the completion of a task within a given time frame, is a known open issue in crowdsourcing. While running tasks on crowdsourcing platforms a requester experiences long tails in execution caused by abandoned assignments (those left by workers unfinished), which become available for other workers only after some expiration time (e.g., 30 minutes in CrowdFlower). These abandoned assignments result in significant delays and a poor predictability of the overall task execution time. In this paper, we propose an approach and an implementation called ReLauncher to identify such abandoned assignments and relaunch them for other workers. We evaluate our implementation with an experiment on CrowdFlower that provides substantive evidence for a significant execution speed improvement with an average extra cost of about 10%.
- Michael S. Bernstein, Joel Brandt, Robert C. Miller, and David R. Karger. 2011. Crowds in Two Seconds: Enabling Realtime Crowd-powered Interfaces. In UIST 2011. 33–42. Google Scholar
Digital Library
- Ioannis Boutsis and Vana Kalogeraki. 2013. Crowdsourcing under Real-Time Constraints. In IPDPS 2013. 753–764. Google ScholarDigital Library
- Alessandro Bozzon, Marco Brambilla, Stefano Ceri, and Andrea Mauri. 2013. Reactive crowdsourcing. In WWW 2013. 153–164. Google ScholarDigital Library
- Dana Chandler and John Joseph Horton. 2011. Labor Allocation in Paid Crowdsourcing: Experimental Evidence on Positioning, Nudges and Prices. In HCOMP 2011.Google Scholar
- Lydia B. Chilton, John J. Horton, Robert C. Miller, and Shiri Azenkot. 2010. Task Search in a Human Computation Market. In HCOMP 2010. 1–9. Google ScholarDigital Library
- Djellel E. Difallah, Michele Catasta, Gianluca Demartini, and Philippe Cudré-Mauroux. 2014. Scaling-up the Crowd: Micro-Task Pricing Schemes for Worker Retention and Latency Improvement. In HCOMP 2014.Google Scholar
- Siamak Faridani, Bjoern Hartmann, and Panagiotis G. Ipeirotis. 2011. What's the Right Price? Pricing Tasks for Finishing on Time.. In HCOMP 2011.Google Scholar
- Jeff Howe. 2008. Crowdsourcing: why the power of the crowd is driving the future of business (1st ed.). Crown Publishing Group, New York, NY, USA. Google ScholarDigital Library
- Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2010. TurKit: Human Computation Algorithms on Mechanical Turk. In UIST 2010. 57–66. Google ScholarDigital Library
- Winter Mason and Duncan J. Watts. 2009. Financial Incentives and the 'Performance of Crowds'. In HCOMP 2009. 77–85. Google ScholarDigital Library
- Massolution. 2013. The crowd in the cloud: exploring the future of outsourcing. White Paper. (January 2013).Google Scholar
- Donna Vakharia and Matthew Lease. 2015. Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms. In Proceedings of the iConference.Google Scholar
- Jing Wang, Siamak Faridani, and Panagiotis G. Ipeirotis. 2011. Estimating Completion Time for Crowdsourced Tasks Using Survival Analysis Models. In CSDM 2011. 31–34.Google Scholar
- Man-Ching Yuen, I. King, and Kwong-Sak Leung. 2011. A Survey of Crowdsourcing Systems. In PASSAT-SocialCom 2011. 766–773.Google Scholar
Recommendations
Involving Senior Workers in Crowdsourced Proofreading
Proceedings of the 8th International Conference on Universal Access in Human-Computer Interaction. Aging and Assistive Environments - Volume 8515Seniors have a wealth of knowledge and free time, so they are a promising workforce for crowdsourced tasks. Currently senior workers are hardly involved in real applications. We have started an experimental project that crowdsources proofreading micro-...
Age-Based task specialization for crowdsourced proofreading
UAHCI'13: Proceedings of the 7th international conference on Universal Access in Human-Computer Interaction: user and context diversity - Volume 2Crowdsourcing can efficiently produce accessible digital books for people with print disabilities. However, particularly in Japan, the proofreading step tends to be expensive because of language-related issues. The elderly population is a promising ...
Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments
The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...
Comments