Steven Dow
Scott Klemmer
Björn Hartmann
ABSTRACT
Micro-task platforms provide massively parallel, on-demand labor. However, it can be difficult to reliably achieve high-quality work because online workers may behave irresponsibly, misunderstand the task, or lack necessary skills. This paper investigates whether timely, task-specific feedback helps crowd workers learn, persevere, and produce better results. We investigate this question through Shepherd, a feedback system for crowdsourced work. In a between-subjects study with three conditions, crowd workers wrote consumer reviews for six products they own. Participants in the None condition received no immediate feedback, consistent with most current crowdsourcing practices. Participants in the Self-assessment condition judged their own work. Participants in the External assessment condition received expert feedback. Self-assessment alone yielded better overall work than the None condition and helped workers improve over time. External assessment also yielded these benefits. Participants who received external assessment also revised their work more. We conclude by discussing interaction and infrastructure approaches for integrating real-time assessment into online work.
- Annett, J. Feedback and human behaviour: the effects of knowledge of results, incentives, and reinforcement on learning and performance. Penguin Books, 1969.Google Scholar
- Bernstein, M.S., Little, G., Miller, R.C., Hartmann, B., Ackerman, M.S., Karger, D.R., Crowell, D., and Panovich, K. Soylent: a word processor with a crowd inside. Proc of ACM Symposium on User Interface Software and Technology (2010), 313--322. Google ScholarDigital Library
- Bigham, J.P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R.C., Miller, R., Tatarowicz, A., White, B., White, S., and Yeh, T. VizWiz: nearly real-time answers to visual questions. Proc of ACM Symp. on User Interface Software and Technology (2010), 333--342. Google ScholarDigital Library
- Boud, D. Sustainable Assessment: Rethinking assessment for the learning society. Studies in Continuing Education 22, 2 (2000), 151.Google ScholarCross Ref
- Chandler, D. and Kapelner, A. Breaking monotony with meaning: Motivation in crowdsourcing markets. University of Chicago mimeo, (2010).Google Scholar
- Cheshire, C. and Antin, J. The Social Psychological Effects of Feedback on the Production of Internet Information Pools. Journal of Computer-Mediated Communication 13, 3 (2008), 705--727.Google ScholarCross Ref
- David, B. Enhancing Learning Through Self-Assessment. Routledge, 1995.Google Scholar
- Ericsson, K.A., Charness, N., Feltovich, P.J., and Hoffman, R.R. The Cambridge Handbook of Expertise and Expert Performance. Cambr. Univer. Press, 2006.Google ScholarCross Ref
- Gawande, A. The Checklist Manifesto: How to Get Things Right. Metropolitan Books, 2009.Google Scholar
- Hanrahan, S.J. and Isaacs, G. Assessing Self- and Peer-assessment: The students' views. Higher Education Research & Development 20, 1 (2001), 53.Google ScholarCross Ref
- Havighurst, R.J. Human Development and Education. Longmans, Green and Co, 1955.Google Scholar
- Heer, J. and Bostock, M. Crowdsourcing graphical perception: using mechanical turk to assess visualization design. Proc of ACM conf on Human factors in computing systems (2010), 203--212. Google ScholarDigital Library
- Hinds, P. The Curse of Expertise: The Effects of Expertise and Debiasing Methods on Predictions of Novice Performance. Journal of Experimental Applied Psychology 5, (1999), 205--221.Google ScholarCross Ref
- Horton, J.J. Employer Expectations, Peer Effects and Productivity: Evidence from a Series of Field Experiments. SSRN eLibrary, (2010).Google Scholar
- Hullman, J., Adar, E., and Shah, P. The impact of social information on visual judgments. Proc of conf on Human factors in computing systems, ACM (2011), 1461--1470. Google ScholarDigital Library
- Ipeirotis, P.G., Provost, F., and Wang, J. Quality management on Amazon Mechanical Turk. Proc of ACM SIGKDD Workshop on Human Computation, (2010), 64--67. Google ScholarDigital Library
- Ipeirotis, P.G. Analyzing the Amazon Mechanical Turk marketplace. XRDS: Crossroads, The ACM Magazine for Students 17, 2010, 16--21. Google ScholarDigital Library
- Karger, D., Oh, S., and Shah, D. Budget-optimal Crowdsourcing using Low-rank Matrix Approximations. Proc. of the Allerton Conf. on Communication, Control, and Computing, (2011).Google ScholarCross Ref
- Kittur, A., Chi, E.H., and Suh, B. Crowdsourcing user studies with Mechanical Turk. Proc of SIGCHI conference on Human factors in computing systems, ACM (2008), 453--456. Google ScholarDigital Library
- Lampe, C. and Resnick, P. Slash(dot) and burn: distributed moderation in a large online conversation space. Proc. of the SIGCHI conference on Human factors in computing systems, ACM (2004), 543--550. Google ScholarDigital Library
- Lave, J. and Wenger, E. Situated Learning: Legitimate Peripheral Participation. Camb. University Press, 1991.Google ScholarCross Ref
- Levenshtein, V.I. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady 10, 8, 707--710.Google Scholar
- Little, G., Chilton, L.B., Goldman, M., and Miller, R.C. Exploring iterative and parallel human computation processes. Proc. of the ACM SIGKDD Workshop on Human Computation, ACM (2010), 68--76. Google ScholarDigital Library
- Masten, A.S., Morison, P., and Pellegrini, D.S. A revised class play method of peer assessment. Developmental Psychology 21, 3 (1985), 523--533.Google ScholarCross Ref
- Mattheos, N., Nattestad, A., Falk-Nilsson, E., and Attstrom, R. The interactive examination: assessing students' self-assessment ability. Medical Education 38, 4 (2004), 378--389.Google ScholarCross Ref
- Musico, C. There's No Place Like Home. destinationCRM.com, 2008.Google Scholar
- Orsmond, P., Merry, S., and Reiling, K. A Study in Self-assessment: tutor and students' perceptions of performance criteria. Assessment & Evaluation in Higher Education 22, 4 (1997), 357.Google ScholarCross Ref
- Sadler, D.R. Formative assessment and the design of instructional systems. Instructional Science 18, 2 (1989), 119--144.Google ScholarCross Ref
- Shute, V.J. Focus on Formative Feedback. Review of Educational Research 78, 1 (2008), 153 -189.Google ScholarCross Ref
- Silberman, M.S., Ross, J., Irani, L., and Tomlinson, B. Sellers' problems in human computation markets. Proc ACM SIGKDD Workshop on Human Computation, (2010), 18--21. Google ScholarDigital Library
- Taras, M. Using Assessment for Learning and Learning from Assessment. Assessment & Evaluation in Higher Education 27, 6 (2002), 501.Google ScholarCross Ref
- Taras, M. To Feedback or Not to Feedback in Student Self-assessment. Assessment & Evaluation in Higher Education 28, 5 (2003), 549.Google ScholarCross Ref
- Viégas, F., Wattenberg, M., and Mckeon, M. The Hidden Order of Wikipedia. In Online Communities and Social Computing. 2007, 445--454. Google ScholarDigital Library
- Zimmerman, B.J. Becoming a self-regulated learner: Which are the key subprocesses? Contemporary Educational Psychology 11, 4 (1986), 307--313.Google ScholarCross Ref
- http://en.wikipedia.org/.Google Scholar
- http://www.thejohnnycashproject.com/.Google Scholar
- Crowdflower. Crowdflower.com/.Google Scholar
Index Terms
- Shepherding the crowd yields better work
Recommendations
Shepherding the crowd: managing and providing feedback to crowd workers
CHI EA '11: CHI '11 Extended Abstracts on Human Factors in Computing SystemsMicro-task platforms provide a marketplace for hiring people to do short-term work for small payments. Requesters often struggle to obtain high-quality results, especially on content-creation tasks, because work cannot be easily verified and workers can ...
The Effects of Sequence and Delay on Crowd Work
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing SystemsA common approach in crowdsourcing is to break large tasks into small microtasks so that they can be parallelized across many crowd workers and so that redundant work can be more easily compared for quality control. In practice, this can result in the ...
Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments
The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...
Comments