Márcia Prante Assmann
ABSTRACT
The development of crowdsourcing software is commonly organized around platforms that allow a requester to submit a task and connect it with a crowd of individuals, who will provide likely solutions to the task. However, these platforms often ignore factors that provide a satisfactory user experience, reflecting, among others, the abandonment of tasks. To this end, it is essential to identify and understand User Experiences (UX) on platforms, assist in the improvement/development of these environments, and minimize barriers imposed by unsatisfactory experiences problems. This paper aimed to investigate long-term UX on a crowdsourcing platform. It is a qualitative exploratory study, carried out from a quasi-experiment, in which 10 undergraduate students selected for convenience participated. Participants performed tasks on the TopCoder platform over 3 months and answered the online questionnaire. Throughout the period, participants reported their experiences with the platform, describing their impressions about the use. The results showed the most satisfactory experiences occurred mainly in the last weeks of study and were mainly associated with the completion and task submission. For less satisfactory experiences, the main reasons identified were mainly associated with the difficulty of use and low usability. All participants informed that they would recommend the platform to others. It is concluded that, even with usability problems, the user's experience when using the platform was satisfactory. The results can be used to suggest mechanisms that reduce the barriers imposed by problems of less satisfactory experiences, spreading the crowdsourcing software and, consequently, expanding its adoption in the software development industry.
- Alexandre Lazaretti Zanatta et al. 2017. Barriers Faced by Newcomers to Software-Crowdsourcing Projects. IEEE Software. 34, 2 (2017), 37--43. DOI:https://doi. org/10.1109/MS.2017.32.Google Scholar
Digital Library
- Alexandre Lazaretti Zanatta et al. 2016. Software Crowdsourcing Platforms. IEEE Software. 33, 6 (2016), 112--116. DOI:https://doi.org/10.1109/MS. 2016.151.Google ScholarCross Ref
- Arnold P.O.S. Vermeeren et al. 2010. User Experience Evaluation Methods: Current State and Development Needs. (2010), 521--530.Google Scholar
- Arthur Fisk et al. 2004. Designing for Older Adults: Principles and Creative Human Factors Approaches. October 2014 (2004). DOI:https://doi.org/10.1201/9781420023862.Google Scholar
- Avinoam Baruch; Andrew May and Dapeng Yu 2016. The motivations, enablers and barriers for voluntary participation in an online crowdsourcing platform. Computers in Human Behavior. 64, (2016), 923--931. DOI:https://doi.org/10.1016/j.chb. 2016.07.039.Google Scholar
- Carmen Lucia Branquinho. 2016. CROWDSOURCING uma forma de inovação aberta.Google Scholar
- Claes Wohlin et al. 2012. Experimentation in software engineering.Google Scholar
- Daiana Biduski and Ericles Andrei Bellei, João Pedro Mazuco Rodriguez, Luciana Aparecida Martinez Zaina, A.C.B.D.M. 2019. Assessing long-term user experience on a mobile health application through an in-app embedded conversation-based questionnaire. Computers in Human Behavior. (2019). DOI:https://doi.org/10.1016/j.chb.2019.106169.Google Scholar
- Daniel Kahneman and Alan B. Krueger Princeton, David Schkade, Norbert Schwarz, A.S.S. 2004. The Day Reconstruction Method (DRM): Instrument Documentation. July (2004), 1--56.Google Scholar
- David Crane et al. 2017. Factors influencing usability of a smartphone app to reduce excessive alcohol consumption: Think aloud and interview studies. Frontiers in Public Health. 5, APR (2017). DOI:https://doi.org/10.3389/FPUBH.2017.00039.Google Scholar
- Evangelos Karapanos and John Zimmerman, Jodi Forlizzi, J.-B.M. 2009. User experience over time: An initial framework. Conference on Human Factors in Computing Systems - Proceedings. (2009), 729--738. DOI:https://doi.org/10.1145/1518701.1518814.Google ScholarDigital Library
- Fershtman Chaim and Uri Gneezy 2011. The tradeoff between performance and quitting in high power tournaments. Journal of the European Economic Association. 9, 2 (2011), 318--336. DOI:https://doi.org/10.1111/j.1542-4774.2010.01012.x.Google ScholarCross Ref
- Hala Magdy Hassan and Galal-Edeen, G.H. 2018. From usability to user experience. ICIIBMS 2017 - 2nd International Conference on Intelligent Informatics and Biomedical Sciences. 2018-Janua, (2018), 216--222. DOI:https://doi.org/10.1109/ICIIBMS.2017.8279761.Google Scholar
- Hanna Schneider et al. 2016. CrowdUX: A Case for Using Widespread and Lightweight Tools in the Quest for UX. ACM. (2016), 415--426. DOI:https://doi.org/10.1145/2901790.2901814.Google ScholarDigital Library
- Javier A. Bargas-Avila and Kasper Hornbæk 2011. Old Wine in New Bottles or Novel Challenges ? A Critical Analysis of Empirical Studies of User Experience. (2011), 2689--2698.Google Scholar
- Jeff Sauro and James R. Lewis 2012. Quantifying the User Experience: Practical Statistics for User Research.Google ScholarDigital Library
- Jinghui Cheng and Jin L.C. Guo 2018. How Do the Open Source Communities Address Usability and UX Issues? An Exploratory Study. ACM. (2018), 1--6. DOI:https://doi. org/10.1145/3170427.3188467.Google Scholar
- Ke mao, Ye yang, Qing wang, Yue Jia, M.H. 2015. Developer recommendation for crowdsourced software development tasks. Proceedings - 9th IEEE International Symposium on Service-Oriented System Engineering, IEEE SOSE 2015. 30, (2015), 347--356. DOI:https://doi.org/10.1109/SOSE.2015.46.Google ScholarDigital Library
- Ke Mao; Licia Capra; Mark Harman; and Yue Jia 2017. A survey of the use of crowdsourcing in software engineering. Journal of Systems and Software. 126, (2017), 57--84. DOI:https://doi.org/10.1016/j.jss.2016.09.015.Google Scholar
- Kristine Bang and Martin Akto Kanstrup, A.K. 2016. What are They Measuring? A Literature Review of Empirical Studies of UX Evaluation Methods.Google Scholar
- Laurence Bardin 2011. Análise de Conteúdo.Google Scholar
- Lee K, Kwon H, Lee B, Lee G, Lee JH, Park YR and et al. 2018. Effect of self-monitoring on long-term patient engagement with mobile health applications. PLoS ONE. 13, 7 (2018), 1--12. DOI:https://doi.org/10.1371/journal.pone.0201166.Google Scholar
- Leticia Machado et al. 2014. Crowdsourcing in the Brazilian IT Industry: What We Know and What We Don't Crowdsourcing in the Brazilian IT Industry: What We Know and What We Don't Know. November (2014). DOI:https://doi.org/10.1145/2666539.2666567.Google ScholarDigital Library
- Mahmood Hosseini et al. 2014. The Four Pillars of Crowdsourcing: a Reference Model. May (2014). DOI:https://doi.org/10.1109/RCIS.2014.6861072.Google Scholar
- Marc Hassenzah 2001. The Effect of Perceived Hedonic Quality on Product Appealingness. International Journal of Human-Computer Interaction. November 2011 (2001), 37--41. DOI:https://doi.org/10.1207/S15327590IJHC1304.Google Scholar
- Marc Hassenzahl and Noam Tractinsky 2006. User experience - A research agenda. Behaviour and Information Technology. 25, 2 (2006), 91--97. DOI:https://doi.org/10.1080/01449290500330331.Google ScholarCross Ref
- Marion SCHMID-DRÜNER 2016. The Situation of Workers in the Collaborative Economy. European Parliameny Employment and Social Affairs. October (2016).Google Scholar
- Maurício Elias Dick et al. 2017. Experiência antecipada: a influência da imagem da marca na relação dos usuários com aplicativo móvel Spotify. Design e Tecnologia. 6, 11 (2017), 79. DOI:https://doi.org/10.23972/det2016iss11pp79-85.Google ScholarCross Ref
- Mohammad Zarour and Mubarak Alharbi 2017. User experience framework that combines aspects, dimensions, and measurement methods. Cogent Engineering. 4, 1 (2017), 1--25. DOI:https://doi.org/10.1080/23311916.2017.1421006.Google ScholarCross Ref
- Natasha Malveira Costa Valentim; Williamson Alison Freitas Silva; and Tayana Conte 2015. Avaliando a experiência do usuário e a usabilidade de um aplicativo web móvel: Um relato de experiência. Cibse. 25010, 2011 (2015), 788--801.Google Scholar
- Patrizia Marti and Iolanda Iacono 2016. Anticipated, Momentary, Episodic, Remembered: the many facets of User eXperience. Proceedings of the 2016 Federated Conference on Computer Science and Information Systems. 8, (2016), 1647--1655. DOI:https://doi.org/10.15439/2016f302.Google ScholarCross Ref
- Pedro Alberto Barbetta 2008. Estatística aplicada às ciências sociais. Editora da UFSC. (2008), 340.Google Scholar
- Pieter Desmet and Martijn Vastenburg, Van Bel, D., N.R. 2012. Pick-A-Mood development and application of a pictorial mood-reporting instrument. 8th International Conference on Design and Emotion: Out of Control - Proceedings. September (2012).Google Scholar
- Ramon R. Palacio; Christian O. Acosta; Joaquın Cortez; Alberto L. Moran 2017. Usability perception of different video game devices in elderly users. (2017), 103--113. DOI:https://doi.org/10.1007/s10209-015-0435-y.Google ScholarDigital Library
- RICARDO RODRIGO MARINHO MELO 2018. CrowdSourcing no Desenvolvimento de Software: Caracterização da Colaboração nos desafios da plataforma TopCoder. (2018).Google Scholar
- Sari Kujala et al. 2011. UX Curve: A method for evaluating long-term user experience. Interacting with Computers. 23, 5 (2011), 473--483. DOI:https://doi.org/10.1016/j.intcom.2011.06.005.Google ScholarDigital Library
- Shashank Khanna et al. 2010. Evaluating and improving the usability of Mechanical Turk for low-income workers in India. Proceedings of the 1st ACM Symposium on Computing for Development, DEV 2010. (2010). DOI:https://doi.org/10.1145/1926180.1926195.Google ScholarDigital Library
- Simon à Campo, Vasssilis-Javed Khan, Konstantinos Papangelis, P. 2019. Community heuristics for user interface evaluation of crowdsourcing platforms. Future Generation Computer Systems. 95, (2019), 775--789. DOI:https://doi.org/10.1016/j.future.2018.02.028.Google ScholarDigital Library
- Tânia Modesto Veludo de Oliveira 2001. Amostragem não Probabilística: Adequação de Situações para uso e Limitações de amostras por Conveniência, Julgamento e Quotas. Administração On Line. 2, 3 (2001), 1--7.Google Scholar
- Thomas D. LaToza et al. 2014. Microtask programming: Building software with a crowd. UIST 2014 - Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (New York, New York, USA, Oct. 2014), 43--54.Google Scholar
- Thomas D. LaToza and André van der Hoek 2016. Crowdsourcing in Software Engineering: Models, Motivations, and Challenges. Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP 2019. (2016). DOI:https://doi.org/10.1109/ICSE-SEIP.2019.00043.Google ScholarDigital Library
- Tiago Moraes Ferreira et al. 2019. CrowdRec: A prototype recomendation system for crowdsourcing platforms using Google Venture Design. ACM, New York, NY, USA. (2019), 8. DOI:https://doi.org/https://doi.org/10.1145/3330204.3330235 1.Google ScholarDigital Library
- Tim Straub et al. 2015. How (not) to Incent Crowd Workers: Payment Schemes and Feedback in Crowdsourcing. Business and Information Systems Engineering. 57, 3 (2015), 167--179. DOI:https://doi.org/10.1007/s12599-015-0384-2.Google ScholarCross Ref
- Topcoder Top Technology Talent On-Demand: https://www.topcoder.com/. Accessed: 2021-05-05.Google Scholar
- Tor Eriksson et al. 2009. Feedback and incentives: Experimental evidence. Labour Economics. 16, 6 (2009), 679--688. DOI:https://doi.org/10.1016/j.labeco. 2009.08.006.Google ScholarCross Ref
- Virpi Roto et al. 2011. USER EXPERIENCE WHITE PAPER Bringing clarity to the concept of user experience. Seminar on Demarcating User Experience. (2011), 1--12.Google Scholar
- Wei-Tek Tsai et al. 2014. Cloud-based software crowdsourcing. IEEE Internet Computing. 18, 3 (2014), 78--83. DOI:https://doi.org/10.1109/MIC.2014.46.Google ScholarCross Ref
- Ye Yang et al. 2016. Who Should Take This Task?: Dynamic Decision Support for Crowd Workers. Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 5777, (2016), 8:1--8:10. DOI:https://doi.org/10.1145/2961111.2962594.Google ScholarDigital Library
- Yunrim Park and Carlos Jensen 2009. Beyond Pretty Pictures: Examining the Benefits of Code Visualization for Open Source Newcomers. (2009), 3--10.Google Scholar
Index Terms
- Long-term user experience in software crowdsourcing platform
Recommendations
The experience of interactive storytelling: comparing “fahrenheit” with “façade”
ICEC'11: Proceedings of the 10th international conference on Entertainment ComputingAt the intersection of multimedia, artificial intelligence, and gaming technology, new visions of future entertainment media arise that approximate the “Holodeck” ® idea of interactive storytelling. We report exploratory experiments on the user ...
The role of platform moderators in software crowdsourcing projects
CHASE '19: Proceedings of the 12th International Workshop on Cooperative and Human Aspects of Software EngineeringCrowdsourcing divides a task into small pieces that are carried out by the crowd. In Software Engineering, crowdsourcing divides the software development tasks of to be carried out online by the crowd and is simply called Software Crowdsourcing (SW CS). ...
Software crowdsourcing reliability: an empirical study on developers behavior
SWAN 2016: Proceedings of the 2nd International Workshop on Software AnalyticsCrowdsourcing has become an emergent paradigm for software production in recent decades. Its open-call format attracts the participation of hundreds of thousands of developers. To ensure the success of software crowdsourcing, we must accurately measure ...
Comments