Skip to main content
SpringerLink
Log in
Book cover

International Conference on Collaboration and Technology

CRIWG 2018: Collaboration and Technology pp 133–154Cite as

Crowdsourcing and Massively Collaborative Science: A Systematic Literature Review and Mapping Study

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11001))

Abstract

Current times are denoting unprecedented indicators of scientific data production, and the involvement of the wider public (the crowd) on research has attracted increasing attention. Drawing on review of extant literature, this paper outlines some ways in which crowdsourcing and mass collaboration can leverage the design of intelligent systems to keep pace with the rapid transformation of scientific work. A systematic literature review was performed following the guidelines of evidence-based software engineering and a total of 148 papers were identified as primary after querying digital libraries. From our review, a lack of methodological frameworks and algorithms for enhancing interactive intelligent systems by combining machine and crowd intelligence is clearly manifested and we will need more technical support in the future. We lay out a vision for a cyberinfrastructure that comprises crowd behavior, task features, platform facilities, and integration of human inputs into AI systems.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    https://www.mturk.com/.

  2. 2.

    In Appendix A will be found a list of all publications included in the final review.

  3. 3.

    https://experiment.com/.

  4. 4.

    https://www.rottentomatoes.com/.

References

  1. Jirotka, M., Lee, C.P., Olson, G.M.: Supporting scientific collaboration: methods, tools and concepts. Comput. Support. Coop. Work 22(4–6), 667–715 (2013)

    Article  Google Scholar 

  2. Gil, Y., Greaves, M., Hendler, J., Hirsh, H.: Amplify scientific discovery with artificial intelligence. Science 346(6206), 171–172 (2014)

    Article  Google Scholar 

  3. Quinn, A.J., Bederson, B.B.: Human computation: a survey and taxonomy of a growing field. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 1403–1412 (2011)

    Google Scholar 

  4. Klippel, A., Sparks, K., Wallgrün, J.O.: Pitfalls and potentials of crowd science: a meta-analysis of contextual influences. In: ISPRS Annals (2015)

    Google Scholar 

  5. Buecheler, T., Sieg, J.H., Füchslin, R.M., Pfeifer, R.: Crowdsourcing, open innovation and collective intelligence in the scientific method – a research agenda and operational framework. In: ALIFE, pp. 679–686 (2010)

    Google Scholar 

  6. Ranard, B.L., et al.: Crowdsourcing – harnessing the masses to advance health and medicine, a systematic review. J. Gen. Intern. Med. 29(1), 187–203 (2014)

    Article  Google Scholar 

  7. Law, E., Gajos, K.Z., Wiggins, A., Gray, M.L., Williams, A.C.: Crowdsourcing as a tool for research: implications of uncertainty. In: ACM CSCW, pp. 1544–1561 (2017)

    Google Scholar 

  8. Good, B.M., Su, A.I.: Games with a scientific purpose. Genome Biol. 12(12), 135 (2011)

    Article  Google Scholar 

  9. Quinn, A.J., Bederson, B.B., Yeh, T., Lin, J.: Crowdflow: integrating machine learning with mechanical turk for speed-cost-quality flexibility. Technical report HCIL-2010-09, University of Maryland (2010)

    Google Scholar 

  10. Absalom, R., Luczak-Rosch, M., Hartmann, D., Plaat, A.: Crowd-sourcing fuzzy and faceted classification for concept search. arXiv preprint arXiv:1406.7749 (2014)

  11. Barowy, D.W., Curtsinger, C., Berger, E.D., McGregor, A.: Automan: a platform for integrating human-based and digital computation. ACM Sigplan Not. 47(10), 639–654 (2012)

    Article  Google Scholar 

  12. André, P., Zhang, H., Kim, J., Chilton, L., Dow, S.P., Miller, R.C.: Community clustering: leveraging an academic crowd to form coherent conference sessions. In: Proceedings of the First AAAI Conference on Human Computation and Crowdsourcing (2013)

    Google Scholar 

  13. Nov, O., Arazy, O., Anderson, D.: Scientists@home: what drives the quantity and quality of online citizen science participation. PLoS ONE 9(4), e90375 (2014)

    Article  Google Scholar 

  14. Chandler, J., Paolacci, G., Mueller, P.: Risks and rewards of crowdsourcing marketplaces. In: Michelucci, P. (ed.) Handbook of Human Computation, pp. 377–392. Springer, New York (2013). https://doi.org/10.1007/978-1-4614-8806-4_30

    Chapter  Google Scholar 

  15. Good, B.M., Nanis, M., Wu, C., Su, A.I.: Microtask crowdsourcing for disease mention annotation in PubMed abstracts. In: Pacific Symposium on Biocomputing, pp. 282–293 (2014)

    Google Scholar 

  16. Brown, A.W., Allison, D.B.: Using crowdsourcing to evaluate published scientific literature: methods and example. PLoS ONE 9(7), e100647 (2014)

    Article  Google Scholar 

  17. Tomlinson, B., et al.: Massively distributed authorship of academic papers. In: CHI EA, pp. 11–20 (2012)

    Google Scholar 

  18. Lease, M.: On quality control and machine learning in crowdsourcing. In: Proceedings of the 11th AAAI Conference on Human Computation, pp. 97–102 (2011)

    Google Scholar 

  19. Schneider, D., Moraes, K., de Souza, J.M., Esteves, M.G.P.: CSCWD: five characters in search of crowds. In: CSCWD, pp. 634–641 (2012)

    Google Scholar 

  20. Zhao, Y., Zhu, Q.: Evaluation on crowdsourcing research: current status and future direction. Inf. Syst. Front. 16(3), 417–434 (2014)

    Article  Google Scholar 

  21. Hosseini, M., Shahri, A., Phalp, K., Taylor, J., Ali, R.: Crowdsourcing: a taxonomy and systematic mapping study. Comput. Sci. Rev. 17, 43–69 (2015)

    Article  MathSciNet  Google Scholar 

  22. Kamar, E.: Directions in hybrid intelligence: complementing AI systems with human intelligence. In: IJCAI, pp. 4070–4073 (2016)

    Google Scholar 

  23. Kittur, A., et al.: The future of crowd work. In: ACM CSCW, pp. 1301–1318 (2013)

    Google Scholar 

  24. Horvitz, E.: Principles of mixed-initiative user interfaces. In: CHI, pp. 159–166 (1999)

    Google Scholar 

  25. Dong, Z., Lu, J., Ling, T.W., Fan, J., Chen, Y.: Using hybrid algorithmic-crowdsourcing methods for academic knowledge acquisition. Clust. Comput. 20(4), 3629–3641 (2017)

    Article  Google Scholar 

  26. Chau, D.H., Kittur, A., Hong, J.I., Faloutsos, C.: Apolo: making sense of large network data by combining rich user interaction and machine learning. In: CHI, pp. 167–176 (2011)

    Google Scholar 

  27. Kondreddi, S.K., Triantafillou, P., Weikum, G.: Combining information extraction and human computing for crowdsourced knowledge acquisition. In: ICDE, pp. 988–999 (2014)

    Google Scholar 

  28. Petersen, K., Feldt, R., Mujtaba, S., Mattsson, M.: Systematic mapping studies in software engineering. EASE 8, 68–77 (2008)

    Google Scholar 

  29. Cruzes, D.S., Dybå, T.: Research synthesis in software engineering: a tertiary study. Inf. Softw. Technol. 53(5), 440–455 (2011)

    Article  Google Scholar 

  30. Kitchenham, B.: Procedures for performing systematic reviews. Keele University, Keele, UK, vol. 33, pp. 1–26 (2004)

    Google Scholar 

  31. Johansen, R.: Groupware: computer support for business teams. The Free Press (1988)

    Google Scholar 

  32. Lasecki, W.S., Murray, K.I., White, S., Miller, R.C., Bigham, J.P.: Real-time crowd control of existing interfaces. In: UIST, pp. 23–32 (2011)

    Google Scholar 

  33. Ponciano, L., Brasileiro, F., Simpson, R., Smith, A.: Volunteers’ engagement in human computation for astronomy projects. CiSE 16(6), 52–59 (2014)

    Google Scholar 

  34. Uchoa, A.P., Esteves, M.G.P., de Souza, J.M.: Mix4Crowds – toward a framework to design crowd collaboration with science. In: CSCWD, pp. 61–66 (2013)

    Google Scholar 

  35. Wiggins, A., Crowston, K.: Distributed scientific collaboration: research opportunities in citizen science. In: ACM CSCW 2010 Workshop on Changing Dynamics of Scientific Collaboration (2010)

    Google Scholar 

  36. Ellis, C.A., Gibbs, S.J., Rein, G.: Groupware: some issues and experiences. Commun. ACM 34(1), 39–58 (1991)

    Article  Google Scholar 

  37. Estellés-Arolas, E., González-Ladrón-de-Guevara, F.: Towards an integrated crowdsourcing definition. J. Inf. Sci. 38(2), 189–200 (2012)

    Article  Google Scholar 

  38. Gadiraju, U., Demartini, G., Kawase, R., Dietze, S.: Human beyond the machine: challenges and opportunities of microtask crowdsourcing. IEEE Intell. Syst. 30(4), 81–85 (2015)

    Article  Google Scholar 

  39. Sakamoto, Y., Tanaka, Y., Yu, L., Nickerson, J.V.: The crowdsourcing design space. In: Schmorrow, Dylan D., Fidopiastis, Cali M. (eds.) FAC 2011. LNCS (LNAI), vol. 6780, pp. 346–355. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21852-1_41

    Chapter  Google Scholar 

  40. Huang, S.W., Fu, W.T.: Don’t hide in the crowd!: increasing social transparency between peer workers improves crowdsourcing outcomes. In: CHI, pp. 621–630 (2013)

    Google Scholar 

  41. Deci, E.L., Ryan, R.M.: The general causality orientations scale: self-determination in personality. J. Res. Pers. 19(2), 109–134 (1985)

    Article  Google Scholar 

  42. Freitag, A., Pfeffer, M.J.: Process, not product: investigating recommendations for improving citizen science “success”. PLoS ONE 8(5), e64079 (2013)

    Article  Google Scholar 

Download references

Acknowledgements

This work is financed by the ERDF – European Regional Development Fund through the Operational Programme for Competitiveness and Internationalisation - COMPETE 2020 Programme within project «POCI-01-0145-FEDER-006961», and by National Funds through the Portuguese funding agency, FCT - Fundação para a Ciência e a Tecnologia as part of project «UID/EEA/50014/2013».

Author information

Authors and Affiliations

  1. University of Trás-os-Montes e Alto Douro, UTAD, Vila Real, Portugal

    António Correia, Benjamim Fonseca & Hugo Paredes

  2. INESC TEC, Porto, Portugal

    António Correia, Benjamim Fonseca & Hugo Paredes

  3. Tércio Pacitti Institute of Computer Applications and Research (NCE), Federal University of Rio de Janeiro, Rio de Janeiro, Brazil

    Daniel Schneider

Authors
  1. António Correia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Schneider
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Benjamim Fonseca
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hugo Paredes
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to António Correia .

Editor information

Editors and Affiliations

  1. Department of Computer Science, Faculty of Science and Technology, Universidade NOVA de Lisboa, Caparica, Portugal

    Armanda Rodrigues

  2. Universidade de Trás-os-Montes e Alto Douro, Vila Real, Portugal

    Benjamim Fonseca

  3. Department of Computer Science, Faculty of Science and Technology, Universidade NOVA de Lisboa, Caparica, Portugal

    Nuno Preguiça

Appendix A

Appendix A

See Table 10.

Table 10. Publications included in the final review (adapted from Cruzes & Dybå [29])
Full size table

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Correia, A., Schneider, D., Fonseca, B., Paredes, H. (2018). Crowdsourcing and Massively Collaborative Science: A Systematic Literature Review and Mapping Study. In: Rodrigues, A., Fonseca, B., Preguiça, N. (eds) Collaboration and Technology. CRIWG 2018. Lecture Notes in Computer Science(), vol 11001. Springer, Cham. https://doi.org/10.1007/978-3-319-99504-5_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-99504-5_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-99503-8

  • Online ISBN: 978-3-319-99504-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation