Yunchao Liu
Bobby Bodenheimer
ABSTRACT
In building a new drug design mode for the popular citizen scientist game Foldit, we focus on creating an easy-to-use and intuitive interface to confer complex scientific concepts to citizen scientist players. We hypothesize that to be efficient in the hands of citizen scientists such an interface will look different from well-established drug-design software used by experts. We used the relaxed think-aloud method to compare citizen and expert scientists working with our prototype interface for Foldit Drug Design Mode (FDDM). First, we tested if the two groups are providing different feedback when it comes to the usability of the prototype interface. Second, we investigated how the difference between the two groups might inform a new game design. As expected, the results confirm that experienced scientists differ from citizen scientists in engaging their background knowledge when interacting with the game. We then provided a prioritization list of background knowledge employed by the expert scientists to derive design suggestions for FDDM.
Supplemental Material
Available for Download
pdf
a20-liu-supplement.pdf (1.6 MB)
Supplement to "Foldit Drug Design Game Usability Study: Comparison of Citizen and Expert Scientists" by Liu et al., Motion, Interaction and Games (MIG '20).
- Rick Bonney, Jennifer L. Shirk, Tina B. Phillips, Andrea Wiggins, Heidi L. Ballard, Abraham J. Miller-Rushing, and Julia K. Parrish. 2014. Next Steps for Citizen Science. Science 343, 6178 (2014), 1436–1437. https://doi.org/10.1126/science.1251554 arXiv:https://science.sciencemag.org/content/343/6178/1436.full.pdfGoogle Scholar
- Anne Bowser, Derek Hansen, Yurong He, Carol Boston, Matthew Reid, Logan Gunnell, and Jennifer Preece. 2013. Using Gamification to Inspire New Citizen Science Volunteers. In Proceedings of the First International Conference on Gameful Design, Research, and Applications(Toronto, Ontario, Canada) (Gamification ’13). Association for Computing Machinery, New York, NY, USA, 18–25. https://doi.org/10.1145/2583008.2583011Google ScholarDigital Library
- Seth Cooper, Firas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popović, 2010. Predicting protein structures with a multiplayer online game. Nature 466, 7307 (2010), 756–760.Google ScholarCross Ref
- K. Crowston and N. R. Prestopnik. 2013. Motivation and Data Quality in a Citizen Science Game: A Design Science Evaluation. In 2013 46th Hawaii International Conference on System Sciences. 450–459. https://doi.org/10.1109/HICSS.2013.413Google ScholarDigital Library
- J A DiMasi, L Feldman, A Seckler, and A Wilson. 2010. Trends in Risks Associated With New Drug Development: Success Rates for Investigational Drugs. Clinical Pharmacology & Therapeutics 87, 3 (2010), 272–277. https://doi.org/10.1038/clpt.2009.295 arXiv:https://ascpt.onlinelibrary.wiley.com/doi/pdf/10.1038/clpt.2009.295Google ScholarCross Ref
- Christopher B Eiben, Justin B Siegel, Jacob B Bale, Seth Cooper, Firas Khatib, Betty W Shen, Barry L Stoddard, Zoran Popovic, and David Baker. 2012. Increased Diels-Alderase activity through backbone remodeling guided by Foldit players. Nature biotechnology 30, 2 (2012), 190–192.Google Scholar
- K Anders Ericsson and Herbert A Simon. 1984. Protocol analysis: Verbal reports as data.the MIT Press.Google Scholar
- Robert Heck, Oana Vuculescu, Jens Jakob Sørensen, Jonathan Zoller, Morten G. Andreasen, Mark G. Bason, Poul Ejlertsen, Ottó Elíasson, Pinja Haikka, Jens S. Laustsen, Lærke L. Nielsen, Andrew Mao, Romain Müller, Mario Napolitano, Mads K. Pedersen, Aske R. Thorsen, Carsten Bergenholtz, Tommaso Calarco, Simone Montangero, and Jacob F. Sherson. 2018. Remote optimization of an ultracold atoms experiment by experts and citizen scientists. Proceedings of the National Academy of Sciences 115, 48(2018), E11231–E11237. https://doi.org/10.1073/pnas.1716869115 arXiv:https://www.pnas.org/content/115/48/E11231.full.pdfGoogle ScholarCross Ref
- Scott Horowitz, Brian Koepnick, Raoul Martin, Agnes Tymieniecki, Amanda A Winburn, Seth Cooper, Jeff Flatten, David S Rogawski, Nicole M Koropatkin, Tsinatkeab T Hailu, 2016. Determining crystal structures through crowdsourcing and coursework. Nature communications 7, 1 (2016), 1–11.Google Scholar
- Ioanna Iacovides, Charlene Jennett, Cassandra Cornish-Trestrail, and Anna L. Cox. 2013. Do Games Attract or Sustain Engagement in Citizen Science? A Study of Volunteer Motivations. In CHI ’13 Extended Abstracts on Human Factors in Computing Systems (Paris, France) (CHI EA ’13). Association for Computing Machinery, New York, NY, USA, 1101–1106. https://doi.org/10.1145/2468356.2468553Google ScholarDigital Library
- Monique W.M. Jaspers. 2009. A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. International Journal of Medical Informatics 78, 5(2009), 340 – 353. https://doi.org/10.1016/j.ijmedinf.2008.10.002Google ScholarCross Ref
- Monique W.M. Jaspers, Thiemo Steen, Cor van den Bos, and Maud Geenen. 2004. The think aloud method: a guide to user interface design. International Journal of Medical Informatics 73, 11(2004), 781 – 795. https://doi.org/10.1016/j.ijmedinf.2004.08.003Google ScholarCross Ref
- Victor Jupp. 2006. The Sage dictionary of social research methods. Sage.Google Scholar
- Firas Khatib, Seth Cooper, Michael D. Tyka, Kefan Xu, Ilya Makedon, Zoran Popović, David Baker, and Foldit Players. 2011a. Algorithm discovery by protein folding game players. Proceedings of the National Academy of Sciences 108, 47(2011), 18949–18953. https://doi.org/10.1073/pnas.1115898108Google ScholarCross Ref
- Firas Khatib, Frank DiMaio, Seth Cooper, Maciej Kazmierczyk, Miroslaw Gilski, Szymon Krzywda, Helena Zabranska, Iva Pichova, James Thompson, Zoran Popović, 2011b. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature structural & molecular biology 18, 10 (2011), 1175–1177.Google Scholar
- Brian Koepnick, Jeff Flatten, Tamir Husain, Alex Ford, Daniel-Adriano Silva, Matthew J Bick, Aaron Bauer, Gaohua Liu, Yojiro Ishida, Alexander Boykov, 2019. De novo protein design by citizen scientists. Nature 570, 7761 (2019), 390–394.Google Scholar
- E. Korpela, D. Werthimer, D. Anderson, J. Cobb, and M. Leboisky. 2001. SETI@home-massively distributed computing for SETI. Computing in Science Engineering 3, 1 (Jan 2001), 78–83. https://doi.org/10.1109/5992.895191Google ScholarDigital Library
- Andrew Leaver-Fay, Michael Tyka, Steven M. Lewis, Oliver F. Lange, James Thompson, Ron Jacak, Kristian W. Kaufman, P. Douglas Renfrew, Colin A. Smith, Will Sheffler, Ian W. Davis, Seth Cooper, Adrien Treuille, Daniel J. Mandell, Florian Richter, Yih-En Andrew Ban, Sarel J. Fleishman, Jacob E. Corn, David E. Kim, Sergey Lyskov, Monica Berrondo, Stuart Mentzer, Zoran Popović, James J. Havranek, John Karanicolas, Rhiju Das, Jens Meiler, Tanja Kortemme, Jeffrey J. Gray, Brian Kuhlman, David Baker, and Philip Bradley. 2011. Chapter nineteen - Rosetta3: An Object-Oriented Software Suite for the Simulation and Design of Macromolecules. In Computer Methods, Part C, Michael L. Johnson and Ludwig Brand (Eds.). Methods in Enzymology, Vol. 487. Academic Press, 545 – 574. https://doi.org/10.1016/B978-0-12-381270-4.00019-6Google Scholar
- Jens Meiler and David Baker. 2006. ROSETTALIGAND: Protein–small molecule docking with full side-chain flexibility. Proteins: Structure, Function, and Bioinformatics 65, 3(2006), 538–548. https://doi.org/10.1002/prot.21086Google ScholarCross Ref
- Josh Aaron Miller, Vivian Lee, Seth Cooper, and Magy Seif El-Nasr. 2019. Large-Scale Analysis of Visualization Options in a Citizen Science Game. In Extended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts (Barcelona, Spain) (CHI PLAY ’19 Extended Abstracts). Association for Computing Machinery, New York, NY, USA, 535–542. https://doi.org/10.1145/3341215.3356274Google ScholarDigital Library
- Garrett M. Morris, Ruth Huey, William Lindstrom, Michel F. Sanner, Richard K. Belew, David S. Goodsell, and Arthur J. Olson. 2009. AutoDock4 and AutoDockTools4: Automated docking with selective receptor flexibility. Journal of Computational Chemistry 30, 16 (2009), 2785–2791. https://doi.org/10.1002/jcc.21256Google ScholarCross Ref
- Michael Shirts and Vijay S. Pande. 2000. Screen Savers of the World Unite!Science 290, 5498 (2000), 1903–1904. https://doi.org/10.1126/science.290.5498.1903Google Scholar
- Jonathan Silvertown. 2009. A new dawn for citizen science. Trends in Ecology & Evolution 24, 9 (2009), 467 – 471. https://doi.org/10.1016/j.tree.2009.03.017Google ScholarCross Ref
- Kristin Siu, Alexander Zook, and Mark O. Riedl. 2017. A Framework for Exploring and Evaluating Mechanics in Human Computation Games. In Proceedings of the 12th International Conference on the Foundations of Digital Games (Hyannis, Massachusetts) (FDG ’17). Association for Computing Machinery, New Yorkk NYk USA, Article 38, 4 pages. https://doi.org/10.1145/3102071.3106344Google ScholarDigital Library
- Joanna Stanczyk, Caroline Ospelt, and Steffen Gay. 2008. Is there a future for small molecule drugs in the treatment of rheumatic diseases?Current opinion in rheumatology 20, 3 (2008), 257–262.Google Scholar
- Brian L. Sullivan, Christopher L. Wood, Marshall J. Iliff, Rick E. Bonney, Daniel Fink, and Steve Kelling. 2009. eBird: A citizen-based bird observation network in the biological sciences. Biological Conservation 142, 10 (2009), 2282 – 2292. https://doi.org/10.1016/j.biocon.2009.05.006Google ScholarCross Ref
- Marcel L. Verdonk, Jason C. Cole, Michael J. Hartshorn, Christopher W. Murray, and Richard D. Taylor. 2003. Improved protein–ligand docking using GOLD. Proteins: Structure, Function, and Bioinformatics 52, 4(2003), 609–623. https://doi.org/10.1002/prot.10465 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/prot.10465Google ScholarCross Ref
- L. von Ahn. 2006. Games with a purpose. Computer 39, 6 (June 2006), 92–94. https://doi.org/10.1109/MC.2006.196Google ScholarDigital Library
- Luis Von Ahn and Laura Dabbish. 2008. Designing games with a purpose. Commun. ACM 51, 8 (2008), 58–67.Google ScholarDigital Library
Recommendations
Is Virtual Citizen Science A Game?
The use of game elements within virtual citizen science is increasingly common, promising to bring increased user activity, motivation, and engagement to large-scale scientific projects. However, there is an ongoing debate about whether or not gamifying ...
A new framework of usability evaluation for massively multi-player online game: case study of "World of warcraft" game
HCI'07: Proceedings of the 12th international conference on Human-computer interaction: applications and servicesAccording to traditional marketing wisdom, customers' preferences are a core issue in designing successful products, and the design process in game is crucial for guaranteeing gamers' satisfaction. This research aimed to explore critical factors for the ...
A Comparative Usability Analysis of Handheld Game Consoles
Design, User Experience, and Usability. Design for Contemporary Interactive EnvironmentsAbstractA handheld game console is a small and portable video game console with a built-in screen, game controls and speakers. Playing a handheld game console involves multiple operations and buttons. The usability of game consoles is influenced by users’ ...
Comments