Valentina Grigoreanu
Kyle Rector
Irwin Kwan
Skip Abstract Section
Abstract
Despite decades of research into how professional programmers debug, only recently has work emerged about how end-user programmers attempt to debug programs. Without this knowledge, we cannot build tools to adequately support their needs. This article reports the results of a detailed qualitative empirical study of end-user programmers' sensemaking about a spreadsheet's correctness. Using our study's data, we derived a sensemaking model for end-user debugging and categorized participants' activities and verbalizations according to this model, allowing us to investigate how participants went about debugging. Among the results are identification of the prevalence of information foraging during end-user debugging, two successful strategies for traversing the sensemaking model, potential ties to gender differences in the literature, sensemaking sequences leading to debugging progress, and sequences tied with troublesome points in the debugging process. The results also reveal new implications for the design of spreadsheet tools to support end-user programmers' sensemaking during debugging.
- Abraham, R. and Erwig, M. 2007. A type system based on end-user vocabulary. In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 215--222. Google Scholar
Digital Library
- Ayalew Y. and Mittermeir R. 2003. Spreadsheet debugging. In Proceedings of the European Spreadsheet Risks Interest Group, Dublin, Ireland, July 24--25.Google Scholar
- Bandura, A. 1986. Social Foundations of Thought and Action. Prentice Hall.Google Scholar
- Bates, M. 1990. Where should the person stop and the information search interface start? Inf. Process. Manag. 26, 5, 575--591. Google ScholarDigital Library
- Beckwith, L., Burnett, M., Wiedenbeck, S., Cook, C., Sorte, S., and Hastings, M. 2005. Effectiveness of end-user debugging software features: Are there gender issues? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 869--878. Google ScholarDigital Library
- Beckwith, L., Kissinger, C., Burnett, M., Wiedenbeck, S., Lawrance, J., Blackwell, A., and Cook, C. 2006. Tinkering and gender in end-user programmers' debugging. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 231--240. Google ScholarDigital Library
- Beckwith, L., Inman, D., Rector, K., and Burnett, M. 2007. On to the real world: Gender and self-efficacy in Excel. In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 119--126. Google ScholarDigital Library
- Boehm, B. and Basili, V. 2001. Software defect reduction top 10 list. Comput. 34, 1, 135--137. Google ScholarDigital Library
- Burnett, M., Cook, C., Pendse, O., Rothermel, G., Summet, J., and Wallace, C. 2003. End-User software engineering with assertions in the spreadsheet paradigm. In Proceedings of the International Conference on Software Engineering. IEEE, 93--103. Google ScholarDigital Library
- Burnett M., Cook C., and Rothermel G. 2004. End-User software engineering. Comm. ACM 47, 9, 53--58. Google ScholarDigital Library
- Busch, T. 1995. Gender differences in self-efficacy and attitudes toward computers. J. Educ. Comput. Res. 12, 2, 147--158.Google ScholarCross Ref
- Butler, R. 2000. Is this spreadsheet a tax evader? How HM customs and excise test spreadsheet applications. In Proceedings of the 33rd Annual Hawaii International Conference on System Sciences. Google ScholarDigital Library
- Byrnes, J. P., Miller, D. C., and Schafer, W. D. 1999. Gender differences in risk taking: A meta-analysis. Psychol. Bull. 125, 367--383.Google ScholarCross Ref
- Dervin, B. 1984. A theoretic perspective and research approach for generating research helpful to communication practice. Public Relat. Res. Educ. 1, 1, 30--45.Google Scholar
- Dervin, B., Foreman-Wernet, L., and Launterbach, E. Eds. 2003. Sense-Making Methodology Reader: Selected Writings of Brenda Dervin. Hampton Press, Cresskill, NJ, 215--231.Google Scholar
- EUSPRIG 2009. Spreadsheet mistakes news stories, european spreadsheet risks interests group site. http://www.eusprig.org/stories.htm.Google Scholar
- Fern, X., Komireddy, C., Grigoreanu, V., and Burnett, M. 2009. Mining problem-solving strategies from HCI data. ACM Trans. Comput.-Hum. Interact. To appear. Google ScholarDigital Library
- Finucane, M., Slovic, P., Merz, C-K., Flynn, J., and Satterfield, T. 2000. Gender, race and perceived risk: The white male effect. Health, Risk Soc. 2, 2, 159--172.Google ScholarCross Ref
- Fisher II, M. and Rothermel, G. 2005. The EUSES spreadsheet corpus: A shared resource for supporting experimentation with spreadsheet dependability mechanism. In Proceedings of the 1st Workshop on End-User Software Engineering. 47--51. Google ScholarDigital Library
- Furnas, G. and Russell, D. 2005. Making sense of sensemaking. Extended Abstracts on Human Factors in Computing Systems (ACM CHI'05). ACM, 2115--2116. Google ScholarDigital Library
- Gallagher, A., De Lisi, R., Holst, P., McGillicuddy-De Lisi, A., Morely, M., and Cahalan, C. 2000. Gender differences in advanced mathematical problem solving. J. Experim. Child Psychol. 75, 3, 165--190.Google ScholarCross Ref
- Grigoreanu, V., Beckwith, L., Fern, X., Yang, S., Komireddy, C., Narayanan, V., Cook, C., and Burnett, M. 2006. Gender differences in end-user debugging, revisited: What the miners found. In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 19--26. Google ScholarDigital Library
- Grigoreanu, V., Cao, J., Kulesza, T., Bogart, C., Rector, K., Burnett, M., and Wiedenbeck, S. 2008. Can feature design reduce the gender gap in end-user software development environments? In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 149--156. Google ScholarDigital Library
- Grigoreanu, V., Brundage, J., Bahna, E., Burnett, M., ElRif, P., and Snover, J. 2009. Males' and females' script debugging strategies. In Proceedings of the 2nd International Symposium on End-User Development. Springer, 205--224. Google ScholarDigital Library
- Hartzel, K. 2003. How self-efficacy and gender issues affect software adoption and use. Comm. ACM 46, 9, 167--171. Google ScholarDigital Library
- Ioannidou, A., Repenning, A., and Webb, D. 2008. Using scalable game design to promote 3D fluency: Assessing the AgentCubes incremental 3D end-user development framework. In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 47--54. Google ScholarDigital Library
- Jeffries, R. A. 1982. Comparison of debugging behavior of novice and expert programmers. AERA Annual Meeting, Department of Psychology, Carnegie Mellon University, Pittsburgh, PA.Google Scholar
- Jin, J. and Dabbish, L. 2009. Self-Interruption on the computer: A typology of discretionary task interleaving. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1799--1808. Google ScholarDigital Library
- Kelleher, C. and Pausch, R. 2005. Lowering the barriers to programming: A survey of programming environments and languages for novice programmers. ACM Comput. Surv. 37, 2. Google ScholarDigital Library
- Kelleher, C., Pausch, R., and Kiesler, S. 2007. Storytelling Alice motivates middle school girls to learn computer programming. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1455--1464. Google ScholarDigital Library
- Kissinger, C., Burnett, M., Stumpf, S., Subrahmaniyan, N., Beckwith, L., Yang, S., and Rosson, M. 2006. Supporting end-user debugging: What do users want to know? In Proceedings of the Working Conference on Advanced Visual Interfaces, ACM, 135--142. Google ScholarDigital Library
- Klein, G., Moon, B., and Hoffman, R. R. 2006. Making sense of sensemaking 1: Alternative perspectives. IEEE Intell. Syst. 21, 4, 70--73. Google ScholarDigital Library
- Ko, A. and Myers, B. 2004. Designing the whyline: A debugging interface for asking questions about program behavior. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 151--158. Google ScholarDigital Library
- Ko, A., Myers, B., and Aung, Htet Htet 2004. Six learning barriers in end-user programming systems. In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 199--206. Google ScholarDigital Library
- Lawrance, J., Bellamy, R., Burnett, M., and Rector, K. 2008. Using information scent to model the dynamic foraging behavior of programmers in maintenance tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1323--1332. Google ScholarDigital Library
- Leedom, D. K. 2001. Final report: Sensemaking symposium. http://www.dodccrp.org/events/2001_sensemaking_symposium/docs/FinalReport/Sensemaking_Final_Report.htm.Google Scholar
- Littman, D. C., Pinto, J., Letovsky, S., and Soloway, E. 1986. Mental models and software maintenance. In Proceedings of the 1st Workshop on Empirical Studies of Programmers on Empirical Studies of Programmers. Ablex Publishing Corporation, 80--98. Google ScholarDigital Library
- Meyers-Levy, J. 1989. Gender differences in information processing: A selectivity interpretation. In Cognitive and Affective Responses to Advertising. R. Cafferata and A. Tybout, Eds., Lexington Books, MA.Google Scholar
- Myers, B., Ko, A., and Burnett, M. 2006. Invited research overview: End-User programming. In Extended Abtracts of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 75--80. Google ScholarDigital Library
- Nanja, N. and Cook, C. 1987. An analysis of the on-line debugging process. In Proceedings of the 2nd Workshop on Empirical Studies of Programmers. Ablex Publishing Corporation, 172--184. Google ScholarDigital Library
- Nardi, B. A. 1993. A Small Matter of Programming: Perspectives on End-User Computing. MIT Press, MA. Google ScholarDigital Library
- O'Donnell, E. and Johnson, E. 2001. The effects of auditor gender and task complexity on information processing efficiency. Int. J. Audit. 5, 91--105.Google ScholarCross Ref
- Pane, J. and Myers, B. 1996, Usability issues in the design of novice programming systems, Tech. rep. CMU-CS-96-132, School of Computer Science, Carnegie Mellon University.Google Scholar
- Panko, R. 1998. What we know about spreadsheet errors. J. End User Comput. 10, 2, 15--21. Google ScholarDigital Library
- Panko, R. and Orday, N. 2005. Sarbanes-Oxley: What about all the spreadsheets? In Proceedings of the European Spreadsheet Research Information Group.Google Scholar
- Phalgune, A., Kissinger, C., Burnett, M., Cook, C., Beckwith, L., and Ruthruff, J. 2005. Garbage in, garbage out? An empirical look at oracle mistakes by end-user programmers. In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 45--52. Google ScholarDigital Library
- Pirolli, P. and Card, S. 1999. Information foraging. Psychol. Rev. 106, 4, 643--675.Google ScholarCross Ref
- Pirolli, P. and Card, S. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of the International Conference on Intelligence Analysis.Google Scholar
- Powell, M. and Ansic, D. 1997. Gender differences in risk behaviour in financial decision-making: An experimental analysis. J. Econom. Psychol. 18, 6, 605--628.Google ScholarCross Ref
- Prabhakararao, S., Cook, D., Ruthruff, J., Creswick, E., Main, M., Durham, M., and Burnett, M. 2003. Strategies and behaviors of end-user programmers with interactive fault localization. In Proceedings of Symposium on Human-Centric Computing Languages and Environments. IEEE, 15--22. Google ScholarDigital Library
- Rode, J. and Rosson, M. 2003. Programming at runtime: Requirements and paradigms for nonprogrammer web application development, In Proceedings of Symposium on Human-Centric Computing Languages and Environments. IEEE. 23--30. Google ScholarDigital Library
- Romero, P., du Boulay, B., Cox, R., Lutz, R., and Bryant, S. Debugging strategies and tactics in a multi-representation software environment. Int. J. Hum.- Comput. Studies 61, 992--1009. Google ScholarDigital Library
- Rosson, M., Sinha, H., Bhattacharya, M., and Zhao, D. 2007. Design planning in end-user web development. In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 189--196. Google ScholarDigital Library
- Russell, D. M., Stefik, M. J., Pirolli, P., and Card, S. K. 1993. The cost structure of sensemaking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 269--276. Google ScholarDigital Library
- Scaffidi, C., Shaw, M., and Myers, B. 2005. Estimating the numbers of end users and end user programmers, In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 207--214. Google ScholarDigital Library
- Shrager, J. and Klahr, D. 1986. Instructionless learning about a complex device: The paradigm and observations. Int. J. Man-Mach. Studies 25, 153--189. Google ScholarDigital Library
- Stefik, M., Baldonado, M., Bobrow, D., Card, S., Everett, J., Lavendel, G., Marimont, D., Newman, P., Russell, D., and Smoliar, S. 2002. The knowledge sharing challenge: The sensemaking. White paper. http://www2.parc.com/istl/groups/hdi/papers/sensemaking-whitepaper.pdf.Google Scholar
- Subrahmaniyan, N., Beckwith, L., Grigoreanu, V., Burnett, M., Wiedenbeck, S., Narayanan, V., Bucht, K., Drummond, R., and Fern, X. 2008. Testing vs. code inspection vs. what else? Male and female end users' debugging strategies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 617--626. Google ScholarDigital Library
- Torkzadeh, G. and Koufteros, X. 1994. Factorial validity of a computer self-efficacy scale and the impact of computer training. Educ. Psychol. Measur. 54, 3, 813--821.Google ScholarCross Ref
- Wagner, E. and Lieberman, H. 2004. Supporting user hypotheses in problem diagnosis on the web and elsewhere. In Proceedings of the International Conference on Intelligent User Interfaces, ACM, 30--37. Google ScholarDigital Library
- Wilcox, E., Atwood, J., Burnett, M., Cadiz, J., and Cook, C. 1997. Does continuous visual feedback aid debugging in direct-manipulation programming systems? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 258--265. Google ScholarDigital Library
Index Terms
- End-user debugging strategies: A sensemaking perspective
Recommendations
Testing vs. code inspection vs. what else?: male and female end users' debugging strategies
CHI '08: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsLittle is known about the strategies end-user programmers use in debugging their programs, and even less is known about gender differences that may exist in these strategies. Without this type of information, designers of end-user programming systems ...
Tinkering and gender in end-user programmers' debugging
CHI '06: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsEarlier research on gender effects with software features intended to help problem-solvers in end-user debugging environments has shown that females are less likely to use unfamiliar software features. This poses a serious problem because these features ...
Debugging support for end user mashup programming
CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsProgramming for the web can be an intimidating task, particularly for non-professional ("end-user") programmers. Mashup programming environments attempt to remedy this by providing support for such programming. It is well known, however, that mashup ...
Comments