Souti Chattopadhyay
ABSTRACT
Computational notebooks - such as Azure, Databricks, and Jupyter - are a popular, interactive paradigm for data scientists to author code, analyze data, and interleave visualizations, all within a single document. Nevertheless, as data scientists incorporate more of their activities into notebooks, they encounter unexpected difficulties, or pain points, that impact their productivity and disrupt their workflow. Through a systematic, mixed-methods study using semi-structured interviews (n=20) and survey (n=156) with data scientists, we catalog nine pain points when working with notebooks. Our findings suggest that data scientists face numerous pain points throughout the entire workflow - from setting up notebooks to deploying to production - across many notebook environments. Our data scientists report essential notebook requirements, such as supporting data exploration and visualization. The results of our study inform and inspire the design of computational notebooks.
- Titus Barik, Robert DeLine, Steven Drucker, and Danyel Fisher. 2016. The bones of the system: A case study of logging and telemetry at Microsoft. In 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C). IEEE, 92--101.Google Scholar
Digital Library
- M. Beth Kery and B. A. Myers. 2017. Exploring exploratory programming. In 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 25--29. DOI: http://dx.doi.org/10.1109/VLHCC.2017.8103446Google ScholarCross Ref
- Jelke Bethlehem. 2010. Selection bias in web surveys. International Statistical Review 78, 2 (2010), 161--188.Google ScholarCross Ref
- Allen Cypher and Daniel Conrad Halbert. 1993. Watch What I Do: Programming by Demonstration. MIT Press.Google ScholarDigital Library
- Robert DeLine, Danyel Fisher, Badrish Chandramouli, Jonathan Goldstein, Michael Barnett, James F Terwilliger, and John Wernsing. 2015. Tempe: Live scripting for live data. In 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 137--141.Google ScholarCross Ref
- Olivier Flückiger, Guido Chari, Jan Jecmen, Ming-Ho Yee, Jakob Hain, and Jan Vitek. 2019. R melts brains: An IR for first-class environments and lazy effectful arguments. In Proceedings of the 15th ACM SIGPLAN International Symposium on Dynamic Languages. ACM, 55--66. DOI: http://dx.doi.org/10.1145/3359619.3359744Google ScholarDigital Library
- Aviral Goel and Jan Vitek. 2019. On the design, implementation, and use of laziness in R. Proceedings of the ACM on Programming Languages 3, OOPSLA (2019), 1--27.Google ScholarDigital Library
- Sumit Gulwani. 2011. Automating string processing in spreadsheets using input-output examples. In ACM SIGPLAN Notices, Vol. 46. ACM, 317--330.Google ScholarDigital Library
- Sumit Gulwani. 2016. Programming by examples. Dependable Software Systems Engineering 45, 137 (2016), 3--15.Google Scholar
- Philip J. Guo. 2012. BURRITO: Wrapping your lab Notebook in computational infrastructure. In 4th USENIX Workshop on the Theory and Practice of Provenance (TaPP). USENIX.Google Scholar
- Philip J. Guo, Sean Kandel, Joseph M. Hellerstein, and Jeffrey Heer. 2011. Proactive wrangling: Mixed-initiative end-user programming of data transformation scripts. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST). ACM, 65--74. DOI: http://dx.doi.org/10.1145/2047196.2047205Google ScholarDigital Library
- Jo Erskine Hannay, Carolyn MacLeod, Janice Singer, Hans Petter Langtangen, Dietmar Pfahl, and Greg Wilson. 2009. How do scientists develop and use scientific software?. In Proceedings of the 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering (SECSE). IEEE, 1--8. DOI: http://dx.doi.org/10.1109/SECSE.2009.5069155Google ScholarDigital Library
- Andrew Head, Fred Hohman, Titus Barik, Steven M. Drucker, and Robert DeLine. 2019. Managing messes in computational notebooks. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI). ACM, Article 270. DOI: http://dx.doi.org/10.1145/3290605.3300500Google ScholarDigital Library
- Eric Horton and Chris Parnin. 2019. DockerizeMe: Automatic inference of environment dependencies for Python code snippets. In Proceedings of the 41st International Conference on Software Engineering (ICSE). IEEE, 328--338.Google ScholarDigital Library
- Jupyter. 2015. Jupyter Notebook UX Survey. (2015). https://github.com/jupyter/surveys/blob/master/surveys/ 2015--12-notebook-ux/analysis/report_dashboard.ipynbGoogle Scholar
- Sean Kandel, Andreas Paepcke, Joseph Hellerstein, and Jeffrey Heer. 2011. Wrangler: Interactive visual specification of data transformation scripts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM, 3363--3372.Google ScholarDigital Library
- S. Kandel, A. Paepcke, J. M. Hellerstein, and J. Heer. 2012. Enterprise data analysis and visualization: An interview study. IEEE Transactions on Visualization and Computer Graphics 18, 12 (Dec 2012), 2917--2926. DOI: http://dx.doi.org/10.1109/TVCG.2012.219Google ScholarDigital Library
- Mary Beth Kery, Bonnie E. John, Patrick O'Flaherty, Amber Horvath, and Brad A. Myers. 2019. Towards effective foraging by data scientists to find past analysis choices. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI). ACM, 92. DOI: http://dx.doi.org/10.1145/3290605.3300322Google ScholarDigital Library
- M. B. Kery and B. A. Myers. 2018. Interactions for untangling messy history in a computational notebook. In 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 147--155. DOI: http://dx.doi.org/10.1109/VLHCC.2018.8506576Google ScholarCross Ref
- Mary Beth Kery, Marissa Radensky, Mahima Arya, Bonnie E. John, and Brad A. Myers. 2018. The story in the notebook: Exploratory data science using a literate programming tool. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI). ACM, Article 174, 11 pages. DOI: http://dx.doi.org/10.1145/3173574.3173748Google ScholarDigital Library
- Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian E Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica B Hamrick, Jason Grout, Sylvain Corlay, and others. 2016. Jupyter Notebooks: A publishing format for reproducible computational workflows. In ELPUB. 87--90.Google Scholar
- Sean Kross and Philip J. Guo. 2019. Practitioners teaching data science in industry and academia: Expectations, workflows, and challenges. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI). ACM, 1--14. DOI: http://dx.doi.org/10.1145/3290605.3300493Google ScholarDigital Library
- Hiroaki Mikami, Daisuke Sakamoto, and Takeo Igarashi. 2017. Micro-versioning tool to support experimentation in exploratory programming. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI). ACM, 6208--6219. DOI: http://dx.doi.org/10.1145/3025453.3025597Google ScholarDigital Library
- Michael Muller, Ingrid Lange, Dakuo Wang, David Piorkowski, Jason Tsay, Q. Vera Liao, Casey Dugan, and Thomas Erickson. 2019. How data science workers work with data: Discovery, capture, curation, design, creation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI). ACM, Article 126, 15 pages. DOI: http://dx.doi.org/10.1145/3290605.3300356Google ScholarDigital Library
- B. A. Myers. 1986. Visual programming, programming by example, and program visualization: A taxonomy. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM, 59--66. DOI: http://dx.doi.org/10.1145/22627.22349Google ScholarDigital Library
- Brad A. Myers. 1998. Scripting graphical applications by demonstration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM, 534--541. DOI: http://dx.doi.org/10.1145/274644.274716Google ScholarDigital Library
- Netflix. 2018. Part 2: Scheduling Notebooks at Netflix. (2018). https://medium.com/netflix-techblog/ scheduling-notebooks-348e6c14cfd6Google Scholar
- Anthony J. Onwuegbuzie and Nancy L. Leech. 2007. Validity and qualitative research: An oxymoron? Quality & Quantity 41, 2 (01 April 2007), 233--249. DOI: http://dx.doi.org/10.1007/s11135-006--9000--3Google ScholarCross Ref
- F. Perez and B. E. Granger. 2007. IPython: A system for interactive scientific computing. Computing in Science Engineering 9, 3 (May 2007), 21--29. DOI: http://dx.doi.org/10.1109/MCSE.2007.53Google ScholarDigital Library
- Fernando Perez and Brian E Granger. 2015. Project Jupyter: Computational narratives as the engine of collaborative data science. Retrieved September 11, 207 (2015), 108.Google Scholar
- F. Perez, B. E. Granger, and J. D. Hunter. 2011. Python: An ecosystem for scientific Computing. Computing in Science Engineering 13, 2 (March 2011), 13--21. DOI: http://dx.doi.org/10.1109/MCSE.2010.119Google ScholarDigital Library
- M. Ragan-Kelley, F. Perez, B. Granger, T. Kluyver, P. Ivanov, J. Frederic, and M. Bussonnier. 2014. The Jupyter/IPython architecture: A unified view of computational research, from interactive exploration to communication and publication. AGU Fall Meeting Abstracts (Dec. 2014), H44D--07.Google Scholar
- B. M. Randles, I. V. Pasquetto, M. S. Golshan, and C. L. Borgman. 2017. Using the Jupyter Notebook as a tool for open science: An empirical study. In 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL). 1--2. DOI: http://dx.doi.org/10.1109/JCDL.2017.7991618Google ScholarCross Ref
- Adam Rule, Aurélien Tabard, and James D. Hollan. 2018. Exploration and explanation in computational notebooks. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI). ACM, Article 32, 12 pages. DOI: http://dx.doi.org/10.1145/3173574.3173606Google ScholarDigital Library
- Johnny Saldaña. 2009. The Coding Manual for Qualitative Researchers. SAGE Publications.Google Scholar
- Helen Shen. 2014. Interactive notebooks: Sharing the code. Nature News 515, 7525 (2014), 151.Google ScholarCross Ref
- April Yi Wang, Anant Mittal, Christopher Brooks, and Steve Oney. 2019. How data scientists use computational notebooks for real-time collaboration. Proceedings of the ACM on Human-Computer Interaction (CSCW) 3 (2019), 1--30.Google ScholarDigital Library
- Aruliah D.A. Brown C.T. Hong N.P.C. Davis M. Guy R.T. Haddock S.H. Huff K.D. Mitchell I.M. Plumbley M.D. Wilson, G. and B. Waugh. 2014. Best practices for scientific computing. PLoS Biology 12, 1 (2014), e1001745.Google ScholarCross Ref
- Jo Wood, Alexander Kachkaev, and Jason Dykes. 2018. Design exposition with literate visualization. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2018), 759--768.Google ScholarDigital Library
Index Terms
- What's Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities
Recommendations
Fork It: Supporting Stateful Alternatives in Computational Notebooks
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing SystemsComputational notebooks, which seamlessly interleave code with results, have become a popular tool for data scientists due to the iterative nature of exploratory tasks. However, notebooks provide a single execution state for users to manipulate through ...
How Data Scientists Use Computational Notebooks for Real-Time Collaboration
Effective collaboration in data science can leverage domain expertise from each team member and thus improve the quality and efficiency of the work. Computational notebooks give data scientists a convenient interactive solution for sharing and keeping ...
Exploration and Explanation in Computational Notebooks
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing SystemsComputational notebooks combine code, visualizations, and text in a single document. Researchers, data analysts, and even journalists are rapidly adopting this new medium. We present three studies of how they are using notebooks to document and share ...
Comments