Software Carpentry: lessons learned

2.1 Version 1: red light

In 1995–96, the author organized a series of articles in IEEE Computational Science & Engineering titled, “What Should Computer Scientists Teach to Physical Scientists and Engineers?”⁹. These grew out of the frustration he had working with scientists who wanted to run before they could walk, i.e., to parallelize complex programs that were not broken down into self-contained functions, that did not have any automated tests, and that were not under version control¹⁰.

In response, John Reynders (then director of the Advanced Computing Laboratory at Los Alamos National Laboratory) invited the author and Brent Gorda (now at Intel) to teach a week-long course to LANL staff. This course ran for the first time in July 1998, and was repeated nine times over the next four years. It eventually wound down as Gorda and the author moved on to other projects, but two valuable lessons were learned:

2.2 Versions 2 and 3: Another red light

The Software Carpentry course materials were updated and released in 2004–05 under a Creative Commons license with support from the Python Software Foundation¹¹. They were used twice in a conventional term-long graduate course at the University of Toronto aimed at a mix of students from Computer Science and the physical and life sciences.

The materials attracted 1000–2000 unique visitors a month. But while graduate students (and the occasional faculty member) found the course at Toronto useful, it never found an institutional home. Most Computer Science faculty believe that this basic material is too easy to deserve a graduate credit (even though a significant minority of their students, particularly those coming from non-CS backgrounds, have no better software development skills than the average physicist). Meanwhile, other departments believe that courses like this ought to be offered by Computer Science, in the same way that Mathematics and Statistics departments routinely offer service courses. In the absence of an institutional mechanism to offer credit courses at some inter-departmental level, this course, like many other interdisciplinary initiatives, was left without a home.

Secondly, despite repeated invitations, other people did not contribute new material beyond an occasional bug report (a point which we will return to in Section 6).

The most important lesson, though, was that while many faculty in science, engineering, and medicine agree that their students should learn more about computing, they won’t agree on what to take out of the current curriculum to make room for it. A typical undergraduate science degree in the US or Canada comprises roughly 1800 hours of class and laboratory time. Anyone who wants to add more programming, statistics, writing, or anything else must either lengthen the program (which is financially and institutionally infeasible) or take something out. However, everything in the program is there because it has a passionate defender who thinks it’s vitally important, and who is likely senior to those faculty advocating the change.

The sweet spot for this kind of training is therefore the first years of graduate school. At that point, students have time to learn (at least, more time than they’ll have once they’re faculty) and real problems of their own that they want to solve.

2.3 Version 4: orange light

The author rebooted Software Carpentry in May 2010 with support from Indiana University, Michigan State University, Microsoft, MITACS, Queen Mary University of London, Scimatic, SciNet, SHARCNet, and the UK Met Office. More than 120 short video lessons were recorded during the subsequent 12 months, and six week-long classes were run for the backers. We also offered an online class three times (a MOOC avant la lettre).

This was our most successful version to date, in part because the scientific landscape itself had changed. Open access publishing, crowd sourcing, the data deluge in the life sciences, and growing concern about reproducible research had convinced a growing number of scientists that knowing how to program was now as important as knowing how to do statistics. Even most of them, though, still (rightly) regarded it as a tax they had to pay in order to get their science done.

Despite this round’s overall success, there were several disappointments:

The biggest take-away from this round was the need to come up with a scalable, sustainable model for delivering training. One instructor simply can’t reach enough people, and cobbling together funding from half a dozen different sources every twelve to eighteen months is risky as well as wearying.

2.4 Version 5: green light

Software Carpentry rebooted again in January 2012 with a grant from the Sloan Foundation to the Mozilla Foundation. This time, the model was two-day intensive workshops like those pioneered by The Hacker Within, a grassroots group of grad students helping grad students at the University of Wisconsin - Madison.

Shortening the workshops made it possible for more people to attend, and increased the proportion of the material they could absorb. It also forced us to think much harder about what skills scientists really needed. Out went object-oriented programming, XML, Make, and other topics. Instead, we focused on a small set of tools that let us introduce higher-level concepts without learners really noticing (Section 3).

Reaching more people allowed us to recruit new instructors from workshop participants, which in turn allowed us to scale. Switching to a “host site covers costs” model was equally important: funding was still needed for 1.5 core staff to lead the project and match instructors to workshops, but everything else funded itself.

2.5 Version 6: A true community project

In July 2014, the author left Mozilla and set up the Software Carpentry Foundation, an independent non-profit foundation under the auspices of NumFOCUS. The SCF held its first elections in January 2015, in which instructors who had taught over the past two years selected seven of their own number as a Steering Committee to oversee the project’s operations. Since then, the SCF has formed partnerships with a growing number of institutions (see Table 1), run an ever-increasing number of workshops, and much more.

While the SCF is only nine months old, we have already learned many things. The most important are:

2.6 Data Carpentry

The biggest recent development, though, has been the foundation of a sibling organization called Data Carpentry in April 2014. Where Software Carpentry’s mission is to help scientists who are programming badly to program better, Data Carpentry’s focus is, as its name implies, to help them manage and analyze their data. Led by Dr. Tracy Teal, Data Carpentry was recently awarded $700,000 by the Moore Foundation, and is expected to grow rapidly over the coming two years.

2.7 Results

As we discuss in Section 8.1, we do not know how to measure the impact of our workshops. However, both their number (Figure 1), and the number of people attending (Figure 2), have grown steadily, as has the number of instructors (Figure 3).

Figure 1. Cumulative number of workshops.

Figure 2. Cumulative number of learners.

Figure 3. Cumulative number of instructors.

We are now a truly global organization (Table 3). And most importantly, feedback from participants is strongly positive. While there are always problems with software set-up and the speed of instruction (Section 8.2), 80–90% of attendees typically report that they were glad they attended and would recommend the workshops to colleagues.

4.1 Feedback loops

Giving each learner two sticky notes of different colors allows instructors to do quick true/false questions as they’re teaching. It also allows real-time feedback during hands-on work: learners can put a green sticky note on their laptop when they have something completed, or a red one when they need help.

We also use them as minute cards: before each break, learners take a minute to write one thing they’ve learned on the green sticky note, and one thing they found confusing (or too fast or too slow) on the red. It only takes a couple of minutes to collate these, and allows the instructors to adjust to learners’ interests and speed.

We frequently also ask for summary feedback at the end of each day. The instructors ask the learners to alternately give one positive and one negative point about the day, without repeating anything that has already been said. This requirement forces people to say things they otherwise might not: once all the “safe” feedback has been given, participants will start saying what they really think.

On a longer timescale, we send a post-workshop assessment questionnaire to attendees shortly after the workshop ends. Response rates vary, but are usually low, and the opt-in nature of the survey undoubtedly biases the data (Section 8.1). Feedback from instructors has proven more insightful. In August 2015, for example, Azalee Bostroem surveyed our instructors to find out what they were actually teaching about Python (http://software-carpentry.org/blog/2015/09/thinking-about-teaching.html). From this, we learned that 60% of our learners are novices with little or no prior programming experience, and that only a third of workshops get through the entire Python lesson.

Finally, starting in January 2015 we began running biweekly debriefing sessions for instructors who have recently taught workshops, in which they can discuss what they actually did, how it worked, how the lessons they actually delivered differed from our templates, what problems arose, and so on. Summaries are posted shortly after each meeting, and Alistair Walsh recently collected and posted information about the same Python lesson discussed above (http://software-carpentry.org/blog/2015/10/python-debriefing-summary.html). We are now (October 2015) beginning a redesign of the lesson to take all this information into account.

4.2 Live coding

We teach via live coding rather than using slides because:

It takes a bit of practice for instructors to get used to thinking aloud while coding in front of an audience, but most report that it is then no more difficult to do than talking off a deck of slides.

4.3 Open everything

Our grant proposals, mailing lists, and everything else that isn’t personally sensitive are out in the open (see 13 for links). We believe that letting people see us succeed, fail, and learn encourages them to be more involved in our community, and inspires them to be open as well.

4.4 Open lessons

This is an important special case of the previous point. Anyone who wants to use our lessons can take what we have, make changes, and offer those back by sending us a pull request on GitHub. As discussed in Section 6, this workflow is foreign to most educators, but allows us to scale and adapt more quickly and more cheaply than the centralized approaches being taken by many high-profile online education ventures.

For example, we recently “published” our core lessons through Zenodo. The number of contributors per lesson is shown in Table 2. The distribution of contributions has the usual long-tail distribution, but the fact remains that our lessons have had more contributors than many “massive” and “open” online courses.

4.5 Use what we teach

We also make a point of eating our own cooking, e.g., we use GitHub for our web site and to plan workshops. Again, this makes us more credible, and gives instructors hands-on practice with the things they’re going to teach. Up until a year ago, the (considerable) downside to this was that it could be difficult for newcomers to contribute material. We have simplified our templates and build procedures considerably to fix this, and will be making more changes early in 2016 to incorporate further insights.

One problem we haven’t solved is the bikeshedding mentioned earlier. Many contributors would rather spend days tweaking the build process for lessons rather than an hour coming up with some new self-test exercises for those same lessons, both because they are on more familiar ground when debating programming issues, and because the feedback loop is much tighter. One of our goals for the coming year is to push the bulk of discussion toward teaching practices and lesson content.

4.6 Meet the learners on their own ground

Learners tell us that it is important to them to leave the workshop with their own machine set up to do real work. We therefore continue to teach on all three major platforms (Linux, Mac OS X, and Windows), even though it would be simpler to require learners to use just one (Section 8.3).

We have experimented with virtual machines (VMs) on learners’ computers to reduce installation problems, but those introduce problems of their own: older or smaller machines simply aren’t fast enough, and learners often struggle to switch back and forth between two different sets of keyboard shortcuts for things like copying and pasting.

Some instructors use VPS over SSH or web browser pages instead. This solve the installation issues, but makes us dependent on host institutions’ WiFi (which can be of highly variable quality), and has the issues mentioned above with things like keyboard shortcuts.

4.7 Collaborative note-taking

We often use Etherpad for collaborative note-taking and to share snippets of code and small data files with learners. (If nothing else, it saves us from having to ask students to copy long URLs from the presenter’s screen to their computers.) It is almost always mentioned positively in post-workshop feedback, and several workshop participants have started using it in their own teaching.

4.8 Pair programming

Pairing is a good practice in real life, and an even better way to teach: partners can not only help each other out during the practical, but can also clarify each other’s misconceptions when the solution is presented, and discuss common research interests during breaks. To facilitate this, we strongly prefer flat (dinner-style) seating to banked (theater-style) seating; this also makes it easier for helpers to reach learners who need assistance.

4.9 Diversity

On June 24–25, 2013, we ran our first workshop for women in science, engineering, and medicine. This event attracted 120 learners, 9 instructors, a dozen helpers, and direct sponsorship from several companies, universities, and non-profit organizations. Our second such workshop ran in March 2014, and we have done half a dozen of varying sizes since. While we do occasionally get complaints (mostly from outsiders) about such events being discriminatory, they are overwhelmed by the uniformly positive response from participants, many of whom say that they would probably not have attended a mixed-gender event because of previous bad experiences with tech meetups.

5.1 Why teach?

But why do people volunteer as instructors?

8.1 Long-term assessment

Our biggest challenge is figuring out whether we are actually helping scientists get more science done, and if so, how, and how much. 5–8 seem to show that we are, but we have not yet done a large-scale, long-term follow-up. This is partly because of a lack of resources, but it is also a genuinely hard problem: no one knows how to measure the productivity of programmers, or the productivity of scientists, and putting the two together doesn’t make the unknowns cancel out.

8.2 Too slow and too fast

Our second biggest challenge is the diversity of our learners’ backgrounds and skill levels. No matter what we teach, and how fast or how slow we go, 20% or more of the room will be lost, and there’s a good chance that a different 20% will be bored.

The obvious solution is to split people by level, but if we ask them how much they know about particular things, they regularly under- or over-estimate their knowledge. We have therefore developed a short pre-assessment questionnaire that asks them how easily they could do a small number of specific tasks. It is useful, in that it gives instructors some idea of who they’re going to be helping, but we have done nothing to validate the questions themselves, i.e., to ensure that respondents are interpreting them the same way that we are, or that their categorization of respondents corresponds in any meaningful way to actual proficiency. As mentioned in Section 8.1, we have been trying for several years to find the support needed to do rigorous assessment of this and other aspects of our program, but if funders are reluctant to invest in training, they are doubly reluctant to invest in measuring its effects.

8.3 “Is it supposed to hurt this much?”

Third, getting software installed is often harder than using it. This is a hard enough problem for experienced users, but almost by definition our audience is inexperienced, and our learners don’t (yet) know about system paths, environment variables, the half-dozen places configuration files can lurk on a modern system, and so on. Combine that with two versions of Mac OS X, three of Windows, and two oddball Linux distributions, and it’s almost inevitable that every time we introduce a new tool, it won’t work as expected (or at all) for at least one person in the room. Detailed documentation has not proven effective: some learners won’t read it (despite repeated prompting), and no matter how detailed it is, it will be incomprehensible to some, and lacking for others.

8.4 Editors

Editing text should be a minor problem, but if you’re standing in class telling three sets of users, "Now open Notepad++ if you’re on Windows, or Kate if you’re on Linux, or TextMate if you’re on a Mac, or whatever you want to use if you’re more advanced", and then demonstrate with whichever you have on your laptop (which looks different from what half of your learners are sitting in front of), you will cause mass confusion.

We therefore still use http://www.nano-editor.org/ as an editor in class, even though none of our instructors use it for real work. Arguments over this are another example of the bikeshedding discussed in Section 7: many people who are passionate about programming are also passionate (some might say “zealous”) about their favorite editor, and will argue about the relative merits of various choices at length.

The choice of editor is also an example of expert blind spot. People who know a subject well often have trouble re-imagining it through novice eyes, and hence underestimate how difficult “simple” tasks actually are for newcomers. For example, every reasonably experienced user of the shell knows that an editor can run inside a terminal window, so that a single fixture on the screen can play multiple roles. This is not obvious to newcomers, who are frequently confused when instructors move back and forth between an editor and a regular shell prompt in a single window.

8.5 Testing

We believe that software testing is important, but no longer include it in our core curriculum. The reason is that while it’s easy to teach the mechanics of using a unit testing framework (https://en.wikipedia.org/wiki/XUnit), we do not know what tests to tell learners from dozens of different disciplines to write for their (very diverse) programs. In addition, while most research communities have a collective notion of what is “close enough” for laboratory work (“Physicists worry about decimal places, astronomers worry about the exponent, and economists are happy if they’ve got the sign right.”), similar heuristics have not yet emerged for key aspects of computational work. An attempt in 2014–15 to collect examples of actual tests from different domains didn’t achieve critical mass, but we hope to take another run at doing this.

8.6 Watching vs. doing

We try to make our teaching as interactive as possible, but we still don’t give learners hands-on exercises as frequently as we should. We also don’t give them as diverse a range of exercises as we should. This is simply due to a lack of time: two eight-hour days are as much as learners’ brains can handle, but not nearly enough to give them all the practice they need.

There is also a constant tension between having students do realistic exercises drawn from actual scientific work-flows, and giving them tasks that are small and decoupled, so that failures are less likely and don’t have knock-on effects when they occur. This is exacerbated by the diversity of learners in the typical workshop.

8.7 Less of a problem

One issue which is less of a problem than it used to be is financial sustainability. The “host site covers costs” model scales naturally with the number of workshops, while a growing number of organizations are keen to partner with us, primarily to build local capacity to run more work-shops when and as needed. While we do not wish to tempt fate, the Software Carpentry Foundation does seem to be headed toward financial stability.