Pair programming productivity: Novice–novice vs. expert–expert

https://doi.org/10.1016/j.ijhcs.2006.04.010Get rights and content

Abstract

Agile Software Development methodologies have grown in popularity both among academic researchers and industrial practitioners. Among the various methodologies or practices proposed, pair programming, which is concerned with two programmers collaborating on design, coding and testing, has become a controversial focus of interest. Even though some success stories have been reported with the use of pair-programming in real software development environment, many people remain rather skeptical of the claims on pair-programming productivity. Previous studies in pair programming have only addressed the basic understanding of the productivity of pairs and they have not addressed the variation in productivity between pairs of varying skills and experience, such as between novice–novice and expert–expert. Statistical productivity measurements reported by different researchers also seem to lead to contradictory conclusions. Until now, the literature has not addressed how those results and experiments were related to each other. In this paper, we propose a controlled experiment called repeat-programming which can facilitate the understanding of relationships between human experience and programming productivity. Repeat-programming can be performed when controversial issues in non-traditional programming methodologies and development productivity need to be investigated into. To illustrate how the proposed empirical experiment can put arguable, divisive problems into perspective, we have examined the productivity in pair programming as a case study. With repeat-programming, we are able to (i) better understand why results of previous pair programming control experiments reached different conclusions as to the productivity of pair programming and (ii) most importantly, present a case in which novice–novice pairs against novice solos are much more productive than expert–expert pairs against expert solos.

Introduction

Studies of the applications of cognitive science and programming in computer science started in the 1970s. This area has focused on empirical studies of observable behavior of people as they program. The results of such studies are useful in understanding natural language texts (Atwood and Ramsey, 1978), in educational models (Traynor and Gibson, 2004), in the development of computer aptitude tests for screening job candidates (Mayer and Stalnaker, 1968) and in the modeling of knowledge (Mckeithen et al., 1981; Soloway and Ehrlich, 1984). Previous approaches to the study of computer programming have formulated it as an intelligence-oriented task undertaken by a single programmer. Rarely it has been considered as a multi-disciplinary topic in group dynamics and human intelligence. Further, understanding a highly intelligent task solved by two people in a collaborative manner may also involve fundamental problems of modern informatics, software engineering, time-to-market product development and psychology. For example, pair programming, which involves two developers collaborating as a single individual on the same programming task, has been shown by two experiments to be productive, and to produce higher quality code than either of them may as single individuals alone (Nosek, 1998; Williams et al., 2000). This can be a case where our knowledge of the software development process can be beneficial to other fields such as cognitive science.

Human programming is in fact an interdisciplinary topic that is concerned with the internal information mechanisms on one side and the brain on the other side. To build highly intelligent systems, we are interested in using process engineering and software engineering to explain the mechanisms of memory and learning. Such developments will have significance for the development of the next generation computer (Wang, 2002). This paper aims at exploring the two ways of programming, pair programming and solo programming and at advancing our understanding of the programming productivity.

The idea behind pair programming, also known as Collaborative Programming, is straight-forward. It involves two programmers collaborating side-by-side on the design, coding and testing of a piece of software. One, the Driver, controls the keyboard/mouse and actively implements the program. The other, the Navigator/Observer, continuously observes the work with a view to identifying tactical defects and providing strategic planning.

Pair programming has recently drawn much attention because of the increase in popularity of eXtreme Programming (XP) (Beck, 2000), but it was first reported in the workplace in 1995 (Constantine, 1995) and was discussed as early as 1991 in Distributed Cognition (Hutchins, 1995). The researchers in distributed cognition reported that the verbal and non-verbal correlated behaviors of a pair of programmers can help them to search through larger spaces of alternatives (Flor and Hutchins, 1992). From 1998 to 2001, scholars in computer science conducted a number of controlled experiments to evaluate the effectiveness of pair programming. The settings of these experiments had a number of similarities in that subjects were divided into a pair-programming group and a solo-programming group and that the two groups were then asked to write the same program so that their results could be directly compared. Interestingly, the experiments produced different and apparently incompatible results.

In pair programming, time productivity can be measured as a pair takes z% more effort than an individual. To avoid confusion between elapsed time and total time per programmer in our discussion, the z% is interpreted here as “Relative Effort Afforded by Pairs” (REAP) against solo programming. The formula for calculating REAP is shown in (1):REAP=(Elapsed_time_of_pair)×2-(Elapsed_time_of_individual)Elapsed_time_of_individual×100%.

If REAP is zero, pair programming halves the time required for solo programming. When REAP is greater than zero but less than 100%, pairs require more total man-hours but do complete tasks faster. This can be useful when the critical issue is time-to-market. The commanding market share advantage that can accrue to early mover companies can make it worthwhile for them to spend more on short-term development costs.

The relative productivities of pair and solo programming have been the focus of much research interest. Nosek (1998) reported that REAP was 42%. In contrast, Williams et al. (2000) reported that REAP was 15% on the same task yet the quality of pair programming was higher. Nawrocki's study (2001) indicated that pair programming could be less efficient as his experiment showed that REAP was 100%.

These experiments do appear to indicate that productivity in pair programming can vary significantly but the absence of an underlying principle makes it hard to account for the productivity variations. There is a need to understand pair programming and the previous programming experiments by an easily reproducible experiment. For example, the findings of an experiment should allow us to judge when pair programming should be adopted over solo programming. In this paper, we propose to investigate into the relative productivities of novice–novice and expert–expert pair programming. Nosek (1998) pointed out that two average or less-experienced workers collaborating to perform tasks may take on a much more challenging task than can two individuals. This is the equivalent of saying that pair programming is much more effective and efficient for inexperienced programmers than for expert programmers. By performing an experiment, we would like to investigate into the differences.

The paper is organized as follows. Section 2 clarifies the meaning of productivity in pair programming and introduces the background to the three experimental studies under consideration in this paper, by Nosek in 1998, Williams in 2000 and Nawrocki in 2001. In Section 3, we propose our experiment, “repeat-programming”, which describes the behavior of novice and expert programmers. The experiment seeks to keep many variables constant including programming capabilities, fast and slow programmers and problem comprehension. Thus, the outcome of the experiment can clearly illustrate the productivity of pair programming. Section 4 discusses our experimental results as well as those of Nosek, Williams, Nawrocki and generalizes them into two guidelines for the adoption of pair programming. Section 5 suggests guidelines for future work in software development using pair programming. The final section offers our conclusion.

Section snippets

Solo programming and pair programming

It is expected that, as a recognized practice in Agile Software Development, pair programming will continue to attract more attention from academic and industrial researchers. Our study here includes three controlled experiments in pair programming. The experiments have been the center of controversial articles in the Agile Software Development community. We would like to provide the background in details and to show the importance of our case study to the research community.

The three

Repeat-programming

Owing to the differences in programming experience, novice and expert programmers should perform the same programming job very differently in terms of time and software quality. A novice programmer takes longer to complete a new program. Once he has gained experience by working on that and other problems of that type, he can write faster and better. When it comes to programming skill and experience in forming pairs for pair programming, therefore, we have two extremes: novice–novice and

Two issues

Repeat-programming shows that pair programming performs well when a pair encounters challenging programming problems. By “challenging programming problems”, we mean problems that require more sophisticated and less straight-forward algorithms to solve. This is rarely related to the programming skills of any particular computer language.

This section discusses some of the implications of our experiment. Particularly, we consider how we can understand the discrepancies of REAP in the three

Future work

Given the limitation that in repeat-programming programmers with “similar-capability” is in fact an ideal case, the experiment requires further tests. It is for this reason that we do not strictly define the term “similar-capability”. Indeed, it is conceivable that different programmers might solve different problem at different speeds.

Repeat-programming described here can provide a basis for future work that seek to develop a pair programming framework that can be embedded in a software method

Conclusion

Previous studies in pair programming have only addressed the basic understanding of the productivity of pairs, rather than the change of productivity between novice–novice pairs and expert–expert pairs. The three controlled experiments, by Nosek, Williams and Nawrocki, all reported statistical productivity measurements, but their conclusions seemed contradictory. Until now, the literature has not addressed how those results and experiments were related to each other. This paper contributes to

Acknowledgements

The authors would like to thank John Nosek for the many insights he contributed to this work. They also thank other anonymous reviewers of IJHCS for their thoughtful comments. In particular, we thank one of them who suggested that we use a term to refer to the relative effort of a pair of programmers. We thank Pekka Abrahamsson and Laurie Williams for their comments on the earlier version of this paper. Finally, we also appreciate very much the Associate Editor, Deborah A. Boehm-Davis, for her

References (29)

  • K.B. Mckeithen et al.

    Knowledge organization and skill differences in computer programmers

    Cognitive Psychology

    (1981)
  • Atwood, M.E., Ramsey, H.R., 1978. Cognitive structure in the comprehension and memory of computer programs: an...
  • Baheti, P., Gehringer, E., Stotts, D., 2002. Exploring the efficacy of distributed pair programming. In: Proceedings of...
  • K. Beck

    eXtreme Programming Explained: Embrace Change

    (2000)
  • L.L. Constantine

    Constantine on Peopleware

    (1995)
  • N.V. Flor et al.

    Analyzing distributed cognition in software teams: a case study of team programming during adaptive software maintenance

  • M. Friedman et al.

    Software Assessment: Reliability, Safety, and Testability

    (1995)
  • Gallis, H., Arisholm, E., Dybå, T., 2002. A transition from partner programming to pair programming—an industrial case...
  • Gallis, H., Arisholm, E., Dybå, T., 2003. An initial framework for research on pair programming. In: Proceedings of...
  • E. Gamma et al.

    Design Patterns: Elements of Reusable Object-Oriented Software

    (1995)
  • Heiberg, S., Puus, U., Salumaa, P., Seeba, A., 2003. Pair programming effect on developers productivity. In:...
  • E. Hutchins

    Cognition in the Wild

    (1995)
  • Keefer, G., 2002. Extreme programming considered harmful for reliable software. In: Proceedings of the 6th Conference...
  • Mayer, D.B., Stalnaker, A.W., 1968. Selection and evaluation of computer personnel—the research history of SIG/CPR. In:...
  • Cited by (93)

    • GitHub Copilot AI pair programmer: Asset or Liability?

      2023, Journal of Systems and Software
    • Fifty years of the psychology of programming

      2019, International Journal of Human Computer Studies
      Citation Excerpt :

      The rise of agile software development methods, where programming would be far more closely integrated into the organisation, became a clear focus (e.g. Sharp and Robinson, 2008). The commercial discovery of “pair programming” introduced not only a need for research into the interaction between programming pairs (e.g. Lui and Chan, 2006), but also a research opportunity through the ability to capture and analyse natural vocalisation of programming knowledge and strategy (e.g. Bryant et al., 2008). There continued to be comparative studies of alternative programming languages, now extending beyond the vogue for object-oriented programming to include logic and functional programming paradigms, as well as evaluation of visual language syntax and of the visual notations and diagrams used in other phases of the software development process.

    View all citing articles on Scopus
    View full text