Predicting software defects in varying development lifecycles using Bayesian nets

Abstract

An important decision in software projects is when to stop testing. Decision support tools for this have been built using causal models represented by Bayesian Networks (BNs), incorporating empirical data and expert judgement. Previously, this required a custom BN for each development lifecycle. We describe a more general approach that allows causal models to be applied to any lifecycle. The approach evolved through collaborative projects and captures significant commercial input. For projects within the range of the models, defect predictions are very accurate. This approach enables decision-makers to reason in a way that is not possible with regression-based models.

Introduction

In [8] we reviewed the various approaches to software defect prediction. We concluded that traditional statistical approaches, such as using regression modelling alone, were inadequate. We proposed that causal models were needed for more accurate predictions. We described a simple Bayesian Net (BN) as an example of the kind of causal model needed for this purpose. Since then a number of authors, for example [3], [7], [24], have used BNs in various related aspects of software engineering management. Our own research has extended the use of BNs to more general software project management [13]. However, the original motivation of more accurate software defect prediction has continued to be an important focus of our research and this paper describes some of the most recent results in this area.

Following on from the early ideas presented in [8], in [12] we have shown how BNs can be used to predict the number of software defects remaining undetected after testing. This work led to the AID tool [22] developed in partnership with Philips, and used to predict software defects in consumer electronic products. Project managers use a BN-based tool such as AID to help decide when to stop testing and release software, trading-off the time for additional testing against the likely benefit.

Rather than relying only on data from previous projects, this work uses causal models of the Project Manager’s understanding, covering mechanisms such as:

• poor quality development increases the number of defects likely to be present,

• high quality testing increases the proportion of defects found.

Causal models are important because they allow all the evidence to be taken into account, even when different evidence conflicts. Suppose that few defects are found during testing – does this mean that testing is poor or that development was outstanding and the software has few defects to find? Regression-based models of software defects are little help to a Project Manager who must decide between these alternatives [8]. BN models allow back propagation of evidence to show the most likely causes of a given outcome. Data from previous projects are used to build the BN, with expert judgements on the strength of each causal mechanism.

In this paper, we extend the earlier work by describing a much more flexible and general method of using BNs for defect prediction. We also describe how the AgenaRisk [1] toolset is used to create an effective decision support system from the BN. An important limitation of the earlier work was the need to build a different BN for each software development lifecycle – to reflect both the differing number of testing stages in the lifecycle and the differing metrics data available. Given the work required to build a BN, this severely limits the practicality of the approach. To overcome this limitation, we describe a BN that models the creation and detection of software defects without commitment to a particular development lifecycle. We then show how a software development organisation can adapt this BN to their development lifecycle and metrics data with much less effort than is needed to build a tailored BN from scratch.

The contents of the remainder of the paper are as follows: in Section 2 we introduce BNs and show how they are used for causal modelling in software engineering. Section 3 introduces the idea of a ‘phase’ as a sub-part of a software lifecycle and shows how several phase models can be combined to model different lifecycles. The phase model is described in detail in Section 4; Section 5 shows how it is adapted to different development lifecycles. An experimental validation of defect predictions is described in Section 6.