1.1 Overview of the prediction framework

To date, the underlying factors responsible for an increase in the number of method-level defects in a software product (for example, the defect acceleration, as characterized by the defect velocity and defect introduction time) have not been fully considered in defect prediction studies. In view of the fact that the existing literature has not demonstrated a way to bridge this gap, this manuscript therefore aims to address this need by presenting a cost-effective data preprocessing approach for enhancing the quality of the method-level data applied in defect prediction studies while revealing the factors that contribute to an increase in the number of defects in a software product. Thereafter, it is demonstrated how the relationship between the defect acceleration and the number of defects can be useful in predicting the number of defects in a new software version. Fig 1 presents the step-by-step approach, from preprocessing the method-level datasets to predicting the number of defects in a new software version using the information obtained from the current software version. In addition, the preprocessed datasets are applied in evaluating the average classifier performance. The preprocessing of a method-level dataset is performed as follows. First, relevant features are selected from the dataset. The reason for such feature selection is to ensure that only meaningful attributes that show sufficient correlation with the target class (defective class) are selected. Redundant and irrelevant features are discarded. Second, outlier removal is performed. Outliers are data points that lie far away from the main cluster(s) of data. Outliers may occur in datasets as a result of measurement variations or may be an indication of experimental error. In this study, outliers were identified and removed during the preprocessing phase because they might impact model performance. The preprocessing time for the method-level datasets applied in the current study is also presented. Following outlier removal, the resulting inliers are further preprocessed by determining the average defect density, average defect velocity and average defect introduction time. Furthermore, the correlations of the average defect density, average defect velocity and average defect introduction time with the number of defects are determined. The preprocessed datasets are divided at a ratio of 80% to 20% for training and testing the defect prediction models, respectively. The reason for training the models with 80% of the data is to ensure that the models learn accurately and independently via cross-validation sampling to avoid bias. The models are then validated with the remaining 20% of the data for predicting the number of defects in a new software version, and the average classifier performance is also determined on the test data. In the current manuscript, we demonstrate how imbalanced datasets can be preprocessed to avoid bias by applying the proposed data preprocessing framework. In addition, the time required to preprocess the method-level datasets is presented. This manuscript also shows how certain variables such as the average defect velocity can be applied in predicting the number of defects at the method level in a new software version. Furthermore, this manuscript demonstrates the need to determine the average performance of classifiers when applied to imbalanced data as well the average information entropy in classification algorithms.

The remainder of this paper is organized as follows. Section 2 presents related work. Section 3 describes the materials and methods. Section 4 presents our experimental methodology and outcomes. Section 5 presents our results and discussion. In Section 6, we present our conclusion and directions for future work.

3.1 Formulation of research questions

This subsection presents the research questions addressed in this study. One of the objectives of this study is to determine how the techniques proposed in our previous work [3] can be applied in data preprocessing to facilitate the estimation of the number of method-level defects. Thus, the research questions (RQs) addressed in our study are as follows:

1. RQ1. How many studies have applied the aforementioned optimal variables when performing data preprocessing for defect prediction at the method level?
2. RQ2. How do these variables address method-level data quality issues?
3. RQ3. What are the limitations of the current research in the supervised machine learning domain?

To address these research questions, we considered several factors. For instance, to address RQ1, we first investigated articles in specific topic areas related to defect prediction to determine whether these optimal variables have been applied in predicting the possible number of defects in a future software version at the method level. To address RQ2, we focused on the results of the individual studies to determine whether these variables are capable of addressing specific data-related issues. For RQ3, which concerns the limitations of the current state of research in this area, we considered one specific question:

RQ3.1. Is there evidence that these optimal variables have not been fully applied in machine learning studies when attempting to predict the number of defects in a new version of software at the method level?

3.2 Definitions of metrics

Defect density is an essential attribute for determining software reliability; a software product can be released on the market only after its defect density is considered low enough to avoid criticism [30]. The quality of a software product can be evaluated based on its defect density [31], [32]. A software product with a high defect density tends to perform less well, while a product with a low defect density has the ability to run without failure [33], [34]. Specifically, a high defect density indicates that a software program contains a large number of defects; as such, the ability to estimate defect density makes it easier for a testing team to focus on identifying and correcting as many defects as possible with limited resources [33]. Software companies tend to benefit from the early detection of defects that are likely to be present in software, which is facilitated by knowledge of the defect density in a software product of interest [24], [35], [36]. In addition, defect density data provide information that can help development teams keep proper records while attempting to reduce the number of defects in upcoming versions of a current product [33].

Moreover, defect density also provides the development team with helpful information for keeping track of the progress made in reducing the number of defects during software project transitions [33]. It is also important to note that while overall software quality is a matter of concern, the aspects that require the greatest attention are mainly the components with the highest defect densities; thus, the ability to determine defect density is key [37]. Furthermore, based on the information provided by the defect density of a software product, the number of defects in an upcoming release of that product can be estimated [37]. Based on the relevant mathematical relationship, the optimal variables chosen as the basis of the current study are derived to provide a means of determining the possible numbers of defects likely to be present in various software products, as reported in [3]. If the number of defects can be determined prior to software testing, it will be easier for the testing team to focus on addressing as many defects as possible with the limited resources available [33]. Typically, throughout the SDLC, software developers aim to meet the project deadline and thus carry out software development as quickly as possible, without proper elaboration of the impact of such speedy transitions. Such high-velocity transitions through the SDLC can expose a software product to defects. Therefore, to improve software quality, it is wise to determine the impact of the average defect velocity on the number of software defects as a software product transitions from one phase of the SDLC to another. Any significant improvement in the SDLC will lead to a reduction in the rate at which defects occur, reduce the need for software rework and ultimately improve software quality and productivity [38]. Hence, it would be in the best interests of the machine learning community if the number of defects in a new version of software could be successfully estimated. In addition, it would be beneficial to have an idea of the rate at which these defects occur and the impact of this rate on software products. In this study, we hypothesized that there is a possibility that the proposed optimal variables have an impact on the number of defects at the software method level since the defect velocity has a strong impact at the class level, as reported in [3]. Further hypotheses were also formulated in the current study. To confirm these hypotheses, an experiment was carried out in which the defect velocity was used to predict the numbers of defects in software programs at the method level, with the goal of assisting software teams in improving the quality of software through proper management. The defect velocity was chosen because it exhibits a higher correlation with the number of defects than either the defect density or defect introduction time does. Notably, a well-managed software development process will result in less defective software by allowing the speed of software development to be increased while ensuring that the demands of the customers are met [39].

At the starting point of a project, the number of defects is zero, as illustrated in Fig 2. The chance of defect introduction increases over time as the project proceeds from one phase to the next. With further phase transitions during software development, the defect acceleration increases, which can lead to an increase in the number of defects. The defect acceleration is the change in the defect velocity at a given instant of time. The increase in the number of defects and the defect density are therefore functions of the defect acceleration.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Rayleigh distribution curve [40].

https://doi.org/10.1371/journal.pone.0229131.g002

If the defect acceleration is low, then the number of defects will also be low, and because the defect acceleration is defined as the change in the defect velocity over time, the defect density, which is affected by the defect velocity, will be low as well. If the defect acceleration increases, this increased acceleration will lead to an increase in the number of defects and, hence, an increase in the defect density; therefore, the defect density g is a function of the defect acceleration, which characterizes defect occurrence given a constant project size. The defect acceleration can be calculated as the change in the defect velocity over the change in time, as follows: (1) Defect density: The defect density g represents the ratio of the number of defects to the size of a software project. An increase in the number of defects in a software project will result in an increase in the defect density [3]. The defect density as applied in this study is measured in number per unit project size and is calculated using the following equation: (2)

Defect velocity: The defect velocity v is the change in the defect position with respect to time t and is measured in units of defects per day. The defect position here ranges from the requirement phase of the SDLC to the implementation phase and indicates when defects might have occurred during software development.

Defect introduction time: The defect introduction time t is the time at which defects occur in a software product and is measured in days.

3.3 Machine learning models for data analysis

As mentioned earlier, one of the objectives of the current study is to extend the implementation of the previously proposed optimal variables to the preprocessing of method-level datasets and predicting the number of defects in a new version of software. Furthermore, we compare the results obtained when various learning algorithms are applied to unprocessed data with the results obtained on preprocessed data. To achieve this objective, both classification algorithms and evaluation metrics are required. For this purpose, in the current study, we have implemented several classifiers, i.e., machine learning models for data analysis tasks. The list of classifiers applied in the current study includes naïve Bayes, logistic regression (LR), neural network, K-nearest neighbors (KNN), support vector machine (SVM) and random forest (RF) classifiers.

Naïve Bayes: A naïve Bayes classifier is an effective tool that is capable of generating accurate prediction results, and it is a suitable choice for classification problems involving variates of independent variables [41]. It is capable of handling large datasets and often outperforms more sophisticated classification models. The naïve Bayes model is derived from Bayes’ theorem under the hypothesis of independence among the variables. Bayes’ theorem provides an avenue for estimating the conditional probability prb(a|b) from prb(a), prb(b) and prb(b|a). For a large dataset, the posterior probability of class a is calculated as (4)

where prb(a|b) = the posterior probability of class a given attribute b,

prb(a) = the prior probability of class a,

prb(b) = the prior probability of attribute b,

n = the number of instances, and

prb(b|a) = the likelihood, that is, the probability, of attribute b given class a. The class with the highest posterior probability is the predicted outcome.

Logistic regression (LR): LR estimates the likelihood of an action that has two possible values (binary classification) and thus is suitable for a predictive analysis conducted when the possible values of the independent variables are 0 and 1. The LR model is employed to analyze data and to interpret the relationship between one dependent binary variable and one or more independent variables.

Neural network: A neural network classifier is a classification model inspired by the neurons in the human brain. The units in a neural network perform tasks similar to those of human neurons. Such a network has an input layer, an output layer, and one or more hidden layers in between. During classification, data pass through the units of the neural network, which perform various mathematical computations. These units interact with each other via connections linking the various layers. Each connection has a number, called a weight, associated with it. When the neural network receives an input, it processes that input by performing calculations based on the connection weights to produce an output. A neural network is trained by adjusting the values of the connection weights; this is achieved by means of a backpropagation algorithm, which trains from the input layer to the output layer and vice versa. Furthermore, a neural network can be trained by reducing the value of a loss function over a training set using a gradient-based method [42].

K-nearest neighbors (KNN): A KNN classifier is a straightforward classification model that considers all possible outcomes and classifies an instance in advance based on a resemblance factor. An instance is classified based on the voting strength of its neighbors. In the classification result, each instance is assigned to the class with the highest number of corresponding votes, as determined by a distance factor. For the distance factor in this paper, we use the Euclidean distance between two neighbors a and b, which also represents the length of (). In Cartesian coordinates, if the coordinates of two different points in a Euclidean k-space are given by a = (a₁, a₂, …, a_n) and b = (b₁, b₂, …, b_n), then the distance d from point a to point b (or from b to a) is determined by the Pythagorean formula: (5) (6)

Support vector machine (SVM): An SVM classifier finds the line of best fit that maximizes the margin between two classes that are linearly separable. The SVM classifier treats every instance of a class as a vector of the corresponding input variables. Thus, (7)

Consequently, a dataset D takes the form of pairs of vectors and classes: (8) where x = the vectors, y = the classes associated with the vectors (either +ve or -ve), and n = the number of instances.

Then, the SVM locates the two closest points between the two classes. The SVM connects these two points with a connection line and draws a perpendicular line bisecting this connection line. Thus, the two closest points define the line of best fit. This line of best fit has an intercept b and a normal vector that represents a weight vector such that all point vectors on the line satisfy the linear equation (9) We can rewrite Eq (9) as (10) such that the scalar product of the weight vector and the point vector is equal to the intercept of the line of best fit. From Eq (8), the training dataset can be written as (11) in the form of point vectors paired with their corresponding classes y_i. Thus, the linear classification as a function of the point vector becomes (12)

Random forest (RF): An RF classifier consists of a large collection of decision trees. These random trees are said to make up a forest. A reasonable number of decision trees are generated based on random selections of data and variables. The RF predicts the class of the dependent variable based on the number of trees. The decision trees are then used to predict the classification outcome. RF classifiers can sometimes outperform other sophisticated classifiers [43], [44].

These learning algorithms have different parameter settings, which we have maintained throughout the experiments. For instance, in the naïve Bayes classifier, the parameter for M-estimation was set to 2.0 with 100 sampling points. For the LR classifier, L2 regularization (square weights) was used, with a training error cost (C) of 100 and preprocessing based on a normal data distribution. The neural network classifier was configured with 20 hidden-layer neurons, a regularization parameter of 10 and a maximum of 300 iterations. For the KNN classifier, the number of neighbors was set to 5, and Euclidean normalization was applied to continuous attributes. For the SVM classifier, the objective function was C−SVMCost(C) = 1.00; the radial basis function (RBF) kernel was used, i.e., exp(−γ∥X − X′∥²), where ∥X − X′∥² is the squared Euclidean distance between the feature vectors of two samples and γ = , with σ being a free parameter; and the numerical tolerance was 0.0010, with estimated class probabilities and normalized data. For the RF classifier, the number of trees in the forest was 10, and the maximum number of nodes per tree was defined to be 5 as the criterion to stop splitting.

4.1 Data collection and preprocessing technique

In the current study, the ELFF open-source Java projects, which contain 289,132 methods, were used. The ELFF dataset is vailable at www.elff.org.uk/ESEM2016, [50]. Furthermore, these datasets contain a significant number of inconsistencies, necessitating proper preprocessing and cleaning to avoid misleading results.

4.1.2 Outlier removal from imbalanced method-level datasets.

A dataset is imbalanced if the categories into which the data are to be classified are not almost equally represented [53]. Thus, data imbalance refers to the case in which the proportions of defective (minority) and defect-free (majority) modules or methods in a project are not equal. In addition to feature selection, an important phase of data preprocessing is the outlier removal. Therefore, we first checked whether the data contained outliers before performing further preprocessing tasks.

Outliers are data points that are isolated from the main cluster(s) of data. Outliers may occur in datasets as a result of measurement variations or may be a manifestation of experimental error. In this study, outliers were identified and eliminated during the preprocessing phase to prevent them from impacting the results. Fig 3 illustrates how the outliers were identified and removed from the data. To identify outliers, first, we loaded the data file into the Orange workspace and visualized the inliers and outliers in the data. An outlier widget was then connected to the data file to enable the separation of the outliers. Thereafter, we reset the outlier widget signal to capture only the inliers in the data. To achieve a reliable, bias-free dataset that would be suitable for use in prediction, it was then necessary to subject the inliers to further preprocessing. The reason for removing outliers is to enable prediction models to learn only from the main clusters of data to ensure an accurate assessment of the models’ performance. If a prediction model is trained on data containing outliers, the resulting noise in the data can cause the model to produce misleading results.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Outlier removal procedure.

https://doi.org/10.1371/journal.pone.0229131.g003

Table 1 presents the detailed statistics of the preprocessed ELFF datasets at the method level. From left to right, the columns present the project name, the number of methods, the number of defective methods, the number of attributes, the percentage of defective methods, the corresponding method-level defect density, the time of defect occurrence and the corresponding defect velocity. Fig 4 presents a flow diagram of the data preprocessing steps applied to reduce data inconsistencies to ensure that the data applied in this study were free from bias. The data preprocessing technique is presented in Algorithm 2. During data preprocessing, we first analyzed the datasets applied in this study to understand their nature in detail. We determined the source, size, and format of each dataset. Second, we assigned a unique identifier to each dataset to enable us to accurately identify the defective methods in the datasets.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Flow diagram of data preprocessing to eliminate inconsistencies in the data.

https://doi.org/10.1371/journal.pone.0229131.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Method-level statistics of the preprocessed ELFF datasets.

https://doi.org/10.1371/journal.pone.0229131.t001

When identifying the defective and defect-free methods, a binary decision (i.e., a choice between two alternatives) was made with an output of (0, 1) to ensure that accurate records were obtained during data preprocessing.

An error count greater than or equal to 1 indicated a defective method, whereas an error count of 0 indicated a defect-free method or module. This process was repeated for all datasets applied in the current study, as presented in Algorithm 2. In total, 80% of the preprocessed data were used as training data, and validation was performed using the remaining 20% of the data in both the regression and classification phases. This data ratio was selected to ensure that the learning algorithms would be adequately trained to avoid bias. Six selected classifiers were implemented on each dataset to determine their individual classification performances; thereafter, the average classification performance of each classifier on all datasets was determined accordingly. The total preprocessing time, which includes the load time, feature selection time, feature ranking time and outlier removal time, is presented in Table 2.

Load time: This is the time required to import the method-level data file into an Orange 2.7 data file. The load time is measured in seconds.

Feature selection time: This the time required to select all features from a dataset before ranking. The feature selection time is also measured in seconds.

Feature ranking time: This is the time consumed for ranking the features in the datasets, measured in seconds.

Outlier removal time: This is the time required to separate the outliers from each dataset.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Preprocessing time for the method-level ELFF datasets.

https://doi.org/10.1371/journal.pone.0229131.t002

Algorithm 2 Preprocessing technique for defect prediction

1: procedure Phases of Data Preprocessing for Predicting Possible Numbers of Defects and Average Classifier Entropy Evaluation

 Input: Datasets DS

 Output: Preprocessed data for determining the numbers of defects and average classifier performance based on 10 × 10 cross-validation sampling in terms of both the number of folds and the number of repetitions of training/testing

2:  Analyze method-level datasets DS_ml

3:  Perform feature selection on each dataset DS_ml

4:  Remove outliers from each dataset DS_ml

5:  for i = 1 to n do

6:   Assign unique IDs to the inliers of each DS_ml

7:   Recheck features in each module of each dataset

8:   Remove incomplete and missing values

9:   Identify defective and defect-free modules MD_1−n among DS_ml

10:   for i = 1 to n do

11:    if number of faults ≥ 1 then

12:     (MD) = 1

13:     MD = defective: f(number of faults present)

14:     Determine and record number and percentage of defects

15:    else

16:     (MD) = 0

17:     MD = defect-free: f(no faults present)

18:     Recheck feature completeness in DS_ml

19:    end if

20:    for i = 1 to n do

21:     First, determine the average defect density

22:     Determine v and t; see Eq (2)

23:     Determine the correlations and record the impacts of g, v and t on the number of defects for the current version in datasets DS_ml

24:     Estimate the possible number of defects in the new version

25:     Second, apply learning algorithms to the training data

26:     Evaluate the individual classifier performances p using the testing data

27:     Calculate the average classifier performance

28:  Evaluate the average classifier entropy performance H(x)

29:    end for

30:   end for

31:  end for

32: end procedure

4.2 Regression model for predicting the number of defects

A regression model was constructed for the purpose of estimating the possible number of defects in a new version of software. The prediction outcomes were then compared with the actual numbers of defects in the current versions.

In the above multiple regression equation, which contains two independent variables, X₁ and X₂, these two variables are treated as being independent of each other. However, multicollinearity will arise if X₁ and X₂ are correlated. In the above expression, Y represents the dependent variable and, as such, depends on X₁ and X₂. This indicates that X₁ and X₂ both contain relevant information about Y. If X₁ and X₂ are collinear, this situation can lead to redundancy, which will hinder the ability to generate accurate prediction outcomes. In such a situation, highly correlated metrics should be eliminated. When collinear variables are applied in a prediction model, the correlation among these supposedly independent variables tends to reduce the accuracy of the model [54]. To avoid the potential influence of collinearity on the prediction accuracy, we considered a simple regression model, which will accurately reflect the performance of the independent variable. B₀ represents the intercept, while B₁ and B₂ are the coefficients of the independent variables X₁ and X₂. ε represents the standard error of our prediction.

We constructed a regression model with only one independent variable X₁, i.e., the defect velocity, because it shows a firm but positive impact on the number of defects in terms of correlation. Based on an ANOVA of the defect velocity at the method level, the intercept and coefficient for the defect velocity were found to be approximately -96 and 17.7, respectively. These values were used to construct the regression equations. The regression equation below was applied to estimate the number of defects based on the above values: (27) The resulting regression equation for estimating the number of defects at the method level is (28)

Example 1. Suppose that a new version of the Unicore project, similar to an existing version, is to be developed. It is possible to estimate the number of defects in the new project at the method level using this regression equation. For instance, when developing a Unicore1.n version that is similar to the existing Unicore1.3 project, information on the existing project, such as the rate at which defects occurred in the current project (i.e., the defect velocity), can be helpful for predicting the defect characteristics of the future project. The Unicore1.3 project was characterized by a defect velocity of 6.50 defects per day. Therefore, the regression equation for estimating the possible number of defects in Unicore1.n at the method level is (29) This means that the predicted number of defects at the method level in the Unicore1.n version is 19 plus the standard error ε. For comparison, the actual number of defects in the Unicore1.3 project is 21. The reader is referred to Table 3 for details.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Correlation coefficients between the predictor variables and the number of defects for the ELFF datasets at the method level.

https://doi.org/10.1371/journal.pone.0229131.t003

Example 2. Suppose that a new version of the Jppf5.1 project, similar to an existing version, is to be developed. It is possible to estimate the number of defects in the new project at the method level using this regression equation. For instance, when developing an Jppf5.n version that is similar to the Jppf5.1 project, information such as the defect velocity of the existing project can again be helpful for predicting the defect characteristics of the future version. The current version of the Jppf5.1 project was characterized by a defect velocity of 6.20 defects per day. Therefore, the resulting regression equation for estimating the possible number of defects in Jppf5.n at the method level is (30) This means that the predicted number of method-level defects in the Jppf5.1 version is 14 plus the standard error ε. The numbers of defects predicted as described above are compared with the actual numbers of defects in the results section.

Example 3. Suppose that a new version of the Jitterbit1.1 project, similar to an existing version, is to be developed. It is possible to estimate the number of defects in the new version at the method level using this regression equation. For instance, when developing a Jitterbit1.n version that is similar to the Jitterbit1.1 project, information such as the defect velocity of the existing project can be helpful for predicting the defect characteristics of the future project. The Jitterbit1.1 project was characterized by a defect velocity of 6.60 defects per day. Therefore, the resulting regression equation for estimating the possible number of defects in Jitterbit1.n at the method level is (31) This means that the predicted number of method-level defects in the Jitterbit1.n version is approximately 21 plus the standard error ε; by comparison, the number of defects in the current version is 22.