2.1 Just-in-Time software defect prediction

SDP closely follows the project release schedule, based on code snapshots and defects found in previous releases. Therefore capable of predicting defect density and defect proneness at multiple prediction levels (modules, classes, changes) [17]. An alternative to prior prediction levels of SDP involves using software change histories to predict potential defects at the time of making changes to the repository (“just-in-time”), or JIT-SDP. The majority of software changes correspond to commits made to a Source Code Management (SCM) system or Version Control System (VCS). Fig 1 shows the overall JIT-SDP workflow. The workflow begins with identifying and assembling data sources that drive the model building process. The data sources include software code changes, issue reports, commit messages, and others from VCS and issue tracking system (ITS). The next step converts the raw data into feature vectors of software metrics. A feature vector obtained from software metrics by using metric filtering of collinearity analyses. Filtering analysis here is regarded as part of data pre-processing. In the following classification stage, model training and optimization take place in order to construct the prediction model. Following the construction of the prediction model, the model’s capabilities for predicting software defects will be evaluated. In recent years, many studies have focused on the application of deep learning techniques to JIT-SDP modelling [18, 19]. Yang et al. [20] proposed an approach called Deeper, which leveraged deep belief network and logistic regression classifier to predict defect-prone changes. Yang et al. [15] present various hybrid deep learning as ensemble models to see if it will improve the performance of JIT-SDP. In the study of Hoang et al. [21] utilize CNN to extract features from commit messages and code changes. Qiao and Wang [22] utilized neural network to select useful features for effort-aware JIT-SDP. Li et al. [23] leveraged an ensemble model of decision trees named EATT by using a greedy strategy to rank changes based on effort awareness. Similarly, Zhu et al. [24] introduced denoising autoencoder for features representation into CNN to construct the basic change features into the abstract deep semantic features. Motivated from these studies that conducted experiments on deep learning approaches is on the rise, in this work, we focus modelling of JIT-SDP using deep neural network algorithm.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Workflow of JIT-SDP process.

https://doi.org/10.1371/journal.pone.0299585.g001

2.2 Resampling approaches

Data from code changes often have class imbalance issues. Particularly, the difference between defect-inducing changesets and clean changesets has substantial implications for software projects. A typical distribution of software changes in a software project follows the Pareto Principle, which states that 80% of defects originate from by 20% of the code [24]. Accordingly, the ratio aligns with the observation that software defects in a project have skewed distribution, with a relatively small number of files or modules containing a significant number of defects. The class imbalance problem remains well-recognized as one of the major causes of the poor performance of defect prediction models [25, 26]. Random under-sampling (RUS) is the simplest and most common approach for undersampling in imbalance defect datasets [2, 14]. RUS approach randomly removes the majority class instances to match number of minority instances. Liu et al. [27] introduced an under-sampling approach based on sequential evaluation to guide the sampling process for subsequent classifiers. For each iteration, the majority of instances classified correctly by the current iteration will be excluded from consideration in the subsequent iteration. In the context of oversampling, Chawla et al. [28] proposed Synthetic minority over-sampling technique (SMOTE) as improvement technique of standard random oversampling. SMOTE randomly creates new samples between several instances within a defined neighborhood. He et al. [6] present ADASYN to assign weights to the minority classes and dynamically adjust the weights in a bid to reduce the bias in the imbalanced dataset. ADASYN incorporates a density distribution in automatically deciding the number of synthetic samples needed for each minority class sample. Barua et al. [7] proposed MWMOTE to divide positive samples into safety data, boundary data, and potential noise data, and then adopt different sampling strategies for different types of samples. MWMOTE adaptively assigns the weights to the selected samples according to their importance in learning. Cabral et al. [29] recently proposed oversampling rate boosting (ORB) to adjust the resampling rate over time rather than always using 1:1 ratio of the balanced defect dataset. ORB automatically readjusts the resampling rate that evolve throughout time according to the ratio of current instances class distribution. For the research community in class imbalance classification, SMOTE serves as a basis for oversampling. Numerous extensions and alternatives have been proposed since SMOTE was released in order to improve its performance in different situations [30]. Despite the fact that greater predictive impact for resampling on the minority class than on the majority class, most of the recent JIT-SDP works pre-processed the imbalance dataset by an under-sampling approach. The preference results from the fact that under-sampling requires shorter training time and a simpler process than oversampling [27]. Consequently, prior JIT-SDP research typically used under-sampling rather than oversampling [17]. For oversampling, JIT-SDP poses a challenge compared to other defect prediction especially in term concept drift which led to nonlinear data distribution [29, 31]. Addressing the challenges of oversampling in JIT-SDP requires the development of specialized techniques that account for the nonlinear data distribution, and complex interactions among software metrics.

3.1 Overview of KCO

Using kernel analysis with spectral clustering and crossover interpolation as a combination method of oversampling, this study recommends Kernel Crossover Oversampling (KCO). The proposed algorithm consists of three phases designed to generate synthetic samples that possess both distinctive and common features, as illustrated in Fig 2. The first phase adopts KPCA to segregate the measurements for the minority samples. In this process, KPCA transforms the original dataset into a simpler dimension dataset to analyze the occupied space in the data distribution. In the second phase, spectral clustering divides the transformed data into several clusters. We then evaluate the fitness of each cluster based on the overlapped spatial distribution. By using the crossover operator of as in genetic algorithms, new samples are continuously synthesized to complete the oversampling of defect instances in the last phase. The newly generated samples combine with the initial data to produce a balanced dataset for training the JIT-SDP model. Algorithm 1 illustrates the full process. During Phase 1, steps 1 to 6 are described, then in Phase 2, steps 7 to 11, and finally, steps 12 to 22 for the last stage. To facilitate replication, we publish the source code of KCO at https://github.com/amuhaimin24/KCO—JIT-SDP. The following sections describe each phase in more detail.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Workflow of KCO.

https://doi.org/10.1371/journal.pone.0299585.g002

Algorithm 1 Pseudo Code of Kernel clustering oversampling (KCO)

Input: Dataset of majority and minority class samples N; desired balanced proportion Pfp

Output: Balanced dataset at a set Pfp value

Procedure Begin

 1) Split dataset N into majority class N_maj and minority class N_min

 2) Compute the number of additional minority class to be generated T to attain Pfp

 3) X_new: array for generated samples, initialized to 0

 4) X_newchk: keeps count of the number of synthetic samples generated

 5) Compute Kernel function of PCA for dataset N, KPCA = KernelPCA(n_components = 2, kernel = ’rbf’) where n_components = dimension of data and RBF = radial basis function

 6) Transform dataset N into KPCA, X_tranformed

 7) Create partitions of dataset X_tranformed using Spectral clustering technique, cluster = {i…10}

 8) For each cluster_i, sequentially compute spatial distribution fitness F(cluster_i) = N_maj / (N_maj + N_min)

 9) End for

 10) Rank clusters according to spatial distribution fitness in increasing order

 11) Cluster_best: Select top three clusters

 12) While length of X_newchk ≤ size of N_min

 13) Select samples parent_a, parent_b from Cluster_best, where parent_a and parent_b are not equal

 14) Generate a minority class synthetic sample X_i where X_i = average(parent_a, parent_b)

 15) Add X_i to X_new and increase X_newchk (i): X_newchk = X_newchk (i) + 1

 16)  End while

 17)  While length X_newchk ≤ T

 18) Select samples parent_a, parent_b from Cluster_best and X_new respectively, where parent_a and parent_b are not equal

 19) Generate a minority class synthetic sample, where X_i =

 20) Add X_i to X_new and increase X_newchk (i): X_newchk = X_newchk (i) + 1

 21)  End while

 22)  Add X_new to dataset N

 23)  Return N

3.2 Diversity measurement

Euclidean distance fails to be effective in nonlinear distributions [32] as presented in JIT-SDP datasets. JIT-SDP data typically exhibit a nonlinear distribution as a result of the uncorrelated relationship between software metrics. Several factors may affect the distribution, including clusters, non-convex shapes, or overlapping regions that are not accurately represented using a linear distance measure. The relationship between data points is unable to be well-represented by a straight line calculated by Euclidian distance. Therefore, the measure does not accurately reflect the diversity of data points. Moreover, the JIT-SDP datasets contain noise or duplicates as a result of the collection process for the metrics [2]. Accordingly, the Euclidean distance measure unable to identify highly correlated or duplicated data samples within nonlinear distribution which failed to provide meaningful during information classifier training. As an alternative to handle highly correlated data, one may utilize feature engineering techniques such as Principal Component Analysis (PCA) [33]. PCA learns the original feature combinations linearly in new dimensional spaces. Nevertheless, PCA assumes that the learning data follow a linear separable Gaussian distribution. For real world data, particularly code changesets, linearly separated data is impractical due to the nonlinear structures of software metrics.

Prior studies have indicated that KPCA perform better than PCA for software engineering tasks [34]. Researchers have investigated the use of KPCA in software defect prediction, especially for the selection of features. Xu et al. [35] found that basic classifiers including KCPA as a feature selection method achieve promising performance when compared to 41 baseline methods. Experimental results indicate that the framework outperforms PROMISE and NASA datasets, particularly in terms of F-measure, MCC, and AUC. Ho et al. [36] utilized KPCA to reduce the dimensions of defect feature spaces from software metrics in order to extract essential information. A deep neural network (DNN) is then built to emphasize the semantic relations between software metrics so that defect data are distinguished from non-defect data using newly generated features from KPCA. Azzeh et al. [37] examine the performance of nonlinear kernel functions and linear kernel functions in the context of different experimental parameters such as the granularity of the data, the imbalance ratio of the dataset, and feature subsets. According to their findings, RBF is the only kernel function that exceeds linear and other nonlinear kernel functions. Nonetheless, reducing the dimensionality of a dataset did not often improve the accuracy of software defects prediction [38, 39]. Therefore, the KPCA should not be limited to measuring the similarity between features in software metrics. In other aspects of JIT-SDP, KCPA presents a promising alternative. As a result of KCPA, patterns in the data are identified that are not apparent by traditional methods of data representation, including handling high-dimensional datasets and capturing non-linear relationships among features. Therefore, the analysis of data distribution can be particularly important for data resampling.

This study employs KPCA to map multivariate of software metrics into a linear projection using a nonlinear kernel function. The process of data projection involves transforming the original data into lower dimension data. Data transformation process converts multivariate data into a new set of uncorrelated variables. Enabling efficient multidimensional scaling of JIT-SDP datasets with varying software metrics. In this way, the diversity analysis of JIT-SDP datasets by KPCA is independent of the data dimensions and becomes a scale-independent measurement. Therefore, the complex structure becomes easier to manage and allows the representation of features to be projected in a linear manner. Using a Radial Basis Function (RBF) kernel, KPCA provides a linear representation of the data while preserving the relative distances between pairs of data points that are close to the original space.

Algorithm 2 Pseudo Code of KPCA

Input: Dataset N x D; Number of principal components: k

Output: Projected_data with an N x k matrix representing the projected data onto the selected principal components

Procedure Begin

 1) Compute Kernel Matrix N x N, K

 2) For i = 1 to N do

 3) For j = 1 to N do

 4)  K[i, j] = RBF(Data[i], Data[j])

 5) Center the Kernel Matrix

 6) for i = 1 to N do

 7)  for j = 1 to N do

 8)   K[i, j] = K[i, j] - mean_rows[i] - mean_cols[j] + mean_total

 9) Compute the Eigenvectors and Eigenvalues of the centered kernel matrix K

 10) Sort the eigenvalues in descending order and select the top k eigenvalues and corresponding eigenvectors

 11) Project Data onto Principal Components

 12) for i = 1 to N do

 13) for j = 1 to k do

 14)  Projected_Data[i, j] = dot_product(alpha[j], K[i, :])

 15) Return Projected_Data

3.3 Data partitioning

KCO relies on spectral clustering to simplify multidimensional nonlinear datasets while reducing them into clusters of data exhibiting similar characteristics on lower dimensions. Spectral clustering treats the data clustering problem as a graph partitioning problem without making any assumptions about the shape of the clusters. Fig 3 illustrates spectral clustering in sample distributions in the JIT-SDP dataset.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Spectral clustering within transformed data.

https://doi.org/10.1371/journal.pone.0299585.g003

The basic premise of spectral clustering in defect datasets is as follows: For a dataset with n samples D = {x₁, x₂, …, x_n} and each sample consist of variables x_i = {v₁,v₂,..,v_m}, where m is the number of software metrics. The clustering is based on dividing each sample into k clusters C = {C₁, C₂,.., C_k}. As a result, the samples in the clusters have a variance that is: (1) Where, μ_i is the mean value of the samples in C_i.

Fitness calculation is made for each cluster based on the number of samples taken from both the minority and the majority classes. The intuition behind fitness assessment for clusters is that regions with a lower proportion of majority samples should have fewer overlapped spatial distributions. The following formula calculates the fitness weight of each cluster: (2) A fitness evaluation is conducted for each cluster, and the three best clusters are chosen. Clusters selected for interpolation have a greater proportion of empty spaces, indicating that there are more areas that can be utilized for new samples. Using selected clusters, a pool of suitable templates for oversampling is established.

3.4 Crossover-interpolation

Interpolation in oversampling generates synthetic samples from existing minority class samples. One of the earliest methods for oversampling was the SMOTE, introduced by Chawla et al. [4]. SMOTE uses interpolation to generate synthetic samples from existing minority class samples. Even so, the use of SMOTE to develop prediction models may still result in overgeneralization as it relies solely on the selection of nearest neighbor instances. Due to the limitations of SMOTE, a variety of modifications have been proposed, including Borderline-SMOTE [13] and MWMOTE [7]. Nevertheless, prior techniques unable to provide a diverse and balanced set of synthetic samples from datasets with high-dimensional input features. Cross-over interpolation provides an alternative way to generate synthetic samples by combining or "crossing over" the features of two existing minority class samples. Consequently, the generation new samples exhibit more representative and diverse to better reflect minority class distributions. In SDP, Bennin et al. [11] first to propose crossover interpolation into oversampling process which named as MAHAKIL. They adopted Mahalanobis distance to rank instances and divide them into two groups. During the generation of new instances, two corresponding instances are chosen from each group. Synthetic instances tend to be more diverse when pairs of selected instances do not have a close distance between them. In comparison to SMOTE-based oversampling techniques, MAHAKIL offers superior performance and greater stability. Even though MAHAKIL intends to improve the diversity of data, it unable to detect defects consistently, thereby reducing its value. Particularly, MAHAKIL ineffective to calculate the Mahalanobis distance when the number of minority class instances is smaller than the dimensionality of these instances. Thus, MAHAKIL unable to function optimally when the number of minority class instances is less than the number of metrics. Zhang et al. [12] extended the work of Bennin et al. [11] by adding K-means clustering to MAHAKIL in order to improve the recognition rate of positive samples. By using K-means, they divide positive samples into K-clusters and crossover interpolate synthetic instances.

This study uses the crossover operator to generate new samples in the same manner as genetic algorithm. In this process, chromosome information contributes by two parents to generate a child. Chromosome information defined in this study as software metrics for JIT-SDP modeling purposes. In order to generate new samples, crossover operators combine the characteristics of two samples. Given two samples of = [a₁,…, a_l] and = [b₁,…, b_l] are two chromosomes crossed in gth generation and l is the length of chromosome or features, the child sample of g + 1 th generation is: (3) Where λ provides a random variable between a range of [0,1].

During the experiment, λ is set to 0.5 for generating the child samples. The setting means that the child samples inherit 50 percent of their characteristics from each of their parent samples. Fig 4 illustrates an example of crossover operation during the generation of a new sample.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Example of multi-point crossover.

https://doi.org/10.1371/journal.pone.0299585.g004

Fig 5 illustrates the generation of new synthetic samples based on the level of inheritance. First, based on diversity measurements obtained from KPCA, the grandparent samples are identified, G₀. The samples from G₀ are then used to generate the G₁ set of new synthetic samples. To prevent new samples from entering the region of the majority class, the first parent node or grandparent act as a boundary such that all children produced in the future reside within the range of the parents. In the second generation G₂, samples from grandparent and samples from G₁ are selected as template to generate new samples. In case of the interpolation at current generation is still not meet with the maximum samples, the process continues to crossover interpolate the samples within the previous generation until maximum number reach. The process of pairing the child nodes with older generations is repeated until the generated samples are sufficient (greater than or equal to the required number of samples). The pairing process is carried out using the sequential information inherited from the immediate parents of the instances beginning at G₁.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Crossover interpolation across generations.

https://doi.org/10.1371/journal.pone.0299585.g005

4.1 Benchmark datasets

We evaluate six imbalanced software project datasets which comprise Bugzilla, Columba, Eclipse.JDT (JDT), Eclipse.Platform (Platform), Mozilla, and PostgreSQL (Postgres). Note that all the datasets are imbalanced. The most imbalanced dataset, Mozilla, contains only 5% defects, while the most balanced dataset, Bugzilla, contains 36% defects. To ease the analysis of prediction results, these datasets are classified into two severity groups as shown in Table 1. Mild imbalanced class refers to datasets that contain more than 25% defects. High imbalanced datasets are based on datasets with fewer than 25% defects. The severity of the imbalance class represents the difficulty for data resampling in the imbalance distribution.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Imbalanced class datasets.

https://doi.org/10.1371/journal.pone.0299585.t001

4.2 Evaluation measures

The evaluation measure is important to reveal the performance of the classifier, especially for imbalanced datasets. Some conventional measures lead to a wrong conclusion owing to the skewness of the class distribution [10]. For example, consider an extremely imbalanced dataset: 99% of instances are of the majority class, and the remaining 1% samples belong to the minority class. In case of using the accuracy measure which indicates how many test samples are correctly classified as the evaluation criterion, even if the classifier ignores all of the minority classes, it still reaches a very high accuracy rate of 99%. Therefore, this experiment also considered F1-score, which is a commonly used measure to evaluate classification performance. F1-score combines Precision and Recall from a confusion matrix. The confusion matrix lists all four possible prediction results. If an instance is correctly classified as a defect, it is a true positive (TP); if an instance is misclassified as a defect, it is a false positive (FP). Similarly, for false negatives (FN) and true negatives (TN). Based on the four numbers, Precision, Recall, and F1-score are calculated. Precision is the ratio of correctly predicted defect instances to all instances predicted as defects (. Recall is the ratio of the number of correctly predicted defect instances to the actual number of defect instances (. Finally, F1-score is a harmonic mean of Precision and Recall, .

4.3 Experimental design

The main goal of the research is to propose a resampling technique named KCO on JIT-SDP datasets. To that end, we conducted experimental investigations based on three research questions in the following order.

1. RQ1: Does KCO contribute to the diversity of the datasets?

To evaluate whether KCO improves the diversity of the data distribution, we conduct an analysis to compare the distribution of the resampled data by oversampling when applied separately to oversample datasets. The analysis considers sparsity of data distribution where the larger sparsity data contain less information across data classes. Oversampling in training datasets reduces the impact of noise and improves sparsity of data distribution toward dense data. Dense data yields more informative data, which results in more accurate predictions due to more data available for model training. By measuring the percentage of non-zero values in the data, sparsity of the data distribution is analyzed using data density measurement. The analysis provides a simple and straightforward way to quantify the sparsity of the data. In measuring the diversity of data for resampled datasets (d), sparsity formulation is utilized. .

1. RQ2: Does KCO improve the diversity within the data distribution at the expense of its ability to provide accurate predictions?

To validate the effectiveness of oversampling providing diverse data distribution, we compare the proposed technique with baseline techniques. Evaluation of accuracy prediction is conducted based on 10-folds stratified within project validation. The validation started with the splitting of data into 8:2 ratio, for both training and prediction datasets. Then, the training dataset undergoes 10-fold stratified within project validation. The datasets are divided randomly into 8-folds, 2-folds serve as training data, and the remaining fold serves as test data. In stratified cross validation, each fold is used as a testing dataset only once. Additionally, the data are folded so that every fold consists of the same proportions as the original dataset. the data need to be folded in such a way that each fold consists of the same proportions as the original dataset. The average result is recorded using StratifiedKFold to strengthen the reliability of the experiment outcomes.

1. RQ3: How does KCO addresses class imbalance problem under different imbalance severity?

To evaluate the performance of oversampling more accurately, the effectiveness of oversampling should be evaluated in the context of the different validation settings. We conducted cross-project prediction and timewise prediction. In cross project prediction, the prediction of software defects is evaluated across different software projects. Specifically, the models are constructed by one source of software project and use these models to predict software defects on another target software project. For a set of n projects, this method produces n * (n—1) prediction effectiveness values. In timewise prediction, we evaluated JIT-SDP within the same project datasets, in which the chronological order of changes based on the commit date is considered. Assuming the changes are divided into n parts, we first construct the models based on the changes from part i and i + 1. Then we use the constructed models to predict the changes from part i + 4 and i + 5. For each fold, it consists of varies in the size of data and imbalance ratio.