Automatic detection of code smells using metrics and CodeT5 embeddings: a case study in C#

Abstract

Code smells are poorly designed code structures indicating that the code may need to be refactored. Recognizing code smells in practice is complex, and researchers strive to develop automatic code smell detectors. An obstacle to developing these solutions is the datasets’ limitations. Manually labeled datasets were collected to investigate the developers’ perceptions of code smells. They are characterized by a high label disagreement that hurts the performance of Machine Learning (ML) models trained using them. Furthermore, all large, manually labeled datasets are developed for Java. We recently created a novel dataset for C# to alleviate these issues. This paper evaluates ML code smell detection approaches on our novel dataset. We consider two feature representations to train ML models: (1) code metrics and (2) CodeT5 embeddings. This study is the first to consider the CodeT5 state-of-the-art neural source code embedding for code smell detection in C#. To prove the effectiveness of ML, we consider multiple metrics-based heuristics as alternatives. In our experiments, the best-performing approach was the ML classifier trained on code metrics (F-measure of 0.87 for Long Method and 0.91 for Large Class detection). However, the performance improvement over CodeT5 features is negligible if we consider the advantages of automatically inferring features. Finally, our ML model surpassed less experienced annotators and nearly matched the most experienced annotator, suggesting it can assist less experienced developers under tight deadlines. To the best of our knowledge, this is the first study to compare the performance of automatic smell detectors against human performance.

Appendix 1: Performance of heuristic-based detectors

This section presents the performance existing metric-based heuristic detectors achieve on our novel C# dataset. We chose the metric-based heuristics following the same approach as in our paper [10], where we compared multiple heuristic-based detectors on the MLCQ dataset [11]. That is, we started from the list of available code smell detection heuristics neatly summarized in a recent Systematic Literature Review by Bafandeh Mayvan et al. [30] . We selected the heuristics for which we could find metrics definitions that aligned with the implementation of metrics in our platform [44].

Tables 9 and 10 present the metric-based heuristic’s performance on the Large Class and Long Method detection task, respectively. For easy comparison of the heuristic-based detectors on the test set, we summarize the performances from Tables 9 and 10 in Figs. 6 and 7, respectively.

Table 9 Performance of heuristic-based approaches for Large Class detection: P—precision, R—recall, F—F-measure of the minority (smell) class

Table 10 Performance of heuristic-based approaches for Long Method detection: P—precision, R—recall, F—F-measure of the minority (smell) class

Fig. 6

Performance of heuristic-based approaches for Large Class detection on the test set. We sort the performances from left to right according to the F-measure, our primary performance metric

Fig. 7

Performance of heuristic-based approaches for Long Method detection on the test set. We sort the performances from left to right according to the F-measure, our primary performance metric

The application of metric-based heuristics does not require model training. Thus, we report the performance on the training set, test set, and the whole dataset. By comparing performances on the test set and whole dataset, we can assure that the code snippets, randomly chosen for the test set, are not particularly “easy” or “hard” for heuristic-based detectors, compared to the rest of the dataset. We can observe from Tables 9 and 10 that this is not the case—the performance of the heuristic-based detectors on the test set is similar to their performance on the whole dataset.

In a realistic scenario, we would choose the best-performing heuristic according to its performance on the training set (we use the minority F-measure as our primary performance measure). Thus, the best-performing heuristic for Large Class detection is $LC_9$ (Table 9), and the best-performing heuristic for Long Method detection is $LM_3$ (Table 10). We compare these two best-performing heuristics with ML-based detectors in our paper. Interestingly, heuristics $LC_9$ and $LM_3$ proposed by Liu et al. [32] achieved the best performance on both our C# dataset and the MLCQ Java dataset, as we showed in [10].