Editing training data for multi-label classification with the k-nearest neighbor rule

Abstract

Multi-label classification allows instances to belong to several classes at once. It has received significant attention in machine learning and has found many real-world applications in recent years, such as text categorization, automatic video annotation and functional genomics, resulting in the development of many multi-label classification methods. Based on labeled examples in the training dataset, a multi-labeled method extracts inherent information to output a function that predicts the labels of unlabeled data. Due to several problems, like errors in the input vectors or in their labels, this information may be wrong and might lead the multi-label algorithm to fail. In this paper, we propose a simple algorithm for overcoming these problems by editing the existing training dataset, and adapting the edited set with different multi-label classification methods. Evaluation on benchmark datasets demonstrates the usefulness and effectiveness of our approach.

Appendices

Appendix 1: Evaluation measures

As discussed in Sect. 3.2, performance evaluation for multi-label learning systems differs from that of single-label classification. Let $\mathcal {H}:\mathbb {X}\rightarrow 2^\mathcal {Y}$ be a multi-label classifier that assigns a predicted label subset of $\mathcal {Y}=\{\omega _1,\ldots ,\omega _{Q}\}$ to each instance $\mathbf {x}\in \mathbb {X}$, and let $f:\mathbb {X}\times \mathcal {Y}\rightarrow [0,1]$ be the corresponding scoring function which gives a score for each label $\omega _q$ which in turn is interpreted as the probability that $\omega _q$ is relevant. The function $f(.,.)$ can be transformed to a ranking function ${\rm{rank}}_f(.,.)$ which maps the outputs of $f(\mathbf {x},\omega )$ for any $\omega \in \mathcal {Y}$ to $\{\omega _1,\omega _2,\ldots ,\omega _Q\}$ so that $f(\mathbf {x}_i,\omega _q)>f(\mathbf {x}_i,\omega _r)$ implies that ${\rm{rank}}_f(\mathbf {x}_i,\omega _q)<{\rm{rank}}_f(\mathbf {x}_i,\omega _r)$.

Given a set $\mathcal {S}=\{(\mathbf {x}_1,Y_1),\ldots ,(\mathbf {x}_m,Y_m)\}$ of $m$ test examples, the evaluation metrics of multi-label learning systems are divided into two groups: prediction-based and ranking-based metrics. Prediction-based measures are calculated based on the average difference of the actual and the predicted set of labels over all test examples. Ranking-based metrics evaluate the label ranking quality depending on the scoring function $f(.,.)$.

1.1 Prediction-based measures

Hamming loss: The Hamming loss metric for the set of labels is defined as the fraction of labels whose relevance is incorrectly predicted:

$$\begin{aligned} \mathcal H{\rm{Loss}}(\mathcal {H},\mathcal {S})=\frac{1}{m}\sum _{i=1}^m \frac{|Y_i\triangle \widehat{Y}_i|}{Q}, \end{aligned}$$

(7)

where $\triangle$ denotes the symmetric difference between two sets.

Accuracy: The accuracy metric gives an average degree of similarity between the predicted and the ground truth label sets:

$$\begin{aligned} \mathcal A{\rm{ccuracy}}(\mathcal {H},\mathcal {S})=\frac{1}{m}\sum _{i=1}^m \frac{|Y_i\cap \widehat{Y}_i|}{|Y_i\cup \widehat{Y}_i|}. \end{aligned}$$

(8)

Precision: The precision metric computes the proportion of true positive predictions:

$$\begin{aligned} \mathcal P{\rm{recision}}(\mathcal {H},\mathcal {S})=\frac{1}{m}\sum _{i=1}^m \frac{|Y_i \cap \widehat{Y}_i|}{|\widehat{Y}_i|}. \end{aligned}$$

(9)

Recall: This metric estimates the proportion of true labels that have been predicted as positives:

$$\begin{aligned} \mathcal R{\rm{ecall}}(\mathcal {H},\mathcal {S})=\frac{1}{m}\sum _{i=1}^m \frac{|Y_i \cap \widehat{Y}_i|}{|Y_i|}. \end{aligned}$$

(10)

F1-measure: F1 measure is defined as the harmonic mean of precision and recall. It is calculated as:

$$\begin{aligned} \mathcal F1(\mathcal {H},\mathcal {S})=\frac{1}{m}\sum _{i=1}^m \frac{2|Y_i\cap \widehat{Y}_i|}{|Y_i| + |\widehat{Y}_i|}. \end{aligned}$$

(11)

Note that the smaller the value of the Hamming loss, the better the performance. For the other metrics, higher values correspond to better classification quality.

1.2 Ranking-based measures

One-Error: This metric computes how many times the top-ranked label is not in the true set of labels of the instance, and it ignores the relevancy of all other labels.

$$\begin{aligned} {\rm{OErr}}(f,\mathcal {S})=\frac{1}{m}\sum _{i=1}^m \left\langle [\underset{\omega \in Y}{\arg \max }~f(\mathbf {x}_i,\omega )]\notin Y_i\right\rangle , \end{aligned}$$

(12)

where for any proposition $H$, $\langle H \rangle$ equals to 1 if $H$ holds and 0 otherwise. Note that, for single-label classification problems, the One-Error is identical to ordinary classification error.

Coverage: Coverage computes the average of how far we need to move down the ranked label list to cover all the labels assigned to a test instance.

$$\begin{aligned} \mathcal C{\rm{ov}}(f,\mathcal {S}) = \frac{1}{m}\sum _{i=1}^m \max _{\omega \in Y_i} {\rm{rank}}_f(\mathbf {x}_i,\omega )-1. \end{aligned}$$

(13)

Ranking loss: This metric computes the number of times that an incorrect label is ranked higher than a correct label.

$$\begin{aligned} {\mathcal {R}}{\rm{Loss}}(f,\mathcal {S}) = \frac{1}{m}\sum _{i=1}^m \frac{1}{|Y_i||\overline{Y}_i|}|{(\omega _q,\omega _r)\in Y_i\times \overline{Y}_i\backslash f(\mathbf {x}_i,\omega _q)\le f(\mathbf {x}_i,\omega _r)}| \end{aligned}$$

(14)

where $\overline{Y}_i$ is the complementary set of $Y_i$ in $\mathcal {Y}$.

Average precision: This metric evaluates the average fraction of labels ranked above a particular label $\omega \in Y_i$ which are actually in $Y_i$.

$$\begin{aligned} {\mathcal {A}}{\rm{vPrec}}(f,\mathcal {S}) = \frac{1}{m}\sum _{i=1}^m \frac{1}{|Y_i|}\sum _{\omega _q \in Y_i}\frac{|\{\omega _r \in Y_i\}\backslash {\rm{rank}}_f(\mathbf {x}_i,\omega _r)\le {\rm{rank}}_f(\mathbf {x}_i,\omega _q)|}{{\rm{rank}}_f(\mathbf {x}_i,\omega _q)}. \end{aligned}$$

(15)

Note that AvPrec$(f, \mathcal {S}) = 1$ means that the labels are perfectly ranked. For the other metrics, smaller values correspond to a better label ranking quality.

Appendix 2: Multi-labeled dataset statistics

Given a multi-labeled dataset $\mathcal {D}=\{(\mathbf {x}_i,Y_i),i=1,\ldots ,n \}$ with $\mathbf {x}_i \in \mathbb {X}$ and $Y_i \subseteq \mathcal {Y}$, this dataset can be measured by the number of instances ($n$), the number of attributes in the input space, and the number of labels ($Q$). In the following, we review some statistics about the multi-labeled dataset $\mathcal {D}$ [36].

Label cardinality: The label cardinality (LCard) of $\mathcal {D}$ is the average number of labels per instance. Label cardinality is calculated as⁴

$$\begin{aligned} {\rm{LCard}}(\mathcal {D})=\frac{1}{n}\sum _{i=1}^n |Y_i| \end{aligned}$$

(16)

Label density: The label density (LDen) of $\mathcal {D}$ is defined as the average number of labels per instance divided by the total number of labels $Q$. Label density is calculated as:

$$\begin{aligned} {\rm{LDen}}(\mathcal {D})=\frac{1}{n}\sum _{i=1}^n \frac{|Y_i|}{Q} \end{aligned}$$

(17)

Both metrics indicate the number of alternative labels that characterize the examples of a multi-labeled dataset. Label cardinality is independent of the total number of labels in the classification problem, while label density takes into consideration the total number of labels. Two datasets with the same label cardinality but with different label densities may present different properties that influence the performance of the multi-label classification methods.

Distinct label sets: The distinct label sets (DL) count the number of label sets that are unique across the total number of examples. Distinct label sets are given by:

$$\begin{aligned} {\rm{DL}}(\mathcal {D})=|\{Y_i \subseteq \mathcal {Y}|\exists ~\mathbf {x}_i\in \mathbb {X}~:~(\mathbf {x}_i,Y_i)\in \mathcal {D}\}| \end{aligned}$$

(18)

This measure gives an idea of the regularity of the labeling scheme.