Abstract Corpus

For the training corpus, Li’s lab selected 1203 pharmacokinetics-related abstracts by searching PubMed using terms from a previously developed ontology for PK pharmacokinetic parameters [38]. Therefore, all retrieved articles describe and contain some form of pharmacokinetic evidence, though not necessarily of DDI. We kept in vitro studies but removed any animal in vivo studies. Abstracts were labeled according to the following criteria: abstracts that reported the presence or absence of drug interaction supported by explicit experimental evidence of pharmacokinetic parameter data were labeled as DDI-relevant (909 abstracts) while the rest were labeled as DDI-irrelevant (294 abstracts). DDI-relevance was established regardless of whether the relevant enzymes were presented or not. Importantly, the concept of DDI-relevance employed here updates the criteria used in a previous preliminary study [39]. Interactions between a drug and food, fruit, smoking, alcohol, and natural products are now classified as drug interactions because their pharmacokinetics studies are designed similarly. For the same reason, studies dealing with interactions between drug metabolites (instead of parent compounds) are now also considered relevant, as well as studies reporting inhibition of induction of a drug on a drug metabolism enzyme or drug transporter. Classification was done by three graduate students with M.S. degrees and one postdoctoral annotator; any inter-annotator conflicts were checked by a Pharm D. and an M.D. scientist with extensive pharmacological training. The corpus is publicly available as “Pharmacokinetics DDI-Relevant Abstracts V0” in [43] (see also [26]).

We extracted textual features from PubMed article title and abstract text fields as well as the following metadata fields: the author names, the journal title, the Medical Subject Heading (MeSH) terms, the ‘registry number/EC number’ (RN) field, and the ‘secondary source’ field (SI) (the latter two fields contain identification codes for relevant chemical and biological substances). For each PubMed entry, the content of the above fields was tokenized, processed by Porter stemming [44], and converted into textual features (unigrams and, in certain runs, bigrams). Strings of numbers were converted into ‘#’, short textual features (with length of less than 2 characters) and infrequent features (that occurred in less than 2 documents) were omitted. Author names, journal titles, substance names, and MeSH terms were treated as single textual tokens.

The corpus was represented as binary term-document occurrence matrices. We evaluated classification performance under two different conditions: in the first—referred to as ‘unigram runs’—only word unigram features were used; in the second—referred to as ‘bigram runs’—word bigram features were used in addition to unigram features. Bigram runs included a much larger number of parameters (i.e. the bigram feature coefficients) that needed to be estimated from training data, which can potentially increase generalization error arising from increased model complexity [45]. Testing the classifiers exclusively with unigram features as well as with both unigram and bigram features evaluated whether the class information provided by bigrams outweighed their cost in complexity.

Sentence Corpus

The evidence sentence task consisted in identifying those sentences within a PubMed abstract that reported experimental evidence for the presence or absence of a specific DDI. For this purpose, Li’s group developed a training corpus of 4600 sentences extracted from 428 PubMed abstracts. All abstracts contained (positive or negative) pharmacokinetic evidence of DDIs. Sentences were manually labeled as DDI-relevant (1396 sentences) if they explicitly mentioned pharmacokinetic evidence for the presence or absence of drug-drug interactions, and as DDI-irrelevant (3204 sentences) otherwise. The same pre-processing and annotation procedures were followed for the sentence corpus as for the abstract corpus (see section “Abstract Corpus”). This corpus is publicly available as “Deep Annotated PK Corpus V1” in [43] (see also [26]).

Classifiers

Six different linear classifiers were tested:

1. VTT: a simplified, angle-domain version of the Variable Trigonometric Threshold Classifier, previously developed in Rocha’s lab [12, 41, 42]. Given a document vector x = <x₁, …, x_K> with features (i.e. dimensions) indexed by i, the separating hyperplane is defined as Here, λ is a threshold (bias) and φ_i is the ‘angle’ of feature i in binary class space: where p_i is the probability of occurrence of feature i in relevant-class documents and n_i is the probability of occurrence of feature i in irrelevant-class documents. The threshold parameter l is chosen so that a neutral ‘pseudo-document’ defined by x_i = (p_i+n_i)/2 falls exactly onto the separating hyperplane.
   The full version of VTT, which includes additional parameters to account for named entity occurrences and which we have previously used in protein-protein interaction classification, is evaluated in combination with various NER tools in section “Impact of NER and PubMed metadata on abstract classification” below. VTT performs best on sparse, positive datasets; for this reason, we do not evaluate it on dense dimensionality-reduced datasets. Notice that in previous work, we used a different version of VTT with a cross-validated threshold parameter; its performance on the tasks was very similar, and is reported in the Supporting Information as the ‘VTTcv’ classifier (section 1 and 2 in S1 Text).
2. SVM: a linear Support Vector Machine with a cross-validated regularization parameter (implemented using the sklearn [46] library’s interface to the LIBLINEAR package [47]).
3. Logistic regression classifier with a cross-validated regularization parameter (also implemented using sklearn’s interface to LIBLINEAR).
4. Naive Bayes classifier with smoothing provided by a Beta-distributed prior with a cross-validated concentration parameter.
5. LDA: a regularized Linear Discriminant Analysis classifier, following [48]. Singular value decomposition (SVD), a dimensionality reduction technique, is first used to reduce any rank-deficiency, after which the covariance matrix is shrunk toward a diagonal, equal-variance structured estimate. The shrinkage parameter is determined by cross-validation.
6. dLDA: a ‘diagonal’ LDA, where only the diagonal entries of the covariance matrix are estimated and the off-diagonal entries are set to 0. A cross-validated parameter determines shrinkage toward a diagonal, equal-variance estimate. This classifier can offer a more robust estimate of feature variances; it is equivalent to a Naive Bayes classifier with Gaussian features [49].

Generally, linear classifiers fall into one of two types. Classifiers of the first type—sometimes called ‘naive’ in the literature, which in our case include VTT, dLDA, and Naive Bayes—learn feature weights without considering feature covariances. While covariance information can be useful for distinguishing classes, naive classifiers often perform well with small amounts of training data, when covariances are difficult to estimate accurately. Classifiers of the second type—which we refer to as ‘non-naive’, and which in our case included SVM, LDA, and Logistic Regression—do consider feature covariances (often in combination with regularization techniques to smooth covariance estimates) and can achieve superior performance when provided with sufficient training data.

Feature Transforms

For both unigram and bigram runs, we evaluated classification performance on several transforms of the document matrices:

1. No transform: raw binary occurrence matrices (see section “Abstract Corpus”).
2. IDF: occurrences of feature i were transformed to its Inverse Document Frequency (IDF) value: $\mathrm{\text{idf}}\left(i\right)=\log \frac{N}{c_i+1}$, where c_i is the total number of occurrences of feature i among all documents. This reduced the influence of common features on classification.
3. TFIDF: the Term Frequency, Inverse Document Frequency transform (TFIDF); same as above, but subsequently divided by the total number of features that occur in each document. This reduced the impact of document size differences.
4. Normalization: the non-transformed, IDF, and TFIDF document matrices underwent a length-normalization transform, where each document vector was inversely scaled by its L2 norm. L2 normalization has been argued to be important for good SVM performance [50].
5. PCA: The above matrices were run through a Principal Component Analysis (PCA) dimensionality reduction step. Projections onto the first 100, 200, 400, 600, 800, and 1000 components were tested.

Feature transforms can improve classification performance by making the surfaces that separate documents in different classes more linear and by decreasing the weight of non-discriminating features. PCA, on the other hand, reduces the number of parameters that need to be estimated from training data. If class membership information is contained in the subspace spanned by the largest principal components, then this kind of dimensionality reduction can improve generalization performance by reducing noise and model complexity.

Performance evaluation

The abstract and sentence corpora described above were used both for training classifiers and for estimating generalization performance on out-of-sample documents. In order to estimate out-of-sample performance, we used the following cross-validation procedure for each possible classifier and feature transform:

1. Each corpus was randomly partitioned into 4 document folds (75%–25% splits). This was repeated 4 times, yielding 16 outer folds. All classifiers and transforms were evaluated using the same partitions.
2. For each fold, the 75% split was treated as the ‘training’ split and the 25% split was treated as the ‘testing split’. If a feature transform was used, it was applied to both splits but was computed using statistics (such as IDF or principal components) from the training split. Finally, classifiers were trained on the training split and evaluated based on their prediction performance on the testing split.
3. Measures of classification performance (see below) on the testing split were collected. The 16 sets of performance measures were averaged to produce an estimate of generalization performance.

Because training and testing documents are always separated, for each cross-validation fold the above procedure is equivalent to calculating performance on an independent testing corpus.

Except for VTT, the classifiers listed in section “Classifiers” used cross-validated regularization parameters. These parameters were not chosen using cross-validation on the outer folds because this would lead to a biased estimate of out-of-sample performance. Instead, regularization parameters were chosen using nested cross-validation within each of the 75% blocks of the above outer folds:

1. The 75%-block was itself partitioned into 4 folds (75%–25% splits of the outer 75% block). This is repeated 4 times, producing a total of 16 inner folds for each outer fold training split.
2. Over a range of values of the cross-validated parameter, the procedure described in step 2 above was used, but now applied to the 75%/25% splits of each inner fold. Mean performance on inner fold testing splits were measured using the Matthews Correlation Coefficient [51] (MCC), which is particularly well-suited for the unbalanced scenarios of our corpora [52].
3. The parameter value giving the highest mean MCC was chosen as the regularization parameter value for training the classifier in the outer fold.

We evaluated the performance of the classifiers using three different measures: the balanced F1 score (the harmonic mean of precision and recall), the iAUC or ‘area under the interpolated precision/recall curve’ [53], and the MCC. In addition, we computed and reported the rank product of these three measures (RP3) as a single inclusive metric of classification performance. The RP3 measure provides a well-rounded assessment of classifier performance, as it combines the ranking of the different individual measures [12, 41].

For displaying results, we focus primarily on the iAUC measure (in cases where only plots of iAUC performance are provided, F1 and MCC plots are found in the Supporting Information, S1 Text). iAUC does not depend on predicted class assignments but rather on the ranking of test set documents according to classifier confidence scores from most relevant to most irrelevant. iAUC offers three major advantages as a measure of classification performance. First, it provides a more comprehensive measure of classifier performance because it evaluates the entire ranking of documents, as opposed to just class assignments. Second, iAUC is less sensitive to variation driven by random-sampling differences in the training corpus, which may lead to fluctuations in the class assignments of low confidence documents and, correspondingly, high variability in measures such as F1 and MCC. Finally, it is more relevant in a frequently-encountered situation where a human practitioner uses a BLM pipeline to retrieve only the most relevant documents (which should have high positive-class confidence scores) or to identify likely-to-be-misclassified documents (which should have low confidence scores).

Both the abstract and sentence classification tasks are characterized by imbalanced datasets, with more relevant-class abstracts and more irrelevant-class sentences respectively. For simplicity, and because we are primarily concerned how ranking performance (as measured by iAUC) changes between different machine learning configurations on the same dataset, we do not perform resampling or re-weighting of training items. We also report MCC values, a measure which is known to be stable in the face of unbalanced classes [52].

The performance of a classifier and feature transform configuration varies both due to random sampling of folds and due to the inherent performance bias of the configuration over the entire distribution of folds. Since we are only interested in the latter, observed performance differences between pairs of configurations were tested for statistical significance using a non-parametric paired-sample permutation test. First, the assignments of performance scores for each of the 16 outer folds were permuted between the two classifier/transform configurations under consideration. For each of the 2¹⁶ possible permutations, the difference in across-fold mean performance was calculated; this formed the distribution of performance differences under the null hypothesis that the two configurations have equal performance. Finally, the p-value was computed as the probability (one- or two-tailed, as indicated) of observing a difference under the null hypothesis distribution equal to or greater than the actual difference.

Abstract classification performance

Fig 2 shows classifier performance on the abstract task for the unigram and bigram runs with no feature transform applied. The best classifier configuration, as well as those configurations not significantly different from the best (p>0.05, one-tailed test), are marked with an asterisk. In addition, the performance results, ranks, and the rank-product (RP3) measure are reported in Table 1. The best classifier achieves F1≈0.93, iAUC≈0.98, MCC≈0.73, which constitutes a substantial and significant improvement over our previous preliminary results reported in [39], where we had reached F1≈0.8, iAUC≈0.88, MCC≈0.6 (notice that these performance values would be well below the lowest reported levels in Fig 2). This demonstrates that the corpus used in this work—which is more carefully curated and now also considers interactions between drugs and food, fruit, smoking, alcohol, and natural products to be relevant (details in “Materials and Methods Section”)—improves the classification of abstracts with pharmacokinetic evidence of DDI. The levels of performance achieved are excellent when compared to similar abstract classification tasks in other biomedical domains. For instance, in the BioCreative Challenge III, considered one of the premier forums for assessment of text mining methods, the best classifiers of abstracts with Protein-Protein Interaction yielded performances of F1≈0.61, iAUC≈0.68, MCC≈0.55 [17]. Naturally, our results are not directly comparable to results obtained on different corpora and on a different problem; rather, these numbers provide guidance on what is typically considered good results in biomedical article classification.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Classification performance on abstracts.

Performance for both unigram and bigram runs on non-transformed features. Left: F1 measure. Middle: MCC measure. Right: iAUC measure. The best classifier configuration, and configurations not significantly different (p>0.05, one-tailed test) from it, marked with asterisk ‘*’.

https://doi.org/10.1371/journal.pone.0122199.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Classification performance on abstracts.

Performance for both unigram and bigram runs on non-transformed features according to F1, MCC, and iAUC performance measures. The rank of the classifiers according to each measure is reported in parenthesis in the respective column. Classifiers are ordered according to the rank product (RP3) of the three measures (last column).

https://doi.org/10.1371/journal.pone.0122199.t001

For each classifier, the inclusion of bigram features improved performance according to the RP3 measure. The best classifier according to all measures was LDA using bigrams. The performance of this classifier was significantly better than all others for the MCC, but not significantly better that Logistic Regression according to iAUC, and not significantly better that Logistic Regression and SVM according to F1 score. According to RP3, these three classifiers using bigrams yield the best performance. Naive Bayes, VTT, and dLDA—classifiers that make a ‘naive’ independence assumption about features (see “Materials and Methods” section)—performed below the top three. However, the performance levels they achieved are still quite high, which indicates that such simple classifiers are also capable of classifying documents with pharmacokinetic DDI evidence in our corpus. The in-house VTT classifier is the only classifier among these that does not use cross-validated parameters; when used with NER features and cross-validated parameters (the configuration for which it was originally designed [12, 41, 42]), its performance improved (see below).

Pharmacokinetics DDI Features in abstract classification.

We looked at which textual features play the largest role in the abstract classification task. A linear classifier separates document classes with a hyperplane defined by a set of feature coefficients. The impact of a feature on classification is quantified by the sign and amplitude of its hyperplane coefficient. A feature with a large positive coefficient contributes strongly to a document’s propensity to be classified as relevant, while a feature with a large negative coefficient contributes strongly to a document’s propensity to be classified as irrelevant. In Table 2, we show the top 20 most distinctive features of the relevant and irrelevant classes in the abstract task, as chosen in the bigrams runs by the LDA classifier (left) and Logistic Regression classifier (right), the two top-performing classifiers in this task according to the RP3 measure (see Table 1). Notice that textual features are stemmed.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Top 20 relevant and irrelevant abstract features.

The stemmed textual features most discriminative of relevant and irrelevant classes on the abstract task, as chosen by two of the top-performing classifiers according to the RP3 measure: LDA with bigrams (left) and Logistic Regression with bigrams (right).

https://doi.org/10.1371/journal.pone.0122199.t002

Some of the most relevant features come from MeSH term metadata (such as the MeSH term Drug interactions) and terms that explicitly indicate interactions (‘interact’, ‘inhibit’, ‘interact between’, ‘decreas’, ‘increas’). Other relevant terms deal with administration protocols and study design (‘oral’, ‘day’, ‘receiv’, ‘mg’, ‘treatment’, ‘alon’, ‘combin’). Some of the irrelevant features concern genetics terminology (‘allel’, ‘genotyp’, ‘polymorph’, and MeSH term Phenotype), indicating that the irrelevant class was enriched with genetics or pharmacogenetics vocabulary. Several generic biomedical terms (such as ‘patient’, ‘healthi subject’, ‘higher’) terms are also highly irrelevant. In addition, highly irrelevant features also contain some non-DDI-specific pharmacokinetic terms (for example, ‘area’, ‘rate’, ‘clearance of’), which is not surprising given that both relevant and irrelevant articles were drawn from pharmacokinetics-related literature. One surprising result is the observation that while the MeSH term Male is one of the top relevant features, the MeSH term Female is one of the top irrelevant features. We have no explanation for the cause of this gender imbalance since the corpus was built from automatic searches to PubMed without any gender-specific query terms.

Further analysis of highly relevant and irrelevant features across all classifiers and feature transforms was performed and reported in the Supporting Information (section 1.2 in S1 Text). We quantified and plotted the contribution of standardized coefficients [54] of different features and show the most positively and negatively loaded features for different classifier and transform configurations. Top textual features obtained from all classifiers include additional terms falling under the categories described above, with features derived from PubMed metadata (MeSH, chemical substances) also appearing among both the most relevant and irrelevant sets. Other relevant MeSH terms, besides Drug Interactions, include Cimetidine/pharmacology, Cross-Over Studies, Enzyme Inhibitors/PK, Kinetics, and Proton Pump Inhibitors. Additionally, a PubMed author entry corresponding to a prominent researcher in the pharmacokinetics DDI field (‘PJ Neuvonen’) appears as highly relevant, as well as three substances from the RN field (see also section “Impact of NER and PubMed metadata on abstract classification” below): Cimetidine, Enzyme Inhibitors, and Proton Pump Inhibitors. For the irrelevant set, additional MeSH (Anti-Ulcer Agents/adm&dos; Injections, Intravenous; Phenotype; Protein Binding; Reference Values) and Substance terms also appear (Anti-ulcer agents; Hydrocarb. Hydroxylas). The Supporting Information S1 Text contains details of the analysis and lists of features. It also shows the results of a Principal Component Analysis of feature weight coefficients chosen by different classifiers.

Impact of NER and PubMed metadata on abstract classification.

We have previously demonstrated improved classification performance on protein-protein interaction BLM tasks by supplementing textual features (such as the word unigram and bigram occurrences) with features built using Named Entity Recognition (NER) and domain-specific dictionary tools [12, 41, 42]. To test if similar techniques are useful in the DDI domain, we counted mentions of named biochemical species (e.g. proteins, compounds and drugs) and concepts (e.g. pharmacokinetic terms) in each document and then included these counts as document features in addition to the bigram and unigram textual features. Counts were extracted using biomedical-specific NER extraction tools and dictionaries, with dictionary matches identified by internally-developed software. A preliminary study of the impact of NER/Dictionary features was reported in [39] using a previous less-refined DDI corpus. Here, in addition to using the more fine-tuned corpus (see “Methods and Data” section), we study the impact of PubMed metadata features on classification performance. We also provide a new comprehensive analysis of the performance impact of including features from several publicly-available NER and metadata resources:

• OSCAR4 [55]: NER tool for chemical species, reaction names, enzymes, chemical prefixes and adjectives.
• ABNER [56]: NER tool for genes, proteins, cell lines and cell types.
• BICEPP [57]: NER tool for clinical characteristics associated with drugs.
• DrugBank database [58]: a dictionary list of drug names
• Dictionaries provided by Li’s lab. i-CYPS: cytochrome P450 [CYP] protein names, a group of enzymes centrally involved in drug metabolism; i-PkParams: terms relevant to pharmacokinetic parameters and studies; i-Transporters: proteins involved in transport; i-Drugs: Food and Drug Administration’s drug names. The dictionaries are available for download from [43].

For each of these NER tools and dictionaries, we counted the number of occurrences of any of its entities/entries in a given abstract. These counts were treated as any other feature for SVM, Logistic Regression, diagonal LDA, and LDA classifiers. Naive Bayes was omitted since NER count features are non-binary. VTT incorporates NER features via a modified separating hyperplane equation: where x_i represents the occurrence of textual feature i, φ_i and λ are textual feature and bias parameters as described in section “Classifiers”, c_j is the count of NER/Dictionary feature from resource j, and β_j is a weight for resource j, which is chosen by cross-validation.

In Fig 3 (left), we plot the relative iAUC changes over the respective classifiers without NER/Dictionary count features (results for MCC and F1 in Supporting Information; section 1.3 in S1 Text). Significant performance changes are indicated with an asterisk (p<0.05, two-tailed test). Some NER/Dictionary features improved performance significantly for several classifiers. However, the inclusion of two dictionary features (DrugBank, and i-CYPS) actually decreased performance significantly for several classifiers, suggesting that these features contain little class information and instead contribute to over-fitting. Table 3 lists performance for configurations in which NER and dictionary features gave a significant performance increase for at least one of the three measures (F1, MCC, or iAUC), along with best classifier performance using only textual features (bigram runs). The BICEPP tool consistently yielded the best improvement for every classifier tested, followed by the i-Drugs dictionary. The OSCAR4 tool also significantly improved the performance of the VTT classifier (especially for the MCC measure as shown in Supporting Information, S1 Text). With the inclusion of NER and dictionary features, the overall top classifiers (LDA and Logistic Regression), significantly improved their performance, now reaching F1≈0.93, MCC≈0.74, iAUC≈0.99. Among the set of naive classifiers, VTT improved performance significantly with the inclusion of NER features, ranking above the other naive classifiers according to the RP3 measure.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Performance impact of abstract NER and metadata features.

Left: Relative changes in iAUC scores on non-transformed bigram runs in combination with different NER/Dictionary features. Significant changes (p<0.05, two-tailed test) in performance over the respective classifiers without NER features are indicated with asterisk ‘*’. Right: Relative changes in iAUC when features from a given PubMed metadata field are included versus omitted (while including features from the other 4 metadata fields). Significant changes (p<0.05, two-tailed test) in performance are indicated with asterisk ‘*’.

https://doi.org/10.1371/journal.pone.0122199.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Abstract classification performance using NER features.

Performance of the best classifiers when specific NER and dictionary features are added; original (bigram runs) classifiers also listed with no NER features (indicated by -). F1, MCC, and iAUC performance measures are listed; the rank of the classifiers according to each measure is reported in parenthesis in the respective column. Classifiers are ordered according to the rank product (RP3) of the three measures (last column).

https://doi.org/10.1371/journal.pone.0122199.t003

As mentioned, word unigram and bigram features were extracted not only from article abstracts and titles, but also from five PubMed metadata fields: author names, journal titles, MeSH terms, and two fields referring to standardized substance names: the ‘registry number/EC number’ [RN] field and the ‘secondary source’ field [SI]. In fact, some PubMed metadata features were among those most distinguishing of relevant and irrelevant abstracts (for greater detail, see Table 2 and section “Pharmacokinetics DDI Features in abstract classification”, as well Supporting Information; section 1.2 in S1 Text). We tested the impact of PubMed metadata fields on abstract classification performance. In Fig 3 (right), we plot the relative iAUC changes when features from a given PubMed metadata field are included versus omitted (while including features from the other 4 metadata fields). Significant changes (p<0.05, two-tailed test) in performance are indicated with an asterisk; results for MCC and F1 can be found in Supporting Information (S1 Text). MeSH terms was the only metadata source whose omission decreased performance significantly. However, the performance increase of including MeSH data is rather small. Therefore, the methodology does not require the availability of human-annotated metadata such as MeSH terms and can still be deployed on recent articles that have not yet been annotated with MeSH terms.

Evidence sentence extraction performance

Fig 4 shows classification performance on the sentence task of the unigram and bigram runs without any feature transforms applied, according to F1, MCC, and iAUC measures. The best classifier configuration, as well as those configurations not significantly different from the best (p>0.05, one-tailed test), are marked with an asterisk. In addition, the numerical results, ranks, and the rank-product (RP3) measure are reported in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Sentence classification performance.

Performance for both unigram and bigram runs on non-transformed features. Left: F1 measure. Middle: MCC measure. Right: iAUC measure. The best classifier configuration, and configurations not significantly different (p>0.05, one-tailed test) from it, marked with asterisk ‘*’.

https://doi.org/10.1371/journal.pone.0122199.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Sentence classification performance.

Performance for both unigram and bigram runs on non-transformed features according to F1, MCC, and iAUC performance measures. The rank of the classifiers according to each measure is reported in parenthesis in the respective column. Classifiers are ordered according to the rank product (RP3) of the three measures (last column).

https://doi.org/10.1371/journal.pone.0122199.t004

As with abstracts, including bigram features tended to improve sentence classification performance. LDA performed best, having the highest RP3 and being the best classifier according to the F1 and MCC measures and one of the two best classifiers (along with SVM) on the iAUC measure. Generally, the classifiers that performed well on the sentence task were those that took into account feature covariances: SVM, Logistic Regression, and LDA. The top classifier (LDA with bigrams) on the evidence sentence task reached performance of F1≈0.75, MCC≈0.64, iAUC≈0.83.

We measured sentence classification performance in combination with different feature transforms and dimensionality reductions (see section 2.1 in S1 Text). In general, the three classifiers that do best on non-transformed features (SVM, Logistic Regression, and LDA) show decreased performance with dimensionality reduction according to all measures, with more extreme dimensionality reduction leading to larger performance decreases. On the other hand, for dLDA (a ‘naive’ classifier that treats features as independent), PCA-based dimensionality reduction—which uses feature covariances to choose optimal projections—led to significant improvements in all measures, with more dimensions giving better performance. These findings indicate that the pattern of feature covariance carries important information about class membership in the sentence task, and that this pattern is distributed across a large number of dimensions. Generally, the LDA classifier achieved the best performance according to all three measures. Its baseline performance according to the iAUC measure was further improved significantly by an IDF-transform, and—according to the F1 measure—by any transform containing an L2 normalization.

In Table 5, we show the top 20 features most distinctive of the relevant and irrelevant classes in the sentence task as chosen by the LDA classifier on bigrams (the top performing classifier according to the RP3 measure; features of other top classifiers are not shown because they were highly similar). Numerical features (indicated by ‘#.#’) were highly indicative of the relevant class, along with expressions of quantitative changes (‘decreas’, ‘increas’) and interaction (‘inhibit’, ‘catalyz’, ‘interact with’) as well as adverbs expressing significance of evidence (‘significantli’). Also highly relevant were features referring to the area under the concentration-time curve (‘auc’), which is often employed in pharmacokinetics to measure differences in drug clearance rates under different experimental conditions. Names of several drugs (‘ketoconazol’, ‘itraconazol’, ‘quiindin’) were relevant in predicting DDI evidence sentences. These drugs are frequently used probe inhibitors for metabolism enzymes CYP3A4/5, CYP3A4/5 and CYP2D6 respectively and are routinely used in drug interaction studies.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Top 20 relevant and irrelevant sentence features.

The most discriminative features of relevant and irrelevant classes in the sentence task, as chosen by the top-performing classifiers according to the RP3 measure: LDA on bigrams.

https://doi.org/10.1371/journal.pone.0122199.t005

Highly irrelevant features refer to more generic pharmacokinetic or biomedical concepts such as ‘investig’, ‘dose’, ‘enzym’, ‘studi’, etc. Interestingly, some terms that are highly relevant in the abstract task are highly irrelevant in the sentence task (e.g., ‘day’). Notably, the unigram ‘interact’ is highly irrelevant for sentences, whereas the bigram ‘interact with’ is highly relevant. This may be because all sentences in this corpus come from abstracts containing pharmacokinetic DDI evidence (see “Materials and Methods” section). Thus, general administration protocols and drug interaction terms are likely to occur in the abstract as a whole but not necessarily in the evidence sentences that actually report outcomes of the pharmacokinetic drug interaction experiments. Similar patterns are observed in the more extensive analysis provided in Supporting Information (section 2.2 in S1 Text), where relevant and irrelevant features are analyzed across a wide range of classifier and feature transform configurations. There we also show the results of a Principal Component Analysis of feature weight coefficients chosen by different classifiers.

Finally, we tested the impact of additional features on sentence classification. Though there is no metadata available in the sentence corpus, features from NER tools can still be computed. Six NER features were tested: BICEPP, DrugBank, i-CYPS, i-Drugs, i-PkParams, i-Transporters (see section “Impact of NER and PubMed metadata on abstract classification” for details). As before, we counted mentions of named biochemical species and concepts specified by different NER tools in each sentence and then included such counts as sentence features in addition to the bigram and unigram textual features. Fig 5 shows relative iAUC changes when features from each of these NER tools were included. Significant improvements (p<0.05, two-tailed test) above the corresponding classifier’s performance without NER features are indicated by an asterisk; performance according to MCC and F1 measures is shown in Supporting Information (section 2.3 of S1 Text). Notice that Naive Bayes was omitted since NER count features are non-binary.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Performance impact of sentence NER features.

Relative changes in iAUC scores on sentence bigram runs (without transforms or dimensionality reductions) in combination with different NER features. Significant changes (p<0.05, two-tailed test) in performance over respective classifiers without NER features are indicated with asterisk ‘*’.

https://doi.org/10.1371/journal.pone.0122199.g005

As in the abstract task, a few NER/Dictionary features improved performance for several classifiers. The iAUC scores of nearly all classifiers were significantly improved by three NER features: BICEPP, DrugBank, and our internally developed i-Drugs dictionary. These three features represent counts of drugs names, showing that drug name counts are helpful for classifying sentences as DDI-relevant vs. DDI-irrelevant. Use of features from the BICEPP tool yielded the largest improvement for every classifier. Table 6 lists the performance according to all measures for classifiers using the BICEPP features; also listed are the corresponding best classifiers using only textual features (bigram runs). The overall top classifiers (LDA and SVM) showed significantly improved performance with the inclusion of these NER features, reaching F1≈0.76, MCC≈0.65, iAUC≈0.83. In addition, VTT performance improved significantly for all three measures with the inclusion of NER features. Here VTT with bigrams performs better than other naive classifiers, as expected given that this classifier was designed specifically to handle such NER features [12, 41, 42]. In contrast, dLDA (another naive classifier) did not benefit much from the inclusion of NER features.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Sentence classification performance with NER features.

Performance of different sentence classifiers with the count features obtained via the BICEPP NER tool; also listed are the corresponding best classifiers using only textual features (bigram runs; indicated by -). F1, MCC, and iAUC performance measures are listed; the rank of the classifiers according to each measure is reported in parenthesis in the respective column. Classifiers are ordered according to the rank product (RP3) of the three measures (last column).

https://doi.org/10.1371/journal.pone.0122199.t006