An integrated text mining framework for metabolic interaction network reconstruction

Construction of Metabolic Entities (ME) corpus

The ME corpus is developed and made publicly available at www.sbi.kmutt.ac.th/ preecha/metrecon. The corpus is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. It is a collection of various abstracts and titles from different databases that have been manually annotated for metabolic events by two annotators who are biologists with different backgrounds. In order to construct the ME corpus, an article list was initially collected from the EcoCyc database (version 16.5). The EcoCyc database was selected as an example source because it contains a comprehensive resource of biological information for the model organism E.coli K12. It contains manually curated and extensive information (e.g., summary comment, regulatory information, literature citation, and extracted evidence type obtained from thousands of publications) (Keseler et al., 2013). From this list, articles were randomly selected and their abstracts and titles were then downloaded from the PubMed database for the subsequent annotation process. Considering the process of annotation in each abstract and title, BANNER (Leaman & Gonzalez, 2008) was first used to annotate gene and protein (GP), as well as metabolite entities according to our annotation guideline (www.sbi.kmutt.ac.th/ preecha/metrecon). Table 1 presents annotated entity types along with reference databases i.e., EcoCyc and ChEBI and Systems Biology Ontology (SBO) ID. Afterwards, manual correction using BRAT (Stenetorp, Pyysalo & Topic, 2012) by individual domain experts was performed. The metabolic events in the abstracts and titles were annotated according to four types of the metabolic events (i.e., metabolic production, metabolic consumption, metabolic reaction, and positive regulation). Eventually, these annotations were merged to create a final annotation set. For the definition and scope of the metabolic event annotation, the Systems Biology Ontology (SBO) and the Gene Ontology (GO) are considered. Table 2 presents the annotated metabolic event types, arguments and their Ontology ID. A hierarchical representation of the metabolic entities and events is illustrated in Fig. 2A. Also, an example of annotation for metabolic entities and events can be seen in Fig. 2B. For metabolic entities, the annotation identifies phosphoglucosamine mutase and GlmM as GP entities and glucosamine-1-phosphate and glucosamine-6-phosphate as metabolite entities. For metabolic events, the annotation identifies event words of catalyzes and formation as event types of positive regulation and metabolic reaction, respectively. Note that this ME corpus focuses only on metabolic interactions (i.e., enzyme–metabolite interactions) throughout metabolic events at the end. Other types of data, e.g., substrates, products, co-enzymes and co-factors, were considered as metabolites. For discussion of the relation between these entities and event types and the other representations applied in ME corpus, Ohta et al. (2013) was used as a reference.

To measure an inter-annotation agreement, 20 abstracts and titles were randomly selected as an example case. The two annotators annotated these abstracts and titles according to the annotation guidelines (www.sbi.kmutt.ac.th/ preecha/metrecon). These annotated abstracts and titles were then used to manually construct consensus annotation.

The annotated results from the two annotators were compared against the constructed consensus annotation and then the F-score was calculated across different data of GP entities, metabolite entities and events. Additionally, the Cohen’s kappa coefficient (Cohen, 1960) was also considered as a statistical measure of inter-annotator agreement for GP entities, metabolite entities and events. The overall performance difference between annotators A and B is not significant when compared to consensus annotation (F-scores range from 74.07% to 96.88%) (see Table 3). Also, the agreement between the two annotators were high across all categories (kappa coefficients range from 0.72 to 0.96). It is worth noting that multiple interpretations of numerous entities and events between the two annotators may cause high variability of the inter-annotator agreement level.

Construction of test corpus

The test corpus (www.sbi.kmutt.ac.th/ preecha/metrecon) was constructed from a collection of introduction, abstract and title sections within two Superpathways articles in the EcoCyc and PubMed databases, i.e., the Superpathway of leucine, valine, and isoleucine biosynthesis (18 articles) and the Superpathway of pyridoxal 5′-phosphate biosynthesis and salvage (9 articles). The abstract, introduction and title sections of an article were used because they provide a major summary of the research article, supporting statement and theoretical context. It is also noted that these two selected Superpathways were relatively large compared to others and therefore a considerable number of abstracts, titles and introductions could be collected for test corpus construction. Regarding the annotation process, it was performed as described in the earlier section about the construction of a ME corpus.

Development of Metabolic Event Extraction (MEE) module

After constructing the ME corpus and test corpus, we developed a MEE module, as illustrated in Fig. 3A. This MEE module architecture was divided into two sub-parts, namely text pre-processing and metabolic event extraction as described below.

Development of Metabolic Interaction Network Reconstruction (MINR) module

Once MEE module was developed, we further developed a MINR module. We divided the MINR module into two sub-sections: the mapping from metabolic event to metabolic interaction and reconstructing the metabolic interaction network by combinations of individual metabolic interactions.

Evaluation of integrated TM framework

To evaluate the integrated TM framework, we assessed the MEE module and the MINR module, separately as described below. MEE module evaluation on three critical sub-parts: ME corpus, ME corpus size and test corpus was performed. In each sub-part, we calculated performance based on standard precision, recall, and F-score as performance measures (Van Rijsbergen, 1979).

To compare the obtained results with manually curated entities, we applied the sloppy span matching criterion to entities, which means that entities must match the types, but are not required to exactly match the entities boundaries (Czarnecki et al., 2012). For metabolic events, the comparison criterion that is termed approximated boundary matching (Kim et al., 1979) was used. In particular, the three criteria used were: (i) identical metabolic event type, (ii) sloppy span matching between metabolic event trigger span, and (iii) at least one matching or all matching of arguments. We used these criteria in this study due to strong supporting evidence that they are better in terms of information retrieval than the alternative exact matching criterion, where boundaries between two entities are required to match exactly (Kabiljo, Clegg & Shepherd, 2009; Shepherd & Kabiljo, 2008).

Application of integrated TM framework

To demonstrate the application of the integrated TM framework, we presented two case studies. The first case study showed a comparative analysis of our integrated TM framework with another TM system developed by Czarnecki et al. (2012) on the test corpus for reconstruction of the Superpathway of leucine, valine, and isoleucine biosynthesis. To elaborate, we ran our integrated TM framework on the test corpus to extract the metabolic events and mapped them to metabolic interactions for the Superpathway reconstruction. Once completed, we then performed a comparative analysis of the reconstructed network with the other results achieved by the TM system developed by Czarnecki et al. (2012).

For the second case study, the integrated TM framework with EcoCyc extraction was applied to reconstruct a metabolic interaction network. EcoCyc extraction is a collection list of references from the EcoCyc database (2,373 abstracts and titles). In brief, we first ran the integrated TM framework on EcoCyc extraction in order to extract a list of enzyme, metabolite, and enzyme–metabolite interactions association with metabolic events. Also, we ran the integrated TM framework on the ME corpus for performance comparison. To evaluate the performance, the predicted entities in terms of enzymes, metabolites, and enzyme–metabolite interactions were compared to the manually-curated metabolic entities in the EcoCyc database for calculating the precision. A published genome-scale metabolic network of E. coli K-12 MG1665 (iJO1366) (Orth et al., 2011) was also used as the manually-curated metabolic pathways for metabolic interaction network reconstruction.

The integrated TM framework could be run using the virtual machine image on one of Mac, Windows, or Linux system with the pre-configured software available at www.sbi.kmutt.ac.th/ preecha/metrecon. The source code in the virtual machine is licensed under Apache License 2.0.

ME corpus statistics

Our constructed ME corpus consists of annotated GP, metabolite entities, and metabolic events. Table 4 shows the basic statistics for the constructed ME corpus. Of the 271 abstracts and titles selected from the PubMed database (see ‘Methods’), we found a total number of 2,513 entities for GP and 1,898 entities for metabolite, corresponding to 9.27 and 7.00 entities per abstract for GP and metabolite, respectively.

In addition, we further examined the basic statistics on the average number of arguments per metabolic event. As also shown in Table 4, the metabolic reaction event has an average number of 2.12 arguments per metabolic event. This is higher than the average number of arguments in metabolic production event (1.10) and metabolic consumption event (1.22). In order to further express the word preference in the metabolic event, we inspected common words in the ME corpus. The top ten list of event words identified in the ME corpus is presented in File S3. Interestingly, we found that a major portion of these common words (45.84% of total metabolic events) are involved in the generic description of biological processes (e.g., Catalyzes, Biosynthesis, Synthesis, Formation, Conversion, Utilization, Catalyzed, Catalyze, and Metabolism). These results showed that most of the event words in the metabolic event were centred around a small set of general keywords. However, it is possible to deduce the types of enzymatic reactions using name of substrate and product in some cases, e.g., the formation of glucosamine-1-phosphate (product name) from glucosamine-6-phosphate (substrate name) (Fig. 2B) suggests a phosphorylation reaction.

Performance of MEE module on ME corpus

Using five-fold cross-validation on the ME corpus, the recall, the precision, and the F-score were calculated for each metabolic event type as measures for the overall performance evaluation. Table 5 shows these measures for each metabolic event type as well as for the total event type which indicates the sum of all metabolic event types. That is, the measures of total event type were calculated by summation of the individual true positives, false positives, and false negatives for each metabolic event type.

The F-scores of the events of metabolic production (59.15%) and metabolic consumption (48.59%) turned out to be higher than both of the events of metabolic reaction (28.32%) and positive regulation (36.69%). It is intuitive to think that complex events (i.e., two or more arguments) as found in metabolic reaction and positive regulation are harder to be classified than simple events (i.e., one argument) as found in metabolic production and metabolic consumption. In such a case, when the average number of arguments per metabolic event is high, a low F-score is clearly shown (see Tables 4 and 6) as in the example of the metabolic reaction and the positive regulation. These above-mentioned results are strongly supported by earlier works in Pathway Curation (PC) task—BioNLP-ST’13 and BioNLPST’11 (Ohta et al., 2013; Kim et al., 2011). In particular, the F-scores achieved from positive regulation event between PC task—BioNLP-ST’13 (Ohta et al., 2013) and our study were compared. Consequently, the F-scores were similar with values of 39.23 and 36.69, respectively.

Performance of MEE module on different ME corpus sizes

To assess the effect of ME corpus sizes, different subsets of 100, 150, and 200 abstracts and titles were randomly extracted and compared to the whole ME corpus size of 271 abstracts and titles. Each subset was evaluated three times, and the results shown were averages (Fig. 4). The corresponding recall, precision and F-score were compared in the form of learning curves in Fig. 4 using five-fold cross-validation. Expectedly, the performance of MEE module with the largest corpus size was better than that of the smaller corpus sizes in all possible cases. Clearly, the best recall and F-score were obtained with the whole corpus size of 271 abstracts and titles (Figs. 4A and 4C). In general, the trend was improved performance with a larger corpus size for these two measures. A similar trend was observed for the precision measure, except for the case of metabolic production which showed no dependence on corpus size (Fig. 4B). From the overall results, we suggest that a minimum of 150 abstracts and titles should be used for development of a ME corpus. Note that the regularity of the metabolic event description is applied for development of ME corpus for easier event extraction.

Performance of MEE module on test corpus

As a further assessment of the proposed integrated TM framework, a constructed test corpus was used for performance evaluation of the MEE module. Basic statistics of test corpus can be seen in Table 4. It contained 27 articles related to two Superpathways from the EcoCyc database (see ‘Methods’). At first, we evaluated MEE module on the test corpus using GP and metabolite entity tagging. As shown in Table 6, the precision, recall, and F-score of the entity tagger for GP and metabolite were very high for more than 80% of GP and metabolite entities identified for the Superpathway of leucine, valine, and isoleucine biosynthesis. However, the recall and F-score of the entity tagger for metabolite was lower than our expectations for the Superpathway of pyridoxal 5′-phosphate biosynthesis and salvage. The recall showed less than 70%, and the F-score showed less than 80%. These results seem to indicate that the entity tagger (i.e., BANNER) has a weakness in detecting metabolite entities in an abbreviated form (e.g., Pyridoxine (PN), Pyridoxal (PL), and 4-hydroxy-l-threonine phosphate (HTP)).

Performance of the MINR module on test corpus

We evaluated the performance of the MINR module using the test corpus. In terms of enzyme–metabolite interaction, we found that the MINR module showed high performance as presented in Table 7. The reconstructed results from the MINR module were then compared with manually curated metabolic interactions list (see File S2). As shown in Table 7, for the two Superpathways, the precisions were 80–90%, the F-scores were 70–80%, and the recalls were 60–70%. These results suggest that the MINR module performed well for the mapping of enzyme–metabolite interactions and can be further used for reconstruction of metabolic interaction networks. Nonetheless, there were still missing interactions which could not be identified by the MINR module. These could be because MINR module was unable to capture the metabolic events that were implicit in the text as seen in the example of PMID-13405870 (see File S4). Moreover, it was also unable to extract exact precedence relationships among metabolic events as seen in the example of PMID-13727223 (see File S4). Further improvements are planned for MINR module after taking into account these limitations.

The integrated TM framework for reconstructed metabolic interaction network

As mentioned in the ‘Methods,’ two case studies were used to evaluate the integrated TM framework. For the first case study, as shown in Fig. 5, our integrated TM framework on the test corpus successfully extracted 11 entities of enzymes and metabolites as well as 10 enzyme–metabolite interactions for reconstruction of the Superpathway of leucine, valine, and isoleucine biosynthesis.

To elaborate how a biologist can apply our integrated TM framework for reconstruction of the Superpathway of leucine, valine, and isoleucine biosynthesis, we show two example sentences extracted from PMID-1646790 that were obtained from MEE and MINR modules. The examples are described below.

Example 1: “leucine synthesis by the tyrosine-repressible transaminase in Escherichia coli K-12.”

Example 2: “2-KIC amination by the tyrB-encoded transaminase and also by the aspC- and avtA-encoded transaminases.”

For explanation of example 1, the MEE module identifies enzyme (tyrosine-repressible transaminase) and metabolite (leucine) throughout metabolic production event. Considering example 2, the MEE module identifies enzyme (tyrB-encoded transaminase) and metabolite (2-KIC) throughout metabolic consumption event. To the end, the MINR module obtains enzyme–metabolite interactions by combining the metabolic production and consumption events from examples 1 and 2, respectively. As a result, 2-KIC can be converted to leucine by tyrB-encoded transaminase (tyrosine-repressible transaminase). Full details of enzymes, metabolites, and enzyme–metabolite interactions can be seen in File S5.

Comparing these results to those obtained by a TM system developed by Czarnecki et al. (2012), we found that a similar number of correctly extracted entities of enzymes and metabolites and enzyme–metabolite interactions were obtained (Fig. 5). As can be seen, the results achieved from both systems are able to extract a different part of network suggesting that the combination of the results and biological intepretation would be an interesting option for a biologist who searches for an alternative way for reconstructing a network.

For the second case study involving large-scale data extraction from EcoCyc, the results are shown in Table 8. We found the precisions for this EcoCyc extraction data in terms of enzymes (69.93%), metabolites (70.63%), and enzyme–metabolite interactions (46.71%). After comparing these precisions to the similar results gained from constructed ME corpus (271 abstracts and titles), we found that the EcoCyc extraction data showed a higher number of false positives. Based on our manual inspection, one source of false positives came from mentions of enzymes, metabolites, or enzyme–metabolite interactions in other species that were not from E. coli despite the fact that our framework was trained using E. coli abstracts and titles. However, this is favorable for the real-world usage since it shows the generality of our method can capture all generic mentioned reactions in text. Another note is that we did not deploy a normalization method in our evaluation, and this might not correctly reflect the performance of the real-world large-scale extraction where the normalization method is critical. Nevertheless, these results illustrate that our constructed ME corpus within the integrated TM framework is solid and can be used as a representative dataset for large-scale data extraction with applications for building metabolic interaction databases and networks as well as for knowledge discovery tasks. The proposed integrated TM framework application is summarized in Fig. 6.