Design and implementation

Galaxy-ML provides key benefits in scalability, reproducibility, and workflow development. Large machine learning analyses, such as optimizing hyperparameters and model evaluation across many different datasets, can require building tens of thousands of models. Galaxy-ML uses Galaxy’s workflow system to execute large-scale analyses by distributing them across one or more computing clusters and running them in parallel. Galaxy ensures reproducibility by recording all parameters and tools used, so all analyses, including those for machine learning, are completely reproducible. This is critical, as reproducibility has become critical in machine learning research [13,14]. Galaxy-ML enables end-to-end machine learning analyses that begin with processing primary biological data and ends with trained machine learning models that can make predictions of phenotypic attributes like demographics or prognosis. For instance, this interactive tutorial [15] uses Galaxy-ML to reproduce a study that predicts an individual’s chronological age from RNA-seq data [16,17]. End-to-end workflows are possible because Galaxy-ML’s machine learning tools can be connected to any of the more than 7,800 tools available in the Galaxy ToolShed [12] for analyzing genomics, proteomics, imaging, and other kinds of biomedical data.

Galaxy-ML supports four major steps in machine learning—preprocessing, modeling, ensembling, and evaluation—by integrating six machine learning libraries (Table 1) together with additional visualization and conversion tools. Scikit-learn [18] provides the foundation for Galaxy-ML with approaches for all four major steps. Additional libraries are included to meet key needs for machine learning in biomedicine, including feature selection, approaches for working with imbalanced datasets, and modeling approaches using gradient boosted decision trees, deep learning, and ensembling. Documentation, along with tutorials, is available at https://galaxyproject.org/community/machine-learning/, and links to the Galaxy-ML code and tool repositories are available in the Methods section.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Software libraries integrated into Galaxy-ML and their applications.

https://doi.org/10.1371/journal.pcbi.1009014.t001

Automatically creating and evaluating thousands of machine learning models

In the first use case, we used Galaxy-ML to extend an analysis of machine learning models across 276 biomedical datasets [25]—164 classification datasets and 112 regression datasets [25]. The original analysis compared performance of 13 models on the 164 classification datasets. We applied 15 models to the classification datasets and 14 models to the regression datasets, creating a total of 4,028 trained models with hyperparameters optimized using grid search (S1 Text and S1 Table). We evaluated all models using 10-fold cross-validation (CV). Because many datasets were imbalanced, F1 scoring rather than ROC AUC was used to evaluate performance of classification models, and Pearson’s R² was used to evaluate performance of regression models. Performance of classification models are concordant with the initial publication: (i) boosted tree models perform best overall (Fig 2A) and (ii) automated hyperparameter optimization improves performance for many models (Fig 2B). Performance of regression models are similar to those in classification, though boosted tree models only modestly outperform random tree models, and hyperparameter optimization often improves results most for models with low overall performance (S2 Text and S1 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Pairwise performance comparisons for use cases 1 and 2.

Use case 1 pairwise comparisons for classification tasks on 164 structured biomedical datasets [25] show decision tree forests perform best (panel A) and hyperparameter optimization can improve the performance of most models (panel B). Use case 2 results for prediction using regression (panel C) and classification (panel D) show ensemble approaches that use stacking perform best, though linear-based gradient boosting also performs. In panels A, C, and D, heatmaps show the percentage of datasets for which the model listed along the row outperforms the model along the column. For instance, in panel A, XGBoost outperforms Gradient Tree Boosting (GTB) from scikit-learn on 38% of datasets, GTB outperforms XGBoost on 11% of datasets, and they perform equivalently on 51% of datasets.

https://doi.org/10.1371/journal.pcbi.1009014.g002

Developing meta-ensembles for predicting cell line drug response

For the second use case, we used Galaxy-ML to apply sophisticated machine learning models, including stacked meta-ensemble predictors, to predict drug response in cancer cell lines from high-throughput gene expression data from RNA-seq (S3 Text). Because cancer cell lines serve as models for patient tumors, accurate predictions of drug response can be used to improve understanding of cancer systems biology and inform patient treatment recommendations [26]. Gene expression and drug response data was obtained from DepMap [27]. There are two key challenges for this dataset: (1) there are ~50,000 gene expression features but only ~1,000 cancer cell lines and ~700 drugs, so preventing overfitting is essential, and (2) the dataset is highly imbalanced because there is a small number of cell lines that respond to each drug.

Using Galaxy-ML, we built a meta-ensemble as well as other learners for each drug. The meta-ensemble included a linear boosted model, tree boosted model, and k-nearest neighbor regression, and principal component analysis (PCA) was used for dimensionality reduction in several learners. Dimensionality reduction was used to address the challenge of using a dataset with a very large number of features. We developed predictors for both regression and classification; labels for classification were generated by thresholding drug response values and labeling cell lines as responders or non-responders to each drug using a cutoff of z-score < -1 for responders. Predictors were scored using average precision to address the challenge of assessing model performance on a highly imbalanced dataset, where the goal is to identify responders (true positives) amongst a very large number of non-responders. To compare regressors and classifiers, average precision for regressors was calculated using rank-ordered predictions, which has been done in past machine learning work in this space [6]. We evaluated each learner using nested CV, with 5-fold CV for 4 repetitions for the outer splits and 5-fold CV with two repetitions for the inner splits. Our results show that stacking regressors performed best for both regression (Fig 2C) and classification (Fig 2D). Linear boosting approaches also performed very well, with results that were on par with the meta-ensembles. Successful completion of these two use cases shows that Galaxy-ML can support large and diverse machine learning experiments.

Reproducing deep learning models for DNA sequence analysis

In the third use case, Galaxy-ML was used to validate key results from Selene [28], a deep learning toolkit for biological sequence data built on the PyTorch library. Using Galaxy-ML, we reimplemented two deep learning architectures originally implemented in Selene that model and predict regulatory elements, including transcription factor binding sites, DNase I hypersensitive sites, and histone marks (S4 Text). Results from these models are within 1% of those reported for Selene (Figs 3 and S2 and S3 and S2 Table). Critical to this work was the implementation of data generators in Galaxy. Data generators meet two important needs: (1) producing new examples from existing data to increase the number of instances available for training, and (2) feeding small sets of examples to the deep learning model so that the entire training dataset does not need to be loaded into memory. This use case demonstrates that Galaxy-ML deep learning tools are general and powerful enough to support realistic use cases and that Selene results validated across different deep learning implementations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3.

(A) Galaxy workflow to create and train a deep learning model, then use the model for visualization and prediction. (B) Precision-recall curve for a deep neural network trained to predict binding sites for a single transcription factor. (C) Precision-recall curves for a deep neural network that predicts 919 regulatory element binding profiles, with each curve in the plot denoting a precision-recall curve for 1 regulatory element.

https://doi.org/10.1371/journal.pcbi.1009014.g003

Reproducibility and extensibility

All analyses in Galaxy are highly reproducible from individual tool executions to complete workflows because Galaxy records and stores all parameter settings, tool versions, and workflow versions used in an analysis. This reproducibility extends to all Galaxy-ML tools as well as Galaxy workflows that include Galaxy-ML tools. There are ongoing efforts to enable reproducibility of Galaxy workflows outside of the Galaxy platform via interoperable workflows. The primary focus of these efforts are to make Galaxy compatible with workflows written in Common Workflow Language (CWL) [29]. CWL is a community standard for workflow definitions that is embraced by many workflow engines. When Galaxy’s CWL features are complete, Galaxy will be able to execute CWL workflows as well as export Galaxy workflows in CWL format. Once in CWL format, Galaxy workflows can be executed in any workflow engine that supports CWL, and this interoperability will extend to workflows composed of Galaxy-ML tools.

It is possible to extend Galaxy-ML with additional machine learning software libraries and custom methods. Galaxy-ML has tools for data preprocessing, data generators, model definition, model training, and model evaluation, providing clear integration points where additional machine learning approaches can be added based on their functionality. These integration points mirror the scikit-learn application programming interface, which is widely used across the machine learning community. For instance, if a new library for creating gradient boosted decision trees becomes available, tools such as Model Fit and Hyperparameter Search can be augmented so users can create and use models from this library. As part of the use cases in the previous sections, we implemented custom modules for preprocessing, modeling, data splitting, and evaluation.

Availability and future directions

The website https://galaxyproject.org/community/machine-learning/ provides a hub for machine learning in Galaxy and access to all Galaxy-ML tools, workflows and tutorials. We anticipate that this hub will serve as a community starting point to foster accessible machine learning in biomedicine. The Galaxy tool wrappers for our machine learning suite are available at the following URLs: (1) main tools: https://github.com/bgruening/galaxytools/tree/master/tools/sklearn and (2) utilities and custom classifiers: https://github.com/goeckslab/Galaxy-ML, and the entire suite can be installed onto any Galaxy server through the Galaxy ToolShed at http://bit.ly/galaxy-ml-toolshed.

Galaxy-ML accelerates biomedical research by making machine learning more accessible, scalable, and reproducible. We applied Galaxy-ML in three complex use cases that yielded novel insights from several large and diverse biomedical datasets. Galaxy-ML’s tools are completely generalizable and have applications well beyond these use cases. With Galaxy’s web-based user interface, an entire machine learning pipeline from normalization, feature selection, model definition, hyperparameter optimization, and cross-fold evaluation can be created and run on large datasets in parallel across a computing cluster using only a web browser. This makes scalable and reproducible machine learning accessible to biomedical scientists regardless of their informatics skills. By leveraging the more than 7,800 analysis tools available in Galaxy, comprehensive end-to-end analyses can be performed, which begins with primary analysis of -omics, imaging, or other large biomedical dataset and continues to downstream machine learning tools that build and evaluate predictive machine learning models from features extracted from the primary data.

Looking forward, additional machine learning tools, libraries, and datasets will be integrated into Galaxy-ML. High priority work includes support for deep learning with imaging datasets and integration of predefined and pretrained models. Future versions of Galaxy-ML will also include additional integration points to make it easier to implement and use new machine learning libraries and individual tools.