Utilizing Heterogeneous Data Sources in Computational Grid Workflows

Abstract

Besides computation intensive tasks, the Grid also facilitates sharing and processing very large databases and file systems that are distributed over multiple resources and administrative domains. Although accessing data in the Grid is supported by various lower level tools, end-users find it difficult to utilise these solutions directly. High level environments, such as Grid portal and workflow solutions provide little or no support for data access and manipulation. Workflow systems are widely utilised in Grid computing to automate computational tasks. Unfortunately, the ways of feeding data into these workflows is limited and in most cases requires additional tools and manual intervention. This paper describes how data can be fed into computational workflows from heterogeneous data sources. The P-GRADE Grid portal and workflow engine have been integrated with the SDSC Storage Resource Broker (SRB) in order to access SRB data resources as inputs and outputs of workflow components. The solution automates data interaction in computational workflows allowing users to seamlessly access and process data stored in SRB resources. The implemented solution also enables the seamless interoperation of SRB, SRM (Storage Resource Manager) and GridFTP file catalogues.