Adding domain data to code profiling tools to debug workflow parallel execution

Highlights

• Database-oriented approach to extract at runtime fine-grain performance data associated to workflow information, provenance and domain-specific data.

• Development of PerfMetricEval component to capture performance and resource consumption data using TAU and SAR profiling tools.

• Specialization of our provenance data model, named as PROV-Df, to include typical data from the code-profiling tools.

• Integration of PerfMetricEval with Chiron, a parallel SWfMS.

• Parallel execution of two workflows in the astronomy and bioinformatics domains showing the analytical potential of our approach.

Abstract

Computer simulations may be composed of several scientific programs chained in a coherent flow running in High Performance Computing and cloud environments. These runs may present different execution behavior associated to the parallel flow of data among programs. Gather insight into the parallel flow of data is important for several applications. The usual way of getting insight into code performance is by means of a code-profiler. Several parallel code-profiling tools already support performance analysis, such as Tuning and Analysis Utilities (TAU), or provide fine-grained performance statistics, e.g., System Activity Report (SAR). These tools are effective for code profiling, but are not connected to the concept of IO-intensive workflows. Analyzing the workflow execution with domain and performance data is important for users because they can identify anomalies, choose suitable machines to run their workflows, etc. This type of analysis may be performed by capturing execution data enriched with fine-grained domain data during the long-term run of a computer simulation. In this paper, we propose a monitoring data capture approach as a component that couples code-profiling tools to domain data from workflow executions. The goal is to profile and debug parallel executions of workflows through queries to a database that integrates performance, resource consumption, provenance, and domain data from simulation programs flow at runtime. We show how querying this database with domain-aware data at runtime allows to identify performance anomalies not detected by code-profiling tools. We evaluate our approach using the astronomy Montage workflow on a cluster environment and the SciPhy bioinformatics workflow on the Amazon cloud. In both cases computing time overhead imposed by our approach for gathering fine-grained domain, performance, and resource consumption data is negligible.

Introduction

Oden et al. [1] define models as mathematical constructions based on physical principles attempting to characterize abstractions of the reality. Computer simulations commonly use mathematical models with high complexity for solving problems of relevance in science and engineering. Commonly, these simulations involve the execution of complex programs chained in a coherent flow. Each program in the flow represents a computational model execution with data dependency from a previous simulation program in this flow. These simulations can be modeled as a scientific workflow (for the sake of simplicity henceforth named as workflow) [2].

A workflow is an abstraction that defines a set of activities and a dataflow among them [2]. Each activity is associated with a simulation program, which is responsible for the consumption of an input dataset and the production of an output dataset. Many workflows process a large volume of data, requiring the effective use of High Performance Computing (HPC) and High-Throughput Computing (HTC) [3] environments allied to parallelization techniques such as data parallelism or parameter sweep [4]. While HPC environments are commonly employed to perform large amounts of computing, HTC environments usually process applications that are more interested in how many tasks can be completed over a long period of time instead of how fast they are completed.

To support the modeling and execution of workflows in HPC/HTC environments, parallel Scientific Workflow Management Systems (SWMS) were developed, such as Swift/T [5], Pegasus [6],ASKALON [7], Triana [8], and Chiron [9], or SWMS embedded in science gateways, such as WorkWays [10]. To foster data parallelism, workflow activities can be instantiated as tasks for processing each input data, known as activations [9]. Each activation executes a specific program or computational service in parallel, consuming a set of parameter values and input data, and producing output data. Besides the activations, a parallel SWMS controls data dependencies among activities. Scheduling, parallel processing and provenance support are some of the advantages of using a SWMS as opposed to executing workflows using Python scripts or Spark [11].

Workflow execution in these SWMS may consume largeamounts of resources and as the workflow becomes increasingly IO- and compute-intensive (measured by the number of activations or the volume of processed data), they tend to execute for days, or even weeks, depending on the amount of input data and the availability of computational resources. Thus, it is important to be aware of execution status and to analyze partial results to check if the current execution complies with some pre-defined quality, performance or financial criteria. Consequently, analytical capabilities of the solutions based on offline analysis are not able to do monitoring, since users can only investigate data after the workflow execution ends and they are not aware of partial results at runtime. With online analysis [12] and data steering support [13], users may decide if they have to interfere in the execution (also known as dynamic workflows). Data steering at runtime allows for the analysis of partial results and to potentially identify failures and problems as early as they happen. Data steering is a common requirement in scientific simulations. In fact, several visualization tools such as ParaView already provide plug-ins so that users can analyze partial results using filters and do in situ visualization [14].

Most of the parallel SWMS have addressed the need of performance and resource consumption monitoring facilities by adding new components to their workflow engines, or loading data into databases (at runtime or after the workflow execution) to be further queried by users [13]. In 2008, Swift [15] adopted Kickstart [16] to successfully improve workflow monitoring and debugging using execution data combined to provenance data. Kickstart has also been coupled to SWMS Pegasus [6] to monitor the execution of workflows in HPC environments at runtime. These monitoring mechanisms can provide information such as the execution time of each activity or activation of the workflow, how many activations were executed, and resource usage. This type of information is very useful to help users to debug an execution or to understand the performance of a workflow in a specific environment. However, this coarse grain provenance information has limitations in monitoring and debugging. In scientific workflows, particularly IO-intensive ones, data values of interest are inside files and provenance just registers input and output of activities, which in this case would be the file name [16]. This is useful for understanding the data derivation behavior (i.e., dataflow file path), but the user still has a lot of additional effort to parse the files to trace and analyze data elements inside datasets.

In previous papers, we have experienced the benefits of domain-data steering with execution data, when running workflows in different domains such as engineering and biology. For example, in [17], the user was able to analyze partial results of the workflow, thus allowing for stopping it or changing parameter values to continue executing. However, one would not be able to check if some problem, error or low-quality result was influenced by environment issues such as low memory or bandwidth. This prevented users to fine tune parameters and avoid these environment problems. In [18], the user was able to verify that a task was taking longer than average and was able to define which parameter and files were corresponding to the anomaly, while running in the cloud environment. However, the user could not be sure if the long-time problem was related to CPU or IO. This limitation of our previous approach also occurs in Kickstart based solutions. For example, in case the long execution is due to CPU, it would be consistent according to the input data, however, if the problem was on IO, then the user should abort the execution. These scenarios motivated us to improve our runtime steering support with HPC/HTC monitoring and debugging tool to complement the existing analytical power with code profiling tools. When users are left with separate data analytic tools each with their own data it is difficult to relate them. It does not make sense to reproduce all that HPC/HTC debugging tools already do, into a SWMS. Therefore, a solution that connects both can benefit from both the SWMS monitoring and debugging as well as the code profiling tool.

In this work, we developed a SWMS component for capturing performance and resource consumption metrics that is designed on top of TAU [19] and SAR [20] tools, which we named as PerfMetricEval. Without PerfMetricEval, we would need to browse the generated files to extract and analyze performance data from TAU and SAR log files. Therefore, we would have to gather those data from log files and develop programs to relate performance and resource consumption data captured from the log with domain data extracted from raw data files [16]. Different from this manual approach, our component enables online query processing considering data obtained from log and raw data files, taking advantage of existing code-profiling tools, such as TAU and SAR. In addition, we coupled PerfMetricEval to Chiron SWMS to present a relational database that integrates provenance, domain, execution, performance, and resource consumption data. This integrated database enables users to perform their analysis by submitting SQL queries to the RDBMS. Therefore, we can enumerate our contributions as:

• The development of PerfMetricEval component to capture performance and resource consumption data using TAU and SAR profiling tools;

• A specialization of PROV-Df provenance database schema to include relevant data from the code-profiling tools;

• Integration of PerfMetricEval with an existing SWMS. In this case, we use Chiron SWMS.

We have presented this database-oriented approach in a paper at the Workflows in Support of Large-Scale Science (WORKS) 2016 Workshop [21]. In our previous paper, we presented workflow elapsed time and CPU usage metrics, and performed experiments in a cluster environment. In this paper, we go deeper in describing the components of PerfMetricEval, in particular we detail memory consumption and disk operations. We also extend [21] with new experiments to show the performance and analytical power of PerfMetricEval using Amazon EC2 cloud. These new results show how the gathered data can improve virtual machines provisioning in a cloud environment.

This paper is organized in five sections. Section 2 presents a motivating example. Section 3 discusses related work. Section 4 describes our approach for performance and resource consumption monitoring; the extended provenance database schema that integrates provenance, domain, performance and resource consumption data, all together; and the integration between PerfMetricEval with Chiron SWMS. Section 5 shows the evaluation of the proposed approach using the Montage workflow in a cluster environment and the SciPhy workflow in a cloud environment. Finally, we conclude the paper in Section 6.