A novel independent job rescheduling strategy for cloud resilience in the cloud environment

1. Introduction

Although a cloud data center often has its own proactive fault handling plan, the threat of unplanned outages still exists [1–4]. Therefore, cloud resiliency is an important issue to successfully keep a stable cloud computing environment under fault scenarios [5]. To rescue a series of operations from a fault event, a variety of fault handling approaches have been proposed [6, 7]. Several parameters such as CPU temperature [6], cloud performance [8] and job importance [9], have been considered in these fault handling approaches. Job execution duration and job deadline, as two of the most significant job attributes, has also been considered when performing job scheduling, job rescheduling and resource allocation. Completing the job beyond its deadline is meaningless [10, 11]. From the point in time when a fault occurs, uncompleted jobs with the deadline requirements are particularly at risk of failing to meet their deadline requirements [12, 13].

Many deadline-constrained job scheduling strategies have been proposed. The HEFT series approaches are one of the most significant series of job scheduling strategies published from 2002 to date [14–23]. Although HEFT series approaches were proposed over past decade, selecting the first available server to enable job to finish early might not be the optimal solution when handling faults [15, 21, 24]. It may cause unnecessary deadline or resource competition between the job with high priority and the job with low priority. As a result, the cloud resiliency may not be optimized with many low priority jobs unsaved. Besides, selecting the first available server may cause a temporary dramatic load increase at some specific time points, which leads to performance bottleneck.

Therefore, in this paper, we propose a novel independent job rescheduling strategy for better cloud resiliency and load balancing performance. Our approach concentrates on independent job rescheduling based on job nature, timeline scenario and overall cloud performance. A job parsing system is deployed to identify the eligible time slots for the jobs. Then a priority assignment system is developed to prioritize the jobs in the cloud environment. To handle different cases, two sub-algorithms are applied in the proposed dynamic job rescheduling algorithm. The simulation results show that our proposed dynamic job rescheduling strategy has better cloud resiliency and load balancing performance than the HEFT series approaches. Besides, our approach can fit both the single-fault scenario and multi-faults scenario when handling faults.

2. Related work and problem statement

The cloud environment is subject to many types of faults, which might lead to a data center being unstable or even unavailable. These faults will disrupt uncompleted jobs [12]. Therefore, it is crucial for a data center to handle faults and rescue uncompleted jobs when faults occur [25, 26]. Fault tolerance can be approached from two different perspectives, proactive fault tolerance and reactive fault tolerance [27]. The main objective of fault tolerance techniques is to rescue the jobs from the faulty data center to working-proper replica-ready data centers [28, 29].

The deadline is one of the most significant parameter to be considered in some contemporary fault handling approaches. The HEFT series approaches are one of the most significant series of deadline-constrained job scheduling strategies, which are published from 2002 to date [14–23]. In 2002, a Heterogeneous Earliest Finish Time (HEFT) algorithm was proposed to minimize its earliest finish time with an inserted-based policy [14]. It firstly assigns the priority to each job in the scheduling list and then assigns each task to the first available server which can enable the job to finish the earliest. In 2013, a Budget and Deadline Constrained scheduling algorithm named BEFT was proposed to find the optimal workflow scheduling solution to satisfy both deadline and budget constraints for avoiding SLA violations [16]. Specifically, the BEFT algorithm only works by reserving and billing a fixed number of resources in heterogeneous grid computing systems. Same year in 2013, another novel list-based task scheduling algorithm called Predict Earliest Finish Time (PHEFT) was proposed to improve the makespan and efficiency to compare with the HEFT, LDCP and LHEFT approach. At the same time, this algorithm keeps the same time complexity to HEFT approach. In 2015, a Budget and Deadline Constrained Heterogeneous Earliest Finish Time (BDHEFT) algorithm was proposed by extending the classic HEFT approach and the BHEFT approach [18]. The BDHEFT approach considers six major variables, such as Spare Workflow Budget, Spare Workflow Deadline, Current Task Budget, Current Task Deadline, Budget Adjustment Factor and Deadline Adjustment Factor, to generate a Budget and Deadline Constrained scheduling plan. In 2016, an Enriched-Look ahead HEFT (E-LHEFT) algorithm was proposed to optimize both QoS and load balancing without considering any constraints [19]. E-LHEFT algorithm updates the processor selection phase of LHEFT algorithm by applying task grouping and Pareto theory for effective load balancing performance. In 2018, jobs with both unconstrained and time deadline constrained cases were taken into account by applying a HEFT technique for order preference called HEFT-T algorithm [20]. A three-stage non-dominated sorting approach is applied to identify the optimal solutions for the unconstrained case, and an adaptive weight adjustment strategy is proposed to adjust weight value for time for the deadline-constrained case. In 2019, a workflow scheduling algorithm named Greedy Resource Provisioning and Modified HEFT (GRP-HEFT) was proposed by developing a resource provisioning mechanism [22]. The resource provisioning mechanism generates the instance type list based on the efficiency ratio of different instance types and selects the most efficient instances constrained by pre-defined budget. The modified HEFT algorithm employs the optimal configuration of instance types with their number of created VMs to obtain the job scheduling plan. Same year in 2019, a Dynamic Variant Rank HEFT (DVR-HEFT) algorithm was also proposed to reduce the scheduler's makespan without increasing the algorithm's time complicity to compare with the classic HEFT approach [23].

The HEFT series approaches tend to select the first available server to enable the earliest finish time. Although HEFT series approaches were developed over a long time period, selecting the first available server might not be the optimal configuration when handling faults [15, 21, 24]. It may cause anabatic deadline or resource competition problems in which the job rescue with the high priority may unnecessarily impact the job rescue with the low priority. Therefore, the cloud resiliency may be degraded. Besides, selecting the first available server may cause a temporary dramatic load increase at some specific time points. Therefore, the timeline processing should be further considered because of the time-varying characteristic of the entire timeline. Firstly, different time-varying resource situations at different cloud data centers should be taken into account. Secondly, the job deadline competition should be addressed when allocating the jobs. Furthermore, the load should be more balanced to avoid traffic spikes and degraded cloud performance.

3. General modeling

In general, a cloud environment contains multiple components such as data centers, servers, data and jobs. In this research, the proposed independent job rescheduling strategy is developed to globally control the job allocation over the cloud environment. We recognize a data center as an independent scheduling entity when doing job allocation. The job scheduling and rescheduling is conducted among a set of data centers. How the jobs are allocated into servers or virtual machines in a specific data center, is not the focus of this research. Instead, we study a cloud environment E=(DC,J) with multiple data centers DC {dc1, dc2, …, dcp} and a set of jobs J {j1, j2, …, jg}.

Each job j∈J is associated with R(j), DEAD(j) and PRO(j), which present resource requirement, job deadline and job operation profit, respectively. Each job j has a fixed job execution duration Len(j). The job urgency value of the job j, ρ(j), refers to the time buffer between current time point and its deadline DEAD(j). In this research, we define the job deadline as a specific point in time without the consideration of the job with infinite deadline. Completing a job beyond its deadline is meaningless. All jobs in this research are independent, which means there is no dependency among the jobs. Besides, we assume all jobs require restarting of the entire job as the traditional job rescheduling approach did [30]. This is because the major concern of this paper is not the way how the job is resumed. The main objective is on how to maximize the overall cloud resiliency and balance the resource load by analyzing the job priority. The traditional restart approach is also the most general approach that is supported by any cloud for any task.

In this research, we focus on the improvement of the cloud resiliency. The cloud resiliency for a faulty data center can be calculated using the formula in Eqn (1).

4. Job parsing system

We develop a job parsing system to find the eligible time slots for jobs. A timeline exists at each data center. In this research, we do not consider the job with an infinite deadline. Therefore, the timeline range refers to [T0, TLatest], where T0 denotes the fault-occurred time point and TLatest denotes the latest deadline time point of the jobs in J.

We define the time slot as a series of continuous time points. The available resource at each time point is the most significant factor for the further reception of the rescheduled jobs from the faulty data center. Therefore, we parse the timeline at each data center site in a two dimensional vector space. The x axis is the discrete time points ranged from [T0, TLatest] and the y axis is the available resource. Thus, the line in this space represents the available resource over time. We call it a resource line. Each job can be parsed into this two dimensional vector space as a rectangle. The height of the rectangle represents the resource requirement of the job and the length of the rectangle corresponds to the job execution duration. The rectangle will horizontally move from T0  to TLatest. An eligible time slot for a job starts from a time point when the rectangle starts to stand completely below the resource line and ends at a time point when the rectangle starts to stand above the resource line. A function Count(S(j)dc) is deployed to count the number of eligible time slots of the job  j at a data center dc. An example of our proposed job parsing system is shown in Figure 1. The final range of the eligible time slot can be recognized when the job rectangle completely stands under the resource line (red line).

5. Job rescheduling strategy

Our job scheduling strategy has three phases, replica selection phase, job prioritizing phase and eligible time slot selection phase. Our strategy is an independent job rescheduling strategy for a bounded number of data centers when faults occur. In the case of single-fault scenario, our strategy can be applied by the faulty data center in one time to rescue the faulty jobs. While in the case of multi-faults scenarios, our strategy should be separately applied in each faulty data center. The main objective of our job rescheduling strategy is to maximize the cloud resiliency and balance the resource load at the same time by applying a priority-based job rescheduling method. Therefore, the starvation of the less privileged jobs may be sacrificed to some extent as our proposed job prioritizing method aims to obtain the job importance by analyzing the job urgency, the job operation profit and the number of eligible time slots of the job. The job urgency describes the urgency degree of the job. The job operation profit demonstrates the economic value of the job completion. The number of eligible time slots of the job denotes the job allocation difficulty. Therefore, the job with higher priority means the job has higher urgency degree, higher economic value and higher job allocation difficulty. In other words, the job with lower priority means the job has lower urgency degree, lower economic value and lower job allocation difficulty. Therefore, from the relation among urgency degree, economic value and job allocation difficulty, the less privileged jobs are less important to be saved. Therefore, the resource will be firstly available to the jobs with higher priority and then the starvation of the jobs is sacrificed to some extent.

6. Simulation results

To evaluate our proposed approach, we performed three simulations on OMNeT++ 5.4.1. We make the following assumptions in our simulations:

1. Traditional three-replicas strategy is deployed.

2. The latency among data centers is insignificant.

3. All data centers are inter-connected.

4. Bandwidth is set as consumed resource.

5. Wδ and Wσ is set to be 0.5 each to simplify the problem.

We implemented three types of workflows, Montage scientific workflow, LIGO Inspiral Analysis workflow and SIPHT program. Each workflow is seen as an independent job instance. The details of these workflows are adjusted and referenced from [31]. We measure the cloud resiliency in all three simulations and the load situations specifically in Simulation 2 to compare with the job scheduling method of the HEFT series approaches. We compare the performance of our approach to the average performance of HEFT series approaches.

7. Conclusions and future work

To conclude, the HEFT series approach is one of the most significant deadline-constrained job scheduling approaches. But selecting the first available server might not the optimal configuration when handling faults. In this paper, we propose a novel job rescheduling strategy for better cloud resiliency and Load balancing performance. This approach concentrates on independent job rescheduling based on job nature, timeline scenario and overall cloud performance to handle the job rescue from the faulty data center. A job parsing system and a priority assignment system are developed to identify the eligible time slots for the jobs and prioritize the jobs, respectively. Two algorithms (Algorithm 2 and 3) are proposed to support the proposed dynamic job rescheduling algorithm (Algorithm 1). The simulation results show that our proposed strategy has better cloud resiliency and load balancing performance than the HEFT series approaches. Besides, both single-fault scenario and multi-faults scenario can adopt our strategy. In the future work, we propose to further develop a fault handling approach for dependent jobs. Apart from that, the check pointing method will be considered instead of the restart method for tasks in this approach.