Optimization of energy consumption in cloud computing datacenters

ABSTRACT

workload prediction. Then, the algorithm modifies and optimizes the set of available servers to best fit the predicted workload. The final step is to schedule the jobs to the available processors using a modified version of Round Robin scheduling method. The authors then simulated their proposed solution and tested it against other online and offline scheduling algorithms and presented their results with a recommendation on which type of scheduling to use for different types of workload. Another technique that is used with scheduling to minimize the energy consumption in cloud computing environment is dynamic voltage and frequency scaling (DVFS). In [14] a DVFS based green energy-efficient scheduling algorithm was proposed. It works on increasing the utilization factor of the different servers in the datacenter in an effort to reduce its energy consumption. The environment consisted of heterogeneous servers with each server having different processor, memory, and storage specifications. Server's has frequency is varied or reduced to minimize the energy meeting the requirements of the workload. Next, a priority-based scheduling algorithm is used to assign jobs to created VMs. Experimental results indicated that the proposed algorithm reduce the energy consumption by a range that varies from 5 to 25% while maintaining the required performance and execution time.
Various combination of DVFS and node vary-on vary-off (VOVO) methods were evaluated in [15]. The objective of the research was to minimize the power consumption in clusters of servers during times where the workload is reduced. The authors explored five different policies with different degrees of complexity at the implementation level. The five policies are:  Independent voltage scaling (IVS): uses processors that have voltage scaling property and can independently change their voltage and frequency according to their workload.  Coordinated voltage scaling (CVS): exploits voltage scaled processors but coordinate their voltage scaling algorithms.  Vary-on vary-off (VOVO): turning the whole server on or off depending on the size of the workload.  Combination of IVS and VOVO.  Combination of CVS and VOVO.
These different policies were studied and compared with a combination of VOVO and CVS producing the most savings in energy consumption.
In [16] authors defined an energy consumption ratio (ECR) to evaluate the efficiency of different frequencies under which to execute a task. They converted the energy-efficient task scheduling problem into minimizing the total ECR. Next they transformed the problem to the variable size bin packing, and showed that the minimization of ECR is NP-hard and proposed propose a task allocation and scheduling methods based on the feature of this problem. They also proposed a processor-level migration algorithm to reschedule remaining tasks among processors on an individual server and dynamically balance the workloads and lower the total ECR on a given server. In [17], researchers propose a consolidation algorithm which favors the most effective migration among Virtual Machines, containers and applications; and investigate how migration decisions should be made to save energy without any negative impact on the service performance.
Linear programming is a mathematical method for maximizing or minimizing a linear objective function under certain conditions or constraints consisting of different variables. Integer linear programming (ILP) is a special type of linear programming where some or all the variables in the objective function and constraints are limited to integer values. ILP problems can be categorized into two categories: 0-1 ILP problems where the variables in the problem are binary variables and have values of 0 or 1, and Generic ILP where the variables are not limited to binary variables [18].
In this work, the generated ILP model will be transformed into a boolean satisfiability problem (SAT) equations. These equations will be solved using SAT solvers and the result will be mapped back to the original scheduling problem. Boolean satisfiability problem refers to finding a satisfying assignment for a problem or proving that none exist. A satisfying assignment for a certain Boolean formula is an assignment of the binary variables in the formula to 0 or 1 to make the overall formula evaluate to 1. The assignment must satisfy all the constraints of the problem simultaneously [19]. Both the SAT problem and the 0-1 ILP problem are represented using binary variables but in SAT, the Boolean formulas used to represent the problem and its constraints must all be in a product-of-sums form or a conjunctive normal form (CNF) and the operators used in these forms are logical operators (AND, OR, and NOT) while in 0-1 ILP problem the constraints are simple inequalities of the form ≤ where ∈ ℤ , ∈ ℤ × ℤ , and ∈ {0,1} . These constraints are referred to as Pseudo Boolean (PB) constraints and they are considered as a general form of the CNF constraints [20].
The SAT problem is considered to be NP complete problem. The solving time increases exponentially with the increase in the number of variables. The SAT solvers are algorithms that determine if a SAT problem is satisfiable or not and find a satisfying assignment in case the problem was satisfiable [19][20][21].
In the past few years the use of SAT models to represent different problems in various domains increased 689 rapidly due to the huge improvement in the SAT solving algorithms and the availability of better and more powerful computing capabilities. One of the great improvements to SAT-based solvers is the extension to handle PB constraints because they are easier to write and understand and can replace an exponential number of CNF constraints. Before this extension SAT models were limited to decision problems but with the help of PB constraints SAT can now model optimization problems and solve them using the different SAT-based PB solvers [20]. Pseudo Boolean SAT Solvers: which are basically SAT solvers that can handle pseudo Boolean constraints allowing the SAT solvers to be used in solving optimization problems. In [22], for example, two power estimation problems were formulated as SAT problems and evaluated using generic ILP and SAT-based solvers. It was shown that the SAT based solver performed better than random based approaches in terms of finding the optimal values of the proposed problem.
In this work, PB-SAT solvers are used to solve the proposed optimization model. The testing process involves using different solvers and comparing their results. The solvers are selected according to the following criteria:  A proven performance of the solver from the reported literature.  The ability of the solver to handle large problem size.
Based on the above criteria, we identified three PB-SAT solvers, namely:  NaPS [23]: Nagoya Pseudo Boolean Solver, is a solver for PB constraints that are linear and contain binary variables only. It was the best performing solver in the Pseudo Boolean competition 2016 in both categories of optimization with small integers and large integers and was among the best solvers in the other categories. The information and results of the competition is available in [24].  Minisat+ [25]: is a modified version of the Minisat solver to be able to deal with pseudo Boolean constraints, and despite being a simple solver it performed well in the pseudo Boolean competition through the years.  Sat4j [26]: is a Boolean reasoning java library that is used to solve Boolean satisfaction and optimization problems. The sat4j solver is different from the other two solvers because it does not prioritize being the fastest in solving a problem. However, the solver is full featured, robust, and user friendly. The solver design follows the Java design guidelines and code conventions. Furthermore, Commercialized and open source optimization tools with various embedded algorithms exist today. Those tools deal with generating a solution to a defined linear, non-linear, or integer models. Some of the popular tools include:  CPLEX [27]: is a high-performance solver for linear programing, mixed integer programing, and quadratic programing. It is a well-known and wildly used commercial solver developed over 20 years ago and is currently owned by IBM. The solver uses the Simplex exact algorithm, and the name CPLEX is actually taken from C-Simplex which is Simplex algorithm implemented in C language. The solver is improved and updated periodically to increase its performance and include different algorithms for different types of problems. some of the various methods used in CPLEX are: primal simplex algorithm, dual simplex algorithm, network simplex algorithm, and the barrier method. This solver will be used in testing the proposed model.  LINGO [28]: is an efficient comprehensive tool used in building and solving different types of optimization models. It was developed by LINDO Systems as an optimization modeling software for linear, nonlinear, and integer programming. LINGO provides its users with a language to express the optimization models, an environment to build and edit their problem, and a set of solvers. These solvers include but not limited to: Primal and dual simplex solvers, Barrier solver, Integer solver, General nonlinear solver, and Quadratic solver. The rest of the paper is organized as follows: in Section 2 we discuss the research methodology followed in this work; Section 3 includes the results obtained and a thorough discussion of it, the paper is concluded in Section 4.

RESEARCH METHOD 2.1. Environment setting and model formulation
This section presents the characteristics of the cloud computing environment, analyses the scheduling problem in the cloud datacenter, and formulates the scheduling problem as an integer linear program (ILP) model.
A cloud computing environment may consist of a single or multiple datacentres that may belong to a single or multiple providers. This work deals with a single cloud datacentre that belongs to a single provider. The cloud datacentre infrastructure contains three resource types: compute, storage, and network resources. Our aim in the work presented here is to focus on the computing component of the resources used since it is shown to consume about 26% of the total energy of a datacentre [17]. The main computing units are the servers. The proposed model is formulated based on the following assumptions:  Each CPU can run two or four tasks at the same time depending on number of cores it contains (dual or quad core); each core will run only one task at 100% utilization.  A server is turned ON if at least a single core in one of its CPUs is executing a task and is switched OFF otherwise.  A core is either ON (switched on and executing a task), idle (switched on but not running any task), or completely OFF (switched off).  Each core is assumed to have two running modes: High and Low.  In High mode, the core runs with the maximum available frequency and voltage, while in Low mode, the core runs at the minimum available frequency and voltage.  Cores inside the same server can run in different modes.  Tasks are allocated in batches that arrive at distinct times (t 0 , t 1 , …, t n ).  All the tasks in a specific batch will be scheduled immediately and will start running simultaneously (no task will be assigned to start later).  A task can only be executed by a core that meets its frequency and memory requirements.  For any task there is at least one core that meets the requirements of the task (no task will be rejected).  The execution of a task on a certain core is assumed to consume a specific amount of energy that will be called the energy cost in this model.  Ne pre-emption of tasks is allowed.

Optimization model formulation
This section discusses the integer linear programming (ILP) model that minimizes the energy consumption of the servers in a cloud datacentre. The intended objective is to find the specific assignment of tasks to cores that gives the minimum energy value. The total power consumption is the power consumed by the cores when a task is allocated to it plus the power consumed by the idle cores. Table 1 presents the variables used in developing the model and their meaning. Figure 1 illustrates a simple datacentre configuration with two servers S 1 and S 2 . S 1 has two dual-core CPUs and S 2 has one quad-core CPU as shown. A server has two states either ON or OFF. Depending on the server state, each core is assumed to be either active, idle, or switched off, as represented by the green, blue, and red coloured-squares respectively. In the scenario depicted by Figure 1, all cores were initially off, then two tasks are assigned to C 1,1,1 and C 1,2,1 with a cost of X 1,1,1,1 and X 1,2,1,2 respectively. Binary variable equals to 0 if CPU j in server i is not active and 1 otherwise. , Binary variable, equal to 1 if CPU j in server i is idle and 0 otherwise. , , Binary variable, equal to 1 if core k in CPU j in server i is active, and 0 otherwise.

, ,
Binary variable, equal to 1 if core k in CPU j in server i is idle, and 0 otherwise. , , , Binary variable, equal to 1 if task n is assigned to core k in CPU j in server i, and 0 otherwise. The energy cost value of assigning task n to core k in CPU j in server i in Low mode.

, , ,
The energy cost value of assigning task n to core k in CPU j in server i in High mode.

, ,
The energy cost value of core k in CPU j in server i when it is idle.
The minimum amount of RAM requested by task n. The minimum amount of Frequency requested by task n.

, ,
The available amount of RAM in core k in CPU j in server i.

, ,
The available Low frequency in core k in CPU j in server i.

, ,
The available High frequency in core k in CPU j in server i.
We describe the formulation steps taken to develop the comprehensive model. Initially, the formulation commences with an objective function whose primary goal is to minimize the energy consumption when assigning tasks to cores in a datacenter. From Figure 1 we observe that the energy consumption comes from the energy cost of active cores. This energy cost consists of the cost of cores running in Low mode ( , , , ) and the cost of cores running in high mode ( , , , ). In addition to active cores there is  ). Furthermore, the off cores are assumed to have zero energy needs. This is represented as follows in (1): The objective function described in (1) has three parts, the first two parts represent the energy consumed by the active cores. The active cores energy is calculated by summing the energy cost of all the cores that are running a task across the whole datacenter. The third component of the objective function sums the energy consumed by the remaining idle cores in the datacenter. This objective function is subject to the following set of constraints:  If a core k is active, then the CPU j that contains the core k is considered active as well.
 If a CPU j is active, then at least one of its cores is active.
 If a CPU is active, all the cores inside the CPU must be either active or idle but not switched off.

Dynamic scheduling
Following the scheduling of the first batch at time t 0 , in the second phase of this work, we assume the arrival of a new batch to be scheduled at time t 1. Here, the remaining free cores are used. Following the scheduling of the first batch and while preparing to schedule the second batch, the constraints below are added:  All the cores that are still running a task from batch t0 are marked as busy, where is the number of tasks from the first batch that are still running at t1.  If a core , , is busy then no task from the second batch is assigned to it, where is the number of the new tasks in the second batch at t 1 .  The total number of active cores is now equal to the total number of tasks to be scheduled plus the number of tasks that are still running from the first batch.

RESULTS AND DISCUSSION
The generated optimization model of (1) is validated and tested using SAT-based 0-1 PB solvers named NaPS, Minisat+, and Sat4j, and the generic ILP solver CPLEX. The solvers compute the minimum energy consumption of the datacenter and the corresponding assignment of tasks to the cores that would achieve this power. To test the model, several datacenter instances were created with different number of servers, different number and types of CPUs (dual-core or quad-core) in each server, and different number of tasks to be scheduled. The tests are carried out as described in the following sections.
The comprehensive model in (1) is tested using the three PB-SAT solvers and CPLEX. The testing instances used consist of a number of servers with each server containing dual or quad core CPUs. The instances also include the number of tasks to be assigned to the cores of the datacenter. The tasks to be scheduled have several characteristics including: task's size (in number of instructions), task's required frequency, and task's required RAM. Each core in the datacenter runs in one of two modes: High, or Low. As mentioned earlier, in High mode the core runs at the maximum frequency and voltage allowed, while in low mode, the core runs at the minimum frequency and voltage possible. The testing instances are generated randomly from a pool of servers. The tasks requirements are created depending on the tasks' size. Table 2 shows the task size ranges and the corresponding tasks' requirements. Table 3 includes partial servers' specifications. Table 4 lists the instances created for this test. Each setup is tested twice using different number of tasks. The instances are first solved using all the solvers to find the minimum energy consumption and compare the solvers' performance. Then, the same instances are solved using round robin and maximum possible value methods to compare their performance against the proposed model.   In the first part of the test, the PB-SAT solvers and CPLEX solved the testing instances to find the minimum energy consumption and the results are presented in Table 5. All solvers managed to find the minimum energy consumption in the first six instances. The CPU times for the PB-SAT solvers are relatively close to each other, with NaPS taking the lead in most of the instances. However, if we include CPLEX in the comparison we can see that except in instance 1 it was always faster than the other solvers with an appreciable difference as can be seen in Figure 2.

Figure 2. The performance of the PB-SAT solvers and CPLEX in Test A
In the last two instances (7 and 8) the PB-SAT solvers are stopped after exceeding the 1000 second mark. The values between the brackets in Table 5 are the best solution found by the solvers up to that point in time. Only CPLEX have successfully found the minimum energy consumption in those instances and within less than 5 seconds. From Figure 3 we notice that the minimum energy consumption is directly related to the total number of instructions of the tasks to be scheduled. The time taken by NaPS is proportional to the total number of instructions as well. Table 6 depicts the results of the second part of the test where the proposed model is compared with maximum possible value (MPV) and round robin (RR). We observe that overall the proposed model has consistently returned the minimum energy consumption when compared with the other two methods. The average amount of energy saved is 27.3% of the maximum energy consumption with a maximum of 35.6% energy saved in instance 7. We also notice that round robin method failed to schedule two of the instances (4 and 6). This can be attributed to the fact that round robin method schedules the tasks to the first available core that can execute these tasks even if the task is small and can be executed on some other lower capability core. This leads to larger tasks not finding any core that can be assigned to them because all the capable cores are executing smaller tasks and that exactly what happened in both instances 4 and 6.
The only way round robin can schedule the tasks of instances 4 and 6 is by waiting until one of the busy cores finishes its task. Hence, in all cases the proposed model proved its superiority. Figure 4 is a graphical representation of the results of Table 6.   In all of the analysis above we assumed that all the tasks arrive as one batch and scheduled to execute simultaneously at time t 0 . In this segment of our tests, we consider the case of another batch of tasks arriving at time t 1 . The new batch of tasks is to be assigned to the remaining free cores of the datacenter while minimizing the energy consumption. The tasks of the first batch (t 0 ) that are still running at t 1 will not be rescheduled. However, their energy consumption will be taken into account when calculating the overall energy consumption of the second batch. Table 7 shows the testing instances used with a second batch of tasks at time t 1 . Number of remaining tasks at t 1 refers to how many tasks from batch t 0 are still running when the second batch of tasks arrived at t 1 . Number of new tasks at t 1 is how many new tasks arrived at t 1 . Both the proposed model and round robin are used to schedule the tasks at t 0 and t 1 and the minimum energy consumption in each case is compared.  Table 8 shows the results obtained. The second column (Min Energy at t 0 ) is the energy consumption value of executing all the tasks in batch t 0. The minimum energy consumption at t 1 is the total of the energy consumed by the tasks that are still running from t 0 plus the energy consumed by the new tasks at t 1 and is listed under the fourth column. From the results we observe that the modification made to the original model has successfully allowed scheduling a second batch of tasks at t 1 while minimizing the overall energy consumption of the datacenter. We also observe that scheduling the second batch of tasks took less time in instance 4 and 5 because some cores were already busy running the remaining tasks from batch t 0 , and hence were removed from the pool of available cores. Further testing is carried out to compare the proposed modified model to round robin algorithm in scheduling a second batch of tasks. The results are tabulated in Table 9. In the small instances (instances 1 and 2), round robin and the proposed model results are close and even equal in some cases, but as the number of servers, CPUs, and cores increase, the proposed model results became better compared to round robin. Considering the largest instance tested (instance 5) we observe that round robin results are 235.7% worse than the proposed model. The difference between the two algorithms' results at t 0 and t 1 is illustrated in Figures 5 and 6.

CONCLUSION
In this work, the energy consumption of a datacentre is minimized using energy efficient tasks scheduling and allocation of resources. We proposed a framework to tackle the energy efficiency in the context of a datacentre operation through energy conscious scheduling and reduction in the number of active servers. The resources considered in this research are the computing resources only. We developed ILP optimization model for the task allocation and scheduling problem in a datacentre with the objective of assigning the tasks to the resources that would execute them with the least amount of energy consumption.
The model included tasks' characteristics such as: Task' frequency requirement, its size and RAM requirement. It also included two different running modes (high and low) for the cores inside the same server. The model was tested with different datacentre setups and compared with round robin and maximum possible value methods. It was found that the proposed comprehensive model was able to find the minimum energy consumption in all of the tested cases. Finally, the proposed model was modified to handle the arrival and scheduling of a second batch of tasks after the initial batch is already assigned to various servers in the datacentre. The performance of this model was compared to round robin algorithm, and it was evident that the modified model was able to allocate the two batches of tasks with less energy consumption than round robin.
Various datacentres configurations are studied using different scenarios. The results of these tests show that the energy consumption of a datacentre was found as expected to be proportional to the utilization level of the datacentre, that is, as the number of active cores increases the energy consumption as well increases. Upon testing the complete model and comparing it with round robin and maximum possible value methods, results demonstrated that the proposed model can save up to 35.6% of the energy consumption in some cases when compared to maximum possible value method. Furthermore, while round robin algorithm was not always successful in finding a solution for allocating all the tasks, still the proposed model outperformed it in the few cases where it did find a solution.
Another contribution in this work involved using the proposed ILP formulations to test the performance of three PB-SAT solvers namely: NaPS, Minisat+ and Sat4j, and a generic ILP solver named CPLEX in solving the energy optimization problem for datacentres. The performance of the three PB-SAT solvers was relatively close to each other, and they were able to find the optimum energy for most of the tested instances. However, as the size of the search space increased the PB-SAT solvers started to time-out. Therefore, for PB-SAT solvers the time needed to explore the search space makes them impractical for finding solutions for large scale datacentres. On the other hand, the generic ILP-solver CPLEX outperformed all the PB-SAT solvers and was able to handle relatively large datacentres configurations and find the optimal solution (the minimum energy consumption) in a shorter amount of time. Potential future directions of this work include extending the ILP models developed in this work to include scheduling of network and storage resources and also extending the proposed model to have a multi-objective function such as reliability and availability of services while optimizing energy consumption.