Energy-efficient Resource Utilization

in Cloud Computing

Giorgio L. Valentini1,2, Samee U. Khan1, and Pascal Bouvry2

1 North Dakota State University, NDSU-CIIT Green Computingand Communications Laboratory, Department of Electrical and

Computer Engineering, Fargo, ND 58108-6050,samee.khan@ndsu.edu

2 University of Luxembourg, Computer Science andCommunications Research Unit, Faculty of Science, Technology

and Communication, Kirchberg L-1359, Luxembourg,{firstname.lastname}@uni.lu

Abstract

In cloud computing systems, the energy consumption of the under-utilized resources accounts for a substantial amount of the actual energyuse. Inherently, a resource allocation strategy that considers the resourceutilization would increase the energy efficiency of the system. Task con-solidation is an effective technique that increase the system resource uti-lization. Recent studies reported that energy consumption (in servers)scales linearly with (processor) resource utilization. The aforementionedfact highlights the significant contribution of task consolidation to reducein turn the energy consumption of the system. In our study, we ana-lyze two existing energy-conscious heuristics for task consolidation. Bothheuristics aim to maximize resource utilization, with the main differencebeing whether the energy consumption to execute the given task is im-plicitly or explicitly considered. To improve the energy efficiency of taskconsolidation, we propose a bi-objective algorithm that combines the twoheuristics, in order to take advantage of both. According to our exper-imental results on cloud computing systems, the proposed algorithm in-creases the energy efficiency of the task consolidation problem, withoutany performance degradation when individually compared with the twoexisting energy-conscious heuristics.

1

1 Introduction

Nowadays, data communications are an important element of our daily lives.Most of our interactions rely on gathering the information through the client-server paradigm [7]. Over time, user demands have rapidly increased in terms ofthe number of requests. To cater to the consistent amount of requests, the com-putational capacities and facilities must be constantly reviewed and improved.As a drawback, the proportional nonnegligible amount of the required energyhas been often left behind to remain competitive.

The recent advocacy of “green” or “sustainable computing” (tightly coupledwith energy consumption) has been getting considerable attention. The scope ofsustainable computing goes beyond the main computing components, expandinginto a much larger range of resources associated with computing facilities asauxiliary equipment, such as the water used for cooling and the physical/floorspace occupied by the resources.

In cloud computing energy consumption and resource utilization are stronglycoupled. Specifically, resources with a low utilization rate still consume anunacceptable amount of energy compared to the energy consumption of a fullyutilized or sufficiently loaded cloud computing. According to recent studies([23], [28], [44], [5]), average resource utilization in most data centers can be aslow as 20%, and the average energy consumption of idle resources can be as highas 60% (or peak power). To increase resources utilization, task consolidationis an effective technique, greatly enabled by virtualization technologies, whichfacilitate the concurrent execution of several tasks and in turn reduce the energyconsumption.

Our study uses two energy-conscious heuristics for task consolidation pre-sented by Lee and Zomaya in [46]: MaxUtil that aims to maximize resourceutilization and ECTC (acronym for Energy-Conscious Task Consolidation —an overview of the most common acronyms used in our study is provided in Table1) that explicitly takes into account both active and idle energy consumption.For a given task, ECTC computes the energy consumption based on an objectivefunction derived from findings reported in the literature. As stated in the find-ings, the energy consumption can be significantly reduced while consolidatingtasks instead of being executed stand alone. Consequently, the two heuristics re-duce energy consumption without any performance degradation while assigninga given task to a selected resource.

To take advantage of both of the methods, while always considered sepa-rately, we propose to combine the heuristics in a bi-objective model. Identifyingthe resource offering the best compromise between the objectives will most likelytruly maximize the utilization rate while minimizing the energy consumption.The main idea of the proposed model being to execute the task on the optimal“energy-efficient” resource.

The remainder of this chapter is organized as follows. Section 2 overviewsthe related work. Section 3 details the cloud computing, energy models, andtask consolidation algorithms. The bi-objective approach and the related math-ematical model are described in Section 4 while the simulation results and the

2

discussions are summarized in Section 5 and Section 6, respectively. Section 7conclude our study.

2 Related Work

Energy efficiency is an emerging research issue, recently addressed by severalresearchers. For example, Khan and Ahmad in [38] were the first to use gametheoretical methodologies to simultaneously optimize system performance andenergy consumption. Since then several research works have used similar modelsand approaches, which have addressed a mix of research problems related tolarge scale computing systems, such as: energy proportionality, memory-awarecomputations, data intensive computations, energy-efficient, grid scheduling,and green networks ([37], [25], [10], [8], [4], [45], [19], [13], [29]).

Cloud computing and green computing paradigms are closely related and aregaining more concerns. The energy efficiency of cloud computing became one ofthe most crucial research issues. Advancements in hardware technologies [41],such as lowpower CPUs, solid state drives, and energy-efficient computer moni-tors have relieved the energy issue to a certain degree. Meanwhile, a considerableamount of software approach researches were conducted such as: scheduling andresource allocation ([45], [22], [9], [32], [18], [20], [30]), task consolidation ([35],[47], [21], [33]).

Virtualization technologies are a key component within task consolidationapproach. Parallel processing have been greatly eased and boosted with theprevalence of many-core processors. That is, multiples tasks are often ran on asingle many-core processor. The parallel processing practice seems at a glanceto inherently increase performance and productivity. But the trade-off betweenthe aforementioned increase and the consequent energy consumption shouldbe carefully investigated. For example, the load imbalance (especially in themany-core processors) is a major source of energy drainage that has motivatedmultiples task consolidation studies ([35], [47], [21], [33]).

Srikantaiah et al. in [35] approached the task consolidation using the tradi-tional bin-packing problem with two main characteristics: (a) CPU usage and(b) disk usage. The proposed algorithm consolidates the tasks relying on thePareto front to balance the energy consumption and the performance. The algo-rithm incorporates two main steps: (1) the determination of the optimal pointsfrom the profiling data and (2) the “energy-aware” resource allocation using theEuclidean distance between the current selection and the optimal point withineach server.

Song et al. in [47] proposed an utility analytic model for Internet-orientedtask consolidation. The model considers task’s request for Web services such ase-books database or e-commerce. The proposed model aims to maximize theresource utilization and to reduce the energy consumption, offering the samequality of services proper to the dedicated servers. The model also measurethe performance degradation of the consolidated tasks through the introduced“impact factor” metric.

3

The task consolidation mechanisms detailed by Torres et al. in [21] andNathuji et al. in [33] deal with the energy reduction using unusual approaches,especially in [21]. Unlike typical task consolidation strategies, the approachused in [21] adopts two interesting techniques: (1) memory compression and(2) request discrimination. The first enables the conversion of the CPU powerinto extra memory capacity to allow more (memory intensive) tasks to be con-solidated, whereas the second blocks useless/unfavorable requests (coming fromWeb crawlers) to eliminate unnecessary resource usage. The VirtualPower ap-proach proposed in [33] incorporates task consolidation into the power man-agement, combining “soft scaling” and “hard scaling” methodologies. Thetwo methodologies (of [21] and [33]) are based on power management facilitiesequipped with virtual machines and physical processors, respectively.

More recently, several noteworthy efforts on energy-aware scheduling (inlarge scale distributed computing systems as grids) using game theoretic ap-proaches have been reported ([34], [38]). Subrata et al. in [34] propose a co-operative game model and the Nash Bargaining solution to address the gridload balancing problem. The main objective is to minimize energy consump-tion while maintaining a specified service quality (i.e. time and fairness). Both,[38] and [34], deal with independent jobs through (semi-)static scheduling modeleveraging DVFS technique to minimize the energy consumption. (For recentliterature reviews the reader is referred to [1], [11], and [12].)

3 The Energy Efficient Utilization of Resourcesin Cloud Computing Systems

3.1 The Cloud Computing

The underlying system consists of a set R = {r0, . . . , rm−1} of m resources(processors) that are fully interconnected in the sense that a route exists betweenany two resources. It is assumed that resources are homogeneous in terms ofcomputing capability and capacity. The aforementioned is achieved throughthe virtualization technologies [14]. Nowadays, as many-core processors andvirtualization tools are commonplace [14]. The number of concurrent tasks ona single physical resource is loosely bounded and a cloud computing can spanacross multiple geographical locations.

The cloud computing model we consider, is assumed to: (a) be confinedto a particular physical location, (b) have the inter-processor communicationsperforming with the same speed on all links without substantial contentions,and (c) allow messages to be transmitted from one resource to another while atask is being executed on the recipient resource.

3.2 Energy Model

The energy model is based on the fact that processor utilization has a linearrelationship with energy consumption. The proportional relationship means

4

Table 1: The most common acronyms used in this book chapter

Acronym Description

aj Arrival time of a taskBTC Bi-objective Task Consolidation (algorithm)d Distance between two pointsD Solution set in the two-dimension search spacedj Due date of a taskδ Normalized complement of the distance result (d)

ECTC Energy-Conscious Task Consolidationej Energy consumption of a task on a resourceεi Minute energy factor of a resourceEi Energy consumption of a resourceER Energy consumption of the systemF Subset (of equivalents solutions) of Dfi,j Generic cost functionfx Normalized result of the ECTC cost functionfy Result of the MaxUtil cost functionm Number of resources

MaxUtil Maximum (rate) Utilizationm(p) Objectives vectorn Number of tasks

normx Normalization functionpi Point in the two-dimensional search space

pmax Power consumption at peak loadpmin Power consumption in the active modep∆ pmax − pminri ith resourceR Set of resourcestj jth taskT Set of tasksui,j Resource usage of a taskUi Utilization rate of a resourceUR Utilization rate of the systemτx Generic time periods of the ECTC cost functionτ0 Total processing time of a task on a resourceτ1 Time period where a task is run aloneτ2 Time period where a task is consolidatedλµ Output / Input (ratio)

[xmin : xmax] ECTC unit range[ymin : ymax] MaxUtil unit range

5

that, for a particular task, the information on the processing time and theprocessor utilization is sufficient to measure the energy consumption for thetask.

At any given time, for a resource ri, the utilization Ui is defined as

Ui =

n−1∑j=0

ui,j , (1)

where n is the number of tasks running at the given time and ui,j is the resourceusage of a task tj .

The energy consumption Ei of a resource ri at any given time is defined as

Ei = (pmax − pmin)× Ui + pmin, (2)

where pmax is the power consumption at the peak load (or 100% utilization)and pmin is the minimum power consumption in the active mode (or as low as1% utilization).

Consequently, at any given time, the total utilization (UR) as the totalenergy consumption (ER) of the system are defined as

UR =

m−1∑i=0

Ui and ER =

m−1∑i=0

Ei, (3)

respectively, where m represents the number of resources.The resources in the underlying system are assumed to be incorporated with

an effective power-saving mechanism for idle time slots. The mechanism resultsfrom the significant difference in energy consumption, between active and idleresources states. Specifically, the energy consumption of an idle resource at anygiven time is set to 10% of pmin. Because the overhead to turn off and back ona resource takes a nonnegligible amount of time, the option for idle resourceswas not considered in our study or by others ([35], [47], [21], [33], [46]).

3.3 The Task Consolidation Problem

The task consolidation (also known as server/workload consolidation) problemis the process of assigning a set T = {t0, . . . , tn−1} of n tasks (service requests orsimply services) to a set R = {r0, . . . , rm−1} of m cloud computing resources,without violating time constraints. The main purpose remains to maximizeresource utilization and ultimately to minimize energy consumption.

Time constraints are directly related to the resource usage associated withthe tasks. More precisely, in the consolidation problem, the resources allocatedto a particular task must sufficiently provide the resource usage of that giventask. For example, a task with its resource usage requirement of 60% cannot beassigned to a resource for which the available resource utilization at the time ofthat task’s arrival is 50%.

6

3.4 The Task Consolidation Algorithm

3.4.1 Overview

Task consolidation is an effective means to manage resources, particularly incloud computing, both in the “short-terms” and “long-term” ([24], [31]). Inthe short-term case, volume flux on incoming tasks can be “energy-efficiently”dealt with by reducing the number of active resources and putting redundantresources into a power-saving mode, or even turning off some idle resourcessystematically. In the long-term case, cloud infrastructure providers can bettersupply power and resources, alleviating the burden of excessive operational costsdue to over provisioning. Lee and Zomaya [46] focused on the short term case,even if the results delivered by the task consolidation algorithms could be usedas an estimator in the long-term provisioning case.

Subsection 3.4.2 presents the energy conscious task consolidation heuristics(ECTC and MaxUtil), more commonly referred to as cost functions [46]. Thetwo cost functions are described side by side to highlight the main differences,being whether the energy consumption is considered explicitly or implicitly.More precisely, MaxUtil makes task consolidation decisions based on resourceutilization, which is a key indicator for energy efficiency.

3.4.2 The Cost Functions (ECTC and MaxUtil)

The cost function, termed ECTC, computes the actual energy consumptionof the current task by subtracting the minimum energy consumption (pmin)required to run a task, if other tasks would be running in parallel with thattask. That is, the energy consumption of the overlapping time period amongthe running tasks and the current task (tj) is explicitly taken into account. Thecost function tends to discriminate the task being executed in a stand alonemode.

The value fi,j of a task tj on a resource ri obtained using the ECTC costfunction is defined as

fi,j = [(p∆ × uj + pmin)× τ0]− [(p∆ × uj + pmin)× τ1 + (p∆ × uj × τ2)], (4)

where p∆ is the difference between pmax and pmin, uj is the utilization rate oftj , and τ0, τ1, and τ2 are the total processing time of tj . That is, the time periodtj is running stand alone and that tj is running in parallel with one or moretasks, respectively. For example, consider two tasks (t0 and t1) that are runningin parallel on the same resource (r0), with t0 arriving first on the resource (seeFigure 1). While computing the result for f0,1

τ0 = the total execution time of t1,

τ1 = τ0 − τ2,τ2 = τ0 − τ1,

where τ1 is the time period where t1 will be running stand alone on r0, andτ2 the time period where t1 will be consolidated with t0 in r0 (the overlappingtime).

7

Figure 1: Time periods of the task t1

The rationale behind the ECTC cost function is that the energy consumptionat the lowest resource utilization is far greater than that in idle state, and theadditional energy consumption imposed by overlapping tasks contributes to arelatively low increase.

Alternatively, the MaxUtil cost function is derived with the average utiliza-tion during the processing time of the current task, as core component. Thecost function aims to increase consolidation density and has a double benefit.That is, (a) the implicit reduction of the energy consumption is directly relatedto (b) the decreased number of active resources. In others words, MaxUtil tendsto intensify the utilization of a small number of resources.

Consequently, the value fi,j of a task tj on a resource ri using the MaxUtilcost function is defined as

fi,j =

∑τ0τ=1 Uiτ0

, (5)

which is the utilization of a resource ri, as defined in Equation (1), divided bythe total execution time (τ0) of task tj .

3.4.3 The Task Consolidation Algorithm

In essence, for a given task, the algorithm checks every resource and identifies themost energy-efficient resource for that task. The evaluation of the most energy-efficient resource is dependent on the used heuristic (ECTC or MaxUtil). Morespecifically, on the employed cost function (referred to as fi,j). Algorithm 1describes the main steps of the task consolidation procedure.

3.5 Application of the Model — A Working Example

As incorporated into the energy model, energy consumption is directly propor-tional to the resource utilization. At a skimmed glimpse, for any two task-resource matches, the one with a higher utilization may be selected. However,because the determination of the right match is not entirely dependent on the

8

input : tj ∈ T = {t0, . . . , tn−1}, R = {r0, . . . , rm−1}output: r∗ ∈ Rbegin

r∗ ←− ∅forall the ri ∈ R do

Compute the cost function value fi,j of tj on riif fi,j > f∗,j then

r∗ ←− rif∗,j ←− fi,j

Assign tj to r∗

Algorithm 1: Task consolidation algorithm

current task, ECTC makes its decisions based rather on the (sole) energy con-sumption of that task.

Table 2 details four tasks properties specifically selected (as the workingexample) to point out the divergent behavior of ECTC and MaxUtil. For eachtask (tj) we specified the arrival time (aj), processing time (τ0), and utilizationor resource usage requirement (uj). For the working example it is assumed thatpmin is set to 20 and pmax to 30. These values can be seen as rough estimates inactual resources and can be referenced as 200 watt and 300 watt, respectively.Conforming to the respective properties presented in Table 2, each task (tj) willbe assigned to the more “energy-efficient” resource (ri) selected through thecost functions.

Figure 2 depicts the allocation of the first three tasks, where task t3 illus-trates the divergence from the results obtained from the respective cost func-tions. Based on the (sole) energy consumption of the task, ECTC assigns t3to the resource r1 (see Figure 2(a)), while based on the available utilizationrate of the resources, MaxUtil assigns t3 to the resource r0 (see Figure 2(b)).The difference between the two functions becomes more prominent when taskt4 must be assigned to a resource. As illustrated in Figure 3, ECTC can onlyassign t4 to the empty resource r2 (see Figure 3(a)), while MaxUtil assigns t4to r1 (see Figure 3(b)). On our specific working example MaxUtil seems to bemore “energy-efficient” than ECTC.

Ref.[46] claimed that the performances of the algorithms can be slightlyameliorated incorporating task migration. At each computational time, the

Table 2: Task properties

Task (tj) Arrival Time (aj) Processing Time (τ0) Utilization (uj)

0 00 (sec.) 20 (sec.) 40%1 03 (sec.) 08 (sec.) 50%2 07 (sec.) 23 (sec.) 20%3 14 (sec.) 10 (sec.) 40%4 20 (sec.) 15 (sec.) 70%

9

(a) ECTC (b) MaxUtil

Figure 2: Depiction of the first three tasks

(a) ECTC (b) MaxUtil

Figure 3: Final depiction for all the tasks

scheduler checks if some of the running tasks would be more “energy-efficient”,when allocated to a different resource. If suitable, then the scheduler proceedswith the migration. Interestingly, the benefit of using migration is not apparent.Migrated tasks tend to be with short remaining processing times and these tasksare most likely to hinder the consolidation of new arriving tasks. Consequently,the incorporation of task migration increased the energy consumption.

4 Bi-Objective Approach

4.1 The Main Idea

The algorithm described in Section 3.4 uses only one of the two cost functionsat a time. In advance, it must be decided whether to use ECTC or MaxUtil.

10

According to the working example described in Section 3.5, for a given task,the result of the two cost functions can converge as diverge. The divergencecomes from the two different considered aspects (energy consumption or resourceutilization).

The idea behind the bi-objective model is to combine the two cost functionsto only benefit from their advantages. The algorithm will then provide, as aresult, the more “energy-efficient” resource based on both of the consideredaspects. That is, the (sole) energy consumption as the resource utilization.

4.2 The Motivation

We must note that ECTC computes the energy consumption of a given task on aselected resource, while MaxUtil looks after the more energy-efficient resource interms of resource utilization. The ECTC cost function is designed to encourageresource sharing. As stated in Subsection 3.4.2, for a given resource, the energyconsumption of two tasks running in parallel is slightly superior than the energyconsumption of a task ran alone ([17], [15], [16]).

To be accurate on the computation of the energy consumption, ECTC usesτ1 and τ2 ( see Subsection 3.4.2). Based on the time periods (τx), the cost func-tion gives priority to resources where concurrent tasks can be fully consolidatedand tends to discard the resources offering only a partial consolidation. Theaforementioned scenario is illustrated in Figure 2. Task t0 do not fully overlaptask t3 on resource r0, then ECTC assigns t3 on r1 because t3 can be fullyconsolidated with the task t2.

The working example presented in Section 3.5 pointed out the main draw-back of ECTC. Intuitively, the resulting divergence from the behavior of MaxUtilcan be seen as a “domino effect” that will temporarily affect the system. Beingenergy efficiency (see Figure 3) the main concern of the presented heuristics, theeventuality of a “domino effect” should not be neglected while considering theECTC cost function for the task consolidation problem as defined in Section3.3. Alternatively, MaxUtil always minimizes the total number of used resourceswithout individually considering the energy consumption of the given task.

Because the objective of our study is to minimize the energy consumption asthe total number of used resources, our proposal combines the two cost functionsto select the resource that will most likely maximize the utilization rate andminimize the energy consumption.

4.3 The Approach

The approach uses the two cost functions described in Equation (4) and Equa-tion (5). The respective results are combined to build a “point” in a two-dimensional search space where ECTC gives the x coordinate and MaxUtil they coordinate.

Originally, Equation (4) returns a value greater than zero only when appliedon a resource allowing task consolidation. Among the collected results, thehighest value identify the most “energy-efficient” resource (if τ1 6= τ0), while

11

the null value identifies empty (non “energy-efficient”) resources (if τ1 = τ0).Figure 4 illustrates the rationale behind the ECTC cost function.

To properly construct the point in the search space, the two cost functionshave to be slightly modified. Defining the energy consumption ej of a task (tj)on a given resource (ri) as

ej = (p∆ × uj + pmin), (6)

the value fi,j of tj on ri obtained using the ECTC cost function is now definedas

fi,j =

{(ej × τ0) ; if τ1 = τ0((ej × τ1) + (p∆ × uj × τ2)) ; otherwise.

(7)

The value of fi,j obtained using the MaxUtil cost function as

fi,j =

dj∑aj

Ui, (8)

where aj is the arrival (or ready) time and dj the due date (given by aj + τ0)of the current task tj .

4.4 The Metric Normalization

To find the optimum point in a two-dimensional search space, the results of thetwo cost functions must be normalized to a homogeneous unit scale. Becausethe range of MaxUtil is defined in a continuous unit scale from 1 to 100, theresult of ECTC will be normalized to the unit scale of MaxUtil, from now onformally referred as [ymin : ymax].

For a given task (tj) the utilization on a selected resource (ri) is directly de-pendent on the speed (operations per seconds) of the CPU on that resource. Theamount of time needed for the resource to accomplish the task is derived fromthe speed of that CPU. Based on the above information (speed and time) of a

Figure 4: Structure of the ECTC cost function

12

selected resource, the maximum value (xmax) returned by the ECTC cost func-tion can be then estimated and the ECTC unit range defined by the boundedinterval going from 0 to xmax.

Defining xmin as the lower bound and xmax as the upper bound of theECTC unit range, the normalization of the (ECTC ) metric on the unit scale[ymin : ymax] is given as

normx : [xmin : xmax] −→ [ymin : ymax],

normx =

[(fi,j − xmin)× (ymax − ymin)

xmax − xmin

]. (9)

As mentioned in the Section 4.3, for a set of resources (R = {r0, . . . , rm−1})the highest value (fi,j) returned by the cost functions identify the most “energy-efficient” resource (see Algorithm 1). Because we modified the original ECTCcost function (see Equation (7)), the complement of normx, denoted as fx, mustbe considered

fx : [ymin : ymax] −→ [ymin : ymax],

fx = ymax − normx, (10)

to coherently build the normalized two-dimensional point in the evaluationspace.

4.5 The Evaluation Space

Renaming the respective returned value from the cost functions as fx for (thenormalized) ECTC and fy for MaxUtil, the coordinates of the point pi for aselected resource ri is defined as

pi = (fx; fy),

and among all the points, the optimum will be the higher point giving the bestcompromise between the results of the two cost functions (fx and fy). The idealoptimum points must then belong to the domain space of the function

f(x) = y. (11)

Therefore, the closer the pi is to the line (given by Equation (11)), the moreprobable it is for that point to be the local optimum. For every pi, the respectivedistance to the corresponding ideal optimum point will be computed and referredto as

d = |fx − fy|. (12)

The smaller the value of d, the closer the pi is to the respective ideal optimumpoint belonging to the domain space (of Equation (11)). This also will be thebest compromise offered by the given point. The value of d will finally bethe main estimator of the selection of the best candidate among the equivalentoptimum solutions.

13

4.6 Selection of The Best Candidate

4.6.1 The Mathematical Model

There exist many methodologies to find the optimum solution among the solu-tion set ([36], [2], [6]). The main idea would be to avoid having to compare allpoints within the solution space at every decision point.

The first step of our approach consists of constructing the solution searchspace based on the results of the two cost functions. The optimum solution willbe identified and updated at the same time the solution set is constructed. Bythe time the solution set is built the optimum solution will be identified. Thesearch operation will rely on the Pareto dominance criteria ([40], [42], [43]). Thefirst point of the solution space will be set as the “current” optimum solution.The “current” solution will then be compared to the next (new) created pointand updated, if needed.

Formally, let D be a finite set. For a fixed natural k, a mapping

m : D −→ Rk,m(p) = (m1(p), . . . ,mk(p)),

can be defined, whose components mi : D −→ R, ∀i : 1 ≤ i ≤ k, are denoted asobjectives and Rk is the evaluation (or measurement) space of the elements ofD. As already mentioned, our approach maximizes the considered objectives.Accordingly, given p and q, the two elements of D, p dominates q, and it isrepresented as p � q, if and only if,(∀i : 1 ≤ i ≤ k : (mi(p) ≥ mi(q)) ∧

(∃j : 1 ≤ j ≤ k : mj(p) 6= mj(q)

)). (13)

From our analysis and discussion of Section 4.5 it follows that

m(p) = (fx, fy),

where m(p) belongs to the homogeneous unit scale [ymin : ymax].

4.6.2 The Algorithm

For each resource, the algorithm constructs the corresponding point in the eval-uation space (D) and verify if the new created solution dominates the currentoptimum solution. If the aforementioned case is verified, then the algorithm up-dates the optimum solution. Algorithm 2 details the main steps of the procedureused to identify the optimal solution from the solution set D.

4.7 Processing of Equivalent Solutions

The design of Algorithm 2 does not identify equivalent solutions. A doubledominance check must be introduced and the equivalent solutions added to asubset F (F ⊆ D). Because the optimum solution may change by the timethe domain space (D) is constructed, F must “reset” each time a new optimum

14

input : tj ∈ T = {t0, . . . , tn−1}, R = {r0, . . . , rm−1}output: r∗ ∈ Rbegin

r∗, optimum←− ∅forall the r ∈ R do

x←− fxy ←− fyresult←− (x, y)if result � optimum then

optimum←− resultr∗ ←− r

Algorithm 2: Bi-Objective procedure

point is identified. This will ensure that the subset only contains the equivalentsolutions to the latest optimum point.

By the time the solution space is constructed, the equivalent optimum pointswill be identified. The selection among the equivalent solutions belonging to F ,if any, will rely on d. Because our approach maximizes the considered objectives,the complement of d, denoted as δ, will be considered according the formula

δ : [ymin : ymax] −→ [ymin : ymax],

δ = ymax − d. (14)

The aforementioned selection process will sequentially compare each f ∈ Fwith the actual optimum point. The actual optimum will then be updatedbased on the δ parameter, or on the sum of the two coordinates ((fx, fy) ∈ pi),if the pair share the same value for the x (energy consumption) or y (utilization)coordinate.

4.8 The Bi-objective Task Consolidation Algorithm

The algorithm constructs, for each resource, the corresponding point in theevaluation space (D) and verifies if the: (a) new created solution dominates thecurrent optimum solution or (b) two solutions are equivalent, otherwise. If thefist aforementioned case is verified, then the algorithm updates the optimumsolution and “reset” the equivalent solutions subset (F). Otherwise the algo-rithm update F with the new created solution if suitable. Among the equivalentsolutions of F , the algorithm identifies the optimum solution through: (c) theδ parameter or (d) the sum of the two objectives. Algorithm 3 describes theentire procedure that identifies the optimal solution from the solution set D andthe related subset F .

15

input : tj ∈ T = {t0, . . . , tn−1}, R = {r0, . . . , rm−1}output: r∗ ∈ Rbegin

r∗, optimum,F ←− ∅forall the r ∈ R do

x←− fxy ←− fyδ ←− (ymax − |x− y|)result←− (x, y)if (result � optimum) then

optimum←− resultr∗ ←− r, F ←− ∅F ←− F ∪ (result, r, δ)

if ((result 6� optimum) ∧ (optimum 6� result)) thenF ←− F ∪ (result, r, δ)

forall the (f, r, δ) ∈ F doif ((fx 6= optimumx) ∧ (fy 6= optimumy)) then

if (δ > δoptimum) thenoptimum←− fδoptimum ←− δr∗ ←− r

elseif ((fx + fy) > (optimumx + optimumy)) then

optimum←− fδoptimum ←− δr∗ ←− r

Algorithm 3: BTC Algorithm

16

4.9 An Intuitive Example

Conforming to the data provided in Table 2, the problem will be now solved us-ing the BTC algorithm, where for the normalization procedure into the MaxUtilunit scale, ymin have been set to 1 and ymax to 100. Table 3 summarizes thenumerical results of Equation (7), Equation (8), Equation (10), and Equation(14) presented in Section 4. For each task (tj) a maximum number of threeresources (ri) are identified. ECTC and MaxUtil represent the values of therespective cost functions (see Equation (7) and Equation (8)) for a given task(tj) on the selected resource (ri). The coordinates of the constructed point inthe two-dimensional search space are denoted by pi (see Equation (10)), and δi(see Equation (14)), which shows the normalized complement of the distance (d)from the point (pi) and the corresponding ideal optimum point for the selectedresource (ri). If at the arrival time of a task (tj) the selected ri had not enoughavailable resource utilization rate for the given tj , then that resource will not beshown in the table. The double right arrow (⇒) identifies the optimum resultselected by the algorithm.

For example, task t0 requires: (a) 40% of available resource utilization, (b)20 seconds to be executed by a selected resource, and (c) arrives at time t equalto zero. In line with the aforementioned task’s properties, when evaluating theresource r0, (d) ECTC returns a (normalized) value equal to 48 and (e) MaxUtila value equal to 40 (because ran in stand alone mode). The coordinates of thecorresponding point p0 are given based on the result obtained in: (d) and (e).The x coordinate is computed substracting 48 (result obtained in (d)) from 100(value set as ymax), equal to 52 (fx). The y coordinate is equal to 40 (fy),as computed in (e). From the coordinates (fx and fy) of p0 we compute (f)the distance (d = |fx − fy|), equal to 12. The value of δ is then computedsubstracting the result obtained in (f) to ymax (δ = 100 − 12) and is equal to88.

Table 3: Results of the BTC’s computations

Task (tj) Resource (ri) ETCT MaxUtil pi (fx; fy) δi

t0

⇒ r0 48 40 (52; 40) 88r1 48 40 (52; 40) 88r2 48 40 (52; 40) 88

t1

⇒ r0 18 90 (82; 90) 92r1 20 50 (80; 50) 70r2 20 50 (80; 50) 70

t2⇒ r1 51 20 (49; 20) 71

r2 51 20 (49; 20) 71

t3

⇒ r0 12 80 (88; 80) 92r1 20 60 (80; 60) 80r2 24 40 (76; 40) 64

t4⇒ r1 20 90 (80; 90) 90

r2 41 70 (59; 70) 89

17

The resulting two-dimensional search space for the evaluation of task t0 onthe three considered resources is illustrated in Figure 5(a). For a given task(tj), if at least two resources (ri) share the same coordinates (e.g. task t0 inTable 3), then that point is only represented once. In the aforementioned tableand figure, the point p0 identifies r0 as the optimum (most “energy-efficient”)resource evaluated through the BTC algorithm. Consequently, the algorithmassigns t0 to r0 as depicted in Figure 6.

Figure 5 illustrates the two-dimensional search spaces for each task, in agree-ment with the informations contained in Table 3. Among the points within theevaluation space, the (black) full dot (within the subset F) identifies the opti-mum point (resource) for the given task, while the line given by Equation (11)represents the ideal optimum points.

Throughout the numerical results summarized in the aforementioned table(visually reflected in Figure 5), the BTC algorithm evaluates and identifiesthe optimum solution simultaneously considering two aspects: (a) the resourceutilization rate and (b) the energy consumption implied by a given task. As aresult, the solution selected by the algorithm is the “energy-efficient” optimumin terms of both (a) and (b).

Figure 6 depicts an additional representation of the optimum scenario pro-vided by the algorithm after completing the evaluation of our intuitive example.The four tasks have been assigned only on two resources. As expected the al-gorithm maximized the resource utilization rate simultaneously minimizing thesystem’s energy consumption. For each given task of our particular example,BTC selected the same solutions than MaxUtil (see Figure 3(b)). The main dif-ference being that BTC always guarantee Pareto optimality in terms of energyefficiency.

5 Simulation Results

5.1 Simulation Setup

The simulations were carried out using the MPICH2 framework [27]. A high-performance and widely portable implementation of the Message Passing Inter-face (MPI) standard [26]. MPICH2 has two main goals. First, to provide anMPI implementation that efficiently supports different computation and com-munication platforms (including commodity clusters, high-speed networks, andproprietary high-end computing systems). Second, to enable cutting-edge re-search in MPI through an easy-to-extend modular framework for other derivedimplementations.

Throughout the simulations, the resource usage of the generated tasks wasrandom and uniformly distributed between 4% and 95% (or 0.04 and 0.95). Theminimum utilization rate of 4% avoids generating processing times lower than1 millisecond (τ0 < 1 ms).

The architecture used to simulate the environment followed the Master-Slavescheme. One selected resource (the Master) was in charge of : (a) dynamically

18

(a) t0 (b) t1

(c) t2

(d) t3 (e) t4

Figure 5: The two-dimensional search space of the BTC algorithm

19

Figure 6: The Final result identified by the BTC algorithm

generating the next task and (b) selecting the “optimum” resource (the Slave)to execute that task, according to the selected energy efficiency policy.

Because cloud computing systems are always ready to perform the nextincoming request, our problem becomes time and space dependent. That is,every future decision will be directly dependent on the previous decision.

To properly emulate the system, the simulation was divided into three steps:(a) the “seeding”, (b) the “training” (or warm-up) period, and (c) the (window)“run”. First, the resources are filled with “concurrent” tasks (seeding opera-tion) generated on a nonlinear “time dependent” distribution, aiming to sim-ulate some ongoing previous requests. Secondly, the new tasks were randomlygenerated and assigned to the “optimum” resource (training period) based onthe current work-flow of the system. Finally, the run evaluates the heuristicsbased on a predefined interval of tasks. More precisely, the seeding operationcreates the environment, the training period stabilizes the simulation, and therun evaluates the algorithm over a window consisting of one hundred thousand(105) tasks dynamically generated and assigned among the system. Figure 7illustrates the aforementioned steps.

The length of the warm up (“training”) period needs to be evaluated, if thestate of the model at starting time does not represent the steady state of theactual system. The point at which the model seems real for the first time couldbe estimated as the warm-up time. In our experiment, the warm-up periodis the amount of (simulated) tasks that need to run before the data collectionbegins [3]. The switch from the “training” to “run” occurs dynamically based onthe result given by the “output to input” ratio (λ/µ), also know as the systemutilization. At each task assignment, λ and µ represent the number of outgoingand incoming tasks, respectively. The ratio (λ/µ) aids in the monitoring of thesystem, allowing a dynamic start to the evaluation of the selected heuristic in anenvironment secured from collapse. The “run” starts as soon as the simulationreaches the steady state. In our environment, we considered the steady sate

20

Figure 7: The three steps of the simulation

from any value of the ratio (λ/µ) greater than 0.50.The performance and behavior of the BTC algorithm were evaluated com-

pared to the individual results obtained with Algorithm 1 presented in Section3. Initially setting MaxUtil and successively ECTC as the cost function.

Given that the main objective of our experiment was to maximize the uti-lization while minimizing the energy consumption, at each task assignment twoglobal factors were logged: (a) the total rate of resources utilization (UR) and(b) the cumulated amount of the energy consumption (ER) of the system (seeEquation (3)). Both (a) and (b) aim to observe the global behavior in terms ofenergy efficiency of the selected algorithms.

To compare the scalability of the different heuristics, at each task assignment,we observed the speed intended as the time (in microseconds (µs)) to select the“optimum” resource. The recorded computation relied on the MPICH2 timefunction (MPI Wtime).

The simulation environment was developed based on four different topolo-gies: (a) 10 and 20 resources of 4 cores and (b) 5 and 10 resources of 8 cores.The number of cores available on a resource bound the maximum number oftasks that can be concurrently executed on a selected resource. The experi-ment follows the specifications of the task consolidation problem as explainedin Section 3.3.

For each resource a central processing unit (CPU) of 2, 048 GHz was assumedand equally shared between the predefined numbers of cores. The process torandomly generate the tasks is as follow. Initially, a bounded number is gener-ated corresponding to the number of operations (per second) needed to performthat given task. Being bounded, the aforementioned prevents the resulting gen-erated task to overflow the computational capabilities of the available resources(i.e. uj > 100% for a given task tj). The bounded interval that generates thenumber of operations (per second) is directly dependent on: (a) the predefined

21

CPU speed and (b) the number of cores. From (a) and (b) we compute the solecore speed. Successively, dividing the number of operations (per second) by thesole core speed we obtain the processing time (in milliseconds (ms)) needed forthe given task to be executed on a selected resource. Finally, the utilizationrate of that task is derived by dividing the processing time by the predefinedCPU speed. Table 4 summarizes the parameters used to set our experiment.

Table 4: Parameters of the simulation environmentType Parameter Value

General

CPU speed 2,048 GHzWindow of tasks 100 000Resource usage [0.04 : 0.95][xmin : xmax] [1 : 20 000][ymin : ymax] [1 : 99]pmin 20pmax 30λµ 0.50

4 coresSingle core speed 512 MHzNumber of resources 10, 20Number of operations per second [1 : 9 400]

8 coresSingle core speed 256 MHzNumber of resources 5, 10Number of operations per second [1 : 4 700]

5.2 Results

5.2.1 Energy Efficiency

The results presented in the following graphs aim to evaluate the energy effi-ciency of the three heuristics: (a) MaxUtil, (b) ECTC, and (c) the Bio-objectiveTask Consolidation (BTC ) algorithm.

For each of the heuristic we depicted four graphs (see Figure 8, Figure 9,and Figure 10) that showed the behavior, in terms of energy efficiency and taskconsolidation, among the four selected topologies: (a) 10 and 20 resources of 4cores and (b) 5 and 10 resources of 8 cores. Within the aforementioned figures,the solid (blue) line represents the total utilization rate of the system while thedashed (red) line the energy consumption. The sampling rate for the collectionof the system’s status was performed at each task assignment. The y axis showsthe utilization as the (normalized) energy consumption unit scale, while the taskindex was reported following a logarithmic scale in the x axis.

Figure 8 and Figure 9 corresponds to MaxUtil and ECTC, respectively, whileFigure 10 to the BTC algorithm. From our results, no significant differencescan be pointed out among the three heuristics conforming to the task consol-idation problem. Consequently, MaxUtil, ECTC, and the BTC algorithm canbe considered “energy efficiently” equal.

22

(a)4co

res-10resources

(b)4co

res-20resources

(c)8co

res-5resources

(d)8co

res-10resources

Fig

ure

8:R

esu

lts

usi

ng

theMaxU

til

cost

fun

ctio

n

23

(a)4co

res-10resources

(b)4co

res-20resources

(c)8co

res-5resources

(d)8co

res-10resources

Fig

ure

9:R

esu

lts

usi

ng

theECTC

cost

fun

ctio

n

24

(a)4co

res-10resources

(b)4co

res-20resources

(c)8co

res-5resources

(d)8co

res-10resources

Fig

ure

10:

Res

ult

su

sin

gth

eBTC

alg

ori

thm

25

5.2.2 Speed Analysis

Figure 11 depicts the behavior of the heuristics in term of speed. The timeneeded to select the “optimum energy-efficient” resource among the system ateach task assignment. The solid (blue) line represents the BTC algorithm, whilethe dashed (red) and the dotted (cyan) lines represent MaxUtil and ECTC, re-spectively. The data are represented following a logarithmic scale on both of theaxis, where the task number is provided on the x and the time (in microseconds(µs)) on the y axis.

Among the considered aspects (utilization rate and energy consumption),MaxUtil only considers the available utilization rate, because developed tomaximize task consolidation. ECTC considers the available utilization rate(as MaxUtil) combined with the time periods (τx) to predict the energy con-sumption of the given task on a selected resource, designed to consolidate taskenergy-efficiently. When compared, MaxUtil proved to be faster than ECTC.

The BTC algorithm uses the solutions generated by MaxUtil and ECTC toconstruct the (bi-objective) evaluation space, conferring to the algorithm themore complex election of the optimum solution. Consequently, our proposedalgorithm was the slowest when compared, but resulted being the heuristic thatprovided the best “energy-efficient” solutions.

6 Discussion of Results

Recalling Section 3, our study described two existing heuristics: (a) MaxUtiland (b) ECTC. The main difference between (a) and (b) being whether theenergy consumption is implicitly or explicitly considered. MaxUtil proved tomaximize task consolidation reducing in turns the number of used resources.Consequently, the decreased number of used resources directly reduces the en-ergy consumption of the system. According to the aforementioned MaxUtil wasdefined as implicitly energy-efficient. Alternatively, ECTC was developed toconsolidate tasks that can be fully overlapped, tending to discard the consoli-dation option for partial-overlapping tasks. The main concern of ECTC remainthe energy consumption of the given task on the selected resource, that mustbe minimal. Consequently, the aforementioned property defined ECTC as ex-plicitly energy-efficient.

The rationale behind the ECTC cost function relies on the energy consump-tion during the time periods (τx) and pmin, the minimum power consumptionin the active mode. Designed to consolidate only full overlapping tasks, ECTCallows the minimum power consumption pmin to be ignored for that given task,independently from the number of consolidated tasks. The estimation of the en-ergy consumption of the given task will be fully dependent on the time periodgiven by τ2, the time period τ1 being respectively null (see Figure 4).

When the consolidated task can only be partially overlapped, for that task,ECTC considers the energy consumption as follows. The time period τ1 withthe implied power consumption in the active mode (pmin) is added to the energy

26

(a)4co

res-10resources

(b)4co

res-20resources

(c)8co

res-5resources

(d)8co

res-10resources

Fig

ure

11:

Com

pu

tati

on

al

spee

dof

the

heu

rist

ics

27

consumption during the time period τ2 (see Figure 1). For example, if n tasksare consolidated based on partial overlap, pmin must be considered n times onn different (time periods) τ1. Formally, let the time period (τ1) correspondingto the task (tj) on a selected resource (ri) defined as τ1j

, the total power con-sumption of that resource must then be increased by the minute energy factor

εi =(pmin ×

n∑j=0

τ1j

), (15)

which is what ECTC tries to avoid.Because the time periods constraints are strongly related to the minimum

power consumption in the active mode, ECTC can “diverge” by taking decisionsthat generate the “domino effect” as mentioned in Section 4.2. From the energyefficiency point of view, MaxUtil saves energy minimizing (when possible) thenumber of used resources of the system, while ECTC prioritizes the consolida-tion of full overlapping tasks. More precisely, ECTC runs as much as possibleconcurrent tasks under the same pmin to save that energy consumption.

The proposed bi-objective algorithm named BTC was built from the twoheuristics (MaxUtil and ECTC ). The BTC algorithm identifies the resourceoffering the best compromise between the results of the two cost functions, con-solidating tasks energy-efficiently. The closer the normalized results (see Section4), the more the offered solutions will be energy-efficiently converging. Conse-quently, divergent solutions will be dismissed by the BTC algorithm discardingthe possibility of generating the domino effect as mentioned above.

7 Conclusion

Task consolidation, especially in cloud computing systems, became an importantapproach to streamline resource usage that in turn improves energy efficiency.Two existing energy-conscious heuristics for task consolidation, offering differ-ent energy-saving possibilities, were analyzed in our study. For both heuristicswe identified the corresponding drawback and proposed, as a (single) solution,the Bi-objective Task Consolidation (BTC ) algorithm. The aforementioned al-gorithm combines the two heuristics to construct the corresponding bi-objectivesearch space. Within the domain space, the optimum solution set and the corre-sponding optimum solution are selected through the Pareto dominance criteriaand the Euclidean distance, respectively. The efficiency of the proposed algo-rithm was proved thought the evaluation study, consisting of different simula-tions carried out using the MPICH2 framework. Concerned about the energyefficiency of the system and the scalability of the proposed algorithm, at eachtask assignment we observed three main aspects: the total energy consumption,the total resource utilization, and the time needed to select the optimum so-lution. To evaluate the performance of the BTC algorithm according to theaforementioned criteria, the two heuristics were individually implemented andused as key indicator for the energy efficiency and the scalability. Despite the

28

more elaborate selection of the optimum solution, our study reported that theproposed BTC algorithm was the slowest when compared, but resulted beingthe heuristic that provided the best “energy-efficient” solution. The result ofour study should not only contribute on the reduction of electricity bills ofcloud computing infrastructure providers, but also promote the combinationsof existing techniques toward optimized models for energy efficient use, withoutperformance degradation.

29

References

[1] A. Beloglazov, R. Buyya, Y. C. Lee, and A. Y. Zomaya, “A Taxonomyand Survey of Energy-Efficient Data Centers and Cloud Computing Sys-tems,”, Advances in Computers, 82, 2011, pp. 47-111.

[2] A. Ghosh and S. Dehuri, “Evolutionary Algorithms for Multi-CriterionOptimization: A Survey”, International Journal of Computing & Infor-mation Sciences, 2(1), April 2004.

[3] A. Mehta, “Smart Modeling - Basic Methodology and Advanced Tools”,In Proc 32nd Conference on Winter Simulation (WSC ’00), Orlando, FL,USA, 1, 2000, pp. 241-245.

[4] A. Y. Zomaya, “Energy-Aware Scheduling and Resource Allocation forLarge-Scale Distributed Systems”, in 11th IEEE International Confer-ence on High Performance Computing and Communications (HPCC),Seoul, Korea, 2009.

[5] C. Lefurgy, X. Wang, and M. Ware, “Server-level power control”, in Proc4th IEEE International Conference on Autonomic Computing (ICAC’07), Jacksonville, FL, USA, June 2007, pp. 4.

[6] C. A. Coello Coello, “A Comprehensive Survey of Evolutionary BasedMultiobjective Optimization Techniques”, Knowledge and InformationSystems, 1(3), Citeseer, 1999, pp. 129-156.

[7] D. Grigoras, “Advanced Environments, Tools, and Applications for Clus-ter Computing”, NATO Advanced Research Workshop, IWCC 2001,Mangalia, Romania, September 2001, Lecture Notes in Computer Sci-ence, 2326, Springer, 2002.

[8] D. Kliazovich, P. Bouvry, and S. U. Khan, “DENS: Data CenterEnergy-Efficient Network-Aware Scheduling”, in ACM/IEEE Interna-tional Conference on Green Computing and Communications (Green-Com), Hangzhou, China, December 2010, pp. 69-75.

[9] D. Zhu, R. Melhem, and B. R. Childers, “Scheduling with dynamic volt-age/speed adjustment using slack reclamation in multiprocessor real-time systems”, IEEE Transactions on Parallel and Distributed Systems,14(7), pp. 686-700.

[10] F. Pinel, J. Pecero, P. Bouvry, and S. U. Khan, “Memory-aware GreenScheduling on Multi-core Processors”, in 39th IEEE International Con-ference on Parallel Processing (ICPP), San Diego, CA, USA, September2010, pp. 485-488.

30

[11] F. Pinel, J. E. Pecero, S. U. Khan, and P. Bouvry, “A Review on TaskPerformance Prediction in Multi-core Based Systems”, in 11th IEEEInternational Conference on Scalable Computing and Communications(ScalCom), Pafos, Cyprus, September 2011.

[12] G. L. Valentini, W. Lassonde, S. U. Khan, N. Min-Allah, S. A. Madani,J. Li, L. Zhang, L. Wang, N. Ghani, J. Kolodziej, H. Li, A. Y. Zomaya,C.-Z. Xu, P. Balaji, A. Vishnu, F. Pinel, J. E. Pecero, D. Kliazovich,and P. Bouvry, “An Overview of Energy Efficiency Techniques in ClusterComputing Systems”, Cluster Computing. (Forthcoming)

[13] G. Valentini, C. J. Barenco Abbas, L. J. Garcia Villalba, L. Astorga,“Dynamic multi-objective routing algorithm: a multi-objective routingalgorithm for the simple hybrid routing protocol on wireless sensor net-works”, IET Communications, 4(14), 2010, pp. 1732-1741.

[14] I. Menken and G. Blokdijk, “Virtualization - The Complete CornerstoneGuide to Virtualization Best Practices: Concepts, Terms, and Tech-niques for Successfully Planning, Implementing and Managing Enter-prise IT Virtualization Technology”, Emereo Publishing, October 2008.

[15] J. Choi, S. Govindan, B. Urgaonkar, and A. Sivasubramaniam,“Profiling,Prediction, and Capping of Power Consumption in Consolidated Envi-ronments ”, in 16th International Symposium on Modeling, Analysis,and Simulation of Computer and Telecommunication Systems (MAS-COTS 2008), Baltimore, MD, USA, September 2008, pp. 3-12.

[16] J. Choi, S Govindan, J. Jeong, B Urgaonkar, and A. Sivasubramaniam,“Power Consumption Prediction and Power-Aware Packing in Consol-idated Environments”, in IEEE Transactions on Computers, 59(12),2010, pp. 1640-1654.

[17] J. G. Koomey, “Estimating total power consumption by servers in theU.S. and the world”, Lawrence Berkeley National Laboratory, StanfordUniversity, 2007.

[18] J. J. Chen and T. W. Kuo , “Multiprocessor energy-efficient schedulingfor real-time tasks with different power characteristics”, in Proc Inter-national Conference on Parallel Processing (ICPP ’05), 2005, pp. 13-20.

[19] J. Kolodziej, S. U. Khan, and F. Xhafa, “Genetic Algorithms for Energy-aware Scheduling in Computational Grids”, in 6th IEEE InternationalConference on P2P, Parallel, Grid, Cloud, and Internet Computing(3PGCIC), Barcelona, Spain, October 2011.

[20] J. Moore, J. Chase, P. Ranganathan, and R. Sharma, “Making schedul-ing cool: temperature-aware workload placement in data centers”, inProc USENIX annual technical conference, 2005.

31

[21] J. Torres, D. Carrera, K. Hogan, R. Gavalda, V. Beltran, and N. Poggi,“Reducing wasted resources to help achieve green data centers”, in Proc4th workshop on High-Performance, Power-Aware Computing (HPPAC’08), 2008.

[22] K. H. Kim, R. Buyya, and J. Kim, “Power aware scheduling of bag-of-tasks applications with deadline constraints on DVS-enabled clusters”,in Proc 7th IEEE international symposium on Cluster Computing andthe Grid (CCGrid ’07), 2007, pp. 541-548.

[23] L. Barroso and U. Holzle, “The case for energy-proportional computing”,IEEE Computer, 40(12), 2007, pp. 33-37.

[24] M. G. Jaatun, G. Zhao, and C. Rong ,“Cloud Computing: First Inter-national Conference, CloudCom 2009”, China, December 2009, LectureNotes in Computer Science, 5931, Springer.

[25] M. Guzek, J. E. Pecero, B. Dorrosoro, P. Bouvry, and S. U. Khan,“A Cellular Genetic Algorithm for Scheduling Applications and Energy-aware Communication Optimization”, in ACM/IEEE/IFIP Interna-tional Conference on High Performance Computing and Simulation(HPCS), Caen, France, June 2010, pp. 241-248.

[26] Message Passing Interface (MPI), http://www.mpi-forum.org/, accessedApril 2011.

[27] MPICH2, http://www.mcs.anl.gov/research/projects/mpich2/, release1.3.2p1, accessed April 2011.

[28] P. Bohrer, E. Elnozahy, T. Keller, M. Kistler, C. Lefurgy, and R. Raja-mony, “The case for power management in web servers”, Power AwareComputing, Kluwer Academic Publishers, 2002, pp. 261-289.

[29] P. Ruiz, B. Dorronsoro, G. Valentini, F. Pinel, P. Bouvry, “Optimizationof the Enhanced Distance Based Broadcasting Protocol for MANETs”,Journal of Supercomputing, Springer, 55, 2011, pp. 1-28.

[30] Q. Tang, S. K. Gupta, and G. Varsamopoulos, “Energy-efficient thermal-aware task scheduling for homogeneous high-performance computingdata centers: a cyber-physical approach”, IEEE Transactions on Par-allel and Distributed Systems, 19(11), 2008, pp. 1458-1472.

[31] R. Buyya, J. Broberg, and A. M. Goscinski, “Cloud Computing: Prin-ciples and Paradigms”, Wiley, 1st edition, March 2011.

[32] R. Ge, X. Feng and K. W. Cameron, “Performance-constrained dis-tributed DVS scheduling for scientific applications on power-aware clus-ters”, in Proc the ACM/IEEE conference on SuperComputing (SC ’05),2005, pp. 34-44.

32

[33] R. Nathuji and K. Schwan K, “VirtualPower: coordinated power man-agement in virtualized enterprise systems”, in Proc 21st ACMSIGOPSSymposium on Operating Systems Principles (SOSP ’07), 2007, pp. 265-278.

[34] R. Subrata, A. Y. Zomaya, and B. Landfeldt, “Cooperative power-awarescheduling in grid computing environments”, Journal of Parallel andDistributed Computing, 70(2), pp. 84-91.

[35] S. Srikantaiah, A. Kansal, and F. Zhao, “Energy aware consolidation forcloud computing”, in Proc USENIX HotPower’08: Workshop on PowerAware Computing and Systems in Conjunction with OSDI, San Diego,CA, USA, December 2008.

[36] R.T. Marler and J.S. Arora, “Survey of multi-objective optimizationmethods for engineering”, Published online: 23 March 2004, Springer.

[37] S. U. Khan, “A Self-adaptive Weighted Sum Technique for the Joint Op-timization of Performance and Power Consumption in Data Centers”, in22nd International Conference on Parallel and Distributed Computingand Communication Systems (PDCCS), Louisville, KY, USA, Septem-ber 2009, pp. 13-18.

[38] S. U. Khan and I. Ahmad, “A Cooperative Game Theoretical Techniquefor Joint Optimization of Energy Consumption and Response Time inComputational Grids”, IEEE Transactions on Parallel and DistributedSystems, 20(3), 2009, pp. 346-360.

[39] T. Kuroda, K. Suzuki, S. Mita, T. Fujita, F. Yamane, F. Sano, A. Chiba,Y. Watanabe, K. Matsuda, T. Maeda, T. Sakurai, and T. Furuyama,“Variable supplyvoltage scheme for lowpower highspeed CMOS digitaldesign”, IEEE Journal of Solid-State Circuits, 33(3), 1998, pp. 454-462.

[40] V. Pareto, “Manuale di Economia Politica”, Societa Editrice Libraria,Milano, Italy, 1906. Translated into English by A. S. Schwier, “Manualof Political Economy”, New York: Macmillan, 1971.

[41] V. Venkatachalam and M Franz, “Power reduction techniques for micro-processor systems”, ACM Computing Surveys, 37(3), 2005, pp. 195-237.

[42] W. Stadler, “A survey of multicriteria optimization or the vector maxi-mum problem, part I: 1776-1960”. Journal of Optimization Theory andApplications, 29(1), 1979, pp. 1-52.

[43] W. Stadler, “Applications of Multicriteria Optimization in Engineeringand the Sciences (A Survey)”, in M. Zeleny (ed.) Multiple Criteria De-cision Making — Past Decade and Future Trends, Greenwich, CT: JAI,1984.

33

[44] X. Fan, X.-D. Weber, and L. A. Barroso, “Power provisioning for awarehouse-sized computer”, in Proc 34th annual International Sympo-sium on Computer Architecture (ISCA ’07), 2007, pp 1323.

[45] Y. C. Lee, and A. Y. Zomaya, “Minimizing Energy Consumption forPrecedence-Constrained Applications Using Dynamic Voltage Scaling,”in 9th IEEE/ACM International Symposium on Cluster Computing andthe Grid (CCGrid), Shanghai, China, 2009, pp. 92-99.

[46] Y. C. Lee and A. Y. Zomaya, “Energy efficient utilization of resources incloud computing systems”, published online: 19 March 2010, Springer.

[47] Y. Song, Y. Zhang, Y. Sun, and W. Shi, “Utility analysis for internet-oriented server consolidation in VM-based data centers”, in Proc IEEEinternational conference on cluster computing (Cluster ’09), 2009.

34