multi-objective vm consolidation based on thresholds and...

Received March 17, 2019, accepted April 15, 2019, date of publication April 23, 2019, date of current version May 2, 2019.

Digital Object Identifier 10.1109/ACCESS.2019.2912722

Multi-Objective VM Consolidation Based onThresholds and Ant Colony Systemin Cloud ComputingHUI XIAO 1, ZHIGANG HU1, AND KEQIN LI 2, (Fellow, IEEE)1School of Computer Science and Engineering, Central South University, Changsha 410083, China2Department of Computer Science, State University of New York, New Paltz, NY 12561, USA

Corresponding author: Zhigang Hu ([email protected])

This work was supported by the National Natural Science Foundation of China under Grant 61572525.

ABSTRACT With the large-scale deployment of cloud datacenters, high energy consumption and seriousservice level agreement (SLA) violations in datacenters have become an increasingly urgent problem to beaddressed. Implementing an effective virtual machine (VM) consolidation methods is of great significanceto reduce energy consumption and SLA violations. The VM consolidation problem is a well-knownNP-hard problem. Meanwhile, efficient VM consolidation should consider multiple factors synthetically,including quality of service, energy consumption, and migration overhead, which is a multi-objectiveoptimization problem. To solve the problem above, we propose a new multi-objective VM consolidationapproach based on double thresholds and ant colony system (ACS). The proposed approach leverages doublethresholds of CPU utilization to identify the host load status, VM consolidation is triggered when the hostis overloaded or underloaded. During consolidation, the approach selects migration VMs and destinationhosts simultaneously based on ACS, utilizing diverse selection policies according to the host load status. Theextensive experiment is conducted to compare our proposed approach with the state-of-art VM consolidationapproaches. The experimental results demonstrate that the proposed approach remarkably reduces energyconsumption and optimizes SLA violation rates thus achieving better comprehensive performance.

INDEX TERMS Ant colony system, double thresholds, energy consumption, quality of service, VM con-solidation.

I. INTRODUCTIONCloud computing provides access to virtualized cloudresources as a service to users over Internet in an on-demandand pay-per-use style [1]. With the maturity of cloud com-puting business models and technology architectures, thenumber of cloud users has increased significantly. New dat-acenters, servers, and cooling equipment have been added tomeet the increasing needs, generating high operational costsand carbon dioxide emissions. The minimization of datacen-ter energy consumption has become a critical challenge [2].At the same time, taking into account the users’ require-ments for cloud service performance, the quality of service(QoS) [3] defined in service level agreement (SLA) needs tobe met to avoid bringing unpredictable losses to users. There-fore, it is an urgent problem to be solved for the development

The associate editor coordinating the review of this manuscript andapproving it for publication was Mianxiong Dong.

of cloud computing to provide users with the desired QoSwhile reducing the energy consumption of datacenters.

The implementation of virtualization technology [4] incloud computing enables hosts to share physical resourcesand offer users services flexibly by creating multiple VMs.Dynamic VM consolidation [5] periodically adjusts the cur-rent mapping relation between VMs and hosts accord-ing to time-varying resource requirements by migratingVMs between hosts to fully and evenly utilize comput-ing resources. For example, migrating some VMs from anoverloaded host targets at reducing SLA violation, and foran underloaded host, all VMs on it should be migratedaway, then it is switched into sleep state to avoid energywaste. Obtaining the optimal mapping relation is beneficialto optimize the resource utilization, improve QoS, and reduceenergy consumption.

The techniques addressing the VM consolidation problemprimarily include heuristic greedy algorithms, constrained

VOLUME 7, 20192169-3536 2019 IEEE. Translations and content mining are permitted for academic research only.

Personal use is also permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

53441

https://orcid.org/0000-0002-0122-233X

https://orcid.org/0000-0001-5224-4048

H. Xiao et al.: Multi-Objective VM Consolidation Based on Thresholds and ACS in Cloud Computing

programming, and meta-heuristic algorithms. Greedy algo-rithms are widely used for dynamic VM consolidationbecause of the low time complexity and simplicity to imple-ment. However, traditional greedy approaches are easy tofall into local optimal solutions and miss the optimal solu-tion. Constrained programming techniques can achieve theoptimal solution, but they cannot be well extended to largedatacenters with the limitation of the problem size. In recentyears, researchers have proposed many bio-inspired meta-heuristic consolidation algorithms, such as Ant Colony Opti-mization (ACO) algorithm, Genetic Algorithm, Artificial BeeColony (ABC) algorithm, which are effective in solvinglarge-scale problems and avoiding local optimal solutions.Ant Colony System (ACS) [6]–[8], a kind of ACO algo-rithm, finds the near-optimal solution in polynomial timecomplexity through probabilistic search in the solution space,which has attracted more and more attention for the excellentperformance in solving NP-hard problems and combinatorialoptimization problems.

As far as we are concerned, most of the existing VM con-solidation approaches only focus on saving energy consump-tion of cloud datacenters. However, SLA violation shouldalso be considered to satisfy the QoS delivered by the cloudsystem. Noteworthily, VM consolidation can decrease energyconsumption by consolidating VMs into a reduced numberof hosts, but excessive consolidation might degrade systemperformance and lead to SLA violations [9]. Therefore, theoptimal VM consolidation approach should strike a balancebetween energy consumption and QoS. In addition, the VMmigration incurs additional workload and increases energyconsumption. The service downtime caused by migrationlikely affects QoS. Hence, VM consolidation should triggeras fewVMmigrations as possible to minimize the consequentnegative influence.

In this paper, we propose a multi-objective VM con-solidation approach based on double CPU utilizationthresholds [10] andACS, called DA-VMC. In ACS, a numberof artificial ants build solutions to the related optimizationproblem in parallel. They exchange quality information ofthese solutions via pheromone to find the optimal solu-tion. Taking advantage of these characteristics of ACS, ourapproach exploits artificial ant colony to seek the optimalmapping relation between VMs and hosts by assuming thata mapping relation between VMs and hosts is a food source.The main contributions of this paper are summarized asfollows.• First, we abstract the VM consolidation problem asa multi-objective combinatorial optimization problemoptimizing three conflicting objectives, including reduc-ing energy consumption, ensuring QoS requirements,and reducing the number of migrations.

• Then, we employ double thresholds of CPU utiliza-tion to determine migration time. Based on doublethresholds, the host load status is judged as overloaded,normal-loaded, or underloaded. In order to optimizeQoS and energy consumption of datacenters, overloaded

hosts and underloaded hosts will perform VM consoli-dation successively.

• Next, we apply ACS in a multi-stage VM consolidation,in which several selection policies corresponding to dif-ferent host status are used to select migration VM anddestination host. The problem of migration VM selec-tion and destination host selection for VM consolidationis globally studied and optimized.

• At last, we evaluate the proposed DA-VMC approach byusing CloudSim platform on real workload. The exper-imental results demonstrate that DA-VMC possessesan obvious advantage in the aspect of reducing energyconsumption, SLA violations, and VM migrations.

The rest of the paper is organized as follows. Section 2introduces the related work of VM consolidation. In Section3, we introduce how to build the VM consolidation probleminto a multi-objective combinatorial optimization problem.Section 4 proposes our VM consolidation approach based ondouble thresholds and ACS. Section 5 presents experimentsresults and performance evaluation; Section 6 concludes thepaper and discusses our future work.

II. RELATED WORKThere are three main problems to be solved in VMconsolidation [11]–[13]. First, VM consolidation time selec-tion, to determinewhenVMs should be consolidated. Second,there is a need to select which VMs should be migrated, thatis, migration VM selection. Third, VM deployment, to deploythose selected migration VMs. Depending on specific prob-lems of VM consolidation, researchers have proposed a vari-ety of different approaches.

A. CONSOLIDATION TIME SELECTIONThe workload in the datacenter fluctuates in real time,which affects the resource utilization and energy efficiencyof hosts [14]. Most of the existing studies determined theconsolidation time on the basis of host resource utilization.VM consolidation is usually carried out for overloaded hostsand underloaded hosts to reduce energy consumption andimprove QoS [15], [16].

Chen et al. [17] presented a method based on the slidingwindow concept to perform consolidation operation whenhost resource utilization, sampled at regular intervals andrecorded in windows, exceeds the pre-defined high resourceutilization threshold continuously. The consolidation actionis also triggered if the host CPU utilization is lower thanthe underload threshold. Minarolli et al. [18] proposed along-term forecast of VM resource demands based on Gaus-sian processes to detect when a host is overloaded or under-loaded. A decision-theoretic approach using utility functionwas applied to execute migration decision considering livemigration overheads.

To adapt to the dynamic variation of the workload inthe datacenter, some approaches dynamically adjust thethresholds of CPU utilization to guide VM consolidation.

53442 VOLUME 7, 2019


Masoumzadeh and Hlavacs [19] presented an adaptivethreshold based approach using fuzzy Q-learning for hostoverloading detection. The fuzzy Q-learning technique learnsthe historical data of overload thresholds in different datacen-ter states, and yields the appropriate threshold for workloaddata input from hosts according to the online decision model.Zhou et al. [20] proposed an adaptive three-threshold frame-work based on K-means clustering algorithm. The hosts inthe datacenter are divided into four categories: less loadedhosts, little loaded hosts, normal-loaded hosts, and over-loaded hosts. Salimian et al. [21] proposed an adaptive algo-rithm based on fuzzy threshold to detect overloaded andunderloaded hosts. The information of host resource usage isapplied to the fuzzy inference engine to estimate the numeri-cal value of the CPU utilization thresholds.

Unlike traditional CPU utilization threshold frameworks,Fard et al. [22] employed the temperature threshold metricto conduct a thermal-aware consolidation. The temperaturethreshold is set as the temperature of the optimum host at90% CPU utilization. For the other host, if its tempera-ture exceeds the temperature threshold, it is identified asoverloaded. For underloaded hosts, the consolidation is pre-ferred to the host with most energy consumption but lowestoperation per second. Abdelsamea et al. [23] proposed amultivariate regression method employing hybrid resourceparameters including CPU, RAM, and bandwidth utilizationto predict host resource utilization and detect overloadedhosts. However, the coefficients of the regression modelare trained once per host, which adds complexity to theprediction.

B. MIGRATION VM SELECTIONVM consolidation adopts different selection strategies toselect VMs for migration from hosts of undesirable work-load status, aiming at different targets, such as optimiz-ing resource utilization, improving QoS or reducing energyconsumption.

Focused on the VM selection problem,Beloglazov et al. [10] presented three VM selection poli-cies. The minimization of migrations (MM) policy selectsthe minimum number of VMs to migrate for overloadedhosts to decrease the CPU utilization. The highest potentialgrowth (HPG) policy migrates the VM with the lowest usageof the CPU resource each time to minimize the potentialenhancement of hosts’ CPU utilization and prevent sec-ondary SLA violations. The Random Selection (RC) policyrandomly selects VMs for migration based on a uniformlydistributed discrete random variable until the overload statusof the host is eliminated. Further, Beloglazov and Buyya [24]presented two other VM selection policies. The minimummigration time (MMT) policy selects VMs requiring theshortest migration time for migration. The idea of the Maxi-mum Correlation (MC) policy is that the higher the resourceutilization correlation between VMs on the same host, themore likely the host overload, hence the VM having thehighest correlation of CPU utilization with other VMs is

selected for migration to reduce the risk of host overload.Cao and Dong [25] proposed the minimum utilization(MU)policy to select the VM with the lowest CPU utilizationfor migration. Masoumzadeh and Hlavacs [26] proposed amaximum utilization (MaxU) policy which selects the VMwith the highest CPU utilization to migrate. In contrast tothe MU policy, the MaxU policy eliminates the host overloadas quickly as possible, whereas greatly increasing migrationoverheads.

Different from the aboveworks which select migrationVMbased mainly on CPU utilization, Shidik et al. [27] proposeda VM selection policy based on RAM and CPU utilization.Fuzzy logic is employed to categorize both resource attributesof VM candidates and the Markov normal algorithm is usedto select the migration VM according to the categoricalattributes. Li et al. [28] selected VMs to migrate by utilizingthe content similarity among the VM memory. The migra-tion VM selection favors the VMs with the highest memorycontent similarity from different overloaded hosts to reducethe amount and time of data transfer during VM migration.Laili et al. [29] presented an iterative budget algorithm tak-ing into account various resources including CPU, memory,disk and network. The algorithm builds a reverse selectionmechanism that finds the most suitable VM from candidateVM sets for each randomly selected target host.

C. VM DEPLOYMENTVM consolidation deploys the migrated VMs on a moresuitable destination host to dynamically optimize themapping relation between VMs and hosts in the clouddatacenter.

Mishra and Sahoo [30] studied how to use heuristic greedyalgorithms, such as BFD (Best Fit Deceasing) and FFD (FirstFit Deceasing), to obtain a quasi-optimal VM deploymentsolution. Murtazaev and Oh [31] proposed the Sercon algo-rithm which inherits some properties of FF (First Fit) and BF(Best Fit) to minimize the number of hosts and migrations.Farahnakian et al. [32] abstracted theVMconsolidation prob-lem into a multi-objective vector packing problem, aiming toreduce energy consumption, minimize migrations and avoidSLA violation. The target host is selected according to thecurrent and future resource utilization of hosts and VMs.The correlation-based strategy of VMconsolidation [33] con-solidates VMs with inter-traffic as closely as possible tominimize the network traffic and alleviate network pres-sure, but it also increases the traffic load of the virtualswitches on hosts. To address this problem, Li et al. [34] pro-posed virtual-switching-aware BFD(VSA-BFD) and virtual-switching-aware FFD(VSA-FFD) with comprehensive con-sideration of the traffic between VMs and the CPU overheadgenerated by virtual switches.

The greedy algorithm has low complexity, but it cannotguarantee to find the optimal solution. Chen et al. [17]formulated the VM deployment problem as a multi-criteriadecision-making problem. The TOPSIS solution is applied tochoose appropriate destination hosts. Huang and Tsang [35]

VOLUME 7, 2019 53443


developed non-linear programming and proposed a dis-tributed framework to automate VM consolidation. Basedon the m-convex optimization theorem, the optimalVM deployment solution is obtained in an incremental man-ner. However, limited by the size and complexity of theproblem, non-linear programming is incapable to scale wellto big datacenters.

With the ability to handle the large-scale problems effi-ciently and avoid suboptimal solutions, the meta-heuristicalgorithm is helpful to make up for the deficiencyof the greedy algorithm and constrained programming.Li et al. [36] constructed the VM consolidation problem as amulti-objective optimization problem with multiple resourceconstraints, and simulated the artificial bee colony foragingbehavior to search for the optimal mapping relation betweenVMs and hosts. Mosa and Paton [37] adopted the utilityfunction considering income, energy cost, and violation coststo calculate the profit of VM deployment solutions. Thegenetic algorithm is used to search candidate deploymentsto maximize the utility function. Li et al. [38] proposed anVM reallocation algorithm based on the adaptive particleswarm optimization to achieve minimal energy consumption,where QoS is ensured by applying multi-resource utilizationthresholds.

Focused on balancing the usage of various resources inhosts, Ferdaus et al. [39] proposed a vector algebra-basedACO algorithm for searching the optimal VM deploy-ment. However, the algorithm reallocates all VMs duringeach VM consolidation, leading to high time complexity.Farahnakian et al. [40] proposed a VM consolidation methodleveraging ACS to deploy VMs into the minimal amountof running hosts, based on the objective function definedwith the number of sleep hosts and VM migrations. How-ever, the energy consumption generated by VM migrationsis not considered. Aryania et al. [41] extended the work ofFarahnakian et al. [40] by considering the energy consump-tion of both running hosts and VMmigrations, which obtainsa better energy saving effect.

It can be known from above studies that most of them con-duct research on VM consolidation from various optimiza-tion perspectives such as energy efficiency, resource usagebalance, VM migration overhead. Accordingly, we constructthe VM consolidation problem as a multi-objective combi-natorial optimization problem taking into account multipleinfluential factors of VM consolidation. To figure out theproblem, we propose a dynamic VM consolidation approachDA-VMC. Static double thresholds of CPU utilization areapplied in the decision-making process of the consolidationtime to reduce computational overhead and avoid the systeminstability. Assuming the mapping relation between VMs andhosts as a food source, we employ ACS which simulatesthe artificial ant colony foraging behavior to simultaneouslyselect the migration VM and the destination host basedon the specified objective function. After several rounds ofsearches and information exchanges, the ant colony acquiresthe near-optimal solution.

III. DYNAMIC VM CONSOLIDATION BASED ONMULTI-OBJECTIVE COMBINATORIAL OPTIMIZATIONA. DATACENTER MODELThere exist many hosts with different configurations in thedatacenter. VMs are deployed to the appropriate hosts accord-ing to their respective resource requirements. Assume that thehost set in the datacenter is H =

{h1, h2, . . . , hj, . . . , hm

},

where m is the number of hosts; The VM set is denotedas H = {v1, v2, . . . , vi, . . . , vn}, where n is defined as thenumber of VMs. The variable xij ∈ {0, 1} indicates whetherthe VM vi is assigned to the host hj. xij = 1 if assigned;otherwise xij = 0. The matrix X =

[xi,j]n×m describes

the mapping relation between the VMs and hosts. The hostdeploying vi is denoted as h (vi), and V

(hj)is the set of VMs

deployed on the host hj. VRi is the demand of the VM vi forthe CPU resource. PRj is the total CPU resource demands ofthe host hj, which is expressed as

PRj =N∑i=1

(xij · VRi

). (1)

CRj is the CPU resource capacity of the host hj. The actualutilization of CPU resource in the host hj, denoted as Uj,is calculated with

Uj =PRjCRj

. (2)

B. ENERGY CONSUMPTION MODELIn general, the host has three states, i.e., running state, sleepstate, and shutdown state. Chou et al. [42] suggested thatdifferent states of the host result in different power consump-tion levels. The energy consumption of hosts in the runningstate, denoted asHPonj , is highly positively correlated with theCPU utilization [43]. Compared to the running state, the hostconsumes only a small amount of energy in the sleep state,denoted as HPspj , and consumes no energy in the shutdownstate. Besides, switching back and forth between differentstates consumes energy and time. It takes a long time to restartthe host from the shutdown state to the running state, whichmay cause degradation of service quality. Therefore, in thispaper, VM consolidation is only considered in hosts that arerunning or sleeping.

Based on the above analysis and assumption, the fol-lowing energy consumption model for VM consolidation isestablished with comprehensive consideration of the energyconsumption generated by both host operation and stateswitching:

ECj =∫t

[sonj · HP

onj +

(1− sonj

)· HPspj

]dt

+ sgsj · ESgsj + s

waj · ES

waj , (3)

where sonj ∈ {0, 1} represents the status of the host, sonj = 1

denotes that the host is running, sonj = 0 denotes that thehost is in the sleep state. sgsj ∈ {0, 1}, s

gsj is set to 1 when

switching the host hj from the running state to the sleep state,otherwise sgsj = 0. swaj ∈ {0, 1}, s

waj is set to 1 when hj is

53444 VOLUME 7, 2019


waked up from sleep, otherwise swaj = 0. ESgsj indicates theenergy consumption of state switching from running to sleep,and ESwaj indicates that from sleep to running.The total energy consumption of the datacenter can be

calculated as

EC =m∑j=1

ECj. (4)

C. MULTI-OBJECTIVE COMBINATORIAL OPTIMIZATIONAn efficient VM consolidation approach needs to developcorresponding strategies of resource management andscheduling for multiples goals of reducing energy consump-tion, improving QoS, and facilitating load balancing, so thatthe VM migration can be triggered under specific conditionsto optimize the mapping relation between VMs and hosts.Accordingly, we take into account multiple factors affectingVMconsolidation and define the followingVMconsolidationoptimization objectives.

In order to reduce energy consumption and save energy,the total power consumption of the datacenter EC should beminimized, i.e., min (EC).There exists resource competition between VMs on the

same host. When the resource competition becomes moreintense, especially when the resource utilization approachesor exceeds a certain threshold, the performance of the hostturns worse, and the QoS level is more likely to decline [44].Apparently, the QoS level of the host has a great relationshipwith the resource utilization. Hence, for purpose of optimiz-ing the QoS more conveniently and effectively, we ensure thehost QoS by limiting the host CPU resource utilization belowthe overload threshold Thro, as shown below:

Uj < Thro, ∀j ∈ {1, 2, . . . ,m} . (5)

Moreover, VM migrations due to consolidating VMs con-sume energy and computing resources, incurring perfor-mance interference and cost on both source and destinationhosts [45]. Data transferred during migration burden networktraffic, which inevitably interferes with the other VMs inthe same datacenter. Thus minimizing the number of VMmigrations, denoted asMG, is very necessary, i.e., min (MG).

Finally, the above objectives are integrated as a minimumcombinatorial optimization problem with the resource con-straint via a linear weighting method:

min (F) = min(EC + ωmg ·MG

)s.t. Uj < Thro,∀j ∈ {1, 2, . . . ,m} , (6)

where ωmg represents the weight of migrations with respectto the total energy consumption.

The optimization problem defined in (6) is studied andsolved in Section IV.

IV. DA-VMC CONSOLIDATION APPROACHThe dynamic VM consolidation enables VMs to be dynami-cally migrated between hosts to accommodate the changeable

workload in the datacenter. In VM consolidation, there areseveral key problems that must be addressed. For example,what kind of condition will trigger VM consolidation (con-solidation time selection); which VMs should be migratedto achieve the best energy consumption and QoS (migrationVM selection); Which destination hosts should be choosedto redeploy the selected migration VMs (destination hostselection).

In order to solve the above problems, we propose a VMconsolidation approach based on double thresholds and ACS.Based on the strategy of static double thresholds, the pro-posed DA-VMC approach sets the host CPU utilization upperthreshold Thro and lower threshold Thru (0 ≤ Thru <

Thro ≤ 1) to decide the consolidation time. The definedstatic thresholds are used to identify the load status of hosts,dividing all hosts into three sets: the overloaded host set Ho,the normal-loaded host set Hn, the underloaded host set Hu.For the host hj, the host is overloaded if its CPU utilizationThro ≤ Uj < 1; the host is normal-loaded if Thru ≤Uj < Thro; the host is underloaded if 0 < Uj < Thru.VM consolidation is triggered when the host is overloaded orunderloaded. According to the study by Beloglazov et al. [10]and experimental verification, the algorithm can obtain excel-lent performance when the upper threshold is set to 80% andthe lower threshold is 40%.

In the datacenter, each host deploys one or more VMs.In this paper, we assume that all the VMs deployed on a hostmay be selected for migration, in which case the host is thesource host of the migrated VMs. Likewise, a migrated VMmay be redeployed to any other host, namely all hosts exceptthe source host are its potential destination hosts. Accord-ingly, a tuple set of mapping relations T is defined as T ={(vm, hd )}, where each tuple consists of two elements: theVMto be migrated vm and the destination host hd . By treating thetuples in T as the ant food, the DA-VMC approach leveragesACS to search tuples in T to update the mapping relationbetween VMs and hosts.

The complexity of the consolidation approach dependsprimarily on the number of tuples in T . To reduce the timecomplexity of the approach, we apply a multi-stage consoli-dation which limits the number of tuples in T at each stage.On the one hand, VM consolidation is firstly performed foroverloaded hosts, and then for underloaded hosts. On theother hand, when selecting the destination host, in orderto minimize the number of underloaded hosts to reducepower consumption, the first choice is made in the set ofnormal-loaded hosts. If it fails, the range of choice turns tothe set of underloaded hosts. Hosts in the sleep mode areactivated only if the VM cannot be redeployed on an alreadyactive PM. In this way can restrict the solution space for eachsearch and improve the efficiency of the ant’s search, whichhelps greatly reduce the computation time of the approachwithout affecting the quality of the solution. Fig. 1 shows themulti-stage consolidation framework.

During the process of VM consolidation, the DA-VMCapproach creates the pheromone information matrix [τi,j]n×m

VOLUME 7, 2019 53445


FIGURE 1. Multi-stage consolidation framework.

to save the experience in past searches of ants, where τi,jdenotes the favorability of selecting the mapping relationtuple

(vi, hj

), namely redeploying the VM vi on the host hj.

Two rules of local pheromone update and global pheromoneupdate are applied to update the pheromone level of tuples.The heuristic factor is defined to guide the ant colony toselect the optimal tuple for minimum power consumption andmigration number based on the pseudo-random-proportionalrule [6]. In the following, the detailed definition of thesefactors will be given.

A. PHEROMONE INFORMATIONThe ant deposits pheromone on the path to finding foodsource, and when other ants smell the deposited pheromone,they tend to choose paths with a higher pheromone concen-tration. The quality of the solution found by ants dependsgreatly on the definition of the pheromone. Thus, choosing arational definition of pheromone is very critical. In the initial-ization phase of the pheromone matrix [τi,j]n×m, we employthe solution quality assessment method used by Dorigo andGambardella [6] to calculate the initial pheromone amount asfollows:

τ0 =1

EC0 ·MG, (7)

where EC0 is the total power consumption of the datacenterin the initial state. MG is the approximate optimal numberof migrations, which can be estimated by using the nearestneighborhood heuristic algorithm [6].

For the pheromone update in ACS algorithm, we presentthe local pheromone update rule and the global pheromoneupdate rule to increase the pheromone amounts correspond-ing to high quality solutions or decrease those correspond-ing to the low quality ones. After selecting a new mappingrelation tuple (vi, hj), the ant updates the pheromone levelof this traversed mapping relation using the following localpheromone update rule:

τij = (1− ρl) · τij + ρl · τ0, (8)

where ρl ∈ [0, 1] is the local pheromone evaporating param-eter. The local pheromone update rule applied to a traversedtuple decreases the tuple’s pheromone concentration by a cer-tain level and weakens its attraction to other ants. Hence, thelocal pheromone update rule can avoid a premature conver-gence of the ACS algorithm towards a suboptimal solution.

After all the ants complete building solutions, the qualityof all current built solutions is evaluated according to theobjective function. The following global pheromone updaterule is performed to preserve the experience of the globaloptimal solution:

τij = (1− ρg) · τij + ρg ·1τ

1τ =

1

F(X+), if xij = 1 in X+

0, otherwise,

(9)

where 1τ is the amount of additional pheromone increment.ρg ∈ [0, 1] is the global pheromone evaporation parameter.X+ is the global best solution.

B. HEURISTIC FACTORIn ACS, the heuristic factor is used in combination withthe pheromone to guide the solution construction of ants.The heuristic factor is expressed as ηi,j, indicating thedesirability of selecting the tuple

(vi, hj

). Calculated in a

problem-specific style, the heuristic factor reduces the blind-ness of ant searches, which is an important factor affectingthe efficiency of ACS.

In order to fully utilize host resources while minimizingdegradation of service quality due to resource competition,the proposed approach defines different calculation criterionsof the heuristic factor according to different consolidationstages, considering two partial contribution of migration VMselection and destination host selection. The calculation cri-terion of the heuristic factor ηi,j for selecting the tuple

(vi, hj

)is defined as

ηi,j = ηv (h(vi),−vi) · ηh

(hj,+vi

), (10)

where ηv (h(vi),−vi) is the heuristic factor that selectsthe VM vi for migration from its source host h(vi), andηh(hj,+vi

)is the heuristic factor that redeploys vi to the

destination host hj. A hybrid heuristic factor is defined withconsideration of both migration VM selection and destinationhost selection.

1) MIGRATION VM SELECTIONBased on the predefined double thresholds, the main ideaof VM selection is to control the CPU utilization of hostsbetween the double thresholds. Inappropriate selection crite-ria may increase VM migrations, aggravate power consump-tion, or affect QoS due to long service downtime inmigration.Hence, selection criteria for migration VM should be rea-sonably defined. In the following, we discuss the proposedVM selection policies with corresponding heuristic factorcalculation rules.

53446 VOLUME 7, 2019


If the host CPU utilization exceeds the upper threshold,some VMs have to be migrated from the host to reduce theutilization, preserving some free resources to prevent SLAviolations. Accordingly, we introduce three VM selectionpolicies for overloaded hosts as follows.

First, the Highest CPU Priority Selection policy (HCPS).The HCPS policy defines that the higher the CPU utilizationof a VM, the higher the priority that the VM is selected formigration. For the overloaded host hj ∈ Ho, the heuristicfactor of selecting the VM vi for migration is calculated as

ηv(hj,−vi

)=VRiPRj

. (11)

Second, the Minimum CPU Priority Selection policy(LCPS). When the upper threshold is violated by the host hj,the idea of the LCPS policy is that a VM vi with lower CPUutilization are selected with higher priority as the migrationVM. The heuristic factor is defined as

ηv(hj,−vi

)= 1−

VRiPRj

. (12)

Third, the Random CPU selection policy (RCS).For theoverloaded host hj ∈ Ho, the RCS Policy randomly selectsthe VM vi for migration. All VMs are selected with the samepriority, and the corresponding heuristic factor is defined as

ηv(hj,−vi

)=

1∣∣VMj∣∣ , (13)

where∣∣VMj

∣∣ represents the number of VMs deployed on thehost hj.

For a underloaded host, all VMs on this host should bemigrated and the host should be switched to the sleep stateto avoid the idle power consumption. Thus, with the aimof minimizing invalid migrations and underloaded hosts, theunderloaded host should prefer to migrate the VM that signif-icantly reduces its resource utilization after migration. Theheuristic factor for selecting the VM vi to migrate in theunderloaded host hj ∈ Hu is defined as

ηv(hj,−vi

)= 1− Uj (−vi) , (14)

where Uj (−vi) represents the CPU utilization of the host hjafter migrating the VM vi.

2) DESTINATION HOST SELECTIONA new destination host should be selected to deploy theselected migration VM. When the host CPU utilization getscloser to the overload threshold, the system QoS drops morerapidly. Therefore, when selecting the destination host inthe normal-loaded host set Hn, choosing the host with lowresource utilization rate after VM redeployment is advanta-geous for avoiding the QoS degradation due to the excessiveCPU utilization of hosts. The heuristic factor definition ofselecting the destination host hj ∈ Hn for the VM vi iscomputed as

ηh(hj,+vi

)=

{1− Uj(+vi), if Uj(+vi) < Thro0, otherwise,

(15)

where Uj(+vi) is the CPU utilization of the host hj afterdeploying the VM vi. Besides, the CPU resource utilizationconstraint in the heuristic factor is to prevent migrations thatcause the overload of destination hosts.

Underloaded hosts have low resource utilization and weakresource competition, which guarantee QoS but waste energy.Hence, when selecting the destination host hj in the under-loaded host set Hu, a host with higher resource utilizationafter VM deployment is more preferable, which contributesto the minimization of underloaded hosts. The correspondingheuristic factor is defined as

ηh(hj,+vi

)=

{Uj(+vi), if Uj(+vi) < Thro0, otherwise.

(16)

C. PSEUDO-RANDOM-PROPORTION RULEBased on the heuristic factor and pheromone information,the ant selects the next tuple for traversal according to thefollowing pseudo-random proportion rule:

(vm, hd ) =

{argmax(vi,hj)∈�k {τ

αij · η

βij }, if q ≤ q0(

vs, hg), otherwise,

(17)

where α and β are parameters that control the influence of thepheromone and the heuristic factor respectively.�k is the setof tuples currently allowed to be traversed by the ant antk . q isa random number uniformly distributed in [0, 1], q0 ∈ [0, 1]is a fixed parameter that determines the relative importanceof cumulative experience with random selection. (vs, hg) is arandom tuple variable selected according to the probabilitydistribution given below:

pkmd =

ταmd · η

βmd∑

(vi,hj)∈�k

(ταij · η

βij

) , if(vi, hj

)∈ �k

0, otherwise,(18)

where pkmd denotes the probability that the ant antk choosesto traverse the tuple (vm, hd ) in the next step.

The pseudo-random proportional rule favors the tuple(vi, hj

)with large heuristic value ηij and high pheromone

level τij. In each iterative step, if the generated random num-ber q is not greater than q0, the tuple

(vi, hj

)with maximum

value of ταij · ηβij in �k is selected according to (17), which

helps ants quickly converge to a high quality solution. Oth-erwise, the tuple is randomly selected in �k in accordancewith the probability distribution pkmd in (18), at which timeants conduct a broader search to avoid premature stagnation.Combining multiple heuristic factor calculation criteria andpseudo-random-proportional rule defined above, we proposethe MTS (Mapping Relation Tuple Selection) algorithm. Thealgorithm selects the next tuple of mapping relation (vm, hd )based on the load status identifier heuP of the source host anddestination host. TheMTS algorithm pseudocode is as shownin Algorithm 1.

VOLUME 7, 2019 53447


Algorithm 1 Mapping Relation Tuple Selection (MTS)Input: T , heuPOutput: (vm, hd )1: switch (heuP)2: // VM from overloaded hosts to normal-loaded hosts.3: case ‘‘ON’’:4: Choose a VM selection policy SP from (11), (12), (13)

5: Compute ηmd , ∀ (vm, hd ) ∈ T with SP, (15), (10)6: Break7: // VM from overloaded hosts to underloaded hosts.8: case ‘‘OU’’:9: Choose a VM selection policy SP from (11), (12), (13)

10: Compute ηmd , ∀ (vm, hd ) ∈ T with SP, (16), (10)11: Break12: // VM from underloaded hosts to normal-loaded hosts.13: case ‘‘UN’’:14: Compute ηmd , ∀ (vm, hd ) ∈ T with (14), (15), (10)15: Break16: // VM from underloaded hosts to underloaded hosts.17: case ‘‘UU’’:18: Compute ηmd , ∀ (vm, hd ) ∈ T with (14), (16), (10)19: Break20: Compute pmd , ∀ (vm, hd ) ∈ T with (18)21: Choose (vm, hd ) ∈ T with (17)

The pseudocode of the proposed VM consolidation algo-rithm DA-VMC is shown in Algorithm 2. In the initializa-tion phase, the pheromone matrix is set as τ0 (line 2).Thealgorithm iterates over nI times (line 3). In each iteration,nA ants build new mapping relation between VMs and hostsin parallel by sequentially performing VM consolidation foroverloaded hosts and underloaded hosts (line 5-31). The antfirstly selects the VM in the overloaded hosts for redeploy-ment (line 5-17). Under the premise of ensuring serviceperformance, VMs in the underloaded hosts are redeployedto achieve energy saving (line 18-30). The local pheromoneupdate rule is applied to each traversed mapping relation(line 16 and 29). After all ants have constructed their solu-tions, all ant-specific solutions are added to the solution setS (line 31). The global optimal solution X+ is selected byusing the objective function in (6) to evaluate each solution inS (line 33). The global pheromone update rule is performedconsequently with X+ (line 34). Finally, when all the antshave iterated through nI rounds, the algorithm outputs theglobal optimal solution of the mapping relation X+.

V. PERFORMANCE EVALUATIONA. EXPERIMENT SETUPIn this section, we conduct the simulations to evaluate theperformance of our proposed approach using CloudSim [46]as the simulation platform. CloudSim is a discrete event sim-ulator that enables virtual environment modeling and virtualresource management. The simulated cloud datacenter in the

Algorithm 2 DA-VMCInput: XOutput: X+

1: S ← φ, X+← φ, X k ← φ, t ← φ

2: Initialize all pheromone values τ0 with (7)3: for i ∈ [1, nI ] do4: for k ∈ [1, nA] do5: while overloaded hosts exist do6: Ton←{(vm, hd ) |h(vm) ∈ Ho ∧ hd ∈ Hn }7: t ← MTS (Ton, ‘‘ON ")8: if t is null then9: Tou←{(vm, hd ) |h(vm) ∈ Ho ∧ hd ∈ Hu }10: t ← MTS (Tou, ‘‘OU")11: if t is null then12: Break13: end if14: end if15: Update mapping relation matrix X16: Apply local update rule with (8)17: end while18: while underloaded hosts exist do19: Tun←{(vm, hd ) |h(vm) ∈ Hu ∧ hd ∈ Hn }20: t ← MTS (Tun, ‘‘UN ")21: if t is null then22: Tuu←{(vm, hd ) |h(vm) ∈ Hu ∧ hd ∈ Hu }23: t ← MTS (Tuu, ‘‘UU")24: if t is null then25: Break26: end if27: end if28: Update mapping relation matrix X29: Apply local update rule with (8)30: end while31: S ← S ∪ {X k}32: end for33: X+← argmaxX k∈S

{F(X k)}

34: Apply global update rule on X+ using (9)35: end for

experiment consists of 800 physical hosts, half of which isHP ProLiant G4 and the other half is HP ProLiant G5. Thehost configuration details are listed in Table 1. In addition,four kinds of Amazon EC2 VMs [47] are used in this exper-iment, whose corresponding characteristics are depicted inTable 2. After creating host instances and VM instances onthe CloudSim platform, the VMs are deployed on differenthosts through the PABFD method [24]. During each VMconsolidation cycle, VM consolidation is performed basedon the new workload and resource requirements of hosts andVMs.

The workload employed in the experiment came from theCoMon project, which is responsible for monitoring the oper-ation of the infrastructure in PlanetLab [48]. The data usedin the experiment came from more than 1,000 VMs at morethan 500 places around the world. The statistical properties

53448 VOLUME 7, 2019


TABLE 1. Host configuration.

TABLE 2. VM types.

TABLE 3. The properties of PlanetLab data.

TABLE 4. DA-VMC parameters.

of the data are shown in Table 3. Since the actual workloadof the real datacenter is periodic and regular, we extractone-day workload from the workload datasets to conductexperiments. The algorithm related parameter settings areshown in Table 4.

B. EVALUATION INDICESIn cloud datacenter, SLA is used to specify the QoS require-ments of the user and the consequences of violation, enablingservice providers and users to reach an agreement on the ser-vices, priorities and responsibilities. Once an SLA violationhappens, the user’s interests may not be guaranteed, for whichthe provider may pay an expensive penalty to the user as com-pensation. For the better optimization of QoS, Beloglazovand Buyya [24] proposed several methods to measure SLAviolations.

When the resource utilization of the host reaches 100%,host overload occurs and the available resources are less thanthe total resource demand of the VMs, which may result inan SLA violation. SLAVO is defined as the proportion of timewhen the host resource utilization reaches 100% to measurethe SLA violation caused by the host overload, as shown

below:

SLAVO =1m

m∑j=1

T ojT aj, (19)

where m indicates the number of hosts, T oj is the total timethat the resource utilization of the host hj experiences 100%,and T aj is the total running time of the host hj.VM migrations cause overall performance degradation.

SLAVM represents the SLA violation due to VM migrations,which is defined as

SLAVM =1n

n∑i=1

Cdi

Cri, (20)

where n represents the number of VMs. Cdi denotes the

unsatisfied requirement for CPU resources of the VM vi dueto the migration. Cr

i is the total CPU resource requirementduring the lifetime of the VM vi. According to the previousresearch [24], the overhead Cd

i caused by VM migration isset to 10% of the VM’s CPU utilization.SLAV is employed to assess the overall QoS level of the

cloud datacenter, which comprehensively reflects the totalperformance degradation caused by host overload and VMmigrations, as shown in

SLAV = SLAVO× SLAVM . (21)

The smaller value of the variable SLAV indicates less SLAviolations and higher QoS level.

VM consolidation should optimize the energy consump-tion and SLA violation of the datacenter in a balanced man-ner. The comprehensive evaluation index ESV is obtained bycombining the energy consumption index EC and the SLAviolation index SLAV , as shown in

ESV = EC × SLAV , (22)

where EC represents the total energy consumption of the datacenter. A lower value of EC denotes higher energy efficiencyof the datacenter. And a low ESV value demonstrates thatthe datacenter has excellent performance in both energy con-sumption and QoS.

Dynamic migration of VMs generates certain overhead,such as the occupation of computing resources and energyconsumption. Further, the service suspension due to the VMmigration may degrade QoS. Therefore, reducing the numberof VM migrationsMG facilitates saving resource and energyconsumption, as well as improving QoS, which helps VMconsolidation achieve the desired performance.

VI. RESULTS ANALYSISIn this section, we firstly evaluate three types of VM selectionpolicies proposed for overloaded hosts, which are the HCPSpolicy, the LCPS policy, and the RCS policy. The evaluationresults of the three VM selection policies in energy consump-tion and SLA violation rates are shown in Table 5. Fromthe results, we can see that among the three VM selection

VOLUME 7, 2019 53449


TABLE 5. Performance evaluation of VM selection policies.

policies, the RCS policy has the lowest energy consump-tion, followed by the LCPS policy and the HCPS policy.With respect to SLAV , the RCS policy achieves the lowestvalue, followed by the HCPS policy and the LCPS policy.Obviously, the RCS policy achieves better performance ofenergy consumption and SLA violation rate. This is due tothat the RCS policy attains better randomness and thus antshave more access to different solutions. In the HCPS policy,VMs with larger CPU utilization have a higher probabilityof being selected for migration, resulting in higher energyconsumption, but at the same time, the risk of host overloadis easier to be mitigated, leading to a lower SLAV . The LCPSpolicy prefers to migrate out VMs with low CPU utiliza-tion, which saves energy whereas increasing the overloadrisk of the host, resulting in the highest SLAV . Accordingto the experimental results, the RCS policy has the bestcomprehensive performance to select migration VM for theoverloaded host. Therefore, we choose the RCS policy as theVM selection policy for overloaded hosts in the subsequentexperiments.

To evaluate the performance of the DA-VMC algorithm,we compare the proposed algorithm DA-VMC with twoheuristic VM allocation algorithms (i.e., ST [10], DT [10])and twoACO-based VM consolidation algorithms (i.e., ACS-VMC [40], EVMCACS [41]). Fig. 2 shows the compari-son of energy consumption using real workload among theapproaches. The value after the algorithm name is the currentparameter value of this algorithm. Compared with ST, DT,ACS-VMC, and EVMCACS, DA-VMC saves 38.3%, 34.1%,17.7%, and 15.1% of energy consumption respectively. SinceDA-VMC prioritizes VM migration to a normal-loaded host,many underloaded hosts are enabled to be switched to sleepmode, thus reducing the number of active hosts in the data-center and saving a lot of energy.

Regarding SLAV of the approaches as shown in Fig. 3,the proposed DA-VMC algorithm achieves the lowest valueof SLAV . In order to prevent SLA violations, DA-VMCensures to keep the CPU utilization of hosts below theoverload threshold by moving VMs from overloaded hosts.Besides, the heuristic criteria guarantee that the destinationhost does not exceed the overload threshold after deployingthe migrated VM. Therefore, DA-VMC obtains better SLAVperformance than other algorithms.

Since SLAV is a comprehensive index obtained fromSLAVO and SLAVM , SLAVO and SLAVM of the approachesare evaluated respectively as follows. Fig. 4 depicts the com-parison of SLAVO, which clearly reveals that DA-VMC hasa lower SLAVO compared with other approaches. This is

FIGURE 2. Comparison of energy consumption.

FIGURE 3. Comparison of SLAV.

FIGURE 4. Comparison of SLAVO.

primarily due to that DA-VMC limits the resource utilizationof the host under the overload threshold and reduces the riskof host resource overload, thus guaranteeing the QoS of therunning host. In addition, when redeploying the migratedVM, the DA-VMC algorithm preferentially selects the hostwith lower resource utilization in the normal-loaded hosts,which guarantees the QoS of running hosts from another per-spective. Fig. 5 shows the comparison of SLAVM among theapproaches. As observed from Fig. 5, DA-VMC has the opti-mal performance in terms of SLAVM among the approaches.

53450 VOLUME 7, 2019


FIGURE 5. Comparison of SLAVM.

FIGURE 6. Comparison of the number of VM migrations.

DA-VMC reduces the impact of migration on service qualitymainly because that DA-VMC effectively reduces the numberof triggered VM migrations as shown in Fig. 6. The combin-ing evaluation of SLAVO and SLAVM proves that DA-VMC’scapability of ensuring QoS is preferable to that of the otherapproaches.

As shown in Fig. 6, DA-VMC improves the efficiency ofmigration and obtains the minimum number of migrations.The DA-VMC approach efficiently ensures remarkable QoSof the active hosts and reduces the risk of host overload, lead-ing to a reduction in the number of VM migrations triggeredby host overload. For another, the objective function definedin the DA-VMC approach tends to minimize the number ofVM migrations.

Fig. 7 depicts the comprehensive performance of theapproaches evaluated by employing the ESV index. FromFig. 7, we can see that DA-VMC has the best overall per-formance. According to the experimental results, the ESVindex obtained by DA-VMC is only 36.1% of that of DT,and the ESV index of ST is the worst, which is up to 3.3times that of DA-VMC. The main reason is that DA-VMCeffectively reduces the risk of host overload and the number ofVM migrations by precisely identifying the host load status.

The above experiment results employing the PlanetLabworkload prove that DA-VMC achieves the goal of reducing

FIGURE 7. Comparison of ESV.

FIGURE 8. Comparison of execution time.

datacenter energy consumption, ensuring QoS and makingmore rational VM migration decisions.

In order to deeply analyze the efficiency of DA-VMC, theexecution time of the five approaches is analyzed and com-pared, as shown in Fig. 8. The heuristic algorithms ST andDThave shorter execution time than other three meta-heuristicalgorithms based onACS because of the low time complexity.Among the three ACS-based approaches, DA-VMC is supe-rior to ACS-VMC and EVMCACS in terms of execution timeperformance, because it consolidates in stages and limits thesize of candidate sets of migration VMs and destination hostsat each consolidation stage. Furthermore, we observed thatthe execution time of DA-VMC is relatively close to that ofthe two heuristic algorithms (i.e., ST and DT).

VII. CONCLUSION AND FUTURE WORKIn this paper, we propose a VM consolidation approach(DA-VMC) based on double thresholds andACS. It addressesthe problems of high PM power consumption and QoSdegradation in datacenters by consolidating VMs into appro-priate hosts. The VM consolidation problem is built as amulti-objective optimization problem. The double thresholdsare used to make decision of the consolidation conditions thattrigger VM consolidation. By treating the mapping relationbetween VMs and hosts as the food source, the mapping

VOLUME 7, 2019 53451


relation is optimized through a multi-stage consolidationbased on ACS. The optimal mapping relation between VMsand hosts is acquired globally through the distributed searchand cooperation of the artificial ants. The performance of theproposed approach is evaluated using real workload. The sim-ulation results indicate that compared with other approaches,our approach effectively reduce energy consumption andguarantee excellent QoS of the datacenter.

In the future work, we plan to conduct a further study ofemploying adaptive thresholds aiming at variable workloadto make reasonable decisions of VMmigration. Furthermore,we intend to conduct more simulations to evaluate the pro-posed approach on the real workload.

REFERENCES[1] Z.-H. Zhan, X.-F. Liu, Y.-J. Gong, J. Zhang, H. S.-H. Chung, and Y. Li,

‘‘Cloud computing resource scheduling and a survey of its evolutionaryapproaches,’’ ACM Comput. Surv., vol. 47, no. 4, p. 63, Jul. 2015.

[2] Z. Zhou et al., ‘‘Fine-grained energy consumption model of serversbased on task characteristics in cloud data center,’’ IEEE Access, vol. 6,pp. 27080–27090, 2018.

[3] D. Ardagna, G. Casale, M. Ciavotta, and J. F. Pérez, and W. Wang,‘‘Quality-of-service in cloud computing: Modeling techniques and theirapplications,’’ J. Internet Services Appl., vol. 5, no. 11, pp. 11–27,Sep. 2014.

[4] N. Saswade, V. Bharadi, and Y. Zanzane, ‘‘Virtual machine monitor-ing in cloud computing,’’ Procedia Comput. Sci., vol. 79, pp. 135–142,Dec. 2016.

[5] M. C. S. Filho, C. C. Monteiro, and P. R. M. Inácio, and M. M. Freire,‘‘Approaches for optimizing virtual machine placement and migration incloud environments: A survey,’’ J. Parallel Distrib. Comput., vol. 111,pp. 222–250, Jan. 2018.

[6] M. Dorigo and L. M. Gambardella, ‘‘Ant colony system: A cooperativelearning approach to the traveling salesman problem,’’ IEEE Trans. Evol.Comput., vol. 1, no. 1, pp. 53–66, Apr. 1997.

[7] M. Dorigo, G. Caro, and L. Gambardella, ‘‘Ant algorithms for discreteoptimization,’’ Artif. Life, vol. 5, no. 2, pp. 137–172, Apr. 1999.

[8] P. R. Theja and S. K. K. Babu, ‘‘Evolutionary computing based on QoSoriented energy efficient VM consolidation scheme for large scale clouddata centers,’’ Cybern. Inf. Technol., vol. 16, no. 2, pp. 97–112, Jun. 2016.

[9] H. Zhao, J. Wang, F. Liu, Q. Wang, W. Zhang, and Q. Zheng, ‘‘Power-aware and performance-guaranteed virtual machine placement in thecloud,’’ IEEE Trans. Parallel Distrib. Syst., vol. 29, no. 6, pp. 1385–1400,Jun. 2018.

[10] A. Beloglazov, J. Abawajy, and R. Buyya, ‘‘Energy-aware resource allo-cation heuristics for efficient management of data centers for cloud com-puting,’’ Future Generat. Comput. Syst., vol. 28, no. 5, pp. 755–768, 2012.

[11] F. Zhang, G. Liu, X. Fu, and R. Yahyapour, ‘‘A survey on virtual machinemigration: Challenges, techniques, and open issues,’’ IEEE Commun. Sur-veys Tuts., vol. 20, no. 2, pp. 1206–1243, 2nd Quart., 2018.

[12] C. D. Martino, S. Sarkar, R. Ganesan, Z. T. Kalbarczyk, and R. K. Iyer,‘‘Analysis and diagnosis of SLA violations in a production SaaS cloud,’’IEEE Trans. Rel., vol. 66, no. 1, pp. 54–75, Mar. 2017.

[13] Z. Zhou, Z. Hu, J.-Y. Yu, J. Abawajy, and M. Chowdhury, ‘‘Energy-efficient virtual machine consolidation algorithm in cloud data centers,’’J. Central South Univ., vol. 24, no. 10, pp. 2331–2341, Oct. 2017.

[14] E. Arianyan, H. Taheri, and S. Sharifian, ‘‘Novel energy and SLA efficientresource management heuristics for consolidation of virtual machines incloud data centers,’’ Comput. Elect. Eng., vol. 47, pp. 222–240, Oct. 2015.

[15] Z. Zhou, J. Yu, F. Li, and F. Yang, ‘‘Virtual machine migration algorithmfor energy efficiency optimization in cloud computing,’’ ConcurrencyComput., Pract. Exper., vol. 30, no. 24, p. e4942, Aug. 2018.

[16] A. Horri, M. S. Mozafari, and G. Dastghaibyfard, ‘‘Novel resource alloca-tion algorithms to performance and energy efficiency in cloud computing,’’J. Supercomput., vol. 69, no. 3, pp. 1445–1461, Sep. 2014.

[17] X. Chen, J.-R. Tang, and Y. Zhang, ‘‘Towards a virtual machine migrationalgorithm based on multi-objective optimization,’’ Int. J. Mobile Com-put. Multimedia Commun., vol. 8, no. 3, pp. 79–89, 2017. doi: 10.4018/IJMCMC.2017070106.

[18] D. Minarolli, A. Mazrekaj, and B. Freisleben, ‘‘Tackling uncertainty inlong-term predictions for host overload and underload detection in cloudcomputing,’’ J. Cloud Comput., vol. 6, no. 4, pp. 4–21, Feb. 2017.

[19] S. S. Masoumzadeh and H. Hlavacs, ‘‘An intelligent and adaptivethreshold-based schema for energy and performance efficient dynamic VMconsolidation,’’ in Proc. Eur. Conf. Energy Efficiency Large Scale Distrib.Syst. Vienna, Austria, 2013, pp. 85–97.

[20] Z. Zhou et al., ‘‘Minimizing SLA violation and power consumption incloud data centers using adaptive energy-aware algorithms,’’ Future Gener.Comput. Syst., vol. 86, pp. 836–850, Sep. 2018.

[21] L. Salimian, F. S. Esfahani, and M.-H. Nadimi-Shahraki, ‘‘An adaptivefuzzy threshold-based approach for energy and performance efficient con-solidation of virtual machines,’’ Computing, vol. 98, no. 6, pp. 641–660,Jun. 2016.

[22] S. Y. Z. Fard, M. R. Ahmadi, and S. Adabi, ‘‘A dynamic VM consolida-tion technique for QoS and energy consumption in cloud environment,’’J. Supercomput., vol. 73, no. 10, pp. 4347–4368, Oct. 2017.

[23] A. Abdelsamea, A. A. El-Moursy, E. E. Hemayed, and H. Eldeeb, ‘‘Virtualmachine consolidation enhancement using hybrid regression algorithms,’’Egyptian Inform. J., vol. 18, no. 3, pp. 161–170, Nov. 2017.

[24] A. Beloglazov and R. Buyya, ‘‘Optimal online deterministic algorithmsand adaptive heuristics for energy and performance efficient dynamicconsolidation of virtual machines in cloud data centers,’’ ConcurrencyComput., Pract. Exper., vol. 24, no. 13, pp. 1397–1420, Sep. 2012.

[25] Z. Cao and S. Dong, ‘‘Dynamic VM consolidation for energy-awareand SLA violation reduction in cloud computing,’’ in Proc. 13th Int.Conf. Parallel Distrib. Comput. Appl. Technol., Beijing, China, Dec. 2012,pp. 363–369.

[26] S. S. Masoumzadeh and H. Hlavacs, ‘‘Dynamic virtual machine consolida-tion: Amulti agent learning approach,’’ inProc. IEEE Int. Conf. AutonomicComput., Grenoble, France, Jul. 2015, pp. 161–162.

[27] G. F. Shidik, A. Azhari, and K. Mustofa, ‘‘Improvement of energy effi-ciency at cloud data center based on fuzzy Markov normal algorithm vmselection in dynamic vm consolidation,’’ Int. Rev. Comput. Softw., vol. 11,no. 6, pp. 511–520, Jun. 2016.

[28] H. Li, W. Li, H. Wang, and J. Wang, ‘‘An optimization of virtual machineselection and placement by using memory content similarity for serverconsolidation in cloud,’’ Future Gener. Comput. Syst., vol. 84, pp. 98–107,Jul. 2018.

[29] Y. Laili, F. Tao, F. Wang, L. Zhang, and T. Lin, ‘‘An iterative budget algo-rithm for dynamic virtual machine consolidation under cloud computingenvironment (revised December 2017),’’ IEEE Trans. Serv. Comput., to bepublished. 2018.

[30] M. Mishra and A. Sahoo, ‘‘On theory of VM placement: Anomalies inexisting methodologies and their mitigation using a novel vector basedapproach,’’ in Proc. IEEE 4th Int. Conf. Cloud Comput., Washington, DC,USA, Jul. 2011, pp. 275–282.

[31] A. Murtazaev and S. Oh, ‘‘Sercon: Server consolidation algorithm usinglive migration of virtual machines for green computing,’’ IETE Tech. Rev.,vol. 28, no. 3, pp. 212–231, 2011.

[32] F. Farahnakian, T. Pahikkala, P. Liljeberg, J. Plosila, N. T. Hieu, andH. Tenhunen, ‘‘Energy-aware VM consolidation in cloud data centersusing utilization prediction model,’’ IEEE Trans. Cloud Comput., to bepublished. doi: 10.1109/TCC.2016.2617374.

[33] W. Fang, X. Liang, S. Li, L. Chiaraviglio, and N. Xiong, ‘‘VMPlanner:Optimizing virtual machine placement and traffic flow routing to reducenetwork power costs in cloud data centers,’’ Comput. Netw., vol. 57, no. 1,pp. 179–196, 2013.

[34] M. Li, J. Bi, and Z. Li, ‘‘Improving consolidation of virtual machine basedon virtual switching overhead estimation,’’ J. Netw. Comput. Appl., vol. 59,pp. 158–167, Jan. 2016.

[35] Z. Huang and D. H. K. Tsang, ‘‘M-convex VM consolidation: Towards abetter VM workload consolidation,’’ IEEE Trans. Cloud Comput., vol. 4,no. 4, pp. 415–428, Oct./Dec. 2016.

[36] Z. Li, C. Yan, L. Yu, and X. Yu, ‘‘Energy-aware and multi-resource over-load probability constraint-based virtual machine dynamic consolidationmethod,’’ Future Gener. Comput. Syst., vol. 80, pp. 139–156, Mar. 2018.

[37] A. Mosa and N. W. Paton, ‘‘Optimizing virtual machine placement forenergy and SLA in clouds using utility functions,’’ J. Cloud Comput.,vol. 5, no. 1, pp. 1–17, Dec. 2016.

[38] H. Li, G. Zhu, C. Cui, H. Tang, Y. Dou, and C. He, ‘‘Energy-efficientmigration and consolidation algorithm of virtual machines in data centersfor cloud computing,’’ Computing, vol. 98, no. 3, pp. 303–317, Mar. 2016.

53452 VOLUME 7, 2019

http://dx.doi.org/10.4018/IJMCMC.2017070106

http://dx.doi.org/10.4018/IJMCMC.2017070106

http://dx.doi.org/10.1109/TCC.2016.2617374


[39] M. H. Ferdaus, M. Murshed, R. N. Calheiros, and R. Buyya, ‘‘Virtualmachine consolidation in cloud data centers using ACO metaheuristic,’’in Proc. Eur. Conf. Parallel Process., Porto, Portugal, 2014, pp. 306–317.

[40] F. Farahnakian et al., ‘‘Using ant colony system to consolidate VMs forgreen cloud computing,’’ IEEE Trans. Services Comput., vol. 8, no. 2,pp. 187–198, Mar. 2015.

[41] A. Aryania, H. S. Aghdasi, and L. M. Khanli, ‘‘Energy-aware virtualmachine consolidation algorithm based on ant colony system,’’ J. GridComput., vol. 16, no. 3, pp. 477–491, Sep. 2018.

[42] L.-D. Chou, H.-F. Chen, F.-H. Tseng, H.-C. Chao, and Y.-J. Chang,‘‘DPRA: Dynamic power-saving resource allocation for cloud data cen-ter using particle swarm optimization,’’ IEEE Syst. J., vol. 12, no. 2,pp. 1554–1565, Jun. 2018.

[43] Q. Zheng, R. Li, X. Li, and J. Wu, ‘‘A multi-objective biogeography-basedoptimization for virtual machine placement,’’ in Proc. 15th IEEE/ACMInt. Symp. Cluster Cloud Grid Comput., Shenzhen, China, May 2015,pp. 687–696.

[44] Q. Chen, J. Chen, B. Zheng, J. Cui, and Y. Qian, ‘‘Utilization-based VMconsolidation scheme for power efficiency in cloud data centers,’’ in Proc.IEEE Int. Conf. Commun. Workshop (ICCW), London, U.K., Jun. 2015,pp. 1928–1933.

[45] Z. Li, C. Yan, X. Yu, andN. Yu, ‘‘Bayesian network-based virtual machinesconsolidation method,’’ Future Gener. Comput. Syst., vol. 69, no. 3,pp. 75–87, Apr. 2017.

[46] R. Calheiros, R. Ranjan, A. Beloglazov, C. A. F. De Rose, and R. Buyya,‘‘CloudSim: A toolkit for modeling and simulation of cloud computingenvironments and evaluation of resource provisioning algorithms,’’ Softw.,Pract. Exper., vol. 41, no. 1, pp. 23–50, 2011.

[47] F. Durao, J. F. S. Carvalho, A, Fonseka, and V. C. Garcia, ‘‘A sys-tematic review on cloud computing,’’ J. Supercomput., vol. 68, no. 3,pp. 1321–1346, Jun. 2014.

[48] K. Park and V. S. Pai, ‘‘CoMon: A mostly-scalable monitoring system forPlanetLab,’’ ACMSIGOPSOper. Syst. Rev., vol. 40, no. 1, pp. 65–74, 2006.

HUI XIAO received the B.S. degree from theSchool of Software, Shandong University, in 2017.She is currently pursuing the master’s degree insoftware engineering with Central South Univer-sity. Her main research interests include cloudcomputing, energy efficient datacenter, and virtualmachine management.

ZHIGANG HU received the M.S. and Ph.D.degrees from Central South University, in 1988and 2002, respectively, where he is currentlya Professor with the School of ComputerScience and Engineering. He has publishedover 200 research papers. His research interestsinclude high performance computing, cloud com-puting, energy-efficient resource management,and virtual machine deployment.

KEQIN LI is a SUNY Distinguished Professorof computer science with the State University ofNew York. He is also a Distinguished Professorwith Hunan University, China. He has publishedover 640 journal articles, book chapters, and refer-eed conference papers. His current research inter-ests include cloud computing, fog computing andmobile edge computing, energy-efficient comput-ing and communication, embedded systems andcyberphysical systems, heterogeneous computing

systems, big data computing, high-performance computing, CPU-GPUhybrid and cooperative computing, computer architectures and systems,computer networking, machine learning, intelligent, and soft computing.He is an IEEE Fellow. He has received several best paper awards. He cur-rently serves or has served on the Editorial Boards of the IEEE TRANSACTIONS

ON PARALLELANDDISTRIBUTED SYSTEMS, the IEEETRANSACTIONSONCOMPUTERS,the IEEE TRANSACTIONS ON CLOUD COMPUTING, the IEEE TRANSACTIONS ON

SERVICES COMPUTING, and the IEEE TRANSACTIONS ON SUSTAINABLE COMPUTING.

VOLUME 7, 2019 53453

multi-objective vm consolidation based on thresholds and...

Documents