Energy-efficient Management of Virtual Machinesin Eucalyptus

Pablo Graubner, Matthias Schmidt, Bernd FreislebenDepartment of Mathematics and Computer Science, University of Marburg

Hans-Meerwein-Str. 3, D-35032 Marburg, Germanye-mail: {graubner,schmidtm,freisleb}@informatik.uni-marburg.de

Abstract— In this paper, an approach for improving the energyefficiency of infrastructure-as-a-service clouds is presented. Theapproach is based on performing live migrations of virtualmachines to save energy. In contrast to related work, the energycosts of live migrations including their pre- and post-processingphases are taken into account, and the approach has beenimplemented in the Eucalyptus open-source cloud computingsystem by efficiently combining a multi-layered file system anddistributed replication block devices. To evaluate the proposedapproach, several short- and long-term tests based on virtualmachine workloads produced with common operating systembenchmarks, web-server emulations as well as different MapRe-duce applications have been conducted. The results indicate thatenergy savings of up to 16 percent can be achieved in a productiveEucalyptus environment.

I. INTRODUCTION

The Climate Group and the Global e-Sustainability Initiative(GeSI) [1] have predicted that the demand for services ofthe information technology and communication sector willhave quadrupled by 2020. At the same time, costs for energy,infrastructure, cooling and power supply will keep rising.

It is therefore not surprising that several companies andorganizations are currently trying to reduce their total numberof servers as well as their number of data centers. Flexibleon-demand infrastructure service providers with pay-as-you-go pricing models, such as Amazon’s Elastic Compute Cloud(Amazon EC2) [2], are becoming an alternative to a risingTotal Cost of Ownership (TCO) for a company’s infrastructure.On the other hand, there is a growing interest of public cloudservice providers to reduce their energy related costs. In fact,increasing the energy-efficiency of cloud installations is oneof the major research fields in cloud computing today.

Although several approaches to save energy by virtual ma-chine (VM) consolidation in infrastructure-as-a-service cloudshave been proposed in the literature, the energy costs ofVM live migrations during their pre- and post-processingphases and during the live migration itself are not taken intoaccount. Furthermore, most approaches do not seem to havebeen implemented in a productive Cloud environment, arebased on theoretical considerations or rely on simulationsonly. To consider the effects of implementation details onenergy efficiency, both algorithm design and implementationare important.

In this paper, an approach for achieving energy savingsin a productive infrastructure-as-a-service cloud computing

environment is presented. The approach is based on per-forming VM live migrations to save energy. The commonlyused cloud computing environment EUCALYPTUS, an open-source reimplementation of Amazon EC2, is the basis of ourwork. EUCALYPTUS offers a simple scheduling algorithm thatautomatically turns off unused servers, but does not providesupport for VM live migrations. Typically, live migration asimplemented by the Xen Virtual Machine Monitor [3] is basedon the availability of the virtual disk image on the source andthe destination server. This can either be achieved with sharedstorage, or by distributing the virtual disk image prior to livemigration. Since EUCALYPTUS uses Walrus, a reimplemen-tation of the Amazon S3 interface, a method of distributingvirtual disk images to servers is already available. To makethe virtual disk available prior to live migration, a mirroringtechnique based on distributed replication block devices isused. Since network bandwidth is a common bottleneck ina cloud computing environment [4], a multi-layered root filesystem is used, aimed at reducing disk image transmissiontime by transmitting only small, separate layers instead ofentire, potentially large, disk images.

To evaluate the proposed approach, several short- and long-term tests based on VM workloads produced with commonoperating system benchmarks, web-server emulations as wellas different MapReduce applications have been conducted. Theresults indicate that energy savings of up to 16 percent can beachieved in a productive EUCALYPTUS environment.

This paper is organized as follows. Section II discussesrelated work. Section III states the problems involved indesigning an energy-efficient approach for infrastructure-as-a-service clouds. Section IV presents our novel approach of VMconsolidation based on live migrations. Section V discussesimplementation issues. Section VI presents experimental re-sults. Section VII concludes the paper and outlines areas forfuture work.

II. RELATED WORK

Server consolidation is an approach to maximize resourceutilization while reducing energy consumption. There areseveral studies about different aspects of server consolidationin a cloud computing environment.

Srikantaiah et al. [5] have studied the inter-relationshipsbetween energy consumption, resource utilization, and perfor-mance of consolidated workloads. The consolidation problem

2011 IEEE 4th International Conference on Cloud Computing

978-0-7695-4460-1/11 $26.00 © 2011 IEEE

DOI 10.1109/CLOUD.2011.26

243

is modeled as a modified bin packing problem. However, thisapproach is application and workload dependent. In contrast,our approach is based on VMs, allowing to control theentire software stack from the kernel upwards, leading togeneric workloads. This is what is usually offered by a cloudcomputing service such as Amazon EC2.

Verma et al. [6] have investigated a power and migrationcost aware application placement controller in virtualizedheterogeneous systems, minimizing power subject to a fixedperformance requirement. The authors have designed andimplemented pMapper, a power-aware application placementframework that continuously optimizes the allocation. VMsare live migrated from one server to another, the performancedegradation due to migrations is taken into account. However,no details about the method for calculating the migration costsare presented. In contrast to our work, the mechanisms for livemigration itself are not evaluated.

Energy efficiency has been addressed by Nathuji et al. [7]on the hypervisor level. VirtualPower, a Xen hypervisor im-plementation to control and coordinate the effects of VM-levelpower management policies on virtualized resources has beendeveloped. It provides soft and hard techniques, referring tosoftware solutions like limiting the hardware usage of a VM orscaling the frequency of a processor, respectively. To achievethis, Nathuji et al. have implemented artificial VirtualPowerManagement (VPM) states, used by VPM mechanisms andserving as a base for more complex VPM rules. Their approachrelies on evaluating a global consolidation strategy based onVM live migration and VPM rules, aimed at mapping multipleinstances of virtual resources to appropriate efficient physicalresources. However, the durations of VM live migrations usedin this work are quite short due to the migration of localpersistent storage. In contrast, EUCALYPTUS or Amazon EC2offer gigabyte-sized disk images to the user.

Van et al. [8] have proposed a resource management frame-work combining a utility-based dynamic VM provisioningmanager and a dynamic VM placement manager. The paperdescribes VM live migration as a technique used for max-imizing resource utilization. The authors consider access toa shared storage space for storing VM disk images. In ourapproach, live migrations are executed without shared storage,avoiding network bottlenecks.

Beloglazov et al. [9] have focused on energy-efficientresource management strategies that can be applied to avirtualized data center by a cloud provider. The authors haveproposed the development of thermal, network and multiplesystem resources optimizations, embedded in a decentralizedarchitecture of a resource management system for cloud datacenters. The approach is based on live migration to dynam-ically reallocate VMs. The architecture consists of a dis-patcher and global managers, allocating thermal, networkingand workload information from local managers as well asrunning distributed reallocation decisions. Beloglazov et al.have used a distributed version of a semi-online multidimen-sional bin-packing algorithm, focusing on the system’s energyefficiency extension as well as considering quality-of-service

(QoS) requirements. The approach has been evaluated usingsimulations, showing a reduced power consumption in com-bination with assured QoS. But in contrast to our approach,its implementation as part of a real-world cloud platformis only planned for the future. This is a fact that nearlyall papers related to energy-efficient resource managementin cloud computing have in common. Therefore, increasingthe potential for power savings in a real implementation of acloud computing system, i.e. with realistic costs for VM livemigration, is an interesting challenge.

III. PROBLEM STATEMENT

The scheduling algorithm implemented in EUCALYPTUS isalready able to suspend or shutdown idle servers in a cluster.Nevertheless, depending on the usage profile of the cloud,VM relocations are able to increase the number of suspendedservers by consolidating VMs on a minimum set of servers.There are several practical problems to solve:

1) Due to the on-demand nature of cloud computing, thereis an unpredictable user behavior in terms of startupsand terminations of VMs. Since it is possible thatVM startups and terminations can happen during VMmigration or its pre- and postprocessing phases, thiscan change the number of idle servers and may leadto additional energy consumption.

2) There is an additional overhead spent for live migrationand its pre- and post processing phases.

3) Consolidated VMs share resources (hardware, cachesetc.), i.e., there are interferences between VMs runningon the same server.

These problems may influence the energy efficiency of acloud significantly. To face them, different approaches can betaken: either optimizing the scheduling algorithm by consider-ing/predicting different workloads of VMs or their executiontimes or optimizing the system in terms of energy efficiency.The first approach requires additional information originat-ing from performance trackers and may lead to additionaloverhead, which is often unacceptable for cloud computingproviders [5]. The second approach is the approach proposedin this paper: developing a solution that reduces energy con-sumption and integrating it smoothly into EUCALYPTUS witha minimum additional performance or energy overhead.

IV. SAVING ENERGY IN EUCALYPTUS

In our approach, live migration is limited to clusters ratherthan a set of clusters in a cloud. This is due to the complexityof live migration between two clusters: A VLAN needs to bespanned between all servers inside a virtual network, usingthe cluster controller as a router between two subnets. In thiscase, network virtualization leads to a significant performancedegradation [10]. Since our approach is based on the synchro-nization of gigabyte-sized disks, this would have a significantimpact on live migration.

244

A. Power Management

In general, power management requires an abstract interfacethat is as independent of hardware vendor specifications.This can be found in the Advanced Configuration and PowerInterface (ACPI) [11] that is the prevailing power managementinterface today.

State Time PowerG0-S0 (100% utilization) / 110 WG0-S0 (idle) / 62 WG1-S1 7.1s 57 WG1-S3 suspend-to-RAM 11.2s 6 WG1-S4 suspend-to-disk (without Xen) 28.8s 4 WG2-S5 51.8s 4 WG2-S5 (without Xen) 37.7s 4 W

TABLE ICOMPARISON OF THE TIME SPENT FOR A SLEEP/WAKE UP CYCLE AND THE

POWER CONSUMPTION IN DIFFERENT ACPI STATES.

The system states available for ACPI differ in thesleep/wake up time and in their power consumption. TableI compares both factors for a recent Intel Pentium 3.00 GHzCPU. It shows a high difference of 56 W between the statesG1-S0 (idle) and G1-S3 (suspend-to-RAM). Setting serversto suspend-to-RAM when they are idle ensures both a smallsleep/wake up penalty as well as a small power consumption.This is the basis for the energy savings proposed in this paper.

B. Storage Synchronization

In contrast to network file system approaches, such asaccessing disk images using the Network File System (NFS)protocol, EUCALYPTUS does not need permanent remote ac-cess to a file server via a network. In fact, this would lead tonetwork traffic during the access to a disk image. Even if therewere no ongoing live migrations, there would be an overhead.

To avoid shared storage, Distributed Replicated Block De-vices (DRBD) [12] are used. This concept originated fromhigh-availability computing and can be used for data storage ina distributed system. DRBD replicates data storage to differentlocations in a stable, fault tolerant way. The DRBD modulecan operate in two modes: stand-alone and synchronized. Instand-alone mode, all disk accesses are simply passed to theunderlying disk driver. In synchronized mode, disk writes areboth passed to the underlying disk driver and sent to thebackup machine via a TCP connection. Disk reads are servedlocally.

Since Armbrust et al. [4] have identified data transferbottlenecks as an obstacle for cloud computing services,network traffic through storage synchronization should bereduced. Therefore, the synchronization of gigabyte-sized diskimages is based on the multi-layered root file system (MLRFS)approach developed by Schwarzkopf et al. [13]. They useda MLRFS to centrally apply security updates and to reducedisk image transmission time. Therefore, only small, separatelayers instead of the whole, potentially large disk images aretransmitted. All layers are overlaid transparently and form

a single coherent filesystem using a Copy-On-Write (COW)mechanism. In our approach, a read-only base layer containingthe disk image downloaded from Walrus is used in combi-nation with a writable memory disk, covering read-/write-accesses to the root file system. Live migration provided byXen automatically synchronizes the virtual memory betweensource and destination host, i.e., there is no need for additionalreplication here.

The size of the memory disk restricts VM reconfigurationsas well as additional software installations. On the other hand,most of the software needed at runtime is installed before it isuploaded to Walrus, hence the installations made in a runningVM are lost after its termination due to the EUCALYPTUSinstance cleanup process.

As described by Haselhorst et al. [14], both mechanismscan be used for live migrations in cloud computing environ-ments. From an energy perspective, both are very efficient,as indicated by our experiments. Stand-alone DRBD devicesand a MLRFS do not increase the power consumption of aserver. Until the relocation of a VM is initiated, the powerconsumption of a server is not affected.

C. Eucalyptus Integration

EUCALYPTUS consists of a set of web services: Walrus(storage controller, a reimplementation of the Amazon S3interface), Cloud Controller (user frontend), Cluster Controller(handling clusters of servers), Node Controller (executes onand controls server) [10]. Although there are other hypervisorssupported by EUCALYPTUS, this paper focusses on Xen, asimplemented in Amazon EC2. The Node Controller maintainsthe state of the instance and is periodically polled by theCluster Controller.

An EUCALYPTUS instance consists of a kernel/ramdisktuple, a root file system, swap space, potentially an ephemeraldevice as well as network interfaces. In the following, the terminstance is used for EUCALYPTUS instance. Analogically, theterm instance relocation is used for the relocation of a wholeinstance, including VM live migration and the transformationof its state.

Instance relocations are initiated and controlled by theCluster Controller. For this purpose, two components areadded: A relocation agent and an instance relocation algorithm.

The relocation agent is a separate component inside theCluster Controller. It is responsible for providing an inter-face enabling other components to initiate and abort instancerelocations, handling state transitions and events during theinstance relocation process. However, the relocation agentis strongly connected to the monitoring thread, which ismodified to periodically poll the state of ongoing instancerelocations. There are two major design goals for the relocationagent: It should be reliable, i.e., timeouts of all kinds, NodeController errors or the loss of a connection should not leadto an undefined state. Furthermore, it should be scalable. Theoverhead should grow linearly with the number of relocations.

The instance relocation algorithm follows a centralizedserver approach, reflecting the architecture of EUCALYPTUS.

245

Sour

ce D

RBD

Lay

er

Des

tinat

ion

DR

BD L

ayer

synchronization

Vir

tual

izat

ion

Laye

r

Vir

tual

izat

ion

Laye

r

live migration

1. .

.

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

t

. . .

. . .

3

2

4 4

. . .

. . .

. . .

Source Server:

Destination Server:

Fig. 1. VM relocation process.

Parameters such as the interval the instance relocation algo-rithm runs and, evidently, the type of the instance relocationalgorithm should be configurable by the EUCALYPTUS admin-istrator. In general, The instance relocation algorithm has tosupervise the available set of servers, their state as well astheir set of available resources such as the number of virtualCPUs, the amount of virtual memory and disk space.

Based on this information, the algorithm initiates instancerelocations that lead to additional server suspensions. It takesinto account general and EUCALYPTUS-specific restrictionsfor instance relocations: in general, only running instancesare relocated. VM instances that have been prepared andnot started yet as well as instances currently terminating areignored.

Figure 1 shows the interconnection between source anddestination server during the instance relocation process. Eachstep is initiated by the Cluster Controller relocation agent,while the Node Controller supervises the operations on thecorresponding server:

1) Instance Startup: The disk, kernel and ramdisk imagesare downloaded from Walrus. MLRFS support is addedand the underlying DRBD devices are initialized, beforethe instance is started.

2) Pre-Processing: On the destination server, the prepara-tion of the instance relocation process is similar to an in-stance startup process: The images are downloaded fromWalrus, MLRFS and DRBD devices are configured.

3) Migration: The DRBD devices are synchronized, af-terwards the live migration from source to destinationserver is performed.

4) Post-Processing: On the source server, all remainingdata is removed. On the destination server, the instanceis activated and maintained by the destination NodeController.

V. IMPLEMENTATION

The implementation of the proposed approach is based onEUCALYPTUS version 1.6.2 and Xen version 3.2.11. In par-ticular, the Node Controller implementation is integrated intothe Xen specific parts of the EUCALYPTUS Node Controller.The source code is available on our website.2

A. Cluster Controller

In general, the Cluster Controller is written in the C pro-gramming language, using Axis2/C Web Service operations tocommunicate with the Node Controllers. Instance relocationsare transparent to the Cloud Controller; it is not affected inthis implementation.

The instance relocation algorithm uses a resource map, ba-sically consisting of a mapping between servers and instancesafter all relocations are done.

The relocation agent handles state transitions of a relocation.It consists of two functions to initiate and abort instancerelocations and a state machine for handling state transitionsand events. The state is stored in a separate cache and isperiodically updated by the monitoring thread, which pollsthe state of ongoing instance relocations from the Node Con-troller by using a newly integrated DescribeRelocationoperation. The relocation agent periodically checks its cacheand initiates the pre-processing, migration and post-processingphases. Multiple instance relocations can be handled concur-rently.

B. Node Controller

The synchronization between Primary and SecondaryDRBD devices is achieved by the synchronous replication(DRDB protocol C) configuration. The network bandwidthused during live migration as well as the TCP ports connectingthe DRBD devices are configurable.

All layers of the MLRFS are overlaid using an AUFSfile system. The writable memory disk, covering read-/write-accesses to the root file system, is implemented as a temporaryfile system (TMPFS) with a configurable size.

The Xen migration process requires identical device names(e.g., loopback mounted disk images) on both source anddestination server. Since all resources, including loopbackdevices and DRBD devices, are acquired independently by theCluster Controller and other Node Controllers, their names canoverlap. Therefore, symbolic links pointing to the device filesystem are used to cover the real device names. This ensuresnormal access to the real device as well as arbitrary naming ofthe symbolic link file. The symbolic links are stored in the userinstance directory and are further used for Xen configuration.

In the following, the different phases of the instance relo-cation process according to Figure 1 are described.

1Xen supports host-based suspend-to-ram since version 3.2; it is requiredfor this implementation.

2http://ds.mathematik.uni-marburg.de/˜graubner/euca/

246

1) Instance Startup: The RunInstance operation ismodified in order to modify both the Xen configurationfor multi-layer root file system support and the swap andephemeral disk creation for initializing the underlying DRBDdevices. For this purpose, loopback devices are initialized,DBRD metadata is created and the DRBD devices are attachedto the loopback devices. Since DRBD devices typically usea meta-disk to store metadata information, the DRBD meta-disk in this implementation consists of a 128 MB sparse imagethat is stored in the meta-disk’s local user instance directory.During the VM startup process, a separate Perl helper scriptgen libvirt xml is used to generate the VM configurationfile.

2) Pre-Processing: A new RelocationPreparationoperation is integrated into the Node Controller Web Service,which is responsible for preparing the disk images and theXen configuration on the destination host. As a parameter,the instance state (containing the VM configuration, e.g., thedisk image location in Walrus or the network configuration)is transferred. The root file system is either downloaded viaWalrus or copied from the local instance cache, the Xen con-figuration is created, and the network and the DRBD devicesare initialized. This process is performed asynchronously, itsstate is polled by the Cluster Controller. The thread waits forthe end of the device synchronization. After that, all DRBDdevices are set to dual-primary mode.

3) Migration: The newly integratedRelocationMigration operation signals to the sourceNode Controller that the destination Node Controller isready for migration. The TCP ports are used to set up theconnection between Primary and Secondary DRBD devices,and the synchronization is started. The thread waits for theend of the device synchronization and starts the live migrationusing the libvirt virDomainMigrateToURI() functionafterwards. Since supervising the state of an instance duringits migration may lead to failures, state monitoring is stoppedduring the live migration. Therefore, the Node Controllermonitoring thread is modified.

4) Post-Processing: The newly integratedRelocationCommit operation signals to both sourceand destination Node Controller that the live migrationprocess has been completed. On the source Node Controllerthe relocated instance has to be removed, on the destinationNode Controller the relocated instance has to be added tothe instance cache. Resources allocated during the relocationprocess are freed and the remaining DRBD devices aredisconnected. The source Node Controller’s cleanup process,i.e., the removal of the local instance directory, the DRBDdevice detaching from the loopback devices and their metadataremoval are performed by the monitoring thread.

VI. EXPERIMENTAL RESULTS

To demonstrate that the proposed approach indeed savesenergy, we conducted a set of measurements. This includesseveral short- and long-term tests based on instance workloads

produced with common operating system benchmarks, web-server emulations, and different MapReduce applications.

The hardware configuration consists of Intel Pentium CPU3.00 GHz test servers equipped with 4096 MB memory. Thestartup/terminate experiments are performed on two servers,the other experiments with three servers. The power andenergy measurements are taken with an EasyMeter [15] elec-tricity meter. It is connected to a separate ethernet network viathe Multi Utility Communication (MUC) tool and periodicallypolled at a 5 Hz rate.

All EUCALYPTUS images are Xen compatible Debian Linuximages, with an instance configuration of one virtual CPU,384 MB memory and 3 GB total disk space, with a root filesystem size of 1537 MB and an ephemeral device size of1019 MB. Additionally, 512 MB swap space is available. Anenabled multi-layer root file system works with a 196 MBsized writeable layer and a DRBD device with a synchro-nization rate of 50 MByte/s.

A. Startup/Terminate Experiments

0 s

625 s

1,250 s

1,875 s

2,500 s

Configuration A Configuration B

2259

1180

2128

1051

Executio

n T

ime (s)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 s

625 s

1,250 s

1,875 s

2,500 s

Configuration A Configuration B

2203

1125

2104

1051

Executio

n T

ime (s)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 Wh

50 Wh

100 Wh

150 Wh

200 Wh

Configuration A Configuration B

144

74

171

82Energ

y (W

h)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 Wh

50 Wh

100 Wh

150 Wh

200 Wh

Configuration A Configuration B

158

83

178

93

Energ

y (W

h)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 Wh

125 Wh

250 Wh

375 Wh

500 Wh

1200s sleep interval 2400s sleep interval

349

403

353

411

Energ

y (W

h)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

Fig. 2. Energy consumption of the Startup/Terminate experiment for twotest configurations.

The first experiment is intended to observe the power andenergy consumption of an EUCALYPTUS cloud with randomlygenerated user requests. Its basic principle is to send requestsfrom an external client to the EUCALYPTUS cloud, either tostart or terminate a random number of instances alternately.The only workload created by the instances are CPU, memoryand disk operations during boot time. Two different mea-surement configurations were used: One with a random sleepinterval of 1200 seconds between client operations, the otherone with a 2400 seconds sleep interval. The sleep intervalensures that there is enough time between a startup/terminatesequence to actually observe power and energy consumption.

The experimental results shown in Figure 2 indicate that theinstance relocation algorithm does not lead to an increasedenergy consumption. The reduction between one and twopercent of the energy consumption for the energy-efficientEucalyptus in comparison to an unmodified Eucalyptus is dueto small differences in power consumption of the differentservers (between one and two Watts). The energy-efficientEucalyptus coincidently relocated instances to a server withthe smaller power consumption.

247

The differences in energy consumption between the twoconfigurations stem from the different number of instancesstarted during the experiment (21 and 13). First, power con-sumption peaks can be observed in correlation to the VMboot process during the start of an instance. Second, the totalnumber of instances running in the first configuration is oftenhigher than in the second configuration. In the latter, theservers can be suspended more often, which leads to a smallerenergy consumption.

B. MapReduce Experiment

This experiment monitors energy consumption of an Euca-lyptus cloud during the execution of a distributed application.Since MapReduce is widely used in cloud computing environ-ments (such as Amazon Elastic MapReduce), it is a good can-didate for this type of experiment. In the following, an ApacheHadoop MapReduce implementation in combination with aHadoop File System (HDFS) [16] is used. As a distributedfile system, HDFS spreads data on different nodes to increasedata availability and reliability. The typical configuration of aMapReduce consists of one master node, controlling severalslave nodes. The master serves as NameNode and JobTracker,while the slaves are serving as DataNodes and TaskTracker.

The test applications are automatically installed and initiatedby an external client on previously started EUCALYPTUSinstances inside the test cloud. The maximum number of tasksspawned simultaneously on a TaskTracker are set to its default,two mappers and two reducers on a TaskTracker. All test data,the HDFS as well as the Hadoop installation itself are stored onthe ephemeral device of the EUCALYPTUS instance. Initially,the instances are placed on three different servers: Server 1hosts two instances (one JobTracker and one TaskTracker),Server 2 hosts two instances (one TaskTracker and an idleinstance) and Server 3 hosts one instance (one TaskTracker).After the experiment has been started, the idle instance onServer 2 is terminated.

1) Wordcount: The Wordcount experiment is aimed to showthe processing of a large amount of data in a cloud. TheWordcount application reads a text file and counts how oftena word occurs. The test data consist of the complete Linuxkernel 2.6.33.1 including source code, configuration files anddocumentation, concatenated and stored in a single file. It isperformed with two different test configurations:

• test configuration A: Wordcount is performed on text fileswith 110.4 MB total size 10 times in a row.

• test configuration B: Wordcount is performed on text fileswith 110.4 MB total size 20 times in a row.

Figure 3(a) shows the energy consumption of Wordcountin both test configurations. The energy-efficient Eucalyptusfinishes 80 seconds after Server 3 is suspended, resulting ina reduced energy consumption of 8 Wh, i.e., 10 percent ofthe total energy consumption of the unmodified Eucalyptus.In configuration B, our implementation saves 27 Wh, i.e., 16percent of the energy consumed by unmodified Eucalyptus.

Figure 3(b) shows the execution time of Wordcount in bothtest configurations. For both configurations, the execution time

of the energy-efficient Eucalyptus is increased (configurationA: 7 percent, configuration B: 5 percent). The importantpart of the difference between the unmodified Eucalyptusand the energy-efficient Eucalyptus comes from the instancerelocation process: the parts of the test executed during DRBDsynchronization and VM live migration are up to 30 percentslower than the other parts. In absolute values, the executiontime is increased by about more than 40 seconds. In fact, a 30percent slower execution time may not be acceptable in somecases. But in long-term calculations MapReduce is typicallyused for, this does not dominate the total execution time.

The absolute execution time of Wordcount is increasedbetween both configuration A and configuration B. The partsof Wordcount executed after the instance relocation are slightlyincreased (smaller than one percent compared to the un-modified Eucalyptus). This can be interpreted as a slightperformance degradation due to resource sharing on Server2. In fact, in the long term, the performance degradation isabout 5 percent.

Figure 3(c) shows the energy consumption over time forboth the unmodified Eucalyptus and the energy-efficient Eu-calyptus in configuration B. Since both configurations showa similar pattern for both energy and power consumption,configuration A is not depicted. The filled region in the figureshows a difference between both energy consumptions. Thiscan be interpreted as a potential energy saving for time t ifthe test was finished at t.

There are two turning points in the figure: The first oneis marked with a vertical dotted line and describes the mo-ment when Server 3 is suspended. Up to this moment, thepotential energy saving is only fluctuating (due to a varyingactualization rate of the electricity meter). Afterwards, thegradient of the energy-efficient Eucalyptus gets lower whilethe gradient of the energy consumption of the unmodifiedEucalyptus remains the same.

The second turning point is the moment when the unmod-ified Eucalyptus test is completed. Until then, the differencebetween the energy-efficient Eucalyptus and the unmodifiedEucalyptus is 35 Wh. Afterwards, the remaining executiontime of the energy-efficient Eucalyptus reduces the potentialenergy savings. Finally, the energy saved is 27 Wh.

This figure shows both the tradeoff between reduced energyconsumption and increased execution time: The more theperformance has been degraded, the less energy is saved.

2) Estimation of π: The PiEstimation experiment is aimedat evaluating CPU- and memory-intensive operations in acloud. Therefore, a MapReduce program to estimate the valueof Pi using the quasi-Monte Carlo method is performed withtwo different test configurations:

• test configuration A: PiEstimation is performed with1,000 maps and 1,000,000 points.

• test configuration B: PiEstimation is performed with2,000 maps and 1,000,000 points.

Figure 4(a) shows the energy consumption of PiEstimationin both test configurations. Although configuration A finishes60 seconds after Server 3 is suspended, our implementation

248

0 s

625 s

1,250 s

1,875 s

2,500 s

Configuration A Configuration B

2259

1180

2128

1051

Executio

n T

ime (s)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 s

625 s

1,250 s

1,875 s

2,500 s

Configuration A Configuration B

2203

1125

2104

1051

Executio

n T

ime (s)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 Wh

50 Wh

100 Wh

150 Wh

200 Wh

Configuration A Configuration B

144

74

171

82Energ

y (W

h)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 Wh

50 Wh

100 Wh

150 Wh

200 Wh

Configuration A Configuration B

158

83

178

93

Energ

y (W

h)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 Wh

125 Wh

250 Wh

375 Wh

500 Wh

1200s sleep interval 2400s sleep interval

349

403

353

411

Energ

y (W

h)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

(a) Wordcount energy consumption.

0 s

625 s

1,250 s

1,875 s

2,500 s

Configuration A Configuration B

2259

1180

2128

1051

Executio

n T

ime (s)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 s

625 s

1,250 s

1,875 s

2,500 s

Configuration A Configuration B

2203

1125

2104

1051

Executio

n T

ime (s)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 Wh

50 Wh

100 Wh

150 Wh

200 Wh

Configuration A Configuration B

144

74

171

82Energ

y (W

h)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 Wh

50 Wh

100 Wh

150 Wh

200 Wh

Configuration A Configuration B

158

83

178

93

Energ

y (W

h)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 Wh

125 Wh

250 Wh

375 Wh

500 Wh

1200s sleep interval 2400s sleep interval

349

403

353

411

Energ

y (W

h)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

(b) Wordcount execution time.

0 500 1000 1500 2000Time (s)

0

20

40

60

80

100

120

140

160

180

Ener

gy (W

h)

0

20

40

60

80

100

120

140

160

180

Energy-efficient EucalyptusUnmodified EucalyptusEnergy Savings

(c) Wordcount energy consumption trend for testconfiguration B.

Fig. 3.

0 s

625 s

1,250 s

1,875 s

2,500 s

Configuration A Configuration B

2259

1180

2128

1051

Execu

tio

n T

ime (s)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 s

625 s

1,250 s

1,875 s

2,500 s

Configuration A Configuration B

2203

1125

2104

1051

Execu

tio

n T

ime (s)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 Wh

50 Wh

100 Wh

150 Wh

200 Wh

Configuration A Configuration B

144

74

171

82En

erg

y (W

h)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 Wh

50 Wh

100 Wh

150 Wh

200 Wh

Configuration A Configuration B

158

83

178

93

En

erg

y (W

h)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 Wh

125 Wh

250 Wh

375 Wh

500 Wh

1200s sleep interval 2400s sleep interval

349

403

353

411

En

erg

y (W

h)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

(a) PiEstimation energy consumption.

0 s

625 s

1,250 s

1,875 s

2,500 s

Configuration A Configuration B

2259

1180

2128

1051

Execu

tio

n T

ime (s)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 s

625 s

1,250 s

1,875 s

2,500 s

Configuration A Configuration B

2203

1125

2104

1051

Execu

tio

n T

ime (s)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 Wh

50 Wh

100 Wh

150 Wh

200 Wh

Configuration A Configuration B

144

74

171

82En

erg

y (W

h)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 Wh

50 Wh

100 Wh

150 Wh

200 Wh

Configuration A Configuration B

158

83

178

93

En

erg

y (W

h)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

0 Wh

125 Wh

250 Wh

375 Wh

500 Wh

1200s sleep interval 2400s sleep interval

349

403

353

411

En

erg

y (W

h)

Test Configurations

Unmodified Eucalyptus Energy-efficient Eucalyptus

(b) PiEstimation execution time.

0 500 1000 1500 2000Time (s)

0

20

40

60

80

100

120

140

160

180

200

Ener

gy (W

h)

0

20

40

60

80

100

120

140

160

180

200

Energy-efficient EucalyptusUnmodified EucalyptusEnergy Savings

(c) PiEstimation energy consumption trend for testconfiguration B.

Fig. 4.

saves 8 Wh, i.e., 10 percent of the total energy consumption.Furthermore, our implementation saves 27 Wh in configurationB, which is 16 percent of the total energy consumption.

Figure 4(b) shows the execution time of PiEstimation inboth test configurations. The execution time of the energy-efficient Eucalyptus is increased in both configuration A(about 12 percent) and configuration B (about 6 percent).The difference between the execution times of the unmodifiedEucalyptus and the energy-efficient Eucalyptus is about 130seconds, for both configurations.

Figure 4(c) shows the energy consumption over time forboth the unmodified Eucalyptus and the energy-efficient Eu-calyptus configuration B. Similar to the Wordcount results,the gradient of the energy consumption of the unmodifiedEucalyptus does not change, but after the suspension of Server3, the gradient of energy-efficient Eucalyptus gets lower. Thesuspension of Server 3 leads to a growing energy savingpotential of up to 32 Wh. After the turning point for the energysaving potential, the potential energy savings are reduced.Since the power consumption difference between G1-S0 (idle)and G1-S3 (suspend-to-RAM) is very high (about 52 W), theadditional energy spent for longer execution times is amortizedvery fast, even with more unfavorable configurations.

The average power consumption of the unmodified Eucalyp-tus was 286.5 W, while the average power consumption of theenergy-efficient Eucalyptus was 250.3 W. Furthermore, after

the suspension of Server 3, the average power consumptionof the energy-efficient Eucalyptus was reduced to 235.3 W.This means that the average power consumption of eachserver was 117.6 W, compared to 95.5 W of the unmodifiedEucalyptus. In other words, the remaining servers run moreenergy-efficiently; they are highly utilized and their powerconsumption comes close to their maximum. Due to DRBDdevice synchronization, there is a power peak of about 353 W,lasting for 136 seconds.

C. KernCompile Experiment

Linux kernel compiles are the basis for commonly usedCPU throughput benchmarks like kernbench [17], designedto compare different operating system kernels on the samemachine or different hardware. Kernel compiles are oftenused as benchmarks because their workload combines CPU-,memory- as well as I/O-intensive parts. For example, kernelcompiles are used to show the slowdown of a live migration[18]. In our test, a vanilla 2.6.33.1 kernel was compiledwith the default configuration and four jobs compiling con-currently. The results of this experiment are: At first, theenergy consumed for an instance relocation is very small. Inthis disadvantageous experimental setup, the additional energyconsumed is about 3 Wh, which is only two percent of thetotal energy consumption. Second, the experimental resultsconfirm the conclusions of the Startup/Terminate experiment.Even though there are power consumption peaks due to DRBD

249

device synchronizations and VM live migrations, instancerelocation itself is not energy-intensive.

D. Apache Benchmark Experiment

The Apache Benchmark [19] is a simple web server bench-mark typically used to show how many requests per secondthe Apache installation is capable of serving. For this purpose,a client repeatedly fetches static web pages from a web server.This experiment is aimed at emulating a web server applicationtypically used by a cloud customer. Due to the concurrentrequests handled by the Apache application, there is a lotof interprocess communication. In general, this test is moreoperating system-intensive rather than I/O-intensive.

The results of this experiment are similar to the KernCom-pile tests described in the previous section. The difference inthe average power consumption between this experiment andKernCompile is only about 3 W in all test configurations, apartfrom the reduced power consumption of 3 W and fluctuationsin the electricity meter measurements. In addition, the patternof power consumption in the different phases of the experimentis similar to the KernCompile power consumption.

VII. CONCLUSIONS

In this paper, we have presented an approach for improvingthe energy efficiency of infrastructure-as-a-service clouds. Theapproach is based on performing live migrations of virtualmachines to save energy. In contrast to related work, theenergy costs of live migrations including their pre- and post-processing phases are taken into account. Support for virtualmachine live migrations has been added to the EUCALYP-TUS open-source cloud computing system. Furthermore, toenable smart disk image transfers between servers, distributedreplication block devices were used. A division of the virtualmachines into multiple layers is achieved by using a virtualfile system. This reduces the amount of data that has tobe transferred. We have evaluated the proposed approachusing several short- and long-term tests based on virtualmachine workloads produced with common operating systembenchmarks, web-server emulations and different MapReduceapplications. The results have indicated that energy savings ofup to 16 percent can be achieved in a productive EUCALYPTUSenvironment. Even if it is not possible to save any energy(due to disadvantageous system conditions or user actions),the presented approach does not consume more energy thanthe original version of EUCALYPTUS.

There are several areas of future work. Currently, ourapproach considers instances as black boxes, i.e., we do nottake the nature of the applications into account (e.g., CPUintensive vs. memory or I/O intensive workload). In a futureversion, it would be desirable to gather additional informationand change the instance relocation algorithms accordingly andto find a trade-off between the additional costs spent and theenergy saved. Furthermore, our instance relocation algorithminitiates live migrations if it has the possibility to shutdownan idle server. Here, it would be interesting to find smarterheuristics that improve the level of energy-efficiency.

VIII. ACKNOWLEDGEMENTS

This work is partly supported by the German Ministry ofEducation and Research (BMBF) (D-Grid Initiative and HPC-Call: Software for Scalable Parallel Computers).

REFERENCES

[1] The Climate Group, “SMART 2020: Enabling the Low Carbon Economyin the Information Age,” http://www.smart2020.org/, 2008.

[2] Amazon Inc., “Amazon EC2,” http://aws.amazon.com/de/ec2/, 2010.[3] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neuge-

bauer, I. Pratt, and A. Warfield, “Xen and the Art of Virtualization,”in SOSP ’03: Proceedings of the 19th ACM Symposium on OperatingSystems Principles. ACM Press, 2003, pp. 164–177.

[4] M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. Katz, A. Konwinski,G. Lee, D. Patterson, A. Rabkin, I. Stoica, and M. Zaharia, “A Viewof Cloud Computing,” Communications of the ACM, vol. 53, no. 4, pp.50–58, 2010.

[5] S. Srikantaiah, A. Kansal, and F. Zhao, “Energy Aware Consolidation forCloud Computing,” in HotPower’08: Proceedings of the 2008 conferenceon Power aware computing and systems. Berkeley, CA, USA: USENIXAssociation, 2008, pp. 10–10.

[6] A. Verma, P. Ahuja, and A. Neogi, “pMapper : Power and MigrationCost Aware Application Placement in Virtualized Systems,” Ifip Inter-national Federation For Information Processing, pp. 243–264, 2008.

[7] R. Nathuji and K. Schwan, “VirtualPower: coordinated power manage-ment in virtualized enterprise systems,” in SOSP ’07: Proceedings oftwenty-first ACM SIGOPS symposium on Operating systems principles.New York, NY, USA: ACM, 2007, pp. 265–278.

[8] H. N. Van, F. D. Tran, and J.-M. Menaud, “Performance and PowerManagement for Cloud Infrastructures,” 2010 IEEE 3rd InternationalConference on Cloud Computing, pp. 329–336, July 2010.

[9] A. Beloglazov and R. Buyya, “Energy Efficient Resource Managementin Virtualized Cloud Data Centers,” in CCGRID ’10: Proceedings ofthe 2010 10th IEEE/ACM International Conference on Cluster, Cloudand Grid Computing. Washington, DC, USA: IEEE Computer Society,2010, pp. 826–831.

[10] D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman,L. Youseff, and D. Zagorodnov, “The Eucalyptus Open-Source Cloud-Computing System,” in CCGRID ’09: Proceedings of the 2009 9thIEEE/ACM International Symposium on Cluster Computing and theGrid. Washington, DC, USA: IEEE Computer Society, 2009, pp. 124–131.

[11] Hewlett-Packard Corp., The ACPI Promoters Corp., Microsoft Corp.,Phoenix Technologies Ltd., and Toshiba Corp., “Advanced Config-uration and Power Interface 4.0 Specification,” http://www.acpi.info/DOWNLOADS/ACPIspec40.pdf, 2009.

[12] DRBD.org, “DRBD,” http://www.drbd.org/, 2010.[13] R. Schwarzkopf, M. Schmidt, N. Fallenbeck, and B. Freisleben, “Multi-

layered Virtual Machines for Security Updates in Grid Environments,”in SEAA ’09: Proceedings of the 2009 35th Euromicro Conference onSoftware Engineering and Advanced Applications. Washington, DC,USA: IEEE Computer Society, 2009, pp. 563–570.

[14] K. Haselhorst, M. Schmidt, R. Schwarzkopf, N. Fallenbeck, andB. Freisleben, “Efficient Storage Synchronization for Live Migration inCloud Infrastructures,” in Proceedings of the 19th Euromicro Conferenceon Parallel, Distributed and Network-based Processing (PDP). IEEEpress, 2011, pp. 511–518.

[15] EasyMeter GmbH, “EasyMeter Smart Metering Device,” http://www.easymeter.com/, 2011.

[16] Apache Software Foundation, “Apache Hadoop,”http://hadoop.apache.org/, 2010.

[17] C. Kolivas, “Kernbench Benckmark,” http://ck.kolivas.org/kernbench,2004.

[18] C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach, I. Pratt,and A. Warfield, “Live Migration of Virtual Machines,” in NSDI’05:Proceedings of the 2nd conference on Symposium on Networked SystemsDesign & Implementation. Berkeley, CA, USA: USENIX Association,2005, pp. 273–286.

[19] Apache Software Foundation, “HTTP Server Project,” http://httpd.apache.org/, 2010.

250