harnessing renewable energy in cloud datacenters: opportunities and challenges

ransitioning from fossil fuels to the use of renewableenergy sources, such as solar and wind, to power cloudcomputing datacenters is a hot research topic today. Inthis article, we review and summarize a series of funda-

mental relevant research problems: why, when, where and howto leverage renewable (green or clean) energy in datacenters.

How Big is the Datacenter Power Consumption? The proliferation of cloud computing services has promotedmassive-scale, geographically distributed datacenters with mil-lions of servers. Large cloud service providers consume manymegawatts of power to operate such datacenters and the cor-responding annual electricity bills are in the order of tens ofmillions of dollars — such as Google with over 1,120 GWhand $67 M, and Microsoft with over 600 GWh and $36 M [1].Reportedly, datacenters now consume about 1.3 percent ofthe worldwide electricity and this fraction will grow to 8 per-cent by 2020 [2]. As a matter of fact, the aggregate datacen-ters in the world consumes more electricity than most nationsin the world except 4 countries [3].

How Clean are the Current Datacenters?High energy consumption not only results in large electricitycost, but also incurs high carbon emission. In the UnitedStates, generating 1 kWh of electricity emits about 500g ofCO2 on average [2]. Each 100MW power station will cost$60-100 million dollars to build and emit 50 million tons ofCO2 during its operation [3]. As a result, IT carbon foot-prints currently occupy 2 percent of global greenhouse gasemissions [4].

To measure how clean is a datacenter, the Green Gridorganization proposes a new sustainability metric, carbonusage effectiveness (CUE), to measure carbon emissions asso-ciated with datacenters. CUE is defined as: CUE = Total CO2Emissions caused by Total Datacenter Energy/IT Energy Con-sumption. The units of the CUE metric are kilograms of car-bon dioxide equivalent (kgCO2e) per kilowatt-hour (kWh).The comparison of carbon emission rate of the most commonenergy sources is shown in Table 1. We observe that therenewable energy sources have much less carbon emissionrate than fossil fuels such as coal, gas and oil. Even though,fossil fuels still counts for 2/3 of electricity of the world [2].

Large IT companies have started to build datacenters withrenewable energy, such as Facebook’s solar-powered data-center in Oregon and Green House Data’s wind-powereddatacenter in Wyoming. In April 2012, Greenpeace releaseda report asking, “How clean is your cloud? [4]” It examinesthe datacenters built by the large Internet companies, andranks them according to how efficient their cloud facilitiesare, and where they get their electricity. It found that bothGoogle (39.4 percent clean energy) and Yahoo (56.4 percentclean energy) are active in supporting policies to driverenewable energy investment and to power clouds with greenenergy. In contrast, many large IT companies, such as Ama-zon, Apple and Microsoft, rapidly expand their cloud busi-ness without adequate attention on the electricity source,and rely heavily on brown energy to power their clouds.Even worse, there are numerous small and medium-sizeddatacenters that consume the majority of energy, yet muchless energy efficient.

T

48 IEEE Network • January/February 2014

AbstractThe proliferation of cloud computing has promoted the wide deployment of large-scale datacenters with tremendous power consumption and high carbon emission.To reduce power cost and carbon footprint, an increasing number of cloud serviceproviders have considered green datacenters with renewable energy sources, suchas solar or wind. However, unlike the stable supply of grid energy, it is challeng-ing to utilize and realize renewable energy due to the uncertain, intermittent andvariable nature. In this article, we provide a taxonomy of the state-of-the-artresearch in applying renewable energy in cloud computing datacenters from fivekey aspects, including generation models and prediction methods of renewableenergy, capacity planning of green datacenters, intra-datacenter workload schedul-ing and load balancing across geographically distributed datacenters. By explor-ing new research challenges involved in managing the use of renewable energy indatacenters, this article attempts to address why, when, where and how to lever-age renewable energy in datacenters, also with a focus on future researchavenues.

Harnessing Renewable Energy in CloudDatacenters: Opportunities and ChallengesWei Deng, Fangming Liu, and Hai Jin, Services Computing Technology and System Lab, Clusterand Grid Computing Lab in the School of Computer Science and Technology, Huazhong Univer-

sity of Science and TechnologyBo Li, The Hong Kong University of Science and Technology

Dan Li, Tsinghua University

T

0890-8044/14/$25.00 © 2014 IEEE

LIU_LAYOUT.qxp_Layout 1 1/15/14 11:42 AM Page 48

Why Leveraging Renewable Energy in CloudDatacenters?There have been numerous academic and commercialendeavors for reducing power costs and carbon footprintsby applying better energy-proportional computing tech-nologies, and more efficient power delivery and coolingsystems. However, energy efficiency alone will slow downthe growth of IT carbon footprint. To maintain safe levelsof global greenhouse gases, renewable energy sources isbecoming a prioritized choice for IT companies to powertheir rapidly expanding datacenter infrastructures. The inter-national environmental organization Greenpeace states that“Green IT = Energy Efficiency + Renewable Energy” [4].

Thus, many governments enact renewable portfolio stan-dards and provide incentives for green power generation andusage. For instance, California aims to generate 33 percent ofits electricity from renewable sources by 2020 [5]. Federal andstate incentives in New Jersey reduce the capital costs ofrenewable energy deployment by 60 percent [6]. Additionally,the improvements in power generating efficiency andcost/Watt reductions of renewable energy will reduce thedeployment cost significantly in the future. For example, theefficiency of solar panels is expected to triple but the cost/Wattof solar panels is expected to halve in 2030 [6].

Apart from the pressures for reducing the huge energy con-sumption and carbon emission, there are unique opportunitiesraised by cloud datacenters to apply green energy:• Cloud service providers typically own geographically dis-

tributed datacenters. They can distribute workloads amonggeo-dispersed datacenters to benefit from the locationdiversity of different types of available renewable energies.

• Cloud datacenters usually support a wide range of IT work-loads, including both delay-sensitive non-flexible applica-tions such as web browsing and delay-tolerant flexibleapplications such as scientific computational jobs. Theworkload flexibility can tackle the challenges in integratingintermittent renewable energy by delaying flexible work-loads to periods when renewable sources are abundantwithout exceeding their execution deadlines.

• Cloud datacenters connect to grids at different locationswith time-varying electricity prices. Renewable energy is ameans to mitigate the risk against the future rise in powerprices.

• Datacenters usually equip with uninterrupted power supply(UPS) in case of power outages. Since UPS battery is usual-ly over-provisioned, UPS can store energy during periods ofhigh renewable generation and supply power when therenewable energy is insufficient.

Pathway to a Cleaner CloudThe most common way to utilize renewable energy is todeploy on-site generation equipments at the datacenter facili-ty. Such on-site generation has negligible transmission anddistribution loss. However, the location of a datacenter doesnot necessarily have a profitable on-site renewable energydeployment. Another way is to deploy off-site renewable ener-gy generation plant at the locations with good wind speed orsolar irradiation. The generated green energy can be dis-patched across the grid to the consuming datacenters [3].

Apart from the above explicit options of provisioningrenewable energy generation plants, there also exist threekinds of implicit options to utilize renewable energy.

• Power purchase agreement (PPA), which purchases a por-tion of green energy from a renewable energy source.

• Renewable energy credits (RECs), which are tradable, non-tangible energy commodities. They represent that 1 MWhof electricity be generated from an eligible renewable ener-gy resource.

• Carbon offsetting, which represents the reduction of oneton of carbon dioxide. Table 2 shows the costs and carbonemission rate of REC, PPA conventional grid, and dieselgenerator (DG) used as backup power source in datacen-ters.

Research ChallengesThe main challenge of utilizing renewable energy is thatrenewable sources are variable, intermittent and unpre-dictable. For instance, wind or solar energy is only availablewhen wind blows or sun shines. Characteristics of cloud data-centers further raise challenges to apply renewable energy:• Global users require 7 × 24 cloud services: However, inter-

mittent renewable energy presents a problem for consistentusers of power like datacenters.

• Cloud demand is dynamic, which requires dynamic powerprovisioning. Thus, the power supply should be elastic. Nev-ertheless, unlike conventional energy from the grid, renew-able energy can not be scheduled on demand.

• High reliability services. This incurs the problem of how toconstruct reliable power supply in the presence of uncertainrenewable energy.

• Automatic management. This requires the power supply sys-tem should choose and supply power automatically amongmultiple power sources.Before deploying renewable energy equipment, we need

to know what kinds of datacenter workloads are amenablefor green energy and whether the datacenter is suitable forgreen energy. Afterwards, we need to know the capacity ofrenewable energy generation for the specific datacenterdemand. To mitigate the variability of renewable energy,datacenters can store the generated renewable energy inbatteries or on the grid. However, it remains significantchallenges when/how to store and supply energy from thebattery to meet demand. In addition, these approachesincur energy loss and high additional costs of purchase andmaintenance. Instead, we can maximize the use of availableenergy by matching the demand to the supply. This promptsmany interesting research problems. Due to the variableavailability of renewable energy, the scheduler has to exploitsuitable forecast techniques to predict the available renew-able energy in an online fashion. How can we scheduleworkload in intra-datacenter to maximize renewable energyusage? Is it profitable to leverage geographical workload forenergy minimization? We will answer these questions innext sections.

IEEE Network • January/February 2014 49

Table 1. Carbon emission rate of major energy sources [2, 3].

Energy Source Nuclear Coal Gas Oil Hydro Wind Solar

Carbon Emission Rate (gCO2e/kWh) 15 968 440 890 13.5 22.5 53

Table 2. Comparison of costs and prices of energy sources [3].

Energy Source Grid PPA REC DG

Cost (¢/kWh) 5 6 0.5 30

Carbon Emission Rate (gCO2e/kWh) 586 0 0 1056


Existing Approaches

Recently, various advanced solutions have been proposed toaddress the above challenges, which can be broadly classifiedinto five topics: generation model and prediction model ofrenewable energy, capacity planning of green datacenter,intra-datacenter workload scheduling and inter-datacentersload balancing. Table 3 summarizes the taxonomy of existingapproaches.

Renewable Energy Generation ModelWind and solar are the most prominent renewable sources,which currently provide 62 percent and 13 percent non-hydrorenewable electricity worldwide, respectively [3]. The solar orwind power output depends on the environmental conditions,such as solar irradiance or wind speed. Consequently, theircapacity factor, which is the ratio of the actual output over aperiod to its potential output if it operated at full nameplatecapacity, can be much lower than grid energy (e.g., 80 per-cent). Specifically, the capacity factor of wind energy is within20~45 percent, while the capacity factor of solar energyranges from 14–24 percent [3].

The wind turbines generation can be modeled as a functionof the wind speed. Let v be the wind speed, then the windpower output pwind can be approximated as follows [8]:

where vin and vout are the cut-in (typically 3~4 m/s) andcut-out (typically 25 m/s) wind speed, and vr and pr are thenameplate speed and wind turbines power. If a wind farmconsists of mw turbine, then the wind power output is thesum of all the turbine power output: PW Smw

k=1 pkwind, where

pkwind is the power output of kth wind turbine at the wind

speed of v , with the assumption that the wind turbineshave the same wind speed in the same wind farm. Howev-er, the speed and angle may differ from turbine to turbinein a farm. A more practical model should be , where vk isthe wind speed that is a vector containing the wind direc-tion.

The solar photovoltaic (PV) generation can be modeledas its current-voltage characteristic equations, which arerelated to solar cell parameters and environmental condi-tions, such as irradiance condition G and temperature T[8]:

=

< >

⋅ −−

< <

< <

⎧

⎨

⎪⎪

⎩

⎪⎪

p

v v v v

pv v

v vv v v

p v v v

0 ,

.wind

in out

rin

r inin r

r r out

IEEE Network • January/February 201450

Table 3. A taxonomy of approaches of leveraging renewable energy in datacenters.

Approaches EnergySource Time Workload Level Input Evaluation

Method Goa Technique

CapacityPlanning

Carbon-AwarePlanning [3]

On/off-site,ESD,Grid Off Pervasive Data

centerHistoricalPrediction

Model,trace-drivensimulation

Optimal energyportfolio tominimize cost

Modeling differentrenewables, carbontarget, policy, tax, ESD

ReRack [12] Wind,solar,ESD,grid Off Interactive Data

centerHistoricalPrediction

Model,ReRack sim-ulator

Evaluate differentenergy portfoliocost

Modeling differentrenewables, ESD,weather, incentive

WorkloadScheduling

GreenSlotGreenHadoop[7, 11]

Solar, grid On ScientificComputing

Datacenter Prediction

Prototype,a micro-datacenter

Maximize solarenergy usage whileguaranteeing delay

Greedy scheduling

Blink [15] Solar, wind On Web request Datacenter Real-time Blink

prototypeFully renewables-driven datacenter Server on/off, ACPI S3

iSwitch [16] Wind, grid On Web request Datacenter Real-time iSwitch

prototype

Maximizerenewables usage,performance

VM migration

SolarCore [9] Solar, grid On HPC Server Real-time Model,trace-driven

Solar-energy-drivenMulti-coreprocessor

DVFS

Energy AgileCluster [14] Wind, grid On Interactive,

batchDatacenter Prediction Prototype Energy-proportional

clusterDefer batch tasks,degenerate interactive

Prediction [10] Solar,wind, grid On Web request

batchDatacenter Prediction Model,

simulationAdapt server loadwith renewables

Two job queues, Deferbatch tasks

LoadBalancing

Greening [18,19] Solar, wind On Interactive Inter-DC Prediction,

real-timeModel,trace-driven

Using entirelyrenewable energy

Follow the renewablesrouting

Not Easy BeingGreen [2]

Fossil fuel,wind,hydro

On Web request Inter-DC Real-time FORTE,trace-driven

Tradeoff delay,energy cost andcarbon footprint

Renewable energyaware requests rout-ing

GreenWare [8] Solar,wind, grid On Web request Inter-DC Real-time,

predictionSimulator,trace-driven

Maximize usage ofavailable renew-ables under budget

Budget, fractionallinear programming

Free Lunch[17] Solar, wind On All VM appli-

cation Inter-DC Real-time Free Lunchframework

Fully use availablerenewable energy VM migration

Some Joules[13]

Solar,wind, grid On Web request Inter-DC Real-time,

predictionModel,simulation

Fully use availablerenewable energy

Renewable energyaware requestsrouting


where short-circuit current

dark saturation current

is the junction thermal voltage and k is Boltzmann’s constant.T is the ambient temperature and q is the charge of the elec-tron. G0 and T0 are the irradiance level and temperature inStandard Test Conditions (STC), i.e., G0 = 1000W/m2, T0 =25°C. Isc, Voc, ki and kv are the rated parameters of short-cir-cuit current, open-circuit voltage, temperature coefficients ofthe open-circuit and the short-circuit in STC, respectively.The solar power output can be estimated as the maximalpower point (mpp) that can be extracted from the PV panel[9], i.e. mmp = i2mp ◊ rmp, where rmp is the optimal load, imp isthe corresponding current,

and ns is the number of the solar cells in the PV panel con-nected in series, e.g., ns = 72. Thus, if there are ms PV panelsin the solar plant, the overall solar power output is estimatedas: PS = Sms

k=1mppk, where mppk is the maximal power valuefrom the kth PV panel with irradiance condition G and tem-perature T.

Renewable Energy Prediction ModelSince renewable energy is highly variable, it is important topredict the available energy when scheduling workloads tomatch renewable energy supply. Solar energy prediction typi-cally applies estimated weighted moving average (EWMA)models, for its relative consistency and periodic patterns.However, with frequent weather changes, the prediction meanerror is over 20 percent [10]. Another prediction model is

weather-conditioned moving average (WCMA),which takes into account the mean value across daysand a solar condition factor in the present day rela-tive to previous days [10]. Differently, load-schedul-ing system GreenSlot [7] and GreenHadoop [11]predict the energy generation as a function inverselyrelated to the amount of cloud coverage. That is,Ep(t) = B(t)(1 – CloudCover), where Ep(t) is theestimated solar energy for time t, B(t) is the idealsolar energy generation, and CloudCover is the esti-mated percentage of cloud coverage compared tosunny days (within 0~1).

Wind energy forecast typically has two majorapproaches: time-series analysis of power data andwind speed prediction. Since the response time con-straints of cloud services can be quite short, the pre-dictor needs to be fast. Aksanli et al. [10] developeda low-overhead wind energy predictor that utilizes

available data to generate a power wind table based on windvelocity and wind direction as follows:

Pnew(v, d) = a × Pobs(v, d, t) + (1 – a) × Pold(v, d)

where Pnew(v, d) is the new power value with a wind velocity vand direction d, Pold (v, d) is the existing value for the samevelocity and direction, and Pobs(v, d, t) is the measured valueat time t. a is within 0~1. When predicting future intervalwind energy, it does a table lookup with the predicted windvelocity and direction: Ppred (v, d, t + k) = P(v(t + k), d(t +k)). For a 30-minute prediction interval, the predictor shows amean error of 17.2 percent, no worse than the time-seriesmodels [10].

Capacity Planning of Green DatacenterRenewable energy can reduce datacenter operational cost andcarbon footprint when correctly selected. Nevertheless, asillustrated in Fig. 1, it is not a trivial task, for the reason thatdatacenter planners should consider different renewableoptions (on-site, off-site, PPA, RCE), different energy sources(solar, wind), different energy storage devices (batteries, fuel-cells, flywheels), markets prices fluctuations, workload vari-ances, weather conditions, incentives and service agreementpenalties. Given all these factors, Jose Renau et.al. [12] pro-posed ReRack, a power simulator that can evaluate the ener-gy cost of a datacenter using renewable energy. For any givenlocation and workload of a datacenter, ReRack can find thebest ratio of renewable energy sources using a genetic algo-rithm.

Differently, Chuangang Ren et al. [3] developed a practicallinear programming based optimization framework for a data-center to achieve a target carbon footprint at minimal cost,among the various options presented in Fig. 1. Under exten-sive experiments with real-world power demand and renew-able energy traces, the key findings are:• Renewable energy can lower both carbon emissions and

costs for datacenters.• On-site renewables can lower datacenter costs by reducing

the peak power draw from grid.• The most cost-effective approaches for carbon reduction

vary with different carbon footprint targets.For a moderate target (upto 30 percent), the best options isusing on-site renewables, while a more carbon reduction goalrequires off-site renewables, and a zero carbon goal mustresort to renewable energy products such as RECs. Unfortu-nately, the model considered a low and constant grid price($0.05/kWh), which made on-site generation relatively expen-sive in comparison.

= ⋅ + ⋅ −I G TG

GI

kT T( , ) (1

100( )),sc

osc

i0

= − ⋅+ ⋅

⋅i G T I G T I T e( , ) ( , ) ( )sc o

v G T i G T R

n V

( ) ( , ) s

s th

= + ⋅ − ⋅ ⋅ =⋅⋅I T I

kT T e V kT q( ) (1

100( )) /o sc

i

V k T T

n Vth0

( – )OC v

s th

0

= + ⋅+ −

r Rn V

I G T I T i( , ) ( ),mp s

s th

sc mp0


Figure 1. Overall vision.

Datacenter

On-site

Off-site

Grid

ESD

Dieselgenerators

On-siterenewable

Fossil fuelOff-site

renewable

Renewableproducts (PPA)

Carbon offsetmarket (REC)


Intra-Datacenter Workload Scheduling

In a modern datacenter with on-site renewable energy, thereare three approaches to match power demand to variablerenewable energy, with respect to job scheduling, serverpower state management and load migration.

Job Scheduling — To study the use of renewable energy indatacenters, a renewable energy powered platform “Para-sol” was constructed [6]. Two load-scheduling systemsGreenSlot [7] and GreenHadoop [11] were designed. Bothassume that the datacenter is powered by solar energy andgrid with no batteries for energy storage. GreenSlot [7] isfor batch jobs while GreenHadoop [11] is for Hadoop jobs.The goal is to maximize the use of solar energy while meet-ing the jobs’ deadline. They predict the available solar ener-gy in the future using historical data and weather forecasts,and then defer workload execution until solar energy isavailable without violating the deadline. If the grid energymust be used to avoid deadline violations, the systemsschedule jobs for times when grid energy is cheap. Todemonstrate these in practice, various experiments are con-ducted in the micro-datacenter Parasol [6] with 16-nodepowered by a solar array and the electrical grid (as a back-up). GreenSlot is robust to different workload characteris-tics, providing green energy increases of at least 19 percentand energy cost savings of at least 25 percent compared tobackfilling scheduling algorithm. The solar energy predictoris reasonably accurate, achieving hour-ahead median and90th percentile errors of 12.9 percent and 24.6 percent,respectively. Interestingly, they found that even perfectknowledge of solar energy availability increases solar energyuse and decreases cost both by only 1 percent.

GreenSlot and GreenHadoop are not suitable for inter-active and delay-sensitive workloads. Aksanli et al. [10]designed an adaptive scheduler for mixed batch and Webservice jobs. The scheduler uses two separate job arrivalqueues: one for the Web services and the other for thebatch jobs. Each server poses a limit on the amount ofworkloads (Web requests and batch jobs) to be served.Web services start execution whenever there are availablecomputing resources, while batch jobs start execution basedon the availability of the renewable energy. Similarly, anenergy agile computing cluster was proposed [14], in whichit also slightly defers long running batch jobs so as to makethe best use of wind energy. In addition, it enables gracefuldegradation of interactive services to adapt power con-sumption.

Server Power State Magement —SolarCore [9] is a solar energy driv-en, multi-core architecture powermanagement scheme that moves thesolar array operating point close tothe maximum power point underchanging atmospheric conditions.SolarCore performs load adaptationutilizing dynamic voltage/frequencyscaling (DVFS) and per-core powergating (PCPG). It intelligently allo-cates the dynamically varying powerbudget across multiple cores to max-imize workload performance. Whenthe solar power supply drops belowa certain threshold, it switches togird. Evaluations with solar tracesand SPEC2000 benchmark show thatSolarCore boosts multi-core proces-

sor performance by 43 percent compared with fixed powerbudget scheme and 10.8 percent compared with round-robinload adaptation. Researchers from UMass at Amherst havebuilt Blink [15], a cluster of 10 laptop motherboards poweredby two micro wind turbines and two solar panels. They usedbatteries only as a small 5-minute energy buffer. Blink modu-lates the server power state using fast sleep to adapt to theavailability of renewable energy. Yet, the proposed cluster isnot connected to the grid, which seems to be unrealistic. Dat-acenters that completely depend on variable green energy cancause unbounded performance degradation.

Load Migration — With the presence of the intermittentrenewable power, inefficient and redundant workload match-ing activities can incur up to 4X performance loss [16].iSwitch [16] explores the design tradeoffs between load tuningoverhead and green energy utilization in renewable energydriven datacenters. Servers are divided into two groups: one ispowered by grid while the other is powered by on-site windenergy. Based on the availability of wind energy, iSwitchmigrates load between these two groups to best utilize therenewable energy generation. iSwitch can mitigate averagenetwork traffic by 75 percent, peak network traffic by 95 per-cent, and reduce 80 percent job waiting time while still main-taining 96 percent renewable energy utilization.

Inter-Datacenters Load Balancing Large cloud service providers usually own geo-distributed dat-acenters for• Better performance, i.e., close to end users• More reliable system, i.e., multiple backups in different dat-

acenters• Lower cost, i.e., leveraging different energy sources, prices,

cooling efficency and taxThe first question is whether geographical load balancing can

exploit renewable energy and reduce environmental impact?FreeLunch was proposed to exploit renewable energy [17]. Itassumes that datacenters are close to renewable energysources, and support the seamless execution and migration ofvirtual machines according to the power availability. Highlatency and storage synchronisation can degrade the efficiencyof VM migration. Thus, FreeLunch might be inefficient forlatency sensitive or interactive applications.

For web requests, Liu et al. exploited whether geographicalload balancing can encourage the use of renewable energyand reduce the use of fossil fuel (brown) energy [18]. It wasfound that if grid energy (with penetration of renewable ener-gy sources) is dynamically priced based on the proportion of


Figure 2. The platform provider GUI.

PriceWind

Green ware

Requestdispatching

Request rate(requests/s)

Monthly budget(dollars)

DatacenterSolarBrown

PriceWindSolarBrown

PriceWindSolarBrown

DatacenterDatacenter

LIU_LAYOUT.qxp_Layout 1 1/15/14 11:43 AM 2

brown energy, then geographical load balancingcan reduce brown energy usage significantly. Espe-cially, the optimal pricing model at ith datacenteris: pi(t) = (1 – ai(t))/b, where ai(t) the fraction ofthe energy that is from renewable sources at time tand b is the relative valuation of delay versus ener-gy. Further, based on trace-driven study, theyinvestigated the feasibility of powering datacentersentirely using renewable energy [19]. It observedthat “follow the renewables” routing could signifi-cantly reduce the required capacity of renewableenergy, especially in corporation with suitable sizeof energy storage. Storage becomes more valuablewith higher capacities of renewables, i.e., >1.5,where renewable capacity is defined as the ratio ofthe average renewable generation to the minimalenergy required to serve the average workload. Moreover,they found that wind energy is more viable than solar. Therational is that wind is available during both day and night,and has little correlation across different locations.

The second question is how many and which user requestsshould be migrated? Stewart et al. [13] proposed an accountingmodel of per-request energy consumption as a guideline formanaging renewable energy. Based on the performance coun-ters, the model collects three metrics including the activitiesof L2 cache, memory and CPU: L2 cache requests per CPUcycle (Ccache), memory transactions per CPU cycle (Cmem),and the ratio of non-halt CPU cycles (Cnonhalt). The requestpower model is given as:

where p* are the coefficient parameters for the linear model,and C*

ceil are the upper bound for the three metrics. With themodel, the power/energy profiles for individual requests canbe constructed, and renewable-aware request distribution canbe conducted.

If incentives for renewable energy usage are not consid-ered, currently renewable energy is often more expensive thanbrown energy. To utilize renewable energy, cloud serviceproviders have two challenges:• How to conduct dynamic request dispatching among geo-

graphical datacenters to maximize the use of renewableenergy

• How to achieve that within allowed operational cost budget.As illustrated in Fig. 2, GreenWare [8] dynamically distributesrequests among datacenters to maximize the percentage ofused renewable energy subject to the desired cost budget.This is formulated as a constrained optimization problem:

where Wi, Si, Bi are the amount of used wind, solar and brownenergy in the ith (1 ≤ i ≤ N) datacenter, and PWi, PSi and PBiare the current prices of the three types of energy. Constraint(6) means that the energy cost should be within the cost bud-get. They proposed an efficient request-dispatching algorithmbased on linear-fractional programming (LFP) and imple-mented GreenWare based on the linprog solver in Matlab.Evaluations with real-world weather, electricity price and

workload traces show that GreenWare can significantlyincrease the use of renewable energy without violating thedesired cost budget.

However, GreenWare ignores a key factor: what set ofdata users are interested in and where the data is? FORTE[2] captures the relationship between user groups, datacen-ters and data. It uses two assignment algorithms to optimallymap users to datacenters and map data to datacenters. Theobjective is to minimize the weighted sum of access latency,electricity cost and carbon footprint. Trace-driven experi-ments show that FORTE can reduce carbon emissions by 10percent without increasing the electricity bill and the meanlatency.

Moving Forward: Future Research TrendsDiverse Green SourcesMost existing works focus on studying the usage of soalr orwind energy. However, many other renewable energy sourcesare ignored, such as tidal energy, hydrogen energy and fuelcells. Especially, fuel cells can generate energy with steadysupply, which is very different from the intermittent solar orwind energy, and thus require different research methods.Besides, off-site renewables such as PPA and REC shown inFig. 1, also deserve deeper theoretical study and practicalapplication, so as to find the best energy portfolio for a specif-ic datacenter.

New Power Supply ArchitecturesIf constructing on-site renewable energy sources, how to inte-grate the renewable energy into the traditional grid circuit?There are two common ways:• Transfer switches. Modern datacenters can automatically

switch power sources via Automatic Transfer Switch (ATS)based on a pre-configured power-transfer threshold. If thepower supply from the primary source falls below a thresh-old, the ATS switches to a secondary energy source.

• Grid ties: Grid ties combine electricity produced from therenewable sources and the grid on the same circuit forpower supply. However, transfer switches isolate electricityproduced on site, keeping it separated from the electricitygrid. In contrast, grid ties allow multiple sources to power aserver. If a new datacenter power supply system desires toisolate variable renewable energy from the stable electricitygrid, the transfer switch is a better choice. The power-trans-fer threshold computation and grid-tie placement are there-fore also important research problems. Further, unlikecurrent centralized cloud model, we can consider peer-to-peer-based architecture to leverage the distributed renew-able energies.

+ ⋅ + ⋅ + ⋅p pC

Cp

C

Cp

C

Cidle cache

cache

cacheceil mem

mem

memceil nonhalt

nonhalt

nonhaltceil

∑∑

∑

+

+ +

⋅ + ⋅ + ⋅ ≤

=

=

=

MaximizeW S

W S B

s t PW W PS S PB B Budget

:( )

( )

. . ( )

i ii

N

i i ii

N

i i i i i ii

N

1

1

1


Figure 3. Business model.

Completeddemand

Served beforedeadlinePostponed demands

Renewable energy

Delay-tolerant

Delay-sensitive Served immediately

Time-varying prices

Energy storage

Grid

+

–

Serve


Demand Side Workload Management

Current works either solely consider delay-sensitive demandor only suit for delay-tolerant workloads, while others neglectthe interaction among uncertain renewable energy generation,energy storage management and time-varying electricityprices. It remains a significant challenge how to operateamong multiple power supply sources in a complementarymanner [20], to deliver reliable energy to datacenter userswith arbitrary demand over time. As shown in Fig. 3, one pos-sible method is to address delay-sensitive demand when it isgenerated, and cater to the delay-tolerant demand in a morestrategical fashion over time, while guaranteeing that it isserved before the deadline. Energy storage can store renew-able energy and cheap grid electricity, while supply powerwhen needed, so as to best utilize the available renewableenergy and the periods with lower electricity prices from thegrid. Another way to address the demand and supply uncer-tainties is demand response mechanism, which is a strategy forachieving energy efficiency through managing power con-sumption in response to supply conditions. For example,dynamic electricity pricing and peak-demand charging caninduce end users to curb their demand at critical times in aneffort to reduce the costs. However, it is still challenging todesign adequate pricing mechanims to reflect the availabilityof renewable energies.

Supply Side Power ManagementThe availability of renewable energy is time-verying. Energystorage devices can alleviate this variability. However, energystorage devices incur energy losses due to self-discharging andinternal resistance. The related costs of energy storage devicesmay dominate in systems powered by renewable energy. It isthus a non-trivial problem. With a plethora of energy storagetechnologies available today, we need to model the tradeoffsin cost, density, lifetime and efficiency when provisioningenergy storage. In addition, most current datacenters use cen-tralized UPS batteries. Newer datacenters are starting to usedecentralized batteries, e.g., rack-level in Facebook or server-level in Google. The centralized vs. distributed and hierarchi-cal placement problems of energy storage devices deservethorough investigation. Moreover, today most datacentershave an inefficient power conversion. Power is delivered usingalternating current (AC) that goes through multiple conver-sions between AC and direct current (DC) , resulting in ener-gy loss up to 30 percent. Future DC-based power distributionsystems can improve renewable energy utilization and reducethe total system energy use.

ConclusionThe main challenge of utilizing renewable energies is the vari-able, intermittent and unpredictable nature. By presenting ataxonomy of the latest research and exploring new challengesinvolved in managing the use of uncertain renewable energy,we intend to answer why, when, where and how to leveragerenewable energy in datacenters. Specifically, we believe thatmatching uncertain power demand and multi-sources supplyin a complementary manner should be one of the highlights infuture research.

AcknowledgmentThe corresponding author is Fangming Liu([email protected]). The research was supported in part by agrant from National Basic Research Program (973 program)under Grant of 2014CB347800, by a grant from The NationalNatural Science Foundation of China (NSFC) under grant

No.61133006 and No.61103176, by a grant from the ResearchFund of Young Scholars for the Doctoral Program of HigherEducation, Ministry of Education, China, under grantNo.20110142120079, by the CHUTIAN Scholar Project ofHubei Province. Prof. Bo Li’s work was supported in part by agrant from RGC under the contract 615613, a grant fromNSFC/RGC under the contract N_HKUST610/11, and a grantfrom ChinaCache Int. Corp. under the contractCCNT12EG01.

References[1] A. Qurush, “Power-Demand Routing in Massive Geo-Distributed Systems,”

Ph.D. dissertation, MIT, 2010.[2] P. Xiang Gao et al., “It’s Not Easy Being Green,” Proc. ACM SIGCOMM,

Helsinki, Finland, 2012, pp. 211–22.[3] C. Ren et al., “Carbon-Aware Energy Capacity Planning for Datacenters,”

Proc. MASCOTS, Washington DC, 2012, pp. 391–400.[4] G. Cook, How Clean is Your Cloud?, Greenpeace International Tech.

Rep., Apr. 2012.[5] R. Wiser and G. Barbose, “Renewables Portfolio Standards in the United

States — A Status Report with Data Through 2007,” LBNL-154E, 2008.[6] R. Bianchini, “Leveraging Renewable Energy in Data Centers: Present and

Future,” Proc. HPDC’12, San Jose, California, 2012, pp. 135–36.[7] I. Goiri et al., “Greenslot: Scheduling Energy Consumption in Green Data-

centers,” Proc. SC’11, Seattle, WA, 2011, pp. 1–11.[8] Y. Zhang, Y. Wang, and X. Wang, “Greenware: Greening Cloud-scale

Data Centers to Maximize the Use of Renewable Energy,” Proc. Middle-ware, Lisboa, Portugal, 2011, pp. 143–164.

[9] C. Li et al., “SolarCore: Solar Energy Driven Multi-core Architecture PowerManagement,” Proc. HPCA’11, San Antonio, TX, USA, 2011, pp.205–16.

[10] B. Aksanli et al., “Utilizing Green Energy Prediction to Schedule MixedBatch and Service Jobs in Data Centers,” Proc. HotPower’11, Cascais,Portugal, 2011, pp. 53–57.

[11] I. Goiri et al., “Greenhadoop: Leveraging Green Energy in Data-process-ing Frameworks,” Proc. EuroSys, Bern, Switzerland, 2012, pp. 57–70.

[12] M. Brown and J. Renau, Rerack, “Power Simulation for Data Centers withRenewable Energy Generation,” Proc. ACM GreenMetrics, San Jose, Cali-fornia, 2011, pp. 77–81.

[13] C. Stewart and K. Shen, “Some Joules Are More Precious Than Others:Managing Renewable Energy in the Datacenter,” Proc. HotPower, Mon-tana, USA, 2009, pp. 15–19.

[14] A. Krioukov et al., “Design and Evaluation of an Energy Agile Comput-ing Cluster,” University of California, Berkeley, Technical Report,UCB/EECS-2012-13, 2012.

[15] N. Sharma, S. Barker, D. Irwin, and P. Shenoy, “Blink: Managing ServerClusters on Intermittent Power,” ACM SIGARCH Computer ArchitectureNews, vol. 39, no. 1, 2011, pp. 185–98.

[16] C. Li, A. Qouneh and T. Li, “iSwitch: Coordinating and OptimizingRenewable Energy Powered Server Clusters,” Proc. ISCA, Portland, OR,USA, 2012, pp. 512–23.

[17] S. Akoush et al., “Free Lunch: Exploiting Renewable Energy for Comput-ing,” Proc. HotOS, Napa Valley, California, 2011, pp. 17–17.

[18] Z. Liu et al., “Greening Geographical Load Balancing,” Proc. SIGMET-RICS, San Jose, California, 2011, pp. 233–44.

[19] Z. Liu et al., “Geographical Load Balancing with Renewables,” Proc.ACM GreenMetrics, SAN JOSE, CALIFORNIA , 2011, pp. 62–66.

[20] W. Deng et al., “SmartDPSS: Cost-Minimizing Multi-source Power Supplyfor Datacenters with Arbitrary Demand,” Proc. IEEE ICDCS, Philadelphia,USA, 2013.

BiographiesWEI DENG ([email protected]) is currently a Ph.D. student in the School ofComputer Science and Technology, Huazhong University of Science and Tech-nology, Wuhan, China. He was the recipient of Best Paper Nominee fromIEEE CloudCom’2012. His research interests focus on cloud computing, data-center networking, green computing, smart grids, modeling and optimization.

FANGMING LIU [M] ([email protected]) is an associate professor in the Schoolof Computer Science and Technology, Huazhong University of Science andTechnology, Wuhan, China; and he is awarded as the CHUTIAN Scholar ofHubei Province, China. He is the Youth Scientist of National 973 BasicResearch Program Project on Software-defined Networking (SDN)-based CloudDatacenter Networks, which is one of the largest SDN projects in China.Since 2012, he has also been invited as a StarTrack Visiting Young Faculty inMicrosoft Research Asia (MSRA), Beijing. He received his B.Engr. degree in2005 from Department of Computer Science and Technology, Tsinghua Uni-versity, Beijing; and his Ph.D. degree in Computer Science and Engineeringfrom the Hong Kong University of Science and Technology in 2011. From2009 to 2010, he was a visiting scholar at the Department of Electrical andComputer Engineering, University of Toronto, Canada. He was the recipient of



two Best Paper Awards from IEEE GLOBECOM 2011 and IEEE CloudCom2012, respectively. His research interests include cloud computing and data-center networking, mobile cloud, green computing and communications, soft-ware-defined networking and virtualization technology, large-scale Internetcontent distribution and video streaming systems. He is a member of IEEE andACM, as well as a member of the China Computer Federation (CCF) InternetTechnical Committee. He has been a Guest Editor for IEEE Network Maga-zine, an Associate Editor for Frontiers of Computer Science, and served asTPC for IEEE INFOCOM 2013–2014 and GLOBECOM 2012–2013.

HAI JIN [SM] ([email protected]) is a Cheung Kung Scholars Chair Professor ofcomputer science at the Huazhong University of Science and Technology(HUST), China. He is now dean of the School of Computer Science and Tech-nology at HUST. He received his Ph.D. degree in computer engineering fromHUST in 1994. In 1996, he was awarded a German Academic ExchangeService fellowship to visit the Technical University of Chemnitz in Germany.He was awarded the Excellent Youth Award from the National Science Foun-dation of China in 2001. He is the chief scientist of ChinaGrid, the largestgrid computing project in China, and chief scientist of the National 973 BasicResearch Program Project of Virtualization Technology of Computing Systems.His research interests include computer architecture, virtualization technology,cloud computing and grid computing, peer-to-peer computing, network stor-age, and network security.

BO LI [F] ([email protected]) is a professor in the Department of Computer Scienceand Engineering, Hong Kong University of Science and Technology. His cur-rent research interests include large-scale content distribution in the Internet,datacenter networking, cloud computing, and mobile sensor networks. Hemade pioneering contributions in the Internet video broadcast with a system,called Coolstreaming (the keyword had over 2,000,000 entries on Google),which was credited as the world first large-scale Peer-to-Peer live video stream-ing system, which spearheaded a momentum in video broadcast industry, withno fewer than a dozen successful companies adopting the same mesh-basedpull streaming technique to deliver live media content to hundreds of millionsof users in the world. He has been an editor or a guest editor for a dozen ofIEEE journals and magazines. He was the Co-TPC Chair for IEEE INFOCOM2004. He received his B. Eng. Degree in the Computer Science fromTsinghua University, Beijing, and his Ph.D. degree in the Electrical and Com-puter Engineering from University of Massachusetts at Amherst.

DAN LI [M] ([email protected]) is currently an associate professor in theDepartment of Computer Science and Technology, Tsinghua University, Bei-jing, China. His research interests include Internet architecture and protocols,cloud computing and data center networking.



harnessing renewable energy in cloud datacenters: opportunities and challenges

Documents