PowerComparisonofCloudDataCenterArchitectures

PietroRuiuAndreaBianco,PaoloGiacconeClaudioFiandrino,DzmitryKliazovich

13thItalianNetworkingWorkshop:SanCandido,ItalyJanuary13-15,2016

Thequestforgreendatacenters•  Datacentersarethenewpollutersof21st

century–  in2012,accountedfor15%oftheglobalICTenergyconsumpSon

–  expectedtoincreaseinthenextyears•  DatacenterconsumpSon

–  75%ICTequipment•  poweringandcooling•  mostlyduetoservers

–  25%powerdistribuSonandfacilityoperaSons•  StronginterestindesigningandoperaSng

datacenterswithhigherenergyefficiency–  notonlytoreduceOPEXcosts

2

KeyquesSon

•  Given•  adatacentertopology•  apowerconsumpSonprofileforeachICTdevice

•  Definethemin-powerjoballocaSonpolicy•  EvaluatepowerconsumpSonasfuncSonofthedatacenterload

ClassicalquesSons

•  Given•  apowerconsumpSonprofileforeachICTdevice•  agenericjoballocaSonpolicy

•  ComparethepowerconsumpSonbehaviorinfuncSonofthedatacentertopology

OurquesSon

3

Datacentermodel

4

Datacenternetwork(DCN)

Power

Load

Power

Load

Power

Load

Servers

ICTdevice Powerprofile

LocalvsglobalenergyproporSonality

• consumeproporSonaltotheload• consume(andpay)onlyifreallyneeded

IdealenergyproporSonality

• Constantpower(CONST)• FullEnergyProporSonal(FEP)• Linear(LIN)

LocalpowerconsumpSonforsingledevice

• maybeverydifferentfromlocalpowerconsumpSon

GlobalpowerconsumpSonforoverallsystem

5

CONST

Power

Load

FEP

Power

Load

LIN

Power

Load

Power

Load

GlobalpowerconsumpSon

6

•  Nresources/devicestobeallocatedforasetofrequests/VMs•  powerconsumpSonprofileforeachresource/device

•  allocaSonpolicy•  consolidate:acSvatetheminimumnumberofresources•  load-balance:distributeloadacrosstheresources

ResourceallocaSon

•  dependsongranularityoftheresources(i.e.thevalueofN)•  dependsonallocaSonpolicy

OverallpowerconsumpSon

GlobalPowerConsumpSon

7

LocalpowerconsumpSon Consolidatepolicy Load-balancepolicy

Normalizedpower=Power/Load

FEP

Power

Load

Power

Load Load

Norm.Pow

er

Power

Load Load

Norm.Pow

er

CONST

Power

Load

N

Load

Power

N

Load

Power

N LoadNorm.Pow

er

NLoad

Norm.Pow

er

•  LocalenergyproporSonalityimpliesglobalenergyproporSonality•  IfNisenoughlarge,consolidatepolicyreachesglobalenergyproporSonalityforCONSTlocalpower

LocalandGlobalEnergyProporSonality

OurcontribuSons

•  Network-awaremin-powerVMonlineallocaSonpolicy

•  Flow-levelC++simulatorofdatacenters•  Powercomparisonofdifferentnetworktopologies

8

•  EnergyprofilesforeachICTdevice(switch,link,server)

•  DCNtopology•  VMarrivalprocess

Flow-levelsimulator

•  GlobalpowerconsumpSon

•  LoadoneachICTdevice

•  VMblockingprobabilityVMallocaSon

policy

DCNtopologies

•  tradiSonalClos-basedswitchtopologies– classical2,3Sers–  Jupiter•  Google’sdisclosedDCNarchitecture•  “JupiterRising:ADecadeofClosTopologiesandCentralizedControlinGoogle’sDatacenterNetwork”,ACMSIGCOMMCCR,Oct.2015

9

TABLE II. DATA CENTER ARCHITECTURES PARAMETERS

PARAMETER TWO-TIER THREE-TIER JUPITER

Small data center Num. Servers 180 180 192scenario Num. Switches 28 27 44

Large data center Num. Servers 4600 4800 4608scenario Num. Switches 530 504 464

traffic between the two VMs is assumed to be bidirectional andis associated a random value. If we define the degree of a VMas the total number of VMs with whom it is communicating,this VM generation model allows to obtain VMs with randomdegree, with an average close to one. The choice of thedestination VM is affecting the degree distribution. We adoptedthe approach shown in Algorithm 2 to generate the trafficflows in a set of V VMs, given an “attachment” probabilityp 2 (0, 1]. In few words, we use geometric trials to findthe destination VM to which the new VM is connected. Thisallows us to distribute fairly the communications among allthe VMs. Actually, the value of p gives the level of varianceon the VMs. When p is close to 1, the maximum degree ofthe VMs is also close to 1. Whereas, when p approaches 0, forenough large V , the maximum degree becomes much largerthan 1.

In our work we do not allow VMs to migrate and forsimplicity we do not consider the case in which a VMfinishes and leaves the server. Thus the overall load of thedata center grows with the number of allocated VMs. Inthe case of a blocking event during the allocation of a newVM, we keep generating new VMs until we reach a givenmaximum number of VMs. We set equal to 50,000 VMs thetotal number of VMs that were generated in each simulationrun. This simulation approach has two advantages. First, it ispossible to test the data center allocation for different valuesof load with just one simulation run; runs are only repeatedto get acceptable confidence intervals for each level of load.Second, it is possible to keep feeding the data center untilit completely saturates either in terms of server resources ornetwork resources. This provides a kind of worst-case loadscenario.

The network topology interconnecting the servers is mod-eled with a directed graph, in which each edge is associatedwith a capacity measured in Gbps. The communication be-tween VM is simulated at flow level, thus by allocating therequested bandwidth on the path connecting the two VMs.Notably, the simulation of the traffic at flow level allowed usto investigate also large data center networks. For performanceevaluation we considered two main scenarios, whose detailsare provided in Table II. All the three architectures have beenscaled down to fit a given number of servers, following theoriginal structure of each interconnection network. Note thatthe actual number of servers has been chosen to be compatiblewith the number of ports of the ToR switches. We devised asmall data center scenario, with around 200 servers, to betterassess and analyze the joint effect of the allocation policies andpower consumption profiles. To validate the results on largerdata centers, we considered also a large data center scenario,with around 5000 servers.

To be able to compare fairly the performance for thedifferent architectures, we propose to consider the average

power per VM. This is an interesting figure of merit to comparedifferent data center architectures, since for the cloud operatorit is directly related to the operational cost of each VM. Inaddition, it ensures our analysis to be completely independentof the number of devices each architecture actually hosts. Inour results, we will consider the total power, obtained bysumming the contribution of the servers and of the networkdevices, and the network power, obtained by considering onlythe contribution of the network devices.

B. Experimental Results

For evaluation purposes we have considered as powerconsumption profiles CONST, FEP and LIN, being the REALprofile very similar to the LIN one.

Fig. 2 compares the per-VM power consumption for theconsidered 3 data center networks and the two allocationpolicies (MNP/RSS). This analysis is performed for smalldata centers. The MNP policy outperforms the RSS policyfor low and medium loads of the data center (25-75%) in allthe three networks. Interestingly, considering the FEP powerconsumption profiles for the equipment and loads that saturatethe data center capacity, the gain using an optimized policysuch as MNP is minimum. It should also be noted thatMNP policy ensures loading the data center in a energy-proportional manner, as the power spent per VM remainsalmost constant. On the other hand, the RSS policy at lowloads of the data center requires more power per VM becausethe workload is not consolidated at the servers. As a resultmore devices need to be powered on. The two- and three-tier architectures in Fig. 2(a) and Fig. 2(b) provides verysimilar performance while adopting the MNP policy. However,if using the RSS policy, the two tier architecture achievesbetter results. To illustrate this behavior, the power per VMspent by the two- and three-tier architectures is 11.57 W and19.15 W respectively, for a data center load equal to 25% andtaking into account the LIN power consumption profile. This isbecause the three-tier design relies on high power consumingdevices in aggregation and core layers. Jupiter achieves betterperformance if compared with the two- and three-tier design(see Fig. 2(c)). Although Jupiter has more switches, the powerper VM that is attributed to the network is lower for medium-low loads of the data center and becomes comparable withthe other architectures only for very high loads. Consideringthat Jupiter exploits low power demanding switches, the reasonbehind such performance is due to the higher number of VMsit can support.

Fig. 3 compares the performance of the architectures bycomparing the amount of power spent to operate the serversand the switches separately. For this analysis, we consider onlythe LIN profile for the IT equipment, being such a profile themost similar to a realistic one (see Fig. 1). As expected, theMNP policy provides higher energy savings, if compared withthe RSS policy. In both policies, the computing servers accountthe most for power consumption. This is especially evident forthe MNP policy in Fig. 3(a), where the network consumptionaccounts for only 1 to 2% of the overall power consumption.This is because through consolidation, the servers powered onbecome fully loaded and they contribute the most to the perVM power consumption (see Table I). For the MNP policy, allthe architectures provides very similar performance, being the

•  Switches:10coresw–18TORsw

•  Servers180

•  TotalICTdevices:208nodes

CORE

TOR

10Gb

ps

40Gb

ps

2-SersDCN

10

•  3coresw–6aggregaSonsw–18TORsw

•  180servers–27switches–207nodes

CORE

AGGREGATION

TOR

10Gb

ps

40Gb

ps

40Gb

ps

3-SersDCN

11

•  24spinesw–16aggregaSonsw–16TORsw

•  192servers–44switches–236nodes

=4p@40Gbps(or16p@10Gbps)SPIN

EAGGREGATIO

N

TOR

10Gb

ps

10Gb

ps

40Gb

ps

40Gb

ps

40Gb

ps

MB MB MB MB

Jupiter-likeDCN

12

OnlineVMallocaSonpolicy

13

•  foreachVM,selectaserveratrandom• connectthroughtheminimumincrementalDCNpower

•  load-balanceontheservers

RSS(RandomServerSelecSon)

•  foreachVM,selecttheserverwithminimumincrementalpower(server+DCN)

• consolidateVMsinthesameserver,inthesamerack,inclosebyracks,etc

• variantofmin-costDijkstraalgorithm

MNP(MinimumNetworkPower)

VMgeneraSon

•  Smeisslooed•  ateachSmelot,anewVMarrivesandmustcommunicateBbpstoapreviouslyrandomlyallocatedVM–  Bisrandomlychosen–  desSnaSonVMischosenwithBernoullitrials

•  simulaSoncanrununSlsaturaSngthedatacenter

14

VM1 VM2 VM3 VM4 VM5

RSS(RandomServerSelecSon)•  smalldatacenter(180-192servers)•  JupiterappearstobethemostenergyproporSonal

–  duetothelargernumberofswitches(44vs27-28)

15

0

5

10

15

20

25

20 30 40 50 60 70 80 90 100

Norm

aliz

ed p

ower

per

VM

[W]

Data Center load [%]

3-Tier

LINFEP

CONST

0

5

10

15

20

25

20 30 40 50 60 70 80 90 100Data Center load [%]

2-Tier

LINFEP

CONST

0

5

10

15

20

25

20 30 40 50 60 70 80 90 100Data Center load [%]

Jupiter

LINFEP

CONST

MNP(MinimumNetworkPower)

16

•  smalldatacenter(180-192servers)•  MNPallowstoachieveglobalenergy-proporSonality•  underFEP,powerjumpsduetoabruptacSvaSonofnewlayers

0

0.5

1

1.5

2

2.5

3

20 30 40 50 60 70 80 90 100

Normalized power per VM [W]

Data Center load [%]

3-Tier

LINFEP

CONST

0

0.5

1

1.5

2

2.5

3

20 30 40 50 60 70 80 90 100Data Center load [%]

2-Tier

LINFEP

CONST

0

0.5

1

1.5

2

2.5

3

20 30 40 50 60 70 80 90 100Data Center load [%]

Jupiter

LINFEP

CONST

Conclusions

• GlobalenergyproporSonalityofanoveralldatacenterdependson•  localpowerprofileofeachdevice•  topology(numberofdevices)• VMallocaSonpolicy

Take-homemessage

•  considerlargetopologieswith10,000servers•  comparedatacenternetworksgiventhesamebisecSonbandwidth

•  considertheallocaSonofclustersofVMs

Futureworks

17