Execution Environments for Distributed Computing

Intelligent placement of datacenters for Internet

Services

EEDC

343

30

Master in Computer Architecture, Networks and Systems - CANS

Homework number: 6

Group number: EEDC-32Francesc Lordan francesc.lordan@gmail.com

2

Popular Internet companies offer services to millions of users everyday.

These services are hosted in geographically distributed datacenters.

No public information about how they select the locations

Introduction

3

Introduction

AustinPUE: 1.39Land: 0.394 $/SFEnergy: 0.066 kWhWater: 0.40 cents/galCO2: 569 g/kWh

4

Introduction

BismarkPUE: 1.20Land: 0.434 $/SFEnergy: 0.062 kWhWater: 0.32 cents/galCO2: 869 g/kWh

5

Introduction

Los AngelesPUE: 1.41Land: 0.638 $/SFEnergy: 0.099 kWhWater: 0.33 cents/galCO2: 286 g/kWh

6

Introduction

New YorkPUE: 1.29Land: 3.460 $/SFEnergy: 0.096 kWhWater: 0.35 cents/galCO2: 960 g/kWh

7

Introduction

OrlandoPUE: 1.42Land: 0.272 $/SFEnergy: 0.081 kWhWater: 0.23 cents/galCO2: 541 g/kWh

8

Introduction

SeattlePUE: 1.19Land: 0.987 $/SFEnergy: 0.041 kWhWater: 0.65 cents/galCO2: 120 g/kWh

9

Introduction

St. LouisPUE: 1.32Land: 0.264 $/SFEnergy: 0.047 kWhWater: 0.21 cents/galCO2: 806 g/kWh

10

Framework for placement - Parameters

Cost Capital Expenses (CAPEX): investments made upfront and depreciated over the lifetime of the datacenter

– CAP_ind: independent of the number of servers. • Bringing the electricity and external networking.

– CAP_max: maximum number of servers that can be hosted• Land adquisition

• Datacenter construction

• Purchasing and installing power delivery infrastructure

• Cooling infrastructure

• Backup infrastructure

– CAP_act: purchasing the servers and internal networking gear

11

Framework for placement - Parameters

Cost Operational Expenses (OPEX): costs incurred during the operation of the datacenters

– OP_act: maintenance and administration of the equipment and external networking bandwith.• Domined by the staff compensation.

– OP_utl: electricity and water costs involved in running the servers

Lower taxes and incentives

12

Framework for placement - Parameters

Response Time: Latency between a population center and a location.

– Latency(c, d): latency between a location d and a center c.

– Pcd: Number of servers at a location d that serve request from c

– Servers(c): Number of servers required by the center c

Consistency Delay: time required for state changes to reach all mirrors

– Latency (d1, d2): one-way latency between the locations d1 and d2.

Availability: depends on the network avalability of all the datacenters

CO2 emissions: determined by the type of electricity consumed

– Emissions(d): carbon emissions (g/Kwh) at location d.

13

Framework for placement – Formulation

Inputs:– Maximum number of servers – Expected average utilization for the servers– Number of user that each server can accomodate– Amount of redundancy– Latencies and availability constraints– CAPEX and OPEX for each location– Latencies between any population center and each location– Latencies between any two locations

14

Framework for placement – Formulation

Outputs:– Optimal cost

– Maximum number of servers at each location– Number of servers that service a population center at a location

15

Framework for placement – Solutions

Simple linear programming (LP0)– Simplifies the equation to check if a datacenter must be placed at a

location and which centers it provides. Proportionally assigns the max number of servers and computes the network costs with the original one

Pre-set linear programming (LP1)– Presets if a location contains a datacenter and its size and removes

the centers which are provided variable.

Bruteforce (Brute)– Generates all the possibilities and tests them using the LP1

approach

16

Framework for placement – Solutions

Heuristic Based on LP (Heuristics)– Generates 10 possible datacenter networks for each number of

datacenters using LP0 applies the LP1 algorithm and sorts the results in increasing order of cost and finally runs the bruteforce method on a small set of solutions to obtain the most efficient.

Simualted Annealing plus LP1(SA+LP1)– SA starts with a configuration that fulfills the constraints and

evaluates the neighbors obtained using LP1. The solution is selected when there is no cost improvement within an iteration interval.

Optimized SA+LP1(OSA+LP1)– Adjusts the results of the LP1: when no servers are assigned to a

datacenter, it is removed.

17

Placement tool

User only specifies:– Area of interest– Granularity of the potentials datacenters– Location of existing datacenters– Max number of Servers– Ratio of user per server– Max latency between – Max delay– Min availability

The toolkit obtains the missing data to compute the best datacenter network in order to fulfill the user constraints.

18

Placement tool

19

Placement tool60k serversLatency <60msDelay <=85 msAvailability >= 0.99999

31789

22712

5501

20

Exploring datacenter placement tradeoffs

Latency– Latencies > 70 ms have the same cost– Latency = 50 ms is the best tradeoff between latency and cost– Latencies < 35 doubles the cost of 50 ms

Availability– Less level Tier datacenters more datacenters– It’s cheaper to achive an avaiability level with more low-level Tier

datacenters than with less high-level datacenters.– TierII datacenters are the best option

21

Exploring datacenter placement tradeoffs

Consistency delay– Low consistency delays and low latency are conflicting goals– Low consistency delays implies less datacenters and lower costs

Green Datacenters– When latencies can be relatively high, a green datacenter is less

expensive than $100K a month.

Chiller-less datacenters– Water chillers increases energy consumption by 20% and

building costs by 30%. Necessary for locations with an average temperature over 20ºC.

– Avoiding chillers is feasable when latencies are over 70 ms. It reduces costs by an 8%.

22

Questions