energy efficient data centres

Liam NewcombeBCS Data Centre Specialist Group Secretary

Energy Efficient Data Centres

Energy Use in the Data Centre

EU Code of Conduct for Data Centre Operators

Cardiff University

Asking the Wrong Questions

Per Service Energy Use

The DCSG Model

Cardiff University

Asking the Wrong Questions

Cardiff University

Utility Input Power

Input Transformer

Master PDU(s)

UPS

IT PDU(s)IT PDU(s)

Lighting, Generator Fire etc.

Chiller Plant CRAC(s)

IT PDU(s)

IT EquipmentIT Equipment

IT Equipment

IT Power Path

Non IT and Power Losses

Total Facility

Power

IT Equipment

Power

The Power Delivery Path

“For each Watt my Data Centre consumes what output do I get?”

vs.

“For each Service I deliver how much energy do I use?”

Cardiff University

Energy

Delivered IT Services

DC

iE /

PU

ED

ata

Cen

ter

Pro

du

ctiv

ity

Us

efu

l Wo

rk

Per

Ser

vice

Acc

ou

nti

ngTraditional Carbon

Accounting

Reporting Metrics

Whole Building

Whole Platform

Apply Targets

Create Incentive to Change

Analysis Metrics

Detailed Breakdown

Why is my IT Inefficient?

How do I Change?

Should I use Product X?

How much £/CO2 will I Save?

Targeting Supply Targeting Demand

What is the marginal economic or environmental benefit of operating this IT service?

Cardiff University

Per Service Energy UseData Centres

Servers

Server in a Data Centre

Cardiff University

Cardiff University

Data Centre Power - Where does it go?

Overall Power Transfer Functions

0

500

1,000

1,500

2,000

2,500

0 200 400 600 800 1,000

kW

IT Electrical Load (kW)

Dat

a C

entr

e In

pu

t P

ow

er

Transformer

Chiller

CRAC

UPS

PDU

Cabling &SwitchgearLinear IT LoadPower

Per Service Energy UseData Centres

Servers

Server in a Data Centre

Cardiff University

Cardiff University

Server Power and Efficiency vs. Workload

Commodity x86 Power and Efficiency by Workload

0

50

100

150

200

250

300

350

0% 20% 40% 60% 80% 100%

Workload

Wa

tts

0.00

0.20

0.40

0.60

0.80

1.00ServerPowerDraw

Efficiency

Per Service Energy UseData Centres

Servers

Server in a Data Centre

Cardiff University

Put the Server in a Data Centre

What does DCiE / PUE tell us?

Cardiff University

Server Power by Fixed and Proportional vs PUE / DCiE

0

250

500

750

1000

1250

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

IT Workload

Po

wer

Dra

wn

(W

atts

)

Fixed Data CentreLosses

ProportionalServer Losses

Server InputPower

Server Power byDesign PUE

Server Power byAchieved PUE

MeteringInstall Detailed Power Metering?

What about Blade Servers and Virtualisation?

Cardiff University

Install Detailed Power Metering?

ServerServer Input Power

User Work

Inaccurate Multiplier Function

Billed Input Power

ServerServer Input Power

User Work

Inaccurate Multiplier Function

Billed Input Power

ServerServer Input Power

User Work

Inaccurate Multiplier Function

Billed Input Power

MeteringInstall Detailed Power Metering?

What about Blade Servers and Virtualisation?

Cardiff University

Metering Fails for Blades or Virtualisation

Blade Server

Blade Server

Blade Server

Blade Server

Blade Server

Blade Server

PSU

PSU

PSU

PSU

Pow

er B

us

Blade Server

Virtual Machine

Virtual Machine

User Work

Input Power

Input Power

Input Power

Input Power

Where to Measure

Utility Input Power

Input Transformer

Master PDU(s)

UPS

IT PDU(s)IT PDU(s)

Lighting, Generator Fire etc.

Chiller Plant

CRAC(s)

IT PDU(s)

IT EquipmentIT Equipment

IT Equipment

Utility Electrical

Load

PDU Level IT

Equipment PowerTransformer

Output

Ancillary Load

Chiller Load

CRAC Load

UPS Input UPS Output

Rack or Device

Level IT Equipment

Power

Simple Measurements

Detailed Measurements

IT Measurements

External and Data

Floor Temperature

MeteringMust be able to link logical to

physical

Cannot solve our problem

Cardiff University

DCSG Data Centre Model

Cardiff University

DCSG Data Centre Model

IT Workload

IT Device Load to Power Function

Data Centre Power Transfer Function

Server Workloads

0 20 40 60 80 100

Percent Load

Pro

po

rtio

n

Server Power Utilisation by Load

0

100

200

300

400

500

0% 20% 40% 60% 80% 100%

Load

Wa

tts

Data Centre Power Transfer Functions

0200400600800

1,0001,2001,4001,6001,800

0 200 400 600 800 1,000

kW

IT Electrical Load (kW)

Inp

ut

Po

wer

EU Code of Conduct

for Data Centre Operators

Cardiff University

Context

Goals and Scope

Development

Best Practice

Release

Cardiff University

Context for the CoC

Cardiff University

Cardiff University

Political Context• Carbon reduction commitment• Energy security

Industry Context• Many activities within Industry• Risk of Confusion and Mixed Messaging

Cardiff University

Economic Context• Rising energy costs• Data Centres represent an increasing proportion of overall

business cost

Social Context• Rising public awareness• Data Centres represent an easy target• ICT can reduce environmental impact• Demonstrate that our industry is not profligate with energy

Goals and Scope of the CoC

Cardiff University

Cardiff University

Goals• Led by Paolo Bertoldi, Directorate General

– “aim is to inform and stimulate Data Centre operators to reduce energy consumption in a cost effective manner without hampering the critical function ….”

• A Voluntary Commitment • Reward best practice with branding and

associated EU Green marketing.

Cardiff University

Scope

• The Code of Conduct covers:– “Data centres” of all sizes – server rooms to

dedicated buildings– Both existing and new– IT power and Facility power– Equipment procurement and system design

Cardiff University

Scope

• The Code of Conduct is for:– Participants: Data centre owners and

operators– Endorsers: Vendors, consultants, industry

associations

Development

Cardiff University

Cardiff University

Substantial Interest• Broad Participation and Support from

– Vendors– Data Centre Operators– End User Organisations

• Will become a Supplier Selection Criterion

Cardiff University

Working Groups• Best Practice

– Focused on design best practice, Software, IT Architecture and Facility

• Metrics & Measurement– Developing a standard method of comparative

measurement of energy efficiency

• Data Collection & Analysis– Performance benchmarking across the industry

Best Practice

Cardiff University

Cardiff University

Best Practice Intent

• Neither a prescriptive nor exhaustive list of specific technologies

• Focus on goals

• Structured to allow the addition of new technologies

Cardiff University

Best Practice Intent

• Some Practices are required for Participants

• Practices that apply to;– Existing estate– New IT equipment– New or refitted Data Centres

• It is understood that not all operators will be able to implement all required practices

Cardiff University

Value of Practices

• Best Practices are guidance to operators on how they might improve energy efficiency

• Practices are scored 1-5 (min-max) based upon their likely energy use benefit

• Practices are ordered by score

Cardiff University

Required Practices• Grid and Virtualisation

– “Processes should be put in place to require senior business approval for any new service that requires dedicated hardware and will not run on a resource sharing grid or virtualised platform”

• Select efficient software – “Make the performance of the software, in terms of the power

draw of the hardware required to meet performance and availability targets a primary selection factor ”

Cardiff University

Required Practices• New IT Equipment

– “Include the Performance per Watt of the IT device as a high priority decision factor in the tender process ”

• Power Provisioning – “Provision power and cooling only to the as-configured

power draw capability of the equipment, not the PSU or nameplate rating ”

Example Scenario

Cardiff University

Same Computing Workload

100 One App Per Server Servers

15 Virtualised Servers

10 HPC Grid Servers

Old N+1 Data Centre, Nameplate Provisioning

Cardiff University

Old N+1 Data Centre, Nameplate

0%25%

50%75%

100%

-100

1020

3040

0.00

0.20

0.40

0.60

0.80

DCiE

IT Electrical Load

Temp

DCiE by IT Electrical Load and External Temperature for Traditional Cooling

0.60-0.80

0.40-0.60

0.20-0.40

0.00-0.20

Cardiff University

Old N+1 Data Centre, Nameplate

Overall IT Device(s) Cost

£0

£200,000

£400,000

£600,000

£800,000

One App Per Server Virtualised HPC Grid

Co

st

Fixed Overhead PowerCost

Proportional OverheadPower Cost

IT Device PSU PowerCost

M&E Infrastructure,Overprovisioning Cost

M&E Infrastructure,Utilised Cost

Maintenance Cost

Hardware Cost

Cardiff University

Old N+1 Data Centre, Nameplate

Overall IT Device(s) Energy Use

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

One App Per Server Virtualised HPC Grid

En

erg

y U

se

d k

Wh

Proportional OverheadPower

IT Device PSU Power

Fixed Overhead Power

Example Scenario

Cardiff University

Same Computing Workload

100 One App Per Server Servers

15 Virtualised Servers

10 HPC Grid Servers

Old N+1 Data Centre, Peak Provisioning

Cardiff University

Old N+1 Data Centre, Peak

Overall IT Device(s) Cost

£0

£200,000

£400,000

£600,000

£800,000

One App Per Server Virtualised HPC Grid

Co

st

Fixed Overhead PowerCost

Proportional OverheadPower Cost

IT Device PSU PowerCost

M&E Infrastructure,Overprovisioning Cost

M&E Infrastructure,Utilised Cost

Maintenance Cost

Hardware Cost

Cardiff University

Old N+1 Data Centre, Peak

Overall IT Device(s) Energy Use

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

One App Per Server Virtualised HPC Grid

En

erg

y U

se

d k

Wh

Proportional OverheadPower

IT Device PSU Power

Fixed Overhead Power

Cardiff University

Required Practices• Cooling

– “Review and if possible raise target IT equipment intake air temperature”

• Above the dew point

– “Review of cooling before IT equipment changes”

– “Rack air flow management”

Cardiff University

Required Practices – Retrofit or New• Cooling

– “Design – Contained hot or cold air”– “Variable Speed Air Fans”

– “Rack air flow management”

• Utilisation, Management and Planning– “Lean provisioning of power and cooling for 18

months worth of data floor capacity”

Example Scenario

Cardiff University

Same Computing Workload

100 One App Per Server Servers

15 Virtualised Servers

10 HPC Grid Servers

New N+1 Data Centre, Free Cooling

Cardiff University

New N+1 Data Centre, Free Cooling 19°C

0%25%

50%75%

100%

-100

1020

3040

0.00

0.20

0.40

0.60

0.80

DCiE

IT Electrical Load

Temp

DCiE by IT Electrical Load and External Temperature for Free Cooling

0.60-0.80

0.40-0.60

0.20-0.40

0.00-0.20

Cardiff University

New N+1 Data Centre, Free Cooling 25°C

0%25%

50%75%

100%

-100

1020

3040

0.00

0.20

0.40

0.60

0.80

DCiE

IT Electrical Load

Temp

DCiE by IT Electrical Load and External Temperature for Free Cooling

0.60-0.80

0.40-0.60

0.20-0.40

0.00-0.20

Cardiff University

New N+1 Data Centre, Free Cooling 25°C

Overall IT Device(s) Cost

£0

£200,000

£400,000

£600,000

£800,000

One App Per Server Virtualised HPC Grid

Co

st

Fixed Overhead PowerCost

Proportional OverheadPower Cost

IT Device PSU PowerCost

M&E Infrastructure,Overprovisioning Cost

M&E Infrastructure,Utilised Cost

Maintenance Cost

Hardware Cost

Cardiff University

New N+1 Data Centre, Free Cooling 25°C

Overall IT Device(s) Energy Use

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

One App Per Server Virtualised HPC Grid

En

erg

y U

se

d k

Wh

Proportional OverheadPower

IT Device PSU Power

Fixed Overhead Power

Example Scenario

Cardiff University

15 Virtualised Servers

Old N+1 Data Centre, Nameplate Provisioning

Old N+1 Data Centre, Peak Provisioning

New N+1 Data Centre, Free Cooling

Cardiff University

Comparison of Virtualised Scenarios

Overall IT Device(s) Cost

£0

£40,000

£80,000

£120,000

£160,000

Old Nameplate Old Peak New Free Cooling

Co

st

Fixed Overhead PowerCost

Proportional OverheadPower Cost

IT Device PSU PowerCost

M&E Infrastructure,Overprovisioning Cost

M&E Infrastructure,Utilised Cost

Maintenance Cost

Hardware Cost

Cardiff University

Comparison of Virtualised Scenarios

Overall IT Device(s) Energy Use

0

100,000

200,000

300,000

400,000

Old Nameplate Old Peak New Free Cooling

En

erg

y U

se

d k

Wh

Proportional OverheadPower

IT Device PSU Power

Fixed Overhead Power

Cardiff University

Required Practices – Retrofit or New• Cooling

– “Design – Contained hot or cold air”– “Variable Speed Air Fans”

– “Rack air flow management”

• Utilisation, Management and Planning– “Lean provisioning of power and cooling for 18

months worth of data floor capacity”

CoC Release

Cardiff University

Cardiff University

Release Target

Draft released 30th April

1st Release September 2008

Thank you

Find out more

Data Centre Specialist Group

http://dcsg.bcs.org

Cardiff University