1

Taking HPC beyond energy efficiency;sustainability, the new measure of success

Michael K Patterson, PhD, PE, DCEP

Eco-Technology Program Office

2

Why HPC is different

• Procurement and refresh timing

• Workload and Utilization

• HPC and infrastructure link

• Best opportunity for Energy Reuse

• Minimal UPS

• Availability needs lower than Enterprise / Financial / IPDC

• Performance measurements higher than Enterprise / Financial / IPDC

3

PUE – simple and effective

PUE is defined in terms of total annual energy and total annual IT energy, allowing a more valid site-to-site

comparison

4

PUEs: Reported and Calculated

PUE

EPA Energy Star Average 1.91

Intel Jones Farm, Hillsboro 1.41

T-Systems & Intel DC2020 Test Lab, Munich 1.24

Google 1.16

Leibniz Supercomputing Centre (LRZ) 1.15

National Center for Atmospheric Research (NCAR) 1.10

Yahoo, Lockport 1.08

Facebook, Prineville 1.07

National Renewable Energy Laboratory (NREL) 1.06

5

PUEs: Reported and Calculated

PUE

EPA Energy Star Average 1.91

Intel Jones Farm, Hillsboro 1.41

T-Systems & Intel DC2020 Test Lab, Munich 1.24

Google 1.16

Leibniz Supercomputing Centre (LRZ) 1.15

National Center for Atmospheric Research (NCAR) 1.10

Yahoo, Lockport 1.08

Facebook, Prineville 1.07

National Renewable Energy Laboratory (NREL) 1.06

It’s all about the “1”!

1.06Focus on driving the 0.06 down? Or work with the 1.0? How?

� Energy reuse� Continued improvement in compute

performance

6

Tick Tock of Energy EfficiencyMoore’s Law and IA Innovation

Source: SPECpower_ssj2008* 2 socket results from SPEC.org as of August 2010Performance and power consumption results are based on certain tests measured on specific computer systems. Any difference in system hardware, software or configuration will affect actual performance. Configurations: Two-socket Systems, Test Results for SPECpower_ssj2008, Testing by Hewlett-Packard. For more information go to http://www.intel.com/performance

Source: SPECpower_ssj2008* 2 socket results from SPEC.org as of August 2010

Double Efficiency every 16 months (67% CAGR)

0

50

100

150

200

250

300

350

400

450

0 200,000 400,000 600,000 800,000 1,000,000

Po

we

r (W

)

SSJ Ops

2 Socket Volume-Server Power vs. Performance

2010Intel® Xeon®

X5670

2009Intel® Xeon®

X55702008

Intel® Xeon®

E5450

2006Intel®

Xeon®

5160

2004Intel®

Xeon®

3.8GHz

Pow

er: L

ower

is b

ette

r

Performance: Higher is better

Higher Efficiency

65�45nm 45�32nmNehalemCore

TICK TOCK TICK TOCKSandy Bridge

TICK

7

LRZ-MunichIBM/Intel supplying the new LRZ-Munich system. 40C water to the servers. Free cooling 100% of the year. PUE ~ 1.15 Supplier must pay the power bill for the first several years of the system operations.

Power Bill

K- Euros/TF

€2308

€113

€7

Energy cost / Tflop plummeting, butPower bill hitting “political” ceilings

8

Free cooling in Atlanta?

Recall that LRZ is using 40 °°°°C cooling

water

9

“I am re-using waste heat from my

data center on another part of my

site and my PUE is 0.8!”

Ever hear someone say something like:

10

“I am reusing waste heat from my data center

on another part of my site and my PUE is 0.8!”

• While re-using excess energy from the data center can

be a good thing to do, it should not be rolled into PUE.

The definition of PUE does not allow this. PUE is ALWAYS

greater than or equal to 1.0

•But there is a new metric to do this; ERE

11

Energy Reuse Effectiveness

A new energy efficiency metric

Similar to PUE but accounts for reuse energy

PUE and ERE can both provide insight

• Different perspectives on efficiency vs reuse

12

Utility

Cooling

UPS PDU

IT

RejectedEnergy

(a)

(b)(c) (d)

(f)

(e)

ReusedEnergy

(g)

ERE – adds energy reuse to the PUE concept

13

ERE Definition

IT

Reused-IT+Lighting+Power+Cooling =ERE

IT

IT+Lighting+Power+Cooling =PUE

Energy IT

Energy Total =PUE

Energy IT

Energy Reused -Energy Total =ERE

14

ERE Alternate Development

PUE ERF)1(IT

Reused-IT+Light+Pwr+Cool =ERE ×−=

Energy Total

Energy Reuse =ERF

Define energy reuse factor (ERF) as:

Then:

PUE ERF)-(1 =ERE ×

And finally:

ERF and PUE are mathematically related, but differ and need to defined and reported clearly.

15

Comparison with PUEOne view of PUE is that is the “tax” or burden in energy costs you must pay above the IT load to run the Data Center; ERE allows the same vision

PUE = 1.0 means 100% of the energy you bring in to the data center goes to the IT

ERE = 1.0 means you only need to bring into the site an amount equal to 100% of the IT energy to support the Data Center

We need both!

Case 1 Case 2

PUE = 2.0 PUE = 1.2

ERF= 0.55 ERF=0.25

ERE = 0.9 ERE=0.9

Case 1 focus on PUE, Case 2 focus on ERF

16

Towards the Net-Zero Data

Center: Development and

Application of an Energy

Reuse Metric

Technical Paper presented last

June: ASHRAE Summer Meeting,

Montreal

17

PUE & ERE resorted….

PUE Energy Reuse

EPA Energy Star Average 1.91

Intel Jones Farm, Hillsboro 1.41

T-Systems & Intel DC2020 Test Lab, Munich 1.24

Google 1.16

National Center for Atmospheric Research (NCAR) 1.10

Yahoo, Lockport 1.08

Facebook, Prineville 1.07

Leibniz Supercomputing Centre (LRZ) 1.15 � ERE <1.0

National Renewable Energy Laboratory (NREL) 1.06 � ERE <1.0

18

Two new metrics for Data Center sustainability

Published by The Green Grid

Development of the Metrics will give better focus on Data Center sustainability

Water and Carbon– increasing focus on sustainability

19

New Metrics… all in the PUE family

��� =���� ����������

��������

��� =�������������� ��������������������������

��������

��� =���� �����������

��������

���!"#$%& =���� ��� ��������������� + ���� �����������

��������

WUE ~ Liters/kWh CUE ~ kgCO2eq/kWh

20

Why two WUE’s?

��� =���� �����������

��������

���!"#$%& =���� ��� ��������������� + ���� �����������

��������

21

The Sustainability Triangle – an example

Water

Technology choices are typically a trade-off

22

Exascale by 2019-2020Business-as-usual (2X perf/watt

every 16 months) ����140 MWTarget ����20 MW

23

Exascale Challenges

� 140 MW just won’t work (~$140M / year to operate)

� Target is 20 MW!

� Even with Moore’s Law like improvements we can’t get there (20 MW) by 2019-2020

� Government / Industry / Academia partnerships must happen to get us there

� If we don’t, HPC growth will stall!

� If we do, amazing things can happen…..

24

TECHNOLOGY AND THE ENVIRONMENT

Use Computing to Improve Energy Savings Outside

Information and Communications Technology

98%The Big Opportunity

Drive Computing to Be More Energy Efficient

~2%Opportunity

Does HPC have a Carbon ROI?Should we explore it?

~2%

25

Carbon Accounting

We will talk about a lot of metrics but here is one that we might want to develop…

���()$*"+ =��������,��� ���������

����������������

26

()$*"+

27ORNL ReView “Supercomputers Help Model Cars in Collisions”

()$*"+

28

Opportunities for the Sustainable HPC Center� HPC is NOT like your typical data center

� Harvest the differences

� Focus on the 1.0!

� Workload

� Architecture

� Carbon ROI

� Exascale R&D

� Beyond Energy Efficiency to Sustainability

� Sustainability Triangles

� Metrics – ERE, CUE, & WUE

29

Thank You! Questions?