24 x 7 Energy Efficiency

February, 2007William Tschudi

wftschudi@lbl.gov

A “research roadmap” A “research roadmap” developed for the California developed for the California Energy Commission outlines Energy Commission outlines key areas for energy efficiency key areas for energy efficiency research, development, and research, development, and demonstration – demonstration –

This includes strategies that This includes strategies that can be implemented today.can be implemented today.

Data Center Research Roadmap

Data Center research activitiesData Center research activities

Benchmarking and 22 data center case Benchmarking and 22 data center case studiesstudies

Best practices identified Best practices identified Self-benchmarking protocol Self-benchmarking protocol Power supply efficiency studyPower supply efficiency study UPS systems efficiency studyUPS systems efficiency study Standby generation lossesStandby generation losses Performance metrics – Computation/wattPerformance metrics – Computation/watt

LBNL data center LBNL data center demonstration projectsdemonstration projects

““Air management” Air management” Outside air economizerOutside air economizer

– Contamination concernsContamination concerns– Humidity control concernsHumidity control concerns

DC poweringDC powering– Facility levelFacility level– Rack levelRack level

LBNL data center LBNL data center Federal projectsFederal projects

Case studies Case studies Technical assistanceTechnical assistance Emerging technologyEmerging technology

Investigating use of infrared thermography as a visualization tool

Benchmarking energy end useBenchmarking energy end use

local distribution lines

to the building, 480 V

HVAC system

lights, office space, etc.

UPS PDU computer racks

backup diesel generators

Electricity Flows in Data CentersElectricity Flows in Data Centers

computerequipment

uninterruptible

load

UPS = Uninterruptible Power Supply

PDU = Power Distribution Unit;

IT equipment load densityIT equipment load density

IT Equipment Load Intensity

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Data Center Number

Wat

ts/s

q.

ft.

2003 BenchmarksAve. ~ 25

2005 BenchmarksAve. ~ 52

Overall power use in Data Overall power use in Data CentersCenters

Courtesy of Michael Patterson, Intel Corporation

Performance varies

Data Center Server Load

51%

Data Center CRAC Units

25%

Cooling Tower Plant4%

Electrical Room Cooling

4%

Office Space Conditioning

1%

Lighting2%

Other13%

Computer Loads67%

HVAC - Air Movement

7%

Lighting2%

HVAC - Chiller and

Pumps24%

The relative percentages of the energy actually doing computing varies considerably.

Percentage of power delivered Percentage of power delivered to IT equipmentto IT equipment

IT Equipment load Index

0.59

0.55

0.66

0.33

0.43

0.68

0.59

0.47

0.38

0.63

0.42

0.49 0.49

0.59

0.670.70

0.75

0.60

0.74

1 2 3 4 5 6 7 8 9 10 11 12 16 17 18 19 20 21 22

Data Center Number

All values are shown as a fraction of the respective data center total power consumption.

Average 0.49

Benchmark results helped to Benchmark results helped to find best practicesfind best practices

The ratio of IT equipment power to the The ratio of IT equipment power to the total is an indicator of relative overall total is an indicator of relative overall efficiency. Examination of individual efficiency. Examination of individual systems and components in the systems and components in the centers that performed well helped to centers that performed well helped to identify best practices.identify best practices.

Lets talk about a few….Lets talk about a few….

Best practices topics identified Best practices topics identified through benchmarkingthrough benchmarking

HVAC – Air Delivery – Water Systems

Facility Electrical Systems

IT Equipment

Cross-cutting / misc. issues

Air management

Cooling plant optimization

UPS systems

Power Supply efficiency

Motor efficiency

Air economizers

Free cooling Self generation

Sleep/ standby loads

Right sizing

Humidification controls alternatives

Variable speed pumping

AC-DC Distribution

IT equip fans Variable speed drives

Centralized air handlers

Variable speed Chillers

Standby generation

Lighting

Direct liquid cooling

Maintenance

Low pressure drop air distribution

Commissioning/ continuous benchmarking

Fan efficiency Heat recovery Redundancies Method of charging for

space and power Building envelope

A word about appropriate A word about appropriate environmental conditions…environmental conditions…

ASHRAE published thermal guidelinesASHRAE published thermal guidelines– Majority of IT suppliers participatedMajority of IT suppliers participated– Guidelines allow most centers to relax Guidelines allow most centers to relax

setpoints over standard practicesetpoints over standard practice Recommended and allowable ranges of Recommended and allowable ranges of

temperature and humidity are provided – temperature and humidity are provided – at the inlet to the IT equipmentat the inlet to the IT equipment

High temperatures in the “hot aisles” and High temperatures in the “hot aisles” and return to air conditioners is desirable.return to air conditioners is desirable.

Temperature guidelines – Temperature guidelines – at the inlet to IT equipmentat the inlet to IT equipment

ASHRAE TEMPERATURE GUIDELINES

40

50

60

70

80

90

100

De

gre

es

F

ASHRAE Allowable Maximum

ASHRAE Allowable Minimum

ASHRAE Recommended Maximum

ASHRAE Recommended Minimum

Humidity guidelines – Humidity guidelines – at the inlet to IT equipmentat the inlet to IT equipment

ASHRAE HUMIDITY GUIDELINES

0

10

20

30

40

50

60

70

80

90

100

% R

ela

tiv

e H

um

idit

y

ASHRAE Allowable Maximum

ASHRAE Allowable Minimum

ASHRAE Recommended Maximum

ASHRAE Recommended Minimum

Best scenario – Best scenario – isolate cold and hotisolate cold and hot

70-75ºF

95-100ºF

Another isolation schemeAnother isolation scheme

Fan energy savingsFan energy savings – 75%– 75%

If mixing of cold supply If mixing of cold supply air with hot return air air with hot return air can be eliminated-can be eliminated-fan speed can be fan speed can be reducedreduced

Better temperature control can Better temperature control can allow raising the temperature allow raising the temperature in the entire data centerin the entire data center

Cold Aisle NW - PGE12813

40

45

50

55

60

65

70

75

80

85

90

6/13/2006 12:00 6/14/2006 0:00 6/14/2006 12:00 6/15/2006 0:00 6/15/2006 12:00 6/16/2006 0:00 6/16/2006 12:00

Time

Tem

per

atu

re (

deg

F)

Low

Med

High

Baseline Alternate 1

Setup

Setup

Alternate 2

ASHRAE Recommended Range

Ranges during demonstration

Best practices – free cooling Best practices – free cooling with air economizerswith air economizers

HVAC – Air Delivery – Water Systems

Facility Electrical Systems

IT Equipment

Cross-cutting / misc. issues

Air management

Cooling plant optimization

UPS systems

Power Supply efficiency

Motor efficiency

Air economizers

Free cooling Self generation

Sleep/ standby loads

Right sizing

Humidification controls alternatives

Variable speed pumping

AC-DC Distribution

IT equip fans Variable speed drives

Centralized air handlers

Variable speed Chillers

Standby generation

Lighting

Direct liquid cooling

Maintenance

Low pressure drop air distribution

Commissioning/ continuous benchmarking

Fan efficiency Heat recovery Redundancies Method of charging for

space and power Building envelope

Encouraging outside air Encouraging outside air economizerseconomizers

Issue: Issue: – Many are reluctant to use air economizersMany are reluctant to use air economizers– Outdoor pollutants and humidity control Outdoor pollutants and humidity control

considered equipment riskconsidered equipment risk Goal: Goal:

– Encourage use of outside air economizers where Encourage use of outside air economizers where climate is appropriateclimate is appropriate

Strategy: Strategy: – Address concerns: contamination/humidity controlAddress concerns: contamination/humidity control– Quantify energy savings benefitsQuantify energy savings benefits

Project objectivesProject objectives

Identify potential failure mechanisms Identify potential failure mechanisms

Measure contamination levels inside and Measure contamination levels inside and

outside of data centersoutside of data centers

Observe humidity controlObserve humidity control

Evaluate economizer effect on cumulative Evaluate economizer effect on cumulative

particulate exposureparticulate exposure

Compare particle concentrations to guidelines Compare particle concentrations to guidelines

Indoor Measurments Fine Particulate Matter

0

20

40

60

80

100

120

140

160

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

Par

ticl

e C

on

c. (

mg

/m3) LBNL

NERSC

Center 3

Center 4

Center 5

Center 6

Center 7

Center 8

Measurements inside the Measurements inside the centerscenters

IBM Standard

EPA Annual Health Standard

EPA 24-Hour Health Standard

and ASHRAE Standard

Outdoor MeasurmentsFine Particulate Matter

0

20

40

60

80

100

120

140

160

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

Par

ticl

e C

on

c. (

mg

/m3) LBNL

NERSC

Center 3

Center 4

Center 5

Center 6

Center 7

Center 8

Outdoor measurementsOutdoor measurements

IBM Standard

EPA Annual Health Standard

EPA 24-Hour Health Standard

and ASHRAE Standard

Indoor measurements Indoor measurements (note scale)(note scale)

Indoor Measurments Fine Particulate Matter

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

Par

ticl

e C

on

c. (

mg

/m3) LBNL

NERSC

Center 3

Center 4

Center 5

Center 6

Center 7

Center 8

Data center w/economizerData center w/economizer

Center 8w/economizer

0.3-5 Particulate Matter

0

10

20

30

40

50

60

70

80

90

100

8/18/060:00

8/18/0612:00

8/19/060:00

8/19/0612:00

8/20/060:00

8/20/0612:00

8/21/060:00

8/21/0612:00

8/22/060:00

8/22/0612:00

8/23/060:00

8/23/0612:00

8/24/060:00

8/24/0612:00

8/25/060:00

8/25/0612:00

8/26/060:00

Part

icle

Co

nc. (m g

/m3)

Outside

Outside (PostFilter)

PreServer

RmAmb

Indoor Relative Humidity

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

RH

(%

)

Center 8

Center 7

Center 5

Humidity measurementsHumidity measurements

ASHRAE Recommended Upper Limit

ASHRAE Recommended Lower Limit

ASHRAE Allowable Lower Limit

ASHRAE Allowable Upper Limit

FindingsFindings

Water soluble salts in combination with high Water soluble salts in combination with high humidity can cause current leakagehumidity can cause current leakage

Static electricity (caused by humans) can Static electricity (caused by humans) can occur with very low humidityoccur with very low humidity

Particle concentration typically is an order of Particle concentration typically is an order of magnitude lower than new ASHRAE limits magnitude lower than new ASHRAE limits (without economizer) (without economizer)

Filtration and humidity control on make-up air Filtration and humidity control on make-up air can provide environments similar to those in can provide environments similar to those in closed data centersclosed data centers

Best practices – Power Best practices – Power conversionconversion

HVAC – Air Delivery – Water Systems

Facility Electrical Systems

IT Equipment

Cross-cutting / misc. issues

Air management

Cooling plant optimization

UPS systems

Power Supply efficiency

Motor efficiency

Air economizers

Free cooling Self generation

Sleep/ standby loads

Right sizing

Humidification controls alternatives

Variable speed pumping

AC-DC Distribution

IT equip fans Variable speed drives

Centralized air handlers

Variable speed Chillers

Standby generation

Lighting

Direct liquid cooling

Maintenance

Low pressure drop air distribution

Commissioning/ continuous benchmarking

Fan efficiency Heat recovery Redundancies Method of charging for

space and power Building envelope

Inverter

In Out

Bypass

Battery/ChargerRectifier

Internal Drive

External Drive

I/O

Memory Controller

mProcessor

SDRAM

Graphics Controller

DC/DCAC/DC

DC/DC

AC/DC Multi output PS

Voltage Regulator Modules

5V

12V

3.3V

12V 1.5/2.5V

1.1V-1.85V

3.3V

3.3V

12V

PWM/PFCSwitcher

Unregulated DCTo Multi Output Regulated DC

Voltages

Data Center power conversions

AC voltage conversions

Research illustrated large losses in power conversionResearch illustrated large losses in power conversion

45%

50%

55%

60%

65%

70%

75%

80%

85%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

% of Nameplate Power Output

% E

ffic

ien

cy

Average of All Servers

Uninterruptible Power Supplies (UPS)

Power Supplies in IT

equipment

Factory Measurements of UPS Efficiency

70%

75%

80%

85%

90%

95%

100%

0% 20% 40% 60% 80% 100%

Percent of Rated Active Power Load

Eff

icie

nc

y

Flywheel UPS

Double-Conversion UPS

Delta-Conversion UPS

(tested using linear loads)

DC powering data centers DC powering data centers

Goal: Goal: Show that a DC system could be Show that a DC system could be assembled with commercially available assembled with commercially available components and measure actual components and measure actual energy savings – a proof of concept energy savings – a proof of concept demonstration.demonstration.

Included in the Included in the demonstrationdemonstration

Side-by-side comparison of Side-by-side comparison of traditional AC system with traditional AC system with new DC systemnew DC system

– Facility level distributionFacility level distribution

– Rack level distributionRack level distribution

Power measurements at Power measurements at conversion pointsconversion points

Servers modified to accept Servers modified to accept 380 V. DC380 V. DC

Artificial loads to more fully Artificial loads to more fully simulate data centersimulate data center

Typical AC distribution Typical AC distribution todaytoday

DC/ACAC/DC480 VACBulk Power

Supply

UPS PDU

AC/DC DC/DC VRM

VRM

VRM

VRM

VRM

VRM

12 V

Loadsusing

LegacyVoltages

Loadsusing

SiliconVoltages

12 V

5 V

3.3 V

1.2 V

1.8 V

0.8 VServer

PSU

480 Volt AC

380 V DC after first stage conversion

Facility-level DC Facility-level DC distributiondistribution

AC/DC480 VACBulk Power

SupplyDC UPS

orRectifier

DC/DC VRM

VRM

VRM

VRM

VRM

VRM

12 V

Loadsusing

LegacyVoltages

Loadsusing

SiliconVoltages

12 V

5 V

3.3 V

1.2 V

1.8 V

0.8 VServer

PSU

380 VDC

380V.DC

480 Volt AC

380 V DC is delivered directly into the server to the same point as in an AC powered server. This eliminates the DC-AC conversion at the UPS and the AC-DC conversion in the server. Also, less equipment is needed.

Rack-level DC Rack-level DC distributiondistribution

DC/ACAC/DC480 VACBulk Power

Supply

UPS PDU

AC/DC DC/DC VRM

VRM

VRM

VRM

VRM

VRM

12 V

Loadsusing

LegacyVoltages

Loadsusing

SiliconVoltages

12 V

5 V

3.3 V

1.2 V

1.8 V

0.8 V

Server

380 VDC

Rack

PSU

480 Volt AC

AC system loss compared to DCAC system loss compared to DC

DC/ACAC/DC480 VACBulk Power

Supply

UPS PDU

AC/DC DC/DC VRM

VRM

VRM

VRM

VRM

VRM

12 V

Loadsusing

LegacyVoltages

Loadsusing

SiliconVoltages

12 V

5 V

3.3 V

1.2 V

1.8 V

0.8 VServer

PSU

AC/DC480 VACBulk Power

SupplyDC UPS

orRectifier

DC/DC VRM

VRM

VRM

VRM

VRM

VRM

12 V

Loadsusing

LegacyVoltages

Loadsusing

SiliconVoltages

12 V

5 V

3.3 V

1.2 V

1.8 V

0.8 VServer

PSU

380 VDC

7-7.3% measured improvement

2-5% measured improvement

Rotary UPS

Energy savings for a typical data Energy savings for a typical data centercenter

20% or more 20% or more facilityfacility level energy savings level energy savings because:because:

– Redundant UPS and server power supplies operate at Redundant UPS and server power supplies operate at reduced efficiencyreduced efficiency

– Cooling loads would be reduced. Cooling loads would be reduced.

– The demonstration comparisons were against “best in The demonstration comparisons were against “best in class” systems which performed better than typical class” systems which performed better than typical systems we benchmarked.systems we benchmarked.

Further optimization of conversion Further optimization of conversion devices/voltages is possibledevices/voltages is possible

Demonstration set-up –

see website for more detail

DC power – next stepsDC power – next steps

DC power pilot installation(s) DC power pilot installation(s) Standardize distribution voltageStandardize distribution voltage Standardize DC connector and power stripsStandardize DC connector and power strips Server manufacturers develop power supply Server manufacturers develop power supply

specificationspecification Power supply manufacturers develop Power supply manufacturers develop

prototypeprototype UL and communications certificationUL and communications certification

Design guidelines available through PG&E’s Energy Design Resources website

Design and Training Resources

A web-based training resource available on LBL’s website

website: website: http://hightech.lbl.gov/datacenterhttp://hightech.lbl.gov/datacenter

s/s/

Discussion/Questions??Discussion/Questions??