Authors: William Tschudi, Lawrence Berkeley Authors: William Tschudi, Lawrence Berkeley National LabNational Lab

Stephen Fok, Pacific Gas and Electric Stephen Fok, Pacific Gas and Electric CompanyCompany

Presented atPresented atASHRAE Winter Meeting in Dallas, TexasASHRAE Winter Meeting in Dallas, Texas

January 31, 2007 January 31, 2007

BEST PRACTICES for ENERGY EFFICIENT DATA CENTERS INDENTIFIED THROUGH CASE STUDIES and DEMONSTRATION

PROJECTS

Northern California service territory Northern California service territory including the Silicon Valleyincluding the Silicon Valley

Estimated 400-500 MW of Data Center Estimated 400-500 MW of Data Center electricity load in our service territoryelectricity load in our service territory

Partnering with LBNL in addressing Data Partnering with LBNL in addressing Data Center Energy Efficiency Opportunities & Center Energy Efficiency Opportunities & Emerging TechnologiesEmerging Technologies

In 2006, major utility EE Program In 2006, major utility EE Program Implementation produced 132 MW and 677 Implementation produced 132 MW and 677 GWh in Electricity SavingsGWh in Electricity Savings

BEST PRACTICES for ENERGY EFFICIENT DATA CENTERS INDENTIFIED THROUGH CASE STUDIES and DEMONSTRATION

PROJECTS

BEST PRACTICES for ENERGY EFFICIENT DATA CENTERS INDENTIFIED THROUGH CASE STUDIES and DEMONSTRATION

PROJECTS OVERVIEW:

LBNL Data Center Benchmarking - Roadmap Best Practices and Design Guidelines Sourcebook Air Management Demonstration Air Economizer Demonstration Centralized Air Handling Case Study Free Cooling using Water-Side Economizer Case Study Efficient Uninterruptible Power Supplies Case Study Q&A

A “research roadmap” was A “research roadmap” was developed for the California developed for the California Energy Commission and Energy Commission and outlined key areas for energy outlined key areas for energy efficiency research, efficiency research, development, and development, and demonstrationdemonstration

LBNL Data Center Benchmarking

LBNL Data Center LBNL Data Center BenchmarkingBenchmarking

23 Data Center Case Studies 23 Data Center Case Studies --

Banks/financial institutionsBanks/financial institutions Web hostingWeb hosting Internet service providerInternet service provider Scientific ComputingScientific Computing Recovery centerRecovery center Tax processingTax processing Storage and router Storage and router

manufacturersmanufacturers

LBNL Data Center LBNL Data Center Benchmarking -IT equipment Benchmarking -IT equipment load densityload density

IT Equipment Load Intensity

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Data Center Number

Wat

ts/s

q. ft

.

2003 BenchmarksAve. ~ 25

2005 BenchmarksAve. ~ 52

Benchmarking energy end useBenchmarking energy end use

local distribution lines

to the building, 480 V

HVAC system

lights, office space, etc.

UPS PDU computer racks

backup diesel generators

Electricity Flows in Data CentersElectricity Flows in Data Centers

computerequipment

uninterruptible

load

UPS = Uninterruptible Power Supply

PDU = Power Distribution Unit;

Overall power use in Data Overall power use in Data CentersCenters

Courtesy of Intel Corporation

Server fans 15W

UPS +PDU 25W

PSU 50W

Load Load 100W100W

VR 15W

HVAC system 40W

Total Total 245W245W

Server fans 15W

UPS +PDU 25W

PSU 50W

Load Load 100W100W

VR 15W

HVAC system 40W

Total Total 245W245W

Data Center performance Data Center performance differencesdifferences

Variation in Data Center Energy End Uses

0%10%20%30%40%50%60%70%80%90%

100%

AVERAG

E 1 2 3 4.1 4.5 5 6.1 6.2 7 8 8.2 9

Facility Number

% o

f tot

al e

nerg

y us

e Other

Lighting

UPS Losses

Total HVAC

DC Equipment

Servers

HVAC system effectivenessHVAC system effectiveness

HVAC Effectiveness Index

0.00.51.01.52.02.53.03.54.0

1 2 3 4 5 6 7 8 9 10 11 12 14 16 17 18 19 20 21 22

Data Center Number

Rat

io o

f IT

Equi

pmen

t Po

wer

to H

VAC

Pow

er

We observed a wide variation in HVAC performance

Best Practices topics identified Best Practices topics identified through Benchmarkingthrough Benchmarking

HVAC – Air Delivery – Water Systems

Facility Electrical Systems

IT Equipment

Cross-cutting / misc. issues

Air management

Cooling plant optimization

UPS systems

Power Supply efficiency

Motor efficiency

Air economizers

Free cooling Self generation

Sleep/ standby loads

Right sizing

Humidification controls alternatives

Variable speed pumping

AC-DC Distribution

IT equip fans Variable speed drives

Centralized air handlers

Variable speed Chillers

Standby generation

Lighting

Direct liquid cooling

Maintenance

Low pressure drop air distribution

Commissioning/ continuous benchmarking

Fan efficiency Heat recovery Redundancies Method of charging for

space and power Building envelope

CA utility Design Guidelines CA utility Design Guidelines SourcebookSourcebook

Guides available through Energy Design Resources Website

Air Management DemonstrationAir Management Demonstration

Goal: Demonstrate better cooling and energy savings through improvements in air distribution in a typical and a high density environment.

Air Management Demonstration Air Management Demonstration ––High Density Data Center High Density Data Center The as-found conditions were monitoredThe as-found conditions were monitored

– TemperaturesTemperatures– Fan energyFan energy– IT energyIT energy

An area containing two high-intensity An area containing two high-intensity rows and three computer room air rows and three computer room air conditioning units was physically conditioning units was physically isolated from rest of the center – isolated from rest of the center – approximately 175 W/sfapproximately 175 W/sf

Air Mgmt Demo, High Density Air Mgmt Demo, High Density Data Center - (First alternate) Data Center - (First alternate) ColdColdAisle IsolationAisle Isolation

Air Mgmt Demo, High Density Air Mgmt Demo, High Density Data Center - (Second Data Center - (Second Alternate): Alternate): Hot Aisle IsolationHot Aisle Isolation

Air Mgmt Demo, High Density Air Mgmt Demo, High Density Data Center - ProcedureData Center - Procedure

Once test area was isolated, air conditioner Once test area was isolated, air conditioner fan speed was reduced using existing VFD’sfan speed was reduced using existing VFD’s

Temperatures at the servers were monitoredTemperatures at the servers were monitored IT equipment and fan energy were monitoredIT equipment and fan energy were monitored Chilled water temperatures were monitoredChilled water temperatures were monitored Hot aisle return air temperatures were Hot aisle return air temperatures were

monitored – monitored – ΔΔT was determinedT was determined

Air Mgmt Demo, High Density Air Mgmt Demo, High Density Data Center - Fan energy Data Center - Fan energy savingssavings:: 75% 75%

Since there was no Since there was no mixing of cold supply air mixing of cold supply air with hot return air -with hot return air -fan speed could be fan speed could be reducedreduced

Air Mgmt Demo, High Density Data Air Mgmt Demo, High Density Data Center - Better temp control would Center - Better temp control would allow raising the temp in the entire allow raising the temp in the entire data centerdata center

Cold Aisle NW - PGE12813

40

45

50

55

60

65

70

75

80

85

90

6/13/2006 12:00 6/14/2006 0:00 6/14/2006 12:00 6/15/2006 0:00 6/15/2006 12:00 6/16/2006 0:00 6/16/2006 12:00

Time

Tem

pera

ture

(deg

F)

LowMedHigh

Baseline Alternate 1

Setup

Setup

Alternate 2

ASHRAE Recommended Range

Ranges during demonstration

Air Mgmt Demo, Air Mgmt Demo, High Density Data CenterHigh Density Data Center

Since air is not mixing, the overall air Since air is not mixing, the overall air supply temperature could be raised supply temperature could be raised (typically centers are below ASHRAE (typically centers are below ASHRAE recommended minimums because of recommended minimums because of mixing)mixing)

Hotter air is returned to the computer Hotter air is returned to the computer room air conditioners - Larger room air conditioners - Larger temperature difference (temperature difference (ΔΔT) allows air T) allows air conditioning unit to have more capacityconditioning unit to have more capacity

Air Economizer DemonstrationAir Economizer Demonstration

Goal: encourage use of outside air Goal: encourage use of outside air economizers where climate is economizers where climate is appropriateappropriate

Strategy: Strategy: – address data center professional’s biggest address data center professional’s biggest

concerns: contamination and humidity concerns: contamination and humidity controlcontrol

– quantify energy savings at one centerquantify energy savings at one center

Air Economizer DemonstrationAir Economizer Demonstration

Total of EIGHT data centers were studied. Total of EIGHT data centers were studied. Snapshot/monitoring over several daysSnapshot/monitoring over several days

Continuous monitoring equipment in Continuous monitoring equipment in place at one center. Data collection over place at one center. Data collection over several monthsseveral months

Before and after capability at TWO Before and after capability at TWO centerscenters

Energy savings estimate based on utility Energy savings estimate based on utility billing records. billing records.

Air Economizer DemonstrationAir Economizer Demonstration

Water soluble salts in combination with high Water soluble salts in combination with high humidity can cause failureshumidity can cause failures

Static electricity can occur with very low humidityStatic electricity can occur with very low humidity Humidity control for make-up air (or no controls) is Humidity control for make-up air (or no controls) is

usually sufficientusually sufficient Particle concentrations with normal filtration are Particle concentrations with normal filtration are

orders of magnitude lower than recommendedorders of magnitude lower than recommended Site No. 8 energy savings was estimated to be

about 1% of total energy depending on economizer set point.

Air Economizer Demo - Air Economizer Demo - Contamination is orders of Contamination is orders of magnitude less at serversmagnitude less at servers

0.3-5 Particulate Matter

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

5/15/060:00

5/15/0612:00

5/16/060:00

5/16/0612:00

5/17/060:00

5/17/0612:00

5/18/060:00

5/18/0612:00

5/19/060:00

5/19/0612:00

5/20/060:00

5/20/0612:00

5/21/060:00

5/21/0612:00

5/22/060:00

5/22/0612:00

5/23/060:00

Par

ticle

Con

c. (

mg/

m3 )

OutsideSupplyReturnrServer

150 mg/m3: IBM Suspended Particulates Limit

Air Economizer Demo - Particle Air Economizer Demo - Particle Concentrations by SizeConcentrations by Size

Particle Concentrationat Server Rack

0.0E+00

2.0E+03

4.0E+03

6.0E+03

8.0E+03

1.0E+04

1.2E+04

1.4E+04

1.6E+04

1.8E+04

2.0E+04

5/5/06 0:00 5/5/0612:00

5/6/06 0:00 5/6/0612:00

5/7/06 0:00 5/7/0612:00

5/8/06 0:00 5/8/0612:00

5/9/06 0:00

Parti

cle

Coun

t

2.0

1.0

0.7

0.5

Centralized Air Handling Case Centralized Air Handling Case StudyStudy

Utilize larger & more Utilize larger & more efficient fans and efficient fans and motors vs CRACmotors vs CRAC

Use of variable Use of variable frequency drivesfrequency drives

Located outside of Located outside of the data center the data center enhances space enhances space allocation and allocation and maintenance of IT maintenance of IT equipmentequipment

Free Cooling using Water-Side Economizer Case Study

Best suited for Best suited for climates with wet bulb climates with wet bulb temp < 55 deg F for temp < 55 deg F for >3000 hrs per year>3000 hrs per year

Very effective for Very effective for chilled water systems chilled water systems designed for >50 deg designed for >50 deg FF

Possible chilled water Possible chilled water plant energy savings plant energy savings of up to 75%of up to 75%

(1) At the same level of redundancy, approx 5% (1) At the same level of redundancy, approx 5% efficiency gains is possible from operating the efficiency gains is possible from operating the UPS at 33% vs 25% of full loadUPS at 33% vs 25% of full load(2) Additional HVAC savings improves life cycle (2) Additional HVAC savings improves life cycle costingcosting

Efficient Uninterruptible Power Supplies Case Study

Measured UPS Efficiency

0

20

40

60

80

100

0 20 40 60 80 100

Load Factor (% of rated load)

Effic

ienc

y (%

)

William Tschudi, P.E.:William Tschudi, P.E.:E-Mail – WFTschudiE-Mail – WFTschudi@lbl.gov@lbl.gov

Stephen Fok, P.E.:Stephen Fok, P.E.:E-Mail – E-Mail – skf2@pge.comskf2@pge.com

Best Practices for Energy Efficient Data Centers Identified through Case Studies and Demonstration Projects - Q & A