Direct 400Vdc forEnergy Efficient Data Centers

27 April 2009Tomm Aldridge, Principal Engineer

Joint work prepared by: Intel Corporate Technology Group

Emerson Network Power EYP Mission Critical Facilities

Energy Systems Research LabIntel Corporate Technology Group

Intel may have referenced third-party information throughout this

presentation. Intel’s use of third-party copyrighted or trademarked materials

are for reference only. Intel does not warrant the adequacy, accuracy or

completeness of the third-party information nor does Intel claim that it has

obtained a license to the third-party material. Copyright to the third-party

referenced materials remain with their respective holders. LIMITED RIGHTS

2Energy Systems

Research Lab

ASD

PSU

Ballast

Direct 400Vdc Facility Vision

60 Hz AC480V

Electronicloads

Lightingloads

Motorloads

VRDC/DC

AC/DC

DC300-400V

DC/AC

DC/AC

AC/DC

DC/DC

Summary Vision

3Energy Systems

Research Lab

Motivation

Inefficiencies in data center power delivery are significant,

expensive, and ecologically unsustainable

– 50% of power is lost in conversions, transformations, and distribution

– US data centers consumed 61B kWh ($4.5B) in 2006, doubling by 2011

– Worldwide data centers consumed 123B kWh in 2005, and is expected

to increase 40% to 76% by 2010

– By 2011 the energy costs over the life of a server will exceed the

purchase price

– Half of all data centers are already power constrained and unable to

expand

– EPA estimates 10 new power plants will be required by 2011 just to

support US data center growth

– By 2011, powering US data centers will be responsible for ~70M metric

tons of atmospheric CO2 per year

4Energy Systems

Research Lab

Goals for Improved Power Delivery

Higher efficiency

Lower equipment and installation costs

Easy integration with alternate power sources (solar, wind, etc.)

Applicable across entire data center (lighting, cooling, energy storage), not just compute loads

Improved reliability

Smaller footprint for power conversion equipment

5Energy Systems

Research Lab

Power distribution - typical US data center today

Electr.loads

PDU 4W

100W

Total 255W –375W

Server fans 13W

PSU 48W

VR 22W

UPS 18W

Room cooling 50W – 150W

Server

MV

480VAC

480VAC

Rack

BLDGPWR

Server

Electr.loads208V

ACL-L

PDU

PSUVR

Fans

?

UPS

12V

~50% of power lost in power distribution

Motivation for improving power distribution

6Energy Systems

Research Lab

Optimizing Data Center Power Delivery

480V

3 ACF

AC/DC DC/AC

>20% improvement in power delivery efficiency

AC/DC

480V

3 ACF 208V AC400V

DC

208V

3 ACF 400V

DC

400V

DC

400V

DC

DC/DC

400V

DC

Intel and partners working to make optimal power delivery a reality

Baseline 480Vac

Direct 400Vdc

7Energy Systems

Research Lab

Comparing Alternatives

Key Conclusions

400Vdc most efficient

DC solutions simplify power management

– no harmonics or phase balancing

– easy to parallel sources

AC solutions require additional conversions which degrade efficiency and reliability

400Vdc requires only modest changes to existing equipment

All techniques suffer increased losses at light loads

48Vdc also promising, but suffers important drawbacks

– Requires ~100x more copper to distribute power

– Suffers greater conversion loss from utility to building entrance

– Other components less efficient at 48 V (e.g. lighting, adjustable speed drives, HVAC)

50.0 55.0 60.0 65.0 70.0 75.0

Baseline AC

High efficiency AC

400Vac

Specialized AC UPS

Rack-level 48Vdc

Rack-level 400Vdc

Facility 48Vdc

Facility 550V/48Vdc

Facility 400Vdc

Power delivery efficiency [%]

~20%

~5%

Power delivery loss reduction:20%

60%

8Energy Systems

Research Lab

Analysis of 2009 Data Center Design*

~7% Facility energy savings, incl. cooling– 7.7% at 50% load; 6.9% at 80% load

33% Space Savings– No PDUs, simplified switchgear

200% Reliability improvement– 2x lower probability of failure in 5 years

– 1000% reliability improvement if direct connect to batteries

15% Electrical facility capital cost savings– Electrical is ~40% of total facility cost, i.e. saves 15% of 40%

~ 6% of total

Using fewer of the earth’s resources– 15% Component volume reduction in every server power

supply (PFC)

– Some additional reduction in battery cabinets without DC/DC++

Why Direct 400Vdc Data Centers?AC and DC power distribution : Benefits of 400Vdc

*Compared to a modern, high efficiency 480-208 Vac design**++DC/DC assumed in other comparisons.

9Energy Systems

Research Lab

And that’s not all …

Additional benefits to DC distribution– No phase balancing => reduces power strip & wiring complexity

– No synchronization required to parallel multiple sources

– No harmonics => no PFC circuits

– Fewer breakers required because of fewer stages

– Simplifies wiring, since only two wires required

– No need for complex interlocks - simpler procedures, less time

– Only resistive voltage drop in wires

Additional benefits specific to higher voltage DC

– Simplifies wiring, especially at higher power densities

– Lower currents than at 48Vdc, so smaller physical wires

– Use fewer natural resources & less energy to process materials

– ~400Vdc bus in light ballasts and Adjustable Speed Drives (ASDs)

– Simpler/more efficient connection to renewable energy sources– Photovoltaics, fuel cells, wind with variable frequency drives

AC and DC power distribution : Benefits of 400Vdc

10Energy Systems

Research Lab

400Vdc Selection

Why use 400Vdc and not a higher or lower voltage?

– 400Vdc link voltage already exists in power supplies

– Straightforward to extend across data center (UPS, PDU)

– Well within 600V safety limit

– Operates over standard 600V wiring and bussing systems

– Commercial solutions already emerging

400Vdc is a natural choice – efficient, safe, and commercially easy to adopt

11Energy Systems

Research Lab

Two-story, vertical flow-through air design

Designed for ~500 watts per square foot

Centralized air cooling plant

6,000 square feet of raised metal floor (RMF) per module

240 cabinets per module (20 network)

Intel facility for comparisons

12Energy Systems

Research Lab

Modules C - E:

Future Full

Build Out

Module AModule B

Loading

Dock

Facility

Electrical

Room

Module CModule E

Low-Temp

Chiller

Plant

High-Temp

Chiller

Plant

Chilled Water

Storage Tanks

Module D

High-Temp

Chiller

Plant

Expansion

Utility Spine

Facility used for comparisons

3.3MW IT Load

400Vdc proposal for Module C- Intel- Emerson Network Power- EYP Mission Critical Facilities (HP)

5.5MW IT Load

Modules A - B:

Existing

(220 racks x

15 kW/rack)

3.3MW IT Load

Module C (Next phase)(220 racks x 25 kW/rack)

13Energy Systems

Research Lab

Estimated total energy savings

Compare proposed 480V/208Vac & 400Vdc designs for Intel facility**

– For 5.5MW facility

– Iso-redundant 480Vac

– Distributed-redundant 400Vdc

– Assumed 2.15% total cable loss for both

– Non-redundant and redundant PSU cases modeled

– Cooling estimates included in total energy savings

Comparisons : Efficiency

400Vdc is the highest efficiency choice.~7.5% Energy savings compared to 480/208Vac distribution

50

55

60

65

70

75

80

0 10 20 30 40 50 60 70 80 90 100

Load [%]

Po

we

r d

elive

ry e

ffic

ien

cy [%

]

0

125

250

375

500

625

750

To

tal in

pu

t p

ow

er

sa

vin

gs [kW

]

High eff 480VAC

Facility 400VDC

Facility 400Vdc (Red PSU)

High Eff 480Vac (Red PSU)

Total savings [kW]

*Facility savings includes cooling power

** Results will vary for different facilities and for different AC architectures

14Energy Systems

Research Lab

Efficient AC Systems

Specialized UPSs avoid double conversion

• Include Delta conversion and Eco-mode double conversion

End user concerns about line disturbance susceptibility remain

480V/277Vac directly to PSU (server) has been proposed

Remove PDU transformer

Expect power supply to be more efficient than at 208V/230Vac

Efficiency data not yet available

Need new power supply since bus voltage > 430Vdc required

480VAC

Rack

208VACL-L

ServerPSU

12VDC/DC

PDU

VR

MV 400VDCAC/DC

UPS

DC/ACAC/DC

Chrg

AC versus DC distribution

AC Claims it can be only 4-6% less efficient than DC… But at what cost?

15Energy Systems

Research Lab

480VAC

Facility-level 480V/208Vac & 400Vdc

Rack

ServerPSU

12V

PDU

VRMV

DC/DC400VDC

Rect

AC/DC

AC and DC power distribution

Proposed distribution bus of 400Vdc directly connected to batteries

Optionally: Batteries connected through bi-directional DC/DC

– Battery voltage can be optimized since not directly connected to bus

400VDC

Rack

208VAC

ServerPSU

UPS12V380-

410VDC

DC/DCAC/DC DC/AC

PDU

VRAC/DC

Bypass

480VACMV

Battery cabinet(s)

Battery cabinet(s)

Chrg

DC/DC

16Energy Systems

Research Lab

480VAC

Direct connection of batteries to 400Vdc bus

Rack

ServerPSU

12V

VR

DC/DC400VDC

(float)

Rect

AC/DC

Batteries connected directly to the distribution bus

– Increased reliability

– Space and cost savings

This implies a wide input range DC/DC

UPS mostly used only till generators on-line, 30sec - 2min

– Typically size banks for 5 minutes won’t reach discharge voltage

– Prototypes rated at 350Vdc for continuous operation

– Standard DC/DC designed to operate 20msec at 300V, bench test showed capable of running several hours at 320Vdc

400VDC

Battery cabinet(s)

Chrg

Recommended implementation

17Energy Systems

Research Lab

75

80

85

90

95

100

0 0.2 0.4 0.6 0.8 1

Load [pu]

Eff

icie

ncy [

%]

0.9

0.92

0.94

0.96

0.98

1

Pin

ra

tio

dc/a

c

208V ac eff

400V dc eff

400V dc est

Pin ratio

Est Pin ratio75

80

85

90

95

100

0 0.2 0.4 0.6 0.8 1

Load [pu]

Eff

icie

ncy [

%]

0

0.5

1

1.5

2

2.5

Incre

ase

in

eff

icie

ncy [

%]

208V ac eff

400V dc eff

400V dc est

Eff delta

Est eff delta

Energy savings at rack

Rack power reduced with higher efficiency PSU

– 400Vdc rack power ~ 0.98 x 208Vac rack power

400Vdc power supplies are prototypes from Delta Electronics

– Will be able to optimize further, but not significantly so

400Vdc concept : Energy savings

Typicaloperating

range

Typicaloperating

range

18Energy Systems

Research Lab

Module AModule B

Loading

Dock

Facility

Electrical

Room

Module CModule E

Low-Temp

Chiller

Plant

High-Temp

Chiller

Plant

Chilled Water

Storage Tanks

Module D

High-Temp

Chiller

Plant

Expansion

Utility SpineUtility Spine extension

3.3MW IT Load

5.5MW IT Load

3.3MW IT Load

Data Center Facility AC designResults & Comparisons

400 Vdc design can fit in the same space as 3.3MW facility

33% Space Savings: Fits within original Module C footprint

AC design requires 50% more space for 5.5MW than 3.3 MW IT load

19Energy Systems

Research Lab

Reliability Analysis

Relex calculations by EYP Mission Critical Facilities

Results & Comparisons: Reliability

2X lower probability of failure compared to equivalent Tier IV AC facility

Option Availability Unavailability Probability of failure in 5 years

AC Tier IV configuration

0.999996 3.9 e -06 13.63%

DC configuration** 0.999998 2.4 e -06 6.72%

DC Improvement 62.5% 200%

** Reliability numbers for Rectifier from simulation of power train only, will be lower with all components included.Assumes DC/DC converter between Battery & Bus – will be higher reliability in systems without converter.

20Energy Systems

Research Lab

Relative Electrical Cost ComparisonDescription AC Data

Center [pu]

DC Data Center

[pu]

Ratio Remarks

Labor 0.193 0.156 0.8 Due to smaller wiring sizes

Materials 0.215 0.227 1.06 Wire, pipes, Supports, Panel boards, Busways etc

Owner furnished items

0.592 0.467 0.79 Equipment, UPS, static switches, transformers, etc.

Electrical Cost of Work (COW)

1.0 0.85 Total cost to implement all the electrical work including equipment

Electrical cost for dc Data Center estimated 15% lower than for ac

AC DC Remarks

UPS / Rectifier 1.0 0.83 Inclusive of batteries, DC/DC converter, input & output switchgear

480/208V Transformers

1.0 0 Distribution transformers are eliminated in DC Data Center

Server Power Supply

1.0 1.0 Assumed same cost although 6% reduction in BOM cost estimated

Cost comparison for a 3.3 MW load rather than the 5.5 MW load due to available data.Cost estimate for maturity, i.e. high volume.

21Energy Systems

Research Lab

“In typical high-power-density ac/dc power supply 60-65% of volume is taken by

EMI filter, PFC, and bulk capacitors”

Estimate ~15% for PFC stage

PFC

(no longer needed)

BULK CAPS

EMI FILTER

3 kW / 48 V

25 W/in3

1U

DC/DC OUTPUT

STAGE

M. Jovanovic, APEC 2006 keynote

Power Supply Unit Volume Reduction

15% Fewer Components by Volume – Conserving the Earth’s Resources

22Energy Systems

Research Lab

Making Direct 400Vdc a reality?

Need new/modified equipment

– Much equipment already exists for other industries

May need new standards

– International Standards already exist

– Unfamiliarity in the Server Industry can leave standards open to interpretation, so have to engage local authorities

Market acceptance

– Need end users to demand it

– Put it in your Request for Proposals

23Energy Systems

Research Lab

400Vdc Server For servers, only power supply enabling required

Remove PFC stage, re-use existing DC/DC stage

Enabling circuit proposed only for input current > 5A

Modified supply to accept 400Vdc input

90 – 264Vac50/60 Hz

LBOOST

COsBOOST

12V

+

-

EMIFilter(AC)

AC fuse

GND

Chassis

DC/DC

CO

12V

+

-GND

Chassis

EMIFilter

(DC)

EN

400Vdc

DC fuse

DC/DC

Equipment : PSU

24Energy Systems

Research Lab

400Vdc Appliance Coupler

Current PSU prototypes from Delta use Anderson Power Products connector from their PowerPak series

+ 400Vdc

400Vdc return

GND (make first)

Enable (make last, break first)

– Power contacts rated for up to 30A @ 600V (Intel’s version is rated 10 A)

– connectors have UL, CSA and TUV approvals.

– receptacle fits standard chassis opening for IEC320 C22 inlet

For the next step

– Receptacle compatible with the IEC320 C14 chassis opening

– Molded rather than “modular” plug for reduced costs

– Proposed 15A continuous rating

– Expect 5 – 8A disconnect rating

– Prototype under development

– Will include 48Vdc keying option

Equipment : PSU

25Energy Systems

Research Lab

No phase balancing

IEC 309 connector

– Standard for international AC power strips

– Rated for hot disconnect

– Recommend mechanical interlock

– IEC specified for >250VDC

– Not currently UL listed in the US

400Vdc rack power stripEquipment : CDU

26Energy Systems

Research Lab

4000A

1600A

1600A

600AAC UPS

12.47kV

FloorBox

LoadBank

2500A

1000A

12.47kV

DC UPS3000A

FloorBox

Busduct

100A

480Vac3W + Grd 400Vdc

2W

480Vac3W + Grd

280Vac4W + Grd

5 breaker in series

10 breaker in series

Elegance in distribution Expect lower conduction losses : 5 vs. 10 breakers

Comparisons : Efficiency

**It may be possible to eliminate the 2,500A breaker at rectifier input

27Energy Systems

Research Lab

Maintenance

Personnel access equivalent to AC system

Power supply/server replacement

– Appliance couplers will have UL, CSA and TUV approvals

Rack connection and disconnect

– DC rated IEC 309 or Anderson connectors

– Rated for hot disconnect, but propose mechanical interlock

Rectifier servicing same as AC UPS

– Addressed by ability to isolate each rectifier

Maintenance

28Energy Systems

Research Lab

Direct 400Vdc Facility VisionSummary Vision

ASD

PSU

Ballast

60 Hz AC480V

Electronicloads

Lightingloads

Motorloads

VRDC/DC

AC/DC

DC300-400V

DC/AC

DC/AC

AC/DC

DC/DC

Vs. Best AC

7% Energy Savings

15% Less Capital Cost

15% fewer PSU components

33% Datacenter Space Savings

200% Reliability Improvement

Vs. Current

28% Energy Savings

29Energy Systems

Research Lab