high density data centers 5'22'03 -...
TRANSCRIPT
High Density Data CentersFraught with Peril
Richard A. Greco, Principal
EYP Mission Critical Facilities®, Inc.
2
Microprocessors Trends
Reprinted with the permission of The Uptime Institute from a white paper titled Heat Density Trends in Data Processing,
computer Systems and Telecommunications Equipment Version 1.1
3
Power Consumptions will Continue to Increase
• More transistors and higher clock speeds will result in more power consumption which in turn lead to greater production of heat.
• Projections of power consumption increases by 2X every four years.
4
Power Consumptions
• The fact is that the majority of servers available today have high power consumptions.
3U 2U
Albatros Server DS/5U 1.3 kW
1U 3U BladeTotal Load (kW) 3.0 - 7.3 3.2 - 5.5 3.2 - 12.6 7.9 - 12.0 6.0 - 14.0
Technology Impact on 42U Enclosure
5U
14 - 28Number of Outlets 8 - 16 14 - 28 21 - 42 42
- - - -
- - .4 kW -
IBM X Series 360 .37 kW
Sun Fire B1600 - -
Compaq Proliant DL380 - -
IBM X Series 300 -
- - 1.0 kW
- - - -
.4 kW - -
5
Cooling Microprocessors
• Within three to five years, researchers at Intel Corp., Hewlett-Packard Co. and IBM predict computer makers will have to move beyond fans and adopt cooling mechanisms such as radiators and liquid cooling systems to avoid potential overheating.
6
How Will These Trends Impact the Data Center?
• High density data centers– Faster chips, more heat, more boxes
• Low to medium density data centers with hot spots– Faster chips, more heat, but fewer total
number of boxes
• Unknown effect
7
What is a High Density Data Center?
“A high compute density data center of today is characterized as one consisting of thousands of racks with multiple computing units…The heat dissipation from a rack containing such computing units exceeds 10kW.”
Source: Thermal Considerations in Cooling Large Scale High ComputeDensity Data Center, Patel et al.
8
What EYP MCF Believes is the Practical Limit to a High Density Data Center
• A high density data center is one in which the power to the raised floor and the computing equipment exceeds 150 watts/sf over the entire raised floor. This is equivalent to approx. 3.5 kW/cabinet in an efficiently arranged data center.
• This is the point where traditional all air data centers begin to have significant limitations.
10
Example of a Traditional All Air Data Center at 150 w/sf
• 15,000 sf, 620 24x36 cabinets, 3.5 kW/cabinet, 2170 kW cooling required
• Cabinets on hot aisle/cold aisle – 16 ft aisle to aisle
• 23 or 24 CRACs (lay-out dependent) total at 110 kW each (375.3 MBH), 3 redundant
• 620 perforated tiles at 530 CFM eachmin. req.
• (24) 200 kW PDU’s (2N configuration)
11
Two Approaches to Cooling the Data Center
• We use Computational Fluid Dynamics (CFD) to determine the preferred arrangement– Option 1: CRACs on opposite walls – Option 2: CRACs in middle and on one wall
12
Option 1: CRACs On Opposite Walls
• 150 w/sf• Typical 15,000 sf
data center• 620 cabinets
@ 3.5 kW/cabinet• (24) 110.2 kW
CRAC units • (24) 225
KVA PDUs
13
Front In, Back Out, Hot Aisle / Cold Aisle Configuration for Cabinet
• Air is drawn through the front of the cabinet (blue) and discharged through the back (red)
Cold Hot ColdAisle Aisle Aisle
Supply Air is 55º F Return Air is 85º F
16
Option 2: CRACs in Middle and On One Wall
• 150 w/sf• Typical 15,000 sf
data center• 620 cabinets
@ 3.5 kW/cabinet• (23) 110.2 kW
CRAC units • (24) 225
KVA PDUs
19
High Density Data Center – 240 w/sf
• Typical 15,000 sf100’ by 150’
• 462 cabinets @ 7.9 kW/cabinet 3650 kW cooling
• 462 air grates @ 1200 cfm ea
• (40) 110.2 kW CRAC units
• (32) 225 KVA PDUs (2N)
22
To Overcome Airflow Variations Use Fan Powered Cabinets
• Ensures airflow to cabinet regardless of pressure in floor and airflow through air grates, however re-circulation will occur if sufficient air from under floor is not delivered.
• Cabinet fans supplement server fans
23
Fan Powered Cabinets: Front In, Back Out, Footprint Vented, Fan Assisted
• Air is drawn through the front of the cabinet (blue) and discharged through the back (red)
Supply Air is 55º F
Discharge Air is 85º F Raised Floor
Cabinet Fan
24
Fan Powered Cabinets: Front In, Back Out, Fan Assisted
• Air is drawn through the front of the cabinet (blue) and discharged through the back (red)
Supply Air is 55º F
Discharge Air is 85º F Raised Floor
Cabinet Fans
25
• Air is drawn through the front of the cabinet (blue) and discharged through duct work (red)
Fan Powered Cabinets: Front In, Top Out, Fan Assisted
Supply Air is 55º F
Raised Floor
Cabinet Fans
Hot Air Plenum
Discharge Air is 85º F
Return Air Ceiling Plenum is 85º F Ceiling
26
Fan Powered Cabinets: Front In, Top Out, Footprint Vented, Fan Assisted
• Air is drawn through the bottom of the cabinet (blue) and discharged through the top (red)
Raised Floor
Supply Air is 55º F
Cabinet FansDischarge Airis 85º F
27
Raised Floor
Supply Air is 55º F
Discharge Airis 85º F Ceiling
Cabinet Fans
Fan Powered Cabinets: Front In, Top Out, Footprint Vented, Forced Ventilated• Air is drawn through the bottom of the cabinet
(blue) and discharged through duct work (red)
28
High Density Data Center without CRACs
• 325 w/sf • Typical 11,200 sf
high density data center
• 462 cabinets @ 7.9 kW/cabinet
• 462 cabinet coolers • (32) 225
kVA PDUs
29
Water Cooled Cabinet: Front In, Back Out
• Air is drawn through the front of the cabinet (blue) and discharged through the back (blue)
Supply Return Piping
Raised Floor
Cooling Coil
Cabinet Fans
Discharge Air is 72º F
30
Supply Return Piping
Raised Floor
Water Cooled Cabinet: Footprint Vented
Cool Air
Cabinet Front
Fan and CoilLocation underCabinet
CabinetRear
Building Floor
31
• Air is drawn through the front of the cabinet (blue) and discharged through the back (red)
Refrigerant Cooled Cabinet
Raised Floor
Heat Exchanger
Refrigerant Pump
Discharge Air is 85º F
45º F Water from Chilled Water Plant
60º F
32
• Air is drawn through the front of the cabinet (blue) and discharged through the back (red)
Refrigerant Cooled Cabinet: Front In, Back Out
Raised FloorSupply Air is 55º F
Piping to Heat Exchanger
Discharge Air is 85º FOverhead Fan Coil
Supply Air
33
Inherent Problems with High Densities
• Air Distribution Problems– Inadequate raised floor height and excessive
openings in perforated tiles can result in high velocity airflows.
– Bypassing of air within cabinets– Re-circulation of air outside of cabinets
• Today users report a greater number of server failures in top 1/3 of cabinets
34
Inherent Problems with High Densities
• Cabinet fans are generally single power supply.
• Cabinet fan power should be from a separate PDU with redundancy.
• Identical cabinets may be required.
• UPS system must be larger or separated to incorporate fan power (up to 350 w/cabinet). For our example of 462 cabinets – 162kW of fan power.
35
Inherent Problems with High Densities
• What about the rate-of-change at the microprocessor level during transitions from utility to generators. To my knowledge this has not be addressed by the industry.
• Is uninterruptible cooling mandatory?– If so, what are the implications?
• Costs• Reliability• Redundancy• Maintainability
36
Other Considerations
• Numerous additional monitoring points.• Addition of cooling liquids in data center
environment.• Fire Zones.• Noise.• Seismic. • Computational Fluid Dynamic modeling is
required for design and recalculation when hardware changes.
37
Other Considerations
• It will be mandatory to have close coordination between facilities department and IT hardware planners.– Does not exist today
• Generally higher risks with lower margins for errors.
38
Prognostications
• “…late 1990’s requirements of 100 watts per square foot of average power now exceed 200 watts per square foot, and will double again by 2004.”
META Report - August 2, 2001 by Rich Evans
39
Prognostications
• “In fact, the 2004 400-watts-per-square-foot requirement will again double by 2008, so designers must work this into longer-range facilities plans so that space can be fully used, as opposed to 40 percent to 50 percent utilization because of insufficient power and cooling.”
META Report – August 2, 2001 by Rich Evans
40
Prognostications
• “… Data centers utilize much less than 50% of the physical and power infrastructure.”
Source: Cost of Overbuilding by Domenic Alcaro
41
Prognostications
• “Current estimates of data center power requirements are greatly overstated because they are based on criteria that incorporate oversized, redundant systems, and several safety factors.”
Source: Data Center Power Requirements: Measurements from Silicon Valley,
Mitchell-Jackson, Koomey, Nordman, Blazek
42
Prognostications
• “Furthermore most estimates assume data centers are filled to capacity. For the most part, these numbers are unsubstantiated.”
Source: Data Center Power Requirements: Measurements from Silicon Valley,
Mitchell-Jackson, Koomey, Nordman, Blazek
43
So What’s Real?
• Today’s data centers incorporate significant amounts of legacy equipment, slow change out due to economy.
• Many data centers have significant amounts of available space on the raised floor.
44
Current Load Densities
DATA CENTER TYPE WATTS/SF KW/CABINET
Enterprise 20 to 60 0.5 to 1.5
Internet/Co-location 40 to 90 1.0 to 2.0
Managed Services 60 to 110 1.5 to 2.5
High Density 150+ 3.5+
45
My View
• Most projections of watts/square foot are overstated.
• Millions of dollars will be spent on unused infrastructure.
• There no longer are real drivers for compaction.
• Generally there is ample raised floor available.
46
My View
• High density data centers will not become a reality.
• Low/medium density data centers will have hot spots due to high density computing equipment.
47
My View of What to Do
• Manage your hardware installations
• Don’t buy IT gear that is problematic
• Don’t build yourself a problem– Existing air delivery systems work– Highly engineered systems are expensive to
build and operate and significantly reduce reliability, redundancy and flexibility.