1

High Heat Loads :Advances in Water Cooling

&

Fire Suppression

Sept, 2004

Tony Sharp M.Eng P.Eng

New Product Design

Enclosure Systems Division

Toronto

SANMINA-SCI Global Presence

North America

Latin America

Europe and Middle East

Asia Pacific

More than 100 manufacturing locations worldwide

The labels seen on our production

SIEMENS

NETWORK SYSTEMS

MOTOROLA

EMCEMC22

The Data Center Cooling Challenge

Source: Zerodowntime Magazine

The rise of heat densities…at the microprocessor level

Data Center Heat Loads

The rise of heat densities…everywhere !

W/ft2

The Data Center Risks

• Interruption of Data Stream

• Temporary Shut-down of Servers

• Damage to Servers, Electrical Supply, or Data cabling

• Damage to Infrastructure or Thermal Management

• Incapacity / unavailability of Operating Staff

7 x 24 Continuous No-break Operation Required

Data Center Stresses

• Temperature High Temp. Temp Cycling, Fire, Smoke.

• Voltage Surges & Spikes. Power quality.

• Vibration Local Source. Seismic.

• Pollutants Water, Conductive particles,Corrosion / Oxidation, Humidity, Fungi, Insects

• Natural Catastrophe Earthquake, Hurricane

• Security Commercial Espionage, Terrorism

CONSEQUENCES OF A ‘THERMAL EVENT’

SourceIT Performance Engineering & Measurement Strategies: Quantifying Performance Loss, Meta Group, October 2000

Risk Management

Business Continuity ?

The Cooling Choices

Air Cooling• Raised Floor as supply duct for cold air• Spot cooling, CRACs• Tailored ductwork

Water Cooling• Central Chiller System• Heat flux through small piping

Mal-Distribution of Plenum Air in Hot/Cold Aisle layout

Slide Courtesy of Dr. Roger Schmidt, PhD.,P.E.,Chief Thermal Architect,Distinguished EngineerIBM

The Ecological Solution ?

• Cool the Cabinet – NOT the Room• Increase Energy Efficiency

Result :

• Cost SavingsCapital Cost ReductionSpace SavingsOperating Cost Reduction

• Increased Efficiency

ECOBAY Thermal Management System

Rear View221 System

Cooled Air Heated Air

Heat ExchangerFront View 221 System

High Performance Closed Loop CoolingHeat removal of 20+ kW per system

◄Warmed Water to ChillerChilled Water from Chiller►

T H E P R E M I E R E M S C O M P A N Y

ECOBAY 221 GRSCFD Airflow Map

(Click Image)

Closed-Loop Water-Cooled Cabinet

Sealed Cabinet improves Reliability of Data Service1. Cooler environment. No “hot spots” in Data Center.2. No contaminants; dust or metallic particles.3. Humidity excluded.4. Better security

Other Advantages1. Lower energy cost. [ Warmer room ]2. Higher equipment density.[ Lower building cost ]3. Reduced noise.4. Better workplace. [ No gales of cold air thru floor ]5. Environmental. Saves energy, re-circulates water.

ECOBAY 118 High Density Single Cabinet

IN TEST LAB

Front View Door Open Rear View Door Open

ECOBAY 118 – Rear Door View

Variable Speed DC Fans,

Temp Controlled

Auto Door Release

in case of Critical Cooling

Failure

ECOBAY 118: Front Door View

Air Metering Plenum on Front Door,

No“Hot Spots” 19” Rack

Mounted Heat

Exchanger

18 kWOne Cabinet

EcoBay 118 Performance

Factory Tests with Dummy Servers

EcoBay 118 H Performance TableSI units Max Min Imperial units Max Min

Inputs Heat Load 20.0 kW 20.0 kWWater Temp 14.0 degC 17 9 57.2 degF 63 48Water Flow 0.76 l/sec 0.76 0.38 12.0 USgpm 12 6

Results Cold Air Supply 25.1 degC 77.1 degFHot Air Disch 47.3 degC 117.1 degF

Air DT 22.2 degC 40.0 degF

Water T. Rise 6.7 degC 12.0 degFWater Disch 20.7 degC 69.2 degF

EcoBay 118 Performance

Field Tests with Blade ServersFive Blade Chassis operating at full power = ~ 18 kW

0.76 l/s ( 12 gpm ) Water at 14 C ( 57 F )

Blades operated within manufacturers specs.

RH 40% - 60% inside cabinet

Acoustic performance – “very good”

Closed-Loop Water-Cooled Cabinet

How do we make the Cabinet Cooling System Reliable ?

• Redundant Auto stand-by Chillers & Pumps.• Matrix piping system.• Redundant Fans.• Redundant power supplies. • Pre-empt problems with a Monitoring & Alarm system.

Advance warning if abnormal conditions occur.

Fall Back Strategy

If the cooling in a cabinet fails for any reason :-• Automatic door release. Room air temporarily removes

heat. [ Door release has dual power & dual thermostats ]

ECOBAY Value Proposition

Processing capacity per ft2

= 324kW/2170 ft2

= 149 W per ft2

Processing capacity per ft2 :

= 324kW/910 ft2

= 356 W per ft2

Costs $898.09/kW to constructCosts $573.89 per kW to construct

and saves 1260 ft2 of infrastructure

Processing Power : 324kW

Total Cost : $290,981.68

Processing Power : 324kW

Total Cost : $185,947.01

AIR COOLEDWATER COOLED

The Water Cooling “Value Proposition”.Capital Cost

$0.00

$100,000.00

$200,000.00

$300,000.00

$400,000.00

1

ECOBAY 118 vs Traditional Room

Capital Cost Reduced 46%

kW Per Square Foot

0

100

200

300

400

1 2

ECOBAY 118 vs Trad. Room

Square Footage

0

500

1000

1500

2000

2500

1 2

ECOBAY 118 vs Conventional Data Center

Square Footage Reduced 58% Power Density Increased 2.4x

The Data Center Fire Challenge

The Fire Risks

• Service Interruption• Asset damage – Heat, Smoke, Water• People – Incapacity, Injury, Death

Risk Management

…fire risks?

80% of extended business down time is through fire or effects offire95% of prolonged business down time is through fire

…smoke risks?

95% of the fire damage in a high risk facility is from combustion products45% of fires occur in IT equipment and areas deemed too small toactivate automatic detection equipment or have no detector present (1994-1998)60% of fires in telephone exchanges caused smoke damage beyond origin (1989-1998)

The Ecological Solution

• Containment – No escape of fire or smoke• Cabinet Level Suppression – Not an

“Occupied Space” per NFPA• Fire suppressant - Benign to environment and

safe to humans• Huge cost saving for cabinet discharge vs

room flooding

Fire Suppression in the Cabinet

Sealed Cabinet System improves Reliability of Data Service1. Early warning – before serious problem occurs.2. Fast Action – reduces intensity of damage.3. Localized – collateral damage limited (smoke &

heat contained within cabinet ).4. Benign – Non-toxic suppressant does no harm.

Other Advantages1. Huge Cost Saving ( discharge cost :- cabinet

volume vs. room volume )

Supplemental Fire Detection and Suppression

Regulatory Controls…don’t address your needs

Aimed at:Life safety of building occupantsLife safety of Fire Brigade personnelPreventing the spread of fire from building to building

NOT aimed at:Protection of building contents such as servers and cabinetsProtection of business continuityProtection of the data, services and maintenance of Service Level ObligationProtection of reputation

Business Continuity: Protecting assets in a High Density Computing Environment

This…… ?

FM 200 ManifoldGas Storage for

The Data Room DumpEn masse.

Room Flooding & DowntimeCopious $$

All “After the Thermal

Event!!

Or THIS!!Fire Detection and Suppression in the Cabinet

Quickly suppresses fire within cabinetEffects contained within one cabinet Either Laser based VESDA Detection/Activation or:Dual Smoke Detection (Ion sensing for early

warning of potential problems. Photo sensing confirms smoke )

5lbs 3M Novec 1230: Clean, Environmental, Non-toxic Agent

Automatic & Manual discharge mechanismsIntegrated panel (Annunciator, solenoid release,local & remote alarms)Optional back-up cylinder

Compartment Gas StorageDetection at the Source

Suppression at the Thermal Event!!

Very Early Smoke Detection

• Very Early detection can pre-empt fire and smoke damage

• Detection of “sparkles” of scattered light due to minute particles of smoke in a laser beam makes it possible to sense the precursors of fire at a very early stage – early enough to suppress the incipient fire and prevent damage.

• The technique is so sensitive that several stages of early warning are available. This enables early intervention to prevent the “thermal incident” from reaching a significant risk level.

Smoke Detection Choices

• Conventional Photoelectric • “Ion” detectors• “Laser” detectors

VESDA smoke detection

Smoke Detection …the value of early warning

Advanced Technology

• ECOBAY enables in-cabinet Monitoring, Smoke/Fire Detection and Fire Suppression.

• Several common Fire Suppression agents have been banned due to toxicity, or concerns over Ozone depletion.

• This has created an opportunity to displace costly, in-room systems with safe, efficient and cost effective in-cabinet systems which use newly developed non toxic, zero ozone depleting Fire Suppressants.

ECOBAY: New Technology Advancement

Novec™ 1230 Fluid Safety Margin

4% - 6% 67% - 150%

• Sanmina, 3M, and VISION Systems have developed a new in-cabinet Fire Suppression System using Novec 1230™, non toxic, zero ozone depleting Fire Suppressant recently developed by 3M.

• Testing of System completed at 3M test facility in St. Paul Minnesota.

• Heptane fire was extinguished in less than 2 seconds. Previous system required 9 seconds to extinguish fire.

The Choice of Fire Suppressant

The Candidates:• FM200 – HFC-227ea

HeptafluoropropaneCF3CHFCF3Boiling Point -16°C ( 3°F ) – Gas @ room temp

• Novec 1230 –FluoroketoneCF3CF2COCF(CF3)2Boiling Point 49°C ( 121°F ) – Liquid @ room temp

Fluorochemicals & the environment

EnvironmentalProperties

Atm. Lifetime +IR Absorbance

Climate Change

Cl or Br + O3

Ozone Depletion

Environmental Footprint Comparison

Halogenated Compounds

0 2 4 6 8 10 12 14

Ozone Depletion Potential

halon1301

HFC-227eaFM200

Glo

bal W

arm

ing

Pote

ntia

l

Halon1211

10000

9000

8000

7000

6000

5000

4000

3000

2000

1000

0

HFC-236faFE36

Area of the circle is proportional to the atmospheric lifetime of the compound.

0 2 4 6 8 10 12 14

Ozone Depletion Potential

halon1301

HFC-227eaFM200

Halon1211

Glo

bal W

arm

ing

Pote

ntia

l

Novec 1230

halon1301

Halon1211

10000

9000

8000

7000

6000

5000

4000

3000

2000

1000

0

HFC-236faFE36

0

0

Environmental Footprint Comparison

Area of the circle is proportional to the atmospheric lifetime of the compound.

Global Warming Potential of Various Compounds

1C2F5C(O)CF(CF3)2

22,200SF6

8,600 - 9,000C3F8, C4F10, C6F14

11,900C2F6

12,000CF3H (HFC-23)

3,500CF3CFHCF3 (HFC-227ea)

1,300CF3CF2H (HFC-134a)

23CH4

296N2O

1CO2

GWP (100 Yr ITH)Compound

Naturally occurring

compounds

HFCsFM200 -

PFCs

Novec™ 1230

Greenhouse Gas Emissions Reductions from replacing HFC Agent with Novec 1230 fluid

HFC227ea

HFC227eaHFC

227eaHFC227ea

=One discharge

from an average size system(approx. 600 lbs of agent)

CO2 savings equal to the annual emissions from 200 U.S. passenger cars

Toxicological Results on Novec™ 1230 fluid

Acute Inhalation Practically Non-Toxic (LC50 >100,000 ppm)

Cardiac Sensitization Not a Sensitizer (NOAEL =100,000 ppm)

Primary Eye Irritation Minimally IrritatingPrimary Skin Irritation Non-IrritatingAcute Oral Toxicity Low Toxicity (LD50 >2000 mg/kg)Acute Dermal Toxicity Low Toxicity (LD50 >2000 mg/kg)Skin Sensitization Not a Skin SensitizerAmes NegativeChromosomal Aberration Negative28-Day Inhalation Study NOAEL of this study:

4,000ppm

Safety Margin for Flooding Agents

3 - 20%9%7.5 - 8.7%HFC-227ea (FM200)

NIL5%5%halon 1301

67 - 150%10%4 - 6%Novec™ 1230

Safety MarginNOAEL*Use Conc.Agent

* No Observable Adverse Effect Level

Test Results

FM200 Novec 1230Quantity 4.4 lbs 7.1 lbs ( 50% margin @ 6.3% conc.)Fans ON ONFuel Heptane Heptane 3 x 3 kW UL2166

Pre-burn 30 sec 30 secDischarge 10 sec 8 secFire out 9 sec 2 sec High cup

4 sec Low cup9 sec Mid cup

VESDA obscuration13 sec Alert 0.01230 sec Alarm 0.03

Fire Suppression in Cabinet

How do we make the Cabinet Fire Suppression System Reliable ?

1. Back-up gas cylinders.2. Dual sensing. No false alarms or discharges,

but early detection. 3. Dual power for redundancy.

What the Water Cooled cabinet offers

RELIABILITYSolutions for Customer Needs:– Improved cooling– Max data speed– Dense packing– Lower energy cost– Sealed system excludes

pollutants – Better security– Smoke/Fire suppression– Improved work area– A future for Blade

Servers

COST REDUCTION

PERFORMANCE

SECURITY

47

Thank You