Optimizing Cooling: 4 Steps to Quickly Improve PUE Through Airflow Management

Lars Strong P.E.Company Science Officer and Senior Engineer

Upsite Technologies, Inc.

Given that cooling typically consumes around half of a data center's power, it's imperative that cooling is optimized to achieve a low PUE. But how can this be done quickly, with all possible benefits realized, and with the fastest return on investment?

Based on research from 45 data centers around the world, we have identified a series of steps to optimize cooling and lower PUE through proper airflow management.

Optimizing Cooling

Research

Identify Your Opportunity

• There are two types of computer rooms• Those without IT equipment intake air temperature

problems• Improving AFM enables them to be more efficient

• Those with IT equipment intake air temperature problems• Improving AFM solves intake air temperature

problems (hot spots and cold spots) and enables them to be more efficient

• More efficient results in: lower OPEX, deferred CAPEX, increased cooling capacity

Total Load

IT LoadPUE =

Mechanical plant + cooling fans: 35% of total load, 73% of non IT Load

Why Airflow Management (AFM)

On Avg. 3.9 times more capacity than load  Raised

floor area (sq ft)

# of running cooling units

Raised floor

bypass open

area (%)

Hot spots (%

of cabinets

)

Cold spots (%

of cabinets) (Data from 6 sites)

Proper perforated tile

placement (%)

Cooling Capacity Factor (CCF)

Averages 7,527 8 48% 20% 35% 77% 3.9Minimum 720 2 13% 0% 0% 7% 1.2Maximum 37,00

040 93% 86% 86% 100% 32.0

Recommended

n/a n/a <10% 0% 0% 100% 1.2Conclusion: Inefficient cooling configuration / airflow management is the problem; capacity is not the problem.

Research

7

Based on a research study conducted by Upsite Technologies of 45 data centers worldwide, we found that the average data center has nearly 4 times more rated cooling capacity than IT load.

Only 4 sites had CCF of 1.3 or less.

The average data center has a cooling capacity nearly 4 times the IT load!

Research

IT Intake Air Temperatures are Paramount

• Airflow is invisible• Cooling capacity is difficult to quantify (ton ???)• Mixed messages in the market• Vendor influence• Lack of education

Why Are Myths and Over Capacity Prevalent

• Money

• OpEx Savings• CapEx Savings

• Capacity• Cooling Capacity

• IT Reliability

• Green Initiatives• CO2 Reduction

“…an average a 10,000 sq ft data center could reduce its operating expense by $42,500 annually, simply by improving airflow management.” - Upsite CCF White Paper

AFM Problems Are An Opportunity

Bypass Airflow Clarified

Any cooling air that does not pass through IT equipment before returning to a cooling unit

Bypass flow rate (CFM) = total cooling supply flow rate (CFM) – total IT equipment flow rate (CFM)

Definition of Bypass Airflow

CFM =

CFM = cubic feet per minute of airflow through the server

3.16 = factor for density of air at sea level in relation to F⁰

ΔT = temperature rise of air through the server in F⁰

3.16 x W ΔT

ΔT Through IT Equipment

Heat Transfer Equation

• “Pizza box” servers• at 20 ⁰F ΔT consume 158 CFM / kW• 500 kW - 1,000 servers• Would require 79,000 CFM of chilled air

• “Blade” servers • at 35 ⁰F ΔT consume 90 CFM / kW• 500kW - 1,600 servers (100 chassis)• Would require 45,140 CFM of chilled air• 43% less than servers with 20 ⁰F ΔT

ΔT Through IT Equipment

UPS load (kW) x avg CFM/kW = Total IT CFM

IT Equipment Required Flow Rate

 IT Equipment Delta T (deg F)

15 20 25 30 35 40Required flow rate (CFM/kW

)

211

158

126

105

90

79

ΔT Through IT Equipment

Typical existing conditions

Bypass Airflow Clarified

After AFM improvements

Bypass Airflow Clarified

Optimized

Bypass Airflow Clarified

Bypass Airflow Revealed

Upsite’s 4 R’s of Airflow Management™

Upsite's 4 R's of Airflow Management™ provide a guide for implementing changes and achieving the greatest benefits in your data center.

Optimizing Airflow

• 3,900 sq. ft.• 81 cabinets (avg. 4 kW / cabinet)• UPS load 332 kW• 10 – 25 ton cooling units• Rated cooling capacity

• 250 ton• 880 kW

CFD Modeling

Raised Floor Airflow Management

• Raised floor open area management

• 2 types (good and bad)

1. Good• Supply tiles in cold aisle, in front of IT equipment

2. Bad• Supply tiles in open areas and hot aisles• unsealed cable openings• PDU and perimeter openings

• It is easier to seal a cable opening before cabinets and IT equipment are in place

Raised Floor AFM Best Practices

• Seal all cable and data cutouts• Seal all conduit penetrations• Seal or reduce open area under PDUs• Seal perimeter penetrations (fair rated?)• Remove unused cables• Use cable trays• Manage type and location

of every supply tile

Raised Floor AFM Best Practices

• CFD Result

• By sealing cable openings flow through supply tiles increased by 30%

• Allowed a 10% reduction in fan speeds

• Data center manager reported $80K savings as a result of just relocating and removing perforated tiles.

Raised Floor AFM Best Practices

Rack Airflow Management

• Blanking panels in 100% of open space• Seal between rails and sides of rack• Seal under the rack• Few sites have done this fully • Cabinet design plays a large role in AFM

• Benefits from IT and facilities cooperation

Rack AFM Best Practices

Rack AFM Best Practices

Hot air is flowing from the hot aisle under the racks and into the cold aisle. The hot air coming from under the racks prevents the conditioned air supplied by the tiles in the cold aisle from reaching the equipment in the bottoms of the racks

Rack AFM Best Practices

Sealing the space under racks prevents hot air from flowing into the cold aisle. The conditioned air supplied by the tiles in the cold aisle can fill the space and support the IT equipment from the bottom to the top of the rack

Rack AFM Best Practices

• CFD results

• By sealing racks the maximum IT intake air temperature dropped 14.6 F. Avg maximums ⁰dropped 6.6 F⁰

• Allowing a 15% reduction in fan speeds• Allowing a 3 F increase in supply temperatures⁰

Rack AFM Best Practices

Row Airflow Management

• Seal spaces between cabinets• Seal spaces where cabinets are missing• Aisle end doors• Block flow over cabinets

• Baffles• Full containment

Row AFM Best Practices

• CFD results

• By installing partial containment (aisle end doors and rack top baffles) the maximum IT intake air temperature dropped 5.5 F. Avg maximums dropped ⁰2.1 F⁰

• Allowing a 10% reduction in fan speeds• Allowing a 7 F increase in supply temperatures⁰

Row AFM Best Practices

Room / Controls Changes

• It is still common to see 72° F cooling unit return set today, and lower set points in the range of 68° F are still not uncommon

• Often driven by IT conservatism or hot spot mitigation

Raise Cooling Unit Set Points

1 Through IT equipment2 IT equipment exhaust to cooling unit3 Through cooling unit4 Cooling unit to IT intake

The 4 Δ T’s

Raise Cooling Unit Set Points

Common situation

Raise Cooling Unit Set Points

Ideal Situation

Improve Efficiency / Release Stranded Capacity

Liebert DX Model VH267W 20-Ton Cooling Unit

Environmental Condition

Total(kW)

Latent(kW)

Sensible(kW)

72°F, 45% Rh 67.0 0 67.070°F, 48% Rh 64.5 4.8 59.7Stranded 2.5 4.8 7.3Stranded (%) 4% 11%

Cooling capacity increase from 2°F return air increase

Raise Cooling Unit Set Points

Cooling capacity increase from 7°F return air increase

Environmental Condition

Total(kW)

Latent(kW)

Sensible(kW)

72°F, 45% Rh 98.0 0 98.065°F, 45% Rh 70.0 0 70.0Stranded 28.0 0 28.0Stranded (%) 40% 40%

Improve Efficiency / Release Stranded CapacityLiebert Chilled Water Model 600C CRAH Unit

Raise Cooling Unit Set Points

• At 50% fan speed• 50% airflow volume• Only 12.5% of fan energy (87.5% savings)

• CFD Modeling • With the raised floor, racks, and rows sealed one

cooling unit was able to be turned off and fan speeds were reduced to 65%

• Annual fan energy savings $36,898

Raise Cooling Unit Fan Speeds

• Maximum IT inlet temperature reduced by 8.4°F• Supply temperature increased 10 °F• Fan speeds reduced 35% and one cooling unit

turned off• Total annual savings $58,163• ROI 1.4 yrs• Partial PUE (pPUE) reduced from 1.54 to 1.34

(chiller plant and cooling unit fans)

Final Results

Conclusions

1. Cooling optimization is about improving data center airflow so the least amount of conditioned air at the highest supply temperature can be used to effectively cool IT equipment

2. AFM fundamentals of sealing Raised floor and Rack, and managing airflow at Row level are essential to effective and efficient cooling

3. Changes need to be made at the room level, to cooling controls, to reduce operating cost and recover stranded capacity

3 Key Takeaways

Thank you!

Lars Strong, P.E.Company Science Officer Senior Engineer, Upsite Technologieslstrong@upsite.com

Follow Upsite for the latest news and information on data center AFM.

@UpsiteTech blog.upsite.com

On LinkedIn