The CDCE Project @ BNLThe CDCE Project @ BNL

HEPIX – LBLHEPIX – LBL

October 28, 2009October 28, 2009

Tony Chan - BNLTony Chan - BNL

BackgroundBackground

Rapid growth in the last few years caused space, power and Rapid growth in the last few years caused space, power and cooling problemscooling problems

Increasing capacity for RHIC/ATLAS and other activities Increasing capacity for RHIC/ATLAS and other activities cannot be accommodated with current facility cannot be accommodated with current facility infrastructureinfrastructure

Search for additional data center space began in 2007Search for additional data center space began in 2007

Update of talk originally given at HEPIX in St. Louis (Nov. Update of talk originally given at HEPIX in St. Louis (Nov. 2007)2007)

Vital StatisticsVital Statistics

Currently housing 165 racks of equipment (disk storage, Currently housing 165 racks of equipment (disk storage, cpu, network, etc) + 9 robotic siloscpu, network, etc) + 9 robotic silos

Approximately 35 PB of tape storage, 9 PB of disk storage Approximately 35 PB of tape storage, 9 PB of disk storage capacity and 10,200 computing corescapacity and 10,200 computing cores

Average power usage ~ 650 kW (~60% of maximum UPS Average power usage ~ 650 kW (~60% of maximum UPS capacity) with peak load ~ 790 kWcapacity) with peak load ~ 790 kW

Cooling capacity for a maximum of ~1000 kW Cooling capacity for a maximum of ~1000 kW

The Growth of ComputingThe Growth of Computing

0

10000

20000

30000

40000

50000

60000

70000

KSpecInt2000

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

(es

t.)

2011

(es

t.)

2012

(es

t.)

Total Distributed Storage Capacity Total Distributed Storage Capacity

0

5000

10000

15000

20000

25000

30000

35000

Sto

rag

e C

apac

ity

(TB

)

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

(es

t.)

2011

(es

t.)

2012

(es

t.)

Evolution of Space Usage Evolution of Space Usage

0

2000

4000

6000

8000

10000

12000

14000

Us

ag

e o

f D

ata

Ce

nte

r (s

q.

ft.)

1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010(est.)

2011(est.)

2012(est.)

Capacity of old data center

Intel dual and quad-core deployed

Evolution of Power Usage Evolution of Power Usage

0

200

400

600

800

1000

1200

1400

1600

1800

2000

Po

we

r U

sa

ge

(k

W)

2003 2004 2005 2006 2007 2008 2009 2010(est.)

2011(est.)

2012(est.)

Existing UPS Capacity

The Search For Solutions (1)The Search For Solutions (1)

Engaged Lab management – Spring 2007Engaged Lab management – Spring 2007

Discussion on possible solutions – Summer 2007Discussion on possible solutions – Summer 2007• CostCost• TimeTime• LocationLocation

Recommendation to Lab management – Fall 2007/Winter Recommendation to Lab management – Fall 2007/Winter 20082008• Two-phase solution to meet cost and time constraintsTwo-phase solution to meet cost and time constraints• Identify and renovate existing space to meet short-term requirementsIdentify and renovate existing space to meet short-term requirements• New building to meet long-term requirementsNew building to meet long-term requirements

Funding for two construction projects approved – Spring Funding for two construction projects approved – Spring 20082008

The Search for Solutions (2) The Search for Solutions (2)

Renovate existing floor space (US $0.6 million)Renovate existing floor space (US $0.6 million)• Tender award – April 2008Tender award – April 2008• Renovations begin – June 2008Renovations begin – June 2008• Renovations end -- October 2008Renovations end -- October 2008• Occupancy – November 2008Occupancy – November 2008

New building (US $5 million)New building (US $5 million)• Finalize design – May 2008Finalize design – May 2008• Tender award – June 2008Tender award – June 2008• Construction starts – August 2008Construction starts – August 2008• Construction ends – August 2009Construction ends – August 2009• Occupancy – October 2009Occupancy – October 2009

From first proposal to occupancy took 2½ yearsFrom first proposal to occupancy took 2½ years

Facility Development TimelineFacility Development Timeline

Recent Past Recent Past (2006-2007)(2006-2007)• More efficient use of facility resourcesMore efficient use of facility resources• Supplemental cooling system in existing facilitySupplemental cooling system in existing facility

Near-Term Near-Term (2008 to present)(2008 to present)• Renovation of (2000 ftRenovation of (2000 ft22) 185 m) 185 m22 of unused floor with 300 kW of of unused floor with 300 kW of

powerpower• New building with (6600 ftNew building with (6600 ft22) 622 m) 622 m22 and 1.0 MW of power and 1.0 MW of power• Mission-specific facility (redundant cooling, deep raised floors, Mission-specific facility (redundant cooling, deep raised floors,

etc)etc)• Room for ~ 150 racks and 7 robotic silosRoom for ~ 150 racks and 7 robotic silos

Long-Term Long-Term (2017 and beyond)(2017 and beyond)• New BNL data center with 25000 ftNew BNL data center with 25000 ft22 (2300 m (2300 m22) after 2018) after 2018

Rack-Top Cooling UnitsRack-Top Cooling Units

Rear Door Heat ExchangerRear Door Heat Exchanger

Data Center Layout in 2007Data Center Layout in 2007

Data Center Layout in 2009Data Center Layout in 2009

DATA CENTER EXPANSION – PART 1

September 12, 2008September 12, 2008

Data Center Expansion – Part 1Data Center Expansion – Part 1

July 15, 2009July 15, 2009

Data Center Expansion – Part 2Data Center Expansion – Part 2

January 14, 2009January 14, 2009

Data Center Expansion – Part 2Data Center Expansion – Part 2

October 10, 2009October 10, 2009

Data Center Expansion – Part 2Data Center Expansion – Part 2

October 10, 2009October 10, 2009

Data Center Expansion – Part 2Data Center Expansion – Part 2

October 10, 2009October 10, 2009

New Data Center Layout New Data Center Layout

Where We Are Today (1)Where We Are Today (1)

Data Center Expansion (part 1) is similar to Data Center Expansion (part 1) is similar to existing facilityexisting facility• 12-in (30.48 cm) raised floor12-in (30.48 cm) raised floor• Redundant cooling capacityRedundant cooling capacity• No support for racks > 10 kWNo support for racks > 10 kW• No support for supplemental coolingNo support for supplemental cooling• Cable trays for power and networkCable trays for power and network

Data Center Expansion (part 2) was designed for Data Center Expansion (part 2) was designed for high density equipmenthigh density equipment• 30-in (76.2 cm) raised floor30-in (76.2 cm) raised floor• Support for racks > 10 kW (blades, half-depths, etc)Support for racks > 10 kW (blades, half-depths, etc)• Redundant cooling capacityRedundant cooling capacity• Support for racks > 2,500 lbs (1,135 kg)Support for racks > 2,500 lbs (1,135 kg)• 13-ft ceiling (4 m) for high-profile racks13-ft ceiling (4 m) for high-profile racks• Cable trays for power and networkCable trays for power and network• Support for supplemental coolingSupport for supplemental cooling• Environmentally-friendly buildingEnvironmentally-friendly building

Where We Are Today (2)Where We Are Today (2)

Facility expanded from 5000 ftFacility expanded from 5000 ft22 (465 m (465 m22) to 13,600 ft) to 13,600 ft22 (1260 (1260 mm22) of floor space) of floor space

Equipment capacityEquipment capacity• from ~150 to ~300 racksfrom ~150 to ~300 racks• from 6 to 13 robotic silosfrom 6 to 13 robotic silos

Infrastructure supportInfrastructure support• from ~1 to ~2.0 MW of UPS-backed power (up to 4 MW capacity)from ~1 to ~2.0 MW of UPS-backed power (up to 4 MW capacity)• Cooling capacity grew from ~1 to ~ 2 MW (up to 4 MW capacity)Cooling capacity grew from ~1 to ~ 2 MW (up to 4 MW capacity)

3 robotic silos and 6 racks of worker nodes first occupants 3 robotic silos and 6 racks of worker nodes first occupants of CDCE (October 2009)of CDCE (October 2009)

Is this sufficient until 2018?Is this sufficient until 2018?

Unresolved IssuesUnresolved Issues

Insufficient funds to:Insufficient funds to:• add 2add 2ndnd flywheel UPS for CDCE flywheel UPS for CDCE• diesel generator to support additionaldiesel generator to support additional flywheel UPS unitsflywheel UPS units• install additional 1 MW of cooling capacity (equipment already install additional 1 MW of cooling capacity (equipment already

purchased) purchased)

Estimated cost is additional US $2-3 millionEstimated cost is additional US $2-3 million• Estimate CDCE will exceed 2 MW of UPS power and cooling by Estimate CDCE will exceed 2 MW of UPS power and cooling by

20122012

Lead time to approve funds and pre-installation is Lead time to approve funds and pre-installation is 12 months 12 months decision by 2011 decision by 2011

Reason for Optimism?Reason for Optimism?

0

20

40

60

80

100

120

140

160

180

Total

Rack

Coun

t

2005 2006 2007 2008 2009 2010 (est.)

Multi-core cpu’s and Multi-core cpu’s and increasing storage increasing storage density have helped density have helped restrain a feared restrain a feared unsustainable growth in unsustainable growth in power and space needspower and space needs

Rack counts have not Rack counts have not increased at the same increased at the same rate as computing and rate as computing and storage deploymentsstorage deployments

Somewhat hopeful that Somewhat hopeful that continued technological continued technological gains will further restrain gains will further restrain data center growthdata center growth

SummarySummary Facility footprint nearly tripled since 2007Facility footprint nearly tripled since 2007

Applied lessons learned in design of data center expansion Applied lessons learned in design of data center expansion (part 2)(part 2)

Must increase cooling efficiency with new technologiesMust increase cooling efficiency with new technologies• Rack-top cooling units Rack-top cooling units • Rear-door heat exchangerRear-door heat exchanger• Hot aisle containmentHot aisle containment

Significant increases in power efficiency and technology Significant increases in power efficiency and technology (power supply, multi-core cpu, etc) is a positive (power supply, multi-core cpu, etc) is a positive development, but some unresolved issues remaindevelopment, but some unresolved issues remain