Build vs. Buy: Internet Datacenter

William B. Norton – Equinix

wbn@equinix.com

Michael Lucking – Epoch Internet

Goals• Create a community white paper on issues

associated with the building & scaling of Internet Datacenters

• Identify the issues that folks should consider when making the build vs. buy decision

Tonight:• Demonstrate the document & models to date and

get feedback

Non-Goals (Out-of-Order)

Not to advocate building you own rack space

Not to advocate buying rack space

(Rather – document the scaling of each and the criteria involved in the decision)

(Warning: Tradeoffs are Religious Topics for many.)

Background…

Toured a lot of data centers

Some with very Secure Fiber Access…

Source: Technology Center of the Americas Tour

Some with less so

Source: Don’t Ask.

Some Very Secure…

http://www.thebunker.net

Some Cat 5 Hurricane proof…

Some not quite cat proof.

Loading Dock

Great religious debate over ties

Velcro vs.Plastic cable ties vs.String

Time to white paper what I’ve seen & Facilitate BOF

Discussion• Datacenter BOF I NANOG Oct ’98 Atlanta

• Build vs. Buy White Paper

• Internet Datacenter Model– Definition of Internet Datacenter

• Focused discussion of Each Component of the Model

Internet Datacenter ModelDependence Relationship

IncreasedValue

Value

Dependence

Internet Datacenter Model Example

Ultimate Goal

• Build Financial models that allow an ISP to say:

• “Given these parameters and our required footprint, it makes sense to build our own.”

Or• “Given these parameters, it

probably makes sense to outsource this to someone who can realize these economies of scale.”

+

+

=

Buy

Build

Real Estate for Internet Datacenter Issues

• Place to put stuff

• Proximity to Fiber

• Space Size

• Utility Power

• Facilities Yard

Increased Value Relationship

Scaling of Datacenter Real Estate?

Scaling of the Real Estate Model

# Sq. Ft.

PremiumReal Estate Market(Tokyo, NYC,…)

Real Estate Market (everywhere else)

$/sq. ft.

Real Estate Size is Important

• Choose Too Small: Run out of space, lose economies of scale• Choose Too Big: Cost per Rack very high

Vs.

• Right of First Refusal (some Internet Datacenters allow this)• Expandability of Space

Commitment to DataCenter

• 1 yr contract

Vs.

• 10 year lease

Time to Market Issue – 6 – 9 months to build

Hidden Charges

• Meet Me Rooms

• Required products: – Must buy port on switch or certain volume of

BW

• Cross Connect Fees

Size & Finding Real Estate

• 100,000 sq ft in downtown DC?

• 10,000 sq ft in downtown DC

• Vs.

• Fiber Miles to Internet Datacenter

Infrastructure Scaling

• Heating Venting Air Conditioning (HVAC)

• Security Infrastructure

• Power Infrastructure

• Fire Suppression

Heating, Venting, Air Conditioning Infrastructure

• Controlled Humidity ~45%

• Too much Humidity: System Failure

• Too Little Humidity: Static Electricity

• Air Temperature:~72 (IBM: 65-85)

• Solutions:• DX Systems• Air Cooled• Water-Based

• How do these Scale?

HVAC ScalingDirect Expansion

Air Cooled

Chilled Water System

HVAC Cost

# Racks

$1,000/ton

$478/ton

$250/ton

HVAC System Cost: Economies of Scale

DX Systems: 25-300 tons of Cooling~ 10K sq ft or ~150-200 racks

Crossover point where Chilled Water lowest per-rack solution

Cost Per Rack

$-

$5,000

$10,000

$15,000

$20,000

$25,000

$30,000

$35,000

$40,000

$45,000

$50,000

1 5 9 13 17 21 25 29 33 37 41 45 49 53Number of 10 Rack Bundles

DX System

Air Cooled

Water Cooled

Security Infrastructure

• 24x7 Manned

• Access: Biometrics, card keys

• Video Surveillance

Sliding Glass

Power Infrastructure

• Monitoring of UPS, DC Power, PDUs,

• N+1 Redundancy

• Power Draw?– 20K=500Kw– (Jason Colangelo,

APC)

Fuel Cells for Generators

Fire Suppression Infrastructure

• VESPA air sampling

• Dry pipe, pre action water system vs. FM-200

Infrastructure Operations

• Operations of HVAC, Security, Power systems

• N+1 Redundancy

Transport

• Dark Fiber Provider(s), Carrier(s), etc.

Internet Service

• ISP(s), Aggregator(s)

Content Providers

• Content Distributor(s), Hoster(s), Manager(s)

Sample Crank of the Model: Assumptions

•Both: $400/sq foot build out costs for conditioned space•Buy: Equinix retail price $1,500/rack used for comparison,•Buy: No discount for buying more or larger cages•Build: Building Real Estate comes in 10,000 sq foot increments•Both: Conditioned space includes infrastructure (power, HVAC, Security systems, etc.)•Both: Does not include network bandwidth, hosting equipment, managed services, etc. •Build: Financing of build costs is available, •Build: Adequate building is available, and build time (time to market) is not an issue•Build: Sufficient network capacity (transport and transit) is available and not an issue•Both: Facility population is not an issue – ignoring collocation benefits and don’t expect to buy or sell services at the facility.

Sample Crank of the Model: Internet Datacenter Build vs. Buy

Conditioned Real Estate Cost

$0.00

$1,000.00

$2,000.00

$3,000.00

$4,000.00

$5,000.00

$6,000.00

$7,000.00

$8,000.00

$9,000.00

1 6 11 16 21 26 31 36 41

Number of 10 Rack Bundles

(Am

ort

ize

d)

Co

st

pe

r R

ack

Build: Equally ConditionedReal Estate Monthly Costper rack

Buy: Equinix ConditionedSpace RetailPrice/Rack/month

Note: Fictitious

References – Don’t Squint…• Ripin & Sayles, Insider Strategies for Outsourcing Information Systems, 1999 Oxford

University Press

• Greaver, Strategic Outsourcing: A structured Approach to Outsourcing Decisions and Initiatives, 1999 AMACOM

• Frenza, J.P. , Buying Web Services, 1999 Wiley Computer Publishing, particularly Chapter 1: An Outsourcing Overview by Frenza and Chapter 5:Infrastructure and Site Hosting by Chris Bryant.

• Norton, Donelan, Adelson, Newton, Data Center Needs Birds of a Feather meeting at NANOG 14 – Atlanta, Georgia – November 9, 1998 7:30-9PM, http://www.nanog.org/mtg-9811/data.bof.html

• Schroepfer, Andrew M., Communication Service Providers, Equity Research, U.S. Bancorp Piper Jaffray Equity Research, December 1999, particularly Ch 12: Facilities Hosting Providers.

• “What every CEO needs to know about Electronic Business”, Business Week, e.biz, Marsh 22, 1999

• IBM White Paper: Building a shared services data center, http://www.as.ibm.com/asus/cwp-liushrdsvcs.html, focuses on shared services enterprise Data Center

• Murray, Steve, “Web hosting Strategies: Market Review and Forecast, 1998-2003”, IDC Report #20953, December 1999.

• Search for “Data Center Construction” on http://www.google.com .

• http://www.as.ibm.com/asus/nordstrom.html

• http://www.gruber.com/compfac/datal.htm

• http://www.edgemail.com/whitepapers/

• Dialtone Internet: http://dialtoneinternet.net/tour/page0.htm

References – cont’d• Yahoo! Directory of Colocation Providers:

http://dir.yahoo.com/Business_and_Economy/Companies/Internet_Services/Web_Services/Network_Service_Providers/Hosting/Colocation/ • Yahoo! Directory of web hosting providers:

http://dir.yahoo.com/Business_and_Economy/Companies/Internet_Services/Web_Services/Network_Service_Providers/Hosting/Web_Site_Hosting/WhitePages/wp_0.html

• Search for Internet Data Center on google.com:• AT&T Internet Data Centers: http://www.ipservices.att.com/datatour/idc_dedicated.html • Web hosting directory site: http://www.webhostdir.com/ • Exodus Internet Data Center descriptions and virtual tour: http://www.exodus.com/idcs/ • .Equinix Internet Business Exchange (IBX) virtual tour http://www.equinix.com • PSINet Colocation Product: http://www.psinet.com.hk/hosting/co_location.html • Flashhost data centers http://www.flashhost.com/data_center.html • Hosting Internet Data Center in Zaire: http://www.is.co.za/solution/hosting/IDC2.html • Level3 colocation facilities: http://www.level3.com/us/services/colocation• Frontier Global Center (now global crossing) http://206.132.184.106/solutions/datacenters.shtml • Exodus Communications  Q1 2000 Quarterly Earnings Announcement 

4/20/2000 2:00 PM Pacific Time: http://webevents.broadcast.com/financecalls/event/index.asp?EarningsID=766&src=StreetFusion • Houses are designed by Geniuses & built by Gorillas, by Bob Johnson, ISBN: 1-886110-48-4• Facilities Operations & Engineering Reference, The Certified Plant Engineer Reference, R.S. Means Company, Inc., ISBN: 0-87629-462-X• Network Equipment-Building System (NEBS) Requirements: Physical Protection, Bellcore GR-63-CORE, Issue 1, October 1995. A module

of LSSGR, FR-64; TSGR, FR-440; and NEBSFR, FR-2063• Central-Station Alarm Services, UL 827 Standard for Safety, Underwriters Laboratories, Inc. ISBN 0-7629-0067-9 , Sixth Edition ©1997•  

Document Availability

• All references are in the BvB white paper

• Send e-mail to wbn@equinix.com

Thanks to

• Sean Donelan (Equinix)• Jason Colangelo (APC)• Stephen Stuart/Phil

Steffora (MFN)• All Colo/ISPs tour

facilitators• Jay Adelson (Equinix)• Josh Horn (NOTA)• Justin Newton (NetZero)

• Friends at Exodus, Frontier Global Center, Level3, etc.

• Daniel Golding (NetRail)• Everyone here! Please e-

mail feedback.