Build vs. Buy: Internet DatacenterOperations Research

INTAPMarch 6, 2001Tokyo

Agenda

• My Background and Who is Equinix? – 15 minutes• Recent Internet Operations Research – 20 minutes• “Build vs.Buy Internet Datacenter” White Paper – 40

minutes• Future: Where our Industry is heading – 5 minutes

NSFNET to Post-NSFNET Era InternetNSFNET to Post-NSFNET Era Internet

MY JOB: 5 Years as Chair of NANOG12 Years in Internet…No Internet Preso…

BARRNet

NWNet

Sesqui

NEARNet

NyserNetMerit

NCAR

OarNet

Regional

RegionalRegionalNSFNET

SDSC

AT&T

MCI

Sprint

UUNet

ANSNWNet NyserNet

MeritOarNet

Regional

Regional Regional

Regional

NSFNET (1987-1994) Regional Tech Meetings

–Performance & Operations

–Reporting & Coordination

–Routing & Network Planning

Privatization (1994-Now) North American Network Operators Group

(NANOG) Meetings

Internet Statistics• 87.5% of all Internet statistics are made up.

Made Up

Real

Source: I made it upSource: I made it up.

Equinix, Inc.

• Scale – size of Internet Business Exchanges, how many, where they are, future plans

• “Virtual Fly Through” Tour of the IBX - 6 minutes

IBX™ CentersIBX™ CentersIndustry Leading

SecurityIndustry Leading

Security• 5 security

checkpoints to customer equipment

• State of the art video imaging and archiving

• On site security personnel 24x365

• 5 security checkpoints to customer equipment

• State of the art video imaging and archiving

• On site security personnel 24x365

• Built to 3x power capacity

• 3 levels of power backup

• Forward looking design for growing industry

• Built to 3x power capacity

• 3 levels of power backup

• Forward looking design for growing industry

Fully RedundantBackup Power

Fully RedundantBackup Power

• 24x365 availability

• Customer trouble-shooting

• Smart Hands

• 24x365 availability

• Customer trouble-shooting

• Smart Hands

IP/CarrierKnowledgeable Staff

IP/CarrierKnowledgeable Staff

Customer FriendlySpaces – Conducting

Business & EquipmentPreparation

Customer FriendlySpaces – Conducting

Business & EquipmentPreparation• Designed from

customer experience

• Ergonomic work spaces

• Designed from customer experience

• Ergonomic work spaces

IBX™ GrowthIBX™ GrowthSites currently open or

Under DevelopmentSites currently open or

Under Development

Equinix Internet Business Exchange Model

Carriers

Internet Service Providers

ContentProviders Neutral Internet

Business ExchangeModel

• Infrastructure

• Players

• Neutrality…

Carrier Internet Exchange ModelCarrier Internet Exchange Model

Internet ServiceProviders

Carrier Operated Internet Exchange

• Infrastructure

• Players

Non Carrier-Neutral

No Content Support

Economic Distortions CarrierOperator

Job Title: Equinix Co-Founder & Chief Technical Liaison

• So, my job is to take friends out for lunches and dinners…

• Internet Operations Research &

• Write White Papers

• Summary: What did I learn?

No Internet Operations DocumentationTo make things more difficult, subtle language differences!

The Internet Operators: All about relationships

My Job: White Papers

Document Internet Operations Research– “Interconnection Strategies for ISPs”– “Internet Service Providers and Peering”– “Peering Case Studies”– “Peering Simulation Game”– “Build vs. Buy Internet Datacenter” today

New Job Title: Equinix Co-Founder & Chief Technical Liaison

White Paper: Interconnection Strategies for ISPs

SSS

GGG

UUU

A A A

S

G

U

A

CCCC

S

G

U

A

C

OC-12

OC-12

OC-12

OC-12

OC-12

Direct circuits replaced with:•OC-12 into Exchange•Fiber cross connects

Cost Drivers:•Big OC-12 Direct Circuit•Cheap Fiber Cross Connects•Aggregation Efficiency over OC-12 (2:1,3:1)•Rack FeesExchange Model Cost Formula:

ExchangeModelCost=BigDirectCircuit()+(n-1)*x/2+RackFees()

So does the Exchange-based model make sense?OC-12 $23,000/mo XC=$200/mo Rack=$1500/mo

Cost Comparison of Interconnection Strategies

$-

$50,000

$100,000

$150,000

$200,000

$250,000

$300,000

$350,000

$400,000

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64# of participants

Mo

nth

ly C

ost

of

Inte

rco

nn

ecti

on

White Paper: Internet Service Providers & Peering Transit

$$

Transit$$

ISP A

ISP B

Transit ISP

Time

Facilities Allocation

100%90%80%70%60%50%40%30%20%10%

First CarrierFirst ISP(s)First CP(s)

Secondary Carrier(s)Multiple ISPs provide CP redundancy

Facility becomes widely knownCritical mass is achievedBulk selling of rack 10-packsCross connect revenue grows rapidly Large Facility

Scaling

“The Startup Hump”

Facility fills

Peering

USNet EastNetWestNet

Peering

RoutingTable

White Paper: Case Study: Peering in the Bay Area

Transit vs. Peering Alternatives

$0

$500

$1,000

$1,500

$2,000

$2,500

$3,000

$3,500

1

11

21

31

41

51

61

71

81

91

10

1

Number of 10 Mbps

Co

st

pe

r M

bp

s

TRANSITIBXCustsGig

XODS3XOOC3XOOC12Telseon GigEXOOC48MAE-East OC-3MAE-East OC-12

Number of Mbps exchanged

Introducing:ISP Peering Breakeven Analysis Graphs

$/MbpsExchanged

Cost of Transit

Cost of Traffic Exchange in Peering Relationship

min(AccessMAX+PortMAX))

Breakeven Point(ISPs Indifferent between

Peering and Transit traffic exchange)

Initial Cost=f(Access,Rack,Port)

Allocated across the “useful capacity” at the exchange

Prefer PeeringPeeringRisk Expansion

at Exchange

Fe“Effective Peering BW”=100MbpsCost 155Mbps/100Mbps=$X/Mbps

Ge“Effective Peering BW”=155MbpsCost 155Mbps/1000Mbps=$Y/Mbps

“Effective Peering Bandwidth”

White Paper: The Peering Simulation Game

A

C

B

D

A

C

B

D

Transit Provider X

Transit Provider Y

IXW IXE

IXN

IXS

Y Y

XX

A A A A

A

A rolls 5, Pays Transit on others squares (3*$1000) Receives revenue on 6 squares (6*$2000)

$12,000 - $3,000 = $9,000

B rolls 3,Can get to IXE

Pays Transit on others squares (8*$1000) Receives revenue on 4 squares (4*$2000)

$8,000 - $8,000 = $0

B

B

B

C rolls 6,Can get to IXW, IXS

Pays Transit on others squares (11*$1000) Receives revenue on 4 squares (7*$2000)

$14,000 - $11,000 = $3,000

C

C

C

C C C

D rolls 1,Late entrant heading to IXE

Pays Transit on others squares (17*$1000) Receives revenue on 4 squares (2*$2000)

$2,000 - $17,000 = -$15,000

D

A rolls 3,Attaches to IXW

Pays Transit on others squares (13*$1000) Receives revenue on 9 squares (9*$2000)

$18,000 - $13,000 = $5,000Wants to peer with C – split costs?

YES: -$1,000 + both lose a turnNeither has to pay transit to each other!

A

AA

C

White Paper: Build Vs. Buy Internet DataCenter

• Modeling:• When does it

make sense to build your own?

• To Outsource?• Issues to consider• Economies of

Scale?• Multiple Business

Models

Service Level Agreements in Principle

• Guarantee a higher level quality• Define Measurement Criteria for adhering

to higher quality service Level• Meeting Performance Level critical to

success of business• Failure to meet Service Levels penalty

significantly worse than cost of service– eBay: $64,000/minute of downtime

SLAs in Practice• No SLA Pricing

– ISPPriceNO SLA=X

• SLA Pricing– ProbabilityOfMissingSLA=p– ISPPriceSLA1=X*(1+p)

• Network does not typically change to meet new criteria• CIO Security Audit Story• Equinix Switch SLA: 99.999% Uptime

SLA-Datacenters

• State-of-the-art Datacenters• Infrastructure Failure unlikely• SLAs rare, but…• Typically Limited to free month service• (Insufficient to potential loss customer faces)

• But what is an Internet Datacenter?• Include hosting? Include Internet Service? Include

Caching, Content Distribution?• Models help Define and Frame the Problem…

SLA Datacenter Parameters

• If there were Internet Datacenter SLAs• Parameters:

– Installation Lead Time– Service Availability– 24/7 Customer Support– Provisioning/Responsiveness

• Independent of Transport and Internet Service Issues.

Creating the Build vs. Buy Internet Datacenter Whitepaper

• Create a community white paper on issues associated with the building & scaling of Internet Datacenters

• Identify the issues that folks should consider when making the build vs. buy decision

Today:• Demonstrate the document & models to date and get

feedbackNOTE WELL:• EARLY STAGE OF BVB WHITE PAPER (v0.4)!• This Model may be of help to you in your SLA Discussions

Toured 32 (now) data centers

Some with very Secure Fiber Access…

Source: Technology Center of the Americas Tour

Some with less secure fiber entry…

Source: Don’t Ask.

Some Very Secure…

http://www.thebunker.net

Some Cat 5 Hurricane proof…

Some not quite cat proof.

Loading Dock

Great religious debate over ties

Velcro vs.Plastic cable ties vs.String

Time to white paper what I’ve seen & Facilitate BOF

Discussion• Datacenter BOF I NANOG Oct ’98 Atlanta

• Build vs. Buy White Paper

• Internet Datacenter Model– Definition of Internet Datacenter

• Focused discussion of Each Component of the Model

Internet Datacenter ModelDependence Relationship

IncreasedValue

Value

Dependence

Customer Business Model

Value

Dependence

Internet Datacenter Model Example

IXP

Ultimate Goal

• Build Financial models that allow an ISP to say:

• “Given our size requirements, it makes sense to build our own.”

Or• “Given our size requirements, it

probably makes sense to outsource this to someone who can realize economies of scale.”

+

+

=

Buy

Build

Size

Unit cost

Flavor of the Real Estate Model

• Place to put stuff

• Proximity to Fiber

• Space Size

• Utility Power

• Facilities Yard

Increased Value Relationship

Scaling of Datacenter Real Estate?

Scaling of the Real Estate Model

# Sq. Ft.

PremiumReal Estate Market(Tokyo, NYC,…)

Real Estate Market (everywhere else)

$/sq. ft.

Real Estate Size is Important

• Choose Too Small: Run out of space, lose economies of scale• Choose Too Big: Cost per Rack very high

Vs.

• Right of First Refusal (some Internet Datacenters allow this)• Expandability of Space

Commitment to DataCenter

• 1 yr contract

Vs.

• 10 year lease

Time to Market Issue – 6 – 9 months to build

Hidden Charges

• Meet Me Rooms

• Required products: – Must buy port on switch or certain volume of

BW

• Cross Connect Fees

Size & Finding Real Estate

• 100,000 sq ft in downtown DC?

• 10,000 sq ft in downtown DC

• Vs.

• Fiber Miles to Internet Datacenter

Flavor of the Infrastructure Model

• Heating Venting Air Conditioning (HVAC)

• Security Infrastructure

• Power Infrastructure

• Fire Suppression

Heating, Venting, Air Conditioning (HVAC) Infrastructure

• Controlled Humidity ~45%

• Too much Humidity: System Failure

• Too Little Humidity: Static Electricity

• Air Temperature:~72 (IBM: 65-85)

• Solutions:• DX Systems• Air Cooled• Water-Based

• How do these Scale?

HVAC ScalingDirect Expansion

Air Cooled

Chilled Water System

HVAC Cost

# Racks

$1,000/ton

$478/ton

$250/ton

HVAC System Cost: Economies of Scale

DX Systems: 25-300 tons of Cooling~ 10K sq ft or ~150-200 racks

Terramark Outsource of HVAC to FPL…

Crossover point where Chilled Water lowest per-rack solution

Cost Per Rack

$-

$5,000

$10,000

$15,000

$20,000

$25,000

$30,000

$35,000

$40,000

$45,000

$50,000

1 5 9 13 17 21 25 29 33 37 41 45 49 53Number of 10 Rack Bundles

DX System

Air Cooled

Water Cooled

Trend? Outsourcing of HVAC

FPL HVAC (Miami FL)Water Chiller

NOTA: 3 Blocks Away

Source: NAP of the Americas Tour

Security Infrastructure

• 24x7 Manned

• Access: Biometrics, card keys

• Video Surveillance

Sliding Glass

Power Infrastructure• Sun Netra 1U=90Watts

– 40U Rack=3600 Watts!

• Power Source– Power Grid (dual feeds)– Co-Generation– UPS/CPS/Generators

• Energy Storage– UPS Batteries– Flywheel (no UPS)– Fuel Cells (service)

• Monitoring Issues• Redundancy Issues• TBD: Scaling Issues

Fuel Cells for Generators

Fire Suppression Infrastructure

• Two Dominant Systems:• VESPA air sampling

• Dry pipe, pre action water system vs. FM-200

• Smoke & Heat Required to activate region

Infrastructure Operations

• Operations of HVAC, Security, Power systems

• N+1 Redundancy

Internet Datacenter Model

Transport Service Model

• Dark Fiber Provider(s), Carrier(s), Aggregators (like Universal Access), etc.

• Business Model?– Owned by Carrier? Will others come in?– Not owned by Carrier? Will Carriers come in?

• What type of services available?• How long to Provision?• Big Problem: Keeping up with BW Demand

Internet Datacenter Model

Internet Service Model

• ISP(s), Aggregator(s) like InterNAP?

• Business Model– Owned by ISP? Will others come in?– Not owned by ISP? Will others come in?

• Capacity Issues

• Latency Issues

• Interconnection Strategies Allowed?

Internet Datacenter Model

Content Provider Model

• Hoster Focus: – Installing New Servers, Provisioning Host Services,– Managing Server Farm, SLAs, Tech Support, 24/7

NOC, Back Office Solutions

• Hoster owns Datacenter?– Pro: Can buy Transit Services– Con: Expensive w/Bandwidth Hungry Apps– Con: Shared bandwidth issues (Yahoo! attack)– Con: Local Loop Expensive

• Business Model Content Distributor(s), Hoster(s)

Internet Datacenter Model

Future of the IXP?

MarketPlace: Stratification of component operations, fast creation & provisioning of packaged operations servicesImplication: IXP Content Sources, bandwidth demands

DSL Access Providers

Dial-in Modem Operators

E-Mail SvcsUSENET Svcs

WebHost SvcsSearch Engines Cache/Content

DistributionCarrier

InternetServiceProvider

Carriers

Carrier

CarrierInternetServiceProvider Carrier

CarrierInternetServiceProvider

Service Providers

InternetServiceProvider

Wireless Access Svc

VirtualISP

Sparing

Future SLA Challenges

1. Bundling of Internet Operations Services…2. Into Cohesive SLA-able services3. Notion: Trickle-down SLA

Bundle Participant Operations AgreementsFinancial Agreements

Example: Bundle Internet DataCenter, Transport, Internet, MSP. What happens when Transport service fails? Who covers the cost of the SLA?

BvB Document Availability

• All references are in the BvB white paper– Hyperlinks work in electronic version of

document– Early Stage of document: Comments Welcome!

• Send e-mail to wbn@equinix.com Subject: Build vs. Buy White Paper

Thanks to

• Sean Donelan (Equinix)• Jason Colangelo (APC)• Stephen Stuart/Phil

Steffora (MFN)• All Colo/ISPs tour

facilitators• Jay Adelson (Equinix)• Josh Horn (NOTA)• Justin Newton (NetZero)

• Friends at Exodus, Frontier Global Center, Level3, etc.

• Daniel Golding (NetRail)• Kenney Huang (AsiaInfra)• Sandip Gupta (Ensim)• Rob Glenn (Sun)• Everyone here! Please e-

mail feedback.