do atm-based internet exchanges make sense anymore? william b. norton co-founder & chief...

Do ATM-based Internet Exchanges Make Sense Anymore?

William B. Norton

Co-Founder & Chief Technical Liaison

Equinix, Inc.

Internet Researcher

• Focused studies on ISP Peering

1. Identify Relevant Internet Operations Topic

2. Speak with prominent Peering Coordinators

3. Write/Evolve Draft White Paper

4. Walk Peering Coordinators through paper

5. Goto Step 3

^Internet Service Providers and PeeringSummary Findings of Peering Research….

Peering is a game of relationships

Today: Research Focus

Many ISPs pulling out of MAE-East

Why? “..Financial Reasons..”

Apply “A Business Case for Peering” WP

Result: “Do ATM-Based Internet Exchange Points Make Sense Anymore” white paper v1.5

~200 ISPs have seen this research paper, ~50 Walk-throughs

Data Points…

The Cost of Transit

• Variable based on usage ($/Mbps)

• Range: $350/Mbps down to $35/Mbps

• Greater volume = Lower unit price

Observation: Over the last 4 years, Transit Prices have dropped like a rock.

$1200/Mbps (AboveNet, 1998) now costs $120/Mbps (AT&T, 2002)

The Cost of Peering

)__(

__

___

____

:

_

CostsEquipmentRouterR

FeesRackMonthlyr

FeesPortIXMonthlyp

IXIntoCostsTransportMonthlyt

where

RrptCostsPeering

Observation: Transport Prices have dropped like a rock. Observation: New Router prices have dropped like a rock.Observation: Used Router Market is also very healthy (cheap).

Apply these to peering at ATM IX…

Transit ISP

ISP A

ISP B

Transit $$$

Transit $$$

1) Transport into ATM Fabric

R

RX

Peering $

ATM Fabric

2) ATM Switch Port (no Private Cross ConnectPossible in this architecture)

ATM-based Peering Model

Ethernet-based Peering Costs…

Transit ISP

ISP A

ISP B

Transit $$$

Transit $$$

1) Transport into Exchange

2) Rack Space atExchange PointFor Router

3) Router at IX

4) Ethernet Switch Port (/Private Cross Connect)

R

RX

Peering $Ethernet-Based Peering Model

When do each of these Peering Infrastructures make sense?

Cos

t of

Tra

ffic

Exc

hang

e in

$/M

bps

Amount of Traffic Exchanged (in Mbps)

Transit Price per Mbps

Peering Price per Mbps

Peering Breakeven Point(Peering=Transit)

EffectivePeering Bandwidth (in Mbps)

Min Cost ofTraffic Exchange(in $/Mbps)

Effective Peering Range

Transit Cheaper-Peering Cheaper

Peering Analysis Graph (axis)

Cos

t of

Tra

ffic

Exc

hang

e in

$/M

bps

Amount of Traffic Exchanged (in Mbps)

Transit Price per Mbps

Peering Price per Mbps

Peering Breakeven Point(Peering=Transit)

EffectivePeering Bandwidth (in Mbps)

Min Cost ofTraffic Exchange(in $/Mbps)

Effective Peering Range

Transit Cheaper-Peering Cheaper

Peering Analysis Graph (EPB)

Definition: The Effective Peering Bandwidth is the maximum bandwidth available for peering, defined as the minimum of the available transport bandwidth and the usable bandwidth on the shared peering fabric.

Cos

t of

Tra

ffic

Exc

hang

e in

$/M

bps

Amount of Traffic Exchanged (in Mbps)

Transit Price per Mbps

Peering Price per Mbps

Peering Breakeven Point(Peering=Transit)

EffectivePeering Bandwidth (in Mbps)

Min Cost ofTraffic Exchange(in $/Mbps)

Effective Peering Range

Transit Cheaper-Peering Cheaper

Peering Analysis Graph (minCost)

Definition: The Minimum Cost of Traffic Exchange is the unit cost of traffic exchange when the Effective Peering Bandwidth is fully utilized.

Cos

t of

Tra

ffic

Exc

hang

e in

$/M

bps

Amount of Traffic Exchanged (in Mbps)

Transit Price per Mbps

Peering Price per Mbps

Peering Breakeven Point(Peering=Transit)

EffectivePeering Bandwidth (in Mbps)

Min Cost ofTraffic Exchange(in $/Mbps)

Effective Peering Range

Transit Cheaper-Peering Cheaper

Peering Analysis Graph (EPR)Definition: The Effective Peering Range (EPR) is the range in which peering at an Internet Exchange makes sense (financially), measured as the range between the Peering Breakeven Point and the Effective Peering Bandwidth.

Q: Do ATM-based Internet Exchange Points Make Sense

Anymore?• Scenario 1: Medium Scale Peering

– ATM-IX– Ethernet-based IX– Transit

• Scenario 2: Large Scale Peering– ATM-IX– Ethernet-based IX– Transit

Medium Scale Peering

Medium Scale Peering

• Assume: OC-3 into IX for Peering

• Assume: Transit Commit of 500Mbps

• Data Point: $125/Mbps Transit Price

• Assume: Used Cisco 7200 w/OC-3+FastE– Price=$9K*2, 36 month term=$500/month– Used Juniper M20 on eBay $14

• Upgrade Peering when at 75% Avg. Util.

Price Survey for ATM Peering

Peering at ATM/OC-3 IX

Transit ISP

ISP A

ISP B

Transit $$$

Transit $$$

1) Transport into ATM Fabric

R

RX

Peering $

ATM Fabric

2) ATM Switch Port (no Private Cross ConnectPossible in this architecture)

WorldCom Local Loop OC-3 DC-MAE $3,000 per month

WorldCom ATM OC-3 Port $8,000 per month

Total Cost of Participation at MAE-East $11,000 per month

Effective Peering Bandwidth 75% * 120 Mbps 90 Mbps Max

(Includes 20% ATM Cell Tax/75% Util)

Minimum cost of traffic exchange $122.22 per Mbps

Assumption: Transit Price $125 per Mbps

Peering Breakeven Point 88 Mbps

Price Survey for FastE Peering

Peering at FastE/OC-3 IX

OC-3 (155 Mbps)Router

Fast Ethernet (100 Mbps)

Eth

Ethernet-based IX

Equipment: Cisco 7200 OC-3 / FastE $500 per month

Local Loop OC-3 DC-EQ-ASH $2,500 per month

1/2 rack + Fast Ethernet at EQ ASH $2,500 per month

Total Cost of Participation at EQ-ASH $5,500 per month

Effective Peering Bandwidth 75% * 94 Mbps 70 Mbps Max

(4% HDLC Framing irrelevent/75% Util)

Minimum cost of traffic exchange $78 per Mbps

Assumption: Transit Price $125 per Mbps

Peering Breakeven Point 44 Mbps

Effective Peering Range

Effective Peering Range

Peering Breakeven Point

Effective Peering Bandwidth

ATM/OC-3 88 Mbps 90 Mbps

FastE/OC-3 44 Mbps 70 Mbps

Shown Graphically…

Effective Peering Range in Mbps

10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

FastE/OC-3

UpgradeNeeded

Effective Peering Range

FastE/OC-3 Effective Peering from

44Mbps-70MbpsMin Peering Cost:$71/MbpsRange:$125/Mbps $71/Mbps

Transit cheaper than Peering

UpgradeNeeded

ATM/

OC-3Transit cheaper than Peering

ATM/OC-3 Peering Effective 88Mbps-90Mbps

Min. Peering Cost: $122/Mbps

Effective Peering Range

Summary…

Peering vs. Transit

ATM/OC-3 and FastE/OC-3 Peering vs. Transit

$0

$500

$1,000

$1,500

$2,000

$2,500

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90

# Mbps

$/M

bp

s MAE-OC3

Transit

EQFastEOC3

Large Scale Peering…

Large Scale Peering

• Assume: OC-12 into IX for Peering

• Assume: Transit Commit of ~900Mbps

• Data Point: $100/Mbps Transit Price

• Assume: New Juniper M5 w/OC-12+GigE– Price=$54K, 36 month term=~$1500/month– Used Juniper M40 on eBay $35

• Upgrade Peering when at 75% Avg. Util.

Price Survey for ATM Peering

Peering at ATM/OC-12 IX

Transit ISP

ISP A

ISP B

Transit $$$

Transit $$$

1) Transport into ATM Fabric

R

RX

Peering $

ATM Fabric

2) ATM Switch Port (no Private Cross ConnectPossible in this architecture)

Price WorldCom OC-12 DC-MAE-East $10,000 per month

Price WorldCom OC-12 Port $16,000 per month

Total Cost of Participation at MAE-East $26,000 per month

Effective Peering Bandwidth 75% * 500 Mbps 375 Mbps Max

(Includes 20% ATM Cell Tax/75% Util)

Minimum price for traffic exchange $69 per Mbps

Assumption: Transit Price $100 per Mbps

Peering Breakeven Point 260 Mbps

Ethernet-Based Peering

Peering at GigE/OC-12 IX

OC-12 (622 Mbps) RouterCisco 7300

gigE Ethernet (1000 Mbps)Eth

Equipment: Juniper M5 OC-12 / GigE $1,500 per month

Local Loop OC-12 DC-EQ-ASH $5,000 per month

1/2 rack + Gigabit Ethernet at EQ ASH $5,000 per month

Total Cost of Participation at EQ-ASH $11,500 per month

Effective Peering Bandwidth 75% * 597 Mbps 448 Mbps Max

(Includes 4% HDLC Framing/75% Util)

Minimum cost of traffic exchange $26 per Mbps

Assumption: Transit Price $100 per Mbps

Peering Breakeven Point 115 Mbps

Peering vs. TransitATM/OC-12 and GigE/OC-12 Peering vs. Transit

$0

$100

$200

$300

$400

$500

$600

50 80 110

140

170

200

230

260

290

320

350

380

410

440

470

# Mbps

$/M

bp

s MAE-OC12

Transit

EQGigEOC12

Effective Peering Range…

Effective Peering Range

Effective Peering Range

Peering Breakeven Point

Effective Peering Bandwidth

ATM/OC-12 260 Mbps 375 Mbps

GigE/OC-12 115 Mbps 448 Mbps

Graphically …

Effective Peering Range

ATM/ OC-3

FastE/OC-3

ATM/OC-12

GigE/OC-12

Effective Peering Range in Mbps100 200 300 400 500 600 700 800 900

ATM/OC-3 Peering effective 88Mbps-90Mbps,

$125/Mbps-$122/Mbps

GigE/OC12 Peering effective from 115Mbps-448Mbps,

$100/Mbps-$26/Mbps

FastE/OC-3 Peering effective from 44Mbps-70Mbps,$125/Mbps-$78/Mbps

ATM/OC-12 Peering effective from 260Mbps-375Mbps at

$100/Mbps-$69/Mbps

Effective Peering Range Gap…

ATM/ OC-3

FastE/OC-3

ATM/OC-12

GigE/OC-12

Effective Peering Range in Mbps100 200 300 400 500 600 700 800 900

EPR Gap

Effective Peering Range Gap

Do ATM-based IXes make sense?

Conclusion• Today’s Economy (Inexpensive Equipment,

Transit, Transport, Ethernet IXes) have conspired to make ATM-based IXes a low value proposition– ATM-based IXes make sense for a very narrow range

– Ethernet-based Peering are always less expensive than ATM-based Peering

• Transit plays a key role in cost-effective traffic exchange (Ethernet Peering+Transit more cost effective than ATM Peering)

“Do ATM-based Internet Exchange Points Make Sense Anymore?”

Author’s Notes

Author’s Note

• Top 5 Reasons ISPs are Peering at ATM-IX1. Haven’t done the math2. Must peer with ATM Participant x not at

Ethernet-based IX – don’t care what it costs3. Contract Term yet to expire4. “To be a player…”5. Let sleeping dogs lie – changes in peering might

trigger re-qualification request from peer.

Acknowledgements…

Acknowledgements

For this white paper I’d like to thank a few folks in particular for their review, insights, and comments on this paper:

Dorian Kim (NTT/Verio), Ingrid Erkman (ICG), Dave McGaugh (ELI), Eric T. Bell (Time Warner Telecom), Chris Parker (StarNet),Lane Patterson (Equinix), Jay Adelson (Equinix), Morgan Snyder (Equinix),John Hardie (Equinix), David Diaz (BellSouth), Joe Wood (Accretive Networks), Robert Seastrom (inter.net), Kevin Epperson (Level3), Petri Helenius (FICIX), Scott Sheppard (BellSouth), Ralph Doncaster (iStop.com), Leo Bicknell (ufp.org), Paul Vixie (vix.com), Ian Somerton and Dave Wodelet (Shaw/BigPipe), Tony Hain (Cisco), Jeff S. Wheeler (five-elements.com), Cliff Hafen, Dory Liefer, Shannon Lake (Omnivergent), Nenad Trifunovic (WorldCom), Andre Gironda (eBay), Jeb Linton (EarthLink), Daniel Golding (SockEye), Peter Moyer (Juniper), and others that preferred no recognition for their contributions to this paper.

Questions…

Questions?

Copies of this research paper available via e-mail to wbn@equinix.com

“Do ATM-based Internet Exchange Points Make Sense Anymore?”

Other Resources Available…

Resources Available for Peering Coordinators

• Gigabit Peering Forums

• Other White Papers document Peering Practices

• Peering Contact Database

Gigabit Peering Forums

Other White Papers

“Interconnection Strategies for ISPs”“Internet Service Providers and Peering”“A Business Case for Peering”“The Art of Peering: The Peering Playbook”“Do ATM-based Internet Exchange Points make

sense anymore?”“The Peering Simulation Game”

Freely available from the author: wbn@equinix.com

Peering Contact Database

For Peering Coordinators OnlyToss in your Business Card &Receive a copy of everyone’s Business CardsEvery 6 weeks (or so)

Managed as a community service.E-mail to wbnorton@wbnorton.org(Or give me your business card)