global traffic exchange among internet service providers (isps)
TRANSCRIPT
www.genuity.com
Global Traffic Exchange among Internet Service Providers (ISPs)
OECD - Internet Traffic Exchange
Berlin, June 7, 2001
J. Scott Marcus,Chief Technology Officer (CTO)
www.genuity.com
Economics of Internet Interconnection
• Background– Internet interconnection in North America– International Internet interconnection
• Economic modeling of Internet backbone peering– The off-net pricing principle– Implications for international interconnection
• Scaling challenges to the peering system• Cross-provider differentiated services,
measurements and SLAs
www.genuity.com
Internet Interconnectionin North America
• Historical roots• Peering and transit• Shared and direct peering• Shortest exit• Hierarchical structure
www.genuity.com
www.genuity.com
Privatization of the Internet (NSF 93-52)
Copyright (c) 1999 by Addison Wesley Longman, Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic,
mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher.
New York
Chicago
Atlanta
Dallas
Los Angeles
San FranciscoWashington DC
NAP
NAP
NAP
NAP
ISP
ISP
ISP
ISP
ISP
ISP
ISP
Backbone ISP
Backbone ISP
Backbone ISP
Backbone ISP
www.genuity.com
Peering and Transit• Peering is usually a bilateral business and technical arrangement,
where two providers agree to accept traffic from one another, and fromone another’s customers (and thus from their customers’ customers).Peering does not include the obligation to carry traffic to third parties.
• Transit is usually a bilateral business and technical arrangement, whereone provider (the transit provider) agrees to carry traffic to thirdparties on behalf of another provider or an end user (the customer). Inmost cases, the transit provider carries traffic to and from its othercustomers, and to and from every destination on the Internet, as part ofthe transit arrangement.
• Peering thus offers a provider access only to a single provider’scustomers; transit, by contrast, usually provides access at a definedprice to the entire Internet.
• Historically, peering has often been done on a bill-and-keep basis,without cash payments, where both parties perceive roughly equalexchange of value; however, there is often an element of barter.
www.genuity.com
Shared and Direct Peering• A few shared global traffic exchange points.• Smaller domestic shared traffic exchange points for
regional concentration and exchange of traffic.• Direct traffic exchange carries most Internet
backbone traffic. Even though shared trafficexchange points are losing market share, their trafficis likely to continue to grow in absolute terms.
• Carrier hotels and fiber interconnects - an emergingtrend that seeks to provide the best of both worlds.
• Whether shared or direct, the prevailing pattern isshortest exit routing - the sending provider hands offtraffic at the point most convenient to the sender.
www.genuity.com
Backbone ISP
Backbone ISP
Backbone ISP
Shared and Direct Peering
Shared TrafficExchange Point
RouterRouter
Router
Router
Router
Router
Router
Router
DirectInterconnection
DirectInterconnection
www.genuity.com
Shortest Exit (“Hot Potato”) Routing
Copyright (c) 1999 by Addison Wesley Longman, Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic,
mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher.
Backbone ISP
Backbone ISP
New YorkChicago
Atlanta
Dallas
Los Angeles
San FranciscoWashington DC
Web Site
Shared Traffic Exchange
Point
Shared Traffic Exchange
Point
www.genuity.com
Backbone and Secondary Peering
• Backbone peering interconnects providers thathave no need for a transit relationship. It providesthe only interconnection between those providers.
• Secondary peering interconnects providers whowould otherwise exchange traffic through sometransit connection.
• Secondary peering is an economic optimization,reducing traffic over the transit connection.
www.genuity.com
Fundamental Economics of aLocal or Regional ISP
Regionalor Local ISP
Many remote locationsconnect to a regional or localISP with individual,low bandwidth connections
Concentration to a larger ISP orbackbone provider with globalconnectivity by means of aconcentrated, high bandwidthconnection
Larger ISP orBackbone
TransitConnection
www.genuity.com
Secondary Peering
Regionalor Local ISP
Larger ISP orBackbone
TransitConnection
Regionalor Local ISP
Larger ISP orBackbone
TransitConnection
The secondarypeering connectionwill tend to exist ifthe cost of the connectionto each ISP is less thanthe money each saves due to reduced transit traffic.
www.genuity.com
A Hierarchical View of the Internet
Copyright (c) 1999 by Addison Wesley Longman, Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic,
mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher.
ISPs
Customers
ISPsBackbone
Backbone
BackboneBackbone
Backbone
Backbone
Backbone
Backbone
“Default Free” Zone
www.genuity.com
A Hierarchical View of the Internet
IBP IBP
ISP
ISP
ISP
Upstream
Downstream
PeeringConnection
cf. Lixin Gao
www.genuity.com
Negotiations for North AmericanBackbone Internet Interconnection
• Typical US backbone interconnection guidelines– Bi-coastal US presence, with multiple potential points of
interconnection– Significant transcontinental bandwidth– Consistent routes at all locations– Competent staff, professional 7 x 24 operation– Rough balance of ingress/egress traffic– Sufficient scale to justify transaction costs
• Where criteria are not met, a backbone may:– decline to exchange traffic, OR– expect cash or non-cash compensation in return
• Backbone providers negotiate traffic exchange terms andconditions on a case by case basis.
www.genuity.com
Emerging Global Trends
• The traditional “hub and spokes” system• Traffic exchange trends in Europe• Global deployment• International diffusion of users and content
www.genuity.com
Traditional “Hub and Spokes” System
New York
Chicago
Atlanta
Dallas
Los Angeles
San FranciscoWashington DC
NAP
NAP
NAP
NAP
NSP #1
NSP #2
NSP #3
vBNS
Regional#1
Regional#2
Regional#3
ISP #1
ISP #2
ISP #4
ISP #3
EuropeanISPs
AsianISP
Non-U.S.ISP pays
Copyright (c) 1999 by Addison Wesley Longman, Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic,
mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher.
Non-U.S.ISP pays
www.genuity.com
Former “Hub and Spokes” in Europe
U.S. Backbone
French ISP
German ISP
www.genuity.com
Internet Traffic Exchange in Europe Today
U.S. Backbone
French ISP
German ISP
Interconnection Point
www.genuity.com
Factors Driving European Evolution
• Decline in “street price” of circuits within Europedue to deregulation.
• Declining cost of transoceanic capacity.• Increased number and density of customers and
content (and caching).• Improved number and distribution of shared peering
points.• Deregulation of European telecoms, and recognition
of the need to minimize regulatory barriers toInternet growth.
• New transit services terminated in Europe andelsewhere.
www.genuity.com
Global Internet Backbone Deployment
New York
Chicago
Atlanta
Dallas
Los Angeles
San FranciscoWashington DC
NAP
NAP
NAP
NAP
NSP #1
NSP #2
NSP #3
vBNS
Regional#1
Regional#2
Regional#3
ISP #1
ISP #2
ISP #4
ISP #3
AsianISP
EuropeanISP pays
Backbonepays
Copyright (c) 1999 by Addison Wesley Longman, Inc. All rights reserved. No part of this publicationmay be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic,
mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher.
Backbonepays
Overseas Point of Presence (POP)
www.genuity.com
International Diffusion
Source: IDC, Merrill-Lynch (Kende, FCC)
Number of Users Online Worldwide
0
50
100
150
200
250
1995 1996 1997 1998 1999E
Mill
ion
s o
f U
sers
United States Rest of World
www.genuity.com
Motivations for InternationalPressure for Cost Sharing (ICAIS)
• Mistaken perception that U.S.-based backbonesdiscriminate against overseas providers in ourinterconnection policies.
• Dissatisfaction with allocation of transoceaniccircuit costs, which often are fully carried by thenon-U.S.-based provider.
www.genuity.com
An Excerpt from Genuity’s Guidelines1. Presence at three or more Shared Interconnection Points listed above
(two of which must be MAE-East ATM and MAE-West ATM), forDomestic ISPs; presence at two or more Shared Interconnection Pointslisted above for International ISPs.
2. For domestic ISPs, United States coast-to-coast nationwide backboneof at least 155Mbps.
3. Consistent route announcements at all exchange locations.4. Experienced, professional Network Operations Center staffed 24x7.5. Loose Source Record Route (LSRR) capability at core border routers
on network.6. For domestic ISPs, roughly balanced traffic.7. A minimum Internet traffic exchange of 1 Mbps with Autonomous
System 1.8. Willingness to enter into a formal Internet Interconnection Agreement.
http://www.genuity.com/infrastructure/interconnection.htm
www.genuity.com
Economic Modeling of InternetBackbone Peering
• Analytical framework• The off-net pricing principle• Implications for international
interconnection
www.genuity.com
Analytical Framework
• Define:co as cost of originationct as cost of terminationa as an access charge levied on
the sender
• Due to shortest exit, ct > co
• Thencost for the originating
network is co + acost for the terminating
network is ct - a
Networki
Networkj
www.genuity.com
Impact of Access Charges
• In a bill-and-keep system, access chargesare zero.
• With equal access charges and symmetrictraffic, net access charges are still zero.
• Providers will, however, view theirmarginal costs quite differently.(Laffont/Rey/Tirole)
www.genuity.com
The Off-Net Pricing Principle
• Under a broad range of conditions, marginalprice should equal marginal cost.
• Providers will tend to charge the same foron-net traffic as for off-net traffic, i.e. thereare no strong incentives for pricediscrimination.
www.genuity.com
Implications for InternationalInterconnection
• Where a group of customers have distinguishable costs thatare higher than those of other customers, the modelpredicts that marginal prices will be higher to exactlyoffset the higher costs.
• In the absence of market power, there is in fact no way forthe ISP to absorb the cost difference without reflecting it inthe price.
• This prediction is consistent with typical practices of US-based ISPs, both within the US and overseas.
www.genuity.com
Scaling Issues
• AS Number growth• IPv4 address growth• Routing table growth
www.genuity.com
Tony Bates’s DataAS Growth
0
1000
2000
3000
4000
5000
6000
7000
8000
9000F
oo
d
Ac
tiv
e A
Se
s
www.genuity.com
Exponential Growth of AutonomousSystem (AS) numbers
R2 = 0.9957
0
10000
20000
30000
40000
50000
60000
70000
80000
10/0
1/19
96
10/0
1/19
97
10/0
1/19
98
10/0
1/19
99
10/0
1/20
00
10/0
1/20
01
10/0
1/20
02
10/0
1/20
03
10/0
1/20
04
10/0
1/20
05
AS N
umbe
rs (ASNs)
ASNs to Oct2000
ASNs since Oct2000
Expon. (ASNs to Oct2000)
Source: Scott Marcus, Genuity
ww
w.genuity.com
Exponential vs Q
uadratic (Bates D
ata)
y = 6E
-14e0.0011x
R2 =
0.9957
y = 0.002x
2 - 138.37x + 2E
+06
R2 =
0.9962
0
5000
10000
15000
20000
25000
10/01/1996
01/01/1997
04/01/1997
07/01/1997
10/01/1997
01/01/1998
04/01/1998
07/01/1998
10/01/1998
01/01/1999
04/01/1999
07/01/1999
10/01/1999
01/01/2000
04/01/2000
07/01/2000
10/01/2000
01/01/2001
04/01/2001
07/01/2001
10/01/2001
01/01/2002
04/01/2002
07/01/2002
10/01/2002
01/01/2003
Active ASes (Clean)
AS
Ns to O
ct2000
AS
Ns since O
ct2000
Expon. (A
SN
s to Oct2000)
Poly. (A
SN
s to Oct2000)
www.genuity.com
Chicken Little was Wrong!• This is far simpler to remedy than IPv4 address
exhaustion, because– the solution need not impact end systems (hosts);– the solution need not impact DNS; and– the solution need not impact routers unless they speak BGP-4.
• Any solution is complicated by the need for backwardcompatibility and phased migration.
• Time until exhaustion is nonetheless sufficient toarchitect, design, implement and deploy solutions.
• Cisco and Juniper are reportedly well intoimplementation.
www.genuity.com
The RIRs Recognize the Need forForecasting
• Continuing need to further refine projections.• Need for forward-looking proactive forecasting on
a regular basis not only for AS numbers, but alsofor route table entries and IPv4/IPv6 addresses.
• Forecasting needs to incorporate allocation datafrom all three RIRs (APNIC, ARIN, RIPE NCC).
• Forecasting needs to be institutionalized by theRIRs themselves, with data readily available toindependent researchers.
www.genuity.com
The Team
• Assembled by ARIN– Frank Solensky Gotham Networks– kc claffy CAIDA– Scott Marcus Genuity
• Active contributions and support by APNICand RIPE NCC
www.genuity.com
Source: Solensky/Marcus/Claffy
IPv4 Address AllocationsCum ula tive IPv4 Addre sses Alloca te d
.
.5
1.
1.5
2.
2.5
1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001
IPv
4 A
dd
res
se
s (
bil
lio
ns
)
RIPE
ARIN
APNIC
www.genuity.com
Route Table Growth
Source: Geoff Huston
www.genuity.com
Applications Have Distinct Needs
• E-mail is tolerant of delay, but every bit of datamust ultimately be delivered correctly.
• For real time WWW traffic, delay must be low,but moderate variability of delay is permissible.
• Real time Voice over IP (VoIP) and video aretolerant of modest loss, but intolerant of largeand variable delay.
www.genuity.com
Cross-Provider Differentiated Services
• Internet services today are best-efforts.• Commoditized service, with marginal price equal to
marginal cost.• Differentiated services are of limited value today.
– Best-efforts are good efforts, so little incrementaladvantage for differentiated services.
– Customers have no demonstrated propensity to pay apremium for better CoS for data services.
– Better CoS for real time voice and video still interesting.– CoS today is limited to a single provider’s network.
www.genuity.com
Cross-Provider Measurements and SLAs
• Potentially valuable for best-efforts traffic;essential for Differentiated Services.
• Provider responsibility ends today at the peeringinterconnection point.
• No consistent measurement framework.• No business arrangements to incent SLA
adherence between peer networks.• Information sharing of proprietary data is a
difficult but not intractable problem.
www.genuity.com
Summary
• The distinction between peering and transit is vitalto an understanding of the underlying businessmodels that drive global Internet connectivity.
• The business models that have evolved aroundpeering, in North America and globally, arecomplex but rational.
• Peering business relationships have adaptedquickly in response to economic or regulatorystimuli, and will continue to evolve.
www.genuity.com
References
Cremer, Rey and Tirole, “Connectivity in the Commercial Internet”, presented at“Competition and Innovation in the Personal Computer Industry”, San Diego, April,1999.
Marcus, Designing Wide Area Networks and Internetworks: A Practical Guide, AddisonWesley Longman, 1999. Chapter 14, which describes peering and transit, is available athttp://www.genuity.com/about/executives/marcus_bio.htm
Huston, “Interconnection, Peering and Settlements”,http://www.isoc.org/inet99/proceedings/1e/1e_1.htm
Laffont and Tirole, Competition in Telecommunications, MIT Press, 2000. General analysisof the economics of telecommunications pricing.
Kende, “The Digital Handshake: Connecting Internet Backbones,” FCC, 2000.Gao, “On Inferring Autonomous System Relationships in the Internet, IEEE GlobeCom
2000, San Francisco, November, 2000.Milgrom, Mitchell, and Srinagesh, “Competitive Effects of Internet Peering Policies”,
presented at the American Economics Association, January, 2001.Laffont, Marcus, Rey and Tirole, “Internet Interconnection”, American Economics Review,
May, 2001.