analysys mason domestic peering paper

8/6/2019 Analysys Mason Domestic Peering Paper

1/47

Public report

Overview of recent changes in the IP

interconnection ecosystem

May 2011

17038-93

.


2/47

Contents

1

Executive summary 1

1.1 Internet trends 11.2 Evolution of Internet interconnection 22 Introduction 43 Internet trends 73.1 Internet traffic has grown exponentially 73.2 Internet traffic has become globalized 93.3 The value of Internet content and applications has increased dramatically 113.4 Bandwidth-intensive types of Internet traffic have become predominant 133.5 Cloud computing services have become widely utilized 163.6 Internet connections are a key feature for many new devices and services 173.7 Conclusion 184 Evolution of Internet interconnection 204.1 Description of the early Internet 204.2 Internet exchange points (IXPs) 214.3 Internet service providers (ISPs) 234.4 Content providers 254.5 Impact on backbone providers 285 Conclusion 34

Annex A: Introduction to the IP interconnection ecosystem

Annex B: About Analysys Mason


3/47

.

Copyright 2011. The information contained herein is the property of Analysys Mason

Limited and is provided on condition that it will not be reproduced, copied, lent or

disclosed, directly or indirectly, nor used for any purpose other than that for which it was

specifically furnished.

Analysys Mason Limited

818 Connecticut Avenue NW

Suite 300

Washington DC 20006

USA

Tel: (202) 331 3080

Fax (202) 331 3083

[email protected]

www.analysysmason.com

Registered in England: Analysys Mason Limited

Bush House, North West Wing, Aldwych

London WC2B 4PJ, UK

Reg. No. 5177472


4/47

Overview of recent changes in the IP interconnection ecosystem | 1

1 Executive summary

Last year saw the 15th anniversary of the commercialization of the Internet backbone as we know it

today, when the US National Science Foundation Network (NSFNET) Backbone Service wasdecommissioned in favor of the current commercial Internet backbone. On that day, April 30, 1995,

the Netscape browser had just been introduced, AOL was the most popular ISP in the US and was still

charging hourly fees, and the DOCSIS standard for cable modem Internet access had not yet been

released. The Internet was mainly used for file transfer, email was simple text, and the Web was not

yet multimedia.

Over the intervening 15 years, there have been significant changes in the Internet ecosystem,

including an enormous increase in Internet traffic and bandwidth along with a corresponding dramatic

increase in the value generated by Internet content. During this period, interconnection arrangements

between all of the players in the ecosystem have continuously evolved to meet the challenges of the

changing Internet ecosystem, driven by commercial considerations and not regulation.1

1.1 Internet trends

The changes in the Internet ecosystem are mainly due to the following interrelated trends:

Internet traffic has

grown

exponentially

There has been an explosion of Internet access and in the traffic generated

by each individual user, for the reasons described below. As a consequence,

market players have adapted by modifying their network architectures andrevamping their relationships with suppliers and peers.

Internet traffic has

become globalized

From its historical origins in the US, Internet usage has grown significantly

around the world, and correspondingly, Internet content creation has also

globalized. As a result, the proportion of Internet traffic originated and

terminated outside the US has significantly increased over time.

Additionally, the amount of Internet traffic exchanged in US-based Internet

exchange points (IXPs) is now much less predominant, as many countries

and regions have expanded their Internet traffic exchange capabilities

through one or more local IXPs.

The value of

Internet content has

dramatically

increased

15 years ago, money flows were mostly directed to ISPs and backbone

providers for Internet access and traffic, while content was mostly free or

had limited value. This has changed significantly, as content has gained in

value thanks to an improved quality of access and the corresponding

availability of premium content. As a result, content providers have been

able to leverage the value of their content in order to reduce their costs for

1The only instances of regulatory intervention in IP interconnection have been during merger proceedings, when agencies have imposed

conditions ranging from divestiture to temporary limits on changes in peering agreements.


5/47


.

delivering their traffic (for instance, by building their own infrastructure to

distribute content around the country closer to end-users).

Bandwidth-

intensive and

quality needy types

of Internet traffic

such as video have

become

predominant

Bandwidth-intensive and quality needy types of traffic, such as video, have

experienced a massive growth, which has had a major impact on bandwidth

requirements and the direction of traffic flows. There has been a significant

increase in the amount of traffic being downloaded from centralized sites

such as YouTube, while at the same time the popularity of peer-to-peer

services has resulted in significant traffic between end users. These

increases in overall traffic, along with changes in traffic patterns, have led

ISPs and content providers to adapt their network policies and architectures.

Cloud computing

services have

become widely

utilized

Cloud computing services, ranging from online email to business

applications, have been embraced as barriers to their adoption have been

progressively lifted, thanks to the improvement of Internet access, the

development of security safeguards, and the resulting increase in awareness

and confidence in cloud computing. This has increased overall usage of the

Internet, while also making consumers more sensitive to latency and other

quality measures that impact access to their cloud applications.

Internet connection

became a

mandatory feature

for many new

consumer

electronic devices

Many new devices have emerged that incorporate Internet access: mobile

devices with features adapted to data traffic, netbooks and tablets designed

to enable Internet access to cloud-computing applications, and new video

streaming devices relying on broadband. As a result, many traditional

communications services have migrated to the Internet, for example TVover IP and Voice over IP (VoIP). This has led market players to increase

mobile access, install additional network capacity, and establish new forms

of interconnection.

The changes described above, individually and in sum, have led to significant changes in the

underlying Internet ecosystem as it was established 15 years ago.

1.2 Evolution of Internet interconnection

15 years ago, the Internet ecosystem had a relatively simple structure, with clear and hierarchical

relationships between end users and content providers buying Internet access from ISPs, ISPs buying

Internet transit from backbone providers, and backbone providers peering with each other. 2 At the

time, given the historical development of the Internet, much of the global traffic originated,

terminated, or transited the US.

But Internet players have steadily adapted to the changing market conditions described above in a

number of ways, in order to carry greater traffic volume with lower latency, and to reduce their transit

2For an introduction to Internet market players and interconnection definitions, please see Annex A.


6/47


.

costs, thereby changing the traditional hierarchy of the ecosystem in the process, while also making it

truly global.

Internet Exchange

Points (IXPs)

facilitated

increased

connectivity

The development of IXPs around the world has benefited backbone

providers by facilitating distributed interconnection between peers and

transit customers, which improves the quality of service and reduces traffic

carrying costs. However, the increase in utilization of IXPs also helped in

marginalizing the role of the backbone providers, as former customers

began to interconnect directly with one another at the IXPs.

ISPs and content

providers

increasingly

connect directly

Many route around options have emerged for ISPs and content providers

to exchange Internet traffic, while avoiding Internet backbone transit costs.

These include secondary peering arrangements between ISPs to exchange

traffic directly, much of it from peer-to-peer applications. Increased traffic

has also led content providers to negotiate paid peering arrangements with

ISPs to deliver content to their end users. Backbones have adapted to these

changes by selling only partial transit to ISPs and content providers, which

have arranged much of their traffic exchange directly. At the same time,

peering relationships have evolved based on volume, traffic flows and

routing.

Content providers

have a variety of

options for

delivering traffic

Increases in content traffic have led to traffic imbalances, such that ISPs

delivered much more traffic from content providers to their end users. This

has led to paid peering arrangements to cover the cost of the traffic. Content

providers have leveraged their expanding financial resources to delivertraffic directly to ISPs by building private networks or using third-party

content delivery networks (CDNs) to deliver content directly to caches at

the edge of ISPs networks, further reducing transit costs and the resulting

revenues for backbones.

From the backbone providers point of view, these changes led to a reduction in demand for transit

services, and an increase in competition from former customers who now have a number of choices

for delivering and exchanging traffic. Further, backbones must compete vigorously on the price of

transit in order to generate the traffic volume to continue to peer with one another. This has resulted in

an increase in the level of competition for Internet transit services, as evidenced for example in the

fall in transit prices over the past five years, with no sign of respite.

In conclusion, in the 15 years since the commercialization of the Internet backbone, the Internet

ecosystem has proven itself to be able to develop and sustain interconnection in the absence of sector-

specific regulation. It has also shown itself to be able to adapt well to rapid and profound market

changes without regulatory intervention, which in telecommunications is typically much slower to

implement changes in interconnection arrangements and has issues with implicit subsidies and

arbitrage. As a consequence of these changes, todays Internet ecosystem is no longer hierarchical, but

rather a dynamic web of interconnections between a variety of Internet players.


7/47


.

2 Introduction

This paper marks the 15th anniversary of the commercialization of the Internet backbone as we know

it today. There have been significant changes in the Internet ecosystem over the last 15 years,resulting in enormous increases in Internet traffic, supported by a massive expansion of the Internet

bandwidth, and accompanied by a major increase in the value generated by Internet services and

applications. Through these changes, one feature remains prominent : interconnection arrangements

between all of the players in the ecosystem have continued to evolve to meet the challenges of the

changing ecosystem, driven almost entirely by commercial considerations and not regulation.3

On April 30, 1995, the US National Science Foundation Network (NSFNET) Backbone Service was

decommissioned in favor of the current commercial Internet backbone. The NSFNET operated a

national backbone network allowing regional networks to connect at supercomputing sites.4 At this

time, the objective was to create an open network linking academic researchers, and allowing them

access to distant supercomputers over the network at no cost. When the NSFNET network went

online in 1986 and allowed commercial traffic (and not just NSFNET traffic) to be carried, it

consisted of six sites interconnected with leased 56kbit/s Digital Data System (DDS) links this was

upgraded to 45Mbit/s links in 1991.

At the time that the NSFNET was decommissioned in 1995, Netscape the first commercial Web

browserhad just been introduced, AOL was the most popular ISP in the US and was still charging

hourly fees, and the DOCSIS standard for cable modem Internet access had not yet been released. The

Internet was used mainly for file transfer, email was simple text, and the Web was not yet multimedia.Since then, traffic has exploded, as shown in Figure 2.1below.

3The only instances of regulatory intervention in IP interconnection have occurred during merger proceedings, when agencies have

imposed conditions ranging from divestiture to temporary limits on changes in peering agreements.

4 From 1987 to 1995, the NSFNET Backbone was designed, managed, and operated on behalf of the NSF by Merit Network, Inc., a non-

profit corporation governed by public universities partnering with IBM, MCI, and the State of Michigan.


8/47


.

Figure 2.1: Monthly

backbone traffic in the

US [Source: Minnesota

Internet Traffic Studies;

Analysys Mason

Estimates]

This increase in traffic comes as a result of a virtuous circle of more powerful computers, the

availability and adoption of broadband access, the increase in bandwidth available for carrying

Internet traffic, and the introduction of multimedia content (particularly video), along with new ways

to share that content, such as peer-to-peer. Figure 2.2 below shows how new categories of Internet

video have arisen in the last five years and are projected to become the dominant category of content

over the next few years.

Figure 2.2: Consumer

Internet traffic forecasts

[Source: Cisco Visual

Networking Index]

Note: in the figure above, the File sharing category includes traffic from peer-to-peer applications such as BitTorrent and eDonkey, as well as

web-based file sharing.

The NSFNET was the backbone over which regional networks exchanged traffic. When it was closed

in favor of commercial backbones, there was no longer a single backbone that could be used for

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

TB/month

0

2,000

4,000

6,000

8,000

10,000

12,000

2006 2007 2008 2009 2010 2011 2012 2013 2014

PBpermonth

Web and Email File Sharing

Internet Gaming Internet Voice

Internet Video com . Internet Video to PC

Internet Video to TV


9/47


.

traffic exchange. Instead, four Network Access Points (NAPs) across the country were designated as

the points at which traffic could be exchanged by the new commercial backbones. Such

interconnection was not regulated, making it possible for commercial arrangements known aspeering

and transit to be negotiated between the backbones and ISPs. The introduction of commercial

backbones not only modified the structure of the Internet infrastructure ecosystem, but also

accommodated to changes to the type of Internet traffic carried, as this opening increased greatly the

carrying and exchange of commercial traffic in addition to the university and research traffic

previously carried.

This document aims at describing changes in the Internet ecosystem since the commercialization of

the Internet backbone. We analyze the changes in terms of the type of Internet traffic carried, the

infrastructure supporting the Internet traffic, as well as the evolution of the different players and their

inter-relationships. We also assess how commercial models adapted to these changes, and analyze the

impact of these changes on competition between the backbones.

The remainder of this document is laid out as follows:

Section 3 provides an outlook of the main trends over the past 15 years that have impacted the

Internet backbone

Section 4 analyses the evolution of the Internet backbone, with a focus on interconnection

arrangements, and describes the current market situation from point of view of each of the main

types of Internet players

Section 5 summarizes the main conclusions reached.


10/47


.

3 Internet trends

The Internet ecosystem has evolved significantly over the past 15 years, with corresponding impacts

on interconnection arrangements. This evolution is partly due to a number of trends that are allinterrelated, but that can be described as follows:

Internet traffic has grown exponentially

Internet traffic has globalized

the value of Internet content and Applications has increased dramatically

bandwidth-intensive types of Internet traffic such as video have become predominant

cloud-computing services have become widely utilized

Internet connections are a mandatory feature for many new consumer electronic devices and

services.

Figure 3.1: Trends that

have impacted the

Internet ecosystem over

the past 15 years

[Source: Analysys

Mason]

We provide in this section an analysis of these trends, which will help to better understand the context

of the evolution of Internet interconnection described in the following section.

3.1 Internet traffic has grown exponentially

The past 15 years have seen an explosion of Internet usage around the world, particularly from

Internet broadband subscriptions that allow end users to use their connections to access the new high

bandwidth services such as video streaming. At the end of September 2009, there were more than 75

million fixed broadband subscriptions in the US, representing a penetration of around 60% of

households, and significant growth from the end of 2001, when there were only around 10 million

broadband subscribers, as shown in the Figure 3.2 below.

Globalization ofInternet traffic

Value of InternetContent

Peer-to-peer andvideo streaming

Cloud computing

Online devices

Volume ofInternet traffic


11/47


.

Figure 3.2: Growth in

number of broadband

subscribers in the US

[Source: Analysys Mason

Research]

In the past 10 years, this dramatic increase in the adoption of fixed broadband has been accompanied

by an explosion of mobile broadband usage. According to CTIA, wireless data revenues in the US

rose from US$211 million in 2000 to US$41.5 billion in 2009. 5 Similar growth has characterized

markets in Europe and much of Asia as well, resulting in demand for Internet services that is

distributed much more evenly across the world than 15 years ago, as shown in Section 3.2.

Not only has the number of Internet access users experienced a significant growth, but also the

bandwidth usage by each individual user has exploded. This can be explained in part by the increase

in the availability of even higher capacity Internet links, and the growing availability of bandwidth-

intensive applications such as video downloads (see section 3.4), the development of cloud-computing

services (see section 3.5), and the take-up of a large variety of devices connected to the Internet (see

section 3.6).

These combined effects have resulted in an exponential growth of Internet traffic in general, and in

the traffic carried by Internet backbone providers in particular, as shown above in Figure 2.1. The fast

growth in traffic has naturally generated significant changes in the Internet ecosystem, as marketplayers have adapted to this demand increase by modifying their network architectures (e.g.

expanding their networks by deploying fiber and upgrading their capacity to accommodate higher

traffic volumes and traffic that is more latency sensitive, and creating new routes to carry the traffic

more efficiently), as well as revamping their relationships with suppliers and peers, as described

below.

5Source: http://www.ctia.org/media/industry_info/index.cfm/AID/10323

0

10

20

30

40

50

60

70

80

90

2001 2002 20032004 2005 20062007 2008 2009

Q3

Millions

0%

10%

20%

30%

40%

50%60%

70%

80%

90%

100%

Penetrationas%households

DSL Cable modem

Residential FTTB Broadband FWABroadband penetration
http://www.ctia.org/media/industry_info/index.cfm/AID/10323http://www.ctia.org/media/industry_info/index.cfm/AID/10323http://www.ctia.org/media/industry_info/index.cfm/AID/10323http://www.ctia.org/media/industry_info/index.cfm/AID/10323


12/47


.

3.2 Internet traffic has become globalized

For historical reasons, in the early 90s Internet usage was biased towards the US, as was Internet

traffic exchange at the time that the NSFNET was decommissioned in 1995. This situation evolved

significantly over time, as demand for Internet access exploded first in all the other developed

countries, and then in developing countries. Indeed, today a significant portion of Internet users are to

be found in large markets like China and India: for example, China today boasts more than 400

million Internet users, and other emerging Asian nations are also fuelling the growth of traffic in this

region.

Figure 3.3: Evolution of

the distribution of Internet

users by region[Source:

ITU]

The increase in the number of Internet users outside the US led to a corresponding increase in Internet

traffic in other regions of world, with Internet content being created, hosted and accessed from outside

the US. As a result, the Internet network topology began to evolve, so as to enable a more efficient

carrying of traffic. The historical use of US-based IXPs to exchange foreign traffic was not only

inefficient, but also led to a call for intervention that went by the rubric of International Charging

Arrangements for Internet Services (ICAIS). In this debate at the Asia-Pacific Economic Cooperation

(APEC) forum and the International Telecommunication Union (ITU), countries underserved by

Internet infrastructure sought to address the issue of bearing the costs of connecting to US-based IXPs

for traffic exchange.

To deal with these issues, national and regional networks started to interconnect to each other and

exchange their traffic at regional IXPs such as AMS-IX in Amsterdam and LINX in London. These

IXPs exchange a significant amount of traffic among a growing number of members. For instance,

AMS-IX has 389 connected networks exchanging 1091.9Gbit/s of traffic at peak times, while LINX

0%

10%

20%

30%

40%50%

60%

70%

80%

90%

100%

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

North America Europe and Central Asia

East Asia and Pacific Latin America and Carabbean

Middle East and North Africa South Asia

Sub-Saharan Africa


13/47


.

has 379 members with a peak traffic exchange rate of 614Gbit/s.6 This traffic is not just national, but

also international, and the result is that the host countries have turned into significant international

hubs. As shown in Figure 3.4 below, the Netherlands, followed by the UK, has a distinct edge in the

amount of international Internet bandwidth per population compared with a wide variety of countries,

based in no small part on the significant role of Dutch and British IXPs in attracting international

backbones and content providers. Similar activity is taking place in Asia and to a lesser degree in

other parts of the world, as IXPs act to help to localize traffic and attract a surrounding ecosystem of

backbones and content providers.

Figure 3.4: Growth in international Internet bandwidth per head of population [Source: Telegeography,

ITU, Euromonitor, company data]

Consequently, the role of the US as a hub for international exchange of Internet traffic has diminished

significantly. Figure 3.5 below shows that even though Latin America still relies heavily on the US

for Internet bandwidth, regional networks in Europe, Asia and Africa are less and less dependent on

US-based facilities to carry and exchange traffic. Africas Internet traffic is now focused morepredominantly on Europe, while Europes traffic is mainly intra-regional, which is also becoming the

norm in Asia.

6In October 2010 AMS-IX had 389 networks connected, measured by autonomous system numbers, with a peak traffic exchange of

1091.9Gbpit/s (source: www.ams-ix.net). LINX provides general statistics showing that in October 2010 it had 379 members exchanging

614Gbit/s peak traffic (source: www.linx.net).

00

20

40

60

80

100

120

140

160

2002

2003

2004

2005

2006

2007

2008

kbit/s

Australia

USA

Canada

Germany

France

Singapore

UK

Netherlands
http://www.ams-ix.net/http://www.ams-ix.net/


14/47


.

Figure 3.5: Share of

Internet bandwidth

connected to US and

Canada, by region

[Source: Telegeography]

Figures represent Internet bandwidth connected across international borders as of mid-year. Domestic routes are excluded.

Today, 89 countries possess their own IXPs to exchange national and regional Internet traffic, and

thus lower the costs associated with international Internet access. 7 It is interesting to note that this

evolution has not required specific regulatory intervention, but is the result of commercial solutions

adopted by national and regional networks.

3.3 The value of Internet content and applications has increased dramatically

As new Internet services have developed, the positioning of the different stakeholders in the Internet

ecosystem has evolved (see Annex A for an introduction to the IP interconnection ecosystem). 15

years ago, ISPs and backbone providers were at the top of the hierarchy, and money flows were

mostly directed to them for Internet access and traffic, while content was mostly free or of limited

value. This has changed significantly in recent years as the value of content has risen, thanks to an

improved quality of service and the development of premium content.

Online music provides a good illustration of the evolution of content values in the past decade.

Initially, music labels were reluctant to sell their music online, for a variety of reasons. As a result, the

first widespread distributor of online music was the peer-to-peer network Napster. This was free, but

content was largely pirated and copyright issues led to its shutdown in 2001. Following this, labels

became comfortable with the level of protection being offered by digital rights management (DRM),

and began to make their music available online, leading to the emergence of paid downloads. A good

7Source: Packet Clearing House Report on Internet Exchange Point Locations (https://prefix.pch.net/applications/ixpdir/summary/)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2002 2003 2004 2005 2006 2007 2008 2009 2010

Africa Asia Europe Latin America
https://prefix.pch.net/applications/ixpdir/summary/https://prefix.pch.net/applications/ixpdir/summary/


15/47


.

illustration is the success of the media platform iTunes Store, launched by Apple in April 2003: as of

February 2010, 10 billion songs have been sold on this platform.8

Other significant value-added services have emerged from mobile Internet access, with the explosion

of the mobile paid applications market. The success of Apples App Store can illustrate this trend: as

of January 2010, 3 billion applications (both free and paid-for) had been downloaded through the

store.9 More generally, the Yankee Group estimates that the total revenues for the mobile paid

applications industry in the US alone reached US$1.6 billion in 2010, against US$537 million the

preceding year. This rise in content value will continue in the coming years, as this market growth

will be supported by devices such as the new iPad designed for the consumption of online multimedia

content. According to the same source, application revenues are forecast to reach US$11 billion by

2014.10

The massive growth in Internet advertising revenue illustrates well the growth in content value. Figure

3.6 below provides an estimate of the evolution of the fixed and mobile Internet advertising markets

in North America. The Internet advertising market is estimated to have been worth around US$25

billion in 2009, compared with approximately US$10 billion five years before, and is projected to

grow to more than US$35 billion by 2013. It should be noted that a large part of these revenues are

captured by content providers such as Google.

Figure 3.6: Internet

advertising market in

North America [Source:

PricewaterhouseCoopers

LLP, Wilkofsky Gruen

Associates]

8Source: http://www.apple.com/pr/library/2010/02/25itunes.html

9Source : http://en.wikipedia.org/wiki/ITunes_Store#Video_2

10Source : http://www.lesechos.fr/info/comm/020439905748-applications-mobiles-l-institut-yankee-group-a-revu-a-la-.htm

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

2004

2005

2006

2007

2008

2009

2010

2011

2012

2013

USD

million

Wired Internet advertising Mobile Internet advertising

Forecasts
http://www.apple.com/pr/library/2010/02/25itunes.htmlhttp://en.wikipedia.org/wiki/ITunes_Store#Video_2http://en.wikipedia.org/wiki/ITunes_Store#Video_2http://www.apple.com/pr/library/2010/02/25itunes.html


16/47


.

An interesting example of the evolution of the value of content is ESPN360.com, a broadband

network for live sports programming in the US and several other countries. ESPN has essentially

adopted a pay-TV model, in which ISPs must sign up and pay ESPN for offering the service, rather

than charging end users directly. The customer pays for access as a part of its ISP fees, whether it uses

the service or not, and may choose an ISP based on the availability of ESPN360.

This example suggests the extent to which the relative positions of content providers and ISPs have

evolved in the Internet ecosystem. In particular, content providers have been able to leverage the

attractiveness of their content in order to reduce their supply costs for accessing the Internet.

Furthermore, as we will see in Section 4.4, content providers can use their fast-growing revenues to

invest in their own network infrastructures, in order to further reduce the transit costs by developing

self-supply solutions, and thus avoid paying other market players to deliver their traffic.

3.4 Bandwidth-intensive types of Internet traffic have become predominant

The nature of Internet traffic has changed dramatically in recent years, and in particular there has been

a massive growth in the proportion of video traffic as a percentage of total Internet traffic. Video

traffic is distributed either via peer-to-peer services or streamed from a centralized server. Projections

show that video will remain a major driver for traffic increase (see Figure 2.2) and sustain the growth

in Internet traffic. We focus in this section on the growth of streaming video and peer-to-peer traffic,

which not only has a major impact on the bandwidth requirement for Internet traffic, but also on the

traffic patterns, as video streaming modifies the traffic pattern from traditional two-way

communication to one-way content delivery).

3.4.1Sustained growth in video traffic

The consumption of video media consists mostly of Internet video to PC (i.e. free or pay-TV viewed

on a PC), and in a more limited scale in Internet video to TV (the same as video to PC, but streamed

to the TV set). Thus today, 82% of Internet users in the US watch videos online, and an average user

views 182 videos per month.11 YouTube, a video-sharing website on which users can upload and view

videos, is a leading symbol of this Internet explosion. YouTube was founded in February 2005 and

bought by Google in 2006. In only five years, this company achieved an almost unimaginable growth:

by October 2009, it was serving more than a billion videos per day worldwide.12 In November 2009,

12.2 billion videos were viewed on YouTube on a monthly basis in the US alone. 13

Not only has YouTube managed to draw a very large audience, it has also succeeded in attracting a

large amount of free user-generated content: YouTube announced on March 2010 that users were

uploading some 24 hours of video to the platform every minute.14 This represents double what was

11Source : http://website101.com/statistics/Internet-stats-video-stats-social-media-usage-traffic-email/#ixzz0jJUhngj3

12Source : http://bits.blogs.nytimes.com/2009/10/09/youtube-were-bigger-than-you-thought/?hp)

13Source: http://bits.blogs.nytimes.com/2009/10/09/youtube-were-bigger-than-you-thought/?hp)

14Source: http://youtube-global.blogspot.com/2010/03/oops-pow-surprise24-hours-of-video-all.html


17/47


.

uploaded only two years before (of course, some of this material is commercial, and YouTube has had

to address the copyright issues that arose first with Napster).

Figure 3.7: Hours of

video uploaded on

YouTube platform per

minute [Source:

YouTube]

But YouTube, while significant, is not the only player in the Internet video market: there are other

major video content providers that are experiencing similar growth levels. Another example would be

Hulu, a website offering commercial-supported streaming video of TV shows and movies from

several major TV channels and studios. In November 2009, Hulu was streaming 924 million videos

per month in the US alone, according to the research company Comscore.15 In the UK, the BBCs

online iPlayer is estimated to have received well over a billion requests for programming in 2010,

with 114 million requests in July 2010 alone.16 Other services such as Netflix stream video directly to

the TV. According to Sandvine, streamed audio and video represent almost 43% of data consumption

in North America during peak periods, with Netflix video alone accounting for over 20% of download

traffic during peak times.17 In addition to these video-streaming sites, a significant amount of video is

consumed using peer-to-peer networks.

15Source:

http://www.comscore.com/Press_Events/Press_Releases/2010/1/November_Sees_Number_of_US_Videos_Viewed_Online_Surpass_30

_Billion_for_First_Time_on_Record

16This includes both video and radio programming. See http://www.bbc.co.uk/blogs/bbcInternet/img/Publicity_Pack_July_2010.pdf.

17Source: Fall 2010 Global Internet Phenomena Report, Sandvine, p.9.

0

5

10

15

20

25

30

Q22007

Q32007

Q42007

Q12008

Q22008

Q32008

Q42008

Q12009

Q22009

Q32009

Q42009

Q12010

Hours


18/47


.

3.4.2Sustained growth in peer-to-peer traffic

Peer-to-peer networks allow a distributed community of users to share digital content or resources,

including video.18 But contrary to a classic clientserver architecture in which all network content is

held on a server in a central location and accessed via a client on the end-users devices, peer-to-peer

resources are located in and provided by devices at the edge of t he network by peers. Peer-to-peer

quickly became a popular way for Internet users to share video and music files, as it was particularly

adapted to transfer large files over the Internet. Indeed peer-to-peer networks use the resources

provided by users (bandwidth, storage space, and computing power) to increase the total capacity of

the system at little or no cost to the originator of the peer-to-peer network. In contrast, in a typical

clientserver architecture, clients make demands on the system, but do not supply any resources (as

such, as more clients join the system, the owner must provide more resources to meet the demand, or

congestion will impact performance).

Peer-to-peer services themselves are not a recent Internet application: they have their origins in many

of the early Internet services, such as Internet Relay Chat (IRC) which was developed at the end of the

1980s. The widespread popularization of peer-to-peer goes back to Napster in 1999. Napsters success

was striking, as only two years after the launch of operations it was reaching 26.4 million users. As

much of the material distributed over Napster violated copyrights, the service was closed down after a

lawsuit instigated by the Recording Industry Association of America (RIAA).19 Nonetheless, peer-to-

peer networks have become more and more popular, and have overcome the perceived drawbacks of

Napsters network (centralized servers distributing the files and requests to the peers) by relying on a

decentralized or a hybrid architecture: without central servers, the technical risks are minimized,

operational costs are reduced (database management in particular is distributed to peers) and the

ability of authorities to shut down the service following legal decisions is much more limited. Kazaa

and Gnutella 0.6 are notable examples of such new peer-to-peer networks.20

As broadband access grew, and music began to shift to commercial sites such as iTunes, peer-to-peer

networks were increasingly used for the exchange of higher-bandwidth video traffic. The result of the

popularization of peer-to-peer was a significant change, in which users became content providers as

well as consumers, and the flow of content shifted from a centralized location to become more

distributed. Unlike the streaming of a video from a centralized website such as YouTube, the

downloading of the same video using a peer-to-peer network has a very different impact in terms of

network utilization: in this case the traffic goes from one or more end users to another end user (andboth ways, as peer-to-peer is based on reciprocal sharing).21

18Even though the rise in peer-to-peer traffic is relatively recent, peer-to-peer services themselves should not be considered to be a recent

Internet application. Peer-to-peer networks have their origins in many of the early Internet services: for instance, the Internet Relay Chat

(IRC), which was developed at the end of the 1980s.

19See http://papers.ssrn.com/sol3/papers.cfm?abstract_id=504062

20See http://en.wikipedia.org/wiki/Timeline_of_file_sharing

21The difference may be most clearly understood by comparing YouTube with a peer-to-peer network. In either case, the content may be

user-generated, but with YouTube the originator uploads it once to YouTube, from which others download it. In contrast, with peer-to-

peer, each time a user downloads the content it is resent from the originator (or other peers).
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=504062http://en.wikipedia.org/wiki/Timeline_of_file_sharinghttp://en.wikipedia.org/wiki/Timeline_of_file_sharinghttp://papers.ssrn.com/sol3/papers.cfm?abstract_id=504062


19/47


.

As a big proportion of traffic goes directly from ISP end users to ISP end users, and large file-sharing

traffic requires very high bandwidth, ISPs have responded by adapting their network policies and

architectures. In particular, they have had to find alternative routes to deliver this video traffic.

Combined with the general growth of Internet traffic, it started to make economic sense for ISP to

adopt solutions like secondary peering to address this traffic issue (see Section 4).

3.4.3Forecasts

Growth in video traffic is not expected to slow down in the medium term, and according to Cisco, the

sum of all forms of video (IPTV, video on demand, Internet video, and peer-to-peer) will account for

over 85% of global consumer traffic by 2013. Internet video excluding peer-to-peer (free or pay TV

and video on demand delivered on a PC or a TV) will account for over 55% of all consumer Internet

traffic by this time.22 It appears therefore that video traffic will be the major driver for the Internet

traffic growth in the coming years, in proportions that will require significant investments in Internet

infrastructures and technology development to carry the traffic and sustain the quality of service (viaan increase in available bandwidth).

It is interesting to note that in relative terms peer-to-peer is actually declining as a percentage of

overall IP traffic, in favor of video streaming from centralized servers. This evolution can be

explained by the growing availability of alternative ways to watch online premium content

(e.g. YouTube, iPlayer and Hulu.com) using centralized servers, and the relative drawbacks of peer-

to-peer downloads compared to video streaming. From video-streaming websites, content is available

with high quality, is often free, and can typically be downloaded quickly and easily. By contrast, peer-

to-peer content, while also free, is not immediately available (content is always downloaded in

random chunks, which makes it difficult to estimate the download completion time), there is no

quality guarantee, and users are subject to an additional risk resulting from possible copyright

violations.

3.5 Cloud computing services have become widely utilized

Online services have also emerged significantly over the past 15 years: examples include data storage,

and remote applications ranging from email or online photo albums, to suites of business applications

including spreadsheets and word processing. Because the content and applications are centralized in

the network, rather than distributed at its edges, this is often referred to as cloud computing.23Although the concept of cloud computing is not new, it has become more popular recently as the

barriers to take-up have been progressively lifted, thanks to the improvement of Internet access (e.g.

development of broadband, availability of cheap dedicated high capacity links), the improvement of

22Source:http://www.cisco.com/en/US/solutions/collateral/ns341/ns525/ns537/ns705/ns827/white_paper_c11-

481360_ns827_Networking_Solutions_White_Paper.html

23As defined by the National Institute of Standards and Technology, cloud computing is a model for enabling convenient, on-demand

network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that

can be rapidly provisioned and released with minimal management effort or service provider interaction. (See

http://csrc.nist.gov/groups/SNS/cloud-computing/)


20/47


.

security (in particular, enhanced encryption techniques), and the increase in awareness and confidence

in cloud-computing techniques.

YouTube is a good example of a popular cloud-computing application, where users store videos on

YouTubes servers, and access them remotely in a streaming mode. The email services offered by

Microsoft and others can also be seen as a very successful popularization of cloud-computing

services. For instance, Googles Gmail became available to the general public on February 7, 2007

(after an initial invitation-only beta release on April 1, 2004), and by December 2009 had already

gained 176 million users.24

This began a transformation of services that were formerly performed at the edge of the network and

perhaps involved some incidental Internet traffic, into ones that were performed in the network itself,

with integral Internet access and only intermittent processing at the edge. In addition to the examples

discussed above, applications such as word processing and spreadsheets began to be offered as

Internet services, threatening traditional software models. From the perspective of this paper, this

increased overall usage of the Internet, and also made consumers more sensitive to latency and other

quality measures that impacted their access to their cloud applications.

3.6 Internet connections are a key feature for many new devices and services

The past 15 years have seen the development (and the wide acceptance by the general public) of a

large variety of devices that are connected to the Internet to enable access to the new services

described above, as well as additional services that were unleashed in response to the new usage

patterns that resulted from these devices.

New connected devices

The release in 2007 of the first popular smartphone Apples iPhone marked the take-off of mobile

Internet access. More and more mobile devices then started to offer specific features adapted to

Internet browsing and data traffic (e.g. mobile applications). By June 2009, there were approximately

6.4 million active iPhone users in the US, with 88% of iPhone users accessing the Internet and 75% of

them downloading mobile applications.25 According to Validas, iPhone users in the US each

consumed 273MB of data per month on average, and 12% of iPhone users consumed at least 500MB

of data per month (as of February 2010).26

Most vendors today are offering similar large touch-screendevices with embedded Internet functionalities.

Netbooks are getting a mass-market status, and according to the research firm ABI Research, 35

million netbooks were shipped worldwide in 2009.27 Netbooks are specifically designed to connect

24Source: http://www.businessweek.com/technology/content/feb2010/tc2010029_989050.htm

25Source : http://blog.nielsen.com/nielsenwire/online_mobile/iphone-users-watch-more-video-and-are-older-than-you-think/

26Source : http://www.ilounge.com/index.php/news/comments/report-iphones-average-over-250mb-of-data-use-per-month/

27Source : http://www.computerworld.com/s/article/9140343/Linux_s_share_of_netbooks_surging_not_sagging_says_analyst
http://www.businessweek.com/technology/content/feb2010/tc2010029_989050.htmhttp://blog.nielsen.com/nielsenwire/online_mobile/iphone-users-watch-more-video-and-are-older-than-you-think/http://www.ilounge.com/index.php/news/comments/report-iphones-average-over-250mb-of-data-use-per-month/http://www.computerworld.com/s/article/9140343/Linux_s_share_of_netbooks_surging_not_sagging_says_analysthttp://www.computerworld.com/s/article/9140343/Linux_s_share_of_netbooks_surging_not_sagging_says_analysthttp://www.ilounge.com/index.php/news/comments/report-iphones-average-over-250mb-of-data-use-per-month/http://blog.nielsen.com/nielsenwire/online_mobile/iphone-users-watch-more-video-and-are-older-than-you-think/http://www.businessweek.com/technology/content/feb2010/tc2010029_989050.htm


21/47


.

remotely to the Internet, either via a WiFi connection or a 3G connection, and are thus optimized for

cloud computing with relatively little memory for installed software.

Tablets can also support new usage intimately related to Internet connection, and the launch of

Apples iPad and other similar products supports the view that expectations are high for this market,

with more than 14 million iPads being sold in 2010.28

New video-streaming devices have also emerged that exclusively rely on a broadband connection:

examples include Vudu, the Boxee Box (so-called Social Media Center), etc. These boxes directly

enable video streaming to the TV, and video file-sharing in competition with pay-TV, and are

beginning to be embedded into other devices such as game consoles or TV sets.

New online multimedia services

Second, many new services related to communications and multimedia have been released that

substitute traditional telecom services with online equivalents. This evolution can be explained by the

fact that an embedded Internet connection can enable advanced features and reduce the costs of

carrying traffic.

In addition to these services offering video on demand, broadcast TV services can also be provided

over an Internet connection (IPTV), in particular as part of the triple-play offerings of telecom

operators.

Voice over IP (VoIP), which consists in delivering voice communication over IP networks, has a

tremendous potential, as it allows customers to make free or very cheap voice calls. So far, qualityissues have restrained a wide acceptance of VoIP, but increases in fixed network capacity are

changing the situation at a rapid pace. Moreover, the development of cheap mobile broadband offers

(unlimited data plans with mobile subscriptions) opens the way to VoIP over mobile: for instance,

the Line2 application for iPhone enables users to call using the data connection instead of the

traditional minutes included in the bundle and paid for by the user.

In summary, the development of electronic devices and services relying on Internet access has

generated significant changes in the Internet ecosystem, as market players have adapted to this

demand increase by modifying and investing in their network architectures: rolling out mobile

broadband networks, installing additional network capacity, and developing new forms of

interconnection as described below.

3.7 Conclusion

As explained previously, and illustrated in Figure 3.8, Internet traffic and revenues have been fed by

the simultaneous growing demand for Internet-based services and the development of Internet-based

solutions to answer and sustain this demand.

28Source: Apple financial results


22/47


.

Figure 3.8: Supply and

demand feed the growth

of Internet traffic and

revenues [Source:

Analysys Mason]

It is not only Internet traffic that has exploded over the past 15 years: so has the Internet market as a

whole. It is interesting to note that, while there has been a significant increase in the revenues

generated by the e-commerce marketand to a smaller scale by the online advertising and the device

manufacturing markets there has been a more measured increase in revenues from Internet access

itself. As shown in Figure 3.9 below, the revenues per month per Internet user generated by e-

commerce and online advertising significantly outweigh those from Internet access.

Figure 3.9: Evolution of the Internet ecosystem in terms of revenues per Internet user per month [Source:

PWC, US Department of Commerce, Bureau of economic analysis, Euromonitor]

Mobilebroadband

Netbooks

Willingnessto pay forcontent

Peer-to-peer

Cloudcomputing

Videostreaming

Inte

rnettraffic

andreve

nues

Demand

Supply

0

20

40

60

80

100

120

2000 2001 2002 2003 2004 2005 2006 2007 2008

$perInternetuserpermonth

Online advertising

E-Commerce

Devices

Internet access


23/47


.

4 Evolution of Internet interconnection

As discussed in the previous section, the trends in Internet usage, particularly high bandwidth video

traffic, have led to significant increases in traffic. This, in turn, has led to a significant increase inbandwidth capacity demands, which provides a corresponding increase in costs, and concerns over

latency, since streamed video content needs to be delivered with minimal delay. At the same time,

content is increasingly monetized, providing content providers with a means to help ensure high-

quality delivery to their users. Here we describe how the main Internet stakeholders have adapted to

these new bandwidth demands, how new players have emerged, and how interconnection

arrangements have evolved accordingly, changing the traditional hierarchy of the Internet ecosystem

in the process.

4.1 Description of the early Internet

The Internet ecosystem 15 years ago had a rather simple structure, with relatively clear and

hierarchical relationships between the different players: end users and content providers bought

Internet access from their local ISP; ISPs bought Internet transit from a limited number of backbone

providers to get access to the whole Internet; and backbone providers peered with each other at the

US-based NAPs to ensure the full extent of their Internet connectivity. The result was a hierarchy

focused largely in the US, with the backbones at the top, interconnecting with one another via peering,

and selling the resulting Internet access to downstream ISPs, who in turn sold access to their end

users. Figure 4.1 below illustrates the architecture of the early Internet 15 years ago. Annex A

provides more background on the relationships between the different players.

Figure 4.1: Simplified architecture of the early Internet [Source: Analysys Mason]

Content provider &aggregator

Backbone1

End-user

Backbone2

Backbone3

ISP 1

ISP 4

ISP 3

ISP 2

Internet Access

Transit

Peering

All traffic passes through thebackbone providers, that carry

the traffic between the ISPs

NAP


24/47


.

In these years, the online market was still very young and dial-up was the predominant means of

Internet access, imposing technical constraints on the services that could be provided. For these

reasons, content providers had a limited size. In this context, we should note that it is only 15 years

ago that the first of todays major Internet companies started to emerge: Amazon (founded in 1994),

Yahoo (1995), eBay (1995), Google (1998), etc.

The Internet ecosystem 15 years ago had a rather simple structure characterized by relatively

clear and hierarchical relationships, with end users and content providers buying Internet access

from ISPs, ISPs buying Internet transit from backbone providers, and backbone providers

peering with each other.

4.2 Internet exchange points (IXPs)

The nature of Internet exchange has been changing for the past 15 years. When the NSF decided to

commercialize the Internet backbone, it designated four dispersed Network Access Points (NAPs), in

San Francisco (operated by PacBell), Chicago (BellCore and Ameritech), New York (SprintLink) and

Washington DC (MFS). The NAPs were used for public peering, whereby traffic was exchanged

using a common switch, which soon congested due to the large volume of traffic. As the historical

home of the Internet, a significant amount of international traffic passed through the US at the time,

either to connect with US providers, or for transit, sometimes back to the same country of

origination.29 At the same time, the operators of the NAPs had the potential to favor their own

services: for instance, when the MAE-East NAP was owned by MCI WorldCom, it required the use of

MCI circuits to access MAE-East services.30

As a result, a migration began towards peering at IXPs around the world. IXPs, such as Equinix,

began to open large data centers, where operators could host their servers and/or connect to one

another using direct cross-connections. In some countries, non-profit IXPs run by a consortium of

members were established today several of the largest IXPs use this model, including LINX in

London, and AMS-IX in Amsterdam. The move to IXPs enabled what is known as private peering,

whereby service providers do not use a common switch and thus can control congestion over each

bilateral link with other service providers. The service providers could also sell or purchase transit

over these direct cross-connections at the IXPs. As a result, these IXPs became focal points for the

entire Internet ecosystem, where the stakeholders could buy and sell their services to one another at

relatively low cost.

There are clearly strong network effects that support the growth of an IXP. The more service

providers there are in an IXP and the more traffic they exchange, the more attractive it is for other

service providers to locate at the same IXP in order to arrange connectivity with the largest number of

29As all international ISPs at the time had to connect to the US for international transit, they sometimes used these links to exchange

domestic traffic, in order to avoid the cost of directly connecting to each other ISP in the country at a time when many domestic telecoms

markets were not yet competitive. This flow of traffic out of the country and then back in is often referred to as tromboning.

30See: Internet Service Providers and Peering, William B. Norton


25/47


.

service providers. However, the growth in traffic described above has provided a strong

countervailing pressure that has caused IXPs to proliferate across populated areas of countries and

regions. Having a dispersion of IXPs has benefits for backbone providers in a peering relationship, in

terms of reducing latency, ensuring quality of service (via redundancy) and controlling costs:

First, where the two parties in a transmission are in the same region, having a nearby IXP enables

the traffic to remain local, rather than tromboning to a more distant location. This enhances the

quality of service thanks to reduced latency. A certain level of redundancy is also provided by

there being multiple IXPs in different locations in the same region, giving some protection to

connectivity in case of natural disaster, for instance.

Second, where the two parties in a transmission are in different regions, having dispersed IXPs

enables backbone providers to effectively share traffic loads, whereby one provider will carry the

originating traffic and the other will carry the return traffic.

This second benefit results from what is sometimes referred to as deflection routing or more

commonly hot-potato routing, whereby each backbone will exchange the traffic with a peer at the

earliest exchange point (for further details see section A.2.3 of the Annex). When the traffic ratio (the

ratio of traffic flowing between peers in one direction compared to the traffic flowing in the other

direction) is reasonably even, the networks will evenly share the costs associated with carrying traffic

exchanged by their users, and thus the agreement is considered fair.

As a result, in their publicly available peering policies, backbone providers may require

interconnection in up to six different points in the US, for instance (typically with dedicated

connections in IXP locations), in order to reduce carriage requirements and balance traffic loads.Backbone providers will also specify a maximum traffic ratio in order to limit the traffic imbalance

between them and fairly split the costs of carrying traffic.31 These types of requirements are designed

to reflect comparable costs and benefits, so that the peering arrangement is mutually beneficial to both

parties.

While IXPs have clearly been beneficial to backbone providers in lowering costs and making more

efficient use of capacity, they have also enabled other trends whereby providers rely less on

backbones, as discussed below.

The development of IXPs has benefited backbone providers by improving the quality of service

and reducing traffic carrying costs. However, the benefits rely on roughly even traffic ratios

between the peers, as any significant traffic imbalance jeopardizes the peering principle that

each party should bear a fair share of the costs of carrying the traffic.

31See Annex A for an example of a current peering policy.


26/47


.

4.3 Internet service providers (ISPs)

ISPs have reacted to the changing mix of traffic described above in several ways, each of which

reduces their reliance on traditional backbones for transit services:

The rise in peer-to- peer usage has dramatically increased the traffic that ISPs customers are

exchanging between each other (acting in the role of both consumers and suppliers of content).

This has led to what is referred to as secondary peering.

At the same time, the rapid growth of centralized video content and traffic in sites such as

YouTube has led ISPs to connect directly to content providers. This typically results in a form of

interconnection known aspaid peering.

We describe each of these changes in turn.

First, the changing mix of traffic led the largest ISPs to reevaluate their need to purchase IP transit

from upstream backbone providers. Traditionally, IP transit was used as a means to access traditional

content providers, which may be connected to the upstream transit provider or another backbone. On

the other hand, IP transit is not necessary to access the customers of another ISP, particularly if that

ISP is located in the same IXP. As a result, many ISPs began to peer with one another, as shown in

Figure 4.2.

Figure 4.2: The principle

of secondary peering

[Source: Analysys

Mason]

This had two impacts for the ISPs. First of all, it lowered the cost of transit, because the traffic

exchanged via secondary peering no longer needed to be sent through the existing transit

relationship(s). As transit is based on a measure of traffic volume, this lowered costs. One estimate is

that up to 40% of a typical ISPs traffic in the US was peer-to-peer that could be exchanged using

secondary peering, with a corresponding reduction of cost.32 In addition, secondary peering reduced

latency for peer-to-peer traffic, as the traffic passed directly between the ISPs rather than via one or

32The Evolution of the US Internet Peering Ecosystem, William B. Norton, Equinix White Paper 11/19/2003. Norton was the Co-Founder

and the Chief Technical Liaison for Equinix. (See http://www.equinix.com/pdf/whitepapers/PeeringEcosystem.pdf)

Backbone

2

Backbone

1

ISP 1 ISP 2

Peering

TransitTransit

Secondary

peering

IXP
http://www.equinix.com/pdf/whitepapers/PeeringEcosystem.pdfhttp://www.equinix.com/pdf/whitepapers/PeeringEcosystem.pdfhttp://www.equinix.com/pdf/whitepapers/PeeringEcosystem.pdf


27/47


.

more upstream backbones. ISPs that have engaged in such secondary peering are sometimes said to

purchase onlypartial transitfrom backbonesnamely only those destinations that they cannot access

via peering.

Second, the rapid growth of centralized video content and traffic in sites such as YouTube has led to a

different form of interconnection known aspaid peering. The genesis of paid peering between an ISP

and a content provider is similar to that of secondary peering between two ISPs, namely it is an

attempt to directly connect in order to reduce transit costs as well as the latency of transmissions.

However, there is a fundamental difference resulting from the ratio of traffic involved. When two

ISPs peer, it is likely that they will exchange roughly equal amounts of traffic, as their end users are

likely to have an equal propensity to generate and consume each others traffic, be it peer-to-peer or

interactive video games or just email. On the other hand, when an ISP and a content provider peer, the

traffic ratios will not be balanced, as for instance the request for a video consumes very little

bandwidth while the video itself takes far more bandwidth: we estimate that the streaming of a three-

minute video on YouTube generates around 35 times more downlink traffic than uplink traffic.33

As aresult, the ISP will deliver far less traffic to the content provider than it carries in return.

When the imbalance of traffic between service providers rises above a certain traffic ratio, the

convention is that the generator of the traffic pays the provider that carries the traffic for the

imbalance, in order to cover the cost of capacity needed. Nonetheless, paid peering is advantageous

for both parties. Previously, both the ISP and content provider used a transit connection to a backbone

to exchange traffic, and each party paid for the traffic that they sent and received. Now, by directly

interconnecting, they each lower their transit payments accordingly. Indeed, in this case the content

provider would have paid transit fees for the traffic that it delivered to the backbone, while the ISP

would have paid similar transit fees for receiving that traffic from the backbone. With paid peering,

the content provider pays the ISP directly for delivering the traffic, probably at a rate lower than it

would have paid for transit.

As a result of these actions on the part of ISPs to lower their transit fees, interconnection has begun to

look like Figure 4.3 below. In order to reduce transit fees, ISPs have begun peering with one another,

typically at IXP locations for efficiency. At the same time, content providers have begun to

interconnect directly with ISPs, using paid peering to compensate for the imbalance of traffic across

the connections. This not only further lowers the ISPs transit fees, but also results in fees from the

paid peering. These new interconnections come at the expense of backbones, which, as a result, endup selling less transit to the ISPs and content providers.

33The video streaming request sent by a user to YouTube generates around 430 kbytes of uplink traffic, while the resulting video stream

generates around 15 Mbytes.


28/47


.

Figure 4.3: Interconnection possibilities that emerged in the past 15 years [Source: Analysys Mason]

Many route around options have emerged for ISPs and content providers to carry Internet

traffic while avoiding Internet backbone transit costs. These options include secondary peering

between ISPs and partial transit agreements. Traffic imbalances have also led to the

development of paid peering, which is advantageous for ISPs and content providers.

4.4 Content providers

Paid peering with ISPs is only one of the means that content providers are using to lower their costs

efforts that go hand in hand with ways to reduce latency and thereby improve the experience for their

users. Another technique used for this purpose is called web caching. A web cache used by an ISP

stores a copy of a web page or video requested by an end user. The ISP can then use the cache to

satisfy subsequent requests for the same content, instead of reloading it from the original source. In

other words, if 500 users view the same popular video, it can be downloaded once from the original

content provider over the transit connection, and the next 499 times it will be viewed from the web

cache within the ISPs network, significantly lowering the amount of traffic over the transit

connection, and therefore reducing transit payments.

Web caching may be used by ISPs, as described above, but this technique is more typically used

either by third-party content delivery networks (CDNs) or by larger content providers, which are

effectively building their own delivery networks in order to deliver content closer to (or from within)

the ISPs networks. Commercial CDNs effectively compete with backbones in order to deliver their

clients content directly to ISPs. CDNs ensure that the content of their clients is distributed to end

users with good performance, independently of the location of the end user. To fulfill this role, a CDN

typically owns a network of caching servers, connected by fiber, which feed copies of content to ISPs

Content provider &

aggregator

Backbone1

End-user

Backbone

2

Backbone

3

ISP 4

ISP 3

ISP 2

IXP

Internet Access

Transit

Peering

Secondary peering

Partial transit Paid peering

ISP

Singleentity


29/47


.

at a limited number of points of presence (PoPs) where networks interconnect.34 The largest CDNs

include Akamai (founded in 1998) and Limelight (2001). According to one estimate, CDNs deliver

close to 10% of all Internet traffic.35 See Figure 4.4 below for a stylized diagram of how a CDN

interacts with the other players in the Internet ecosystem. The use of CDNs to carry content traffic

effectively reduces the traffic to be carried by the backbone provider, and avoids any traffic imbalance

issues. However, at the same time it reduces the transit revenues of backbone providers.

Figure 4.4: Structure of a content delivery network [Source: Analysys Mason]

Content providers themselves may use a similar configuration to build a content delivery network for

their own content. For instance, Google has embarked on a Google Global Cache project with similar

goals to a CDN, in this case to locate Google servers within ISP networks.36 The difference from a

CDN is that Google servers are dedicated to Google content. Other content providers such as

Microsoft are also building such networks to deliver content, such as Microsofts video games. The

difference in the traffic carried over the backbone can be significant: Arbor Networks estimates that

Google traffic represents at least 6% of global Internet traffic, and more than 60% of that traffic is

delivered to ISPs over direct connections rather than via third-party providers.37 Figure 4.5 shows how

the percentage of Google traffic using private peering has increased in recent years.

34CDNs usually try to connect directly to ISPs for cost and performance reasons, but may need in some instances to buy transit from

backbone providers (when they do not own the fiber network), and also buy transit from ISPs (when they cannot get free peering from

large ISPs).

35ATLAS Internet Observatory, 2009 Annual Report, Arbor Networks, Inc., University of Michigan, Merit Network, Inc., Pre-Publication Draft,

page 15.

36See http://www.afnog.org/afnog2008/conference/talks/Google-AFNOG-presentation-public.pdf

37 Source: How Big is Google?, Craig Labovitz, http://asert.arbortnetworks.com/2010/03/how-big-is-google/

Contentprovider

Backbone

ContentDelivery

Network

End-user

Backbonenetwork B

Backbonenetwork A

RegionalISP

Mobileoperator

Aggregation & AccessAggregation & Access

RegionalISP
http://www.afnog.org/afnog2008/conference/talks/Google-AFNOG-presentation-public.pdfhttp://www.afnog.org/afnog2008/conference/talks/Google-AFNOG-presentation-public.pdfhttp://www.afnog.org/afnog2008/conference/talks/Google-AFNOG-presentation-public.pdf


30/47


.

Figure 4.5: Percentage of

Google traffic using

direct peering [Source:

Arbor Networks38]

Even smaller content providers can also use technology to lower their costs, avoiding not just transit

charges, but also the use of CDNs. For instance, Vudu delivers high-quality video on demand to end

users TVs via a dedicated set-top box that is external or integrated into a TV or Blu-ray player. Each

device has a large hard drive, which contains the first few seconds of each title in order to enable the

title to begin as soon as it is chosen. The rest of the chosen content is pieced together via an Internet

connection to the device, using peer-to-peer networking from other Vudu devices that have stored thatcontent. As a result, a user is able to watch high-definition movies with no delay with as little as a

4Mbit/s broadband connection. In this way, Vudu uses its customers own broadband connections to

deliver content to other users in lieu of a more expensive CDN or backbone service.

As a result of the volume of traffic that they deliver, and the associated cost of delivering that traffic,

content providers have a strong incentive to build out their networks in order to reduce their transit

requirements and deliver the traffic directly to the ISP network, if not from within the ISPs network.

As a result of the increasing monetization of content, providers also have the means to engage in such

network build-outs. But even if CDNs are increasingly used or built to deliver Internet content to

ISPs, in particular to deliver the growing share of video traffic for which caching is appropriate,

backbones and/or ISPs still provide significant infrastructure to deliver the content. As explained

earlier, CDNs typically distribute their caching servers into specific PoPs to connect with backbones

and ISPs networks. The selection of these points by the CDNs is optimized in terms of coverage, i.e.

CDNs will primarily set up PoPs in regions concentrating significant amounts of Internet traffic.

However, backbones and ISPs then ensure the delivery of content to the end users from these PoPs to

the end users, regardless of final location.

38 http://gigaom.com/2010/03/17/stat-shot-googles-growing-infrastructure-advantage/

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

A

pr-08

Jun-08

Aug-08

O

ct-08

Dec-08

Feb-09

A

pr-09

Jun-09

Aug-09

O

ct-09

Dec-09

Feb-10

Percentage
http://gigaom.com/2010/03/17/stat-shot-googles-growing-infrastructure-advantage/http://gigaom.com/2010/03/17/stat-shot-googles-growing-infrastructure-advantage/


31/47


.

Overall, the traffic carriage effort supported by backbones and ISPs is significant in comparison with

the CDNs. CDNs typically connect at tens of PoPs: for instance, Limelight notes that in 2010 its

network consisted of 75 PoPs where it interconnected globally, in order to deliver traffic to more than

900 ISPs and backbones. This is broadly similar to the number of PoPs that each single major

backbones typically possesses.39 However, the backbones and/or ISPs carry the Internet traffic further

into the network, in order to reach the 24 000 central offices40 counted in the US (as of February

2010), from where it is carried over the last mile to end users for DSL services. 41 On average, each

PoP may connect to up to 300 or more central offices, requiring significant network resources to carry

the content from the CDN to the last-mile connection.

In conclusion, as explained in the previous section, there are now many situations where content

providers can bypass the backbone providers transit fees. Content providers also have the opportunity

to rely on their own private network or a third-party CDN to carry their traffic to the ISP. Either way,

the consequence has been the continuing trend of providers relying less on backbones, which may

now be avoided for a significant amount of traffic between content providers and end users, even ifthey keep a key role in delivering content to and from end-user networks.

Content providers can employ many different options to carry their Internet traffic to the ISP

(e.g. using CDNs or building their own private networks), and can leverage their expanding

market power and financial resources to rely less on backbone providers.

4.5 Impact on backbone providers

From the backbone providers point of view, these market evolutions have had impacts at two levels :

First, changes in traffic patterns have led to changes in the arrangements between backbone

providers and their traditional customers, the ISPs and content providers. These changes have

impacted the traditional hierarchy of the Internet ecosystem and put downward pressure on transit

prices.

Second, although many traditional backbone customers are relying less on backbones to

accommodate changing traffic patterns, the same trends are impacting the traffic that they still

deliver to the backbones as customers, which puts pressure on the relationship between backbone

39The number of PoPs on backbones is usually not publicly revealed. However, studies providing solid estimates suggest that large

backbones in the US typically own around 50 to 100 PoPs

(Seehttp://www.cs.washington.edu/research/networking/rocketfuel/papers/sigcomm2002.pdf )

40Central offices are network facilities where end users lines get locally connected to the core network (local loop) for xDSL services. To

deliver the traffic to end users, ISPs carry aggregated Internet traffic to the central offices, where it is then distributed to the end users

over the last portion of the network (last mile).

41Source: FCC, Trends in Telephone Service, FCC, September 2010

(http://www.fcc.gov/Daily_Releases/Daily_Business/2010/db0930/DOC-301823A1.pdf ). Similar arrangements are true for cable

networks.
http://www.cs.washington.edu/research/networking/rocketfuel/papers/sigcomm2002.pdfhttp://www.cs.washington.edu/research/networking/rocketfuel/papers/sigcomm2002.pdfhttp://www.cs.washington.edu/research/networking/rocketfuel/papers/sigcomm2002.pdfhttp://www.fcc.gov/Daily_Releases/Daily_Business/2010/db0930/DOC-301823A1.pdfhttp://www.fcc.gov/Daily_Releases/Daily_Business/2010/db0930/DOC-301823A1.pdfhttp://www.fcc.gov/Daily_Releases/Daily_Business/2010/db0930/DOC-301823A1.pdfhttp://www.fcc.gov/Daily_Releases/Daily_Business/2010/db0930/DOC-301823A1.pdfhttp://www.cs.washington.edu/research/networking/rocketfuel/papers/sigcomm2002.pdf


32/47


.

providers. In particular, the increase in content traffic being delivered to end users is impacting

the conditions under which backbones agree to peer with one another.

With respect to the first issue, the changes impacting ISPs and content providers as described in the

previous sections lead to a reduction in demand for transit. This is due to the changes with regards to

backbones customers at both ends of the value chain: namely ISPs and content providers, which are

able to self-supply a growing portion of their needs, and which are also entering into multiple

agreements with peers and other suppliers. This is true not just at a national level, but also

internationally as more and more traffic is regionalized through large IXPs that reduce a reliance on

international transit for traffic exchange. As a result, this puts pressure on transit prices (shown below)

as backbones effectively face greater competition from the self-supply of their customers.

Further, the changing nature of Internet traffic also impacts the relationships between the backbone

providers themselves. In particular, with the increase of bandwidth-intensive content, particularly

video, comes an increase in an imbalance of traffic sent from the content providers to the customers of

the ISPs. As a result, without a change in the mix of customers, backbones with content providers as

customers will send more traffic to backbones with ISP customers. As noted in Section 4.2, peering

policies typically specify a maximum traffic ratio governing the traffic flow between peering partners;

this has been a feature of peering policies at least since backbones began making their peering policies

publicly available a decade ago.42 Increases in traffic ratios resulting from delivering content can

begin to overrun the traffic ratio requirements in the corresponding peering policies.

With the default hot-potato routing that is common for peering arrangements, the sender of the traffic

quickly hands over the traffic to the receiving network for the latter to carry across the network to the

recipient, as described below in Annex A.2.3. As a result, a high traffic ratio can impose costs on thereceiving network, as it carries far more traffic than it hands off to the other network in return.

Specifically, the backbone providing the content will quickly hand it to the backbone with ISP

customers that are receiving the content. As a result, the latter backbone has greater network costs in

transporting the incoming traffic from the backbone providing the content, which could be seen as a

cost subsidy from the former to the latter.

As these costs result from content being exchanged near to the source under hot-potato routing, one

solution is what is known as cold-potato, or best-exitrouting, in which the sending network keeps the

high-bandwidth traffic on its own network longer before exchanging it closer to the consumers.

However, given the ratio between PoPs and the number of distribution points in an ISPs network,

cold-potato routing may still not alleviate the imbalance of costs, implying that the ISP may subsidize

the delivery of traffic from the content provider, on behalf of its customers who may already be

paying for the relevant content (directly or via associated advertising).

42For instance, in January 2001 WorldCom introduced a publicly-available peering policy specifying the requirements that a network must

meet in order to peer with World Com. This policy included a clause with the following requirement: The ratio of the aggregate amount of

traffic exchanged between the Requester and the WorldCom Internet Network with which it seeks to interconnect shall be roughly

balanced and shall not exceed 1.5:1.

See http://drpeering.net/a/Internet_Service_Providers_and_Peering___Peering_Policy_files/UUNET%20Global%20Site%20WorldCom%

20Policy%20for%20Settlement-Free%20Interconnection%20with%20Internet%20Networks.htm


33/47


.

Other solutions to the imbalance of costs issue include paid peering, discussed above, or even moving

to a transit relationship whereby the backbone whose traffic causes the imbalan

analysys mason domestic peering paper

Documents