strategic technology infrastructure for regional ... · volume 6: leveraging advanced optical and...

Strategic Technology Infrastructure for Regional

Competitiveness in the Network Economy

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

eCorridors Program

2003 Virginia Polytechnic Institute and State University, Blacksburg, VA. All rights reserved.


i

Preface This series of reports, entitled Strategic Technology Infrastructure for Regional

Competitiveness in the Network Economy and packaged in eleven Volumes, is the

culmination of a dedicated effort of the following individuals and organizations. Each

Volume can be viewed as a stand-alone publication; however, it should be noted that

each Volume was written in the context of the overall project. The project utilized the

Southside and Southwest Virginia regions as a model for a low-cost Geodesic Mesh

network design and viable financial model that could be replicated in any region of the

U.S.

Volumes 1) Rationale, Environment, and Strategic Considerations

2) Connecting the Regional Infrastructure to National and International Networks

3) A Fiber Optic Infrastructure Design for Southside and Southwest Virginia

4) Fiber Optic Infrastructure Design Guide

5) Financial Feasibility and Investment Rationale

6) Leveraging Advanced Optical and Ethernet Technologies

7) Speculative and Alternative Technologies

8) Community, Applications and Services

9) Demographics for Southside and Southwest Virginia

10) Health Information Technology and Infrastructure

11) Education in the 21st Century

Volume 1: Rationale, Environment, and Strategic Considerations

ii

Acknowledgements The following individuals and organizations contributed to the development and

preparation of this series of reports.

Allen, Morgan

Arellano, Christian

Aughenbaugh, John

Bevis, Jeff

Blythe, Erv

Bohland, James

Bottom, Beth

Bowden, Phillip

Brown, Eric

Charlton, Garland

Chen, Daniel

Cohen, Marc

Colbert, Joy

Croasdale, Hud

Crowder, Jeff

Dalton, Jody

de Vries, Marten

Dwyer, Sharon

Fisher, Tommy

Franklin, Nancy

Gaylord, Clark

Hach, Richard

Hall, Shannon

Hares, Glynn

Harris, Carl

Hey, Bryan

Hoover, Maynard

Horton, Helen

Jones, Brian

Jones, Doug

Kanter, Theresa

Kidd, Jeff

Lee, Steven

Lilly, Judy L.

Martin, David

Martin, Evelyn

Mathai, Mathew

McCann, Jessica

Morrison, Brandon

Neidigh, Brenda

Nichols, John

Pelt, Ranson

Perry, Mike

Pheley, Al

Plymale, V. Jean

Plymale, Bill

Pollard, John

Rodgers, Pat

Sanghvi, Harsh

Shepherd, Scott

Sheppard, Scott

Shumaker, Richard

Stewart, Jeb

Stock, Doris

Tyree, Charles

Waddell, Bobby

Wenrich, John

Woods, Cindy

Zirkle, Mary


iii

Organizations and Companies 3com

Adelphia

Advance Fiber Optics

Advanced Network Infrastructure & Services, VA Tech

AEP

AFL Telecommunications

Alcatel

Anderson & Associates

Asia Venture Partners

AT&T

Avante

Bristol Virginia Utilities Board

Celion

Center for Wireless Telecommunications, VA Tech

Chamber of Commerce, Richlands

Chilson Enterprises

Cisco Systems

Corning Cable Systems

Cox Communications

Danvilleonline.com

Dominion Telecom

Economic Development Assistance Center, VA Tech

EngHouse Systems

Enterasys

Extreme Networks

Economic Development Assistance Center, VA Tech

Floyd County High School

Floydva.com

Force 10

Foundry

Future of the Piedmont Foundation

Gamewood, Inc.

GeoTel

Grant County Public Utility District

Hatteras

Hewlett-Packard

IBM

Institute for Advanced Learning and Research, VA Tech

Institute for Connecting Science Research to the Classroom, VA Tech

ION Consulting

KMI Corporation

LENOWISCO

Level 3 Communications, Inc.

MapInfo

Manticom

Marketing Dept., VA Tech

Micrologic, Inc.

Nexans

Nortel

Old Dominion Electric Coop

Pirelli

Prince Edward County Office of Economic Development

Progress Telecom

Qwest

RACO, Inc.

Rinderva.com

Riverstone

Salira

Sprint

Terabeam

Urban Affairs and Planning Dept., VA Tech

Valleynet

Verizon

Wiltel

Worldcom

Worldwide Packets


iv

Table of Contents Introduction .....................................................................................................................1 Historical Perspective......................................................................................................3 What is Metropolitan/Regional Ethernet? ........................................................................6

Ethernet in the First Mile...........................................................................................7

Backbone Networks for Non-Ethernet Broadband Access......................................11

Interconnecting Service Provider Networks ...................................................................14

Cross-connects as Meet Points ..............................................................................14

Packet Switching or Multiplexing as Meet Points ....................................................16

Collocation at Meet Points......................................................................................16

Ethernet-Based Internet Exchange Points ..............................................................17

Cost and Manageability Benefits of Metro Ethernet .......................................................19 Limitations of Current Metro Ethernet Technology.........................................................22

Slow Recovery From Link Failures .........................................................................22

Lack of Remote Fault Isolation ...............................................................................24

Lack of In-Service Performance Monitoring and OAM ............................................25

Limited VLAN Tag Space .......................................................................................25

Spanning Tree Inefficiencies on Highly Meshed Networks .....................................26

Lack of End-to-End Service Guarantees.................................................................26

Is Metro Ethernet Ready for Prime Time?...............................................................27

Best Practices for Metro Ethernet Networks ..................................................................29

Architecture: Link Layer or Network Layer Switching..............................................29

Spanning Tree Configuration..................................................................................30

Forwarding Table Considerations...........................................................................31


v

Protocol Filtering ....................................................................................................32

Rate Limits on Broadcast and Multicast Frame Flooding ........................................32

IP Multicast Frame Flooding and Rate-Limiting ......................................................33

Quality-of-Service Controls.....................................................................................33

Acronym Glossary.........................................................................................................35 Acknowledgements .......................................................................................................38 References....................................................................................................................39


vi

List of Figures Figure 1: MSAP extending access network .....................................................................8 Figure 2: MSAPs connecting Ethernet rings ....................................................................9 Figure 3: Schematic of traditional DSL access network .................................................11 Figure 4: MSAP connecting multiple ISPs and access networks ...................................12 Figure 5: Schematic of cross-connect pedestal as meet point .......................................15 Figure 6: Collocation at meet points ..............................................................................17 Figure 7: Link Aggregation between MSAPs .................................................................23


1

Introduction Telecommunications services delivery over the last ten years has been dominated by

digital circuit-based technologies such as Time Division Multiplexing (TDM) and

Synchronous Optical Network (SONET). These technologies are readily adapted to

large-scale voice communications, as they effectively multiplex resources with fixed

units. For the same reasons, they are less suited for data communications, where the

resource demands of the applications are highly variable. Packet switching technologies

such as Frame Relay and ATM played an important role in the unprecedented growth in

the data telecommunications market, ushered in by the rise of the Internet. With the

realization of digital voice and video transmission, the efficiencies of packet switching

can also be applied to these applications.

Inarguably, all of these technologies have been extraordinarily effective. Yet, they where

designed and developed for a very different telecommunications economy than that

which exists today. In today’s telecommunications market, service providers must be

agile to seize opportunities. The Internet and all things around it move at a very fast

pace. Service providers must be able to scale capacities upward to meet surging

demands in order to remain competitive. Today’s service provider cannot assume a

decades-long return on capital investments in switching, multiplexing, and line

termination equipment.

While SONET, ATM, and Frame Relay will continue to play an important role in

telecommunications for many years to come, increasingly, service providers are looking

to Ethernet technology as the platform for cost-effective delivery of converged voice,

video, and data telecommunications services. Originally developed for use in local area

network environments, Ethernet has in recent years evolved such that it is a cost-

effective, robust, scalable, manageable platform for metropolitan and regional

telecommunications. The ubiquity of Ethernet technology in enterprise networks and its

increasing role in metropolitan networks has created an enormous and highly

competitive market for Ethernet equipment. This, along with the relative simplicity of the

technology, has led to twenty-year history of ten-fold increases in link capacities for

approximately three times the cost of the preceding generation of equipment.[4]


2

Furthermore, the near-total market adoption of each new generation of Ethernet

technology has ultimately driven down costs even further.

In this report, we examine the role that Ethernet technology can play delivering

telecommunications services on a metropolitan and regional scale. We examine the

case in favor of so-called “metro Ethernet” networks, and consider the foremost

limitations of the current generation of metro Ethernet equipment.


3

Historical Perspective Early wide area data telecommunications networks utilized modems on dial-up or leased

telephone lines. The point-to-point nature of such services greatly limited scalability,

since a new line was needed for each concurrent data connection to another remote

location. True scalability in wide-area data telecommunications networks was ushered

in by packet switching. Packet switching protocols such as X.25 became popular

because a single dial-up or leased line connection to the network allowed

communication with multiple remote locations via the concept of virtual circuits.

The first packet switching protocols were burdened with the need to operate effectively

over analog leased lines with very high bit-error rates. The X.25 protocol and its many

predecessors and relatives were greatly complicated by error detection and correction

mechanisms. With the advent of digital transmission lines, the need for a lightweight

packet switching protocol resulted in the development and large scale deployment of

Frame Relay. Frame Relay networks retained the fundamental label-swapping

techniques of X.25 virtual circuit switching while dispensing with the error correction

mechanisms. The simplicity and elegance of Frame Relay allowed it to easily operate at

speeds up to 1.5 megabits per second, which was quite impressive at the time.

Motivated by a desire to converge applications such as voice, video, and data

communications on the same network, and to meet the future needs for increased

bandwidth, in the late 1980s the telecommunications industry developed specifications

for Broadband ISDN (B-ISDN). The B-ISDN specification consisted of two significant

components: Synchronous Digital Hierarchy (SDH, known in the U.S. and hereafter in

this document as SONET) and Asynchronous Transfer Mode (ATM).

SONET provided a robust ring-based architecture for synchronous digital transmission

over fiber-optic cable. SONET was critical to meeting the capacity demands imposed by

rapid growth and competition in the long distance voice market, as well as the

unprecedented demand for bandwidth that followed the privatization and subsequent

commercialization of the Internet.


4

ATM was intended to address the inefficiencies of time division multiplexing (TDM) that

were inherent to SONET and its T–1 roots. Through the use of statistical multiplexing

and virtual-circuit-level class-of-service parameters, it allowed delay sensitive traffic

classes such as voice and video to be mixed freely with other traffic types on a common digital transmission line. Proponents of ATM believed that it would become the

fundamental means of supporting voice, video, and data telecommunications.

Carrier deployments of ATM in the mid-to-late1990s further fueled the rapid growth of

the Internet and the use of the Internet Protocol (IP) for intranet, extranet, and other

applications, by providing more bandwidth and more flexibility than had previously been

available. ATM also contributed to the continued success of Frame Relay, by providing

a scalable backbone network that was interoperable with Frame Relay and would allow

carriers to meet the growing demands for Frame Relay service that accompanied the

rise of the Internet.

While ATM continues to be an important component of carrier networks, it did not

achieve the goal of convergence for which it was originally intended. The success of

ATM in achieving convergence depended on end-to-end deployment of the technology.

In particular, this meant that the enormous installed base of personal computers in

enterprise networks would need to be directly connected to local area ATM networks in

order to fully leverage the capabilities of ATM. However, in local area networks, there

was already a dominant technology that would not be easily displaced: Ethernet.

When ATM emerged as a potential LAN technology, it offered significant advantages

over Ethernet. At that time, Ethernet was relatively primitive, relying on broadcast

media, and providing only 10 megabits per second of bandwidth shared between all

users on the LAN. Furthermore, Ethernet provided no capability for differentiated levels

of service that would allow mission-critical or time-sensitive applications to be prioritized

above routine traffic on the network. Ethernet networks were interconnected using

transparent bridges and multi-protocol routers, which partitioned the Ethernet into

smaller shared segments. Partitioning the network in this manner added stability and

scalability, at the expense of greater end-to-end packet delay, and increased packet

loss. Despite these shortcomings, Ethernet was inexpensive and easy to implement and

was adopted by enterprise networks of all sizes.


5

The installed base of existing shared Ethernet might not have precluded ATM from

taking over the LAN environment. However, the development and rapid deployment of

Ethernet switching technology and 100 megabit per second Ethernet delivered a

crushing blow to any hope that ATM might one day rule the enterprise network

environment. Ethernet switching was a relatively simple variation on the transparent

bridge. Advances in semiconductor technology enabled the development of inexpensive

high-density Ethernet switches that could be used to replace existing shared hubs. This

allowed the existing Ethernet base to migrate to networks providing much greater

amounts of bandwidth, while not requiring wholesale replacement of existing

components. In particular, existing desktop computer hardware and software could

continue to be used on a switched Ethernet network. Fast Ethernet, as the 100 megabit

per second variant is known, provided the means to increase the bandwidth on

backbone segments by an order of magnitude, in addition to providing very high capacity

links for network intensive server applications.

Enterprise network managers found that by simply eliminating congestion on their

networks, virtually all applications could be made to work successfully, without the need

for ATM’s advanced traffic management capabilities. Any remaining hope for ATM in the

enterprise was lost as Ethernet switches evolved to include features such as traffic

prioritization, rate limiting, and advanced queuing. These features are critical to

supporting real-time applications such as voice and video. The maximum Ethernet link

speed also increased by an order of magnitude, to 1000 megabits per second (1 gigabit

per second). The extraordinary capacity and advanced traffic management capabilities

of Ethernet obviated any need for a technology other than Ethernet at the data link layer

of the enterprise.

Indeed, by the turn of the century, rapid advances in Ethernet technology made it

feasible for use in metropolitan area networks where SONET, ATM, and Frame Relay

technologies ruled. Today’s Ethernet technology supports line speeds from 10 megabits

per second to 10,000 megabits per second (10 gigabits per second). Only SONET itself

offers line speeds that are comparable to Ethernet, but the inefficiencies of the time

division multiplexing inherent to SONET combined with the historically higher capital

costs for SONET equipment make Ethernet a far more cost-effective alternative in

metropolitan and regional area networks.


6

What is Metropolitan/Regional Ethernet? A metropolitan or regional Ethernet (known herein as a “metro Ethernet”) is a packet

switched network that employs Ethernet technology for wide-area connectivity,

especially within a metropolitan area or at a similar regional scale. Metro Ethernet

services are typically used by enterprise networks and access service providers for

connectivity to the public Internet and to extend the functionality of corporate networks

between geographically separate sites.

Nodes in a metro Ethernet are switches operating at either the OSI data-link layer or at

the OSI network layer (where the Internet Protocol is the universal network-layer

protocol choice). Often, some combination of the services provided by each of these

OSI layers is used. Links between nodes in a metro Ethernet are typically a physical

point-to-point connection, provisioned over fiber or optical services (such as DWDM),

and can operate at any Ethernet line speed (10 megabits per second to 10,000 megabits

per second) depending on the service requirements.

Typically, metropolitan and regional fiber networks have ring architectures. While metro

Ethernet can successfully operate over ring topologies, today’s standards-based

Ethernet is better suited to mesh topologies. Consequently, much of the emphasis in

metro Ethernet development is in efficiently supporting ring topologies with a robust,

highly-resilient Ethernet technology.

Metro Ethernet network services are provisioned using virtually any combination of

logical point-to-point, point-to-multipoint, or multipoint-to-multipoint configurations, over

links operating at any Ethernet line speed. Bandwidth can be allocated to such services

in increments as small as 1 kilobit per second. Bandwidth can be dedicated or shared

between multiple service users.

Quality-of-service is implemented in metro Ethernet networks using a combination of

many techniques operating at both the data-link layer and the network layer. These

techniques include most of the same capabilities found in Frame Relay and ATM


7

networks, including packet classification, marking, rate limiting or policing, and transmit

queue scheduling with multiple queues.

Some examples of common and proposed uses of metro Ethernet networks are:

• Residential and commercial Internet access (so-called Ethernet in the First Mile)

• Backbone networks for other broadband access technologies (e.g. DSL, cable

modem, wireless broadband)

• Corporate LAN extension

Ethernet in the First Mile

Residential and commercial Internet access can be delivered using metro Ethernet

networks. In this model, Ethernet connections are extended to the customer premise

using a 1000Base-X Ethernet connection over fiber-optic cable. Ethernet switches are

located in central-office-like facilities known as Multimedia Service Access Points

(MSAP) or in fiber termination pedestals located in utility easements. Figure 1 shows an

Ethernet switch located in a community MSAP that is providing 1000Base-X Ethernet

connections to homes and businesses in the local community.


8

Ethernet

Internet

MSAP

Router

MSAPIP PhoneWireless

SONET ADM

EthernetSwitch

EthernetSwitch

ONT

10/100/1000TX

1000Base-LX

1000Base-X

Figure 1: MSAP extending access network

Using the regional fiber architecture recommended in this report, Figure 1 shows the

metro Ethernet network extending the access network in the community to a distant

MSAP (perhaps in a larger city) where an Internet Service Provider can deliver high-

capacity access to the Internet using prevailing SONET technologies.

Because of the prevalence of ring topologies in metropolitan and regional fiber networks,

Ethernet in the first mile will typically be implemented using a combination of MSAP

facilities and smaller distribution switches located in fiber termination pedestals along

utility easements. As shown in Figure 2, small Ethernet switches are arranged on fiber

rings passing through residential communities and commercial districts. The logical ring

topology minimizes the number of fiber pairs consumed on the physical fiber ring. These

smaller distribution switches are located close to customers, such that the costs for

additional fiber “laterals” to reach each customer premise are substantially lower.


9

1000Base-X

1000Base-X

Internet

MSAPMSAP

1000Base-X1000Base-X


Figure 2: MSAPs connecting Ethernet rings

A community may have multiple logical Ethernet rings providing access services in

different areas of the community. In this case, the MSAP typically acts as the hub of the

community-area distribution networks. Leveraging the regional fiber infrastructure

proposed in this report, each MSAP is interconnected to other MSAPs, allowing Internet

and application service providers to be located wherever it is most advantageous.

The IEEE Ethernet in the First Mile working group (IEEE 802.3ah) is drafting

specifications to make Ethernet-based access networks scalable, manageable, and fault

tolerant. Technical proposals for OAM (operations, administration, and management),

customer premise network termination, and for both fiber- and copper-based physical

layer specifications are being considered by the working group.


10

Efforts are underway in the IEEE 802.17 Resilient Packet Ring working group to define

fault-tolerant ring standards for Ethernet that will ultimately provide a robust architecture

that meets or exceeds the resiliency of venerable SONET ring architecture.

The Multimedia Service Access Point is described in greater detail in the section entitled

Interconnecting Service Provider Networks.


11

Backbone Networks for Non-Ethernet Broadband Access

In small communities, broadband access providers employing DSL, cable modem, or

even wireless broadband technologies can exploit a regional fiber infrastructure and

metro Ethernet to reduce operating costs, and offer revenue-generating broadband

access services – even on a small scale.

As shown in Figure 3, traditional approaches to deploying DSL technology have

employed SONET-based ATM backbone networks interconnecting telco central offices.

The DSL access multiplexer (DSLAM) used to provide connections to residential or

commercial subscribers is connected to this ATM backbone network. The ATM

backbone provides the means of interconnecting service points to backbone resources

such as tier 1 Internet service providers and application service providers. Because of

the high infrastructure costs associated with the SONET/ATM architecture, these

solutions are not cost effective for smaller communities, where the potential subscriber

base is small.

ATM

Internet

ATMSwitch

Central Office

Router

ATMSwitch

POP

DSL Modem

IP Phone Wireless

SONET ADM

DSLAM

Figure 3: Schematic of traditional DSL access network


12

Using the community MSAP model and regional fiber infrastructure described in this

report, it is feasible for even a small rural telephone cooperative to cost-effectively

deploy DSL Internet access services. In the lower-left of Figure 4, we see the central

office continuing to serve as the termination point for residential and commercial DSL

connections. Using 1000Base-X Ethernet, the DSLAM connects to the community

MSAP, where access customers are interconnected with an Internet service provider.

The MSAP containing Internet service providers (labeled “ISP A” and “ISP B” in Figure

4) could be co-located in the same building facility as the telephone cooperative’s central

office, or it could be in some other part of the region, where the costs for traditional high-

capacity SONET-based access to the Internet are lower.

1000Base-X

MSAP


Central Office

DSLAM

1000Base-X

ISP A ISP B

InternetInternet

Broadband Wireless

1000Base-X

EthernetSwitch

Figure 4: MSAP connecting multiple ISPs and access networks


13

Figure 4 also illustrates Ethernet-based network access, DSL access, and wireless

broadband access all coming together at the community MSAP. This emphasizes a key

function performed by the MSAP: interconnection.


14

Interconnecting Service Provider Networks Both the Internet and the public switched telephone networks are intricate meshes of

connectivity between different service provider networks. Interconnection allows service

providers to specialize in network access or higher-layer services, extends the potential

market for all service providers, provides better performance for traffic between these

networks, and reduces the cost of upstream connectivity to higher-tier providers. There

are two basic strategies for interconnecting between such networks: private direct

connections and connections at common meet points. When there is more than one

other service provider to connect with, the expense of having direct connections to each

can be significantly more than connecting to all or most of them at a common meet point.

One of the motivations for direct connections is the desire for service providers to

exercise more control over resources and limit the exposure to risk from other entities

managing those connections. In order to engender the trust of numerous service

providers, meet points are best operated by neutral entities. Such neutrality addresses

the concern over potential competitors having control over a service provider’s critical

interconnection resources.

The technologies for these meet points may be variable, both in terms of the physical

medium and the link-layer technologies used. While the interconnection medium could

be copper for some applications, we will focus on fiber as the principal physical

interconnection medium. The choice of link-layer is independent from the perspective of

interconnection methodology, though the most cost-effective and suitable choices will be

important to successful implementation.

Cross-connects as Meet Points

The simplest form of a common public meet point is a cross-connect pedestal in a public

right-of-way. In such a scenario, service providers need to make only one fiber build of

sufficient capacity to this meet point. Having made this investment once, a service

provider can then connect with any others who have likewise constructed facility to this


15

point. This may reduce costs significantly over constructing separate facilities to each

potential other service provider.

The value of the cross-connect meet point may be increased by having numerous such

meet points distributed throughout a geographic region. The entity operating the cross-

connect point could connect these distributed meet points via fiber optic cable, and lease

access to this dark fiber facility to further reduce the costs to service providers who are

closer to some of these. This distributed cross-connect extends the reach of service

providers beyond what they may have otherwise been able to cost-effectively construct.

In the model of cross-connect meet points, each connection between service providers

is still a dedicated connection and still consumes resources from switching electronics,

lasers, etc. For N service providers connecting at such a meet point to each other, this

is a total of N(N-1) such connection points; each of the N service providers would have

N-1 connections to deploy this “full mesh” approach. The cost of the connecting

electronics may still dominate the cost for such interconnections. If the meet point could

more efficiently use these connections via shared media, this may further reduce the

costs associated with service provider interconnection.

Provider A

Provider C

Provider D

Provider BPedestal provides full mesh cross-connect between providers.

Patch bay

Provider A Provider B Provider C Provider D

Figure 5: Schematic of cross-connect pedestal as meet point


16

Packet Switching or Multiplexing as Meet Points

Instead of having dedicated facilities for each possible connection between service

providers at meet points, the meet point operator could offer packet-switching or

multiplexing services. Using such services, the N service providers may require only

one connection each in order to exchange traffic with any of the other service providers.

The multiplexing technology could be ATM, Ethernet, WDM, IP, or any of a number of

other such technologies. All of these would more efficiently use the physical connection

to the meet point, with the trade-off that services would need to be compatible with the

chosen multiplexing technology. Most multiplexing technologies, such as SONET or

WDM, are based on dedicating virtual resources, for example time slices or frequencies,

respectively. To use these dedicated virtual resources, these technologies still may

have a significant amount of dedicated resources from the connecting service provider.

On the other hand, using a packet-switching technology, such as ATM or Ethernet,

allows for more flexible multiplexing of virtual connectivity. Of these, the cost

advantages of Ethernet are significant.

The operation of such multiplexing or switching services would require more involvement

by the meet point operator. In this scenario, an enclosed space with power would be

required, in addition to regular monitoring and management of the switching service

provided by the meet point operator. However, the cost efficiencies of such a scenario

may be compelling. Offering switching services at the meet point does not preclude the

possibility of having physical cross-connects. Those applications or service providers for

whom this is more suitable could still use such a meet point strategy and still derive the

cost savings relative to independently constructed facilities.

Collocation at Meet Points

So far we have considered the meet point as an isolated point or distributed points to

which the service providers would construct fiber facility, keeping all their electronics at

their own facilities. Once a meet point operator has made the investment in building,

power, HVAC, etc, to operate a switching service, the expansion of such a space to

accommodate equipment owned and operated by the constituent service providers may

not be significant. This may also provide opportunities for other service providers who


17

are better suited by collocation service to connect to the meet points. To more cost-

effectively connect these collocated providers, the meet point operator may provide

copper-based services in addition to the fiber-based services needed to support

connections from outside the facility.

Provider A

Provider C

Provider B

Provider D

Meet Point 2 Meet Point 3

Meet Point 1

WWW Server Mail Server

Media Server

Provider A Provider B

Provider C


Media Server


Media Server

Figure 6: Collocation at meet points

Ethernet-Based Internet Exchange Points

As discussed in other sections of this report, the cost benefits of Ethernet as a

multiplexing and switching technology make it a compelling choice for meet point

services. Using Ethernet as a medium for Internet Protocol connectivity between service

providers provides a basis for establishing the meet point as an Internet Exchange Point.

In such a scenario, connecting service providers advertise their IP addresses to each

other and share a common Ethernet network for exchanging Internet traffic. By

connecting to the same Ethernet network, service providers can decide what routing

policy they want to have with each other, whether they want to provide transit access to

upstream Internet service providers, etc. Just as the cross-connect meet point can be

distributed over a geographic region, so too can the Ethernet exchange point. By


18

building multiple locations and interconnecting them via Ethernet switches, the meet

point operator can distribute the Ethernet exchange point over a region.

Typically, service providers have certain “local” IP addresses they would like all

connecting peers to be able to reach via the Ethernet exchange point. To better enable

this strategy, the meet point operator can also operate route servers available to all

participating service providers for the purpose of exchanging “local” routes.

In addition to having a common Ethernet broadcast domain available for service

providers who wish to exchange IP traffic, an Ethernet based meet point can be used to

establish virtual private connections, using Ethernet’s Virtual LAN technology, 802.1Q.

Using VLANs, connecting service providers can use the same physical Ethernet

connection to virtually connect to other providers, in much the same fashion as ATM and

Frame Relay provide virtual circuit functionality.

Such an Ethernet meet point, with available route servers and virtual LANs, has been

dubbed a “Multimedia Service Access Point” (MSAP). In addition to these stated MSAP

services, there may be dark fiber and collocation facilities available, as well as

management access to collocated equipment. By expanding from the role of simple

cross-connect to an open architecture for Internet based services, the MSAP leverages

the cost-effective and ubiquitous nature of Ethernet. This allows for the myriad of

Internet applications to be offered by providers at the MSAP: electronic mail, web

hosting, streaming media, fiber-based residential and business Internet access … all

these become enabled by having an open, provider-neutral infrastructure for service

providers.


19

Cost and Manageability Benefits of Metro Ethernet

Ethernet links operating at 1 gigabit per second can be extended over fiber optic cable at

distances of up to 70 kilometers, without amplifiers or regeneration. For less than

US$40,000 in capital expenditures, it is possible to “light” a fiber span of up to 70

kilometers and immediately provide gigabit Ethernet services between two distant

locations, with all of the physical redundancy and fault resiliency of SONET1.

The cost of lighting the same fiber span using OC-12 SONET (which operates at only

622 megabits per second) is almost US$130,0002. This difference in capital cost is at

least partly reflective of the different economies of scale for the manufacture of

components needed by each technology. The nearly ubiquitous adoption of Ethernet in

enterprise networks has created a much larger market and far more competitive pricing

for Ethernet products than for SONET products. SONET sales are typically limited to the

service provider market sector.

In addition to the clear cost advantage, Ethernet provides other benefits not available in

the SONET model. Constraints imposed by the SONET architecture preclude using the

entire capacity of the facility (622 megabits per second) for any particular connection.

Typically, only as much as one quarter of the available ring capacity (155 megabits per

second) can be provisioned for any logical circuit on the ring.

Because SONET uses time division multiplexing with very coarse-grained bandwidth

divisions, the capacity of the ring will typically be underutilized even when the ring is fully

provisioned. Service providers must charge customers for more capacity than they are

actually using because the provider has no means with which to recover unused

1 Based on an implementation using Cisco Catalyst 3550 gigabit Ethernet switches and 1000Base-ZX optical transceivers, with two fully redundant physical connections between the switches. 2 Based on an implementation using Cisco 15454 SONET multiplexers with dual rings for redundancy.


20

capacity in a SONET-based service delivered to one customer for use in meeting

another customers needs. The Ethernet approach uses statistical multiplexing and

highly granular bandwidth allocation, just like its cousins, ATM and Frame Relay. These

attributes combine to give the service provider an extraordinary degree of flexibility to

squeeze as much revenue as possible out the link capacity.

While ATM and Frame Relay can provide similar statistical multiplexing with granular

allocation of bandwidth, they cannot compete with Ethernet on the basis of capital cost.

The cost for lighting the same fiber span using ATM or Frame Relay (while providing the

same level of fault resiliency) includes all of the costs for the SONET solution plus the

costs for ATM or Frame Relay switches. Furthermore, neither of these solutions can

provide the same bandwidth. As previously noted, the maximum link speed available to

ATM or Frame Relay implemented over an OC-12 SONET ring is only 155 megabits per

second.

In addition to lower capital costs, the Metro Ethernet Forum cites rapid provisioning as a

key benefit of Ethernet in the metropolitan/regional space. “The present lack of

customer-centric flexibility, as well as the coarseness of bandwidth granularity for

[SONET- and ATM-based] legacy systems are considered major impediments to

providing promising, revenue-generating services”[4]. Citing “service velocity” as a “key

competitive differentiator”, the Metro Ethernet Forum promotes Ethernet’s ability to offer

services with line speeds ranging from 10 megabits per second to 10 gigabits per

second, and configurable bandwidth provisioning, provided quickly and on-demand.

Metro Ethernet has other advantages when applied to enterprise LAN extension

applications. Because of the cost-effectiveness of the metro Ethernet solution (largely a

product of lower equipment costs), it is possible for service providers to offer customers

much greater bandwidth for the same money. This allows enterprise networks to be

distributed over larger geographic regions without the “WAN penalty” – the traditional

difference in bandwidth available between the LAN and the WAN, due largely to the cost

of WAN bandwidth. Additionally, because the enterprise network has traditionally been

implemented using Ethernet technology, the metro Ethernet solution avoids complex,

costly, and difficult-to-manage internetworking solutions to adapt applications to


21

prevailing legacy WAN technologies. Ethernet end-to-end results in enterprise network

extension that is far more seamless than traditional WAN solutions.


22

Limitations of Current Metro Ethernet Technology

The Metro Ethernet Forum has identified the following limitations to the use of Ethernet

in metropolitan and regional networks:

• Slow recovery after link failures

• Lack of remote fault management

• Lack of in-service performance monitoring and OAM

• Limited VLAN tag space

• Inefficiencies of spanning tree relative to redundant link utilization in highly

meshed topologies

• No end-to-end service guarantees.

In the following sections, we will address each of these limitations in some detail, along

with a description of some of the protocols and proposals that seek to overcome these

limitations.

Slow Recovery From Link Failures

The Spanning Tree protocol (IEEE 802.1D) used in Ethernet networks detects link

failures and makes topological adjustments needed to restore network service with a

convergence time that is between 50 and 130 seconds. While adequate for some

applications, this is a far cry from the 50 millisecond link failure recovery time of

SONET’s automatic protection switching (APS). Multiple efforts are underway using

different approaches to address this shortcoming. So-called “carrier grade” services are

implemented with the fundamental assumption that service restoration occurs in less

than 50 milliseconds. Therefore, many in the metro Ethernet camp feel that it is

imperative that Ethernet networks be able to match SONET in this regard.

On the near horizon, the Rapid Reconfiguration protocol (IEEE 802.1w) represents an

incremental improvement. Using this protocol, Ethernet networks with particular


23

topological characteristics can recover from link failures in approximately 1 second. The

faster recovery provided by the 802.1w protocol greatly increases the number of

applications that can be supported on metro Ethernet. Pre-standard support for the

802.1w protocol is appearing in new Ethernet switches from many vendors, allowing

service providers to begin leveraging metro Ethernet to deliver services today.

Ratification of the 802.1w specification is expected in the near future.

The Link Aggregation protocol (IEEE 802.3ad) can also be used to vastly improve the

resiliency and recovery time of metro Ethernet networks. By employing parallel links

between Ethernet switches, and utilizing diverse fiber paths, the IEEE 802.3ad protocol

can provide load sharing between the parallel links when both links are operational.

When a link is broken, failover to the remaining link occurs with a convergence time on

the order of 500 milliseconds. See Figure 7.

MSAP

MSAP

MSAP

EthernetSwitch

EthernetSwitch

Passive FiberPass Through

1000Base-X802.3ad LinkAggregation

Figure 7: Link Aggregation between MSAPs

This is obviously an improvement over 802.1w Rapid Reconfiguration (though an order

of magnitude worse than SONET), and is very simple to implement. However, many

metro fiber networks have ring topologies that do not lend themselves to using Link

Aggregation protocol as a means to improve resiliency.

Several vendors (e.g. Extreme, Riverstone) have implemented proprietary approaches

to matching or at least approaching the 50 millisecond recovery time in metro Ethernet


24

networks. These approaches tend to borrow from the architecture of SONET, creating a

dual ring topology that transports Ethernet frames. Extreme claims that their “Ethernet

Automatic Protection Switching (EAPS)” dual-ring architecture has a recovery time no

worse than 100 milliseconds. The IEEE Resilient Packet Ring working group is

presently standardizing such ring-based approaches (as IEEE 802.17), with a goal of

matching or even improving upon the SONET benchmark.

Proponents of network layer (IP) switching promote Ethernet-over-MPLS (EoMPLS) as

the means by which metro Ethernet networks can achieve the resiliency of SONET

protection switching, while at the same time addressing other shortcomings in the area

of service-level guarantees. The MPLS functions that provide traffic engineering over an

MPLS cloud can be used to provide Ethernet transport with guaranteed bandwidth and

50 millisecond recovery from link failures.

Lack of Remote Fault Isolation

The SONET architecture provides a very effective means of isolating faults to the

particular SONET path, line, or section that is experiencing a fault, through the use of

overhead bytes in the SONET frame, as well as the Remote Defect Indicator (RDI) and

Loss of Signal (LOS) indication at each SONET interface. The 10 gigabit Ethernet

standard includes, in the wide area network physical interface specification (the so-

called WAN PHY for running over a SONET OC-192c carrier), the ability to map SONET

fault isolation into meaningful concepts at the logical interface.

In general, however, Ethernet does not presently possess analogous functionality. In

the long-haul applications for which SONET is often employed, remote fault isolation is

imperative to network manageability. In metro applications, the degree to which the lack

of these capabilities is a detriment to the manageability of Ethernet technology is

debatable. Remote fault isolation is less critical in Ethernet in part because its

architecture is far less complex. Large enterprise networks, based on Ethernet and

spanning very large campuses, have been operated for many years without remote fault

isolation. Few enterprise network managers would argue that the lack of remote fault

isolation makes their networks more difficult to manage.


25

Remote fault isolation is an area of active interest and research in the Metro Ethernet

Forum and certain IEEE working groups.

Lack of In-Service Performance Monitoring and OAM

Customer services provisioned over SONET-based services such as DS1, DS3, and

OC3c are terminated at a demarcation point (typically on the customer premise) using

an intelligent network termination device. This device typically provides the means by

which overhead bits in frames traveling on the circuit can be used to direct the

termination device to loop back the circuit and report the bit error rate (BER). This

capability allows the provider to monitor and test the loop extending to the customer

premise, prior to dispatching a technician, at great cost savings to the provider.

Two alternative proposals to providing this capability are being considered by the IEEE

802.3ah Ethernet in the First Mile working group. One proposal suggests the use of the

Ethernet preamble, and the other offers a frame-based approach.

While there is presently no standards-based approach to providing analogous

functionality for metro Ethernet customer access loops, many vendors are developing

proprietary approaches to solving this problem. For example, Cisco has an Optical

Network Terminator device for use with their Cisco Catalyst 4000 series switches that

provides remote OAM functionality for metro Ethernet networks.

Limited VLAN Tag Space

The IEEE 802.1Q standard defines a VLAN tag address space of only 4096 tags. This

may be insufficient for a large service provider. Many equipment vendors are

implementing so-called “Q-in-Q” approaches to stacking VLAN tags that, along with

careful planning and partitioning of the network to allow some tag reuse, should allow

networks to grow to reasonably large proportions.

While tag stacking approaches are proprietary, various MPLS techniques being

considered by IETF working groups hold the promise of providing standards-based

approaches to better scalability of metro Ethernet networks.


26

Spanning Tree Inefficiencies on Highly Meshed Networks

The advent of standards-based virtual LAN support in the IEEE 802.1Q specification

was not accompanied by a change in the Spanning Tree protocol (IEEE 802.1D).

Standards-based Ethernets with multiple virtual LANs continued to use a single

spanning tree, shared by all virtual LANs in a common broadcast domain. Since there

can be only one loop-free path in a spanning tree, this limitation can result in inefficient

use of the network – redundant paths in a meshed network topology must remain

completely idle by design of the protocol.3

Per-VLAN Spanning Tree (PVST) is an approach implemented by Cisco and other

vendors. With PVST, each VLAN has a distinct spanning tree. Per-VLAN Spanning Tree

allows for load balancing across VLAN trunks. Each spanning tree instance has its own

configuration messages and other overhead, which can be quite expensive (in terms of

CPU cycles) as the number of VLANs increase. By contrast, the single common

spanning tree of standards-based Ethernet alleviates concerns about protocol overhead

but does not allow VLAN load balancing.

The IEEE 802.1s Multiple Spanning Trees specification will standardize the use of

multiple spanning trees. The draft 802.1s specification combines the best aspects of

per-VLAN spanning tree and the common spanning tree of 802.1Q. In 802.1s, each

spanning tree creates a loop-free logical topology for a configured subset of the VLANs

in the network. This allows VLAN load sharing on highly meshed networks, while limiting

the number of spanning tree instances and associated overhead.

Lack of End-to-End Service Guarantees

Unlike ATM, Ethernet does not have inherent quality-of-service guarantees. This is not

to say that an Ethernet network cannot provide engineered quality assurance to specific

application subsets. Most Ethernet switches designed for metro Ethernet applications

3 An alternative approach that allows parallel redundant links to share the network load is the use of the IEEE 802.3ad Link Aggregation protocol. In this case, the parallel redundant links appear to the Spanning Tree protocol as a single logical link.


27

have the ability to classify and mark 802.1Q frames and/or IP packets for elevated

priority, to police traffic classes at particular rates, and to provide multi-queue transmit

disciplines such as Weighted Round Robin, and Strict Priority. By implementing

appropriate trust boundaries, and using these mechanisms, it is possible to provide an

appropriate service level for delay-sensitive applications (e.g. voice, video) in the

presence of bursty, best-effort data applications.

It is not possible, using current Ethernet technology to, implement quality-of-service

guarantees specifying acceptable packet loss, delay, and jitter parameters, with

dynamic admission control and optimal path selection. One could argue that while ATM

has the capability of signaling QoS requirements in the call setup, it has seen little use in

part because of scalability, interoperability, manageability, and policy concerns in the

service provider space. Most often, ATM QoS has been applied to permanent virtual

circuits, where admission control and optimal path selection are manually determined.

Over the long term, proponents of metro Ethernet point to MPLS and its ability to provide

QoS guarantees that are analogous to ATM, with dynamic signaling of QoS parameters

and constraint-based routing as the solution to service guarantees for metro Ethernet.

Using much of the existing hardware and software used to provide traffic classification

and transmit scheduling, and by mapping prioritization between the 802.1Q priority bits

and the MPLS EXP bits, metro Ethernet can easily be adapted to provide true end-to-

end QoS.

Is Metro Ethernet Ready for Prime Time?

Given these limitations, service providers might be reluctant to deploy metro Ethernet-

based services now. While these limitations and their possible solutions are important to

understand, it is also important to note that metro Ethernet, even given these limitations,

can support the vast majority of today’s applications.

Service providers might well choose to implement a simpler, less robust metro Ethernet

network today, and begin generating revenues from the enormous array of applications

that can tolerate near-carrier-class service at a substantial price advantage. Indeed, the


28

success of providers such as Yipes, Telseon, and Cogent in metro Ethernet services is a

testament to the business case for such an approach.

Historically speaking, advances in Ethernet technology have come at a steady pace, and

have been quickly adopted by the industry. As the works-in-progress of IEEE working

groups and other industry forums come to fruition, the future of Ethernet technology

looks very bright, indeed.


29

Best Practices for Metro Ethernet Networks Metro Ethernet networks have topological constraints, as well as management and

security considerations that are unique to the service provider environment. Enterprise

network managers have discovered, through many years of experience with Ethernet

technologies, that these networks can provide extraordinary service levels, with very

high performance, and excellent resiliency. But, in order to achieve these benefits,

careful consideration must be given to the planning and implementation of any Ethernet

network.

Given that many incumbent and new service providers have not previously deployed

Ethernet-based technologies on any scale, this section focuses on best practices and

implementation considerations for metro Ethernet networks. It highlights the pertinent

techniques and technology decisions that can benefit from the lessons learned from

large-scale Ethernet deployments in both the service provider and enterprise network

environments.

Architecture: Link Layer or Network Layer Switching

In planning a metro Ethernet network, one of the foremost considerations is whether the

majority of the switching nodes in the network will operate at the OSI data-link layer or at

the OSI network layer. Link layer Ethernet switches (also known as Layer 2 or “pure”

Ethernet switches) have the functionality needed to perform the role of access network

aggregation points in Ethernet-in-the-First Mile deployments, as well as the interconnect

capabilities needed in the MSAP. Ethernet switches from a variety of different vendors

provide support for QoS mechanisms needed to support the vast majority of

applications.

Network layer switches (traditionally known as “routers”) offer much greater functionality,

but at a higher cost – typically two to four times the cost of link layer switches. Network

layer switches from several vendors can support all of the IP and MPLS functionality

needed to provide a robust, high-performance, and cost-effective solution for virtually


30

every customer application. Moreover, network layer switches can work over almost

any combination of Ethernet and legacy WAN technologies (ATM, PPP/HDLC over

SONET), as well as dark-fiber and optical (e.g. DWDM) networks.

Most metro Ethernet service provider networks will employ a combination of switches

operating at both the data-link and network layers. In first-mile access and distribution

networks, the use of link layer switches interconnected via 1000Base-X Ethernet is a

cost-effective and manageable solution. In the core of the network, where access and

distribution networks must meet Internet and application service providers, network layer

switches provide the greatest flexibility, scalability, and manageability. Additionally,

network layer switching solutions support transparent operation over nearly any

combination of Ethernet and other link layer environments.

Spanning Tree Configuration

Careful implementation of the Spanning Tree Protocol (IEEE 802.1D) is essential to the

success of any metro Ethernet. Even when the nodes of the metro Ethernet are

network-layer packet switches (e.g. IP/MPLS switches), there are spanning tree

implications whenever Ethernet connections are extended to customer networks. The

Spanning Tree protocol is deceptively simple. The implications of the network topology

and switch configuration are not always obvious, particularly in larger networks. The

perils of giving inadequate consideration to the planning and implementation of

Spanning Tree protocol in your network range from inefficient use of valuable (e.g. fiber

line) assets, to incredibly disruptive and difficult-to-resolve anomalies known as

“forwarding loops”.

The Spanning Tree root bridge election is of critical importance. In many cases, the

customer’s local area network will participate, to some degree, in the spanning tree

protocol within your domain. To ensure stability of the network service for each

customer, it is imperative that the root bridge be completely under the control of the

service provider. The root bridge should be at or near the topological center of the

bridging domain, and should be a switch with adequate CPU resources to run multiple

spanning tree instances.


31

Service providers are strongly advised to establish internal practices and reviews that

ensure proper control over the root bridge election via the following means:

• Configuration of root bridge priority parameter values that ensure that the

protocol will elect an administratively and topologically appropriate root bridge for

each bridging domain.

• Configuration of the common so-called “root guard” feature on all customer

facing ports. This feature prevents a bridge in the customer’s network from

becoming the root bridge in the service provider’s domain.

Many metro Ethernet networks will implement the Spanning Tree protocol using

switches interconnected over physical ring topologies. Over this physical topology,

Spanning Tree will create a logical hub-and-spoke topology, where one of the switches

on the ring is the hub (the root bridge), and one link in the ring (most distant from the

root bridge) will be blocked. In this case, best results are achieved when the hub is a

bridge that interconnects many such rings, such as the switch shown in the MSAP in

Figure 2. Furthermore, such a topology is an ideal candidate for deployment of the IEEE

802.1w Rapid Reconfiguration protocol, which will ensure restoration of the ring in

approximately one second, should a fiber cut occur anywhere along the ring.

Forwarding Table Considerations

The forwarding table in a switch contains a list of MAC addresses and corresponding

egress ports, typically on a per-VLAN basis. There are three basic considerations

regarding the forwarding table:

1) Overall capacity. Switches used in service provider networks must provide ample

capacity to support the maximum number of end systems that might be connected to

the provider’s network. Most carrier class switches support on the order of 65,000 or

more entries in the forwarding table per VLAN.

2) Per port capacity. No single port should be allowed to consume all of the available

capacity of the forwarding table, since otherwise a denial of service attack is possible

by flooding a large number of source addresses into the network from a single port.


32

3) MAC address security. Switches used by service providers should provide the

capability to accept and lock-down a limited number of source addresses from

customer-facing ports. This can, in some cases, address the denial-of-service

vulnerability mentioned in (2), above. This capability can also mitigate the effects of

undiscovered topological loops (e.g. resulting from Spanning Tree protocol

misconfiguration), by preventing addresses from being learned on any port other

than the “correct” port.

Protocol Filtering

While metro Ethernet networks can support any higher-layer protocol that can be

encapsulated in an Ethernet frame, service providers may wish to filter unwanted or

unsupported protocols received from customer-facing ports. For example, residential

access service providers may want to filter all Ethernet frame types other than the

conventional encapsulation for IP and ARP. This would ensure that only the protocols

needed for supported services are transported on the network and that customers

cannot subject the network to protocol families (e.g. AppleTalk) that tend to needlessly

consume network resources with overhead traffic, or that otherwise contribute to network

instability.

Most Ethernet switches include protocol filtering support. Service providers should

implement protocol filtering as appropriate to the services delivered.

Rate Limits on Broadcast and Multicast Frame Flooding

Broadcast and multicast frames are, by default, flooded by switches to all ports on the

spanning tree except the port from which the frame was received. Switches deployed by

service providers must include the ability to apply rate limits to broadcast and multicast

frames. Broadcast and multicast rate controls can mitigate the effects of broadcast

storms and undiscovered topological loops and thus allow the network to continue to

deliver service even in the face of severe misconfiguration and/or misbehavior on the

part of switches in the network.


33

Service provider Ethernet switches must provide independent configurable rate limits for

broadcast and multicast frame flooding. Ideally, the limits should be configurable on a

per port basis. True broadcast frame traffic (addressed to the “all ones” destination

address) should, under normal conditions, have a very low bit rate per port. Thus, it is

desirable to set broadcast rate limits to be a very small portion of the available

bandwidth on each port to ensure that broadcast traffic cannot consume valuable

resources or destabilize the network. If controls are provided on a per-port basis, it is

possible to provide an engineered solution for broadcast frames from edge to core.

IP Multicast Frame Flooding and Rate-Limiting

Since IP multicast (used for many forms of one-to-many multimedia content delivery)

uses multicast Ethernet frames, it is not desirable to impose the same highly restrictive

rate limiting on IP multicast frames as should be applied to other multicast and broadcast

frames. Ideally, switches should provide independent rate-limiting and scope-limiting

functionality for IP multicast. Rate-limiting for IP multicast frames could provide an upper

bound for all IP multicast frames, or on a per-multicast-group basis, or both. Scope-

limiting should provide a means to ensure that most IP multicast frames are flooded only

to those ports with downstream IP multicast group receivers, rather than flooding

throughout the broadcast domain.

In lieu of fully independent IP multicast rate/scope controls, independent rate controls for

multicast frames (of all kinds) on a per port basis can suffice by allowing the multicast

frame rate limit to be set to a higher limit than broadcast frames. When combined with

protocol filtering (to filter non-IP multicast frames), this can achieve the same effect as

an independent rate limit for IP multicast, at the expense of other non-IP protocols

(which would in this case be summarily discarded by protocol filters).

Quality-of-Service Controls

In order to provide a reasonable foundation for providing differentiated services,

appropriate for a mix of multimedia applications, switches used by service providers

should have the following capabilities:


34

• Ability to classify traffic at ingress port based on Ethernet frame fields and

network layer attributes. In particular, the classification should allow the

classification of traffic by 802.1Q priority, source or destination MAC address, IP

precedence and/or DSCP, IP source/destination address, IP protocol field, and

transport-layer addresses (i.e. ports) for TCP and UDP.

• Ability to set the 802.1Q priority and optionally (strongly recommended) the ability

to set the IP DSCP field of outbound frames.

• Ability to police at ingress to specified bit rates based on classification as

described above. Policing should minimally provide granularity on the order of

100 kilobits per second, and burst sizes ranging from 32 kilobytes up to hundreds

of megabytes.

• At least two fully independent transmit queues per port. If only two queues are

provided, the ability to select either strict priority or weighted round robin

disciplines is strongly recommended. Additionally, the ability to direct outbound

frames to the appropriate queue of the egress port based on classification as

described above.


35

Acronym Glossary ARP – Address Resolution Protocol. A link-layer protocol used to discover the

associations between Internet Protocol (IP) addresses and Media Access Control (MAC)

addresses in an Ethernet network.

ATM – Asynchronous Transfer Mode. A cell-switching paradigm developed as part of

the ITU Broadband Integrated Services Digital Network (B-ISDN) specification.

BER – Bit Error Rate

BPDU – Bridge Protocol Data Unit. Refers to an Ethernet frame containing a Spanning

Tree protocol message.

DSCP – DiffServ Code Point. A IP packet header field defined to contain a quality-of-

service indicator defined by the Differentiated Services (DiffServ) IETF working group.

DSL – Digital Subscriber Line. A technology used to provide digital services on the

copper customer loop extending between a customer premise and a telephone company

central office.

DWDM – Dense Wave Division Multiplexing.

EoMPLS – Ethernet over MPLS. Refers to the transparent transport of Ethernet frames

over an MPLS switching cloud.

IEEE – Institute of Electrical and Electronic Engineers.

IETF – Internet Engineering Task Force.

IP – Internet Protocol. The OSI network layer protocol used on the Internet and in the

vast majority of corporate intranets and extranets.


36

ISDN – Integrated Services Digital Network.

ITU – International Telecommunications Union, formerly CCITT.

LAN – Local Area Network.

LOS – Loss Of Signal. A SONET fault isolation indicator.

MAC – Media Access Control. Most often used in the context of MAC address, which

refers to a link layer frame address (e.g. an Ethernet node’s hardware address).

MST – Multiple Spanning Trees. The approach to multi-VLAN spanning tree specified

by the IEEE 802.1s working group, wherein multiple spanning trees are operated, each

one providing a loop-free logical topology for a subset of the virtual LANs within the

bridged network.

MPLS – Multi-Protocol Label Switching, also cited as Multi-Protocol Label Swapping.

Provides label stack switching of IP packets in manner similar to that used in Frame

Relay and ATM networks, but with the ability to work over virtually any link layer protocol

(PPP/HDLC over SONET, Frame Relay, ATM, Ethernet, etc).

OAM – Operations, Administration, and Management. Sometimes specified OAMP,

where the ‘P’ represents Provisioning.

OSI – Open Systems Interconnect. An idealized model for representing the peer and

interface interactions between communications protocols, arranged in a stack. Specified

by the International Standards Organization (ISO).

PHY – A physical electrical or physical optical network interface component.

PVST – Per-VLAN Spanning Tree. A proprietary approach to implementing the

Spanning Tree Protocol (STP) in multi-VLAN networks, wherein each VLAN runs its own

instance of STP.


37

QoS – Quality of Service. Refers to the mechanisms, specifications, and/or service-level

agreements associated with providing end-to-end or node-to-node service guarantees or

assurances on the basis of packet loss, delay, and/or jitter.

RDI – Remote Defect Indicator. A SONET fault isolation indicator.

SONET – Synchronous Optical Network. Specified as the Synchronous Digital

Hierarchy (SDH) in the ITU Broadband Integrated Services Digital Network (B-ISDN)

specification. It provides the basis for synchronous transport services in traditional telco

carrier networks and is the underlying transport for both ATM and Frame Relay services,

as well as DS1, DS3, OC3c, and higher speed point-to-point services.

STP – Spanning Tree Protocol. A data-link layer protocol for estabilishing a loop-free

logical topology over an arbitrary interconnecting of data-link layer bridges.

TDM – Time Division Multiplexing.

TCP – Transport Control Protocol. A transport layer protocol providing reliable bulk data

transfer over the Internet Protocol (IP).

UDP – User Datagram Protocol. A transport layer protocol providing message passing

(datagram) capability over the Internet Protocol (IP).

VLAN – Virtual Local Area Network. A means of supporting multiple distinct bridging

domains on a common Ethernet switching network. While developed for local area

networks, the VLAN concept is used in metro Ethernet networks to provision distinct

services, providing a virtual private network for each customer.

WAN – Wide Area Network.


38

Acknowledgements The authors would like to express their gratitude to Cisco Systems, Inc, and, in

particular, Richard Shumaker and Scott Shepard, for their generous contribution of time,

effort, and content to this report.


39

References [1] Black, Ulyess and Waters, Sharleen. Sonet and T1: Architectures for Digital

Transport Networks. Prentice Hall, 1997.

[2] Clark, Kennedy and Hamilton, Kevin. Cisco LAN Switching. Cisco Press, 2001

[3] Goralski, Walter J. Introduction to ATM Networking. McGraw-Hill, 1995.

[4] Metro Ethernet Forum. Metro Ethernet Networks: A Technical Overview. 2002

[5] Norton, William B. Interconnection Strategies for ISPs. Equinex, Inc, 1999.

[6] Perlman, Radia. Interconnections, 2nd Edition. Addison-Wesley, 2000.

[7] Spurgeon, Charles. Ethernet: The Definitive Guide. O’Reilly, 2000.


40

Quick Reference to Frequently Asked Questions

1) Why is it difficult for an established telecommunications company to make this investment? (Volume 1, Volume 5)

2) There is already too much fiber in the ground. Why not use what’s there? (Volume 1, Volume 2, Volume 6)

3) The principal design criterion driving the development of this infrastructure is that every user has the potential to be a “producer” in the network economy. Is this the same as “broadband”, as it is currently hyped in the industry? (Volume 1)

4) Can we quantify the potential jobs that will be created if a region invests in building advanced telecommunications infrastructure? (Volume 1)

5) What should be the Tobacco Commission’s role in the deployment of first mile technologies? (Volume 1, Volume 3, Volume 5, Volume 7, Volume 8)

6) How can localities ensure that they get early access to the network? (Volume 1, Volume 5, Volume 8)

7) What kind of success have other regions had with the development of network infrastructure for economic development? (Volume 1)

8) What regulatory factors should be considered when investing in wireless technologies? (Volume 1, Volume 7)

9) Why do we need to connect to network points outside of the tobacco regions? (Volume 2)

10) Once the network is in place, what do we do with it? (Volume 2, Volume 8)

11) Since the business model for inter-regional and inter-county infrastructure did not include the use of conduit facilitating blown fiber strands, what are the circumstances in which this technology is appropriate and financially feasible? (Volume 3, Volume 7)

12) How do existing community networks fit into the overall design? (Volume 3, Volume 5, Volume 6)

13) What are some examples for deployment in the first/last mile? (Volume 3, Volume 7)

14) What type of fiber is recommended? (Volume 3)

15) What would a network design for my county look like? (Volume 3)


41

16) How much would all this cost? (Volume 3, Volume 5)

17) What is the appropriate organization model for managing and sustaining the Tobacco Commission’s investment in critical technology infrastructure? (Volume 5)

18) Tobacco region communities are underserved because the private sector does not see a profitable business case. What makes this feasible from a business perspective? (Volume 5)

19) If the traditional investment model for developing critical technology infrastructure has failed, what is the alternative? (Volume 5)

20) How much would it cost for consumers in the region to use the network? (Volume 5)

21) What technologies enable use of the fiber? (Volume 6)

22) How does the choice of technology to light the fiber impact the cost? (Volume 6)

23) How do wireless technologies fit into this framework? (Volume 7)

24) What is meant by the term “open access”? (Volume 8)

25) What is the difference between the broadband hype and the “next generation” networks? (Volume 8)

26) What are some next generation Internet (NGI) applications? (Volume 8)

strategic technology infrastructure for regional ... · volume 6: leveraging advanced optical and...

Documents