ethernet performance measurement

Ethernet Performance Measurementon an MPLS platform

Michiel Appelman2011

Ethernet Performance Measurementon an MPLS platform

By:Michiel APPELMAN

Supervisor:Attilla DE GROOT

January 14, 2011

Summary

The Amsterdam Internet Exchange (AMS-IX) is an organization to facilitate the exchangeof IP traffic between its participants and is set up as a not for profit membership basedassociation. The members consist of large ISPs, mobile operators, content providersand other communication companies. AMS-IX has been searching for a way to measurevalues like delay and jitter over this platform, after a need for a new service calledInter-IP eXchange (IPX) arose. Mobile operators are planning to use this new exchangestructure to separate their communication-specific data from the ‘regular’ internet. TheIPX standard requires Service Level Agreements (SLAs) to be met by all parties involvedin providing end-to-end connections, including AMS-IX as an traffic exchange platform.

To provide its customers with a reliable network and prove that the SLA will be met,AMS-IX must be able to collect the statistics of the network. Because the network uses acombination of Ethernet, MPLS and VPLS in exchanging traffic, traditional higher levelOperations, Administration and Maintenance (OAM) utilities could not be used to measureits performance reliably. At the moment there is one new standard defining measure-ment of the various Key Performance Indicators (KPIs) over Ethernet and is developedby the ITU-T in coordination with the IEEE. It’s referenced by Y.1731 and is commonlyknown as Ethernet OAM Performance Measurement.

With this knowledge, a fitting hardware solution supporting Performance Measurementwas searched for among different vendors. Accedian Networks already provided AMS-IX with some test Network Interface Devices (NIDs) to set a baseline. Other devices werelooked at as well and a list of requirements was compiled. Eventually the test devicesby Accedian were the only models which could pass all criteria and had a performancethat would be of use on the AMS-IX platform. Accedian was the choice as a manufac-turer and the first devices for 1 Gigabit Ethernet (GigE) connections were ordered andinstalled.

While testing these new devices it became clear that an important external part of themeasurements is the synchronisation of the clocks between them. The slightest differ-ence between clocks will result in a significant variation in the one way delay values,the only KPI influenced by this. Because the performance of the platform is of such ahigh level, differences in milliseconds are not acceptable. Using the Accedian NIDs theclocks can be set to within a couple of microseconds but a new external time source willbe needed to provide them with a valid time. A search for this new NTP-server is stillon-going.

The data collected by the devices using ITU-T Y.1731 over the platform is graphed toprovide the customers and the company itself with easy to interpret images. This isdone by polling all the devices every minute through SNMP, retrieving their averagevalues and graphing them. The images are then displayed in a web portal where userscan select source and destination of the measurements. The stored values are consoli-dated over time and at the end of each month the data is processed and the values are

i

then reported to customers through a 95th percentile.

At the moment the measurements on the platform are all operational and the valuesare being collected. The Inter-IPX VLAN has yet to have customers connect to it but thedata is enough to give AMS-IX itself a valuable insight in their platform. The Inter-IPX

service will be provided in the coming months and SLAs will also be made available toother customers who wish to have such a contract. All this is enabled by the relativelynew ITU-T Y.1731 standard and the NIDs manufactured by Accedian.

The remaining tasks to make the whole measurement system complete, are the imple-mentation of new and reliable time sources and the expansion of the devices to alsoinclude 10 GigE connections to the network. Doing so will provide a full picture of thenetwork, from and to every switch. Both recommendations will be implemented in thefirst quarter of this year.

ii

Contents

Summary i

1 Introduction 1

2 Background 32.1 AMS-IX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1.1 The platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.1.2 Organisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.2 Inter-IPX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3 Project Details 63.1 Service Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63.2 Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63.3 Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4 Operations, Administration and Maintenance 84.1 Ethernet OAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84.2 IEEE 802.3ah . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84.3 IEEE 802.1ag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.3.1 Maintenance Domains and Levels . . . . . . . . . . . . . . . . . . . 94.3.2 Maintenance Association and Maintenance End Point . . . . . . . . . 94.3.3 Inter-Domain Operations, Administration and Maintenance (OAM) . 10

4.4 ITU-T Y.1731 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.5 Performance Assurance Agent . . . . . . . . . . . . . . . . . . . . . . . . . . 114.6 Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

5 Research 125.1 Probe Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5.1.1 Vendors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125.1.2 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

5.2 Platform Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165.2.1 Multicast Frames . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165.2.2 Label Switched Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . 185.2.3 Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

5.3 Timing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205.3.1 Hardware Timestamping in NTP . . . . . . . . . . . . . . . . . . . 20

6 Implementation 236.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

6.1.1 Network Interface Device (NID) Configuration . . . . . . . . . . . . 246.2 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

6.2.1 Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276.3 Timing Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

iii

7 Conclusion 317.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317.2 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327.3 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Appendices

A Performance measurement infrastructure project 34A.1 Introduction and Purpose of the project . . . . . . . . . . . . . . . . . . . 34A.2 Stage 1: Measurements and reporting on Key Performance Indicators (KPIs)

between a selected subset of the AMS-IX access routers. . . . . . . . . . . 35A.3 Stage 2: Measurements and reporting on KPIs between a selected subset

of the AMS-IX access routers . . . . . . . . . . . . . . . . . . . . . . . . . . 36

B Plan van Aanpak 37B.1 Inleiding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

B.1.1 AMS-IX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37B.1.2 Inter-IPX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

B.2 Doelstelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37B.3 Projectopdracht . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38B.4 Activiteiten . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38B.5 Producten . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

B.5.1 Plan van Aanpak . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39B.5.2 Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40B.5.3 Onderzoek Apparatuur . . . . . . . . . . . . . . . . . . . . . . . . 40B.5.4 Verslag en Presentatie . . . . . . . . . . . . . . . . . . . . . . . . . 40

B.6 Randvoorwaarden . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40B.6.1 Randvoorwaarden . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

B.7 Kwaliteit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41B.8 Kosten/baten overzicht . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

B.8.1 Kosten . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41B.8.2 Baten . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

B.9 Risico analyse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41B.9.1 Tijdnood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41B.9.2 Extra BV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41B.9.3 Expertise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

B.10 Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

C Configuration Example 45

D Graph Generating Bash Script 47

E Acronyms 49

F Bibliography 51

iv

List of Figures

1 Topology of the Amsterdam Internet Exchange (AMS-IX) platform . . . . . 32 The GPRS Roaming eXchange (GRX)-structure . . . . . . . . . . . . . . . . . 53 Structure of Ethernet OAM . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Inter-Domain structure in OAM . . . . . . . . . . . . . . . . . . . . . . . . 105 Overview of a Continuity Check Message Ethernet OAM frame. . . . . . . . 176 The Label Switched Paths (LSPs) over the AMS-IX platform . . . . . . . . . 197 Structure of an NTP packet. . . . . . . . . . . . . . . . . . . . . . . . . . . 218 Synchronisation process between a client and the NTP-server. . . . . . . 229 The path that an NTP-packet travels through the OSI-layers. . . . . . . . . 2210 Graph generated by Round Robin Database (RRD)tool . . . . . . . . . . . . 2711 Overview page KPI measurements . . . . . . . . . . . . . . . . . . . . . . 2812 Overview of a One Way Delay Measurement Ethernet OAM frame. . . . . . 29

List of Tables

1 Comparison of different vendors . . . . . . . . . . . . . . . . . . . . . . . 162 Device configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 OAM threshold values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254 Object Identifiers (OIDs) for various KPIs . . . . . . . . . . . . . . . . . . . 265 Comparison of Network Time Protocol (NTP) time sources . . . . . . . . . . 30

v

Introduction Chapter 1

1 Introduction

This report is a product of a research project that was commissioned by the Amster-dam Internet Exchange (AMS-IX), a provider of an Internet Protocol (IP) traffic exchangeplatform. A significant number of companies connect to this platform such as Inter-net Service Providers (ISPs), content providers, mobile operators, etc. They connect toone of the eight Point of Presences (POPs) in the Amsterdam area which enables them toestablish peering sessions with other members and exchange traffic with them.

Reason

Mobile operators connect to each other through GPRS Roaming eXchange (GRX) providersto provide authentication to their customers when they are roaming on another net-work. A trend within the telecommunications industry is to consolidate all traffic intoone transport mode, be it voice, data, video or any other format.[1] This is called theNext Generation Network (NGN) and operators are now looking at connecting with eachother over a network which supports IP-traffic, called IP eXchange (IPX). AMS-IX rolein this network would be to provide IPX-providers with a similar exchange platformas is now in place for GRX providers. This project was started while developing thisnew service that would require Service Level Agreements (SLAs) to guarantee delivery ofthis vulnerable traffic. The values of these SLAs need to be measured to provide thecustomers and AMS-IX itself with a clear overview of the performance of the network.

Research Questions

This research paper will need to provide answers to the following questions:

1. How can the performance of an OSI-layer 2 platform, such as provided by AMS-IX

to its customers, be measured?

2. How can AMS-IX provide their customers with the measured values and includethis data in an SLA?

Structure

Further project details including other requirements set by AMS-IX will be looked at inchapter 3. In the following chapter a more detailed view of the structure and operationof AMS-IX will be given.

Chapter 4 discusses the techniques which will be used to measure the SLA values. Thisincludes standards by the IEEE and ITU-T. The Y.1731 standard by the ITU-T will be dis-cussed in detail in chapter 5 along with the devices that were considered in finding a

1

Introduction Chapter 1

fitting solution. The difficulties with regards to the Multiprotocol Label Switching (MPLS)platform and timing between the devices are also brought up and solved where possi-ble.

The findings of this research are then implemented in chapter 6. The devices whichhave been purchased are installed and configured to support the measurements of thedefined Key Performance Indicators (KPIs). Also shown is how through various tools, thedata collected by the devices is stored and then visualised in graphs, as well as a moredeeper study in timing sources for the network.

The conclusion summarises the results of the research and answers the afore mentionedquestions but also provides some recommendations for tasks still to be finished.

2

Background Chapter 2

2 Background

2.1 AMS-IX

The Amsterdam Internet Exchange (AMS-IX) provides an Multiprotocol Label Switching(MPLS)-platform to which a customer can connect their network and exchange datawith the networks of others. AMS-IX has several Point of Presences (POPs) distributedaround the Amsterdam-area where customers have the possibility to connect their routersto the platform. There are two datacenters between which the core of the network isdivided, it consists of four switches with connections to every other access switch in thenetwork (see figure 1).

MLX8

Telecity 2

10/100/1000

10GE

MLX32MLX32MLX32 MLX32

PXC

GlobalSwitch

10/100/1000

10GE

MLX8

SARA

10GE

10/100/1000

MLX8

MLX32 MLX32

NIKHEF

10/100/1000

10GE

10/100/1000

10GE

MLX16MLX16

MLX8

euNetworks

MLX8

MLX16MLX16

MLX8

MLX16MLX16

Equinix

GE

10GE

MLX32 MLX32

MLX8

MLX16MLX16

Interxion AMS5

GE

10GE

MLX32MLX32 MLX32MLX32

MLX32 MLX32

PXC PXC PXC PXC PXCpxc-eqx-118PXC

Telecity 4

10/100/1000

PXC

Figure 1: Topology of the AMS-IX platform

2.1.1 The platform

AMS-IX was founded by several Internet Service Providers (ISPs) who could save a lotof money if they exchanged traffic directly instead of paying their upstream-providersto reach each other. What they did was buy a shared infrastructure device on whichthey could all connect to each other. This concept of a shared platform holds true todayalthough the scale has changed significantly. Because of this, a separate, neutral entitywas needed to maintain this piece of shared infrastructure. AMS-IX became this entityand it’s neutrality is ensured through an organisational structure which is described insection 2.1.2. The platform uses MPLS[2] for transporting data from one customer to theother and using Virtual Private LAN Service (VPLS), it appears as a single large broadcastdomain[3] from a customer’s point of view, so they can setup their peering sessions.

To peer is for two customers to exchange information about their networks throughBGP. Although the customers can reach each other over the platform, this does notmean that they immediately can start exchanging traffic. They will have to talk to the

3


other customers to peer with. When they have established a session they can startsending and receiving traffic to each other.

The optical link of a customer with a 10 Gigabit Ethernet (GigE) connection is beingterminated on a so called Photonic Cross Connect (PXC). This device receives the lightfrom the customer router and is able to direct that light to one of two Provider Edge (PE)-router. This is all done in hardware without converting the light to an electrical signal.To do this the device essentially uses micromirrors to switch it from one fiber output tothe other. This system is able to provide a redundant connection to the network whichcan recover from a broken switch within 20 milliseconds and without any downtimefor the customer.

2.1.2 Organisation

AMS-IX is split up in two legal entities: a corporation and an association. The associ-ation is the one and only stockholder of the corporation and consists of the memberswhich are the parties connecting to the AMS-IX platform. Each member has an equalvote in the decision-making of the corporation. Twice a year the members get togetherfor a general meeting to discuss the future of the company. Being a member – andthus stockholder – also means that if the corporation makes any profit this could bebeneficial because prices may go down.

2.2 Inter-IPX

Over time AMS-IX also started offering other services than just the ISP peering, one ofwhich are Inter-GRX connections. A GPRS Roaming eXchange (GRX) is an exchange pointfor mobile operators to exchange roaming traffic and authenticate subscribers of net-work A when they are using network B, eg. when they are in a different country. Thisexchange is maintained by a GRX-provider, which connects several operators, mostlyfrom the same continent. Not all mobile operators are connected to the same GRX-provider though, and so the need for a connection between those exchanges arises. Acompany providing that service is called an Inter-GRX provider, of which three exist inthe world: Singapore, Ashburn (VS) and the one at AMS-IX, which is the largest. TheGRX providers connect to each other using a separate VPLS-instance on the platform –similar to a Virtual LAN (VLAN). Figure 2 illustrates the structure of the various GRX-elements.

A new standard that has been developed by the Global System for Mobile CommunicationsAssociation (GSMA) is IP eXchange (IPX). The GSMA is the developer of standards likeGeneral Packet Radio Service (GPRS) and GRX and has been working on IPX since thetrend of Next Generation Networks (NGNs) started, networks which use the same mode oftransport regardless the payload.[1] More and more traffic between operators is being

4


Figure 2: The GRX-structure

sent over Internet Protocol (IP), be it data, voice or video. The goal of IPX is to supersedeand include GRX and provide the following advantages:

1. connections between every kind of service provider instead of mobile operatorsonly;

2. end-to-end Quality of Service (QoS) agreements for IP traffic;

3. any service based on IP can be used between peers;

4. connections can be sold as upstream to other providers.

To make the transition to IPX without much impact to the traffic the current GRX struc-ture is essentially upgraded. The diagram from figure 2 would not change drasticallyafter the change. AMS-IX started a trial period of the Inter-IPX service in the end of 2010and eventually wants to provide this service to all potential customers.

5

Project Details Chapter 3

3 Project Details

In chapter 1 two research questions were proposed and will need to be answered butthere are some requirements that the answers of the questions will also need to meet.

3.1 Service Levels

As was shown in the previous chapter, there are some advantages to the new IPX stan-dard. One of these is the fact that there will be QoS-agreements that have to be met.These agreements are commonly referred to as Service Level Agreements (SLAs) and forman important aspect to this project.

Traditionally AMS-IX has provided their members with an Service Level Description (SLD),a document that contains some basic information about connecting to the platform. Itdefines terms like the demarcation point, availability, performance, and maintenanceperiods. Although it includes some performance targets, there are no penalties whenfailing to meet them and therefore is called a Description instead of an Agreement.

The Inter-IPX service will require a penalty scheme because of the end-to-end QoS agree-ments that will need to be met at all times. This means that the targeted values that arepresent in the SLD will become maximum values in the new SLA that should not beexceeded. To ensure this doesn’t happen AMS-IX will need to start measuring thesevarious indicators.

Before the beginning of the trial AMS-IX has been using three demo probes capable ofdoing so. The probes were tested on three different locations and produced by AccedianNetworks. They were directly connected to the access switches, ran some standardmeasurements and helped baseline the platform to get some initial values for the SLA.

3.2 Assignment

This project will need to make sure that the Key Performance Indicators (KPIs) includedin an SLA can be measured over time. If hardware is to be purchased all options willneed to be looked at and compared. The data that results from the measurements needsto be stored and visualised in a clear matrix with the data from each probe opposite ofeach other. This way a clear image of the performance of the network is formed forboth AMS-IX and it’s members in the earlier stage of testing. When expanding to morethan twenty devices this will become less useful.

Eventually a solution has to be provided that gives a clear insight in the measurementof the KPIs, can provide monthly reports to customers with an SLA and also performsreliably in the platform. The Inter-IPX working group within AMS-IX starting the trialwill also be followed closely to map their requirements for this assignment. The project-leader is Luisa Garcia Montoya although this research project is supervised by Attilla

6

Project Details Chapter 3

de Groot, a member of the Network Operations Center (NOC). A detailed descriptionof the assignment provided by Chief Technology Officer (CTO) Henk Steenman has beenincluded in Appendix A.

The implementation and visualisation within the web-portal will need to be coördi-nated with the web developers, monthly reports will be drafted with the Member Re-lations department and the technical details will be discussed with the NOC.

This project will be marked as complete when it’s possible to:

1. measure the defined KPIs over the platform;

2. show this data in the web portal;

3. present aggregated stats publicly;

4. show each customer data that is of interest to them, and

5. scale the solution up to a device per customer.

The time schedule and agreements made are also included in the so called ‘Plan vanAanpak’ which is included in appendix B (in Dutch).

3.3 Hardware Requirements

A potential hardware solution has to meet the following requirements:

1. It can measure:

(a) One Way Average Delay;

(b) Two Way Average Delay;

(c) One Way Average Delay Variation also known as jitter;

(d) Two Way Average Delay Variation also known as jitter;

(e) Frame Loss, and

(f) Uptime.

2. Values from these measurements can be stored locally or remotely;

3. There is a possibility to place the device between the customer router and theAMS-IX PE-router.

7

Operations, Administration and Maintenance Chapter 4

4 Operations, Administration and Maintenance

Operations, Administration and Maintenance (OAM) is a technique used by network ad-ministrators to manage their long-distance networks, which can spread over multiplecities, countries or even continents. It enables them to be notified when one of the linksin the network shows errors, high delays or any other issues. Doing this manuallywould be a huge and tiresome task. Based on the events detected by the OAM processother actions can be triggered – eg. taking another routing path – although this is notan internal function of OAM. These kind of functionalities are included in AsynchronousTransfer Mode (ATM) for example, and MPLS includes some tools to troubleshoot LabelSwitched Paths (LSPs). Higher up the stack, IP also includes some of these manual toolswhich are included with Internet Control Message Protocol (ICMP).

4.1 Ethernet OAM

Ethernet wasn’t developed with long distances in mind, it is traditionally used as a Lo-cal Area Network (LAN) protocol. The links in this case are not of great length and easilymanageable by a single person. When Ethernet-networks started to expand, more per-sonnel was needed to look after these links, a company could have multiple locationswith different engineers but the network was still in their control. The demand for OAM

functionalities remained quite low.

These days Ethernet is also being used to connect networks from different companies toeach other. Instead of a LAN protocol it is more and more used for Wide Area Networks(WANs), replacing traditional protocols like ATM while users still expect the same high-quality connections. Around the begin of the century the demand for the same OAM

functionalities grew and were starting to be added to the Ethernet-protocol. In thischapter, four different standards will be discussed that try to fulfill this demand.

4.2 IEEE 802.3ah

In November 2000 a new study group was founded after 300 representatives of differentcompanies got together on a “Call for Interest.” The goal of this group was to addfunctionalities to Ethernet to use it as Ethernet in the First Mile (EFM). The First Mileessentially being the local loop to the customer. [4]

It added various new features that were of importance to Service Providers (SPs) one ofwhich were new types of media that could be used, eg. Passive Optical Networks (PONs)and legacy voice-grade copper wiring. It also provided discovery and monitoring OAM

tools on link level to the Institute of Electrical and Electronics Engineers (IEEE) 802.3 stan-dard for Carrier Sense Multiple Access (CSMA)/Collission Detection (CD) networks in 2008.[5]

8


4.3 IEEE 802.1ag

Although IEEE 802.3ah adds some OAM features they are still limited to one link, ie. itcan’t provide monitoring for a whole path. This is why in 2007 the IEEE approved anew draft to be included in the 802.1Q standard. This section is known as ConnectionFault Management (CFM) and provides end-to-end OAM functionality. [6]

4.3.1 Maintenance Domains and Levels

802.1ag defines a new hierarchy within the different networks that it covers. The topmost level consists of a Maintenance Domain (MD) with a specified MD-level. There a8 levels that can be chosen of which the lowest levels are for the companies on thelowest level of the network, eg. SPs and carriers. The top most levels are for end usersand customers. This means that higher level data may be carried over lower levels butlower level data may not propagate to higher level domains.

4.3.2 Maintenance Association and Maintenance End Point

Below MDs there exist Maintenance Associations (MAs): groups that consist of two ormore Maintenance End Points (MEPs). A MEP is a device that resides at the edge of thedomain, has it’s own unique MEP IDentifier (MEPID) and knows all of its peers througha list defined in the MA. It maintains connections called Maintenance Entitys (MEs), withall the other MEPs provided that they share the same MA and MD-level. Figure 3 tries toillustrate this structure.

Figure 3: Structure of Ethernet OAM

Over the ME-paths the MEPs keep each other informed about their status and the linksin between. These paths are represented by the dotted lines in figure 4. Errors can bepropagated and paths can be traced to check for any faulty links. Intermediate devices

9


can also participate in these checks. These would be called Maintenance IntermediatePoints (MIPs) and are passive in the OAM domain, ie. they only respond to incomingmessages and never originate messages.

4.3.3 Inter-Domain OAM

OAM measurements can be setup between any two MEPs, even if they are located ontwo different continents. But it is possible that this connection traverses several othernetworks which also use OAM. To distinguish between frames from both networks thepreviously mentioned MD-levels are used. Any lower level MD transparently passes onthe frames from the higher levels until the frame reaches a MEP of the same level and isthen processed. But it’s also the other way around: lower level frames are not allowedto enter the higher level MDs. For this to work a device which connects a higher levelnetwork to a lower level one would need to appear as a MIP to the higher level. This is apartial function of this OAM entity and is therefore also called a MIP Half Function (MHF).See also figure 4.

Figure 4: Inter-Domain structure in OAM

4.4 ITU-T Y.1731

At the same time the IEEE was developing 802.1ag, the International TelecommunicationUnion - Telecommunication Standardization Sector (ITU-T) was also looking into EthernetOAM functionalities. The two bodies made some agreements as not to overlap eachother, while both standards would be using the same EtherType (0x8902) and frameformat. They did not agree on the naming of the different parts of the OAM hierarchyhowever, and thus the ITU-T has a different naming convention. A MA is called a Main-tenance Entity Group (MEG) by their standard and although the acronym MEP remainsthe same, in this case it’s short for MEG End Point. The Y.1731 standard also doesn’tdefine MDs and only considers the MEG-levels. Figure 3 includes both types of naming.[7]

10


The ITU-T additionally added Performance Measurement (PM) to their standard, an im-portant functionality for the mobile operators. These companies require to know howlong it would take for a frame to get from one side of the network to the other. Mea-surements for delay, jitter and frame loss were being implemented and eventually thisstandard was made official even before the IEEE approved 802.1ag in 2007.

4.5 Performance Assurance Agent

While testing the demo devices from Accedian that AMS-IX had been using, there alsowas a proprietary protocol that was looked at capable of measuring these indicators,developed by Accedian and called Performance Assurance Agent (PAA). According to theIEEE assignment database [8] EtherType 0x88fc contains the following:

The protocol will use a common header multiplexingvarious subprotocols into a single ethertype.The format is as follow:

Version 8 bits Version of the common headerSubtype 8 bits Subprotocol identifierFlag 8 bits Dependent subprotocol flag bitsOffset 8 bits Offset to the subprotocol PDU

The actual subprotocol PDU location is computedby adding the Offset value to the addressof the Offset field itself. For examplean Offset of 0 point to the byte followingthe Offset field.

Further details about the contents of the subprotocol Protocol Data Units (PDUs) areunfortunately not available.

4.6 Solution

The protocol of choice on the AMS-IX platform would be the two supporting the mea-surements of delays from end-to-end: ITU-T Y.1731 or PAA. Because the latter lacksdocumentation and therefore the way the measurements work can not be checked, itis deprecated. With Y.1731 AMS-IX will be able to measure their required KPIs and thestandard is both open and more widely adopted.

11

Research Chapter 5

5 Research

After knowing the basics of the theory and a good understanding of the demands set byAMS-IX it is possible to search for a fitting solution. Possible vendors for measurementdevices will be compared and further details about the OAM standards will be provided.

5.1 Probe Hardware

To know what kind of hardware to use it is important to understand which require-ments it must fill. The more important ones were already discussed in paragraph 3.3and in summary consisted of:

• One Way measurements;

• Two Way measurements;

• Frame Loss;

• Uptime;

• Storage of values and

• using it as an intermediate device.

At the moment, only 1 GigE customers are taken in to consideration but eventuallythere might be an upgrade to include 10 GigE ports as well. This has to be taken in toaccount when searching for possible solutions. It is also important for these devices tosync to each other so that they all share the same time, this is an essential requirementfor the One Way delay measurements.

On the other hand, uptime is a value which can not be calculated using one device. Thisis a result of the requirement that has been set in a draft Inter-IPX SLA, that every cus-tomer has to have two connections on two different locations to the AMS-IX platform.This means that the results of both devices need to be compared and only when bothof them are down outside of a scheduled maintenance window, it counts as downtime.For this reason, this indicator is not taken into account but will be calculated after thedata has been collected and is to be presented to the customer.

5.1.1 Vendors

Knowing the requirements which have to be fulfilled by a certain device makes search-ing for one a more manageable task. In this section the devices that were looked at willbe discussed.

12

Research Chapter 5

Accedian

Accedian is a vendor of service level assurance devices which are primarily built forEthernet backhauls of large mobile carrier networks. For these operators measuringdelays between their sites is essential to ensure high quality voice connections. Theyhave models which support 1 GigE and others that also support 10 GigE, an interestingoption when scaling is necessary.

These devices had the advantage to already be in use by AMS-IX before starting thisproject. The NIDs were performing correctly and purchasing them was already a dis-cussed option. They nevertheless had to be tested further and models from other ven-dors needed to be considered in case these did not fit the requirements after all. Two 1GigE MetroNIDs and one 10 GigE MetroNODE were provided to use in the platformand provide it with some baseline-values. This also enabled the engineers to see howthe network would react to the data exchanged by the Network Interface Devices (NIDs).

The acronym NID is used to describe the actual location of the device in the network:essentially it provides the customer with a network interface to connect to the platform.This is the demarcation point for the equipment of the provider. In section 5.2.3 it willexplained why AMS-IX would not use the devices in this way, but the term NID will stillbe used to refer to these devices.

The devices include the standard OAM Performance Measurement but also add the pro-prietary PAA protocol which was discussed in 4.5. Both provide one and two way mea-surements and counters for frame loss. The PAA protocol was considered as an optionbut after testing it became apparent that it would not scale to a large number of devices.Because of the fact that all destination Media Access Control (MAC)-addresses have to bemanually configured it would make it a tedious task to configure a large amount NIDs.This is in contrast to the IEEE and ITU-T standards which auto-detect MAC-addresses oftheir peers.

A NID has two active forwarding ports, one facing the network and the other facinga customer. This means that as an intermediate device it can be used to measure thewhole path that the customer data takes.1 Models exist with copper or fiber ports, ora combination of both. The model that would fit AMS-IX is the EtherNID GE, it has acombination of two Small form-factor pluggable (SFP) fiber ports and three copper portsof which one is the management port. Using this combination, both customers withfiber as well as with copper connections can be connected to the device.

Brocade

The devices that are in place right now for switching data are from Brocade. This ven-dor has minimal support built-in for OAM Performance Measurement. The hardware has

1With some exception on the AMS-IX platform, see 5.2.2.

13

Research Chapter 5

the functionality to act as a MIP and as such can participate in any Link Traces originat-ing from MEPs. They can also be configured as MEPs themselves but although this hasbeen implemented, one way delay measurements are not supported because they cannot synchronise their clocks properly. [9]

RIPE Test Traffic Measurement (TTM) servers

For some years AMS-IX has been working with Réseaux IP Européens (RIPE) to providethem with statistics regarding the quality of the Internet. They do this using servershosted by a variety of SPs or other IT-related companies throughout the world. Allthese servers exchange ICMP ping-messages to measure their delays and frame lossto each other. AMS-IX has been hosting four of these devices in datacenters NIKHEF,SARA, GlobalSwitch and TeleCity.

Expanding this setup to all eight datacenters would provide AMS-IX with an overviewof their internal network as well. Although this would be very helpful, the adminis-tration of the servers and the data collected would be done by another company. Thissituation would not be desirable and also wouldn’t make it possible to collect the mea-sured values to calculate downtime periods.

On top of that the devices have stopped working reliably in the network ever sinceAMS-IX started using the MPLS platform. This was caused by the different paths avail-able to the device to get out of the network. It could take a different path each timeand because it only sent out a message every 30 seconds, the Provider (P)-routers in thepath sometimes had to relearn their MAC-address. This would mean that the framewould have to pass over the Central Processing Unit (CPU) for the switch to learn theMAC-address again and this would cause a much higher delay.

Besides these problems the devices weren’t very scalable because they all needed aGlobal Position System (GPS) antenna, they couldn’t be placed as an intermediate devicesand also don’t support 10 GigE connections.

MRV

Another supplier that is familiar within AMS-IX is MRV. Their products are used forDense wavelength division multiplexing (DWDM) connections between datacenters, thesedevices multiplex different light-streams onto one fiber strand using different wave-lengths. This allows AMS-IX to multiply the capacity of a single fiber strand by a factoreight.

MRV recently started producing the OptiSwitch 940 which builds on their Carrier Eth-ernet portfolio. It is marketed as a demarcation device and so multiple customers canbe aggregated on the device to one or more uplinks. The 940 model includes OAM for

14

Research Chapter 5

the first time to measure one and two way delays, in addition to frame loss. The down-side is that the series at this moment lack a stable clock synchronisation method.[10] Ina later hardware revision it will be able to use Synchronous Ethernet (SyncE) for timingbut that would also require additional hardware in the management network. Therealso exists an OptiSwitch 930 model which suffers from the same limitation but doessupport 10 GigE interfaces.

RAD

Besides the known names in the business there are also two new vendors which pro-vide similar solution. One of them is RAD, a manufacturer of products to support car-riers and mobile operators. They provide Ethernet OAM functionality with their ETXdevices, one of which is the ETX-202A. This devices allows connections up to 1 GigEand provides them with one and two way delays measurement, frame loss and SimpleNetwork Management Protocol (SNMP) polling for the collected values.

As with the other devices, an important factor is the synchronisation between clocks.The supplier of the RAD devices couldn’t give a specific value for the average varia-tion between the 202A devices but did mention the 204A model, which similar to theMRV OS940 could sync through SyncE. This would require the additional networkinghardware as well.

To support 10 GigE connections the supplier could provide AMS-IX with the RAD 1002model which is a similar demarcation device. Unfortunately this relatively new productis not yet able to support Ethernet Performance Measurement.

Omnitron iConverter

Another new vendor capable of providing AMS-IX with OAM devices is Omnitron. Itsprimary product line is a range of media-type converters called iConverter. An impor-tant feature that these devices also support is Ethernet Performance Measurement. TheGM3 model supports the same measurements for one and two way delay and frameloss while also making it possible to poll these values through SNMP.

The devices are used as a media-type converter and thus always sit in between thecustomer and the platform. GigE customers are supported but the vendor lacks supportfor 10 GigE connections. Apart from this limitation the clock synchronisation is notspecific: the vendor could not specify a variation value for Network Time Protocol (NTP)and didn’t provide any information about other synchronisation methods.

15

Research Chapter 5

5.1.2 Comparison

With all specifications considered a matrix can be compiled comparing the features ofthe various vendors. Table 1 gives an overview of the devices that were looked at.

10GigE One Way Two Way Loss Intermediate SNMP Sync AccuracyAccedian EtherNID yes* yes yes yes yes yes <20 microseconds

Brocade internal yes no yes yes yes yes noneRIPE TTM boxes no yes yes yes no no 100 nanoseconds

MRV OS940 yes yes yes yes yes yes none yet**RAD 202A no yes yes yes yes yes N/A

iConverter GM3 no yes yes yes yes yes N/A

Table 1: Comparison of different vendors

* Available on another model called MetroNODE.** Should be available in a next hardware revision.

From this table it can be concluded that only two options pass all criteria: Accedian andMRV. Both vendors would fulfill the requirements passed by AMS-IX although the MRVOptiSwitch lacks an important detail: clock synchronisation. At this moment this is notoptimal and would later be improved by adding SyncE support but this would requiremore investment in adding hardware to the management network that also supportSynchronous Ethernet.

Accedian on the other hand claims a synchronisation between clocks using their NTP

implementation of less than 20 microseconds. Thanks to the demo models that were inuse this claim could and has been verified. These observations allow for the conclusionthat the Accedian EtherNIDs and MetroNODEs would be the preferable solution tomeasure the various KPIs over the AMS-IX platform.

5.2 Platform Behavior

Testing continued with the Accedian NIDs to see how the network would react. Someunexpected behaviour was experienced while running the Performance Measurementsconsidering the paths that the OAM PDUs would take over the platform. To understandthis a more detailed view of the OAM frames is need.

5.2.1 Multicast Frames

In chapter 4.1 Ethernet OAM has already been shown to add some important features tothe lower levels of the Open System Interconnection (OSI)-model. The protocol has beendeveloped as a network layer independent function and as such does not rely on theTCP/IP-stack. The OAM data gets an Ethernet-header with its own specific EtherType

16

Research Chapter 5

(0x8902) before being put on the wire. Of course this Ethernet Frame also needs adestination MAC-address to send it to which varies depending on the type of messagebeing sent.

The configuration of an OAM Maintenance Domain does not require to manually set theMAC-addresses of all the peers. These can be auto-discovered through sending out amulticast Ethernet frame which announces the MEG End Point (MEP) MAC-address toall the peers. These message also function as a keep-alive message and are thus calledContinuity Check Messages (CCMs). They are sent out at a certain interval and are notreplied to by the other MEPs in the MEG. If a MEP stops receiving CCMs it marks thepeer MEP as ‘failed’ after 3.5 times the interval. Besides these two functions, the CCM

PDUs also provide the counters for the frame loss measurements, an essential KPI.0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Multicast destination MAC-address0180.C200.003y

Source MAC-addressEtherType 0x8902 MEL Version OpCode (CCM=1)

Flags TLV Offset (CCM=70) Sequence Number (0)

Sequence Number (cont.) MEPID

MEG ID (48 octets)

TxFCfRxFCbTxFCb

Reserved (0)

End TLV FCS

FCS (cont.)

Figure 5: Overview of a Continuity Check Message Ethernet OAM frame.

An example of a complete CCM Ethernet frame is pictured in figure 5. The multicastdestination MAC-address of the CCM PDUs is 0180.C200.003y, where y is the MEG

Level (MEL). Apart from the CCM frames there exists one other type of PDU which usesmulticast but which is not used within AMS-IX: Link Trace Messages (LTMs). All othermessages and replies to those messages – including the delay measurement messages

17

Research Chapter 5

– are unicast. The source address is always the devices own MAC-address and theEtherType is defined as the hexadecimal-value 0x8902. The PDU also includes theMEG-level, Ethernet OAM version number and an OpCode which defines the type ofPDU, which for CCM is 0 (zero). The flags include the Remote Defect Indicator-bit and thespecification of the interval. The sequence number remains empty for this specification,as is the reserved field at the the end of the frame. In between these fields the configuredMEP and MEG IDs are included, as well as the counters for dual-ended frame loss.[7]

The differences in multicast and unicast destination addresses causes some unexpectedbehaviour on the AMS-IX MPLS platform. Because the platform learns all destinationMAC addresses and bases its hardware forwarding on that information, it can quicklyswitch any unicast Delay Measurement Message (DMM) from one MEP to the port onwhich the other resides, provided that the destination MAC has already been learned.With multicast addresses this is different, these frames will have to pass through theCPU for rate-limiting purposes and as such have a much higher delay. For measuringframe loss this shouldn’t be much of a problem, until the CPU becomes loaded withother tasks or other broad/multicast frames. When the CPU on the module is too busyit might occur that a CCM PDU gets dropped. This won’t affect the OAM service in gen-eral but would count for loss of a frame when this shouldn’t be true for standard unicastpackets.

To solve this problem the VPLS instance in which the devices reside can be configuredwith the vpls-cpu-protection option. This enables the switches to flood all broad-cast and multicast frames in hardware without sending it up in to the CPU.[11] Al-though this enables the switches to forward the multicast frames directly it also pre-vents the CPU from rate-limiting the traffic. This could potentially allow for high levelsof broadcast traffic and therefore rate-limiting layer 2 Access Control Lists (ACLs) mustbe configured on the physical ports which connect to the NIDs.

5.2.2 Label Switched Paths

Another problem arose while talking to fellow NOC engineers. They noted that theDMMs which were being exchanged by the MEPs would always take the same pathover the platform as long as this measurement would persist. This of course wouldnot matter if this path would be the same for all traffic but within the AMS-IX the pathsare different for all kinds of traffic streams. Because the OAM stream sends a frameevery 100 milliseconds this stream never changes and its path wouldn’t either (as seenin figure 6a). This means that traffic travelling another path could encounter higherdelays which would not be measured by the NIDs.

These paths are the four LSPs which run from PE-router to PE-router over every one ofthe four P-routers. In figure 6 there are two 10 GigE customers presented on the top withpaths over the core in the middle, to a 1 GigE customer on the bottom. Different traffic-streams are load-balanced over four of those so different streams may take different

18

Research Chapter 5

paths. Besides these LSPs there are other paths that won’t change if the streams won’t:the physical links in a Link Aggregation Group (LAG). A backbone connection betweenan AMS-IX PE and a P-router consists of multiple fiber links which aggregate into onelarge logical link. The OAM-stream will start sending data over one of these physicallinks and won’t change, unless interrupted.

core-loctation-1MLX-32

core-location-1MLX-32



PE-1-redMLX-16

PXC

PE-1-blueMLX-32

PE-3MLX-8

(a) OAM measurement using only one LSP witha backup path.

core-loctation-1MLX-32




PE-1-redMLX-16

PXC

PE-1-blueMLX-32

PE-3MLX-8

(b) OAM measurements using all LSPs

Figure 6: The LSPs over the AMS-IX platform

To mitigate these circumstances Virtual Leased Lines (VLLs) can be configured whichcan be set to use only one of the LSPs. Every NID should then be able to give a differentVLAN-tag to each frame so every one of them could be sent over the different VLLs. Thiswould mean all LSPs would be used, as seen in figure 6b. Unfortunate the Accediandevices at the moment do not support configuring more than one VLAN on the OAM

measurements. This has been requested as a feature with the vendor and has beenpromised to be included in the firmware by Quarter 2 of 2011.

The second problem regarding the physical links will be a lot harder to solve. This hasto do with the way the traffic is load-balanced over the links in a LAG. The algorithm

19

Research Chapter 5

used for this is Brocade proprietary and so it can not be known. Reverse-engineeringthe algorithm would be a possibility, were it not that it is most likely changed withevery firmware revision, according to NOC engineers. Because of this obstacle it wasdecided not to go to deep in to this hardware limitation and leave the KPIs as they wereintended: as indicators of the network performance.

5.2.3 Redundancy

One of the design requirements is also to be able to put the devices directly in theconnection between the customer and the platform. AMS-IX provides a very redundantnetwork which uses the PXCs to switch a customer from one of the PE-routers to anotherin case of a failure (see also section 2.1.1). It also uses MPLS to provide backup-pathsover each core switch. This very redundant and resilient network is as weak as it’sweakest link. If a device like a EtherNID would be placed between a customer and theplatform a new single point of failure would be created. For this reason this require-ment has been let go for the time being. Because the NIDs are all connected in the sameway that a customer would and the traffic they send is completely separate from theirtraffic, there will not be a significant difference in the measurements for a new singlepoint of failure to be created.

5.3 Timing

While running test measurements on the equipment provided by Accedian there weresome unusual patterns to be seen in the graphs between the MetroNIDs and the Metro-NODE. The MetroNODE one way delay graphs were shaped like a sawtooth from oneof the NIDs with an inverse of that graph to that NID. This could indicate some kind ofunstable path, but the two way delay measurements were unaffected by this problem.

After consulting with the vendor it became clear that there was a clock drift issue be-tween the two models; a typo in a constant used by the time syncing algorithm. Solvingthis problem helped to understand the importance of synchronised clocks between thedifferent NIDs. Because of the high speed network used by AMS-IX variations of a cou-ple of microseconds are noticeable in the graphs.

5.3.1 Hardware Timestamping in NTP

The probes use NTP to synchronise to a common time source which is a TTM server.These distributed servers all use GPS antennas on the roof of their datacenters to receivetheir proper timing. To reduce the variations Accedian recommended to look for a timesource which could time stamp their NTP-packets in hardware, instead of letting thesoftware do it and having to travel all the way through the Operating System (OS) andnetworking-stack which is the case with the Linux-based RIPE TTM devices. In figure 7

20

Research Chapter 5

the structure of an NTP packet is shown. The timestamps consist of 64-bits and the lasttwo are used to sync a client to a server. The server stamps the Received field whenit processes the incoming request and stamps the Transmit field when it sends out areply.[12] The synchonisation process as well as the formula the client uses to set itsown time, can be seen in figure 8.

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

LI Status Type PrecisionEstimated Error

Estimated Drift RateReference Clock Identifier

Reference Timestamp

Originate Timestamp

Receive Timestamp

Transmit Timestamp

Figure 7: Structure of an NTP packet.

The RIPE TTM devices would stamp their packets in the Application layer of the OSI-model but with a hardware timestamping device the packet would receive its time-stamp just before being put on the wire (as seen in figure 9). Although this is an OSI-layer violation, it would improve the quality of the timing and above all make it morestable. The variations should become much smaller because CPU load wouldn’t affectthe accuracy of the timestamps.

To test this claim, one of the Accedian MetroNIDs was set to be an NTP-server itself.These devices work with hardware-based Field-programmable gate arrays (FPGAs) andalways use hardware timestamps. The MetroNODE and other MetroNID were thensynchronised to this NID. After letting the synchronisation take place over 24 hours theone way delay graph appeared more smooth and continued with less spikes and dipsover the following days.

21

Research Chapter 5

request

reply

response time

Client Server

network latency

Network/OS stack delay

Transmit Timestamp

Received Timestamp

Originate Timestamp

= network latency + TransmitTS= (RecTS - OrigTS) + TransmitTS

Current Time

Figure 8: Synchronisation process between a client and the NTP-server.

ApplicationPresentationSessionTransportNetworkDatalinkPhysical

NTP

Software Timestamp

Hardware Timestamp

Figure 9: The path that an NTP-packet travels through the OSI-layers.

22

Implementation Chapter 6

6 Implementation

In this chapter the physical implementation of the devices and their configuration willbe discussed. The way of collecting data will be looked as well.

6.1 Hardware

In section 5.1.2 it was shown that the only fitting solution would be the EtherNID de-vices from Accedian. After consulting with the supervisor and CTO eight of these de-vices were ordered. One for every 1 GigE switch on the eight colocations in the Amster-dam area, these switches are represented by the bottom row of devices in figure 1. The10 GigE MetroNODEs will be ordered and implemented in a later stadium after moretesting with the GigE devices.

Together with Hardware Implementation Manager Ariën Vijn it was coördinated whereand how the devices would be placed within the AMS-IX racks. For the most part thedevices were connected through copper Gigabit Ethernet links except for the deviceson TeleCity 4 and Interxion which connect to the platform using fiber links. All themanagement ports were connected to the different management switches and givenan IP address in the AMS-IX space. The ports connecting to the production networkremain without IP-addresses because OAM does not need them and can be put to betteruse when given to customers.

The Domain Name System (DNS)-records were already configured for the managementports and the layer 2-ACLs on the switch-ports were also configured with the corre-sponding MAC-address to allow the NIDs to send traffic.

Location IP Hostname MAC-addressEquinix 91.200.16.181 nid-eqx-015 0015.ad07.171f

Interxion 91.200.16.182 nid-ixn-016 0015.ad07.1647NIKHEF 91.200.16.183 nid-nik-011 0015.ad06.a2df

SARA 91.200.16.184 nid-sar-010 0015.ad06.a5cfTeleCity 2 91.200.16.185 nid-tel-012 0015.ad06.a6dfTeleCity 4 91.200.16.186 nid-tc4-137 0015.ad06.a19f

GlobalSwitch 91.200.16.187 nid-glo-013 0015.ad07.174feuNetworks 91.200.16.188 nid-eun-014 0015.ad07.176f

Table 2: Device configuration

The hostnames were chosen to reflect the location of the NID and the switch to whichit connects. Within the AMS-IX platform, this is a standard naming convention. Theports belonging to the NIDs were all put in a separate VPLS-instance on which CPU

Protection was configured so that multicast frames could be flooded in hardware (see

23


section 5.2.1). This separate VPLS instance was chosen so that not all ports had to beconfigured with layer 2-ACLs to rate limit broadcasts. This choice won’t change themeasurements in any way because the Delay Measurement Messages will still take oneof the same paths that the customer-traffic would take. When the vendor has imple-mented the function to support more VLANs per MEG, separate VLLs will be configuredover all four LSPs.

6.1.1 NID Configuration

The Accedians do not support exporting their configuration in a standard text-file,therefore a list of Command Line Interface (CLI) commands to execute on each of thedevices was composed. An example of such a config is given in Appendix C.

The networking configuration would be as shown in table 2 (lines 1 through 11 in Ap-pendix C). All ports except for the Network and Management ports are disabled andonly the later gets an IP-address. To poll the values measured by the NIDs SNMP is setto accept connections with specific communities (l. 12-17 ) and also to send traps to alog server (l. 18 ). On top of these traps the devices send their errors and messages to asyslog server on the management network (l. 19-20 ). This double logging is part of thetest trial and new logging-method based on the Zabbix software.

An important part of the configuration are the NTP time settings (l. 21-24 ). All of theNIDs are made to listen to the NID on the NIKHEF location which gets its timing from aRIPE TTM server on the same location. The DNS server contains a record to point to thatNID which is time-nid.noc.ams-ix.net so that changing the source would merelybe a change to the DNS record. The synchronisation-mode is set to high-resolutionwhich means that each device polls the NTP server every minute to check its time.

The configuration of the OAM measurements follows the same structure which has beenshown in figure 3. It starts with defining a MD with a Level and a name which followsa DNS-like convention (l. 25 ). It is set on internal index-number 9 below the defaultMDs in the device. Below this domain, a MA is created that includes the MEPIDs of thedevices that will participate in the measurements (l. 26 ). It also receives a name, anindex-number, a reference to the previously configured MD-index and a value for theCCM-interval. This is the time in milliseconds between each CCM-message between theMEPs. Accedian’s lowest supported interval is 1000 milliseconds.

After these global settings are configured every device needs to get it’s own MEPID.This is configured together with specifying a port on which OAM needs to be enabled,a reference to it’s top MA and which errors it should propagate to it’s peers (l. 27 ).When a MEP is configured it can start measurements to the rest of the MEPs, but somevalues need to be defined (l. 28-34 ). The interval is set to 100 milliseconds – againthe lowest value supported by Accedian – which means that one and two way delaysare measured ten times per second. All these values are aggregated within a specificreference period. This is set to 1 minute: the average value over 600 measurements is

24


taken. To allow alarms to be generated and traps to be sent, three types of thresholdsare defined as shown in table 3

Type One Way Two Way DefinitionMax Delay 2 ms 2 ms This is the absolute maximum de-

lay value which can be measuredby one Delay Measurement Message.

Delay Threshold 3 ms 3 ms If the times the maximum delayvalue is exceeded reaches this valuean alarm is sent.

Average Delay 1 ms 1 ms If during the reference period thisvalue is exceeded an alarm is im-mediately sent.

Max Delay Variation 1 ms 1 ms This is the absolute maximum jit-ter value which can be measured byone Delay Measurement Message.

Delay Variation Threshold 2 ms 2 ms If the times the maximum jittervalue is exceeded reaches this valuean alarm is sent.

Average Delay Variation 1 ms 1 ms If during the reference period thisvalue is exceeded an alarm is im-mediately sent.

Table 3: OAM threshold values

For the SLA measurements the Average Delay values are the most important, while themaximum values are mostly used as indicators when a large error occurs on the plat-form. The average thresholds are set to their lowest supported values of 1 millisecondwhile the draft SLA contains much lower values. A feature request has therefore beensent to Accedian to support the values below 1 millisecond.

Frame loss is also defined in the settings but here a reference period can not be set. Asshown before, the CCMs include the counters for packet loss and so this measurementinherits that value of 1000 milliseconds. A threshold is also set to 1 percent and thereference period is kept the same as the delay measurements (l. 35-41 ).

6.2 Data Collection

A Virtual Machine (VM) running Linux has been set up to collect the data using MultiRouter Traffic Grapher (MRTG) which was written to poll routers for their traffic statistics.The program is adaptable to any situation and can also be used to collect the infor-mation from the NIDs every minute. To do this special databases called Round Robin

25


Databases (RRDs) are used to store the values over time. These databases store valuesevery interval (eg. one minute) and consolidate these values after a certain period oftime. This way not all data has to be kept at a very low resolution, disk space can besaved and computation with those values takes less time. By default MRTG uses it’sown text-based database but to set the interval to lower than 5 minutes it can call uponRRDtool, the software handling these database-files.

An RRD-file can store two KPIs in separate datasets. For the Accedian EtherNID setupthe one and two way delays are stored in one file, one and two way jitter in anotherand frame loss is alone in a third file. Three files are needed for any measurement fromon NID to the other. For 8 devices this would correspond to 8 devices * 7 peers * 3 files= 168 RRD-files. Every file needs a configuration within MRTG and thus 168 sections ofconfiguration are needed. Such a section would look like this:

Target[5min_delay_nid-eqx-015_nid-ixn-016]:.1.3.6.1.4.1.22420.2.7.1.1.2.1.10.1&.1.3.6.1.4.1.22420.2.7.1.1.4.1.10.1:[email protected]

Title[5min_delay_nid-eqx-015_nid-ixn-016]: One and Two Way DelayMaxBytes[5min_delay_nid-eqx-015_nid-ixn-016]: 100000000

The first line contains the target specification between square brackets and after thatspecifies the SNMP Object Identifiers (OIDs) to be polled with a specific community stringat a certain host. This line is the only useful line, the other lines containing Title andMaxBytes are settings required by the MRTG software but are not used by RRDtool.MaxBytes for example is nothing more than the maximum value to be considered real-istic but is expressed in bytes because the program was designed to collect exchangedbytes. Another property which shows this specific purpose is that MRTG by default onlyexpects values it’s measuring to grow over time. Delay measurement vary constantlyand so all the datasets are set to the ‘gauge’ type, to allow the OID values to fluctuate.The OIDs to be polled are listed in table 4.

OID Value.1.3.6.1.4.1.22420.2.7.1.1.2.1.10.index One Way Average Delay in microseconds.1.3.6.1.4.1.22420.2.7.1.1.3.1.10.index One Way Average Jitter in microseconds.1.3.6.1.4.1.22420.2.7.1.1.4.1.10.index Two Way Average Delay in microseconds.1.3.6.1.4.1.22420.2.7.1.1.5.1.10.index Two Way Average Jitter in microseconds.1.3.6.1.4.1.22420.2.7.1.2.2.1.18.index Packet Loss in millions of a percent

Table 4: OIDs for various KPIs

After the whole MRTG configuration is complete and the daemon is started, values startto get collected. The RRD-files which did not exist yet are automatically generated andfilled with the average values every minute. By default the values get consolidated toaverages over 30 minutes, then 2 hours and finally to 1 day.

26


6.2.1 Scripts

All these collected KPI measurements will eventually need to be presented to the mem-bers, the NOC and other parties. RRDtool includes function for that: rrdtool graph.It can visualise the data from the files by graphing it and writing a standard image file.To do this rrdtool graph is given parameters to know which RRD-file it must use,where it should write the image, what kind of values must be written in it, which time-period much be shown, etc. Eventually a graph similar to the one in figure 10 would bethe result for the one and two way delays from Interxion to NIKHEF

Sat 12:00 Sun 00:00 Sun 12:00 Mon 00:00 0

100

200

300

400

500

microseconds

delay from nid-ixn-016 to nid-nik-011

RRDTOOL / TOBI OETIKER

95th SLA maximum minimum One Way 210.65 µsec 500 µsec 226.61 µsec 174.42 µsec Two Way 434.51 µsec 1000 µsec 435.61 µsec 428.35 µsecCopyright © 2011 AMS-IX B.V. [updated: 10-Jan-2011 10:58:10 +0100 ]

Figure 10: Graph generated by RRDtool

These images can be generated every reference period for all devices which means 168images would be replaced each minute. This would not be very efficient because theywill not be looked at all the time. To make this more efficient a bash-script is written tobe called from the web browser which takes arguments for source and destination NID,time period and KPIs. This script is included in appendix D. This script then generatesthe graph image on-the-fly using the raw output from the RRDtool command. This way,there is no image written to disc but directly to the requesting browser. This also addsthe advantage that the image is always up-to-date, with every refresh it looks at newdata.

For one minute devices this would be an improvement in efficiency but to report thevalues to customers each month this method would be too much, considering that theseimages would not change over the course of one month. A script has been written torun on the first day of each month to generate a graph and calculate the values of thevarious KPIs over the previous month. The images are all written to a directory whichcan be reached from the internal network to include them in the SLA reports to be sentto customers, the values are reported to customers through a 95th percentile. This isa formula in which the highest 5 percent of the data is not included in computing amaximum value over that month. This allows for some fluctuations in the graphs.

For the NOC engineers an overview page has also been created with a matrix contain-

27


ing all the measurement graphs that will enlarge when the cursor moves over it. Ascreenshot of this page is shown in figure 11.

Figure 11: Overview page KPI measurements

6.3 Timing Sources

In the measurement graphs it is obvious that the line representing one way delay is notas consistent as the two way delay. Because this value is the only one that is influencedby difference in time between the NIDs, this pattern is not seen in any of the othergraphs. Two way measurements base their departure-time on the originating clock andbecause the packet makes a round-trip it’s arrival-time is also based on the same clock.One way jitter measurements do not base their timing on both devices. One of theNIDs receives two packets within a certain period of time and if the period in which itreceives the third packet is higher or lower, this would count as delay variation, or jitter.This is shown by the Reserved place for RxTimeStampf in figure 12, here the receivingNID places it’s timestamp to compare to the previous received values.

28


Using this figure it can also become clear how the one way delay measurements work.The fields in the header of the frame are all discussed in section 5.2.1, apart from thedestination MAC-address which is now a unicast address of the receiving NID. Theoriginating device sends this frame with a timestamp of his clock in the TxTimeStampffield. It travels through the network until the frame reaches it’s destination address.Here it gets timestamped again immediately after being received in the RxTimeStampffield. It can then be processed by the OAM system, it subtracts TxTimeStampf fromRxTimeStampf which results in the one way delay between the two NIDs.

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Unicast destination MAC-address

Source MAC-addressEtherType 0x8902 MEL Version OpCode (CCM=1)

Flags TLV Offset (CCM=16)

TxTimeStampf

Reserved for receiving device (for RxTimeStampf)

End TLV FCS

FCS (cont.)

Figure 12: Overview of a One Way Delay Measurement Ethernet OAM frame.

One way measurements essentially compare the timestamp of the originating devicewith the arrival time at the end-device. Subtracting those two values results in a timedifference, called delay. Because these delay values are quite low within the AMS-IX

platform it becomes quite essential to have the clocks synchronised as precisely as pos-sible. In section 5.3.1 it was already discovered that hardware timestamping wouldimprove these measurements. A new time source would be needed because the RIPE

TTM boxes did not support hardware timestamping and were reaching their end-of-life.

Near the end of the project a still on-going search was started to find such a newtime source. The requirements for such a device are in the first place to support hard-ware timestamping, it should be easily implemented within the current network andit should be as reliable as possible with a small budget. Table 5 lists the options thatwere looked. Note that only two devices support NTP hardware timestamping. This isbecause by default NTP is an application layer protocol and thus inserting a hardwaretimestamp at the data link layer would be an OSI-layer violation. The only possibility tosolve this is to implement the whole function in hardware using FPGAs. This is preciselywhat both the Z-1000 and MetroNID with GPS use to support NTP timestamping. Theformer is an appliance integrated with the antenna that can be placed directly on theroof. It is powered through the Ethernet cable so there doesn’t have to be a power outletoutside and doesn’t take up rackspace. The latter is a familiar name and was discov-ered when talking with Accedian about accurate time-syncing. The vendor produceda small number of devices with a GPS receiver which would enable them to distribute

29


that accurate time.

Device GPS GigE HW Stamp Accuracy CostEndRun Unison yes no no <10 microseconds e2,775.00

Oscilloquartz 5220 yes no no <20 microseconds N/AMeinberg M300 yes no no <50 microseconds e2,482.00

Symmetricom S200 yes yes no 7 microseconds e3,995.00Brilliant Telecom Z-1000 yes no yes 250 nanoseconds e5,570.00

Accedian MetroNID GPS yes yes yes 250 nanoseconds e1,229.00

Table 5: Comparison of NTP time sources

All devices support GPS as a time source and their accuracy is based on the differencebetween the current GPS time and the time in the NTP-packet when it is being put onthe wire. With hardware timestamping this is as low as possible because the time is setright when the NTP-packet leaves the device. But this is only true when the timestamphas just left the device, while taking it’s path along the network the packet can facesome delay which will make it less accurate. NTP tries to consider this delay in settingthe time for a host but only if the delays do not vary a lot over time.

A solution that could be implemented to solve the variation in the network is IEEE

1588v2 or Precision Time Protocol (PTP). This relatively new protocol is said to delivertime around a network in sub-second accuracy.[13] The disadvantage of this new stan-dard is that it requires some new hardware within the management network whichwould also support PTP. Although this is a possibility in the future, at the moment theAccedian NIDs do not support PTP but will do so in the coming year.

The four devices that do not support hardware timestamping are for the most partembedded Linux servers with a standard NTP daemon and are subjectable to delayscaused by the OS and networking stack. Vendors like Oscilloquartz and Symmetricomdo produce hardware timestamping devices but they are targeted at big operators whoneed synchronisation for their Synchronous Digital Hierarchy (SDH) network, for AMS-IX

the performance increase would not be worth the investment.

A requirement which was taken for granted is the maximum length of the antennacable. For roof-access in one of the datacenters this has to be at least 100 meters butpreferably longer. The Brilliant Telecom Z-1000 has a maximum of 100 meters whichis consistent with the maximum length of Ethernet cabling. The Accedian MetroNIDswith GPS can use any standard antenna and depends on that manufacturer to specifya maximum length of the cable. This can be as short as 30 meters so a reliable vendormust be found.

This search for a new reliable time source is still ongoing. In the next chapter the find-ings of the research that has been carried out will be summarised and compared to thepreviously set requirements.

30

Conclusion Chapter 7

7 Conclusion

In the introduction of this paper two questions were stated that needed to be answered:

1. How can the performance of an OSI-layer 2 platform, such as provided by AMS-IX

to its customers, be measured?

2. How can AMS-IX provide their customers with the measured values to be in-cluded in an SLA?

The answers to these questions were subject to five requirements listed in section 3.2which are discussed in the following paragraphs.

7.1 Requirements

The first requirement that the solution had to fulfill was to measure the KPIs as definedin section 3.3. The possible solutions for this were researched by comparing differentstandards with each other. The one that could provide the platform with measurementsof nearly all KPIs was Y.1731 developed by the ITU-T. Only the ‘uptime’ measurementcould not be performed because this would need some additional checking whetherthe other provided port of a customer also had the same problems.

Showing the data of the other KPIs and making the stats publicly accessible, were thesecond and third requirements. These were solved by using SNMP polling over themanagement network to collect the values and graph those on-the-fly. The values areone-minute-averages and provide a performance overview of the whole platform. Cus-tomers can select between which two devices they want to see the values for delay, jitterand frame loss. The same principle will be true for the public stats, but these will beprovided by Synleaf. This company has worked with AMS-IX in a subproject whichenables them to show the real-time values measured on the platform.

The use of a web portal in which customers can see the data that they need is only partof the fourth requirement. The customer would also be interested in the values over acertain month. This would enable them to see if AMS-IX has met their predefined SLA.This is done using the 95th percentile formula, which sorts the values from the pastmonth from high to low, discards the top five percent and takes the maximum value ofthe remaining 95 percent. This allows for some sudden changes in the values withoutaffecting the performance indicators over the whole month.

The last requirement was actually not taken in to account, as has been discussed insection 5.2.3. It would require to insert a NID in every connection between a customerand the network. Because the measurements are not based on the traffic of the customerand the device is connected in the same manner that any other customer is, this willnot improve the results of the measurement significantly. And because this adds a new

31


single point of failure in an otherwise completely redundant network, this requirementwas dropped.

7.2 Research Questions

In summary, the first question has been answered by using two standards providedby the IEEE and ITU-T. The measurement functionalities are provided by ITU-T Y.1731and meet all the measurement demands from section 3.3. The devices meeting thesedemands as well and also supporting the ITU-T standard are produced by AccedianNetworks, where new devices also have been ordered.

The values collected by the devices are stored and processed by a server with RRDtool.This answers the second questions as well, because this tool is also being used to graphthe data and calculate the monthly values to be provided to the customer with an SLA.

All requests by AMS-IX for this project have been met but there are two tasks whichhave yet to be checked off.

7.3 Recommendations

One of the important observations made while further testing the new devices was thatthe synchronisation of the clocks would need to be very precise. Because the platformhas such a high performance, the smallest differences between the clocks of the deviceswould be visible in the one way delay graphs. The time sources AMS-IX uses at themoment had their maintenance contracts ended and are not able to timestamp NTP

reliable enough, therefore a new set of NTP servers is being searched for. These will beneeded in the coming months to provide the platform with a reliable source of time andminimise the variations in the one way delay graphs.

Another task that has yet to fulfilled is the implementation of 10 GigE NIDs, calledMetroNODEs. These devices will be needed to supply the customers connecting througha high speed link with reliable values as well. Adding these devices also means that thedata stored will increase tremendously. At the moment there are 18 switches with 10GigE connections, add to them the 8 existing devices and the count of RRD-files will riseto (24 * 23 * 3 =) 1656. This shouldn’t pose a problem for the server but the VM-guestmight need some speed and memory upgrades.

The 10 GigE devices are already ordered with Accedian but only under the conditionthat three features will be added to the software:

Polling of the Instantaneous Values from the OAM DMM measurements. This allowsfor the generation of real-time graphs, a subproject that was started together withSynleaf[14]. At the moment these values are acquired through the proprietaryPAA protocol (section 4.5).

32


Multiple VLAN-tagging in a MA/MEG of OAM. Using this feature, the measurementsof the devices can be switched over each LSP. Without it, the measurement streamwill only use one of the paths which generates a less representative result.

Threshold for alarms on DMM lower than 1 millisecond. This enables for traps to gen-erated quicker, should a problem arise on the platform.

The manufacturer conceded with these requirements, which should all be implementedby the second quarter of 2011. By that time, the new 10 GigE devices would be fullyimplemented and tested and a reliable time source will be found as well.

33

Performance measurement infrastructure project Appendix A

A Performance measurement infrastructure project

Stagiair: Michiel AppelmanBegeleider: Attilla de Groot

A.1 Introduction and Purpose of the project

This project aims at building an infrastructure that measures and reports on the perfor-mance of the Amsterdam Internet Exchange (AMS-IX) Metro Ethernet Network. Perfor-mance of the network can be identified by the following parameters:

• Availability of the service

• Packet loss

• One way or round trip packet delay

• One way or round trip packet delay variation (jitter)

A number of different applications can be distinguished that need or do not need to beimplemented.

1. An overall overview of the performance of the network by adding measuringdevices to the each of the access routers and measure the performance betweenthese. This will provide an average performance overview as can be experiencedby every customer and will set a baseline on AMS-IX service levels.

2. For specific customer groups it might be necessary to have more direct mea-surements between the participants in these groups. The idea is to put a mea-suring device in the access connection of the customer and measure the perfor-mance between this device and similarly connected devices from customers inthe same group. The purpose of this setup is to be able to define Service LevelAgreements (SLAs) and measure adherence to this SLA

3. The definition and specific implementation of Operations, Administration and Main-tenance (OAM) on the AMS-IX platform to allow both AMS-IX Network OperationsCenter (NOC) and possible customers to see performance of the AMS-IX platformthe L2 level.

For each of the three mentioned applications it is necessary to:

• investigate how it can be implemented

• test implementation in the lab

34


• evaluate impact in the platform

• after agreement with the AMS-IX NOC implement the solution

An important aspect of the project is presentation of the measurement results and con-formance on the defined service levels on the AMS-IX website The project will be splitinto different stages:

A.2 Stage 1: Measurements and reporting on Key Performance Indicators(KPIs) between a selected subset of the AMS-IX access routers.

Initial devices selected are Accedian MetroNID en MetroNode 10G. For the initial trailwe have two metroNIDs connected to the GE access routers at Equinix and Interxion.The single MetroNode10G need can be connected to any suitable 10GE access router onany of the other sites. The following need to be done:

1. Setup all three devices to measure the critical KPIs (Layer 3)

• Availability of the service• Packet loss• One way or round trip packet delay• One way or round trip packet delay variation (jitter)

2. Describe how the parameters are derived and give an estimate of the reliabilityof the measured parameters. Specifically this is the case for the one way delayparameter as this is dependent on a synchronized clock or a vendor specific algo-rithm.

3. Create a web page where for the different KPIs performance matrices are dis-played.

4. Based on the experience with the devices decide if these are the right devices tobuild the service on.

(a) If “yes” continue with Stage 2(b) If “no” select and repeat with different device.

5. An additional test and of influence for the final decision to go on with the Acce-dian boxes is the performance of the box in a customer connection.To be tested in the lab:

(a) Simulate customer connection to the platform terminated on the MetroNidand/or MetroNode10G instead of directly to the platform.

(b) Test performance via this indirect connection between two in such a wayconnected customers.

(c) Repeat measurement of the KPIs as above (in this stage no need for website)

35


A.3 Stage 2: Measurements and reporting on KPIs between a selected subsetof the AMS-IX access routers

Stage 2 is essentially the generalization of stage 1 point 1 to 4

1. Order devices for all access switches

2. Create web based performance overview between all access routers (see A.2.3)

(a) Since there are a lot of devices it is essential to pay specific attention to theuser interface to the data. Cooperation with member relations and or mar-keting might be required here.

Other stages to follow later.

36

Plan van Aanpak Appendix B

B Plan van Aanpak

B.1 Inleiding

Voor de AMS-IX zal in deze afstudeerperiode een onderzoek worden uitgevoerd metbetrekking tot performance measurement op het platform. Dit document zal daarbijals leidraad worden genomen. In de volgende pagina’s zal een beschrijving wordengegeven van het project.

B.1.1 AMS-IX

AMS-IX is één van de grootste Internet Exchanges in de wereld en biedt aan haar klanteneen Layer 2 verbinding aan met overige (internationale) netwerken. Het kantoor bevindtzich in de binnenstad van Amsterdam en de apparatuur is verdeeld over meerdere dat-acentra in en rondom de hoofdstad. Op deze punten bestaat de mogelijkheid voorbijvoorbeeld Internet Service Provider (ISP)’s om verbinding te maken met het platform.Dit onderzoek wordt begeleid door Attilla de Groot van het NOC maar heeft betrekkingop het gehele bedrijf. In het NOC zitten de network engineers die zich bezig houden methet beheren van het platform van AMS-IX. Ook zorgen zij ervoor dat het netwerk meegaat met de tijd en kan voldoen aan de steeds groeiende stroom data.

B.1.2 Inter-IPX

Dit onderzoek wordt onderdeel van een groter project voor een nieuwe service bijhet bedrijf. De service, Inter-IPX, zorgt voor een verbinding tussen meerdere IP eX-change (IPX)es en is door de Global System for Mobile Communications Association (GSMA)gestandaardiseerd. IPXes zijn netwerken van mobiele operators die data uit willen wis-selen buiten het publieke internet om. Deze operators stellen ook eisen met betrekkingtot Quality of Service (QoS) aan het platform van AMS-IX. Voorlopig wordt de service alstrial aangeboden maar de eisen moeten worden vastgelegd in een nieuwe SLA. Om teweten of aan deze SLA kan worden voldaan zal meetapparatuur nodig zijn.

B.2 Doelstelling

Op dit moment is er nog geen exacte meting mogelijk op het netwerk van AMS-IX. Ditis echter wel nodig als men een SLA heeft waaraan voldaan moet worden. Het doel isom een infrastructuur op te zetten die de performance van het AMS-IX netwerk meet enrapporteert.

Onder performance worden de volgende KPI’s verstaan:

37


• Availability (up-time)

• Reliability (packet loss)

• One Way Delay and Variation (jitter)

• Two Way Delay and Variation (Round Trip Time (RTT))

B.3 Projectopdracht

Het bedrijf heeft al een drietal apparaten (zgn. probes) geselecteerd van AccedianNetworks die als demo worden gebruikt. Deze zijn direct aangesloten op de accessswitches en zullen in de beginfase van het project worden gebruikt om te zien of ze vol-doen aan de gestelde eisen. Mocht dit niet zo zijn dan wordt een lijst van alternatievenopgesteld. Tevens wordt de nauwkeurigheid van de probes met betrekking tot de OneWay Delay (OWD) metingen onderzocht.

Zodra de geschikte leverancier gevonden is, zal de projectgroep gaan kijken naar demogelijkheid om de probes direct tussen de klant-aansluiting te plaatsen. Hierdoorkomen ze direct in het pad tussen de twee klanten en worden de metingen meer be-trouwbaar.

Een ander onderdeel houdt in dat de probes uit te lezen moet zijn. De data die opges-lagen is zal overzichtelijk moeten worden gepresenteerd in een website. De data diehiervoor nodig is bestaat uit de verschillende KPI’s die al genoemd werden op pagina37. De gegevens uit de probes zouden tegenover elkaar uitgezet moeten worden perKPI een matrix wordt getoond.

Uiteindelijk moet dit leiden tot een implementatie die voldoet aan alle eisen met be-trekking tot meting van verschillende KPI’s, rapportage aan de klant en stabiliteit inhet platform. Van belang is een afstemming met de projectgroep die gaat over de on-twikkeling van de Inter-IPX service. Projectleider hiervan is Luisa Garcia Montoya.Dit onderzoek valt echter direct onder projectlid Attilla de Groot die vanuit het NOC

meewerkt. Een gedetailleerde beschrijving van deze opdracht door AMS-IX is te vindenin Bijlage B.

B.4 Activiteiten

De volgende activiteiten worden tijdens de loop van de afstudeerstage uitgevoerd:

1. Informatie verzamelen

Opdracht AMS-IX heeft een opdracht geformuleerd (zie ook Bijlage A) die uit-gevoerd moet worden. Deze zal allereerst moeten worden vernomen.

38


Achtergronden Om de achterliggende gedachte van de opdracht te kunnen achter-halen is het van belang om op de hoogte te zijn van de voorgeschiedenis. Ditis te achterhalen door te praten met medewerkers die hier meer ervaring meehebben.

2. Plan van Aanpak

Opstellen Het Plan van Aanpak wordt als leidraad gebruikt voor de rest van hetonderzoek. Deze wordt opgesteld aan de hand van informatie die verkregenis over het bedrijf, de afdeling en de opdracht.

Bevestigen Met de school- en bedrijfsbegeleider wordt overlegd of het opgesteldedocument correct is en deze kan worden gebruikt.

3. Onderzoek

Opties Is het Plan van Aanpak akkoord dan wordt het onderzoek gestart. Ver-schillende oplossingen worden bekeken en vergeleken met de eisen van hetbedrijf.

Oplossing De beste oplossing wordt gekozen met de daarbij behorende toelicht-ing. Deze wordt daarna uitgewerkt.

4. Implementatie

Plaatsing Zodra duidelijk is hoe de gekozen oplossing zou behoren te werkenwordt deze in het platform toegepast. Dit houdt in dat eventuele hard- ensoftware in gebruik zal worden genomen.

5. Evaluatie

Gebruik binnen infrastructuur Verschillende problemen kunnen naar voren komentijdens het gebruik van de gekozen oplossingen en die zullen opgelost moetenworden.

6. Rapportage

Presenteren Alle voorgaande stappen zullen uiteindelijk worden gedocumenteerdin een verslag en gepresenteerd worden aan het bedrijf en de hogeschool.

B.5 Producten

Verschillende producten zullen worden opgeleverd naast het uiteindelijke verslag.

B.5.1 Plan van Aanpak

Dit document is het eerste product dat zal worden opgesteld. Hierin worden de plan-ning, andere producten en deelnemers beschreven. Het biedt een houvast gedurendede looptijd van het project.

39


B.5.2 Planning

Voor AMS-IX is het van belang te weten hoe de komende tijd wordt ingedeeld. Dezeplanning zal gepresenteerd moeten worden op de project vergadering van maandag 20september.

B.5.3 Onderzoek Apparatuur

Om een beslissing te maken tot aanschaf van apparatuur zal er allereerst een gedegenonderzoek moeten worden gedaan. De uitslag hiervan beïnvloedt deze beslissing uit-eraard.

B.5.4 Verslag en Presentatie

Dit zijn twee producten in één die te maken hebben met de afronding van het projecten de stageperiode. Afsluitend wordt het verslag gepresenteerd aan de hogeschool.

B.6 Randvoorwaarden

Het project loopt vanaf 1 september 2010 en zal 20 weken in beslag nemen. Het eindevan het project zal na oplevering van het eindproduct zijn of na afloop van 100 werkda-gen. Als eindproduct wordt een volledig werkend systeem bedoeld waarmee kan wor-den bepaald of de gestelde SLA’s worden gehaald.

B.6.1 Randvoorwaarden

Het eindproduct kan als succesvol worden beschouwd als,

1. de gestelde KPI’s kunnen worden gemeten over het platform;

2. de data hiervan kan worden weergegeven op een website;

3. geaggregeerde statistieken publiekelijk weergegeven kunnen worden;

4. klanten hun eigen specifieke gegevens overzichtelijk kunnen benaderen;

5. het uiteindelijk mogelijk is om metingen tussen specifieke klanten in te zien, en

6. deze opzet schaalbaar is naar alle klanten.

40


B.7 Kwaliteit

De kwaliteit van het project staat of valt door de terugkoppeling en overzichtelijkheid.Voor het Plan van Aanpak en andere documenten geldt dat deze eerst enkel als conceptworden opgesteld. Deze wordt dan ter beoordeling gestuurd naar de begeleider vanAMS-IX en na zijn goedkeuring terug naar school. De verschillende aanpassingen enversies worden apart bewaard.

B.8 Kosten/baten overzicht

B.8.1 Kosten

Voor AMS-IX worden in de eerste fase kosten gemaakt door de inkoop van testappa-ratuur en de inzet van manuren in het gehele project. Mocht de apparatuur bevallendan wordt overgegaan tot een grotere aankoop in een later stadium.

B.8.2 Baten

Belangrijk is om een extra service aan te bieden voor klanten. Hierdoor gaat AMS-IX mee met de ontwikkelingen in de markt en kan het blijven groeien. Ook stelt ditonderzoek het bedrijf in staat om in een later stadium SLA’s aan alle klanten aan tebieden.

B.9 Risico analyse

Door een overzicht te maken met problemen waar tegen aan kan worden gelopen,wordt de kans op een succesvolle afronding vergroot. Op deze manier kan hier opin worden gespeeld.

B.9.1 Tijdnood

Het project moet binnen een bepaalde tijd af zijn en dit gaat met de nodige druk gepaard.Om te voorkomen dat hierdoor fouten worden gemaakt is het belangrijk om te commu-niceren met de overige projectleden, zo kunnen zij rekening houden met een eventuelevertraging of assisteren bij de afronding.

B.9.2 Extra BV

Al voordat dit project gestart werd sprak het management team met de leden van deAMS-IX vereniging over het vormen van een extra BV. Dit met als doel om de risico’s,

41


die SLA’s en overheidsregulering met zich meebrengen, in te perken. De uitwerkinghiervan is echter nog niet volledig en het project wordt nu binnen de bestaande vormuitgevoerd. Het is onduidelijk in welke bedrijfsstructuur de service uiteindelijk aange-boden gaat worden. Voor het project is dit geen risico aangezien de service in elk gevalgeleverd moet worden.

B.9.3 Expertise

Wat wel een probleem kan vormen is dat de student te weinig kennis heeft over hetonderwerp of gebruikte technieken. Dit risico kan worden beperkt door gebruik temaken en te leren van de expertise van de werknemers van AMS-IX.

B.10 Planning

Zie hiervoor pagina 43.

42

Werkzaam bij AMS-IX 100d

Plan van Aanpak 11d

Goedkeuring PvA

IPX VPLS Instance 11d

Inhangen Accedian MetroNODE 2d

Voorbereiden Presentatie 10d

Presenteren Planning

Onderzoek mogelijkheden Accedians 7d

Beslissing tot aanschaf

Uitwerking van mogelijkheden 11d

Plaatsen overige Accedians 2d

Overzichten in My AMS-IX 5d

Welke specificaties in rapportage

Rapportage maken 1d

Start Trail 1d

Evalueren Accedians en Stats 21d

Grotere Implementatie Opzetten 1d

Rapport schrijven 85d

Rapport ter beoordeling 5d

Rapport verbeteren 5d

Rapport

Presentatie voorbereiden 5d

Presentatie

2010, Kw 4 2011, Kw 1 2011, …

sep okt nov dec jan feb mrt apr

Naam Werk

Michiel Appelman

Michiel Appelman

Michiel Appelman

Michiel Appelman

Ger Brouwers, Henk Steemans, Attilla de Groot

Michiel Appelman

Michiel Appelman

Michiel Appelman

Attilla de Groot

Attilla de Groot, Michiel Appelman

Michiel Appelman


Michiel Appelman

Henk Steemans, Attilla de Groot, Michiel Appelman

Michiel Appelman

Michiel Appelman

Michiel Appelman


Attilla de Groot

Ger Brouwers

Michiel Appelman

Michiel Appelman

WBS Naam Start Finish Werk Toegewezen aan

1 Werkzaam bij AMS-IX 1 sep 18 jan 100d Michiel Appelman

2 Plan van Aanpak 1 sep 15 sep 11d Michiel Appelman

3 Goedkeuring PvA 15 sep 15 sep n.v.t. Ger Brouwers

4 IPX VPLS Instance 1 sep 15 sep 11d Attilla de Groot

5 Inhangen Accedian MetroNODE 17 sep 17 sep 2d Attilla de Groot, Michiel Appelman

6 Voorbereiden Presentatie 7 sep 20 sep 10d Michiel Appelman

7 Presenteren Planning 20 sep 20 sep n.v.t. Michiel Appelman

8 Onderzoek mogelijkheden Accedians 21 sep 29 sep 7d Michiel Appelman

9 Beslissing tot aanschaf 29 sep 29 sep n.v.t. Attilla de Groot, Henk Steemans, Michiel Appelman

10 Uitwerking van mogelijkheden 30 sep 14 okt 11d Michiel Appelman

11 Plaatsen overige Accedians 15 okt 15 okt 2d Attilla de Groot, Michiel Appelman

12 Overzichten in My AMS-IX 18 okt 22 okt 5d Michiel Appelman

13 Welke specificaties in rapportage 22 okt 22 okt n.v.t. Attilla de Groot, Michiel Appelman

14 Rapportage maken 25 okt 25 okt 1d Attilla de Groot

15 Start Trail 1 nov 1 nov 1d

16 Evalueren Accedians en Stats 2 nov 30 nov 21d Michiel Appelman

17 Grotere Implementatie Opzetten 1 dec 1 dec 1d Michiel Appelman

18 Rapport schrijven 16 sep 12 jan 85d Michiel Appelman

19 Rapport ter beoordeling 13 jan 14 jan 5d Attilla de Groot, Ger Brouwers, Henk Steemans

20 Rapport verbeteren 14 jan 21 jan 5d Michiel Appelman

21 Rapport 21 jan 21 jan n.v.t. Michiel Appelman

22 Presentatie voorbereiden 21 jan 28 jan 5d Michiel Appelman

23 Presentatie 28 jan 28 jan n.v.t. Michiel Appelman

Configuration Example Appendix C

C Configuration Example

This section gives an example of the configuration used for the Accedian EtherNIDs.1 media-selection select RJ45-A_RJ45-B2 port edit Monitor-1 state disable3 port edit Monitor-2 state disable4 port edit Client state disable5 interface edit Auto state disable6 interface delete Management7 interface add Management address 91.200.16.181 port Management netmask 255.255.255.192

gateway 91.200.16.1298 dns edit hostname nid-eqx-0159 dns edit domain noc.ams-ix.net

10 dns edit server1 91.200.16.3611 dns edit server2 91.200.16.3712 snmp edit ro-community ***********13 snmp edit rw-community ***********14 snmp edit use-host-name enable15 snmp edit system-location Equinix16 snmp edit contact-info [email protected] snmp state enable18 snmp-trap edit v2c 1 host-name zabbix.ams-ix.net host-community ********* host-state

enable19 syslog edit host interipx.stats.ams-ix.net20 syslog edit priority Notice21 ntp add time1.noc.ams-ix.net22 ntp add time-nid.noc.ams-ix.net23 ntp enable time-nid.noc.ams-ix.net24 ntp edit sync-mode highres25 cfm add domain level 1 index 9 format dns-name name noc.ams-ix.net26 cfm add ma name oam ccm-interval 1000 mepid-list 1,2,3,4,5,6,7,8 md-idx 9 index 127 cfm add mep active yes ma-idx 1 mep-id 1 port Network cci-enable yes lowest-alarm-pri

remErrXconAis28 cfm add dmm enable yes mep-idx 1 index 1 rmep-id 2 reference-period 1 interval 100 ow-

delay enable ow-max-delay 2 ow-delay-threshold 3 ow-ad-threshold 1 tw-delay enabletw-max-delay 2 tw-delay-threshold 3 tw-ad-threshold 1 ow-dv enable ow-max-dv 1

ow-dv-threshold 2 ow-adv-threshold 1 tw-dv enable tw-max-dv 1 tw-dv-threshold 2tw-adv-threshold 1

29 cfm add dmm enable yes mep-idx 1 index 2 rmep-id 3 reference-period 1 interval 100 ow-delay enable ow-max-delay 2 ow-delay-threshold 3 ow-ad-threshold 1 tw-delay enabletw-max-delay 2 tw-delay-threshold 3 tw-ad-threshold 1 ow-dv enable ow-max-dv 1








33 cfm add dmm enable yes mep-idx 1 index 6 rmep-id 7 reference-period 1 interval 100 ow-delay enable ow-max-delay 2 ow-delay-threshold 3 ow-ad-threshold 1 tw-delay enable

45

Configuration Example Appendix C

tw-max-delay 2 tw-delay-threshold 3 tw-ad-threshold 1 ow-dv enable ow-max-dv 1ow-dv-threshold 2 ow-adv-threshold 1 tw-dv enable tw-max-dv 1 tw-dv-threshold 2tw-adv-threshold 1



35 cfm add pkt-loss enable yes mep-idx 1 index 1 rmep-id 2 reference-period 1 threshold-ratio 1







42 user edit admin password

46

Graph Generating Bash Script Appendix D

D Graph Generating Bash Script

1 #!/bin/bash2 # Written by [email protected] # Working Directory for the RRD files5 DIR="/opt/mrtg/nids"6 WWWDIR="/var/www/stats"78 ## Strip HTML GET String into local variables ##9 hour=‘echo "$QUERY_STRING" | sed -n ’s/^.*hour=$[^&]*$.*$/\1/p’ | sed "s/%20/ /g"‘

10 end=‘echo "$QUERY_STRING" | sed -n ’s/^.*end=$[^&]*$.*$/\1/p’ | sed "s/%20/ /g"‘11 kpi=‘echo "$QUERY_STRING" | sed -n ’s/^.*kpi=$[^&]*$.*$/\1/p’ | sed "s/%20/ /g"‘12 nid1=‘echo "$QUERY_STRING" | sed -n ’s/^.*srcnid=$[^&]*$.*$/\1/p’ | sed "s/%20/ /g"‘13 nid2=‘echo "$QUERY_STRING" | sed -n ’s/^.*dstnid=$[^&]*$.*$/\1/p’ | sed "s/%20/ /g"‘14 # If hour variable is empty set it to 4815 if [ -z $hour ] ; then16 hour="48"17 fi18 # If end time is empty set it to now or else to starting time + end time19 if [ -z $end ] ; then20 end="now"21 else22 end="start+${end}h"23 fi2425 # Check to see if two different NIDs are selected26 if [ $nid1 == $nid2 ] ; then27 exit 128 fi2930 # Delay or jitter function31 if [ "$kpi" == "delay" ] || [ "$kpi" == "jitter" ]32 then33 # Change colors depending on KPI34 if [ "$kpi" == "delay" ] ; then35 color1="#00ff00"36 color2="#0000ff"37 sla1="500"38 sla2="1000"39 elif [ "$kpi" == "jitter" ] ; then40 color1="#00ffff"41 color2="#ff00ff"42 sla1="50"43 sla2="100"44 fi45 ## Run RRDtool an send output to web page ##46 echo "Content-type: image/png "47 echo48 rrdtool graph -X0 -l0 -Y --imginfo ’’ --title "$kpi from $nid1 to $nid2" --start now-${

hour}h --end ${end} --vertical-label microseconds - \49 DEF:oadv_$nid1\_$nid2=$DIR/5min_$kpi\_$nid1\_$nid2.rrd:ds0:AVERAGE \50 DEF:tadv_$nid1\_$nid2=$DIR/5min_$kpi\_$nid1\_$nid2.rrd:ds1:AVERAGE \51 VDEF:oadvmax_$nid1\_$nid2=oadv_$nid1\_$nid2,MAXIMUM \52 VDEF:oadvavg_$nid1\_$nid2=oadv_$nid1\_$nid2,AVERAGE \53 VDEF:oadvmin_$nid1\_$nid2=oadv_$nid1\_$nid2,MINIMUM \54 VDEF:oadv95t_$nid1\_$nid2=oadv_$nid1\_$nid2,95,PERCENT \55 COMMENT:" 95th SLA maximum minimum\l" \56 LINE1:oadv_$nid1\_${nid2}${color1}:"One Way " \57 GPRINT:oadv95t_$nid1\_$nid2:"%6.2lf Âtsec " \58 COMMENT:"$sla1 Âtsec " \

47

Graph Generating Bash Script Appendix D

59 GPRINT:oadvmax_$nid1\_$nid2:"%6.2lf Âtsec " \60 GPRINT:oadvmin_$nid1\_$nid2:"%6.2lf Âtsec\l" \61 VDEF:tadvmax_$nid1\_$nid2=tadv_$nid1\_$nid2,MAXIMUM \62 VDEF:tadvavg_$nid1\_$nid2=tadv_$nid1\_$nid2,AVERAGE \63 VDEF:tadvmin_$nid1\_$nid2=tadv_$nid1\_$nid2,MINIMUM \64 VDEF:tadv95t_$nid1\_$nid2=tadv_$nid1\_$nid2,95,PERCENT \65 LINE1:tadv_$nid1\_${nid2}${color2}:"Two Way " \66 GPRINT:tadv95t_$nid1\_$nid2:"%6.2lf Âtsec " \67 COMMENT:"$sla2 Âtsec " \68 GPRINT:tadvmax_$nid1\_$nid2:"%6.2lf Âtsec " \69 GPRINT:tadvmin_$nid1\_$nid2:"%6.2lf Âtsec\l" \70 COMMENT:"Copyright Âl’ ‘date +%Y‘ AMS-IX B.V. [updated\: ‘date "+%d-%b-%Y %H\:%M\:%S %z

"‘ ]\c" 2>/dev/null71 # Exit without errors72 exit 07374 # Frame Loss function75 elif [ "$kpi" == "frameloss" ]76 then77 ## Run RRDtool and send output to web page ##78 echo "Content-type: image/png "79 echo80 rrdtool graph -X0 -l0 -Y --imginfo ’’ --title "$kpi from $nid1 to $nid2" --start now-${

hour}h --end ${end} --vertical-label percentage - \81 DEF:fl0_$nid1\_$nid2=$DIR/5min_frameloss_$nid1\_$nid2.rrd:ds0:AVERAGE \82 CDEF:fl_$nid1\_$nid2=fl0_$nid1\_$nid2,1000000,/ \83 VDEF:flmax_$nid1\_$nid2=fl_$nid1\_$nid2,MAXIMUM \84 VDEF:flavg_$nid1\_$nid2=fl_$nid1\_$nid2,AVERAGE \85 VDEF:flmin_$nid1\_$nid2=fl_$nid1\_$nid2,MINIMUM \86 VDEF:fl95t_$nid1\_$nid2=fl_$nid1\_$nid2,95,PERCENT \87 COMMENT:" 95th SLA maximum minimum\l" \88 LINE1:fl_$nid1\_$nid2#000000:"Frame Loss " \89 GPRINT:fl95t_$nid1\_$nid2:"%3.6lf %% " \90 COMMENT:"0.005000 % " \91 GPRINT:flmax_$nid1\_$nid2:"%3.6lf %% " \92 GPRINT:flmin_$nid1\_$nid2:"%3.6lf %%\l" \93 COMMENT:"Copyright Âl’ ‘date +%Y‘ AMS-IX B.V. [updated\: ‘date "+%d-%b-%Y %H\:%M\:%S %z

"‘ ]\c" 2> /dev/null94 # Exit without errors95 exit 09697 # Invalid $kpi - exit with error98 else99 exit 1

100 fi

48

Acronyms Appendix E

E Acronyms

1DM One Way Delay Measurement

ACL Access Control List

AMS-IX Amsterdam Internet Exchange

ATM Asynchronous Transfer Mode

BGP Border Gateway Protocol

CCM Continuity Check Message

CD Collission Detection

CFM Connection Fault Management

CLI Command Line Interface

CPU Central Processing Unit

CSMA Carrier Sense Multiple Access

CTO Chief Technology Officer

DMM Delay Measurement Message

DNS Domain Name System

DWDM Dense wavelength divisionmultiplexing

EFM Ethernet in the First Mile

FCS Frame Check Sequence

FPGA Field-programmable gate array

GigE Gigabit Ethernet

GPS Global Position System

GPRS General Packet Radio Service

GSMA Global System for MobileCommunications Association

GRX GPRS Roaming eXchange

ICMP Internet Control Message Protocol

IEEE Institute of Electrical andElectronics Engineers

IP Internet Protocol

IPX IP eXchange

ISP Internet Service Provider

IT Information Technology

ITU-T International TelecommunicationUnion - TelecommunicationStandardization Sector

KPI Key Performance Indicator

LAG Link Aggregation Group

LAN Local Area Network

LI Leap Indicator

LSP Label Switched Path

LTM Link Trace Message

MA Maintenance Association

MAC Media Access Control

MD Maintenance Domain

ME Maintenance Entity

MEG Maintenance Entity Group

MEL MEG Level

MEP Maintenance End Point

MEP MEG End Point

MEPID MEP IDentifier

MHF MIP Half Function

MIP Maintenance Intermediate Point

MPLS Multiprotocol Label Switching

MRTG Multi Router Traffic Grapher

49

Acronyms Appendix E

NGN Next Generation Network

NID Network Interface Device

NIST National Institute of Standards andTechnology

NOC Network Operations Center

NTP Network Time Protocol

OAM Operations, Administration andMaintenance

OAD One Way Average Delay

OADV One Way Average Delay Variation

OID Object Identifier

OS Operating System

OSI Open System Interconnection

OWD One Way Delay

P Provider

PAA Performance Assurance Agent

PDU Protocol Data Unit

PE Provider Edge

PM Performance Measurement

PON Passive Optical Network

POP Point of Presence

PTP Precision Time Protocol

PXC Photonic Cross Connect

QoS Quality of Service

RDI Remote Defect Indicator

RFC Request for Comment

RIPE Réseaux IP Européens

RRD Round Robin Database

RTT Round Trip Time

SDH Synchronous Digital Hierarchy

SLA Service Level Agreement

SLD Service Level Description

SFP Small form-factor pluggable

SNMP Simple Network ManagementProtocol

SP Service Provider

SyncE Synchronous Ethernet

TAD Two Way Average Delay

TADV Two Way Average Delay Variation

TCP Transmission Control Protocol

TLV Type, Length, Value

TTM Test Traffic Measurement

VLAN Virtual LAN

VLL Virtual Leased Line

VM Virtual Machine

VPLS Virtual Private LAN Service

WAN Wide Area Network

50

Appendix F

F Bibliography

[1] Global System for Mobile Communications Association, “Evolution toip,” December 2010. http://www.gsmworld.com/our-work/programmes-and-initiatives/evolution_to_ip.htm.

[2] E. Rosen, A. Viswanathan, and R. Callon, “Request for Comment 3031: MultiprotocolLabel Switching,” January 2001. http://www.ietf.org/rfc/rfc3031.txt.

[3] M. Lasserre and V. Kompella, “Request for Comment 4762: Virtual Private LAN Ser-vice using Label Distribution Protocol (LDP) signaling,” January 2007. http://www.faqs.org/rfcs/rfc4762.html.

[4] Institute of Electrical and Electronics Engineers (IEEE), “IEEE 802.3 WorkingGroup Approves New Study Group for Ethernet in the First Mile (EFM),”November 2000. http://web.archive.org/web/20070217122603/http://standards.ieee.org/announcements/8023nsg.html.

[5] Institute of Electrical and Electronics Engineers, “IEEE Standard 802.3: Carrier SenseMultiple Access (CSMA) with Collission Detection (CD) (CSMA/CD) access methodand Physical Layer specifications – Section Five,” 2008. http://standards.ieee.org/getieee802/802.3.html.

[6] Institute of Electrical and Electronics Engineers, “Virtual Bridged Local Area Net-works - Amendment 5: Connectivity Fault Management,” 2007. http://www.ieee802.org/1/pages/802.1ag.html.

[7] International Telecommunication Union - Telecommunication Standardization Sector,“OAM functions and mechanisms for Ethernet based networks,” 2008. http://www.itu.int/rec/T-REC-Y.1731/en.

[8] Institute of Electrical and Electronics Engineers, “EtherType Field Public Assign-ments,” 2004. http://standards.ieee.org/regauth/ethertype/eth.txt.

[9] Brocade, NetIron Configuration Guide – Supporting Multi-Service IronWare Unified Ne-tIron R05.0.00. April 2010. page 2239.

[10] MRV, “OptiSwitch 940 - 10GE Carrier-Ethernet Access Data Sheet,” Octo-ber 2010. http://www.mrv.com/datasheets/OS/PDF300/MRV-OS-OS940_A4_HI.pdf.

[11] Brocade, NetIron Configuration Guide – Supporting Multi-Service IronWare Unified Ne-tIron R05.0.00. April 2010. page 1557.

51

http://www.gsmworld.com/our-work/programmes-and-initiatives/evolution_to_ip.htm

http://www.gsmworld.com/our-work/programmes-and-initiatives/evolution_to_ip.htm

http://www.ietf.org/rfc/rfc3031.txt

http://www.faqs.org/rfcs/rfc4762.html


http://web.archive.org/web/20070217122603/http://standards.ieee.org/announcements/8023nsg.html

http://web.archive.org/web/20070217122603/http://standards.ieee.org/announcements/8023nsg.html

http://standards.ieee.org/getieee802/802.3.html

http://standards.ieee.org/getieee802/802.3.html

http://www.ieee802.org/1/pages/802.1ag.html

http://www.ieee802.org/1/pages/802.1ag.html

http://www.itu.int/rec/T-REC-Y.1731/en

http://www.itu.int/rec/T-REC-Y.1731/en

http://standards.ieee.org/regauth/ethertype/eth.txt

http://standards.ieee.org/regauth/ethertype/eth.txt

http://www.mrv.com/datasheets/OS/PDF300/MRV-OS-OS940_A4_HI.pdf

http://www.mrv.com/datasheets/OS/PDF300/MRV-OS-OS940_A4_HI.pdf

Appendix F

[12] D. L. Mills, “Request for Comment 958: Network Time Protocol,” September 1985.http://www.faqs.org/rfcs/rfc958.html.

[13] National Institute of Standards and Technology, “IEEE 1588 - Standard for a Preci-sion Clock Synchronization Protocol for Networked Measurement and ControlSystems,” May 2010. http://ieee1588.nist.gov/.

[14] A. Ramkisoen, “Synleaf,” December 2010. http://www.synleaf.com.

52


http://ieee1588.nist.gov/

http://www.synleaf.com

Acknowledgements

Thanks to Attilla de Groot, Henk Steenman, Martin Pels, Ariën Vijn, Niels Bakker,Steven Bakker and all the other staff at AMS-IX, for providing me with the opportu-nity, knowledge and skills to successfully complete this research project.

ethernet performance measurement

Documents