router characterisation - ucl computer science - home · web view-10 april 2003-document...

MB-NG Project

Template for MB-NG reports-09 May 2023-

Document identifier: MB-NG-Doc-T0-v0.1-Template

Note that the document name is of the for MB-NG-[Document type]-[Task number]-

[version number]-[brief description]

Document types are Doc, Man, Tech (for document, management and technical)

Relevant Task Task 0

Date: 09/05/2023

Version: Version 0.1

Document status: Draft

Description Discription of the document.

Editors: Editors name

Change Record:

TABLE OF CONTENTS

1 INTRODUCTION............................................................................................3

2 CONTEXT/BACKGROUND...........................................................................3

2.1 MB-NG project......................................................................................................3

2.2 QoS..........................................................................................................................3

3 OBJECTIVES.................................................................................................3

4 TEST SETUP.................................................................................................4

4.1 Metrics and procedure..........................................................................................54.1.1 Maximum throughput without loss..................................................................54.1.2 “Blast test”.......................................................................................................64.1.3 Minimum latency.............................................................................................6

5 TESTS ON CISCO 12016 GSR.....................................................................6

5.1 Results between Engine 2 and Engine 2+............................................................7

5.2 Results between Engine 2 and Engine 3.............................................................105.2.1 Maximum throughput without loss................................................................105.2.2 The “blast test”...............................................................................................13

5.3 Conclusions...........................................................................................................17

5.4 Future Measurement...........................................................................................17

6 APPENDIX...................................................................................................18

6.1 Theoretical rate of the OC48 POS link..............................................................18

7 References...................................................................................................19

MB-NG-Doc-T0-v0.1-Template 2/20

1 IntroductionIn this report, we present the results of characterising a Cisco 12016 GSR router and a Cisco 7609 OSR router. The aim is to provide the standalone performance measurements of these devices such that when they are connected in a wider network, the composite behaviour can be understood. This work has been supported by Spirent1 and by Cisco2.

2 Context/background

2.1 MB-NG projectThe Managed Bandwidth Next Generation (MB-NG3) is a UK based “e-science” project. The aims are firstly to demonstrate end-to-end managed bandwidth services in a multi-domain environment, in the context of Grid project requirements. Secondly, to investigate and develop high performance data transport mechanisms for Grid data transfer across heterogeneous networks.

2.2 QoSThe deployment of QoS—the separation and unequal treatment of different traffic flows based on the application requirements and the agreement between different administrative domains—is a significant part of this project.For the deployment of QoS and defining sensible Service Level Specification (SLS) and Service Level Agreement (SLA), it is important that the network behaviour is quantified and understood.

3 ObjectivesFor the tests presented here, our objectives were to obtain the performance limits of the routers in standalone mode. We are interested in the maximum packet forwarding rate, the maximum throughput and the latency, all as a function of the packet size. An indication of the queue lengths is also useful.

Figure 1 shows how each of the site network within the MB-NG network is connected to the core network. Each site network has a Cisco 7609 which connects to the core via a 2.5 Gbit/s POS interface to a Cisco 12016 GSR. On the other side of the 7609 there are Gigabit Ethernet connections to other site devices. The GSR in the core has an Engine 3 line card connecting to the site’s 7609 and two Engine 2 line cards connecting to the other core devices.

Thus for the 12016 GSR, we are interested in the performance between Engine 3 and Engine 2 line cards and the performance between Engine 2 OC48 POS line cards. For the

1 http://www.spirentcommunications.com/2 http://www.cisco.com3 http://www.mb-ng.net


7609, we are interested in the performance between Gigabit Ethernet and the OC48 POS interfaces.

Figure 1. The MB-NG site network connection to the core.

4 Test setupThe setup of the test is shown in Figure 2. This shows the Cisco router device under test and the Adtech AX4000 tester from Spirent. For the Cisco 12016, both input and output interfaces were 2.5Gbit/s POS, while in the case of the 7609, we used Gigabit Ethernet as the input interfaces and 2.5Gbit/s POS as the output interface. This is consistent with the setup of the MB-NG network. There were up to two input interface and a single output interface in the case of the GSR. While for the 7609, we required three Gigabit input interfaces to congest the 2.5 Gbit/s POS output interface.

A small but significant point is that to ensure that the router did table lookups rather than simply forwarding when the IP address is in the same subnet. To do this, the packets transmitted from the Adtech AX4000 were configured to have a different IP address compared to its interface. That is, no directly connected IP addresses.

The Adtech AX4000 is able to achieve line rate for all packet sizes and is able to provide accurate delay measurements to a resolution of 40ns.


Cisco 7609 OSRCisco 12016 GSR

Engine2

Engine2

Engine3

Core Network Site Network

GigabitEthernet

2.5Gbit/sOC48 POS

2.5Gbit/sOC48 POS

Figure 2. The setup for characterising a router

There were two generic types of experiments we wanted to perform. The first set of experiments was aimed at characterising the baseline port-to-port performance of the router. That is without any applied QoS features or policies. The second set of experiments was performed with QoS features enabled and policies applied.

In both cases mentioned above, two situations were looked at: maximum rate without packet loss “blast test” at maximum rate allowing for packet loss the minimum latency with no load.

The results enabled us to assess if QoS was working properly and whether enabling QoS had any detrimental effects on the router performance.

4.1 Metrics and procedureWhat follows is a description of the procedure used to characterise the baseline port-to-port performance of the router.

4.1.1 Maximum throughput without lossThe first test is the maximum throughput through the device without loss for a fixed packet size. For this, we use the Adtech AX4000’s RFC2544 test suite. The idea is that a binary search is performed to find the maximum throughput at which no packet loss is observed. Thus for a given measurement, the packet rate is reduced if losses are observed, if not then it is increased. The amount by which rate is reduced or increased is halved at each step, starting from 50% of the maximum load. From this we measure the throughput, the packet rate and the router port-to-port latency at that packet size. The setting for the AX4000’s RFC2544 test suite we used were:

The wait time = 2 seconds. This is the extra time at the end of the test to allow for the packets still in the system to be forwarded.

Load = 100%. This is the initial load for the tests. Duration = 12 seconds. This is the length of each trial.


Cisco router Router

AX4000

OutputInterface

IntputInterface

Tolerance = 10%. The maximum difference in rate between the rate without loss and the next rate with loss before ending the test.

Loss tolerance = 0%. The allowed packet loss per flow. Interval = 6 seconds. For the delay measurements, this was the number of seconds

to wait before sending the test frame. Repetition = 5. For the delay measurements, this was the number of time the test

was performed. The reported value is the average.

4.1.2 “Blast test”The second test involves transmitting packets at the maximum rate (100% of the load) and measuring the throughput, frame rate, latency and also the packet loss. The setting for the AX4000’s RFC2544 test suite we used were:

The wait time = 2 seconds. This is the extra time at the end of the test to allow for the packets still in the system to be forwarded.

Load = 100%. Kept at this value. Duration = 12 seconds. This is the length of each trial. Tolerance = 0%. Not applicable. Loss tolerance = 100%. The allowed packet loss per flow. Interval = 6 seconds. For the delay measurements, this was the number of seconds

to wait before sending the test frame. Repetition = 5. For the delay measurements, this was the number of time the test

was performed. The reported value is the average.

Comparing the first and the second tests will tell us whether the router’s performance deteriorates in the transition area just below and at its maximum load.

4.1.3 Minimum latencyA third test measuring the port-to-port latency without any other traffic present in the system. This should give us the minimum latency through the router.

In the GSR tests that follow, we performed test in Section 4.1.1 and Section 4.1.2.

5 Tests on Cisco 12016 GSR

The Cisco 12016 GSR (R5000 CPU at 200Mhz, 512KB L2 Cache) came with 256 MBytes of memory was running the Cisco IOS version 12.0(23)S. It was equipped with an Engine 3 POS interface card and two Engine 2 POS interface cards. The first Engine 2 card (R5000 CPU at 200Mhz) had 128 MBytes of memory and the second Engine 2 card (R5000 CPU at 200Mhz) had 256 Mbytes of memory. We henceforth refer to the latter as Engine 2+ due to it larger memory size. For reference, the Engine 3 card (R7000 CPU at 400Mhz) was equipped with 256 MBytes of memory.


In the graphs following on this section, it is shown a “Theoretical value” for different metrics. The way this “Theoretical value” has been calculated is explained in the Appendix.

5.1 Results between Engine 2 and Engine 2+The test setup of the test is shown in Figure 3. In our first experiment, we wanted to find out if there were any distinctions in the direction of the traffic flow given the differences between the input and output interface cards. Figure 4 to Figure 7 show the results of a single flow between Engine 2 and Engine 2+ interface cards.

Figure 4 shows maximum achieved throughput without loss, Figure 5 shows the corresponding packet rate. Figure 6 and Figure 7 shows the port to port latency for the “blast test” described in Section 4.1.2. (Figure 7 being a magnification of Figure 6).

The results are identical, that is, the performance of the router is the same whether the traffic flow is from Engine 2 to Engine 2+ or vice versa. At above 83 bytes, a peak throughput of 2.35 Gbit/s is achieved. The maximum achievable packet rate is 3.54 million packets/s.

Notable are the troughs in the throughput measurements at packet sizes of 98, 148 and 203 bytes. We do not know what causes these performance dips.


E2

E2+

GSRAX4000

Figure 3 Test setup to check E2 and E2+ performance

0.00E+00

5.00E+08

1.00E+09

1.50E+09

2.00E+09

2.50E+09

0 100 200 300 400 500

Packet size. Bytes

Thro

ughp

ut. b

it/s

Theoretical

E2 to E2+

E2+ to E2

Figure 4.The GSR's lossless throughput between Engine 2 and Engine 2+ POS interfaces.

0.00E+00

5.00E+05

1.00E+06

1.50E+06

2.00E+06

2.50E+06

3.00E+06

3.50E+06

4.00E+06

0 100 200 300 400 500

Packet size. Bytes

Pack

et ra

te. P

acke

ts/s

Theoretical

E2 to E2+

E2+ to E2

Figure 5. The GSR's lossless packet rate between Engine 2 and Engine 2+ POS interfaces.


0

1000

2000

3000

4000

5000

6000

0 50 100 150 200 250 300 350 400 450 500

Packet size. Bytes

Ave

rage

end

-to-e

nd la

tenc

y. u

s

E2 to E2+E2+ to E2

Figure 6. The GSR's latency between Engine 2 and Engine 2+ POS interfaces.

Figure 7. The GSR's latency between Engine 2 and Engine 2+ POS interfaces. Magnified.


28

29

30

31

32

33

34

35

36

37

38

0 50 100 150 200 250 300 350 400 450 500

Packet size. Bytes

Ave

rage

end

-to-

end

late

ncy.

us

E2 to E2+

E2+ to E2

The Latency plot of Figure 6 also shows increased latencies at those same packet sizes and also below 83 byte packets. From Figure 7, we see that the latencies normally at 30 microseconds (over two orders of magnitude less than the high values), increasing with packet size up to 35 microseconds for the shown range (500 bytes). The gradient of the latency lines (ignoring the high values) indicated by the dotted line in Figure 7 is around 89 Mbytes/s or 712 Mbit/s. We interpret this as the bandwidth used to carry a packet from the input port of the router to the output port. This suggests that some degree of parallel transfers (from the input to output port) are used to achieve the peak throughput of 2.35 Gbit/s.

The intercept of the dotted line in Figure 7 with the y-axis is 28.5. We interpret this as the fixed processing delay for a packet. This is typically due operations such as the packet destination address to router output port lookup. Hence the equation for the dotted line is:

5.2 Results between Engine 2 and Engine 3

In this section, we assess the symmetry of the performance when a single flow is sent between Engine 2 and Engine 3 interface cards. Figure 8 shows the setup used.

5.2.1 Maximum throughput without lossFigure 9 shows the maximum achieved throughput without loss from Engine 2 to Engine 3 and vice versa. When compared with the performance between Engine 2 and Engine 2+, we note that Engine 3 to Engine 2 has identical performance from packet sizes above 83 bytes, including the troughs at 98, 148 and 203 byte packet size. Below 83 byte packet size, the performance between Engine 2 and Engine 2+ is identical to that of Engine 2 to Engine 3. In the latter case, the troughs are shifted by a few bytes at 93, 138 and 188 bytes.


E3

E2 or E2+GSR

AX4000

Figure 8 : Testbed between E2/E2+ and E3 cards

0.00E+00

5.00E+08

1.00E+09

1.50E+09

2.00E+09

2.50E+09

0 100 200 300 400 500

Packet size. Bytes

Thro

ughp

ut. b

it/s

Theoretical

E2 to E2+

E2 to E3

E3 to E2

Figure 9 GSR's Maximum throughput without loss for various card combinations

From Figure 10, also obtained from the maximum throughput without loss, we note that Engine 3 to Engine 2 achieves the highest packet rate (3.82 million packets/s) and Engine 2 to Engine 3 is the combination with the slowest performance. This suggests a higher forwarding rate from the Engine 3 card to the router’s switching fabric and also same type cards perform better than mixing different cards. It would be interesting to compare the performance between Engine 3 cards.


0.00E+00

5.00E+05

1.00E+06

1.50E+06

2.00E+06

2.50E+06

3.00E+06

3.50E+06

4.00E+06

0 100 200 300 400 500

Packet size. Bytes

Pac

ket r

ate.

Pac

kets

/s

Theoretical

E2 to E2+

E2 to E3

E3 to E2

Figure 10 The GSR's maximum packet forwarding rate for various card combinations

Figure 11 shows the latency for the maximum throughput without loss for the Engine 2 and Engine 3 combination taken to the maximum MTU of 4470 bytes. This measurement was done as described in Section 4.1.1. The Engine 3 to Engine 2 line shows a different behaviour for packet sizes below 500 bytes compared to that above 500 bytes. Furthermore, the Engine 3 to Engine 2 has a more jitter than Engine 2 to Engine 3. We do not know the reasons why. For packet sizes above 500 bytes, the equation for Engine 3 to Engine 2 is:

E3 to E2: Equation 1

and the equation for Engine 2 to Engine 3 is:


For packet sizes below 500 bytes, the latency equation is the same as we observe in the blast tests following.


30

40

50

60

70

80

90

0 500 1000 1500 2000 2500 3000 3500 4000 4500

Packet size. Bytes

Ave

rage

end

-to-e

nd la

tenc

y. u

s

E2 to E3 Without congestion

E3 to E2 Without congestion

Figure 11. The GSR's end-to-end latency in a congested state between Engine 2 and Engine 3. Without congestion.

5.2.2 The “blast test”The results shown in Figure 12 to Figure 15 were obtained by the “blast test” as described in Section 4.1.2.

Figure 12 shows the throughput obtained from the “blast test” and Figure 13 shows the corresponding packet rate. These graphs show great similarity to the case of the maximum throughput without loss shown in Figure 9 and Figure 10. The troughs seen earlier are also present here at the same packet sizes. The only difference is negligibly higher rates for the blast test. This can be accounted for by the “Tolerance” setting of 10% described in Section 4.1.1. The conclusion to draw from this is that the router’s forwarding performance does not deteriorate when it is oversubscribed.

The packet loss rate for the “blast test” is shown in Figure 14. The observed packet loss rate is consistent with the achieved rate and the theoretical limit.


0.00E+00

5.00E+08

1.00E+09

1.50E+09

2.00E+09

2.50E+09

0 50 100 150 200 250 300 350 400 450 500

Pck Size (Bytes)

Thro

ughp

ut (b

ps)

E2 to E3E3 to E2

E2 to E2+

Figure 12. The GSR's Maximum throughput with loss (blast test) for various card combinations

0.00E+00

5.00E+05

1.00E+06

1.50E+06

2.00E+06

2.50E+06

3.00E+06

3.50E+06

4.00E+06

4.50E+06

0 50 100 150 200 250 300 350 400 450 500

Pck Size (Bytes)

Pac

ket R

ate

(Pkt

s/s)

E2 to E3

E3 to E2

E2 to E2+

Figure 13. GSR's Maximum packet rate with loss (blast test) for various card combinations.


0.00E+00

5.00E+05

1.00E+06

1.50E+06

2.00E+06

2.50E+06

3.00E+06

0 50 100 150 200 250 300 350 400 450 500

Pck Size (Bytes)

Pac

ket L

oss

Rat

e (P

kts/

s)

E2 to E3

E3 to E2

E2 to E2+

Figure 14. The GSR’s packet-loss-rate from the blast test for various card combinations.

Figure 15 shows that the highest latencies of the Engine 2 and Engine 3 combination is an order of magnitude higher than the Engine 2 to Engine 2+ combination. A magnification of Figure 15 showing the lower latencies is shown in Figure 16. Here we see that the E2/E2+ combination also has the smallest minimum latency. Engine 3 to Engine 2 has at least an extra four microseconds and Engine 2 to Engine 3 has an extra seven microseconds.


0.00E+00

2.00E+04

4.00E+04

6.00E+04

8.00E+04

1.00E+05

1.20E+05

1.40E+05

1.60E+05

1.80E+05

2.00E+05

0 100 200 300 400 500

Packet size. Bytes

Ave

rage

end

-to-e

nd la

tenc

y. u

s

E2 to E3

E3 to E2

E2 to E2+

Figure 15 The GSR's end-to-end latency in a congested state (blast test) for various card combinations.

Figure 16. The GSR's end-to-end latency in a congested state (blast test) for various card combinations. Magnified from Figure 15.

The equations for the dotted lines illustrated in Figure 16 are:


28

30

32

34

36

38

40

42

0 100 200 300 400 500

Packet size. Bytes

Ave

rage

end

-to-e

nd la

tenc

y. u

s

E2 to E3

E3 to E2

E2 to E2+



E2 to and from E2+: Equation 5

5.3 ConclusionsWe have performed baseline tests on the GSR. The results are summarised in Table 1. This table shows the peak throughput, peak packet rate and the minimum packet size at which the peak throughput is achieved. The table also shows the latency variables for an equation of the form:

Equation 6

From this table we can predict the minimum latency and maximum throughput through the router for most packet sizes.

Peak throughput.Gbit/s

Peak packet rate. Millions of packets/s

Minimum packet size for peak throughput/packet rate. Bytes

Latency variables. Below 500 bytes

Latency variables. Above 500 bytes

c(u s) m(us/byte)

c(us) m(us/byte)

E2 & E2+

2.36 3.54 83 28.5 1/89 NA NA

E2 to E3

2.36 3.54 83 35.5 1/117 35.0 1/91

E3 to E2

2.36 3.82 83 33.0 1/98 38.5 1/91

Table 1. Summary of the GSR 12016 performance.

5.4 Future MeasurementFor the future, we would like to perform the test described in Section 4.1.3. That is, the latency with minimum load.The GSR architecture means that we cannot easily saturate the output ports by combining the traffic from multiple input ports. The crossbar shares the bandwidth to each output port equally between the input ports such that the combined input rate equals the output rate of the output port. At a later date, we will try various scenarios, which we can use to


build queues within the router in order to implement differentiated treatment of traffic, i.e. QoS. We will also test the ability of the GSR to police and re-classify traffic.


6 Appendix

6.1 Theoretical rate of the OC48 POS link

Figure 17 Shows the format of the SONET frame. It has 90 columns by 9 rows. Each square represents a byte of data. For an STS-1 (OC-1 optical), a frame is output every 125 us or 8000 frames/s. Higher data rates are multiple of the STS-1 rates.

From Figure 17, the actual available user data is 86*9 bytes. Thus the user data rate is 86*9*8000 = 6.192 Mbytes/s or 49.536 Mbit/s.

For our experiments, we are using OC-48 links, so the data rate available to the user is 49.536*48= 2377.728Mbit/s.

Figure 17. The SONET frame

Figure 18 shows the PPP frame format. This is how the user data is encapsulated to go over the OC48 link.


86 Columns user data

3 Columns for overhead

Path overhead

Sonet frame. 125 us.9 rows

. . .

Figure 18. The PPP frame format.

The theoretical throughput (referring to the PPP frame format of Figure 18) is calculated as follows:(Data+IP overhead+PPP header + PPP trailer)*(link rate)/ (data+IP overhead+PPP header + PPP trailer+Flag+Flag)

The Data is the variable user dataThe IP overhead is 20 bytesThe PPP header is 4 bytesThe PPP trailer is 4 bytesThe link rate is 2377.728Mbit/sThe Flag is 1 byte (Note that there are two of them).

7 References

[ ] “MB-NG technical goals: (experiments) for QoS and Managed bandwidth” Document ID MB-NG-Doc-T1-1.3-TechnicalGoals http://www.mb-ng.net

[ ] Spirent Communications 2001 “RFC 2544 Performance Test Suite User’s Guide” Adtech Division Software Part Number 200104, Version 1.0 http://www.spirentcom.com/search/literature.cfm?dprimary=2&dtid=4

[ ] S. Bradner, J. McQuaid March 1999 “RFC2544 Benchmarking Methodology for Network Interconnect Devices” http://www.faqs.org/rfcs/rfc2544.html

[ ] S. Bradner July 1991 “RFC1242. Benchmarking Terminology for Network Interconnection Devices” http://www.faqs.org/rfcs/rfc1242.html


Flag1 byte

Flag1 byte

PPP Header4 byte

IP Header20 byte

Dataup to 4472 byte

PPP Trailer4 byte

router characterisation - ucl computer science - home · web view-10 april 2003-document...

Documents