QoS Testing between CNAF and Pisa

Test results

Nov 29 2005

Outline

• Objectives

• QoS scenarios

• Best-effort TCP performance results

• Test on differentiation of outgoing traffic from INFN-Pisa to CNAF

• Best-effort UDP performance results

• Future work

Objectives

• Test traffic differentiation techniques on the customer edge routers of the INFN local area networks, for allocation of a minimum guaranteed bandwidth to a number of traffic classes– Traffic classification and marking– Scheduling: Weighted Round Robin (WRR)– No policing (for the moment)

Testbed configuration

• CNAF:– Juniper M10 (dedicated to testing)– GigaEthernet switch Extreme Summit 400– Two end-nodes (64 bit PCI slot network interface, 1 GEthernet),

connected to the Service Challenge GigaEthernet switch– Capacity to/from GARR: 2 Gbit (boundling of two GEthernet

interfaces)

• PISA– Juniper M7 (production router)– Two end-node (64 bit PCI-X slot network interface, 1 GEthernet;

1 Fast-Ethernet interface )– Capacity to/from GARR: 1 Gbit

Traffic Classes1. User traffic:

• Bandwidth range: [300, 1000] Mb/s• Minimum 30% of link capacity guaranteed in case of congestion• TOS Preference codepoint: 000 (best-effort)

2. Service Challenge traffic:• Bandwidth range: [700, 1000] Mb/s• Minimum 70% of link capacity guaranteed in case of congestion• TOS Preference codepoit: 001 (assured-rate)

• Purposes of traffic differentiation:– Allocation of minimum guaranteed bandwidth to input/output legacy

and Service Challenge traffic classes in case of congestion– Fair distribution of link capacity in case of congestion– Possibility to get more bandwidth than the minimum guaranteed in

case of spare link capacity

Scenario 1: differentiation of outgoing traffic (from Pisa)

GARR

GARR

CNAF INFN Pisa

70%

30%

Juniper M10 Juniper M7

1 Gb/s2.0 Gb/sService Challenge

Service Challenge

bottleneck

Users

Scenario 2: differentiation of incoming traffic (to Pisa)

GARR

GARR

CNAF INFN Pisa

70%

30%

Juniper M10 Juniper M7

1 Gb/s2.0 Gb/sService Challenge

Service Challenge

bottleneck

Users

Scenario 3: differentiation of outgoing traffic (from CNAF)

GARR GARR

CNAF

PisaJuniper M10

Torino

Legnaro

MilanoBari

2.0 Gb/s

20%

20%

20%

20%

20%

bottleneck

Scenario 4: differentiation of incoming traffic (to CNAF)

GARR

GARRCNAF

PisaJuniper M10

Torino

Legnaro

Milano

Bari

2.0 Gb/s

1Gb/s

1Gb/s

1Gb/s

100Mb/s1Gb/s

bottleneck

Best-effort TCP performance

• Initially asymmetric throughput to/from Pisa (probably due to misconfiguration in the PISA MAN), now solved

• Performance (with network configuration fixed):

Best-effort TCP performance:test CNAF INFN Pisa

Test TCP CNAF --> Pisa

860870

880890

900910

920930

940

1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97 103 109 115

n° istances

Th

rou

gp

ut

(Mb

/s)

Test TCP Pisa --> CNAF

850860870880890900910920930940

1 9 17 25 33 41 49 57 65 73 81 89 97 105 113

n° istances

Th

rou

gh

pu

t (M

b/s

)

Differentiation of outgoing traffic (from Pisa)

GARR

GARR

CNAF INFN Pisa

SC1

70%

Juniper M10 Juniper M7

1 Gb/s2.0 Gb/s

Service Challenge

bottleneck

Users

30%SC2

BE1

BE2

BE2

BE1

AR

AR

Test on differentiation of outgoing traffic (1/2)

Differentiation of outgoing traffic in router M7 (Pisa)

0%

20%

40%

60%

80%

100%

1 2 3 4

n° test

thro

ug

hp

ut

%

Traffico AR

Traffico BE2

Traffico BE1

N° test

Sender(command - line) [Mb/s]

Riceiver[Mb/s]

ProtocolDroppedPackets

Percentuage

1BE1 = 90

BE2 = 250AR = 750

BE1 = 72BE2 = 198AR = 670

UDPUDPUDP

√BE1 = 8

BE2 = 21AR = 71

2BE1 = 90

BE2 = 280AR = 700

BE1 = 67BE2 = 208AR = 663

UDPUDPUDP

√BE1 = 7

BE2 = 22AR = 71

3BE1AR

TCPTCP × BE1 = 10,5

AR = 89,5

4BE1 =

BE2 = 280AR =

BE1 = 68,5BE2 = 280AR = 565

TCPUDPTCP

×BE1 = 7,5

BE2 = 30,7AR =61,8

Differentiation of outgoing traffic on router M7 (Pisa)

0%

20%

40%

60%

80%

100%

5 6 7 8 9

n° test

Thro

ughp

ut % Traffico AR

Traffico BE2

Traffico BE1

N° test

Sender(command

- line) [Mb/s]

Riceiver[Mb/s]

ProtocolDroppedPackets

Percentuage

5BE1 = 90

BE2 =AR = 750

BE1 = 90,5BE2 = 47AR = 783

UDPTCPUDP

×BE1 = 9,9BE2 = 5,1AR = 85

6BE1 = 90

BE2 = 250AR =

BE1 = 90,4BE2 = 250AR = 583

UDPUDPTCP

×BE1 = 10,6BE2 = 27AR = 62,4

7BE1 = BE2 =

AR = 750

BE1 = 76,3BE2 = 68AR = 784

TCPTCPUDP

×BE1 = 9,7BE2 = 7,2AR = 83,1

8BE1 = 80

BE2 = 250AR =

BE1 = 80BE2 = 250AR = 600

UDP UDPTCP (*20)

×BE1 = 8,4 BE2 = 26,9AR = 64,5

9BE1 = 80

BE2 = 250AR =

BE1 = 80BE2 = 250AR = 600

UDP UDPTCP

(*100)

×BE1 = 8,4 BE2 = 26,9AR = 64,5

Test on differentiation of outgoing traffic (2/2)

Test Marker

Traffic Tos binary Tos hexadecimal

Best-effort 000 (0x00)

Assured-rate 001 0x20

[datatag1test] /home/bencivenni > tcpdump -c 20 host 212.189.152.177 -i eth0

• tcpdump: listening on eth0• 11:18:33.254874 192.135.23.200.20000 > qos1.pi.infn.it.33233: .

ack 2468187583 win 18432 (DF)• 11:18:33.254878 qos1.pi.infn.it.33233 > 192.135.23.200.20000: .

1:1461(1460) ack 0 win 22 (DF) [tos 0x20]

Best-effort UDP performance (1/3)

• Performance issues in the LAN CNAF, probably due to the NIC hardware used

Test LAN CNAF TCP/UDP

0

200

400

600

800

1000

1 13 25 37 49 61 73 85 97 109

n° istances

Th

rou

gh

pu

t (M

b/s

)

UDP b=800 Mb/s

TCP

Best-effort UDP performance 1/3:traffic profile

• Purpose of test: verify the inpact of production traffic on the UDP constant bit rate traffic profile generated at the source (constant inter-packet gap)

• Production traffic mixing with test traffic on most of the end-to-end path under test

Best-effort UDP performance 3/4

Test UDP INFN Pisa --> CNAFbandwidth = 800 Mb/s

0

200

400

600

800

1000

1 17 33 49 65 81 97 113

n° istances

Th

rou

gh

pu

t (M

b/s

)

HOST PISA (Sender)

HOST CNAF (Riceiver)

Differences between CNAF and INFN Pisa

PISA

• Operating System:

Linux 2.4.21-32.0.1.ELsmp

• iperf version 1.7.0 • MTU = 1500 Byte• BUS PCI

CNAF

• Operating System:

Linux 2.4.21- 20.EL.cernsmp

• iperf version 2.0.1• MTU = 5000 Byte• BUS PCI-X

Xeon

GARR GARR

Xeon

opteron

opteron

datatag1test

test7200a

nettetst1

nettest2

192.135.23.200

Eth0 1Gb/s 64 bit

192.135.23.201

Eth0 1Gb/s 64 bit

192.135.23.202

Eth1 1Gb/s 64bit

192.168.1.11

Eth1 1Gb/s 32 bit

192.168.1.14

Eth11Gb/s 32 bit

Eth2 10Gb/s 64 bit/133 Mhz

192.168.1.101

192.168.1.102

Eth2 10Gb/s 64 bit/133 Mhz

Switch Extreme Summit 400-48pt

Jumbo frames 9216

Switch Extreme Summit 400-48pt

Jumbo frames 9216

RouterJuniper M10

131.154.99.32

131.154.99.201

131.154.100.1

131.154.100.2

Test SC

2 Gb/s Balanced Boundling

SC

Eth2

Eth2

Eth0

Eth0

Athlon

RouterJuniper M7

INFNPisa

212.189.152.177 Eth0

1gb/s 64 bit/133 Mhz

1 Gb/s

Iperf: Command line option

• -l (iperf_len): The length of buffers to read or write. Iperf works by writing an array of len bytes a number of times. Default is 8 KB for TCP, 1470 bytes for UDP. Note for UDP, this is the datagram size and needs to be lowered when using IPv6 addressing to 1450 or less to avoid fragmentation.

UDP Test : Pisa CNAFSender

(command line) (Mb/s)

-l

(Byte)

Sender

(real) (Mb/s)

Riceiver

(Mb/s)

-l

(Byte)Packet loss

800 × 840470-819

× 20%

900 × 905422-800

× 16%

800 1500 800 800 1500 0%

900 1690 901 900 1690 0%*

900 1690 901 900 2000 0%

UDP Test : CNAF Pisa

Sender

(command line) (Mb/s)

-l

(Byte)

Sender

(real) (Mb/s)

Riceiver

(Mb/s)

-l

(Byte)Packet loss

800 ×463-805

449-788

× 2,2%

900 ×500-785

482-780

× 2,2%

800 1500 800 × 1500 ×

900 1690 901 × 1690 ×

900 1690 901 × 2000 ×

Sender

(command line) (Mb/s)

-l

(Byte)

Sender

(real) (Mb/s)

Riceiver

(Mb/s)

-l

(Byte)Packet loss

800 ×466-696

460-648

× 0,2%

900 ×509-739

503-718

× 16%

800 1500 800 670 1500 14%

900 1690 901 702 1690 22%

900 1690 901 702 2000 22%

UDP Test : LAN CNAF

Router Configuration: Class of Service (1/2)

class-of-service { classifiers { # Definisco traffico best-effort tutto ciò inet-precedence inet-precedence-service-challenge{ che entra con un TOS = 000 # forwarding-class best-effort { loss-priority high code-points 000; } } forwarding-classes { # Definisco le 2 code # queue 0 best-effort; queue 2 ip-premium; } interfaces { ge-y/y/y { scheduler-map service-challenge-ar-be; unit 0 { rewrite-rules { # All’interfaccia d’uscita marco i pacchetti del inet-precedence ar-mark; traffico assured-rate con il TOS opportuno } che verrà poi specificato nelle rewrite-rules # } ............ # Associo alle interfacce fastethrnet il traffico best-effort #

Router Configuration: Class of Service (2/2)

rewrite-rules { inet-precedence ar-mark { forwarding-class assured-rate { loss-priority low code-point 001; # TOS = 001 → Assured forwarding # } } } scheduler-maps { service-challenge-ar-be { forwarding-class assured-rate scheduler sch-assured-rate; forwarding-class best-effort scheduler sch-best-effort; } } schedulers { # Servo per il 70% del tempo la coda del sch-assured-rate traffico assurde-srate e per il 30% quella transmit-rate percent 70; del traffico best-effort # buffer-size percent 70; priority high; } sch-best-effort { transmit-rate percent 30; buffer-size percent 30; priority low;

Router Configuration: Firewall

firewall { filter assured-rate-traffic { #Etichetto come assured-rate- term term-192.135.23.200{ traffic il traffico generato da from { 212.189.152.177 e destinato source-address { a 192.135.23.200 # 212.189.152.177/32; } destination-address { 192.135.23.200/32; } protocol [ udp tcp ]; } then { accept; forwarding-class assured-rate; } } term term-192.135.23.201{ #Etichetto come best-effort- from { traffic il traffico generato da source-address { 212.189.152.177 e destinato

212.189.152.177/32; a 192.135.23.201 # } destination-address { 192.135.23.201/32; } protocol [ udp tcp ];then { accept; forwarding-class best-effort; } }

term default-action { # Tutto ciò che non è stato specificato

then { precedentemente viene accettato e accept; classificato come traffico best forwarding-class best-effort; effort # } } } }