1 nznog 2006 – victoria university wellington, march 23 rd, 2006 10 gigabit networking at...

1NZNOG 2006 – Victoria University Wellington, March 23rd, 2006

10 Gigabit Networking at TelstraClearJörg Micheel


Outline

• Some Metro Ethernet equipment in use at TelstraClear• Test equipment, testing process• Project: University of Auckland 10 Gigabit Campus Ring• Measurement results and performance figures• Outlook: REANNZ Advanced Network implementation


Extreme Networks Black Diamond – metro core switch

• Black Diamond 10808 (BD10K)• 22 rack mount units• 1280 Gbps capacity• Up to 48 10-Gigabit ports• Up to 480 10/100/1000 ports• Powerful VLAN, Virtual router Layer2

and Layer3 capabilities• Proprietary EAPS link-protection

protocol provides continuity in case of fiber cut

• L2/L3 Quality-of-Service• L2/L3 hardware filtering and priority• Jumbo frames at 9212• High availability, hardware

redundancy


Extreme Networks Summit X450 – edge switch

• X450-24t with 24 ports 10/100/1000 copper, four combined SFP GigE ports

• X450-24x with 24 ports 1-GigE SFP, four combined 10/100/1000 ports

• Optional dual 10-Gigabit Ethernet uplinks• 1 RU form factor• 160 Gigabits-per-second capacity• 65 million packets-per-second forwarding performance• Coming: X450a with stacking capability• Other features similar to Black Diamond series


Network test equipment – EXFO PacketBlazer

• 8510 – wire speed full duplex 10/100/1000

• 8510G – wire speed full duplex 10G LAN/WAN PHY

• BERT testing at Layer 1• Frame testing at Layer 2• RFC 2544 Performance

test suite for network devices

• Throughput, back-to-back packets, frame loss and latency, jitter

• Detailed QoS analysis• Can extend concept of

device testing to (parts of) a network


University of Auckland 10 Gigabit campus ring (1)

Requirements:• Resilient infrastructure to interconnect eight University of

Auckland campus sites• Provide multiple 1-Gigabit services at Layer 2 between

campuses

Solution:• Dual-implementation with Summit X450 at six sites within

Auckland metropolitan domain• Dark fiber between the sites• Supports multiple unconstraint 1-Gigabit VLAN services with

QoS up to the maximum 10 Gigabit backbone capacity• Resiliency via Ethernet Advanced Protection Switching (EAPS)• SLA: 99.99% end-to-end throughput• SLA: 500 milliseconds maximum round-trip-time• SLA: 250 milliseconds maximum packet delay variation


University of Auckland 10 Gigabit campus ring (2)

ITSS

Symonds St

Newton Data

Medicine

Epsom

Tamaki


University of Auckland – test setup

ITSS

Symonds St

Newton Data

Medicine

Epsom

Tamaki

12

V

LAN

S

12 VLANS

12

VLANS

12 VLANS

12

VLANS

12

V

LAN

S


University of Auckland – throughput results

• 10 Gigabit backbone performance: full line rate for any packet sizes between 128 and 9212 bytes

• Any single 1-Gigabit loop VLAN: loss free at any packet size

• 10 loop 1-Gigabit VLAN performance (saturated core, with all services active):

100% 99% 98% 97% 96%64 8.12 6.10 4.07 2.02 0.00

128 4.71 3.54 1.16 0.00 0.00256 2.54 0.60 0.00 0.00 0.00512 1.31 0.00 0.00 0.00 0.00

1024 0.66 0.00 0.00 0.00 0.001280 0.51 0.00 0.00 0.00 0.001518 0.42 0.00 0.00 0.00 0.00

Packet size in bytes

Link utilization

Percent loss


University of Auckland – delay and jitter results

• Fiber run length 44.7 km – 218.7 microseconds (RI=1.467)

• OTD(min) = 258.4 microseconds• Switching, buffering accounts for 39.6 microseconds (15%

of total)• Per switch latency 3.3 microseconds, inclusive

serialization

• RTTD(64) 517.0 microseconds at 100% of 1-Gigabit load• PDV(64) 1.2 microseconds

Meets SLA requirements comfortably, after adjustment for additional 10 km fiber spur


University of Auckland – latency as a factor of packet size

A -> B B -> A A -> B B -> A A -> B B-> A64 258,436 258,385 258,549 258,483 259,465 258,591

128 261,985 261,934 262,047 262,001 262,089 262,088256 265,226 265,175 265,296 265,239 265,381 265,278512 273,662 273,611 273,717 273,699 273,817 273,766

1024 289,866 289,814 289,920 289,882 289,969 289,9691280 299,537 299,485 299,581 299,565 299,691 299,6401518 306,687 396,687 306,777 306,739 306,893 306,7919212 579,630 579,629 579,727 579,689 579,784 579,835

RTTD 64PDV 64

Latency (in nanoseconds)Packet size (in bytes)

Summary results (in nanoseconds)

516,821 517,032 518,0561,235

Minimum Average Maximum

A -> B -> A A -> B -> A A -> B -> A


University of Auckland – EAPS protection switching

• Measured the impact of a ring break upon site-to-site traffic

• Worst case scenario – re-routing once around the entire ring

• Measurements conducted at single site• Metric – number of packets lost• Worst case timing was measured at 217 milliseconds• Typical timing is around 100-150 milliseconds

Exceeds SLA expectations for sub-second failover


University of Auckland – test summary

• Installation meets / exceeds original design parameters• Network is capable of delivering unconstraint full 1-

Gigabit services across the metro domain• Fiber spur by far dominates latency (85%)• Switching latency is negligible, a few microseconds per

packet at most• Serialization delay can become the dominating factor with

large packet sizes and many hops

Consider the QoS impact when designing applications for either bulk data transfer or sensitive audio/video conferencing applications


REANNZ Advanced Network

• 10 Gigabit backbone overlay on TelstraClear DWDM

• Gigabit access in all major cities

• Layer 2 network with QoS

• Routers in Auckland, Wellington, Christchurch

• International access: 155 Mbits/sec to Sydney, 622 Mbits/sec to California

• Rollout in 2006 and 2007


Acknowledgements and references

• University of Auckland ITSS department – Robert Beattie, Manager

• REANNZ – Charles Jarvie, CEO• A cast of dozens of hands at TelstraClear and JazzTech

Thank you!

1 nznog 2006 – victoria university wellington, march 23 rd, 2006 10 gigabit networking at...

Documents