Shawn P. McKeeShawn P. McKee University of MichiganUniversity of MichiganUltraLight Meeting, NSFUltraLight Meeting, NSF

January 26, 2005January 26, 2005

Network Working Group ReportNetwork Working Group Report

UltraLight Network ReportUltraLight Network Report

http://atlas.physics.lsa.umich.edu/docushare/dsweb/Get/Document-2546

Overview of UltraLight NetworkOverview of UltraLight NetworkBackboneBackboneUltraLight SitesUltraLight Sites

UltraLight PartnersUltraLight PartnersNetwork EngineeringNetwork Engineering

Effective Protocols for “Best Effort”Effective Protocols for “Best Effort”MPLS/QoS for “Sizeable Pipes”MPLS/QoS for “Sizeable Pipes”Optical Path Management Plans for “Optical Circuits”Optical Path Management Plans for “Optical Circuits”

MonitoringMonitoringDisk-to-Disk TransfersDisk-to-Disk TransfersMilestonesMilestonesStatus and SummaryStatus and Summary

(MS Word, for PDF use 2547 rather than 2546)

S. McKee (UM, Team Leader)S. McKee (UM, Team Leader)

S. Ravot (LHCNet)S. Ravot (LHCNet)

R. Summerhill (Abilene/HOPI)R. Summerhill (Abilene/HOPI)

D. Pokorney (FLR)D. Pokorney (FLR)

S. Gerstenberger (MiLR)S. Gerstenberger (MiLR)

C. Griffin (UF)C. Griffin (UF)

The UltraLight Network The UltraLight Network Engineering TeamEngineering Team

J. Ibarra (WHREN, AW)J. Ibarra (WHREN, AW)

C. Guok (ESnet)C. Guok (ESnet)

L. Cottrell (SLAC)L. Cottrell (SLAC)

D. Petravick (FNAL) D. Petravick (FNAL)

R. Hockett (UM)R. Hockett (UM)

E. Rubi (FIU)E. Rubi (FIU)

UltraLight BackboneUltraLight Backbone

UltraLight has a non-standard core network with dynamic UltraLight has a non-standard core network with dynamic links and varying bandwidth inter-connecting our nodes. links and varying bandwidth inter-connecting our nodes. Optical Hybrid Global NetworkOptical Hybrid Global Network

The core of UltraLight will dynamically evolve as function of The core of UltraLight will dynamically evolve as function of available resources on other backbones such as NLR, available resources on other backbones such as NLR, HOPI, Abilene or ESnet. HOPI, Abilene or ESnet.

The main resources for UltraLight:The main resources for UltraLight: LHCnet (IP, L2VPN, CCC)LHCnet (IP, L2VPN, CCC) Abilene (IP, L2VPN)Abilene (IP, L2VPN) ESnet (IP, L2VPN)ESnet (IP, L2VPN) Cisco NLR wave (Ethernet)Cisco NLR wave (Ethernet) HOPI NLR waves (Ethernet; provisioned on demand)HOPI NLR waves (Ethernet; provisioned on demand) UltraLight nodes: Caltech, SLAC, FNAL, UF, UM, UltraLight nodes: Caltech, SLAC, FNAL, UF, UM,

StarLight, CENIC PoP at LA, CERN StarLight, CENIC PoP at LA, CERN

UltraLight Network UltraLight Network Infrastructure ElementsInfrastructure Elements

Trans-US 10G Trans-US 10G s Riding on NLR, Plus CENIC, FLR, MiLRs Riding on NLR, Plus CENIC, FLR, MiLR LA – CHI (2 Waves): HOPI and Cisco Research WavesLA – CHI (2 Waves): HOPI and Cisco Research Waves CHI – JAX (Florida Lambda Rail)CHI – JAX (Florida Lambda Rail)

Dark Fiber Caltech – L.A.: 2 X 10G Waves (One to WAN In Lab);Dark Fiber Caltech – L.A.: 2 X 10G Waves (One to WAN In Lab); 10G Wave L.A. to Sunnyvale for UltraScience Net Connection 10G Wave L.A. to Sunnyvale for UltraScience Net Connection Dark Fiber with 10G Wave: StarLight – FermilabDark Fiber with 10G Wave: StarLight – Fermilab Dedicated Wave StarLight – Michigan Light RailDedicated Wave StarLight – Michigan Light Rail SLAC: ESnet MAN to Provide 2 X 10G Links (from July): SLAC: ESnet MAN to Provide 2 X 10G Links (from July): One for Production, and One for Research One for Production, and One for Research Partner with Advanced Research & Production NetworksPartner with Advanced Research & Production Networks

LHCNet (Starlight- CERN), Abilene/HOPI, ESnet, NetherLight, LHCNet (Starlight- CERN), Abilene/HOPI, ESnet, NetherLight, GLIF, UKLight, CA*net4GLIF, UKLight, CA*net4

Intercont’l extensions: Brazil (CHEPREO/WHREN), GLORIAD, Intercont’l extensions: Brazil (CHEPREO/WHREN), GLORIAD, Tokyo, AARNet, Taiwan, ChinaTokyo, AARNet, Taiwan, China

UltraLight SitesUltraLight Sites

UltraLight currently has 10 UltraLight currently has 10 participating core sites (shown participating core sites (shown alphabetically)alphabetically)

The table provides a quick The table provides a quick summary of the near term summary of the near term connectivity plansconnectivity plans

Details and diagrams for each Details and diagrams for each site and its regional networks site and its regional networks are shown in the technical reportare shown in the technical report

((see URL on second slidesee URL on second slide))

SiteSite DateDate TypeType StorageStorage Out of Out of BandBand

BNLBNL MarchMarch OC48OC48 TBDTBD TBDTBD

CaltechCaltech January January 10 GE10 GE 1 TB May 1 TB May YY

CERNCERN JanuaryJanuary OC192OC192 TBDTBD YY

FIUFIU JanuaryJanuary OC12OC12 TBDTBD YY

FNALFNAL MarchMarch 10 GE10 GE TBDTBD TBDTBD

I2I2 March March MPLSMPLS

L2VPNL2VPN

TBDTBD TBDTBD

MITMIT MayMay OC48OC48 TBDTBD TBDTBD

SLACSLAC SeptSept 10 GE10 GE TBDTBD TBDTBD

UFUF FebruaryFebruary 10 GE10 GE 1 TB May1 TB May YY

UMUM MarchMarch 10 GE10 GE 1 TB May1 TB May YY

International PartnersInternational Partners

One of the UltraLight program’s strengths is the large number of One of the UltraLight program’s strengths is the large number of important international partners we have:important international partners we have:

AMPATH AMPATH http://http://www.ampath.netwww.ampath.netAARNet AARNet http://http://www.aarnet.edu.auwww.aarnet.edu.auBrazil/UERJ Brazil/UERJ http://http://www.hepgridbrazil.uerj.brwww.hepgridbrazil.uerj.br// CA*net4 CA*net4 http://www.canarie.ca/canet4http://www.canarie.ca/canet4GLORIAD GLORIAD http://http://www.gloriad.orgwww.gloriad.org//IEEAF IEEAF http://http://www.ieeaf.orgwww.ieeaf.org// Korea/KOREN Korea/KOREN http://www.koren21.net/eng/network/domestic.phphttp://www.koren21.net/eng/network/domestic.php

NetherLight NetherLight http://http://www.surfnet.nl/info/innovatie/netherlight/home.jspwww.surfnet.nl/info/innovatie/netherlight/home.jsp

UKLight UKLight http://http://www.uklight.ac.ukwww.uklight.ac.uk//As well as collaborators from China, Japan and Taiwan.As well as collaborators from China, Japan and Taiwan.

UltraLight is well positioned to develop and coordinate UltraLight is well positioned to develop and coordinate globalglobal advances to networks for LHC Physics advances to networks for LHC Physics

UltraLight envisions a 4 year program to deliver a new, UltraLight envisions a 4 year program to deliver a new, high-performance, network-integrated infrastructure:high-performance, network-integrated infrastructure:

Phase I will last Phase I will last 12 months12 months and focus on and focus on deploying the initial deploying the initial network infrastructurenetwork infrastructure and and bringing up first services bringing up first services ((Note: we are well on our way, the network is almost up Note: we are well on our way, the network is almost up and the first services are being deployedand the first services are being deployed))

Phase II will last Phase II will last 18 months18 months and concentrate on and concentrate on implementing all the needed servicesimplementing all the needed services and and extending the extending the infrastructure to additional sites (infrastructure to additional sites (We will be entering this We will be entering this phase starting approximately this summerphase starting approximately this summer))

Phase III will complete UltraLight and last Phase III will complete UltraLight and last 18 months18 months. The . The focus will be on a focus will be on a transition to production in support of transition to production in support of LHC Physics; + eVLBI AstronomyLHC Physics; + eVLBI Astronomy

Workplan/Phased DeploymentWorkplan/Phased Deployment

GOAL:GOAL: Determine an effective mix of bandwidth-management Determine an effective mix of bandwidth-management techniques for this application-space, particularly:techniques for this application-space, particularly:

Best-effort and “scavenger” using “effective” protocolsBest-effort and “scavenger” using “effective” protocolsMPLS with QOS-enabled packet switchingMPLS with QOS-enabled packet switchingDedicated paths arranged with TL1 commands, GMPLSDedicated paths arranged with TL1 commands, GMPLS

PLAN: PLAN: Develop, Test the most cost-effective integrated combination Develop, Test the most cost-effective integrated combination of network technologies on our unique testbed:of network technologies on our unique testbed:

1.1.Exercise UltraLight applications on NLR, Abilene and campus Exercise UltraLight applications on NLR, Abilene and campus networks, as well as LHCNet, and our international partnersnetworks, as well as LHCNet, and our international partners

Progressively enhance Abilene with QOS support to Progressively enhance Abilene with QOS support to protect production traffic protect production traffic

Incorporate emerging NLR and RON-based lightpath Incorporate emerging NLR and RON-based lightpath and and lambda facilitieslambda facilities

2.2.Deploy and systematically study ultrascale protocol stacksDeploy and systematically study ultrascale protocol stacks (such as FAST) addressing issues of performance & fairness (such as FAST) addressing issues of performance & fairness

3.3.Use MPLS/QoS and other forms of BW management, and Use MPLS/QoS and other forms of BW management, and adjustments of optical paths, to optimize end-to-end adjustments of optical paths, to optimize end-to-end performance performance among a set of virtualized disk serversamong a set of virtualized disk servers

UltraLight Network EngineeringUltraLight Network Engineering

UltraLight: Effective ProtocolsUltraLight: Effective Protocols

The The protocolsprotocols used to reliably move data are a critical used to reliably move data are a critical component of Physics “end-to-end” use of the networkcomponent of Physics “end-to-end” use of the network

TCP is the most widely used protocol for reliable data TCP is the most widely used protocol for reliable data transport, but is becoming ever more ineffective for transport, but is becoming ever more ineffective for higher and higher bandwidth-delay networks.higher and higher bandwidth-delay networks.

UltraLight will explore extensions to TCP (UltraLight will explore extensions to TCP (HSTCP, HSTCP, Westwood+, HTCPWestwood+, HTCP, , FASTFAST) designed to maintain fair-) designed to maintain fair-sharing of networks and, at the same time, to allow sharing of networks and, at the same time, to allow efficient, effective use of these networks.efficient, effective use of these networks.

UltraLight plans to identify the most effective fair UltraLight plans to identify the most effective fair protocol and implement it in support of our “Best Effort” protocol and implement it in support of our “Best Effort” network components.network components.

Selecting the data Transport protocolSelecting the data Transport protocol

Tests between CERN and CaltechTests between CERN and CaltechCapacity = OC-192 9.5Gbps; 264 ms round trip latency; 1 flowCapacity = OC-192 9.5Gbps; 264 ms round trip latency; 1 flowSending station:Sending station: Tyan S2882 motherboard, 2 x Opteron 2.4 Tyan S2882 motherboard, 2 x Opteron 2.4

GHz, 2 GB DDR.GHz, 2 GB DDR.Receiving station:Receiving station: (CERN OpenLab): (CERN OpenLab): HP rx4640, 4 x 1.5GHz HP rx4640, 4 x 1.5GHz

Itanium-2, zx1 chipset, 8GB memoryItanium-2, zx1 chipset, 8GB memoryNetwork adapter:Network adapter: S2io 10 GE S2io 10 GE

Linux TCP Linux Westwood+ Linux BIC TCP FAST

3.0 Gbps 5.0 Gbps 7.3 Gbps4.1 Gbps

MPLS/QoS for UltraLightMPLS/QoS for UltraLight

UltraLight plans to explore the full range of UltraLight plans to explore the full range of end-to-end end-to-end connectionsconnections across the network, from across the network, from best-effortbest-effort, , packet-packet-switchedswitched through through dedicated end-to-end light-pathsdedicated end-to-end light-paths. . MPLS paths with QoSMPLS paths with QoS attributesattributes fill a middle ground in fill a middle ground in this network space and allow fine-grained allocation of this network space and allow fine-grained allocation of virtual pipes, sized to the needs of the application or virtual pipes, sized to the needs of the application or user.user.

Network QoS with Three Classes: Best Effort, Class 4 and EF

0

200

400

600

800

1000

1200

0 100 200 300 400 500 600 700 800 900 1000

Time (Seconds)Best Effort Class 4 Express Forwarding TOTAL Wire Speed

UltraLight, in conjunction UltraLight, in conjunction with the DoE/MICS with the DoE/MICS funded TeraPaths effort, funded TeraPaths effort, is working toward is working toward extensible solutions for extensible solutions for implementing such implementing such capabilities in next capabilities in next generation networksgeneration networks

TeraPaths Initial QoS test at BNL

Optical Path PlansOptical Path Plans

Emerging “light path” technologies are becoming Emerging “light path” technologies are becoming popular in the Grid community:popular in the Grid community:

They can extend and augment existing grid computing They can extend and augment existing grid computing infrastructures, currently focused on CPU/storage, to infrastructures, currently focused on CPU/storage, to include the network as an integral Grid component. include the network as an integral Grid component.

Those technologies seem to be the most effective way to Those technologies seem to be the most effective way to offer network resource provisioning on-demand between offer network resource provisioning on-demand between end-systems.end-systems.

A major capability we wish to develop in Ultralight A major capability we wish to develop in Ultralight network nodes is the ability to dynamically switch optical network nodes is the ability to dynamically switch optical paths across the node, bypassing electronic equipment paths across the node, bypassing electronic equipment via a fiber cross connect.via a fiber cross connect.The ability to switch dynamically provides additional The ability to switch dynamically provides additional functionality and also models the more abstract case functionality and also models the more abstract case where switching is done between colors (ITU grid where switching is done between colors (ITU grid lambdas).lambdas).

Network monitoring is Network monitoring is essentialessential for UltraLight. for UltraLight. We need to understand our network infrastructure and We need to understand our network infrastructure and track its performance both historically and in real-time to track its performance both historically and in real-time to enable the network as a managed robust component of enable the network as a managed robust component of our overall infrastructure. our overall infrastructure. There are two ongoing efforts we intend to leverage to There are two ongoing efforts we intend to leverage to help provide us with the monitoring capability required: help provide us with the monitoring capability required:

IEPM IEPM http://www-http://www-iepm.slac.stanford.edu/bwiepm.slac.stanford.edu/bw// MonALISA MonALISA http://monalisa.cern.chhttp://monalisa.cern.ch

Both efforts have already made significant progress Both efforts have already made significant progress within UltraLightwithin UltraLight

Monitoring for UltraLightMonitoring for UltraLight

Monitoring for UltraLightMonitoring for UltraLight

Shown is one Shown is one example of some of example of some of the progress that the progress that has been made in has been made in monitoring for monitoring for UltraLight. UltraLight. MonALISAMonALISA has been has been augmented with a augmented with a real timereal time module module that provides that provides complete picture of complete picture of the connectivity the connectivity graphs and delay on graphs and delay on each segment for each segment for routersrouters, , networksnetworks and and AS AS

ROUTERS

ROUTERS

NETWORKS

AS

End-Systems performance End-Systems performance

Latest disk to disk over 10Gbps WAN: Latest disk to disk over 10Gbps WAN: 4.3 Gbits/sec (536 MB/sec)4.3 Gbits/sec (536 MB/sec) - 8 TCP - 8 TCP streams from CERN to Caltech; windows, 1TB file streams from CERN to Caltech; windows, 1TB file

Quad Opteron AMD848 2.2GHz processors with 3 AMD-8131 chipsets: 4 64-Quad Opteron AMD848 2.2GHz processors with 3 AMD-8131 chipsets: 4 64-bit/133MHz PCI-X slots.bit/133MHz PCI-X slots.

3 Supermicro Marvell SATA disk controllers + 24 SATA 7200rpm SATA 3 Supermicro Marvell SATA disk controllers + 24 SATA 7200rpm SATA disksdisks Local Disk IO – Local Disk IO – 9.6 Gbits/sec9.6 Gbits/sec (1.2 GBytes/sec read/write, with <20% (1.2 GBytes/sec read/write, with <20%

CPU utilization)CPU utilization)10GE NIC10GE NIC

10 GE NIC – 10 GE NIC – 7.5 Gbits/sec7.5 Gbits/sec (memory-to-memory, with 52% CPU (memory-to-memory, with 52% CPU utilization)utilization)

2*10 GE NIC (802.3ad link aggregation) – 2*10 GE NIC (802.3ad link aggregation) – 11.1 Gbits/sec11.1 Gbits/sec (memory-to- (memory-to-memory)memory)

PCI-Express, TCP offload enginesPCI-Express, TCP offload enginesA 5U server with 24 disks A 5U server with 24 disks (9 TB)(9 TB) and a 10 GbE NIC is capable of 700 and a 10 GbE NIC is capable of 700

MBytes/sec in the LAN and 330 MBytes/sec in a WAN is $ 25k today.MBytes/sec in the LAN and 330 MBytes/sec in a WAN is $ 25k today.Small server with a few disks Small server with a few disks (1.2 TB)(1.2 TB) capable of 120 Mbytes/sec (matching capable of 120 Mbytes/sec (matching

a GbEport) is $ 4K.a GbEport) is $ 4K.

Network Connect MilestonesNetwork Connect Milestones

January 2005:January 2005: NLR Cisco wave connecting LA to CHI NLR Cisco wave connecting LA to CHI BGP peering with Abilene & ESnet at CHI.BGP peering with Abilene & ESnet at CHI. MPLS upgrade at CHI.MPLS upgrade at CHI. Two virtual fibers connecting CERN to CaltechTwo virtual fibers connecting CERN to Caltech End-systems at Chicago in UltraLight domain. End-systems at Chicago in UltraLight domain. Layer 2 connection to Abilene at Los-AngelesLayer 2 connection to Abilene at Los-Angeles Extension of UltraLight Network to CERNExtension of UltraLight Network to CERN

February 2005:February 2005: Connection to HOPI at 10 GE Connection to HOPI at 10 GE Connection to FLR at 10 GE Connection to FLR at 10 GE

March 2005:March 2005: 10 GE link to UM via MiLR10 GE link to UM via MiLR Connection to BNL at OC48Connection to BNL at OC48

May 2005:May 2005: Connection to MIT at OC48 Connection to MIT at OC48April 2005: April 2005: “Manual” provisioning across UltraLight “Manual” provisioning across UltraLightAugust 2005:August 2005: Semi-autonomous provisioning process. Semi-autonomous provisioning process. September 2005:September 2005: Connection to SLAC at 10 GE Connection to SLAC at 10 GE

NLR Cisco wave from exclusive to scheduled NLR Cisco wave from exclusive to scheduled useuse

UltraLight Near-term MilestonesUltraLight Near-term MilestonesProtocols:Protocols:

Integration and of FAST TCP (V.1) (Integration and of FAST TCP (V.1) (July 2005July 2005) ) New MPLS and optical path-based provisioning (New MPLS and optical path-based provisioning (Aug 2005Aug 2005)) New TCP implementations testing (New TCP implementations testing (Aug-Sep, 2005Aug-Sep, 2005))

Optical Switching:Optical Switching: Commission optical switch at the LA CENIC/NLR and CERN (Commission optical switch at the LA CENIC/NLR and CERN (May 2005May 2005)) Develop dynamic server connections on a path for TB transactions (Develop dynamic server connections on a path for TB transactions (Sep 2005Sep 2005))

Storage and Application Services:Storage and Application Services: Evaluate/optimize drivers/parameters for I/O filesystems (Evaluate/optimize drivers/parameters for I/O filesystems (April 2005April 2005)) Evaluate/optimize drivers/parameters for 10 GbE server NICs (Evaluate/optimize drivers/parameters for 10 GbE server NICs (June 2005June 2005) ) Select hardware for 10 GE NICs, Buses and RAID controllers (Select hardware for 10 GE NICs, Buses and RAID controllers (June-Sep 2005June-Sep 2005)) Breaking the 1 GByte/s barrier (Breaking the 1 GByte/s barrier (Sep 2005Sep 2005))

Monitoring and Simulation:Monitoring and Simulation: Deployment of end-to-end monitoring framework. (Deployment of end-to-end monitoring framework. (Aug 2005Aug 2005)) Integration of tools/models to build a simulator for network fabric. (Integration of tools/models to build a simulator for network fabric. (Dec 2005Dec 2005))

Agents: Agents: Start development of Agents for resource scheduling (Start development of Agents for resource scheduling (June 2005June 2005)) Match scheduling allocations to usage policies (Match scheduling allocations to usage policies (Sep 2005Sep 2005) )

WanInLab: WanInLab: Connect Caltech WanInLab to testbed (Connect Caltech WanInLab to testbed (June 2005June 2005)) Procedure to move new protocol stacks into field trials (Procedure to move new protocol stacks into field trials (June 2005June 2005))

Summary: Network ProgressSummary: Network ProgressFor many years the For many years the Wide Area NetworkWide Area Network has been the has been the bottleneckbottleneck;;

this is no longer the case in many countries thus making deployment this is no longer the case in many countries thus making deployment of a data intensive Grid infrastructure possible!of a data intensive Grid infrastructure possible! Recent I2LSR records show for the first time ever that the network Recent I2LSR records show for the first time ever that the network

can be truly transparent; can be truly transparent; throughputs are limited by end-hoststhroughputs are limited by end-hosts Challenge shifted from getting adequate bandwidth to Challenge shifted from getting adequate bandwidth to deploying deploying

adequate infrastructureadequate infrastructure to make to make effectiveeffective use of it! use of it!

Some transport protocol issues still need to be resolved; however there Some transport protocol issues still need to be resolved; however there are many encouraging signs that practical solutions may now be in are many encouraging signs that practical solutions may now be in sight.sight.

1GByte/sec disk to disk challenge. 1GByte/sec disk to disk challenge. Today: 1 TB at Today: 1 TB at 536 MB/sec 536 MB/sec from CERN to Caltechfrom CERN to Caltech Still in Early Stages; Expect Substantial ImprovementsStill in Early Stages; Expect Substantial Improvements

Next generation network and Grid systemNext generation network and Grid system Extend and augment existing grid computing infrastructures Extend and augment existing grid computing infrastructures

(currently focused on CPU/storage) to include the network (currently focused on CPU/storage) to include the network as an integral component.as an integral component.

Conclusion: Conclusion: We’re Ready for the next stepWe’re Ready for the next step

The network technical group has been hard at work on The network technical group has been hard at work on implementing UltraLight.implementing UltraLight.

Significant progress has been made, allowing us to build Significant progress has been made, allowing us to build upon these achievements.upon these achievements.

Our global collaborators are ready to work with us on Our global collaborators are ready to work with us on achieving the UltraLight vision.achieving the UltraLight vision.

We are ready to ramp up our efforts and we need the We are ready to ramp up our efforts and we need the additional resources of our second year funding to keep our additional resources of our second year funding to keep our momentum.momentum.

We look forward to a busy productive year working on UltraLight!We look forward to a busy productive year working on UltraLight!