8 May 2002
Abilene Update SessionAbilene Update Session
Steve Corbató
Director, Backbone Network Infrastructure
HENP Working Group
Washington DC
Steve Corbató
Director, Backbone Network Infrastructure
HENP Working Group
Washington DC
8 May 2002 2
Agenda
Network status & events of interest
10-Gbps upgrade plans
Optical networking on the regional & national scale
8 May 2002 3
Abilene – May, 2002
IP-over-SONET backbone (OC-48c, 2.5 Gbps) 53 direct connections
• 4 OC-48c connections • 1 Gigabit Ethernet trial • 23 will connect via at least OC-12c (622 Mbps) by 1Q02• Number of ATM connections decreasing
215 participants – research universities & labs• All 50 states, District of Columbia, & Puerto Rico• 15 regional GigaPoPs support ~70% of participants
Expanded access• 50 sponsored participants
– New: Smithsonian Institution, Arecibo Radio Telescope
• 23 state education networks (SEGPs)
8 May 2002 4
Abilene international connectivity
Transoceanic R&E bandwidths growing!• GÉANT – 5 Gbps between Europe and New York City now
Key international exchange points facilitated by Internet2 membership and the U.S. scientific community
• STARTAP & STAR LIGHT – Chicago (GigE)• AMPATH – Miami (OC-3c OC-12c)• Pacific Wave – Seattle (GigE)• MAN LAN - New York City (GigE/10GigE EP soon)• CA*NET3/4: Seattle, Chicago, and New York• CUDI: CENIC and Univ. of Texas at El Paso
International transit service• Collaboration with CA*NET3 and STARTAP
Sacramento
Los Angeles
Washington
Abilene International PeeringSTAR TAP/Star LightAPAN/TransPAC, Ca*net3, CERN, CERnet, FASTnet, GEMnet, IUCC, KOREN/KREONET2, NORDUnet, RNP2, SURFnet, SingAREN, TAnet2
NYCMBELNET, CA*net3,
GEANT*,HEANET,
JANET, NORDUnet
Pacific WaveAARNET, APAN/TransPAC, CA*net3, TANET2
SNVAGEMNET, SINET, SingAREN, WIDE
LOSAUNINET
AMPATHREUNA, RNP2 RETINA,
ANSP, (CRNet)
OC3-OC12
El Paso (UACJ-UT El Paso)CUDI
San Diego (CALREN2)CUDI
* ARNES, CARNET, CESnet, DFN, GRNET, RENATER, RESTENA, SWITCH, HUNGARNET, GARR-B, POL-34, RCST, RedIRIS
09 March 2002
8 May 2002 6
Packetized raw High Definition Television (HDTV)
Raw HDTV/IP – single UDP flow of 1.5 Gbps• Project of USC/ISIe, Tektronix, & U. of Wash (DARPA) • 6 Jan 2002: Seattle to Washington DC via Abilene
–Single flow utilized 60% of backbone bandwidth• 18 hours: no packets lost, 15 resequencing episodes• End-to-end network performance (includes P/NW & MAX
GigaPoPs)– Loss: <0.8 ppb (90% c.l.)– Reordering: 5 ppb
• Transcontinental 1-Gbps TCP requires loss of
– <30 ppb (1.5 KB frames)– <1 ppm (9KB jumbo)
8 May 2002 7
End-to-End Performance:‘High bandwidth is not enough’
Bulk TCP flows (transfers > 10 Mbytes) • Current median flow rate over Abilene: 1.9 Mbps
–95th percentile: 7.0 Mbps
8 May 2002 8
Netflow information sources
Weekly summaries• http://netflow.internet2.edu/weekly/
Raw data manipulation• http://www.itec.oar.net/abilene-netflow/
8 May 2002 9
Jumbo frames are supported here
Default Abilene MTU: 4.5 kB
Now we also support 9 kB MTUs on per connector basis
Motivation: support for HPC computing
Interested connectors? • Contact the NOC
8 May 2002 10
Future of Abilene
Original UCAID/Qwest agreement amended on October 1, 2001
Extension of MoU for another 5 years – until October, 2006
• Originally expired March, 2003
Upgrade of Abilene backbone to optical transport capability - ’s (unprotected)
• x4 increase in the core backbone bandwidth–OC-48c SONET (2.5 Gbps) to 10-Gbps DWDM
8 May 2002 11
Key aspects of next generation Abilene backbone - I
Native IPv6• Motivations
– Resolving IPv4 address exhaustion issues
– Preservation of the original End-to-End Architecture model• p2p collaboration tools, reverse trend to CO-centrism
– International collaboration
– Router and host OS capabilities
• Run natively - concurrent with IPv4• Replicate multicast deployment strategy• Close collaboration with Internet2 IPv6 Working Group on
regional and campus v6 rollout– Addressing architecture
8 May 2002 12
Key aspects of next generation Abilene backbone - II
Network resiliency• Abilene ’s will not be ring protected like SONET• Increasing use of videoconferencing/VoIP impose tighter restoration requirements (<100 ms)
• Options:–MPLS/TE fast reroute (initially)–IP-based IGP fast convergence (preferable)
8 May 2002 13
Key aspects of next generation Abilene backbone - III
New & differentiated measurement capabilities• Significant factor in NGA rack design
– 4 dedicated servers at each nodes– Additional provisions for future servers– Local data collection to capture data at times of network
instability
• Enhance active probing – Now: latency & jitter, loss, reachability (Surveyor)– Regular TCP/UDP throughput tests – ~1 Gbps
• Separate server for E2E performance beacon
• Enhance passive measurement– Now: SNMP (NOC) & traffic matrix/type (Netflow)– Routing (BGP & IGP)– Optical splitter taps on backbone links at select location(s)
8 May 2002 14
Abilene Observatories
Currently a program outline for better support of computer science research
• Influenced by discussions with NRLC members
1) Improved & accessible data archive• Need coherent database design • Unify & correlate 4 separate data types
– SNMP, active measurement data, routing, Netflow
2) Provision for direct network measurement and experimentation
• Resources reserved for two additional servers– Power (DC), rack space (2RU), router uplink ports (GigE)
• Need process for identifying meritorious projects• Need ‘rules of engagement’ (technical & policy)
8 May 2002 16
8 May 2002 17
Next generation router selection
Extensive router specification and test plan developed
• Team effort: UCAID staff, NOC, NC and Ohio ITECs– Chris Heerman, Matt Davy, Lee Graham, John Moore, Paul Schopis,
Matt Zekauskas
• Discussions with four router vendors
Tests focused on next gen advanced services• High performance TCP/IP throughput • High performance multicast• IPv6 functionality & throughput• Classification for QoS and measurement
3 routers tested & comm. ISPs referenced
New Juniper T640 platform selected
8 May 2002 18
Two leading national initiatives in the U.S.
Next Generation Abilene• Advanced Internet backbone
– connects entire campus networks of the research universities
• 10 Gbps nationally
TeraGrid• Virtual machine room for distributed computing (Grid) • Connecting 4 HPC centers initially
– Illinois: NCSA, Argonne– California: SDSC, Caltech
• 4x10 Gbps: Chicago Los Angeles
Ongoing collaboration between both projects
8 May 2002 19
Deployment timing
Ongoing – Backbone router procurement
Detailed deployment planning
July – Rack assembly (Indiana Univ.)
Aug/Sep – New rack deployment at all 11 nodes
Fall – First Wave ’s commissioned
Fall meeting demonstration events• iGRID 2002 (Amsterdam) – late Sep.• Internet2 Fall Member Meeting (Los Angeles) – late Oct.• SC2002 (Baltimore) – mid Nov.
Remaining ’s commissioned in 2003
Please let us know now of 2002 upgrade plans
8 May 2002 20
Abilene cost recovery model
Connection (per connection) Annual fee
OC-3 (155 Mbps)* $110,000
OC-12 (622 Mbps) $270,000
Gigabit Ethernet (1 Gbps)** $325,000
OC-48 (2.5 Gbps) $430,000
OC-192/10 GigE** (10 Gbps) $490,000
Participation (per university) $20,000
8 May 2002 21
Abilene program changes
10-Gbps (OC-192c POS) connections backhaul available wherever needed & possible
– Only required now for 1 of 4 OC-48c connections
• 3-year connectivity commitment required
Gigabit and 10-Gigabit Ethernet• Available when connector has dark fiber access into Abilene router
node• Backhaul not available
ATM connection & peer support • TAC recommended ending ATM support by fall 2003• Two major ATM-based GigaPoPs have migrated • 2 of 3 NGIXes still are ATM-based
– NGIX-Chicago @ STAR LIGHT is now GigE
• Urging phased migration for connectors & peers
8 May 2002 22
Conclusions – Abilene future
Backbone upgrade project underway• Partnership with Qwest extended thru 2006• Juniper T640 routers selected for backbone• 10-Gbps backbone deployment starts this fall
Advanced service foci• Native, high-performance IPv6• Enhanced, differentiated measurement• Network resiliency
Incremental, non-disruptive transition
Complementary to and collaborative with NSF’s TeraGrid
8 May 2002 23
For more information
Web: www.internet2.edu/abilene
E-mail: [email protected]
Again please let us know now of 2002 connection upgrade plans
8 May 2002 24
Optical network project differentiation
Distance scale (km)
Examples Equipment
Metro < 60
UW(SEA),
USC/ISI(LA)
Dark fiber & end terminals
State/
Regional
< 500
(ULH: <2500)
I-WIRE (IL),
CENIC ONI,
I-LIGHT (IN)
Add OO
amplifiers
Extended
Regional/
National
> 500
PLR,
TeraGrid
Abilene
Add OEO
regenerators
& O&M $’s
8 May 2002 25
Regional optical networking
Regional (state-based) optical networking projects are critical for next generation architecture:
• Three-level hierarchy: – National backbones, GigaPoPs, Campuses
• Leading examples of state-based initiatives– CENIC ONI (California), I-WIRE (Illinois), I-LIGHT (Indiana), NC
Close collaboration with the Quilt Project• Regional Optical Networking effort
U.S. carrier DWDM access is now not nearly as widespread as with SONET
• 30-60 cities for DWDM vs. ~120 cities for SONET (ca. 1998)
8 May 2002 26
Pacific Light Rail (Source: Greg Scott, CENIC/UCSC)
8 May 2002 28
National optical networking options
1 – Provision incremental wavelengths• Obtain 10-Gbps ’s as with SONET• Exploit smaller incremental cost of additional ’s
– 1st cost is ~10x than subsequent ’s
2 – Build dim fiber facility• Partner with a facilities-based provider
– Acquire 2 fiber pairs on a national scale– Outsource operation of transmission equipment
• Needs lower-cost optical transmission equipment– Find ELH/ULH optical kit partner
The classic ‘buy vs. build’ decision in Information Technology
• Option 1 selected for TeraGrid and Next Gen Abilene
8 May 2002 29
National Light Rail
Project objectives• form lightweight, but highly coordinated, collaboration to provision,
acquire, and/or operate optical networking assets and services• leverage collective buying power and experience of the consortium
(ANL, CENIC, P/NW, UCAID) from the metropolitan to the national scales
• serve as optical infrastructure substrate for e-science projects proposing to a diverse array of funding agencies
• facilitate advanced network measurement and academic research
Initial collaboration• TeraGrid (Argonne), UCAID, CENIC and P/NW GigaPoPs• UCSD, UIC
8 May 2002 30
National Light Rail – an evolvingview
Key Functions brokerage service using established relationships with multiple facilities-based carriers
• Ongoing evaluation of potential acquisition and operation of national fiber optical network facility in partnership with the corporate sector
www.internet2.edu