8 May 2002

Abilene Update SessionAbilene Update Session

Steve Corbató

Director, Backbone Network Infrastructure

HENP Working Group

Washington DC

Steve Corbató

Director, Backbone Network Infrastructure

HENP Working Group

Washington DC

8 May 2002 2

Agenda

Network status & events of interest

10-Gbps upgrade plans

Optical networking on the regional & national scale

8 May 2002 3

Abilene – May, 2002

IP-over-SONET backbone (OC-48c, 2.5 Gbps) 53 direct connections

• 4 OC-48c connections • 1 Gigabit Ethernet trial • 23 will connect via at least OC-12c (622 Mbps) by 1Q02• Number of ATM connections decreasing

215 participants – research universities & labs• All 50 states, District of Columbia, & Puerto Rico• 15 regional GigaPoPs support ~70% of participants

Expanded access• 50 sponsored participants

– New: Smithsonian Institution, Arecibo Radio Telescope

• 23 state education networks (SEGPs)

8 May 2002 4

Abilene international connectivity

Transoceanic R&E bandwidths growing!• GÉANT – 5 Gbps between Europe and New York City now

Key international exchange points facilitated by Internet2 membership and the U.S. scientific community

• STARTAP & STAR LIGHT – Chicago (GigE)• AMPATH – Miami (OC-3c OC-12c)• Pacific Wave – Seattle (GigE)• MAN LAN - New York City (GigE/10GigE EP soon)• CA*NET3/4: Seattle, Chicago, and New York• CUDI: CENIC and Univ. of Texas at El Paso

International transit service• Collaboration with CA*NET3 and STARTAP

Sacramento

Los Angeles

Washington

Abilene International PeeringSTAR TAP/Star LightAPAN/TransPAC, Ca*net3, CERN, CERnet, FASTnet, GEMnet, IUCC, KOREN/KREONET2, NORDUnet, RNP2, SURFnet, SingAREN, TAnet2

NYCMBELNET, CA*net3,

GEANT*,HEANET,

JANET, NORDUnet

Pacific WaveAARNET, APAN/TransPAC, CA*net3, TANET2

SNVAGEMNET, SINET, SingAREN, WIDE

LOSAUNINET

AMPATHREUNA, RNP2 RETINA,

ANSP, (CRNet)

OC3-OC12

El Paso (UACJ-UT El Paso)CUDI

San Diego (CALREN2)CUDI

* ARNES, CARNET, CESnet, DFN, GRNET, RENATER, RESTENA, SWITCH, HUNGARNET, GARR-B, POL-34, RCST, RedIRIS

09 March 2002

8 May 2002 6

Packetized raw High Definition Television (HDTV)

Raw HDTV/IP – single UDP flow of 1.5 Gbps• Project of USC/ISIe, Tektronix, & U. of Wash (DARPA) • 6 Jan 2002: Seattle to Washington DC via Abilene

–Single flow utilized 60% of backbone bandwidth• 18 hours: no packets lost, 15 resequencing episodes• End-to-end network performance (includes P/NW & MAX

GigaPoPs)– Loss: <0.8 ppb (90% c.l.)– Reordering: 5 ppb

• Transcontinental 1-Gbps TCP requires loss of

– <30 ppb (1.5 KB frames)– <1 ppm (9KB jumbo)

8 May 2002 7

End-to-End Performance:‘High bandwidth is not enough’

Bulk TCP flows (transfers > 10 Mbytes) • Current median flow rate over Abilene: 1.9 Mbps

–95th percentile: 7.0 Mbps

8 May 2002 8

Netflow information sources

Weekly summaries• http://netflow.internet2.edu/weekly/

Raw data manipulation• http://www.itec.oar.net/abilene-netflow/

8 May 2002 9

Jumbo frames are supported here

Default Abilene MTU: 4.5 kB

Now we also support 9 kB MTUs on per connector basis

Motivation: support for HPC computing

Interested connectors? • Contact the NOC

8 May 2002 10

Future of Abilene

Original UCAID/Qwest agreement amended on October 1, 2001

Extension of MoU for another 5 years – until October, 2006

• Originally expired March, 2003

Upgrade of Abilene backbone to optical transport capability - ’s (unprotected)

• x4 increase in the core backbone bandwidth–OC-48c SONET (2.5 Gbps) to 10-Gbps DWDM

8 May 2002 11

Key aspects of next generation Abilene backbone - I

Native IPv6• Motivations

– Resolving IPv4 address exhaustion issues

– Preservation of the original End-to-End Architecture model• p2p collaboration tools, reverse trend to CO-centrism

– International collaboration

– Router and host OS capabilities

• Run natively - concurrent with IPv4• Replicate multicast deployment strategy• Close collaboration with Internet2 IPv6 Working Group on

regional and campus v6 rollout– Addressing architecture

8 May 2002 12

Key aspects of next generation Abilene backbone - II

Network resiliency• Abilene ’s will not be ring protected like SONET• Increasing use of videoconferencing/VoIP impose tighter restoration requirements (<100 ms)

• Options:–MPLS/TE fast reroute (initially)–IP-based IGP fast convergence (preferable)

8 May 2002 13

Key aspects of next generation Abilene backbone - III

New & differentiated measurement capabilities• Significant factor in NGA rack design

– 4 dedicated servers at each nodes– Additional provisions for future servers– Local data collection to capture data at times of network

instability

• Enhance active probing – Now: latency & jitter, loss, reachability (Surveyor)– Regular TCP/UDP throughput tests – ~1 Gbps

• Separate server for E2E performance beacon

• Enhance passive measurement– Now: SNMP (NOC) & traffic matrix/type (Netflow)– Routing (BGP & IGP)– Optical splitter taps on backbone links at select location(s)

8 May 2002 14

Abilene Observatories

Currently a program outline for better support of computer science research

• Influenced by discussions with NRLC members

1) Improved & accessible data archive• Need coherent database design • Unify & correlate 4 separate data types

– SNMP, active measurement data, routing, Netflow

2) Provision for direct network measurement and experimentation

• Resources reserved for two additional servers– Power (DC), rack space (2RU), router uplink ports (GigE)

• Need process for identifying meritorious projects• Need ‘rules of engagement’ (technical & policy)

8 May 2002 16

8 May 2002 17

Next generation router selection

Extensive router specification and test plan developed

• Team effort: UCAID staff, NOC, NC and Ohio ITECs– Chris Heerman, Matt Davy, Lee Graham, John Moore, Paul Schopis,

Matt Zekauskas

• Discussions with four router vendors

Tests focused on next gen advanced services• High performance TCP/IP throughput • High performance multicast• IPv6 functionality & throughput• Classification for QoS and measurement

3 routers tested & comm. ISPs referenced

New Juniper T640 platform selected

8 May 2002 18

Two leading national initiatives in the U.S.

Next Generation Abilene• Advanced Internet backbone

– connects entire campus networks of the research universities

• 10 Gbps nationally

TeraGrid• Virtual machine room for distributed computing (Grid) • Connecting 4 HPC centers initially

– Illinois: NCSA, Argonne– California: SDSC, Caltech

• 4x10 Gbps: Chicago Los Angeles

Ongoing collaboration between both projects

8 May 2002 19

Deployment timing

Ongoing – Backbone router procurement

Detailed deployment planning

July – Rack assembly (Indiana Univ.)

Aug/Sep – New rack deployment at all 11 nodes

Fall – First Wave ’s commissioned

Fall meeting demonstration events• iGRID 2002 (Amsterdam) – late Sep.• Internet2 Fall Member Meeting (Los Angeles) – late Oct.• SC2002 (Baltimore) – mid Nov.

Remaining ’s commissioned in 2003

Please let us know now of 2002 upgrade plans

8 May 2002 20

Abilene cost recovery model

Connection (per connection) Annual fee

OC-3 (155 Mbps)* $110,000

OC-12 (622 Mbps) $270,000

Gigabit Ethernet (1 Gbps)** $325,000

OC-48 (2.5 Gbps) $430,000

OC-192/10 GigE** (10 Gbps) $490,000

Participation (per university) $20,000

8 May 2002 21

Abilene program changes

10-Gbps (OC-192c POS) connections backhaul available wherever needed & possible

– Only required now for 1 of 4 OC-48c connections

• 3-year connectivity commitment required

Gigabit and 10-Gigabit Ethernet• Available when connector has dark fiber access into Abilene router

node• Backhaul not available

ATM connection & peer support • TAC recommended ending ATM support by fall 2003• Two major ATM-based GigaPoPs have migrated • 2 of 3 NGIXes still are ATM-based

– NGIX-Chicago @ STAR LIGHT is now GigE

• Urging phased migration for connectors & peers

8 May 2002 22

Conclusions – Abilene future

Backbone upgrade project underway• Partnership with Qwest extended thru 2006• Juniper T640 routers selected for backbone• 10-Gbps backbone deployment starts this fall

Advanced service foci• Native, high-performance IPv6• Enhanced, differentiated measurement• Network resiliency

Incremental, non-disruptive transition

Complementary to and collaborative with NSF’s TeraGrid

8 May 2002 23

For more information

Web: www.internet2.edu/abilene

E-mail: abilene@internet2.edu

Again please let us know now of 2002 connection upgrade plans

8 May 2002 24

Optical network project differentiation

Distance scale (km)

Examples Equipment

Metro < 60

UW(SEA),

USC/ISI(LA)

Dark fiber & end terminals

State/

Regional

< 500

(ULH: <2500)

I-WIRE (IL),

CENIC ONI,

I-LIGHT (IN)

Add OO

amplifiers

Extended

Regional/

National

> 500

PLR,

TeraGrid

Abilene

Add OEO

regenerators

& O&M $’s

8 May 2002 25

Regional optical networking

Regional (state-based) optical networking projects are critical for next generation architecture:

• Three-level hierarchy: – National backbones, GigaPoPs, Campuses

• Leading examples of state-based initiatives– CENIC ONI (California), I-WIRE (Illinois), I-LIGHT (Indiana), NC

Close collaboration with the Quilt Project• Regional Optical Networking effort

U.S. carrier DWDM access is now not nearly as widespread as with SONET

• 30-60 cities for DWDM vs. ~120 cities for SONET (ca. 1998)

8 May 2002 26

Pacific Light Rail (Source: Greg Scott, CENIC/UCSC)

8 May 2002 28

National optical networking options

1 – Provision incremental wavelengths• Obtain 10-Gbps ’s as with SONET• Exploit smaller incremental cost of additional ’s

– 1st cost is ~10x than subsequent ’s

2 – Build dim fiber facility• Partner with a facilities-based provider

– Acquire 2 fiber pairs on a national scale– Outsource operation of transmission equipment

• Needs lower-cost optical transmission equipment– Find ELH/ULH optical kit partner

The classic ‘buy vs. build’ decision in Information Technology

• Option 1 selected for TeraGrid and Next Gen Abilene

8 May 2002 29

National Light Rail

Project objectives• form lightweight, but highly coordinated, collaboration to provision,

acquire, and/or operate optical networking assets and services• leverage collective buying power and experience of the consortium

(ANL, CENIC, P/NW, UCAID) from the metropolitan to the national scales

• serve as optical infrastructure substrate for e-science projects proposing to a diverse array of funding agencies

• facilitate advanced network measurement and academic research

Initial collaboration• TeraGrid (Argonne), UCAID, CENIC and P/NW GigaPoPs• UCSD, UIC

8 May 2002 30

National Light Rail – an evolvingview

Key Functions brokerage service using established relationships with multiple facilities-based carriers

• Ongoing evaluation of potential acquisition and operation of national fiber optical network facility in partnership with the corporate sector

www.internet2.edu