regional networking update and national network backbone maps

Southeastern Universities Research Association

Regional Networking Updateand

National Network Backbone Maps

SURA SE Networking Summit MeetingFebruary 20, 2007

Atlanta

Don Riley

SURA IT Fellow

NLR Today

NLR Topology, showing also POP-territories(gray areas unserved) & key Federal lab sites

3


NLR Today

NLR Layer-1 Wave Availability Jan 23, 2007NLR Layer-2 Network Status Monday, Jan 23,2007

NLR Layer-3 Network Status Jan 23, 2007

Internet2 Today

Internet2 HOPI Testbed Topology, ~ 2006,showing overlay with NLR Backbone

Current Abilene Backbone Topology Abilene Weather Map – Jan 23, 2007

Abilene Logical Map – January 2007

Internet2 Tomorrow - “NewNet”

to replace Abilene summer 2007

Internet2 Tomorrow - “NewNet”

Internet2 HOPI Testbed Topology

Internet2 DWDM Layer1 - Optical Circuit Services Internet2 Optical Circuit Services

Internet2 IP Services

7

Phase 1Phase 1

PhasePhase22Phase 3Phase 3

Phase 4Phase 4

Rollout

88

DRAGON/HOPI Control PlaneProvisioning Environment

•• GMPLS Multi-layer, Multi-DomainGMPLS Multi-layer, Multi-Domain•• Ethernet Service ProvisioningEthernet Service Provisioning•• Dynamic dedicated VLAN basedDynamic dedicated VLAN based

connectionsconnections

EthernetLayer

Switched WDMOptical Layer

DRAGONMulti-Layer GMPLS Network

HOPIDynamic Ethernet Network

DomainBoundary

GMPLS Provisioned LSPDedicated Ethernet VLAN “Circuit”

GWU CLPK

LA

SEADC

CHI

Static Optical Layer

MCLNARLG DCNE

NY

HOUEthernet Layer

ENNI

IGP-TE IGP-TE

UNIUNI

99

Heterogeneous NetworkTechnologies

Complex End to End Paths

EndSystem

AS 1AS2 AS 3

VLSR

Ethernet SegmentVLSR EstablishedVLAN

Ethernet overWDM

Ethernet overSONET

EndSystem

Ethernet SegmentVLSR EstablishedVLAN

VLSR

Router MPLSLSP

IP Control Plane

IP Control Plane

IP Control Plane

Ethernet

Router

Lambda Switch

SONET Switch

“horizontal” multi-layer adaptations for multi-domain

CA*net4FranceGLORIAD (Russia, China)Korea (Kreonet2)

Japan (SINet)Australia (AARNet)Canada (CA*net4Taiwan (TANet2)Singaren

ATL

DC

MAE-EPAIX-PAEquinixMAE-West

Connecting DOE Labs to the World’s R&E and Commercial Nets:ESnet’s Domestic, Commercial, and International Connectivity (Spring 2006)

Abilene

Abilene

CERN(USLHCnet

CERN+DOE funded)

GÉANT - France, Germany, Italy, UK, etc

NYC

Starlight C

HI-S

L

Ab

ilene

SNV

SDSC

MAXGPoP

SoXGPoP

High Speed International Connection

Commercial and R&E peering points

Abilene High-speed peering points with Abilene

ESnet core hubs IP SDN

CHI

MRENNetherlandsStarTapTaiwan (TANet2)UltraLight

NGIX-W

Australia

SEA

SINet (Japan)Russia (BINP)

AMPATH(S. America)

AMPATHS. America

MA

N L

AN

Ab

ilen

e

ESnet provides:• High-speed peerings with Abilene, CERN,

and US and international R&E networks• Management of the full complement of global

Internet routes (about 180,000 unique IPv4routes) in order to provide DOE scientists richconnectivity to all Internet sites

Australia

Equinix

Equinix

PNWGPoP/PacificWave

NGIX-E

PacificWave

UNM ALB

SNV

USN

USN

11


10-50 Gb/s circuitsProduction IP coreScience Data Network coreMetropolitan Area NetworksInternational connections

MetropolitanArea Rings

Primary DOE Labs

IP core hubs

possible hubs

SDN hubs

Europe(GEANT)

Asia-Pacific

New York

Chi

cago

Washington, DC

Atl

anta

CERN

Seattle

AlbuquerqueAu

s.A

ust

ralia

San Diego

LA

Su

nn

yval

e

Denver

South America(AMPATH)

South America(AMPATH)

Canada(CANARIE)

CERN

Loop offBackbone

Canada(CANARIE)

Europe(GEANT)

SDN Core

IP Core

ESnet Target Architecture: IP Core+Science Data Network Core+Metro Area Rings

ESnet3 today: A National IP Network Built onVarious Circuit Infrastructure

Primary DOE Labs

Backbone hubs

10 Gbps circuitsProduction IP coreNLR coreMetro Area NetworksLab suppliedInternational connections

Canada CERN

Major research andeducation (R&E)network peering points

R&E

ESnet network architecture consists of1) Circuits2) Circuits interconnects

• hubs with routers and switches3) Connected sites4) Connections to other networks

• US R&E, international, and commercial

2700 miles / 4300 km

1200

mile

s / 1

900

km

Europe

Asia-Pacific

New York

Chi

cago

Washington, DC

Atl

anta

CERN

Seattle

AlbuquerqueAu

s.A

ust

ralia

Su

nn

yval

e

South America

Latin America

Canada Russia andChina

Canada CERN

San Diego

R&E

R&E

R&E

R&E

R&E

R&EQwest supplied10Gbps backbone

NLR supplied10Gbps circuits

ESne

t Sc

ienc

e Dat

a Net

work

(SDN) co

re

TWC

SNLL

YUCCA MT

BECHTEL-NV

PNNLLIGO

INEEL

LANL

SNLAAlliedSignal

PANTEX

ARM

KCP

NOAA

OSTI ORAU

SRS

JLAB

PPPLLab DCOffices

MIT

ANL

BNL

FNALAMES

NR

EL

LLNL

GA

DOE-ALB

OSC GTNNNSA

International (high speed)10 Gb/s SDN core10G/s IP core2.5 Gb/s IP coreMAN rings (≥ 10 G/s)Lab supplied linksOC12 ATM (622 Mb/s)OC12 / GigEthernetOC3 (155 Mb/s)45 Mb/s and less

Office Of Science Sponsored (22)NNSA Sponsored (12)Joint Sponsored (3)

Other Sponsored (NSF LIGO, NOAA)Laboratory Sponsored (6)

42 end user sites

ESnet IP core

SINet (Japan)Russia (BINP)CA*net4

FranceGLORIAD (Russia, China)Korea (Kreonet2

Japan (SINet)Australia (AARNet)Canada (CA*net4Taiwan (TANet2)Singaren

ESnet IP core: Packet overSONET Optical Ring and Hubs

ELP

ATL

DC

commercial and R&E peering points

MAE-E

PAIX-PAEquinix, etc.

PN

WG

Po

P/

PA

cifi

cWav

e

ESnet3 Layer 2 Architecture Provides Global High-Speed Internet Connectivityfor DOE Facilities and Collaborators (spring, 2006)

ESnet core hubs IP

Abilene high-speed peering points with Internet2/Abilene

Abilene

CERN(USLHCnet

CERN+DOE funded)

GÉANT - France, Germany, Italy, UK, etc

NYC

Starlight

Chi NAP

SNV

Ab

ilene

SNV SDN

JGI

LBNL

SLACNERSC

SDSC

Equinix

MA

N L

AN

Ab

ilen

e

MAXGPoP

SNV

ALB

ORNL

CHI

MRENNetherlandsStarTapTaiwan (TANet2, ASCC)

NA

SA

Am

es

AU

AU

SEA

CH

I-SL

14


LHC Tier 0, 1, and 2 Connectivity Requirements Summary

Denver

Su

nn

yval

e

LA

KC

Dallas

Albuq.

CE

RN

-1G

ÉA

NT

-1G

ÉA

NT

-2C

ER

N-2

Tier 1 Centers

ESnet IP core hubs

ESnet SDN/NLR hubs

Cross connects with Internet2/Abilene

CE

RN

-3

Abilene/GigaPoP nodes

USLHC nodes

ESnetSDN

Abilene / GigapopFootprint

Seattle

FNAL(CMS T1)

BNL(Atlas T1)

New York

Wash DC

Jacksonville

Boise

San DiegoAtlanta

Vancouver

Toronto

Tier 2 Sites

Chicago

ESnetIP Core

TRIUMF(Atlas T1,Canada)

CANARIE

GÉANT

USLHCNet

Virtual Circuits

• Direct connectivity T0-T1-T2

• USLHCNet to ESnet to Abilene

• Backup connectivity

• SDN, GLIF, VCs

15


Esnet4 Plan for 2012 Build-out

16


Quilt Fiber-based RON Map

17


AT&T NexGen Routes

Use of SURA Donated AT&T Fiber

International: Atlantic Wave

19

It must be real - we have a website….

20

And we’ve connected all the International PeeringPoints on the East Coast, but also in North America

ToSao Paolo

21

And with the Latin American backbone, CLARA, weactually have completed a “Ring of the Americas”

22

And we did a demo (and what a demo….):SC06 Demos used AtlanticWave

The following demos used the Atlanticwave circuit at SC06:• DRAGON, Dynamic Resource Allocation via GMPLS Optical

Networks• Nortel - VMT: Enhanced Virtual Machine Testbed Demo• Caltech - High Speed Data Gathering, Distribution and Analysis

for Physics Discoveries at the Large Hadron Collider• International High Performance Digital Media With Dynamic

Optical Multicast• Teraflow Testbed o SDSS Data Distribution• Goddard Space Flight Center’s High End Computer Networks

Team participated on four of these demo/testbeds

23

A-Wave Made a Difference at SC06:1 of 10 10G Waves Available for Demos

A-Wave was heavilyused for majority of

SC06

24

SC06 Bandwidth Utilization“Weathermap”

25


SURA Region and RONs:Good news, bad news

• Good News– 7 of NLR Members in

Region - 7 NLR Nodes– Strong Internet2 Gigapops in

region– A number of strong RONS

evolving– International exchange points

and access– Well represented on national

scene– Strong base of technical

expertise in region now

• Bad News– Still several states without

RONs, without NLR access

– Under Internet2 “NewNet”plans, access to I2 backbonemay be more difficult,expensive

– Correlation to state economicbase - disadvantagedstill/again

26


SURA Region Fiber-based RON Map

27


Internet2 SEGP (K-20 nets) Map

28


EPSCOR States

29


Good News, Maybe: Tools for RON andState Network Development

• FiberCo– Prroviding means and tools

for the acquisition of darkfiber

– Also provides professionalservices to assist inmanaging and operatingoptical and networkinginfrastructure investments

– Level 3 footprint (long-haul,metro) now includes WilTeland Progress Telecom;acquiring Telcove, ICG,LookingGlass

– [email protected]

• WaveCo– Will provide access to long-

term dedicated wavelengths(OC-3 to 10G) on extensivenationwide and metropolitanfootprint covering more than33,000 miles of optical fiber.

– Level 3 footprint (long-haul,metro) now includes WilTeland Progress Telecom;acquiring Telcove, ICG,LookingGlass

– [email protected]

30


Level3/Wiltel/Progress Fiber Footprint

31


Level3/Wiltel/Progress SURA Fiber Footprint

32


Telcove

33


ProgressTelecom

34


Looking Glass - Metro Fiber

35


Road Tour: State/RON updates

36


Delaware

37


Maryland-DC-Virginia

McLean LVL3

UVA

VT

E-Lite

ODU

W&MJeffLab

VCU

Ring completesvia Pittsburgh

Ring completesvia Durham

VRoanoke

VArlington

VRichmond

VNorfolk

VLynchburg

VCharlottesville

VAshburn

GMU

MAXDC metro

WDM

GWUGU UMD

VMATP OSR

NLR

CLPK

Baltimore

Abilene

ESnet

NREN NISN DREN

NASA

BoSSnet

JHU

NGIX

ATDnet

MAX T640

NetworkVirginia

MAX Members & Projects

MATP Members & Resources

V MATP VORTEX DWDM Node

OC-48 POS

10 GE

KEY:

Combined Participants (partial)

Primary Members (MAX and MATP)GMU GUGWU JeffLabs/SURAODU NASAUMD UVAVCU VTW&M

Selected Connectors of 39 ConnectorsLibrary of Congress NASA GSFCNat. Archives & Records Ad. NSFNat. Inst. Of Health ATDnetNat. Library of Medicine USGSNOAA USDANaval Research Lab US CensusHoward Hughes Med. Inst. JHUSmithsonian Institution ISI EastUCAID World BankUniv. System of Maryland (11 campuses)NetworkVirginia (651 campuses)

DRAGON

NASA

HOPI

UMB

UMBC GEANT

MAX – MATP Combined Regional Infrastructure

VT BlacksburgCampus

National LambdaRail

Internet2 Abilene / HOPI

Mid Atlantic Crossroads

NLR / MCNCRaleigh/Durham

NLR / SOX Atlanta

Mid-Atlantic Broadband Coop

MATP Washington DC

VORTEX 10 Gbps LANPHY(1 wave lit, 3 waves dark)

MBC Private DWDM System(Infinera DWDM on XO Fiberunder development, Fall 06)

VT Private Dark Fiber/DWDM Program with MBC

40


Jefferson Laboratory Connectivity

Eastern LITE

(E-LITE)

Old DominionUniversity

W&MJLAB

JTASC

VMASC

Bute StCO

MATPNYC

Atlanta

ODU

NASA

Lovitt

MATPVirginiaTech

MAXGIGAPOP

ESnet Router

10GEOC192OC48

ES

net c

ore

JLAB SiteSwitch

DC - MAXGiga-POP

T320T320

41


The DRAGON Testbed

42


DRAGON-HOPI SC05 Demo:Intercontinental eVLBI Collaboration

4343

DRAGON Application Specific Topologies

•• Identify endpoints, control plane sets up topologiesIdentify endpoints, control plane sets up topologies

•• Set up global multi-link topologiesSet up global multi-link topologies

•• ~30 seconds~30 seconds

44


NorthCarolina

Abilene

Research Triangle

Internet

Internet

Internet

Morehead City

WinstonNet

Greensboro

Charlotte

WinstonNetAccess Router

Winston-Salem RTP/Raleigh

Wilmington

GreenvilleAsheville

North Carolina Research & Education Network

OC48 (2.5 Gigabits/second)

Gigabit Ethernet

OC12 (622 Megabits/second)

smj Apr'05

UNC-C

ASU

UNC-A

WCU

UNC-G ECSU

FSU

UNC-P

NCA&T

UNC TV

Duke

NCSUCentennial

UNC-CH

UNC GA

MCNCNCCU

NCSU

WSSU

NCSAWFU

WFU-SM

Multi-Gigabit Research ApplicationsNorth Carolina

Distributed Visualization Power Walls

Interactive HDTV & Petabyte Storage10 Gig NLR Links

Level 3 POP

Atlanta

WashingtonDC

UNCRENCI

UNCA

DUKENCSU

ECU

MCNC

46

Testbed Connectivity DiagramNational footprint that consists of four Calient GMPLS-Enabled MEMs switches

NSF Project PartnersOptIPuter, UltraLight, WAN-in-LAB,DRAGON, Cheetah

To Canada

EnLIGHTened wave(Cisco/NLR)

LONI waveCisco/NLR

CHI

HOU

DAL

TUL

KAN

PIT

WDC

OGD

BOI

CLE

POR

DEN

SVL

SEA

To AsiaTo Europe

San Diego

NLR FrameNet

VCL@NCSU

Official Partners:Members:- MCNC GCNS- LSU CCT- NCSU- RENCI

International PartnersG-Lambda in JapanPHOSPHORUS is Europe

47


South Carolina Conceptual Slidefrom Jul06

48


South Carolina LightRail•South Carolina LightRail is a new initiative that is evolving.

•The driver of SC LightRail is Health Sciences South Carolina (HSSC), a501c3 collaborative research organization whose 6 members are Clemson, USC,MUSC, Palmetto Health, Greenville Hospital System, and Spartanburg RegionalHealth. Clemson, USC and MUSC are research universities. MUSC’s hospitalis also one of the four largest in SC, along with Palmetto, Greenville andSpartanburg.

•HSSC engaged the services of a consultant in November, 2006, to identify fiberassets, begin negotiations with vendors, and develop the architectural framework.His work has been successful and is ongoing.

•HSSC believes that financial and political support necessary to proceed toimplementation will be in place during the current session of the SC GeneralAssembly, which generally concludes its deliberations in June.

•Other major partners will be invited to join SC LightRail upon successfulimplementation of the initial backbone connecting HSSC partners.

49

Georgia: Southern LightRail (SLR) – Atlanta

Connectivity

• Telx – 56 Marietta– DWDM– Ethernet Switch

• Level 3 – 345 Courtland– DWDM– NLR Access

• WaveNet• FrameNet• PacketNet

• Georgia Tech– DWDM– Ethernet Switch

50

SLR – SoXRegional

ConnectionsChicagoPittsburgh

Atlanta

Baton Rouge

Jacksonville

NLR BACKBONE- FLR- Atlantic Wave- PSC- NCREN

Univ Sys of AL BackboneNASA MSFC

ORNL BackboneTennesseeMississippi

South Carolina

Raleigh

Wash DC

NYC

STRK

OXFD

KNOX

StarLight- Imperial College-London- LONI- GLORIAD- MREN

TO FLR

TUSCBHAM

HUNT

CHL

COLCLM

Atlantic WaveExchange Points

51

Southern Crossroads GigaPopSoX Participants:AlabamaUniv. of Alabama – Birmingham, Univ. of Alabama –Huntsville, Univ. of Alabama – Tuscaloosa, Univ. of Auburn,Alabama Supercomputer Authority (SEGP)FloridaFlorida LambdaRail - State AggregatorGeorgiaGeorgia Institute of Technology, Georgia State Univ., EmoryUniv., Univ. of Georgia, Georgia SEGP-BOR/PeachNet,Morehouse School of MedicineKentuckyUniv. of Kentucky, Univ. of Kentucky SEGPNorth CarolinaNorth Carolina MCNC - State AggregatorSouth CarolinaUniv. of South Carolina, Medical Univ. of South Carolina,Clemson Univ., SCINET (South Carolina SEGP)TennesseeUniv. of Tennessee, Vanderbilt Univ., Oak Ridge National Lab,Tennessee SEGP

SoX Services:• PacketNet Access• Internet2 Access• Commodity Internet Services• Redundancy/Backup Services• Multiple Peerings

52


Florida - Florida LambdaRail

53


Florida - FLR Optical Network

54


Florida - FLR Layer2

55


FIU/AMPATH - Latin America Connectivity(under NSF IRNC program)

56

AtlanticWave

• AtlanticWave is provisioning a 10GigE wave to support a distributedinternational exchange and peering fabric along the Atlantic coast of Northand South America, following the GLIF GOLE model

• AtlanticWave will connect the key exchange points on the U.S. East Coast:– International Exchange Points MANLAN in NYC and AMPATH in Miami– MAX gigapop and NGIX-East in Washington, DC– SoX gigapop in Atlanta

• A-Wave is an integral component of the NSF IRNC WHREN-LILA proposalto create an open distributed exchange and transport service along theAtlantic rim

• A-Wave partners include SURA, FIU-AMPATH, IEEAF, FLR, MAX,SLR/SoX, Internet2/MANLAN, and the Academic Network of Sao Paulo(ANSP)

40

57

Connecting Western-HemisphereInternational Exchange Points

♣ Collaboration withTransLight and CANARIEto extend connectivity toStarLight andPacificWave

♣ International ExchangePoints at Sao Paulo,Miami, Washington DC,NYC, Chicago, Seattle,LA

♣ Exchange and Peeringcapabilities with nationaland internationalnetworks

41

58


Louisiana

Research University

Multiple 10GE

Logical View of Connectivity

59


External Network Connections

Cisco/UltraLight wave

EnLIGIHTened wave(Cisco/NLR)

LONIwave

CHI

HOU

DAL

TUL

KANPIT

WDC

OGD

BOI

CLE

POR

DEN

SVL

SEA

Baton Rouge

Raleigh

To Asia To Canada To Europe

L.A.

San Diego

CAVE wave

Chicago

60


Texas - Learn

Midland

Odessa

Lubbock

Amarillo

El Paso

Denton

Fort Worth Dallas

Longview

Tyler

NacogdochesWaco

Austin

San Marcos

San Antonio

Laredo

Edinburg

Harlingen

Brownsville

Kingsville

Corpus Christi

Houston

Galveston

Beaumont

College

Station

Waller

Production Fiber

Testing FiberCharacterized awaiting Equipment

Contracted – NOT characterized

NO Contract

LEARN Fiber IRU (solid line )

LEARN Leased Lambda or

Service (dashed line )

Wichita

Falls

NLR

Wavenet

FrameNet

PacketNetNLR

Wavenet

NLR

Wavenet

FrameNet

NLR

Wavenet

Abilene

NewNet

61


Texas - SE Texas Gigapop (SETG)

62


Oklahoma

DarkFiber

63


Arkansas Research & Education Optical Network(ARE_ON)

64

Alabama

65


Mississippi

66


Tennessee

67


Tennessee

68


Kentucky

• State wide Higher Education Network is an MPLS basedmanaged services network with Universities attaching at100mb to 1 Gig ( 2.4 million annually)

• Dark Fiber has proved to be elusive (currently workingwith Insight); lowest proposal to date for Lexington –Louisville is 3,000,000 for 100 miles.

• Currently preparing to deploy a K-20 network withfunding from last legislative session. $8.9 million forcapital purchases & $15.3 million operating

• Desire to join NLR & participate in the SURAgrid project.• Talking to Ohio and Tennessee about developing RON

69


West Virginia

•West Virginia University R&E network connectivity is throughan OC-3 to Abilene, of which WVU purchases 75mb of I2traffic. This OC3 is shared with other federal governmentagencies.

•WVNET, a sister higher ed agency, provides Internet servicesfor WVU, K-20 education facilities and state agencies. Thisgroup also serves as the network communications hub forintrastate agency traffic.

•WVU has a gigabit connection to WVNET, of which we have200mb of commodity internet and 50mb of intrastate traffic.Most other agencies in the state connect to WVNET via leasedlines.

70


What’s next?What can we do together?

How can we help?


The End or The Beginning

regional networking update and national network backbone maps

Documents