nlr layer2/layer3 users bof nlr status update albuquerque internet2 joint techs 8 february 2006...
TRANSCRIPT
NLR Layer2/Layer3 Users NLR Layer2/Layer3 Users BOFBOF
NLR status updateNLR status update
Albuquerque Internet2 Joint Albuquerque Internet2 Joint TechsTechs
8 February 20068 February 2006Brent Sweeny, Indiana UniversityBrent Sweeny, Indiana University
Jon-Paul Herron, Indiana UniversityJon-Paul Herron, Indiana UniversityJohn Moore, North Carolina State John Moore, North Carolina State
UniversityUniversity
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 22
AgendaAgenda
1.1. Progress Report Progress Report
2.2. Questions (non how-do-I-connect)Questions (non how-do-I-connect)
3.3. What’s the Layer2/3 connection What’s the Layer2/3 connection process?process?
4.4. Experiments case studiesExperiments case studies
5.5. ““Ask the experts”—questions, special Ask the experts”—questions, special situations, discussion, “how-do-I”, “I situations, discussion, “how-do-I”, “I need…”, etc.need…”, etc.
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 33
NLR Engineering/Support NLR Engineering/Support OrganizationOrganization
A very A very distributed, coordinateddistributed, coordinated organization: organization: Service desk at IndianaService desk at Indiana Layer1 NOC and engineering at CENICLayer1 NOC and engineering at CENIC Layer2/3 NOC and engineering at IndianaLayer2/3 NOC and engineering at Indiana
Also: Measurement, monitoringAlso: Measurement, monitoring Tech mailing lists for Layer2 and Layer3 usersTech mailing lists for Layer2 and Layer3 users Technical documentationTechnical documentation
Experiments support center at North CarolinaExperiments support center at North Carolina
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 44
The Big News about NLR The Big News about NLR Layers 2 and 3:Layers 2 and 3:
It’s working!It’s working! People are connected, and using it!People are connected, and using it!
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 55
Who’s Who—Who’s Who—aboutabout 150 150 institutionsinstitutionsNLR Members NLR Members (for participants see(for participants see http://http://
www.nlr.net/members_participants.htmlwww.nlr.net/members_participants.html))
Corporation for Education Network Initiatives in California (CENIC)
Pacific Northwest GigaPop (PNWGP)Pittsburgh Supercomputing Center and the University of
PittsburghDuke University, representing a coalition of North Carolina
universitiesMid-Atlantic Terascale PartnershipInternet2® (no participants currently) Florida LambdaRail, LLCSouthern Light Rail, Inc.Committee on Institutional Cooperation (CIC)Cornell UniversityLouisiana Board of RegentsOklahoma State Board of RegentsLonestar Education and Research Network (LEARN)University of New Mexico, on behalf of the state of New MexicoUniversity Corporation for Atmospheric Research (UCAR),
representing a coalition of universities and government agencies from Colorado, Wyoming, and Utah
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 66
NLR PacketNet/FrameNet Current Status
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 77
Review:Review:NLR architecture and NLR architecture and
servicesservices
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 88
National LambdaRail designNational LambdaRail design
NLR WaveNet PoP NLR WaveNet & FrameNet PoP NLR WaveNet, FrameNet & PacketNet PoP
NLR owned fiberBAT
SAN
STA
CHISLC
HOU
DAL
SYR
TUL
PENELP
KAN
PHOLAX
ALB
PIT
WDC
OGD
BOI
CLE
ATL
POR
RAL
NYC
SAA
DENSVL
SEA
JAC
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 99
Generic NLR L1, L2 and L3 PoP Layout
CRS-1
Colo
EastWest
NLR demarc
15808
6509
15454
DWDM
1G wave, link or port10G wave, link or port
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 1010
DAL
SYR
TUL
PENELP
PHO
BAT
ALB
HOU
WDC
OGDCLE
NYC
SAA
4
4
JAC4
4
4
44
4
4
4
4
4
4
4
4
4
4
4
44
44
4
44
4 4
SLC4
4
Level3 fiberWilTel fiber
4
4
RAT4
Cisco 15808 terminalCisco 15808 OADMCisco 15454 terminalCisco 15454 OADM
8
4
4
8
STA
CHIKAN
PIT
BOI
CLE
ATL
POR
RAL
DENSVL
SEA
8
8
8
8
8 8
8
8
8
8
88
8
8
88
88
8
8
8
8
8
4
4
LAX 8
8
NLR Layer 1 “WaveNet”NLR Layer 1 “WaveNet”
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 1111
Layer 1 Phase 2 Layer 1 Phase 2 DeploymentDeployment
DAL
SYR
TUL
PENELP
KAN
PHO
BAT
ALB
HOU
WDC
OGDCLE
NYC
SAA
DEN
4
4
JAC4
4
4
44
4
4
4
4
4
4
4
4
4
4
4
44
44
4
44
4 4
SLC4
4
Level3 fiberWilTel fiber
LAX4
4
RAT4
Cisco 15808 terminalCisco 15808 OADMCisco 15454 terminalCisco 15454 OADM
8
4
4
8
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 1212
Layer 1 baselineLayer 1 baseline
Opportunity to connect into lambda fabricOpportunity to connect into lambda fabric Point to pointPoint to point Other endpoint could be anywhereOther endpoint could be anywhere
Early examples:Early examples: HOPIHOPI UltralightUltralight iGRIDiGRID SC05 (Supercomputing)SC05 (Supercomputing)
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 1313
Layer 2 Network Design Layer 2 Network Design “FrameNet”“FrameNet”
HOU
TUL
ELP
KAN
BAT
LAXALB
PIT
WDC
CLE
ATL
RAL
CHINYC
DENSVL
SEA
JAC
10GE wave10GE managed waveYellow sites are done
Cisco 6509 switch
SVL
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 1414
Layer 2 installation statusLayer 2 installation statusAll Switch installations completed:
Sunnyvale Denver Kansas City ChicagoCleveland Pittsburg Raleigh Washington DCJacksonville Atlanta Los Angeles TulsaEl Paso Houston New York City Baton RougeAlbuquerque
Layer2 backbone interconnections status:
All layer2 backbone interconnects are done except:•Tulsa-Kans•Tulsa-Hous (that is, both directions out of Tulsa)•New York City-Wash•New York City-Clev (e.g. both directions out of NYC)•Los Angeles-Sunnyvale
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 1515
Layer 2 service baselineLayer 2 service baseline(see also FrameNet Technical Guide)(see also FrameNet Technical Guide)
1GE Connection into local 65001GE Connection into local 6500 1GE Access to “national exchange fabric”1GE Access to “national exchange fabric” Additional Additional OptionsOptions::
Dedicated point to point Etherrnet, Nx1GEDedicated point to point Etherrnet, Nx1GE Best-effort point to multipoint (no dedicated Best-effort point to multipoint (no dedicated
bw)bw) Soon:Soon:
10GE ports10GE ports Dedicated point-to-multipointDedicated point-to-multipoint
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 1616
GoalGoal
Provide circuit-like options for users Provide circuit-like options for users who can’t use, can’t afford, or don’t who can’t use, can’t afford, or don’t need, a 10G Layer1 wave via point-need, a 10G Layer1 wave via point-to-point layer2 VLANs.to-point layer2 VLANs.
Experiment with large-scale layer2 Experiment with large-scale layer2 capabilities.capabilities.
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 1717
National Exchange FabricNational Exchange Fabric Multipoint, public, best-effort, resilient, permanent, stableMultipoint, public, best-effort, resilient, permanent, stable
Non-dedicated bandwidth Non-dedicated bandwidth NLR allocated addresses, peer with any other member across the layer 2 exchange policy-free.NLR allocated addresses, peer with any other member across the layer 2 exchange policy-free. Possible to have more than one (say by MTU).Possible to have more than one (say by MTU). Ready to go today.Ready to go today. Current participants: Duke, MATP, SLR—several more coming…Current participants: Duke, MATP, SLR—several more coming…
Backup connections to networks such as Abilene, commodity Backup connections to networks such as Abilene, commodity providers, etc.providers, etc.
Point-to-point, private, permanent, stablePoint-to-point, private, permanent, stable Could be best-effort or guaranteed.Could be best-effort or guaranteed. Could be nailed-up or resilient.Could be nailed-up or resilient. Load-balance or leave idle until needed.Load-balance or leave idle until needed. Ready to go today, though we only have pricing for the guaranteed nailed-up case.Ready to go today, though we only have pricing for the guaranteed nailed-up case. Example: NRENExample: NREN
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 1818
To enable a flexible topology for NLR To enable a flexible topology for NLR layer3layer3
Best-effort, private, temporary, experimentalBest-effort, private, temporary, experimental We have 8 layer3 nodes, but the topology between We have 8 layer3 nodes, but the topology between
them can be made much more interesting by creating them can be made much more interesting by creating various connections over the layer2 network.various connections over the layer2 network.
Enables layer3 experimentation.Enables layer3 experimentation.
To provide members with a second path To provide members with a second path into the NLR layer3 networkinto the NLR layer3 network
Point-to-point, private, permanent, experimentalPoint-to-point, private, permanent, experimental Connect to a second node on the layer3 backbone.Connect to a second node on the layer3 backbone. Load-balance or leave idle until needed.Load-balance or leave idle until needed. Included in membership.Included in membership.
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 1919
Temporary connections for special Temporary connections for special projectsprojects
Guaranteed, private, temporary, stableGuaranteed, private, temporary, stable For remote instrumentation, where the member For remote instrumentation, where the member
only has the remote resource reserved for a only has the remote resource reserved for a limited window.limited window.
For conferences, demos, and other special For conferences, demos, and other special events.events.
Provides a low latency/jitter path if needed.Provides a low latency/jitter path if needed. Nailed-up if latency is critical, probably resilient Nailed-up if latency is critical, probably resilient
if not.if not. Could be point-to-point or multipoint.Could be point-to-point or multipoint. Technically, this is possible today, but we have Technically, this is possible today, but we have
no pricing model for it.no pricing model for it.
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 2020
Bootstrapping circuit-like researchBootstrapping circuit-like research Point-to-point, private, guaranteed, Point-to-point, private, guaranteed,
temporary, nailed-up, experimentaltemporary, nailed-up, experimental To enable a researcher to get started while waiting To enable a researcher to get started while waiting
for funding or provisioning of a layer1 circuit.for funding or provisioning of a layer1 circuit. Similar to a special event, but more experimental, Similar to a special event, but more experimental,
a probably a stronger need for it to be nailed-up.a probably a stronger need for it to be nailed-up. Technically, this is possible today, but we have no Technically, this is possible today, but we have no
pricing model for it.pricing model for it.
Provide control plane network for optical Provide control plane network for optical experimentsexperiments
Permanent, resilient, experimentalPermanent, resilient, experimental A topology could be created for the oob A topology could be created for the oob
management network needed for some dynamic management network needed for some dynamic optical networking experiments (GMPLS, etc.)optical networking experiments (GMPLS, etc.)
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 2121
Cluster/Grid LANCluster/Grid LAN Multipoint, private, guaranteed, experimentalMultipoint, private, guaranteed, experimental
To enable remote clusters to appear on the same LAN.To enable remote clusters to appear on the same LAN. It is not known if spanning tree would be wanted.It is not known if spanning tree would be wanted. It could evolve into a more production-like It could evolve into a more production-like
requirement.requirement. Technically, this is possible today, but we have no Technically, this is possible today, but we have no
pricing model for it.pricing model for it.
Experiment directly with Layer 2Experiment directly with Layer 2 Could be of any type (experimental, obviously)Could be of any type (experimental, obviously)
Web-based provisioning, direct user requests, etc.Web-based provisioning, direct user requests, etc. Concern about interaction with more production-like Concern about interaction with more production-like
requirements.requirements.
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 2222
ALBbackhaul
BATbackhaul
RALbackhaul
JACbackhaul
TULbackhaul
PITbackhaul
HOU
LAX
WDC
ATL
CHINYC
DEN
SEA
Cisco CRS-1 router
10GE waveYellow sites are installed
Layer 3 Network Layer 3 Network “PacketNet”“PacketNet”
LAX
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 2323
Layer 3 installation statusLayer 3 installation status
All layer3 router installations are complete.
All interconnections between layer3 backbone routers are complete now except:
•New York City (both directions)
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 2424
Layer3 connections are up Layer3 connections are up to:to:
NLR member sites: MATP UCAR Duke/NC PNW CENIC PSC
Peer networks:Peer networks: CAnetCAnet USGSUSGS TranspacTranspac
Coming peers:ESnetDRENStarLightNREN
Exchange points:StarLight (now)Pacific Wave (signed)(others soon)
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 2525
PacketNet Peering PrinciplesPacketNet Peering Principles
Very simple: “AUP-free”Very simple: “AUP-free”
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 2626
Layer3 Peering DetailsLayer3 Peering DetailsMember ConnectionsMember Connections
• Prefix Lists are used, with no approval process for updatesPrefix Lists are used, with no approval process for updates• Only routes NLR will normally prevent are:Only routes NLR will normally prevent are:
• Bogons and private addresses Bogons and private addresses • Transit of other upstream providers Transit of other upstream providers
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 2727
Layer3 Peering DetailsLayer3 Peering DetailsNon-Member R&E PeersNon-Member R&E Peers
• ASN Lists are usedASN Lists are used• Only routes NLR will normally prevent are:Only routes NLR will normally prevent are:
• Bogons and private addresses Bogons and private addresses • Other than that, NLR will cater its peering to meet the expectations and needs for Other than that, NLR will cater its peering to meet the expectations and needs for
each peereach peer
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 2828
Layer3 service baselineLayer3 service baseline(see also PacketNet Technical Guide)(see also PacketNet Technical Guide)
Each member gets two routed connectionsEach member gets two routed connections ““local” 10GE local” 10GE VLAN backhauled to 2VLAN backhauled to 2ndnd node node
BGP peering with NLR L3 networkBGP peering with NLR L3 network IPv4 unicastIPv4 unicast IPv4 multicast (MBGP/PIM/MSDP)IPv4 multicast (MBGP/PIM/MSDP) IPv6 unicast (multicast later)IPv6 unicast (multicast later) An ‘experimental’ (changeable, changing) An ‘experimental’ (changeable, changing)
L3 networkL3 network
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 2929
Layer 3 coming servicesLayer 3 coming services
Likely eventual logical routersLikely eventual logical routers More 1GE optionsMore 1GE options More 10G optionsMore 10G options Pre-emptable connectionsPre-emptable connections MPLSMPLS More user control—scheduling, testing, More user control—scheduling, testing,
etcetc User access to measurement dataUser access to measurement data
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 3030
NOC webpagesNOC webpagesfor Layer2/Layer3for Layer2/Layer3
(noc.nlr.net)(noc.nlr.net) ToolsTools
ProxyProxy Weathermap (layer2, layer3)Weathermap (layer2, layer3) UtilizationUtilization
Documents, notably:Documents, notably: FrameNet Technical GuideFrameNet Technical Guide PacketNet Technical GuidePacketNet Technical Guide PacketNet BGP CommunitiesPacketNet BGP Communities
NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 3131
NLR User ResourcesNLR User Resources http://www.nlr.nethttp://www.nlr.net http://noc.nlr.nethttp://noc.nlr.net [email protected]@nlr.net [email protected]@nlr.net [email protected]@nlr.net