lambdastation phil demar don petravick nesc oct. 7, 2004
TRANSCRIPT
LambdaStation LambdaStation
Phil DeMar Don Petravick
NeSCOct. 7, 2004
10.07/2004 Demar, Petravick: Lambda Station 2f
Fermilab
LambdaStation Overview
• The Problem Space
• Functional Description of LambdaStation
• Components of LambdaStation
• Technical Approach
• Case Study
• Test Environment
• Project Outline
• Current Status
• Summary
10.07/2004 Demar, Petravick: Lambda Station 3f
Fermilab
LambdaStation The Problem
• Very high (1-10 Gb/s) bandwidth research networks becoming available Typically not intended for general network traffic
Alternate paths for high impact data movement
Vision of the future: optical networks & lots of λ’s
• Not integrated with facility production LANs Production systems & storage facilities not directly
accessible Customized local access usually required…
WAN paths difficult to effectively use
Used mostly with prototypes or in demonstrations
10.07/2004 Demar, Petravick: Lambda Station 4f
Fermilab
LambdaStation
• Proposed implementations don’t scale Multi-homed end system model breaks down with
multiple alternate paths Graceful failover capability missing or very complex
Virtual interface model difficult to administer & adds complexity on host systems
• Applications may need customization for each new technology or high-performance path
• A last mile problem between advanced WAN & facility production LANs…
The Problem (II)
10.07/2004 Demar, Petravick: Lambda Station 5f
Fermilab
LambdaStation
Proposed Solution:
LambdaStation
• LambdaStation: an alternate path selection service Agent to coordinate use of alternate WAN paths: Dynamic configuration adjuster of local network devices Goal is per-flow granularity for alternate path selection
• LambdaStation is a research project A 2-3 yr US Dept of Energy, Office of Science Fermilab and Caltech In very early stages of development
Still in design phase Production network use is a ways off
10.07/2004 Demar, Petravick: Lambda Station 6f
Fermilab
LambdaStation
How LambdaStation
Would Help• Advanced research networks
Offer alternate, high bandwidth WAN paths
May be a scheduled resource Typically demands customized
local access
• LambdaStation intended to: Facilitates application request for
high bandwidth WAN path Coordinates alternate WAN path
establishment Dynamically reconfigures LAN
forwarding of application’s dataLocal NetworkInfrastructure
Local NetworkInfrastructure
Internet
LambdaStationLambdaStation
Sto
rag
e s
yste
ms S
torag
e syste
ms
AdvancedResearch Network
Local NetworkInfrastructure
Local NetworkInfrastructure
Sto
rage system
sSto
rag
e sy
stem
s
Internet
AdvancedResearch Network
10.07/2004 Demar, Petravick: Lambda Station 7f
Fermilab
LambdaStation
Components of LambdaStation
• A network path controller with modules interfacing to: Advanced WANs Local network infrastructure Peer LambdaStations Local applications &
middleware AA infrastructure
• Software on host systems to coordinate with LambdaStation
LambdaStation
Local networkequipment
ResourcePredictiveService
Advancednetwork
scheduling&
provisioning
VirtualOrganization(s)
Peerlambdastation
Layer 4: TCP, UDP, advanced stacks
Storage System(s)
Layer 5:direct fileaccess
Layer 5:filetransport
Layer 5:management(SRM)
Networkawareness
module
Middleware & Applications
Networkawareness
module
WANSpace
LocalNetworkSpace
Localnetwork
operatingenvironment
Storage &Application Space
local demand module(s)
local steering service
WAN module
VO module
10.07/2004 Demar, Petravick: Lambda Station 8f
Fermilab
LambdaStation
Host Interactions
• Path request primitives: Flow identification:
Source/dest address/port DSCP code point Possibly IP version
Transfer characteristics Bytes, peak rate, duration Schedule (range)
• Path status primitives: Allocated bandwidth Path setup status Path termination status
Normal or early abort…
LambdaStation
Local networkequipment
ResourcePredictiveService
Advancednetwork
scheduling&
provisioning
VirtualOrganization(s)
Peerlambdastation
Layer 4: TCP, UDP, advanced stacks
Storage System(s)
Layer 5:direct fileaccess
Layer 5:filetransport
Layer 5:management(SRM)
Networkawareness
module
Middleware & Applications
Networkawareness
module
WANSpace
LocalNetworkSpace
Localnetwork
operatingenvironment
Storage &Application Space
local demand module(s)
local steering service
WAN module
VO module
10.07/2004 Demar, Petravick: Lambda Station 9f
Fermilab
LambdaStation LAN Interactions• LAN device forwarding
reconfigurations Graceful alternate path
setup & teardown
• ACL modifications at site egress point
• Customizable interface to fit site specifics
• Non-objectives: QoS Traffic policing
LambdaStation
Local networkequipment
ResourcePredictiveService
Advancednetwork
scheduling&
provisioning
VirtualOrganization(s)
Peerlambdastation
Layer 4: TCP, UDP, advanced stacks
Storage System(s)
Layer 5:direct fileaccess
Layer 5:filetransport
Layer 5:management(SRM)
Networkawareness
module
Middleware & Applications
Networkawareness
module
WANSpace
LocalNetworkSpace
Localnetwork
operatingenvironment
Storage &Application Space
local demand module(s)
local steering service
WAN module
VO module
10.07/2004 Demar, Petravick: Lambda Station 10f
Fermilab
LambdaStation
WAN Interactions
• Capability for multiple alternate WANs Customizable interface to fit
each WAN Provide abstract view for
applications
• Path reservation Resource availability Scheduling
• Path setup/teardown Graceful…
• Non-objectives: QoS Link efficiency
LambdaStation
Local networkequipment
ResourcePredictiveService
Advancednetwork
scheduling&
provisioning
VirtualOrganization(s)
Peerlambdastation
Layer 4: TCP, UDP, advanced stacks
Storage System(s)
Layer 5:direct fileaccess
Layer 5:filetransport
Layer 5:management(SRM)
Networkawareness
module
Middleware & Applications
Networkawareness
module
WANSpace
LocalNetworkSpace
Localnetwork
operatingenvironment
Storage &Application Space
local demand module(s)
local steering service
WAN module
VO module
10.07/2004 Demar, Petravick: Lambda Station 11f
Fermilab
LambdaStation
Remote Site Interactions
• Establish path symmetry Exchange traffic selectors Coordinate WAN path
setup/teardown Ack LAN path
reconfiguration
• Verify path continuity Implies concurrent
connectivity via reserved & commodity network paths
Inform of: Path termination Traffic termination
LambdaStation
Local networkequipment
ResourcePredictiveService
Advancednetwork
scheduling&
provisioning
VirtualOrganization(s)
Peerlambdastation
Layer 4: TCP, UDP, advanced stacks
Storage System(s)
Layer 5:direct fileaccess
Layer 5:filetransport
Layer 5:management(SRM)
Networkawareness
module
Middleware & Applications
Networkawareness
module
WANSpace
LocalNetworkSpace
Localnetwork
operatingenvironment
Storage &Application Space
local demand module(s)
local steering service
WAN module
VO module
10.07/2004 Demar, Petravick: Lambda Station 12f
Fermilab
LambdaStation
Authorization & Authentication
• Initial AA capability: LambdaStation internal Site-controlled
But some trust-level between sites
• Longer term goal: Utilize VO AA
infrastructure Raises site-level control
issues
LambdaStation
Local networkequipment
ResourcePredictiveService
Advancednetwork
scheduling&
provisioning
VirtualOrganization(s)
Peerlambdastation
Layer 4: TCP, UDP, advanced stacks
Storage System(s)
Layer 5:direct fileaccess
Layer 5:filetransport
Layer 5:management(SRM)
Networkawareness
module
Middleware & Applications
Networkawareness
module
WANSpace
LocalNetworkSpace
Localnetwork
operatingenvironment
Storage &Application Space
local demand module(s)
local steering service
WAN module
AA module
10.07/2004 Demar, Petravick: Lambda Station 13f
Fermilab
LambdaStation
Technical Approach
• Flow identification: Source/destination IP and DSCP code point tag Source/destination IP & Source/destination port #
• Local path forwarding reconfigurations: Based on policy routing ACLs
• Advanced WAN interface model: Site egress/ingress router to alternate path(s) Layer 2 site-to-site connections (vlan) or MPLS paths
• Transparent failover back to production path: Bandwidth limits or QBSS on fall-back traffic
10.07/2004 Demar, Petravick: Lambda Station 14f
Fermilab
LambdaStation Use Case1. FNAL SRM gets request to copy 3.5TB data set to CMS tier-2 site
- Transfer involves 3500 1GB files stored on 50 dCache systems
2. SRM initiates data transfer optimized for production network path3. SRM makes alternate path request to local LambdaStation4. LambdaStation:
- Checks authentication & authorization- Schedules & coordinates establishment WAN path- Coordinates with remote site LambdaStation for symmetrical path- Verifies alternate WAN path continuity- Reconfigures local LAN for selective flow forwarding- Notifies SRM that alternate path is established
5. SRM makes parallelism changes to re-optimize data transfer• SS Potentially shapes traffic to avoid congestion at pipe entrance.
6. Path reservation expires; LambdaStation gracefully tears path down- Remaining data movement traffic reverts to production network path- LambdaStation notifies SRM of alternate path teardown
7. SRM re-optimizes data transfer parallelism to complete the transfer
10.07/2004 Demar, Petravick: Lambda Station 15f
Fermilab
LambdaStation
Test Environment
• LambdaStation Project Sites: FNAL (US-CMS Tier-1 Center) Cal Tech (US-CMS Tier-2 Site) Working to include UCSD as
well (US-CMS Tier-2 Site) CERN CMS Edge Computing
also planned
• LambdaStation Test WANs: US DOE UltraScience Net
Two 10 GE (OC192) lambda’s MPLS tunnels over DOE’s
ESnet Sub-10GE chunk out of ESnet
production network
FNALANL
tSunnyva
leFNAL
StarlightSunnyvale
Cal Tech
UltraScience Net (2 x 10 GE)
UCSD
UltraScience Net
MPLS path over ESnet
(<10 GE)
To CERN
(10
GE)
10.07/2004 Demar, Petravick: Lambda Station 16f
Fermilab
LambdaStation Project OutlineYear 1 LAN integration: Dynamic LAN reconfiguration
WAN integration: Interface to DOE UltraScience Net;Establish alternate path to Cal Tech
Application integration: SRM allocated bandwidth awareness ---------------------- ---------------------------------------------------------------------------- --------------------------------------------------------------------------------------------------------------------------
Year 2 LAN integration: Automated LAN path setup/teardown, & scheduling; Peer LambdaStation coordination
WAN integration: Automated UltraScience Net WAN path setup/teardown & scheduling
Application integration: VO sensitivity; Adapt storage systems to schedule transfers to path avail.
---------------------- ---------------------------------------------------------------------------- --------------------------------------------------------------------------------------------------------------------------
Year 3 LAN integration: Harden to production use quality
WAN integration: Full integration w/ UltraScience Net;
Application integration: Harden to production use quality
10.07/2004 Demar, Petravick: Lambda Station 17f
Fermilab
LambdaStation Current Status• A “Pre-LambdaStation” prototype exists:
Dynamically reconfigures LAN devices to forward select traffic over alternate paths
End system-applied DSCP tags for special-treatment flows
• Prototype work-arounds: Manual input of reservation parameters Static alternate WAN path – no setup required Asymmetric – no reverse path setup
• Results: Alternate traffic path established & torn down cleanly Path reroute & failback unnoticed by application
10.07/2004 Demar, Petravick: Lambda Station 18f
Fermilab
LambdaStation Summary• LambdaStation’s role in data-intensive science:
Dynamically connect production end-systems to advanced high-performance wide-area networks
Bring production-use systems to advanced networks Bring advanced networks to production-use applications
• LambdaStation’s status: Prototyping has demonstrated feasibility of dynamically
reconfiguring local network paths for specific data flows
Much work remains to be done for complete proof-of-concept
It’s a research project…