TeraPathsTeraPaths: Flow-Based End-to-End QoS Paths through Modern Hybrid WANs

Presented byPresented by

Dimitrios Katramatos, BNLDimitrios Katramatos, BNL

2

Outline

Background: the TeraPaths projectBackground: the TeraPaths project Objective

View of the world (network)

System architecture

Establishing flow-based end-to-end QoS pathsEstablishing flow-based end-to-end QoS paths Domain interoperation

Experience and encountered issues

Project status and future workProject status and future work

3

Objective

Provide QoS guarantees at the individual data flow level, all Provide QoS guarantees at the individual data flow level, all the way to the end hosts, transparentlythe way to the end hosts, transparently Data flows have varying priority/importance

Video streams Critical data Long duration transfers

Default “best effort” network behavior treats all data flows as equal Capacity is not unlimited

Congestion causes bandwidth and latency variations Performance and service disruption problems, unpredictability

Dynamic flow-based SLAs = schedule network utilizationDynamic flow-based SLAs = schedule network utilization Regulate and classify (prioritize) traffic Select routing (if possible)

4

View of the Network

WAN

ctrl

WAN 1

WAN 2

WAN 3

TeraPaths

Domain ctrl

TeraPaths

RN

RN

TeraPaths

WAN

ctrl

WAN

ctrl

Site A

Site B

Site C

Site D

MPLS tunnelDynamic circuitDomain control

5

TeraPathsTeraPaths Web Services Architecture

Domain Controller

DSM

Web Interface

NDCNDCNDC • • •

Database

protected network

API

local

WAN controllers

• • •

Domain controllers(non-TeraPaths)

WAN serviceclients (proxies)

CLI s/w clientWeb browser

NDC database

Domain service clients (proxies)

Site controller

Site service

hardware

“virtualnetwork

engineer”

remote

6

Establishing E2E QoS Paths

Multiple administrative domainsMultiple administrative domains Cooperation, trust, but each maintains full control

Heterogeneous environment

Domain controller coordination through web services

Coordination modelsCoordination models Star

Requires extensive information for all domains

Daisy chain Requires common flexible protocol across all domains

Hybrid (end-sites first) Independent protocols Direct end site negotiation

…

…

…

7

Path Setup

WAN

WAN web services

TeraPaths

TeraPaths1

2

3

8

Path Setup (ii)

End site subnets are configured by TeraPaths software End site subnets are configured by TeraPaths software instances (TeraPaths Domain Controllers or TDCs)instances (TeraPaths Domain Controllers or TDCs) TDCs configure end site LANs to prioritize and regulate authorized

flows via the DiffServ framework at the network device level Source site polices/marks authorized flow packets Destination site admits/re-polices/re-marks packets End site LANs tx/rx marked packets to/from the WAN

WAN provides MPLS tunnels or dynamic circuitsWAN provides MPLS tunnels or dynamic circuits Initiating TDC requests MPLS tunnel or dynamic circuit with

matching bandwidth and lifetime, or… TDC groups flows with common src/dst into MPLS tunnel or

dynamic circuit with aggregate bandwidth and lifetime WAN preserves packet markings

9

Path Setup (iii)

WAN domains interoperateWAN domains interoperate Each end site’s TDC has a single point of contact for WAN services

TDCs have no knowledge of WAN internals other than what is

exposed by the WAN services End sites have no direct control over the WAN

Either tunnel or circuit through WANEither tunnel or circuit through WAN Cannot mix and match

10

Interoperating with WAN Services

TeraPaths “proxy” serversTeraPaths “proxy” servers Implement interface required by TeraPaths core

Hide WAN service differences

Clients to WAN web services (currently OSCARS / DRAGON) Close cooperation with ESnet and I2 development teams

Submit reservations for MPLS tunnels or dynamic circuits

Handle security requirements

Handle errors

MPLS tunnels vs. dynamic circuitsMPLS tunnels vs. dynamic circuits Utilization requires different approach

11

L2 vs. L3 (i)

MPLS tunnel starts and ends within WAN domainMPLS tunnel starts and ends within WAN domain Packets are admitted into the tunnel based on flow ID information

(IPsrc, portsrc, IPdst, portdst)

WAN admission performed at the first router of the tunnel (ingress)

WANborder routerborder router

MPLS tunnel ingress/egress

router

MPLS tunnel ingress/egress

router

12

L2 vs. L3 (ii)

Dynamic circuit appears as VLAN connecting end site Dynamic circuit appears as VLAN connecting end site

border routers with single hopborder routers with single hop Cannot use flow ID data directly

Flow must be directed to the proper VLAN

WAN admission performed within end site LAN

Select VLAN with Policy Based Routing (PBR) at both ends

Route can be selected on a per-flow basisRoute can be selected on a per-flow basis

WANswitch switch

border routerborder router

13

Site LAN Setup (DiffServ, PBR)

PBR

14

3rd Party Network Segments

Some network segments may not be automatically Some network segments may not be automatically configurable configurable Regional providers Campus segments Border routers

Static (once only) configuration required Static (once only) configuration required Allow DSCP bits to go through

Only allow specific interfaces ACLs and aggregate policers

Configure VLANs to be used for dynamic circuits Trunked VLAN pass-thru

Virtual border router

15

Alternative Site LAN Setup (DSCP, VLAN pass-thru)

PBR

VLAN pass-thru

16

VLAN Setup for L2

TeraPaths-controlled“virtual border” router(directs flows w/PBR)

e.g.,1 to X, 2 to Y

WAN Site’sBorderRouter

trunked VLAN pass-thru50 VLAN ids (3550-3599)

3550 X Y 3599interfaces trust DSCP

TeraPaths-controlledhost router

#X

#Y

DSCP-friendly LAN

host 1 host nhost 2 . . .

1 to X

2 to

Ycan be the same device

RegionalProvider’s

Router

17

L2-Specific Issues

Limitations with VLANsLimitations with VLANs Tag range (tentatively selected 50 VLANs – 3550 to 3599)

Each site may have its own range

Tag conflicts Rely on WAN service Eliminate by synchronizing site databases VLAN renaming (if/when possible)

Scalability issuesScalability issues Flow grouping

Forward flows through same virtual WAN circuit Create circuit with new parameters / switch current flows / cancel old circuit Modify WAN reservations (if/when possible)

PBR overhead Virtual border router

Sensitive/3Sensitive/3rdrd party network segments party network segments VLAN pass-thru

18

Status

Currently: basic software ready, infrastructure testedCurrently: basic software ready, infrastructure tested API and web interface, simple negotiation

Multiple service classes per site with statically allocated bandwidth

Utilization of L3 paths (MPLS tunnels) through ESnet (since 2006)

Utilization of L2 paths (dynamic circuits) through ESnet and

Internet2 (demonstrated at SC’07)

“Circle of trust” security model, X.509 certificates

Simple user AAA

BNL, UMich, BU, SLAC

Multiple successful pass-thru configurations (BNL, UMich, NoX,

Merit, MiLR)

19

TeraPaths Testbed during SC’07

US ATLAS T2 sites

BNL

OU

UC/IU UMich BU

SLAC ESnet

UTA

I2

NLR

NoX

StarLight

UltraLight

MiLR/Merit

L2 (dynamic circuit)

L3 (MPLS tunnel)

L2 and L3

20

Weather Map

21

Traffic Regulation (demo)

1

2

2

22

In Progress / Future

TestbedTestbed Expansion to more US ATLAS Tier 2 sites and beyond BNL testbed router upgrade to 10Gbps

Support for different hardwareSupport for different hardware

Dynamic bandwidth allocation within service classesDynamic bandwidth allocation within service classes

Flow grouping through WAN circuits Flow grouping through WAN circuits

CLI, extended API, configurable negotiationCLI, extended API, configurable negotiation

Grid-style AAA (GUMS/VOMS)Grid-style AAA (GUMS/VOMS)

Plug-ins: SRM (dCache), othersPlug-ins: SRM (dCache), others

Expand collaboration/interoperationExpand collaboration/interoperation

http://www.terapaths.orghttp://www.terapaths.org