TeraPathsTeraPaths: Establishing End-to-End QoS Paths through L2 and L3

WAN Connections

Presented byPresented by

Dimitrios Katramatos, BNLDimitrios Katramatos, BNL

2

Outline

The TeraPaths projectThe TeraPaths project Motivation

Concept and implementation

View of the world (network)

Interoperating with WAN servicesInteroperating with WAN services

L2 vs. L3L2 vs. L3

What is required from the site LAN?What is required from the site LAN?

Status/futureStatus/future

3

Motivation

The problem: The problem: support efficient/reliable/predictable peta-scale support efficient/reliable/predictable peta-scale

data movement in modern high-speed networksdata movement in modern high-speed networks Capacity is not limitless

Multiple data flows with varying priority

Default “best effort” network behavior can cause performance and

service disruption problems

Solution:Solution: enhance network functionality with QoS features to enhance network functionality with QoS features to

allow prioritization and protection of data flowsallow prioritization and protection of data flows Classify traffic

Schedule network usage

4

Prioritized vs. Best Effort Traffic

5

TheTeraPathsTeraPaths Service: Reserve End-to-End Paths with Guaranteed Bandwidth

WAN

WAN web services

TeraPaths

TeraPaths1

2

3

6

Data Flow Information

Owner info (user)Owner info (user)

Data flow IDData flow ID Source IP and port Destination IP and port IPs and ports can be ranges (multiple flows) Direction (unidirectional/bidirectional) Protocol

Bandwidth (class of service)Bandwidth (class of service) Multiple flows will share (best effort within the class)

Start time and durationStart time and duration Minute resolution

7

Path Setup

Participating end site subnets are controlled by TeraPaths software Participating end site subnets are controlled by TeraPaths software

instances (TeraPaths Domain Controllers or TDCs)instances (TeraPaths Domain Controllers or TDCs) TDCs configure end site LANs to prioritize authorized flows via the DiffServ

framework at the network device level

Source site polices/marks authorized flow packets

Destination site admits/re-polices/re-marks packets

End site LANs hand over/receive marked packets to/from the WAN

WAN provides MPLS tunnels or dynamic circuitsWAN provides MPLS tunnels or dynamic circuits Initiating TDC requests MPLS tunnel or dynamic circuit with matching

bandwidth and lifetime, or…

TDC funnels several flows into MPLS tunnel or dynamic circuit with

aggregate bandwidth and lifetime

WAN preserves packet markings

8

Path Setup (ii)

WAN domains must interoperateWAN domains must interoperate Each end site’s TDC has a single point of contact for WAN services

TDCs have no knowledge of WAN internals other than what is

exposed by the WAN services End sites have no direct control over the WAN

Either tunnel or circuit through WANEither tunnel or circuit through WAN Cannot mix and match

9

Conceptual View of the Network

TeraPaths

TeraPaths

TeraPaths

TeraPaths

Site A

Site B

Site C

Site D

WAN 1

WAN 2

WAN 3

service invocation

data flow

peering

WAN chain

10

TeraPaths Testbed ( )

current

US ATLAS T2 sites

11

TeraPathsTeraPaths Web Services Architecture

Internal Services

Public Services

Web Interface

Admin Module

NDCNDCNDC • • •

Database

protected network

API

remote

local

WAN Services

• • •

WAN Services

• •

•

proxy

proxy

• •

•

CLI s/w client

12

Interoperating with WAN Services

TeraPaths “proxy” serversTeraPaths “proxy” servers Implement interface required by TeraPaths core

Hide WAN service differences

Clients to WAN web services (OSCARS and DRAGON) Close cooperation with ESnet and I2 development teams

Submit reservations for MPLS tunnels or dynamic circuits

Handle security requirements

Handle errors

MPLS tunnels vs. dynamic circuitsMPLS tunnels vs. dynamic circuits Utilization requires drastically different approach

13

L2 vs. L3 (i)

MPLS tunnel starts and ends within WAN domainMPLS tunnel starts and ends within WAN domain Packets are admitted into the tunnel based on flow ID information

(IPsrc, portsrc, IPdst, portdst)

WAN admission performed at the first router of the tunnel (ingress)

WANborder routerborder router

MPLS tunnel ingress/egress

router

MPLS tunnel ingress/egress

router

14

L2 vs. L3 (ii)

Dynamic circuit appears as VLAN connecting end site Dynamic circuit appears as VLAN connecting end site

border routers with single hopborder routers with single hop Cannot use flow ID data directly

Flow must be directed to the proper VLAN

WAN admission performed within end site LAN

Select VLAN with Policy Based Routing (PBR)

WANswitch switch

border routerborder router

15

Site LAN Setup (DiffServ)

16

Site LAN Setup (DiffServ w/pass-thru)

17

3rd Party WAN Segments

Some WAN segments may not be Some WAN segments may not be

automatically configurableautomatically configurable

Static configuration allows DSCP Static configuration allows DSCP

bits to go throughbits to go through Only allow specific interfaces

ACLs and aggregate policers

18

L2-Specific Issues

Limitations with VLANsLimitations with VLANs Tag range - tentatively selected 3550-3599 (50 VLANs)

Tag conflicts - eliminate by synchronizing site databases

Scalability problemsScalability problems Flow grouping

Logistics

PBR overhead Virtual border router

Sensitive/3Sensitive/3rdrd party network segments party network segments VLAN pass-thru

19

Additional Setup for L2

20

Summary

TeraPaths stitches together virtual paths with guaranteed TeraPaths stitches together virtual paths with guaranteed bandwidth…bandwidth… through end-site LANs (direct control)… and end-site interconnecting WANs (indirectly, automatically)… from end host to end host

TeraPaths…TeraPaths… utilizes DiffServ for LAN QoS… makes arrangements for WAN MPLS tunnels or dynamic circuits by

interfacing with WAN (web) services… schedules bandwidth usage with advance reservations… utilizes “pass-thru” techniques for sensitive or 3rd party network

segments

21

Status and Future

Currently: basic software ready, infrastructure testedCurrently: basic software ready, infrastructure tested API and web interface, simple negotiation Statically allocated bandwidth classes L3 paths (MPLS tunnels) through ESnet Elementary AAA BNL UMich

In the works, futureIn the works, future Testbed expansion to US ATLAS Tier 2 sites Utilization of L2 paths (dynamic circuits) through ESnet and Internet2 Dynamic bandwidth allocation within service classes CLI, extended API, configurable negotiation Grid-style AAA (GUMS/VOMS) Admin module to facilitate end site LAN setup

http://www.racf.bnl.gov/terapathshttp://www.racf.bnl.gov/terapaths