exterior routing 201 howard c. berkowitz [email protected] [email protected] (703)998-5819 esn...

Exterior Routing 201Exterior Routing 201

Howard C. BerkowitzHoward C. [email protected] [email protected]

[email protected]@clark.net

(703)998-5819 ESN 451-5819(703)998-5819 ESN 451-5819

NANOG 21 Exterior Routing tutorial 2/17/20012

AgendaAgenda

• What's the problem?— Formal and informal clue

— ISP service offerings

• Quirks, Defnitions, and Issues

• ISP External Scenarios

• POP and other infrastructure

• Router requirements

• Playing in the Club

• Turning it On

If there's time...full employment for consultants: path selection

What is the Problem to be What is the Problem to be Solved?Solved?


Good little boys and girlsread RFC1771

and live happily ever after


• Noah.

• Noah.

• (yawn) MMMmmmmhp?

• Noah.

• Yeahh?

• Build an ISP.


ISPs Facing End UserISPs Facing End User

• Entry— Basic Internet Access

— Hosting

— Availability and QoS

— Dealing with specialized access providers (DSL, CATV, etc.)

— Dealing with content providers

— Voice services?

• Improvement for Users– Improving capacity– Improving availability– Adding services– Perceptions of end-to-end

SLA


Before the AnimalsBefore the Animals

Uplinks

Routers

User Hosts

Downlinks

Management

Facilities

HVAC Staff


Load the ArkLoad the Ark

Policies

Traffic

From Downstreams

Policies

Traffic

From Upstreams

From Users

Traffic AAA

Traffic

From Virtual Hosts

Quirks, Definitions and Quirks, Definitions and IssuesIssues


I said "peer," not "peer"I said "peer," not "peer"

• Peer relationship 1— Basic BGP session

• Peer relationship 2— Mulual benefit customers reach one another

— No monetary exchange

— Each advertises customer routes

• Transit Provider relationship— Customer pays for service

— Full routes available to customer


(C) O'Leary Museum and Library Association Ltd. Inc.


Closest Exit RoutingClosest Exit RoutingHot potatoHot potato

• Paths are not optimized end-to-end

• Paths are optimized for each AS

src

dest


Asymmetrical RoutingAsymmetrical Routing

• No guarantee that traffic leaving your AS at one point

• Will return at the same point

• Remember

—Each AS in both directions makes decisions on its information

ISP ISP ScenariosScenarios


Basic Internet Access ISPBasic Internet Access ISP

POP2 POP3

Core

POP1HostedServers

InternalServers

ISP #2 ISP #1

/18 /18 /18

/16/16

8x/23

To 70-90% of customers Default routeTo 5-10% of customers Partial routesTo 10% of customers Full routes

From customersFew # public routes ??? VPN


Bilateral PeeringBilateral Peering

BigISP 1

BigISP 2

eBGP Relationship

Exchange of customer routes only Some aggregation No infrastructure routes

Highest bandwidth requirement

"Tier 1 Provider" Does not buy transit service from anyone Has default-free routers Gets all routes from bilateral/multilateral peering Total RIB size of 1.3-1.5 * DefaultFreeZone (D)


Large Content ProviderLarge Content Provider

10/100

Server

L4 distributionGE ports

Firewall,etc.

Provider

Server Server

L7 Distribution

L3 Path Determination• Sometimes bandwidth limited

• Provider may be default free

• Often high touch processing limited

• Possible SLA and VPN agreements

May participate in content distribution, caching


Layer2

Fabric

ISP 1

ISP 2

ISP 3

ISP 4

ISP 5

ISP 6RouteServer

Multilateral PeeringMultilateral PeeringeBGP Relationships

Depending on exchange rules Exchange of customer routes only Most common case Some aggregation No infrastructure routes Some ISPs buy transit services Can receive full routes Private peerings

Largest carriers tend to avoid due tocongestion

ISPs can peer with route server rather than a mesh of ISPs May be done to reduce BGP peers Or simply for statistics collection


Special Case: Local ExchangesSpecial Case: Local Exchanges

• Entry— Who's in charge?

— Connectivity

— Facilities

— Allow content providers?

— Allow end users?

— Peering model?

— Supplementary services?

• Improvements

Layer2? 3?Fabric

ISP 1

ISP 2

ISP 3

ISP 4

ISP 5

ISP 6RouteServer

POP and Other Internal POP and Other Internal DesignDesign


Typical Basic POP ImplementationTypical Basic POP Implementation

Gigabit Ethernet Frame Interfaces ATM Interfaces

PSTN

LAN Switch

ManagementServers

AccessServer

DialupCustomers

Router Fabric

ISP CoreRouter 1

ISP CoreRouter 2

DedicatedCustomers

CustomerSite Routers

FrameDS3

2x/25

32x/30

DedicatedCustomers

CustomerSite Router

FullDS3

1 per POP25 per POP

450 users per POP

/18 /18


Transit Provider POP, Transit Provider POP, Intra-POPIntra-POP

Design Alternatives 1. POP is a route reflector cluster Core is higher-level cluster

2. Each POP is a private or public AS Full mesh iBGP or route reflectors inside POP Confederation between POPs

3. IGP within POP Controlled redistribution inside POP to BGP Prefer intra-POP of same metric

AccessRouter

AccessRouter

POPRouter

POPRouter


Public AS

POP ConfederationsPOP Confederations

POP AS65000 POP AS65111 POP AS65222


Public AS

POP ReflectorsPOP Reflectors

POP AS65000 POP AS65111 POP AS65222


Open Access/Specialized AccessOpen Access/Specialized Access

Layer1/2

FabricSubscribers

ISP 1

ISP 2

ISP 3InternalRoutedNetwork

TunnelServer

ContentServers


InternalRouting

&Switching

Tunneled AddressingTunneled Addressing

CLE

Data Provider1

Voice Provider1

EnterpriseVPN NAS

CLE

CLE

AccessGateway

Data Provider2

L2TP, Differv High

L2TP, Differv High

VoIP

Access OAM address space

PPPoE or GRE

ISP address space

Data 1Data 2VPN

DHCPDNS

Router Router RequirementsRequirements

Big part of the solution...but not Big part of the solution...but not all.all.


Routing ParadigmsRouting Paradigms

Number of Routes

Forwarding Bandwidth

Hello Processing

Number of Interfaces

Policy Analysis

QoS Awareness

Low High Medium

Low High Medium

Low Medium High

Medium High Medium

End to End EtE & PHB PHB

Low High Low

L4/7 Processing Medium High Low

Enterprise Edge Core


Observations on Routing Table SizeObservations on Routing Table Size

• Global default-free table continues to grow exponentially— 96509 routes as of Tony Bates' CIDR report 2/11/2001

— Let the default routing table size be D

• Large provider often has 1.3 to 1.5 D active routes— additional routes are more-specific customer & internal

— may also have substantial numbers of inactive routes


Growth in Global Routing Table SizeGrowth in Global Routing Table Size

184K

368K

Sep 01

Sep 02

Sep 03

Sep 04

736K

85K public


Growth in Typical Tier 1 Routing Growth in Typical Tier 1 Routing Table SizeTable Size(external + customer, not infrastructure)(external + customer, not infrastructure)

276

552

Sep 01

Sep 02

Sep 03

Sep 04

1104

85K public42K internal


Observation: More than RoutesObservation: More than Routes

• Customer routes

• Paths per route

• Route validity


ConvergenceConvergence

• Global routing system

• Intra-AS

• Single Router


Single Router ConvergenceSingle Router Convergence

• Initialization— Time to add new route

— Time to add better route

— Time to withdraw route

— Time to withdraw and replace route

• Parameters— Matrix: number of peers

versus– Routes advertised– Routes accepted

• Performance Modifiers— Route filtering

— Route flapping

— Packet vs. route filtering

draft-berkowitz-bgpcon-0x.txt


Distinguish among casesDistinguish among cases

• Failover of link or router between customer and provider

• Rerouting to intranet/adjacent provider resources

• Rerouting to arbitrary internet destnation

More multihoming in next tutorial

S-T-R-E-T-C-HS-T-R-E-T-C-H

Joining the Joining the ClubClub


More than Just Addresses, Protocol...More than Just Addresses, Protocol...

Routing Registry

Routeobjects

ASobjects

Maintainerobjects

RoutingSystem

Configs

Customer

DNS

NAT

Hosts

SpecifyPolicy

Route Track Service

ISP with Prefixes

Allocate

Directories

SWIP

ReverseDNS

AddressRegistry

RouteRegistry


ComplexityComplexity

• BGP itself is fairly simple

• Additional attributes it carries are more complex

• Policy actions taken inside router (BGP sender or receiver) far more complex than the protocol itself


"BGP Transmits Policies""BGP Transmits Policies"


OOppeerraattiioonnaall RReellaattiioonnsshhiippss 11

AAddddrreesssseess aanndd DDeelleeggaattiioonn

Addressauthority

Addressdelegation

Prefixes

Hosts

DNS

ReverseDNS


Obtain routable address spaceObtain routable address space

• Apply to registry— RIPE, APNIC, ARIN

— If immediate need for /19 or /20*

• Obtain addresses from upstream ISP— If /19 or /20 cannot be justified

• Registry needs— Network design

— Justification for address space


Origination vs. AdvertisingOrigination vs. Advertising

128.0.0.0/20

/23POP Dialups

/23Internal

/23Customers

/23Customers

/25/25/25/2532*

/30

32*

/28/24 /24/25/25

AS 65000

192.0.0.0/16AS64444

an AS65000Customer

AS65000

128.0.0.0/19

AS65000

192.0.0.0/16

AS64444


Aggregating your Own TrafficAggregating your Own Traffic

128.0.0.0/20

/23POP Dialups

/23Internal

/23Customers

/23Customers

/25/25/25/2532*

/30

32*

/28/24 /24/25/25

AS65000

128.0.0.0/19

Suppress more specific routesunless required by multihoming


Advertising with NO-EXPORTAdvertising with NO-EXPORT

AS6333364.0.0.0/12

Assigns64.0.0.0/22

Assigns64.0.4.0/22

AS62222 AS61111

AS6100096.1.0.0/16

Advertises64.0.0.0/22 NO-EXPORT

Advertises64.0.4.0/22 NO-EXPORT96.1.0.0/16


Aggregation is better than Aggregation is better than AggravationAggravation

• Blackhole routes for your blocks— Avoid more-specifics— Use NO-EXPORT when controlling load to upstream

• Encourage customers to aggregate— Proxy aggregation hard to administer

• Understand which blocks you can advertise— And do ingress/egress filtering


Preparing for Address Request (1)Preparing for Address Request (1)

• Address requirements of services are you offering

• Dynamic addressing— Dialup

— Residential broadband

• Private addressing— Enterprises homed only to you

— Dialup/broadband not offering servers

• Globally addressable


Prepare for Address Request (2) Prepare for Address Request (2) An ISP TopologyAn ISP Topology

POP11 internal LAN

100Dial

Ports

8smallLANs

1med.LAN

POP21 internal LAN

100Dial

Ports

8smallLANs

1med.LAN

POP31 internal LAN

100Dial

Ports

8smallLANs

1med.LAN

POP41 internal LAN

100Dial

Ports

8smallLANs

1med.LAN

CoreRouter 1

CoreRouter 2

Hosting Farm 1 Hosting Farm 2InfrastructureServers

Switch


EEssttaabblliisshhiinngg aann AASS ((11))

AASS NNuummbbeerr RReeqquueesstt

• In request to AS number registry— Administrative and technical contacts

— Autonomous system name

— Router description

— Deployment schedule

— Networks (by name) connected by the router(s)

— Internet addresses of the routers


Establishing an AS (2)Establishing an AS (2)Registering in Routing RegistryRegistering in Routing Registry

• Minimum requirements— Maintainer object

— AS object

— Route object (s)


Establishing an AS (3)Establishing an AS (3)Operational deploymentOperational deployment

• Build configuration— Policy implementation

— Ingress/egress filtering

• Establish security procedures

• Start BGP connections


RRoouuttiinngg RReeggiissttrryy OObbjjeeccttss

• Basic— AS

— Route

— Maintainer

• Additional— Inter-AS Network

— Community

— Router

Refinements


OOppeerraattiioonnaall RReellaattiioonnsshhiippss 33::

RReeggiissttrriieess,, DDoommaaiinnss,, eettcc..

Addressauthority

Addressdelegation

Prefixes

Hosts

Routeobjects

ASDNS

ReverseDNS


Autonomous SystemAutonomous System

• Basis of exterior routing

• AS originate routes for some prefixes they want to be visible

• AS advertise routes to one another— Advertisement may not contain all addresses

— Not all advertisements need be accepted


Current AS DefinitionCurrent AS DefinitionRFC 1930RFC 1930

• Connected group of IP CIDR blocks

• Run by one or more network operators

• Single routing policy — announced to the general Internet

— announced with BGP-4


AASS NNuummbbeerr

• 16 bit number— 32 bit under discussion

• Numbers assigned by registries— Routing policy should be stored in registry

— ISPs can mirror routing registry -- place for sensitive data

• Private ASNs— 64512 through 65535

— Private AS stripping, confederations


OOppeerraattiioonnaall RReellaattiioonnsshhiippss 22::

AAddddeesssseess aanndd AAuuttoonnoommoouuss SSyysstteemmss

Addressauthority

Addressdelegation

Prefixes

Hosts

ASDNS

ReverseDNS


Full Employment for Consultants: Full Employment for Consultants: Policies are Policies are insideinside Routers Routers

• Advertising Policies— Outbound to other AS

— BGP advertisement sources

— Outbound route filters

— Route must be in internal routing table

• Acceptance Policies— Inbound AS filters

— Inbound route filters


Stop! Stop! What are you going to What are you going to

Advertise?Advertise?•Routes Assigned/Allocated to You

•Routes Assigned/Allocated to Customers

•Routes for which you provide Transit


Advertising AffectsAdvertising Affects

• The way the world sees you/sends to you

• Binary— Routes to which you provide routing

• Quantitative Preferences— Multi-Exit Discriminators to your Neighbors

— AS Path Manipulation to all


Routes Eligible to AdvertiseRoutes Eligible to Advertise

• Are reachable by your IGPor static routes

• Unless they are black holes— Which conceptiually are reachable

• Do not advertise— Spoofed source addresses

— Your internal addresses

— RFC1918 space

— Known rogues?– RBL?


Stop! Stop! What are you going to What are you going to

Accept?Accept?

•It depends

•Only those routes you will do something about

•Otherwise default


Do Not AcceptDo Not Accept

• RFC1918 source or destination

• Unexpected sources not assigned/allocated to peers

• Your internal addresses from peers

Turning it Turning it OnOn


BBGGPP CCoonnffiigguurraattiioonn OOvveerrvviieeww

• Plans and policies first!

• Define system of BGP speakers

• Specific BGP speaker configuration— Identifier

— BGP process

— Neighbors

— NLRI to advertise

— Filters and other policy mechanisms

Cisco commands used as examples


Policy Implementation FlowPolicy Implementation Flow

MainBGPRIB

AdjRIBOut

Outgo-ing

AdjRIBIn

Incom-ing

MainRIB/FIB

IGPs

Static&

HWInfo


AS1R1

AS21R1

AS1R1

AS21R1

AS1R1

AS21R1

AS1R2

AS21R2

All equivalent from a policy standpoint!

Policy vs. Protocol FlowPolicy vs. Protocol Flow


BGP ConfigurationsBGP Configurations

• Know global information (AS, policies, etc.)

• Establish router ID

• Create BGP process

• Identify internal and external peers


RRoouutteerr IIDD aanndd llooooppbbaacckk iinntteerrffaaccee

interface loopback 0ip address 192.168.0.1 255.255.255.0

RReeffiinniinngg tthhee CCoonnffiigguurraattiioonn

Single and Multiple Links Single and Multiple Links

to a Single Providerto a Single Provider


TThhee BBGGPP TTuunnnneell

Serial 0 Serial 0

Serial 1 Serial 1

Loop 0 Loop 0

ebgp-multihop needed whenneighbor is not on same subnet


LLooaadd BBaallaanncciinngg 11::

IIPP LLeevveell ttoo SSiinnggllee PPrroovviiddeerr RRoouutteerr

Serial 0 Serial 0

Serial 1 Serial 1

Loop 0 Loop 0

CustomerAS

ProviderAS


LLooaadd BBaallaanncciinngg 11::

MMuullttiippllee RRoouutteerrss

CustomerAS

ProviderAS


AAnnootthheerr NNoonn--BBGGPP AAlltteerrnnaattiivvee

OOSSPPFF RRoouuttiinngg DDoommaaiinn

Default Route (0.0.0.0/0) Metric Type 1 Equal Metrics

Static routesD1-A0ASBR1

D1-A0ASBR2

ISP 1


Multiple OSPF DefaultsMultiple OSPF Defaults

ISP 1POP

ISP 2POP

Default Route (0.0.0.0/0) Metric Type 2

Higher Metric to ISP 2 (Backup)

Static routesD1-A0ASBR1

D1-A0ASBR2


Blackhole RouteBlackhole Route

• Establish static route to your block(s) ip route 1.2.3.4 255.255.240.0 null0

• Redistribute/import into BGP

• Suppress more-specific prefix advertising


Effects of BlackholingEffects of Blackholing

• No route flapping outside your AS

—If your internal routes go up or down

• Incoming traffic for specific routes that are down

—Doesn’t match any internal route

—Automatically discarded without concerning anyone else

BGP Path BGP Path SelectionSelection


Next Hop AccessNext Hop Access

R2

R1X

Advertised routevia R1



SSccooppee::

MMEEDD vvss.. LLooccaall PPrreeffeerreennccee vvss.. WWeeiigghhtt

Weight

Local Preference

Weight

AS1 AS2

MED


AAddmmiinniissttrraattiivvee WWeeiigghhtt ((CCiissccoo eexxtteennssiioonn))



Rules in this router set R1 weight to 100, R2 weight to 500

R2

R1X


Weight exampleWeight examplefor load sharingfor load sharing

PrimaryISP

Default local preference 500All routes ^ AS_Backup + local preference 100

BackupISP

Default local preference 200


TTiieebbrreeaakkeerr ffoorr EEqquuaall WWeeiigghhtt::

LLooccaall PPrreeffeerreennccee

R2

R1

Advertised routevia R1, local preference 100

Advertised routevia R2, local preference 500


Local Preference Local Preference example for load sharingexample for load sharing

PrimaryISP

Default local preference 500All routes ^ AS_Backup + local preference 100

BackupISP

Default local preference 200


PPrreeffeerr llooccaallllyy oorriiggiinnaatteedd rroouutteess

R2

R1


Locally definedvia R2


AS PathAS Path

1

1

1 2

2 4

1 2 4

5

1 2 4 5


SShhoorrtteesstt AASS PPaatthh ((CCiissccoo eexxtteennssiioonn))

R2

R1

AS AS AS AS Route

AS AS Route


Full Employment For Consultants:Full Employment For Consultants:Interpreting AS PathInterpreting AS Path

• Default assumption: local preference set based on AS_PATH

• Cisco considers it as part of the algorithm


AS Path PrependingAS Path Prepending

• Applies to routes you advertise

• Makes them less attractive to others

• Increases AS_PATH length— your AS put in the path twice


Limitations of PrependingLimitations of Prepending

6

1

1 1

1

1 2

2

3

1 1 3

4

1 2 4

5

1 2 4 5


Route Learned from eBGP

Route Learned from iBGP

R2

R1

External Paths PreferredExternal Paths Preferred


RemoteAS

MED=100

MED=500

R2

R1

Lowest MEDLowest MED


Full Employment For Consultants:Full Employment For Consultants:Weight, Local Preference & MEDWeight, Local Preference & MED

• HIGHER value wins— Weight

— Local preference

• LOWER value wins— MED

— Cisco default: route with no MED preferred

— IETF: route with no MED least preferred


Full Employment For Consultants:Full Employment For Consultants:Scope of MEDScope of MED

• Default assumption: — MEDs only compared between exits to the same adjacent AS

• Alternate: always-compare-MED— Useful at exchange points, possibly private peerings

— Cisco knob


CClloosseesstt NNeeiigghhbboorr

IGP metric to R1=100

IGP metric to R1=500

R2

R1


LLoowweesstt BBGGPP rroouutteerr IIDD

R22.2.2.2

R11.1.1.1

exterior routing 201 howard c. berkowitz [email protected] [email protected] (703)998-5819 esn...

Documents

exterior routing tutorial

customer slide

information slide

vpn slide

asymmetrical routing

path selection slide

defaultfreezone d slide

src dest slide