OLD DOG CONSULTING

www.mpls2007.com

MPLS-TE Doesn’t ScaleAdrian Farrel

Old Dog Consultingadrian@olddog.co.uk

2OLD DOG

CONSULTING

Is the Sky Falling?

The only way to get your attention is to be alarmist

MPLS-TE is perfectly functional in today’s networks

But: MPLS-TE will not scale indefinitely The problem is the well-known “full mesh” or

“n-squared” problem The number of LSPs scales as the square of the number

of PEs

3OLD DOG

CONSULTING

What Do We Want to Achieve?

MPLS-TE is an important feature for many SPs Allow traffic to be groomed Optimize use of network resources Provide quality of service guaranties

Carriers look to provide edge-to-edge tunnels across their core networks Differentiated Services VPNs VLANS and pseudowires Multimedia content distribution Normal IP traffic

4OLD DOG

CONSULTING

What is the Scope of the Problem?

Consider a service provider network with 1000 PEs This is not outrageously large Such a network may be broken into areas or ASes

Consider a full mesh of PE-PE TE-LSPs Consider parallel tunnels for different services,

QoS levels, and for protection May give rise to multiples of 999,000 LSPs in the

core What is the capacity of a core LSR? What is the capacity of a management system?

5OLD DOG

CONSULTING

What Are the Scaling Limits? Management

NMS How many LSPs can the NMS process

Management protocols Reporting on large numbers of LSPs may overload the

management network LSR issues

Memory capacity Per LSP data requirements

CPU capacity – largely an RSVP-TE protocol issue Degradation of LSP setup times Soft state addressed by Refresh Reduction

MPLS forwarding plane Number of labels (Only 1048559 per interface)

6OLD DOG

CONSULTING

The Snowflake Topology Example network for analysis Meshed core of P nodes

Called P1 nodes Each Pi+1 node connected to

just one Pi node PE nodes connected to just one

Pn node Well-defined connectivity and

symmetry allows many important metrics to be computed

Number of levels & number of nodes per level may be varied We can vary the number of P1 nodes We can vary the ratio of Pi+1 to Pi We can vary the value n We can vary the number of PE nodes per Pn node

PE

P1

P2

7OLD DOG

CONSULTING

Analysing the Snowflake Topology Define

Pn a node at the nth level (level 1 is core) Sn the number of nodes at the nth level Mn the multiplier at the nth level (how many Pn+1 nodes are

connected to a Pn node) Ln number of LSPs seen by a Pn node

Discover LPE = 2*(SPE - 1) L2 = M2*(2*SPE - M2 - 1) L1 = M1*M2*(2*SPE – M2*(M1 + 1))

Practical numbers S1 = 10, M1 = 10, and M2 = 20 SPE = 2000 LPE = 3998 L2 = 79580 L1 = 756000

8OLD DOG

CONSULTING

The Ladder Topology

Example network for analysis Core of P1 nodes looks

like a ladder Similar to many national

networks Symmetrical trees subtended

to core Each Pi+1 node connected to just one Pi node Each PE node connected to just one P node

Again: Well-defined connectivity and symmetry allows many

important metrics to be computed Number of levels & number of nodes per level may be varied

9OLD DOG

CONSULTING

Analysing the Ladder Topology Same definitions as for snowflake network

E the number of subtended edge nodes (PEs) to each spar-node (E = M1*M2)

Discover LPE = 2*(SPE - 1) L2 = 2*M2*(SPE - 1) - M2*(M2 - 1) L1 ≈ E*E*S1*S1/2 + E*E*S1 + 3*E*E - E*M2

Practical numbers S 1 = 10, M1 = 10, and M2 = 20 E = 200 SPE = 2000 LPE = 3998 L2 = 79580 L1 = 2516000

10OLD DOG

CONSULTING

Option 1 – Solve a Different Problem!

If a full mesh of PE-PE LSPs is too big, don’t build it! This is the bottom line if we don’t fix the problem

The suggestion is to build a full mesh of Pn-to-Pn LSPs, and perform routing or routing-based MPLS between Pn and PE

Scaling improves from O(10002)to O(1002)

But we lose functionality Why did we want a PE-PE mesh? How do we handle private address spaces? What if the traffic is not routable?

This may simply not be good enough to provide the function

11OLD DOG

CONSULTING

Option 2 – LSP Hierarchies

Well-known, core MPLS function Label stacks Forwarding Adjacencies (RFC 4206) Configured or automatic grooming

Possible to build a full or partialmesh of hierarchical tunnels

For example connect all P2 nodes Each P2 node must encapsulate each PE-PE LSP

in the correct tunnel Each P1 node only sees the P2-P2 tunnels

12OLD DOG

CONSULTING

Scaling Properties of Hierarchies - Snowflake

Note that PE-PE tunnels don’t help P1-P1 tunnels are also no benefit (core is fully meshed) P2 nodes see all PE-PE LSPs and new tunnels

L2 = M2*(2*SPE - M2 - 1) + 2*(S2 - 1) Situation at P1 nodes is much better

L1 = M1*(2*S2 - M1 - 1) Numbers (S1 = 10, M1 = 10, and M2 = 20)

Flat 2-Level HierarchySPE 2000 2000LPE 3998 3998L2 79580 79778L1 756000 1890

Maybe insert another layer (P3 ) to increase the scaling? L3 remains high

13OLD DOG

CONSULTING

Scaling Properties of Hierarchies - Ladder

Note that PE-PE tunnels don’t help But P1-P1 tunnels are good because core is not fully-meshed

L1 ≈ S1*S1/2 + 2*S1 + 2*E*E*(S1 - 1) - E*M2 - 2 Another level of hierarchy is also possible

Add a mesh of P2-P2 tunnels L1 = S1*S1/2 + 2*S1 + 2*M1*M1*S1 - M1(M1 + 1) – 2 L2 = 2*M2*(S(PE) - 1) - M2*(M2 - 1) + 2*(S(1)*M(1) - 1)

Numbers (S 1 = 10, M1 = 10, and M2 = 20)Flat 2-Level 3-LevelHierarchy HierarchySPE 2000 2000 2000LPE 3998 3998 3998L2 795807958079778L1 2516000 716060 1958

14OLD DOG

CONSULTING

Issues and Drawbacks for Hierarchies

Scaling is not good enough! Impact on layer adjacent to PEs is negligible

Actually impact is slightly negative Management burden

Plan and operate a secondary mesh Effectively the same burden as managing PEs or a

layered network Possible to consider auto-mesh techniques

Fast Reroute protection is a problem FRR struggles to protect tunnel end-points

Not obvious how to arrange the hierarchy when the network is not symmetrical E.g., some PEs closer to the core

15OLD DOG

CONSULTING

Option 3 – Multipoint-to-Point LSPs LSPs merge automatically as they converge on the

destination Reduces the number of LSPs toward the egress Other LSP properties (e.g.,

bandwidth) must be cumulative TE is still possible, but

de-merge is not considered Should count “LSP state” not number of LSPs

New definition Xn the amount of LSP state

held at each Pn node For flat and hierarchical networks:

Each LSP adds one state at ingress or egress Each LSP adds two states at each transit node

16OLD DOG

CONSULTING

Scaling Properties of MP2P LSPs - Snowflake

XPE = 2*(SPE - 1)X2 = SPE*(M2 + 1)X1 = M1*M2*(S1 - 2) + SPE*(M1 + 1) Numbers (S1 = 10, M1 = 10, and M2 = 20)

Flat 2-Level Hierarchy P2MPSPE 2000 2000 2000XPE 3998 3998 3998X2 159160 159358 42000X1 1512000 3780 23600

17OLD DOG

CONSULTING

Scaling Properties of MP2P LSPs - Ladder

XPE = 2*(SPE - 1)X2 = (M2 + 1)*S1*EX1 ≤ (4 + M1)*S1*E - M1*E Numbers (S1 = 10, M1 = 10, and M2 = 20)

Flat 2-Level 3-LevelP2MP

Hierarchy HierarchySPE 2000 2000 2000 2000XPE 3998 3998 3998 3998X2 159160 159160 159358 42000X1 5032000 1433998 3898 26000

18OLD DOG

CONSULTING

Issues and Drawbacks for MP2P LSPs Clear scaling benefits

Better than flat networks Only thing that improves the situation adjacent to PEs

But… Data plane support

This will only ever be a packet/frame/cell technology Control plane support

RSVP does have MP2P support RSVP-TE features not yet specified or implemented

De-aggregation and disambiguation May be necessary to use label stack so that egress can

detect sender of data OAM may be more complex and require source labels New management applications needed FRR still to be designed

19OLD DOG

CONSULTING

Other Topics for Investigation Cost-effectiveness of the network

Revenue only generated by PEs K = S(PE)/(S(1)+S(2) + ... + S(n)) Many ways to improve scaling reduce cost-effectiveness

Fast Reroute What are the implications of FRR to scaling? Can scaling contributions be designed that can be protected by

FRR? Point-to-multipoint

What are the scaling properties of P2MP MPLS-TE? Domain boundaries (in particular AS boundaries)

Boundaries such as at area and AS borders cause constrictions How can we reduce the number of LSPs seen by ABRs and

ASBRs?

20OLD DOG

CONSULTING

Conclusions, Next Steps, and References

MPLS-TE is not a scaling issue today But it won’t scale arbitrarily

We need to plan now for tomorrow’s scalability Hierarchical LSPs are not as good as expected MP2P LSPs may offer a better solution More research and implementation is needed

draft-ietf-mpls-te-scaling-analysis-01.txt Seisho Yaukawa (NTT) Adrian Farrel (Old Dog Consulting) Olufemi Komolafe (Cisco Systems)

21OLD DOG

CONSULTING

Questions?

adrian@olddog.co.uk