cisco fabricpath · mac if/sid b e1/2 (local) 8 miss if dmac is known, then learn remote mac...

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Connect 1 1 © 2012 Cisco and/or its affiliates. All rights reserved.

Cisco FabricPath Technology and Design

Max Ardica

Technical Leader

Dubrovnik, Croatia, South East Europe

20-22 May, 2013

© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 2

FabricPath The Requirements

© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 3 3

Cisco DC Fabric Evolution of the Intelligent Layer 2 Domain POD

Shipping Nexus 5k/6k/7k

Shipping

Nexus 7k


Some protocols rely on the functionality

Simple, almost plug and play

No addressing

Required for implementing subnets

Allows easy server provisioning

Allows virtual machine mobility

Application clustering support

Because customers request it!

Cisco DC Fabric Why Extended Layer 2 in the Data Center?


“FabricPath brings Layer 3 routing benefits to

flexible Layer 2 bridged Ethernet networks”

Easy Configuration

Plug & Play

Provisioning Flexibility

Multi-pathing (ECMP)

Fast Convergence

Highly Scalable

Switching Routing

FabricPath

Cisco DC Fabric Introducing FabricPath


Shipping Since 2010: turn your network into a Fabric

N7K(config)# interface ethernet 1/1

N7K(config-if)# switchport mode fabricpath

FabricPath

Connect a group of switches using an arbitrary topology

With a simple CLI, aggregate them into a Fabric:

An open protocol based on Layer 3 technology provides Fabric-wide intelligence and ties the elements together

FabricPath, an Ethernet Fabric


Why Migrate from vPC to FabricPath? Flexible Topologies Support

L3 Cloud Compute

Folded CLOS

Full Mesh

L3 Cloud Compute Compute

Three Tiers (Fat Tree)

Compute

L3 Cloud

Traditional Access/Aggregation

7


FabricPath The Technology


CE Edge Ports

FP Core Ports

Spine Switch

Leaf Switch

Send/receive regular Ethernet frames

Run STP, do MAC address learning using a MAC

address table

Classical Ethernet (CE)

S10 S20 S30 S40

S100 S200 S300

1/1 1/2

A B

Send/receive FabricPath frame

No STP, no MAC learning, no MAC address table

Using a routing table computed by IS-IS

Clos Fabric

FabricPath Terminology

FabricPath


IS-IS assigns addresses to all FabricPath switches automatically

Compute shortest, pair-wise paths

Support equal-cost paths between any FabricPath switch pairs

L1

FabricPath

Routing Table

L2 L3

L4

Switch IF

S10 L1

S20 L2

S30 L3

S40 L4

S200 L1, L2, L3, L4

… …

S400 L1, L2, L3, L4

S100 S200 S300 S400

S10 S20 S30 S40

New Control Plane Plug-n-Play L2 IS-IS Manages Forwarding Topology

FabricPath


The association MAC address/Switch ID is maintained at the edge

Traffic is encapsulated across the Fabric

Classical Ethernet (CE)

S10 S20 S30 S40

S100 S200 S300

1/1 S100: CE MAC

Address Table

MAC IF

A 1/1

B S300

1/2

S100: FabricPath

Routing Table

Switch IF

… …

S100 L1, L2, L3, L4

FabricPath

Switch ID space:

Routing decisions

are made based on

the FabricPath

routing table

MAC address space:

Switching based on

MAC address tables

S100 S300 A B

A B

New Data Plane


FabricPath

Frame

Classical Ethernet Frame

• Switch ID – Unique number identifying each FabricPath switch

• Ftag (Forwarding tag) – Unique number identifying topology and/or distribution tree

• TTL – Decremented at each switch hop to prevent frames looping infinitely

DMAC SMAC 802.1Q Etype Payload CRC

(new)

FP

Tag

(32)

Outer

SA

(48)

Outer

DA

(48)

Endnode ID

(5:0)

Endnode ID

(7:6)

U/L

I/G

RS

VD

OO

O/D

L

Etype

0x8903

6 bits 1 1 2 bits 1 1 12 bits 8 bits 16 bits 10 bits 6 bits 16 bits

Switch ID Sub

Switch ID Ftag TTL LID

Original CE Frame 16 bytes

DMAC SMAC 802.1Q Etype CRC Payload

FabricPath Encapsulation 16-Byte MAC-in-MAC Header


I/O Modules CE Edge Port FP core port Routing for FP

VLAN

N7K-M108X2-12L

N7K-M132XP-12(L)

N7K-M148GS-11(L)

N7K-M148GT-11(L)

✗ ✗ ✓

N7K-M224XP-23L

N7K-M206FQ-23L

N7K-M202CF-22L

✗ ✗ ✓

N7K-F132XP-15 ✓ ✓ ✗

N7K-F248XP-25* ✓ ✓ ✓

N7K-F248XT-25E**

N7K-F248XP-25E** ✓ ✓ ✓

*F2 module needs to be in their own VDC or system. It is not possible to mix F1 and F2 LC in the same VDC

**F2E module supported in proxy-routing mode (mixed in a VDC with Mx cards) from 6.2 release. No mix of F1 and F2E LC in the same VDC

For Your Reference

13

FabricPath Hardware Support Nexus 7000


Switch CE Edge

Port FP Core

port Routing for

FP VLAN

✗ ✗ ✗

✗ ✗ ✗

✓ ✓ ✓

✓ ✓ ✓

✓ ✓ ✓

* L3 daughter card is ‘not’ needed in 5550 to run FabricPath, only L3 routing for FabricPath VLANs

FabricPath Hardware Support Nexus 5000, 5500 & 6000

Nexus 5020

Nexus 5548P/UP*

Nexus 5596UP/5596T*

Nexus 6004

14

Nexus 5010

Nexus 6001

For Your Reference


The Nexus 7000 features two kinds of IO Modules: M series and F series

M1/M2 I/O Modules cannot switch FabricPath traffic

When running FabricPath, FP Core and CE Edge ports must be on an F module or 5500/6000

New FabricPath/CE locally significant VLAN mode:

FabricPath VLANs can only be enabled on F modules (FEX+F2 with NX-OS 6.0 release and FEX+F2E with NX-OS release 6.1 as well) or 5500/6000 (5500/6000 + FEX as well)

FabricPath

Leaf1(config)# vlan 10

Leaf1(config-vlan)# mode ?

ce Classical Ethernet VLAN mode

fabricpath Fabricpath VLAN mode

Leaf1(config-vlan)# mode fabricpath

Leaf1(config-vlan)#

E

F FabricPath

Core Port

Classical

Ethernet

Edge Port

FabricPath VLAN Mode CE versus FP VLAN’s

15


S10 S20 S30 S40

Root for

Tree 1

S100 S2

0

Root for

Tree 2

S1

0 S200

S300

S3

0

S4

0 Logical

Tree 1

Root

S4

0

S100

S200

S300

S1

0

S2

0

S3

0 Logical

Tree 2

Root

S100 S200 S300

FabricPath

FabricPath Key Concept Multi-destination Trees

Multi-destination traffic constrained to loop-

free trees touching all FabricPath switches

Root switch elected for each multi-

destination tree in the FabricPath domain

Loop-free tree built from each Root

assigned a network-wide identifier (Ftag)

Support for multiple multi-destination trees

provides multipathing for multi-destination

traffic

Two multi-destination trees currently supported

in NX-OS

16


Multi-destination

Trees on Switch 100

Tree IF

1 L1

2 L1,L2,L3,L4

S1

0

S2

0

S3

0

S4

0

Root for

Tree 1

Root for

Tree 2

S100 S200 S300

FabricPath

Ingress FabricPath switch determines which tree to use for each flow

Other FabricPath switches forward based on tree selected by ingress switch

Broadcast and unknown unicast typically use first tree

Hash-based tree selection for IP multicast, with several configurable hash options

Multi-destination Trees Role of the Ingress FabricPath Switch

17


S100: CE MAC

Address Table

S1

0

S2

0

S3

0

S4

0

S100 S200 S300

FabricPath

e1/

1 S300: CE MAC

Address Table

MAC IF

B 1/2

… …

S200: CE MAC

Address Table

MAC IF

… …

… …

S100 M A B

Lookup B: Miss

Don’t learn

Lookup B: Miss

Flood

Lookup B: Hit

Learn source A

MAC IF

B e1/2

A S100

MAC IF

… …

… …

MAC IF

A e1/1

… …

e1/2

FabricPath Key Concept Unknown Unicast Flooding

A B

18


Conversational

Learning

S100: CE MAC

Address Table

A

S10 S20 S30 S40

S100 S200 S300

B

e1/1 S300: CE MAC

Address Table

MAC IF

B 1/2

… …

S200: CE MAC

Address Table

MAC IF

… …

… …

MAC IF

B e1/2

A S100

MAC IF

… …

… …

MAC IF

A 1/1

… …

e1/2

S300: FabricPath

Routing Table

Switch IF

… …

S100 L1, L2, L3, L4

S300 S100 B A

Lookup A: Hit

Send to S100 Lookup A: Hit

Learn source B

MAC IF

A e1/1

B S300

FabricPath Key Concept Known Unicast, Conversational MAC Learning

19


FabricPath

MAC Table on S100

MAC IF/SID MAC IF/SID

A e1/1 (local)

S10 S20 S30 S40

Root for

Tree 1

Root for

Tree 2

S100 S200 S300

MAC A MAC B

Multi-destination

Trees on S100

Tree IF

1 po10

2 po10,po20,po30,po40

Broadcast →

DMAC→FF

SMAC→A

Payload

Multi-destination

Trees on S10

Tree IF

1 po100,po200,po300

2 po100

po10 po20

po40

po30

Ftag →

DMAC→FF

SMAC→A

Payload

DA→FF

Ftag→1

SA→100

DMAC→FF

SMAC→A

Payload

po100

po300

po200

po10

po20 po30 po40

1

3

2

4

6

Learn MACs of directly-connected

devices unconditionally

Don’t learn MACs from

flood frames

FabricPath

MAC Table on S200

MAC IF/SID

Multi-destination

Trees on S300

Tree IF

1 po10,po20,po30,po40

2 po40

5

Putting it all together – Host A to Host B (1) Broadcast ARP Request

e1/1 e1/2

20

Ftag →

DMAC→FF

SMAC→A

Payload

DA→FF

Ftag→1

SA→100


S10 S20 S30 S40

Root for

Tree 1

Root for

Tree 2

S200 S300

MAC A MAC B

po10 po20

po40

po30

po100

po300

po200

po10

po20 po30 po40

Multi-destination

Trees on Switch 100

Tree IF

1 po10

2 po10,po20,po30,po40

Ftag →

Multi-destination

Trees on Switch 10

Tree IF

1 po100,po200,po300

2 po100

Ftag →

11

10

DMAC→A

SMAC→B

Payload

DA→MC1

Ftag→1

SA→300

DMAC→A

SMAC→B

Payload

7

A →

FabricPath

MAC Table on S300

MAC IF/SID MAC IF/SID

B e1/2 (local)

8

MISS

If DMAC is known, then

learn remote MAC

Multi-destination

Trees on Switch 300

Tree IF

1 po10,po20,po30,po40

2 po40

9 FabricPath

MAC Table on S100

MAC IF/SID

A e1/13 (local)

MAC IF/SID

A e1/11 (local)

B 300 (remote)

12 DMAC→A

SMAC→B

Payload

Putting it all together – Host A to Host B (2) Broadcast ARP Reply

21

e1/1 e1/2 Unknown →

DMAC→A

SMAC→B

Payload

DA→MC1

Ftag→1

SA→300


FabricPath Routing Table on

S100

Switch IF

S10 po10

S20 po20

S30 po30

S40 po40

S200 po10, po20,

po30, po40

S300 po10, po20,

po30, po40 FabricPath

MAC Table on S300

S10 S20 S30 S40

S200 S300

MAC A MAC B

po10 po20

po40

po30

po10

po20 po30 po40

S100

DMAC→B

SMAC→A

Payload

FabricPath

MAC Table on S100

DMAC→B

SMAC→A

Payload

13

MAC IF/SID

A e1/1 (local)

B 300 (remote) B →

14

S300 →

DMAC→B

SMAC→A

Payload

DA→300

Ftag→1

SA→100

MAC IF/SID

B e1/2 (local)

po300

Putting it all together – Host A to Host B (3) Unicast Data

22

e1/1 e1/2 15

Hash

MAC IF/SID

A S100 (remote)

B e1/2 (local)

17

DMAC→B

SMAC→A

Payload

DA→300

Ftag→1

SA→100.

FabricPath Routing

Table on S30

Switch IF

… …

S300 po300 S300 → 16

If DMAC is known, then

learn remote MAC

FabricPath Design Considerations vPC+ at the Edge

Using vPC+ at the edge enables several

options

Attaching servers with Port-channels

Attaching other Classic Ethernet Switches in

vPC mode

Attaching FEX in Active/Active mode

Each pair of Nexus 5500 configured for

vPC+ runs FabricPath with the spine

AND on the vPC peer-link too

The vPC domain is advertised as a

different Switch-id (Emulated switch)

The area highlighted (i.e. its MAC

addresses) are advertised as belonging

to the “Emulated switch”

vPC+

Emulated Switch-Id 1000

23

vPC+ configuration applied to both FP Leaf devices

vPC+ at the Edge Configuration

Emulated Switch-Id 1000

Leaf1(config)# vpc domain 1

Leaf1(config-vpc-domain)# fabricpath switch-id 1000

Leaf 2 Leaf 1

Spine 1 Spine 2

e1/1 e1/2

Leaf devices advertise MAC_A and MAC_B of hosts attached in vPC+ mode as associated to the emulated switch-ID 1000

North-to-south traffic directed to the hosts has two equal cost paths available

Spine1# sh fabricpath route

FabricPath Unicast Route Table

'a/b/c' denotes ftag/switch-id/subswitch-id

'[x/y]' denotes [admin distance/metric]

<snip>

1/1000/0, number of next-hops: 2

via Eth1/1, [115/40], 1 day/s 04:12:00, isis_fabricpath-default


vPC+

24

MAC A MAC B

Active HSRP Standby HSRP

FabricPath Design Considerations vPC+ at the Spine (Dual Active Gateway)

Emulated Switch-Id 2000 It is possible to distribute routed traffic

to both spines by using vPC+

Spines function as HSRP Active/Standby on

the control plane, Active/Active on the data

plane

The HSRP MAC is advertised with the same

Emulated Switch-Id to all leaf devices

Each edge switch has an equal cost

path to the Emulated Switch ID (via

both spine devices)

All you need to do on the spines is to

configure a vPC domain and a peer-

link

No need to define any vPC port

vPC+

Leaf1# show mac address-table vlan 101

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay

MAC

age - seconds since last seen,+ - primary entry using vPC Peer-

Link

VLAN MAC Address Type age Secure NTFY Ports

---------+-----------------+--------+---------+------+----+--------------

----

* 101 0000.0c07.ac01 dynamic 0 F F 2000.0.1054

Leaf1# sh fabricpath route switchid 2000

<snip>

FabricPath Unicast Route Table for Topology-Default

1/2000/0, number of next-hops: 3



Leaf 1

e1/1 e1/2

Layer 2

Layer 3

25


IGMP Reports

IGMP Reports

FabricPath

S100

S300

S200

Root of

Tree 1

Root of

Tree 2

Mrouter

IGMP

snooping

IGMP

snooping

IGMP

snooping

GM-LSPs

GM-LSPs

Receiver G2

Receiver G1

Receiver G2

Source G1

Source G2 PIM Hellos

Ftag 1

Ftag 2

IGMP Snooping in FabricPath

IGMP snooping learns of interested receivers on FabricPath edge switches

Membership tracked on CE ports based on receiving IGMP reports/leaves

Only locally connected receivers tracked on a given edge switch

Group membership advertised in FabricPath IS-IS using using GM-LSPs

26


Src-G1

S100

FabricPath

S100

S300

S200

Root

S300

S200

Root

G1 Pruned Tree

G2 Pruned Tree

Multidestination Tree 1

Src-G2

Rcvr-G2

Rcvr-G2

Multidestination Tree 2

FabricPath

Mrouter

Mrouter

FabricPath

S100

S300

S200

Root

G1 Pruned Tree

G2 Pruned Tree

Src-G1

Mrouter

FabricPath

S100

S300

S200

Root

Src-G2

Rcvr-G2

Mrouter

Rcvr-G2

Ftag 1

Ftag 2

Rcvr-G1 Rcvr-G1

FabricPath IP Multicast Control Plane IS-IS Creates Pruned Forwarding Trees Using GM-LSPs

27


FabricPath

Topology 0 – Default Topology

Topology 1

Topology 2

FabricPath Multiple Topologies

Topology: A group of links in the Fabric.

By default, all the links are part of

topology 0

A VLAN is mapped to a unique topology

Other topologies can be created by

assigning a subset of the links to them.

A given link can belong to several topologies

Topologies can be used for migration

designs (i.e. VLAN localization, VLAN

re-use), traffic engineering, security

etc…

2 topologies currently supported on

Nexus 5500 (from release 5.2(1))

2 multi-destination trees are supported for

each topology

2 topologies supported on Nexus 7000

starting from release 6.2 (Q2CY13)

28


spine1(config)# show fabricpath topology vlan

Topo-Description Topo-ID Configured VLAN List

------------------------------------ ------------ -----------------------------

0 0 1-4095

s100(config)# fabricpath topology 1

s100(config-fp-topology)# member vlan 10

s100(config)# int po1

s100(config-if)# fabricpath topology 1

s100(config-fp-topology)# sh fabricpath topology vlan

Topo-Description Topo-ID Configured VLAN List

-------------------------------- --------- -------------------------------------

0 0 1-9, 11-4095

1 1 10

S2

S100 S300 S200

VLAN5

S400

S1

VLAN 10 VLAN5 VLAN 10

Po1

VLAN

10

S100

S1 S300

S400

S20

0

Topology 0

Tree 1

S2 Root

S200

S1 S300

S400

S10

0

Topology 0

Tree 2

S2 Root

S10

0

S2

00

Topology 1

Tree 3

Root S20

0

S1

00

Topology 1

Tree 4

Root

29

Multi-Topology Support Understanding VLANs and FabricPath Topologies

29


POD 1 VLANs 100-199 VLANs 2000-2099

POD 2 VLANs 200-299 VLANs 2000-2099

POD 3 VLANs 300-399 VLANs 2000-2099

CORE VLANs 100-199 VLANs 200-299 VLANs 300-399 VLANs 2000-2099

Core must include VLANs from

all PODs due to multidestination

tree behavior

Single Topology Design FabricPath Topology

30


POD 1 T1: VLANs 10,100-199 T0:VLANs 2000-2099

POD 2 T1: VLANs 10, 200-299 T0: VLANs 2000-2099

POD 3 T1: VLANs 10,300-399 T0: VLANs 2000-2099

CORE VLANs 2000-2099

Layer 2 FP Default Topology POD 1 Topology + Default Topology POD 2 Topology + Default Topology POD 3 Topology + Default Topology

Only DC-wide VLANs

exist in FabricPath core

POD-local VLANs exist only

in POD switches, mapped to

POD-specific topology

Core ports in POD belong to

default topology and also mapped

to POD-local topology Core ports

belong only to

default

topology

POD-local VLAN 10 can be

re-used between pods. VLAN

10 is unique to each POD

Multi-Topology Design Localized Forwarding

31


FabricPath and STP FabricPath Simple STP Interaction

FP leaves send and receive STP BPDUs on

CE edge ports

No BPDUs is sent out of FP core port links

FP leaves must be configured as STP root

for all VLANs configured on vPC+ member

ports

All FP switches in the FP domain have the

same STP bridge ID: c84c.75fa.6000

There is no need to explicitly configure

peer-switch in vPC+; both vPC+ peer

devices can send/receive BPDU to/from CE

access switches

FabricPath

BP

DU

s

Logical STP Topology

STP Root STP Root

32


L2 Gateway Inconsistent (port STP blocked)

BID 0030.f275.a20e Bridge Priority 32768

BPDU

Superior BPDU

BID c84c.75fa.6000 Bridge Priority 32768

BPDU

FabricPath and STP Root-Guard

CE edge ports (i.e vPC+ member ports) have FabricPath root guard-like

function enabled implicitly

If superior BPDU received on edge port, port is placed in “L2 Gateway

Inconsistent” state until condition cleared

%STP-2-L2GW_BACKBONE_BLOCK: L2 Gateway Backbone port inconsistency

blocking port port-channel100 on VLAN0100

FabricPath

S1 S2

CE-1

33

FP Leaf

33


spanning-tree pseudo-information

vlan 100 root priority 4096

FabricPath

S1 S2

CE-1 CE-2

When connecting a Classical

Ethernet (CE) switch to FP Leaves,

pay attention to STP priority on

vPC+ domain

STP priority for the VLAN on FP

leaves must be better than on CE

access switch

Recommendation is to decrease STP priority

value on FP leaves

Configure pseudo-information just to cover

scenarios where the CE devices are

attached in STP mode (no vPC)

FabricPath and STP Setting STP Priority on FP Leaves

BID c84c.75fa.6000 Bridge Priority

4096

BPDU

34

34


FabricPath The Design


Check out the corresponding design guide:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/guide_c07-690079.html

Evolutionary extension of current design practices

Design benefits:

Simplified configuration

Removal of STP

Traffic distribution over all uplinks without VPC port-channels

Active/active gateways

“VLAN anywhere” at access layer

Topological flexibility

Scalability considerations

Today: 16K unique host MACs across all routed VLANs

With NX-OS 6.2 release: 128K MACs with “proxy L2 learning” on M1/M2+F1/F2E (Nexus 7000)

128K MACs with Nexus 6000

SVIs

L3

L2/L3 boundary

FabricPath

SVIs

Routed

core

Aggregation

Access

Routing at Aggregation Natural Evolution of the vPC Design

37


Number of Spine Switches N

eed f

or

HA

Spine-Leaf Design Changes to the Approach to Structured High Availability

38


Split VLANs

Some polarization

Inter-VLAN traffic can

be suboptimal

VLAN 100-

200

VLAN 300-

400

FabricPath

GLBP

Host is pinned to a

single gateway

Less granular load

balancing

VLAN 100-400

FabricPath

Anycast HSRP

All active

Available in NX-OS

6.2 release

VLAN 100-400

FabricPath

Routing at Aggregation Options to Scale-Out the Spine Layer

39


FabricPath

L3

L2/L3 boundary Layer 3

services leaf

switches

FabricPath

spine

Server access

leaf switches

FabricPath

L3

Layer 3 services

border leaf switches

FabricPath

spine

All VLANs

available at all

leaf switches FHRP between L3 services

switches for FHRP

L2/L3 boundary

Centralized Routing Design

Alternate View

= Run VPC+

for

active/active

HSRP

Centralized Routing Removing Routing from the FP Spine Layer

Integrated L2/L3 Across All Nodes

Leaf

L3

Spine node

Border leaf

Leaf node

Compute/Services

Spine

Extensible Network Topologies

Distributed Control Plane

Enhanced Forwarding

Simplified Management

Secure Multi-Tenancy

Automation and Simplicity

S1 S2 S3 Sk

L1 L2 L3 Ln BL BL L3

L2

Cisco Unified Fabric Evolution Leading The Data Center Flexibility and Innovation With Nexus

http://www.cisco.com/go/fabricpath

4

2

Additional Resources


Thank you.

cisco fabricpath · mac if/sid b e1/2 (local) 8 miss if dmac is known, then learn remote mac...

Documents