fabricpath operation and troubleshooting · fabricpath operation and troubleshooting ... acronyms...
TRANSCRIPT
Acronyms / Definitions
Acronyms Definitions Acronyms Definitions
ACL Access Control List FP FabricPath
ASIC Application Specific Integrated Circuit FTAG Forwarding Tag
ASID Anycast Switch Identifier LID Local Identifier
BD Bridge Domain LTL Local Target Logic
CE Classical Ethernet MIM MAC-in-MAC (common reference to FP
header)
DBUS / RBUS Data Bus / Result Bus PACL Port-based ACL
DRAP Dynamic Resource Allocation Protocol RACL Router-based ACL
DSID Destination Switch Identifier RPF Reverse Path Forwarding
ELAM Embedded Logic Analyzer Module SoC Switch-On-Chip
ES Emulated Switch SSID Source Switch Identifier
FE Forwarding Engine VACL Vlan-based ACL
FF Flood to Fabric VDC Virtual Device Context
Reference Slide
• FabricPath Overview
Benefits, Restrictions, and Configuration
• Key Concepts
Encapsulation, Trees, Topologies, STP
• Data Plane
Forwarding, Load-Balancing, MAC Learning
• vPC+
Challenges and Operation
• Troubleshooting
Verification steps, tools, and examples
Agenda
FabricPath Benefits
Single path between 2 points in L2 network
• Stability/Resilience at scale
• Disruptive convergence
Shortest path between switches + equal-cost load-balancing
• Core does not need to learn end host MAC addresses
• More resilient to loops
• No topology constraints, L3 anywhere
• Easy scaling / Non-disruptive merge
Existing
Layer2
FabricPath
Fabricpath OverviewUnicast: Known Destination MAC
CE FabricPath CE
MAC A MAC B
Ingress
FabricPath
(Edge) Switch
Egress
FabricPath
(Edge) Switch
DSID comes from
MAC address
table for MAC B
SSID comes
from S10’s
own switchID
TTL
decremented at
every FP switch
Intermediate
switches forward
based on DSID
DMAC B
SMAC A
Payload
DMAC B
SMAC A
Payload
DSID 20
SSID 10 DMAC B
SMAC A
Payload
FabricPath OverviewMultidestination (broadcast, multicast, unicast flood)
MAC A MAC B
DMAC B
SMAC A
Payload
DMAC B
SMAC A
Payload
SSID comes
from S10’s
own switchID
Root switch
for Tree 2
MAC B is
unknown DSID
= FloodSID
Ingress FP
Switch selects
Tree (FTAG)
Root switch
for Tree 1→ FabricPath interface
→ CE interface
→ Tree 1
→ Tree 2DMAC B
SMAC A
Payload
DMAC B
SSID
FTAG 1 DMAC B
SMAC A
Payload
DMAC B
SSID
FTAG 1
CE FabricPath CE
FabricPath support & configuration
• N7K with N7K-F1 linecard as of 5.1.1
• N7K with N7K-F2 linecard as of 6.0.1• N7K + FEX as of 6.1.1 (with N7K-F2) for CE
ports
• F2E as of 6.1.2
• N7K with N7K-F3 linecard as of 6.2.6
• N5500 as of 5.1.3 • no L3 module required
• N5500 + FEX as of 5.1.3 for CE ports
• N6K as of 6.0.2
• Enhanced L2 license required FabricPath
• Packaged as feature-set (plugin)
N7K(config)# install feature-set fabricpath
N7K(config)# feature-set fabricpath
N7K(config)# interface Ethernet4/1
N7K(config-if)# switchport mode fabricpath
...
N7K(config)# vlan 3002
N7K(config-vlan)# mode fabricpath
FabricPath & CE Vlans
• Two types of vlans
CE (Classic Ethernet, default)
FabricPath (FP)
• FP vlans cannot go on M1, M2 modules
• Only FP vlans will be carried over FP interfaces
• FP vlans can be mixed with CE vlans on edge interfaces
N7K(config)# vlan 3002
N7K(config-vlan)# mode ?
ce Classical Ethernet VLAN mode
fabricpath Fabricpath VLAN mode
Classic Ethernet
FabricPath
Port Type VLANs allowed
to be configured
VLANs allowed to
be brought up
N7K-M1, N7K-M2 FP, CE CE
N7K-F1, N7K-F2, N7K-F3 Edge FP, CE FP, CE
N7K-F1, N7K-F2, N7K-F3 Core FP, CE FP
N5500, N6000 Edge FP, CE FP, CE
N5500, N6000 Core FP, CE FP
Core = switchport mode fabricpath
Edge = switchport mode access || trunk
• FabricPath Overview
Benefits, Restrictions, and Configuration
• Key Concepts
Encapsulation, Trees, Topologies, STP
• Data Plane
Forwarding, Load-Balancing, MAC Learning
• vPC+
Challenges and Operation
• Troubleshooting
Verification steps, tools, and examples
Agenda
Encapsulation
Outer SA: SwitchID ingress FP switch system ID
SubswitchID is used in some cases of VPC+
LID is specific to the implementation
• N7K the LID is generally the port index of the ingress interface
• N5K/N6K LID most of the time will be 0
• EndnodeID is not currently used
Outer DA: For known SA/DA is taken from MAC table for DMAC
For broadcast and multicast is the same as DMAC
For unknown unicast DA is 010f.ffc1.01c0 (flood to vlan)
For known unicast DA, but unknown SA is 010f.ffc1.02c0 (flood to fabric)
Example
DMAC SMAC 802.1Q Etype PayloadCRC
(new)
FP
Tag
(32)
Outer
SA
(48)
Outer
DA
(48)
Switch_ID SubSwitch_ID LID
100 1 65535
N7K# show fabricpath switch-id | include SYS|\*
Legend: '*' - this system
SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED
*2028 b414.89e3.a041 Primary Confirmed No No
N7K# sh mac address-table address 0000.1234.5678
VLAN MAC Address Type age Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------------------
3000 0000.1234.5678 dynamic 0 2.0.1054
Local IDSubSwitch
IDSwitch ID
EndnodeID
[ 7:6]
EndnodeID
[ 5:0]
U/L I/G
0
16 bits8 bits12 bits26 1 1
47
1 1
RSVD
OOO
Ethernet II, Src: 02:00:64:01:FF:FF, Dst: 01:00:5e:00:00:02, Type: 0x8903
FabricPath Switch IDs, System IDs … and DRAP
4 5
• Each FP switch is identified by unique number (ID), dynamically assigned or static
• Dynamic Resource Allocation Protocol (DRAP) is responsible for allocating switch IDs and resolving duplicate-ID conflicts. Conflicts are resolved by renumbering switches with higher systemID(DRAP can only auto resolve non-static switch ID)
• When partitioned FP network is merged (or new switch joins the fabric) connecting interface is not enabled for data before all conflicts are resolved
1 2 3
3
+
=
1 2 3
64 5
N7K# show fabricpath switch-id
FABRICPATH SWITCH-ID TABLE
=========================================================================
SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED
----------+----------------+------------+-----------+--------------------
*3 c062.6bac.e343 Primary Confirmed Yes No
30 547f.ee02.ce3c Primary Confirmed Yes No
40 547f.ee04.5cfc Primary Confirmed Yes No
N7K(config-if-range)# no shut
%FABRICPATH-2-
FABRICPATH_LINK_BRINGUP_STALLED_STATIC: Link
bringup stalled due to conflicts
N7K# show fabricpath conflict all
Port State
---------------+------------------------
Ethernet3/31 Suspended due to conflicts
==============================================
Fabricpath Conflicts
SYSTEM-ID SWITCH-ID STATIC
---------------+--------------+---------------
c062.6bac.e343 3 Yes
c062.6bac.e342 3 Yes
Network Merges / Conflict resolution• Goal is to connect two networks with conflicting switch IDs
without incurring packet loss
1) Allocate new switch-id as secondary – tentative• Wait allocate delay time
2) Make new switch-id as secondary - confirmed• Wait transition delay time
3) Swap primary and secondary switch-ids• Wait transition delay time
4) Delete old switch-id (now a secondary switch-id)
More About Graceful Merge
Graceful merge changes the switch-id of a switch to
resolve switch-id collisions
The switch-id to change is based on the system-id
being higher value, or being dynamic
For a time period the switch is identified by two switch-
ids, packets for both are accepted but outgoing packets
only carry the primary switch-id
N7k# show fabricpath switch-id
Legend: '*' - this system
SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED
----------+----------------+------------+-----------+--------------------
*332 b414.89e3.a042 Primary Confirmed Yes No
N7k# show fabricpath isis switch-id
Legend: C - Confirmed, T - tentative, W - swap
S - sticky, E - Emulated Switch
'*' - this system
System-ID Primary Secondary Reachable Bcast-Priority
MT-0
b414.89e3.a042* 332 [C] 0[C] Yes 222 [S]
N7k# show fabricpath timers
Allocate Delay Timer : 10
Transition Delay Timer : 10
Link-up Delay Timer : 10
FabricPath Trees• Known unicast traffic is load-balanced across equal-cost
routes
• FabricPath uses two loop-free trees for unknown unicast, broadcast and multicast traffic
• Two trees are for load-balancing
• For each packet, tree is selected by ingress FP switch and choice is carried in the packet header
• Root of tree1 is the switch with highest Priority (highest sysID for tie)
• Root of tree2 is the switch with 2nd highest Priority (highest sysID for tie)
• Tree is a least-cost-to-the-root graph, with lower sysID used as tie-breaker
• In case of Tree1 root failure both roots are reelected
• Up to 16 trees starting in 7.0 on Nexus 5000 and 6000
→ FabricPath interface
→ Tree 1
→ Tree 2
R
Lower SysID wins
S2
SysID 10S3
SysID 20
S1
SysID 50
S4
SysID 30
Root Election / Tree construction• Every switch advertises its system ID
& Priority
• Once all nodes have spoken Broadcast Root is elected (Highest priority then Highest Mac address wins)
• Broadcast root system will Elect & Advertise Roots for additional multicast Trees (currently only 2 trees)
• Each node will independently run SPF with Tree Root and create 2 Trees
• Since Multicast roots are advertised by Broadcast Root system (Tree 1), in case of failure of the latter both Tree 1 and Tree 2 will re-converge
S101# show fabricpath isis database detail Fabricpath IS-IS domain: default LSP database
LSPID Seq Number Checksum Lifetime A/P/O/T
S1.00-00 0x000000E2 0x0FBB 1054 0/0/0/1
Instance : 0x000000DD
Area Address : 00
NLPID : 0xC0
Hostname : S1 Length : 2
Extended IS : S202.00 Metric : 40
Extended IS : S101.00 Metric : 40
Extended IS : S102.00 Metric : 40
Extended IS : S2.00 Metric : 40
Extended IS : S201.00 Metric : 40
Capability : Device Id: 1 Base Topology
Base Topo Ftag : Graph 1: Root: S1 Primary: 1, Secondary: 0 Nickname 1Graph 2: Root: S2 Primary: 2, Secondary: 0 Nickname 2
Base Topo Trees :
Trees desired: 2 Trees computed: 2 Trees usable: 2
Base Topo Roots : Graph 1: Root Nickname: 1Graph 2: Root Nickname: 2
Version :
Version: 1 Flags: 0
Nickname :
Priority: 0 Nickname: 1 BcastPriority: 255
Nickname Migration :
Swid: 1 Sec. Swid: 0
Encapsulation
• Ethertype for FabricPath packets is 0x8903
• TTL set to 32 and is decremented at every hop. Packet is discarded when TTL reaches 0.
• FTAG: (Forwarding TAG) Used for multidestination traffic; carries the ID of the tree chosen at the FabricPath ingress switch. DRAP is responsible to keep FTAGs unique/consistent. For known unicast, FTAG carries topology ID
Nexus# show fabricpath isis topology summary
Fabricpath IS-IS domain: default FabricPath IS-IS Topology Summary
MT-0
Configured interfaces: Ethernet4/4
Number of trees: 2
Tree id: 1, ftag: 1, root system: 001b.54c2.4244, 4
Tree id: 2, ftag: 2, root system: 001b.54c2.4243, 3
Root for Tree 1, FTAG 1
Root for Tree 2, FTAG 2
Wireshark decodes FP encapsulation (tested on 1.8.3) : EditPreferencesProtocolsCFPEnable Dissector
DMAC SMAC 802.1Q Etype PayloadCRC
(new)
Outer
SA
(48)
Outer
DA
(48)
6 bits10 bits16 bits
FTAG TTLEthertype 0x8903
FP
Tag
(32)
Reverse Path Forwarding Check• RPF: check where the source switch of the packet is
and only accept packets from the interface we would have used if we were to send packet to that source
• At each FP hop RPF check is performed for multidestination traffic against source switchID + FTAG
N7K# show l2 multicast trees
(ftag/2, topo/0, Switch-id 40), uptime: 1w0d, isis
Outgoing interface list: (count: 1, '*' is the preferred interface)
* Interface Ethernet3/39, [admin distance/115] uptime: 1d23h, isis
(ftag/2, topo/0, Switch-id 30), uptime: 1w0d, isis
Outgoing interface list: (count: 1, '*' is the preferred interface)
* Interface Ethernet3/35, [admin distance/115] uptime: 02:56:04, isis
(ftag/2, topo/0, Switch-id 100), uptime: 1w0d, isis
Outgoing interface list: (count: 1, '*' is the preferred interface)
* Interface Ethernet3/39, [admin distance/115] uptime: 1d23h, isis
(ftag/1, topo/0, Switch-id 30), uptime: 02:56:06, isis
Outgoing interface list: (count: 1, '*' is the preferred interface)
* Interface Ethernet3/35, [admin distance/115] uptime: 02:56:06, isis
May also use
show fabricpath isis trees
1 2
34
root
Accept packets from 3
Accept packets from 4,1,2
Accept packets from 1,4
Packets with FTAG==2 from
switch 30 will be accepted from
interface e3/35
Packets with FTAG==1 from
switch 30 will be accepted from
interface e3/35
N7K# show fabricpath topology vlan
Topo-Description Topo-ID Configured VLAN List
-------------------------------- --------- -------------------------------------
0 0 1-99, 200-4095
1 1 100-199
N7K# show fabricpath topology interface
Interface Topo-Description Topo-ID Topo-IF-State
------------------- -------------------------------- ---------- -------------
port-channel1 0 0 Up
Ethernet6/4 0 0 Up
Ethernet6/5 0 0 Up
port-channel1 1 1 Up
Topologies• Routing table & Trees (FTAGs) are per topology
• Switch ID is shared across all topologies
• FP interface may belong to several topologies
• N7K: up to 8 topologies support starting in 6.2
• N5K/N6K: As of 5.2.1 default + 1 extra topology is supported; main use is to permit separate L2 pods to use same local vlanset
Pod 1
Vlan 100-199
Vlan 1000-1099
Pod 2
Vlan 200-299
Vlan 1000-1099
Default Topology allowed
on all FP links
FP links in Topology 0
and Topology 1
fabricpath topology 1
member vlan 100-199
!
interface Port-channel1
switchport mode fabricpath
fabricpath topology 1
Topologies + Vlans• Flood/Multicast/Broadcast trees are per-vlan, made by pruning Topology Tree
• If vlan is not present on the switch, that switch will not be part of per-vlan tree
• This may lead to connectivity issues when not all transit switches in topology have all vlans
• similar to connectivity issues caused by liberal pruning vlans off trunks with MST
• Make sure each vlan exists in every transit switch in a topology
VL10
VL10
VL30
VL20
Topology TreeVLAN 10
VL10
VL20
VL30
VLAN 20 VLAN 30
FabricPath Software Architecture & Hardware tables
on the Supervisor Engine:
• FabricPath IS-IS routing protocol process that forms the core of the FabricPath control plane
• DRAP Dynamic Resource Allocation Protocol, ensures network-wide unique and consistent Switch IDs and FTAGs
• Resolves switch id conflicts
• U2RIB Unicast Layer 2 RIB, containing the “best” unicast Layer 2 routing information
• L2FM Layer 2 forwarding manager, controls MAC address table
on the Linecards:
• U2FIB – Unicast Layer 2 FIB, managing the hardware unicast routing table
• MTM – MAC Table Manager, managing the hardware MAC address table
MAC TableSwitch Table
I/O Module
SupervisorEngine
U2FIB
FabricPath IS-IS
U2RIB L2FM
MTM
DRAP
Other HW
Hardware Drivers
Fabric Path Control Plane initialization flowS101# show processes cpu | egrep "2rib|drap|fab|l2fm|PID"
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
9169 750 16723 0 0.00% 0.00% 0.00% - l2fm
9215 1050 7843 0 0.00% 0.00% 0.00% - m2rib
9555 1050 36161 0 0.00% 0.00% 0.00% - u2rib
9556 14740 163944 0 0.00% 0.00% 0.00% - isis_fabricpath
9557 820 31339 0 0.00% 0.00% 0.00% - drap
----------------------------------------------------------------------------
S101# show fabricpath isis
Fabricpath IS-IS domain : default
System ID : 8478.ac0e.4743 IS-Type : L1 Fabric-Control SVI: Unknown
...
Process is up and running
...
Interfaces supported by Fabricpath IS-IS :
port-channel1
Ethernet6/27
Ethernet6/28
----------------------------------------------------------------------------
S101# show fabricpath switch-id
Legend: '*' - this system
SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED/
ANYCAST
--------------+----------------+------------+-----------+--------------------
* 101 8478.ac0e.4743 Primary Confirmed Yes No
...
Processes start (isis, u2rib, m2rib, drap)
System ID obtained from backplane MAC
Switch ID is obtained from DRAP
As FP interfaces links come up, hellos sent and adjacencies formed
Switch ID conflicts (if any) resolved
FP Interfaces allowed to forward data
Unicast SPF is calculated
Routes installed to U2RIB
Fabric Path Control Plane initialization flowS101# show fabricpath isis interface
Fabricpath IS-IS domain: default
Interface: port-channel1
Status: protocol-up/link-up/admin-up
…
LSP interval: 33 ms, MTU: 1500
P2P Adjs: 1, AdjsUp: 1, Priority 64
Hello Interval: 10, Multi: 3, Next IIH: 00:00:03
Level Adjs AdjsUp Metric CSNP Next CSNP Last LSP ID
1 1 1 40 60 Inactive ffff.ffff.ffff.ff-ff
Topologies enabled:
Level Topology Metric MetricConfig Forwarding
0 0 4000 no UP
1 0 40 no UP -------------------------------------------
---------------------------------
S101# show fabricpath isis adjacency
Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:
System ID SNPA Level State Hold Time Interface
S102 N/A 1 UP 00:00:25 port-channel1
S1 N/A 1 UP 00:00:28 Ethernet6/27
S2 N/A 1 UP 00:00:27 Ethernet6/28
----------------------------------------------------------------------------
S101# show fabricpath isis spf-log
Fabricpath IS-IS domain: default SPF information
SPF log for Topology 0
Total number of SPF calculations: 55
Log entry (current/max): 20/20
Ago Level Reason Count Total
1d09h 1 New LSP S201.00-00 3 0.001141
1d09h 1 Updated LSP S2.00-00 2 0.000965
…
Processes start (isis, u2rib, m2rib, drap)
System ID obtained from backplane MAC
Switch ID is obtained from DRAP
As FP interfaces links come up, hellos sent and adjacencies formed
Switch ID conflicts (if any) resolved
FP Interfaces allowed to forward data
Unicast SPF is calculated
Routes installed to U2RIB
Fabric Path Control Plane initialization flowS101# show fabricpath isis route
Fabricpath IS-IS domain: default MT-0
Topology 0, Tree 0, Swid routing table
1, L1
via Ethernet6/27, metric 40
2, L1
via Ethernet6/28, metric 40
200, L1
via Ethernet6/27, metric 80
via Ethernet6/28, metric 80
...
----------------------------------------------------------------------------
S101# show fabricpath route
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
FabricPath Unicast Route Table for Topology-Default
...
1/102/0, number of next-hops: 1
via Po1, [115/40], 1 day/s 10:01:12, isis_fabricpath-default
1/200/0, number of next-hops: 2
via Eth6/27, [115/80], 1 day/s 10:02:32, isis_fabricpath-default
via Eth6/28, [115/80], 0 day/s 10:20:17, isis_fabricpath-default
Processes start (isis, u2rib, m2rib, drap)
System ID obtained from backplane MAC
Switch ID is obtained from DRAP
As FP interfaces links come up, hellos sent and adjacencies formed
Switch ID conflicts (if any) resolved
FP Interfaces allowed to forward data
Unicast SPF is calculated
Routes installed to U2RIB
How to read
To reach switch 200 in topology 1
send packets to either Eth6/27 or
Eth6/28
FabricPath IP Multicast
• Control plane:• IGMP snooping operates as usual in FabricPath edge switches
• FabricPath IS-IS learns multicast group membership from IGMP snooping on edge switch
• FabricPath edge switch announces group interest by using GM-LSPs, creating “pruned trees” for each group on each multidestination tree
• Data plane:• Hardware selects which multidestination tree to use for each flow based on hash
function
• Once tree is selected, traffic constrained to pruned tree (FTAG) for that IP multicast group, based on MAC table lookup
Key FabricPath Multicast Processeson the Supervisor Engine:
• FabricPath IS-IS routing protocol that forms the core of the FabricPath control plane
• DRAP Dynamic Resource Allocation Protocol, extension to FabricPath IS-IS that ensures network-wide unique and consistent Switch IDs and FTAGs
• IGMP Provides IGMP snooping support for building multicast forwarding database
• M2RIB Multicast Layer 2 RIB, contains the multicast Layer 2 routing information
• L2FM Layer 2 forwarding manager, controls the MAC address table
• MFDM Multicast forwarding distribution manager, connects platform-independent control-plane processes and platform-specific processes on I/O modules
on the Linecards:
• M2FIB – Multicast Layer 2 FIB, manages the hardware multicast routing table
• MTM – MAC table manager, manages the hardware MAC address table
MAC TableSwitch Table Other HW
I/O Module
SupervisorEngine
IGMP
Hardware Drivers
M2FIB
FabricPath IS-IS
MFDM
M2RIB L2FM
MTM
DRAP
FabricPath Multicast Control Plane• IGMP/IGMP snooping tracks connected hosts/routers interest in
receiving multicast
• ISIS distributes information from igmp snooping to other FP nodes using GM-LSPs. Intermediate nodes flood GM-LSPs
• A pruned subtree is created for each group (+flood, OMF) per vlan per FTAG
MAC A MAC B
S10 S30
S1 S2
S20MAC A MAC B
S10 S30
S1 S2
S20
Root
Tree1
Root
Tree2
Vlan FTAG MAC Switches Interfaces
1 1 0100.5e01.0203 S10,S30 E1/1
Vlan FTAG MAC Switches Interfaces
1 1 0100.5e01.0203 S10,S30 E1/10,E1/30
1 2 0100.5e01.0203 S10,S30 E1/2
Vlan FTAG MAC Switches Interfaces
1 1 0100.5e01.0203 S10,S30 E1/1
1 2 0100.5e01.0203 S10,S30 E1/10,E1/30
E1/10 E1/30
E1/2
E1/1 E1/2
E1/10 E1/30
E1/1
Receiver
239.1.2.3
S10 S30
S1 S2
S20Receiver
239.1.2.3
Source
239.1.2.3
Vlan FTAG MAC Switches Interfaces
1 2 0100.5e01.0203 S10,S30 E1/2
STP & FabricPath
• No STP inside FP network
• BPDUs do not traverse FP network
(dropped at FP edge, with the exception of TCNs, see next slide)
• FP network pretends to be 1 switch from STP point of view: all FP edge
switches send BPDUs with the same Bridge ID c84c.75fa.60xx (xx is domain ID
in hex, default 00)
• Before FP ports are up, switch will use its own Bridge ID
(like STP without FP would do)
• Ports inside FP cannot be blocked, FP edge switches will always want to have
STP designated role, if superior BPDU is received such port will be blocked as
L2GW inconsistent
FabricPath
N7K# show spanning-tree interface e3/1 detail
Port 385 (Ethernet3/1) of VLAN2000 is broken (L2 Gateway Backbone Port Inconsistent)Designated root has priority 34768, address c84c.75fa.6000
…
N7K(config)# spanning-tree vlan 2000 priority 8192
22:27:28 %STP-2-L2GW_BACKBONE_UNBLOCK: L2 Gateway Backbone port inconsistency cleared unblocking port Ethernet3/1 on VLAN2000.
STP, FabricPath & TCNs• When CE STP domains are connected to multiple FP switches STP
TCN handling might be needed to maintain accuracy of MAC address tables inside CE
• Example if link CE1-CE2 goes down, link CE2-CE3 will become forwarding. Now to reach MAC B, switches inside FP need to send traffic to S5 instead of S4…
• To achieve this, FP switches when receiving a TCN from CE will propagate it to all FP switches in the network (via ISIS)
• Each FP switch will flush all remote MAC addresses learned from switches in the same STP domain as domain originating the TCN
• In addition, if FP switch is also part of the same STP domain, it will propagate TCN to the CE domain
• TCNs are not propagated to CE in domain 0 (default domain)
MAC A
MAC B
S1
S3
FabricPath
S3
S4 S5
STP Domain 1
STP Domain 2
CE1 CE3CE2
T
C
N
N7K# conf t
N7K(config)# spanning-tree domain ?
<1-1023> Domain Identifier
N7K# sh spanning-tree summary
Switch is in rapid-pvst mode
L2 Gateway Domain ID: 100...
X
Flush MACs learned from
S4,S5
T
C
N
Flush MACs learned on CE
T
C
N
T
C
N
T
C
N
T C N
Control Plane Protection• Both N7K, N6K, and N5K recognize and protect FP ISIS traffic at COPP level
• COPP needs to be updated when deploying FabricPath; standard profiles are FP-aware as of 5.2(1)
• In case of complex CE-side STP topologies (with blocking ports), usual STP safeguards are recommended (Bridge Assurance & Dispute / UDLD)
• On N7K-F1 cards: rate-limiters allow up to 4500 PPS worth of control plane FabricPath packets
Note: These 4500 PPS include also transit packets
N7K# show policy-map interface control-plane
Control Plane
service-policy input: copp-policy-strict
class-map copp-class-critical (match-any)
…
match access-group name
copp-acl-mac-fabricpath-isis
…
set cos 7
police cir 39600 kbps , bc 250 ms
module 1 :
conformed 5136527710 bytes; action: transmit
violated 0 bytes; action: drop
7KN5K# show policy-map interface control-plane class
copp-system-class-isis
Control Plane
service-policy input: copp-system-policy-default
class-map copp-system-class-isis (match-any)
match protocol isis_dce
police cir 1024 kbps , bc 4800000 bytes
conformed 751957 bytes; action: transmit
violated 0 bytes;
5K
6K
• FabricPath Overview
Benefits, Restrictions, and Configuration
• Key Concepts
Encapsulation, Trees, Topologies, STP
• Data Plane
Forwarding, Load-Balancing, MAC Learning
• vPC+
Challenges and Operation
• Troubleshooting
Verification steps, tools, and examples
Agenda
FabricPath: Forwarding Tables
• FabricPath uses 3 tables to forward frames
• MAC address table
VLAN, MAC Address, Port (local or remote), FTAG (for non-unicast)
• Switch-ID table
remote switch-ID, local next-hop interfaces (up to 16)
• Multidestination tree table
Per Tree: remote switch-ID, local next-hop/RPF interface
Tree#1 (broadcast, unknown unicast, IP multicast)
Tree#2 (IP multicast)
MAC TableSwitch Table
I/O Module
SupervisorEngine
U2FIB
FabricPath IS-IS
U2RIB L2FM
MTM
DRAP
Other HW
Hardware Drivers
Forwarding: unicast CEFP
DAKnown
ODA = L2_lookup (DA)
Forward
SAKnown
ODA = MC2 (FF)
Ftag = F(Vlan,SA/DA,…)
Ftag == Vlan2Ftag(Vlan)
ODA = MC1 (Flood2BD)
TTL = 32
unicast
OSA.SW/SubSW = local OSA.LID=LID(ingress_port)
N
Y
N
Y
Unknown unicast
Unknown source Flood to update MACs
Choose FTAG
This is meant to illustrate key decisions in forwarding, some details are abstracted away
FTAG for unicast
is topology ID
DA = Destination Address
SA = Source Address
ODA = Outer Destination Address
OSA = Outer Source Address
MC1 = 010F.FFC1.01C0
MC2 = 010F.FFC1.02C0
Forwarding: broadcast/multicast CEFP
Frame is flooded on CE side as well (based on DA)
Each egress port decides whether to encapsulate the frame in MIM depending on port type (FP,CE)
Forward
Ftag = Hash(Vlan,SA/DA,…)
TTL = 32
BC || MC
OSA.SW/SubSW = local OSA.LID=LID(ingress_port)
Broadcasts are flooded along FTAG1* Exception in vPC+
ODA = DA
Forwarding: FP->FP or FP->CE
Multicast lookups are done using VLAN, FTAG, and ODA(each multicast mac appears twice)
SubSwitchID lookups are omitted here
Remember about special LIDs (Sup, Flood, …)
FF frames are forwarded out of CE ports only when DA is locally learned
TTL<1
Forward
ODA is unicast
Dest = LID orDest = L2_table(DA,VLAN)
NY
RPF is checked against
OSA.SwID + FTAGDecrement(TTL)
RPF
Y
N
Drop
Destination =
L2_Table(Vlan, FTAG, ODA)
Pass
Fail
ODA.SwIDis local
Destination =
Sw_Table(FTAG, ODA.SwID)
N
Y
MIM packet
Load-balancing
• N7K: Unicast and Multicast load-balancing are separate
• N5K/N6K: Unified load-balancing mechanism for unicast and multicast
N7K# show fabricpath load-balance
ECMP load-balancing configuration:
L3/L4 Preference: Mixed
Hash Control: Symmetric
Rotate amount: 6 bytes
Use VLAN: TRUE
Ftag load-balancing configuration:
Hash Control: Symmetric
Rotate amount: 6 bytes
Use VLAN: TRUE
N7K# show fabricpath load-balance unicast forwarding ftag 1 switchid 30 flow l2 src-mac 001c.57ad.ecc3
dst-mac 547f.ee02.ce3c ether-type 0x800 vlan 2000 module 3
128b Hash Key generated : 1ffb80b38f02000019000715eb7b30d5
This flow selects interface Eth3/25
• Symmetric: idea is to make ab and baflows take same path by sorting addresses, before feeding them to hash
• Rotate: polarization avoidance; hash result is rotated by specified number of bytes. Number is derived from unique system MAC
Reducing impact of forwarding loops
• Transient loops might occur during convergence (as with L3 routing)
• To contain impact of these loops FabricPath uses TTL. Starting in 6.2(2), can set the initial TTL via fabricpath [multicast | unicast] ttl
• For Multidestination Trees Reverse Path Forwardingcheck performed on source switch ID
Nexus5k# show platform fwm info asic-errors 0
DROP_TTL_EXPIRED: res0 = 23 res1 = 0 [10]
Nexus7K-F2# show hardware internal errors module 4 | inc ign ttl
47 Ingress redirect due to dtag_ttl check 0000000000000002 41-44 -
MAC Address Learning
• Learning MAC addresses is not required in FabricPath Core as switching is based on Switch ID
• FP Edge switches learn local MAC addresses (behind edge ports) conventionally
• FP Edge devices learn remote addresses (behind Core-facing ports) using conversational learning
• For packets arriving from FP, source MAC (not outer SA!) is learned when destination MAC of the frame is already known on any Edge port of this switch
• No learning from broadcasts (though existing entries will be updated)
• Normal Learning from multicasts (example: HSRP address)
Conversational learning is
disabled on L3 edge
switches (when SVI is up
on FP VLAN)
This does not apply to a
case where F-series is
connected to M-series in
different VDC by external
cable
When M and F are in the
same VDC, special
handling is needed to
forward packets from
MFP core – this is
orchestrated by MCM
(mixed chassis manager)
Conversational MAC Address Learning
A B
• A sends an ARP for B (broadcast)
S1 S2 S3
A BS1 S2 S3
MAC Port
A 1
MAC Port MAC Port
• B sends ARP reply (unicast) to A
A BS1 S2 S3
MAC Port
A 1
B S3.0.1
MAC Port MAC Port
B 1
• A sends unicast packet to B
A BS1 S2 S3
MAC Port
A 1
B S3.0.1
MAC Port MAC Port
B 1
A S1.0.1
MAC Port MAC Port MAC Port
FabricPath Scale
Leaf Layer Optimized conversational learning
Spine No MAC learning (forwarding based on SWID)
VLAN 100
Leaf
VLAN 100VLAN 200 VLAN 200
SpineL2
L3
VLAN 100
Leaf
VLAN 100VLAN 200 VLAN 200
Spine
L3 Spine
Leaf Layer Optimized conversational learning
Spine Learns all MAC addresses in order to route between VLANs
Nexus5500 Nexus6000 N7K-F1 N7K-F2 N7K-F3 N7K-M
series
32K MACs 128K MACs* 16K MACs
per SoC
16K MACs
per SoC
64K MACs
per SoC
128K MACs
Potential bottleneck if
F1/F2 used in L3 Spine
FabricPath Proxy L2 Learn
• Goal: Increase MAC table size in FabricPath for F1/F2E modules
• Solution: Offload MAC learning to M-series module at L2/L3 boundary
• Prerequisites: 6.2(2) on N7K (Spine and Leaf) , M1/M2 + F2E or M1/M2 + F1
L2
L3
VLAN 100
Leaf
VLAN 100VLAN 200 VLAN 200
Spine
SoC
M1/M2 Learn All
Remote MACs
No MAC
Learning
! From default VDC (Prevents F2E/F1 from learning on multicast frames)
no hardware fabricpath mac-learning module <x> [port-group <y>]
! From fabricpath VDC (prevents F2E/F1 from learning remote MACs)
no mac address-table fabricpath remote-learning
Configuration
! If you are using F2 for Leaf core ports to prevent learning from
broadcast/multicast
no hardware fabricpath mac-learning module <x> [port-group <y>]
FabricPath MAC Learning Changes: Why?
• M-Series MAC tables contain VLAN, MAC, and port index (no concept of SWID, SSWID, LID in M-Series MAC table)
• For FP MACs, the destination SWID is mapped to an internal gateway port-channel (GPC) index which is programmed in the M-series MAC table
• FP SoC will translate GPC to SWID before sending out FP port.
• Challenge: No way for FP SoC to determine LID for packet from M-Series module if MAC is not present in local MAC table. Therefore, packet from M-Series sent out FP with flood LID.If FP SoC on destination switch has not learned MAC, then packet will be flooded out local CE ports
• Solution: Sync MACs on CE SoC to FP SoC.
FP
SoC
FP
SoC
CE
SoC
VLAN 100
M1/M2
L2
L3
FP
SoC
VLAN 200
CE
SoC
X, Y, Z A, B, C
S1
S101 S201
VLAN MAC Index
200 A gpc1
GPC SWID
gpc1 S201
M sends
frame to gpc1
F translates
frame to
SWID 201, LID
FFFFMAC miss,
causes flood to
local CE ports
FabricPath MAC Learning Changes
FP
SoC
FP
SoC
CE
SoC
CE
SoC
Learn all MACs on CE
ports. Learn remote
MACs via
conversational learning
No MACs
Learned
FP
SoC
FP
SoC
CE
SoC
CE
SoC
Learn all MACs on CE
ports. Learn remote
MACs via
conversational learning
X, Y, Z X, Y, Z
Sync local CE
MACs to FP SoC
Learns MAC
X,Y,Z
Learns MAC
A,B,C
6.1(2) for F2/F2E
6.2(2) in F1
• To support L2 proxy learning, MACs learned on CE ports will be synced to all SoCs
A, B, CA, B, C
• FabricPath Overview
Benefits, Restrictions, and Configuration
• Key Concepts
Encapsulation, Trees, Topologies, STP
• Data Plane
Forwarding, Load-Balancing, MAC Learning
• vPC+
Challenges and Operation
• Troubleshooting
Verification steps, tools, and examples
Agenda
Fabric Path
VPC+: Why, What and How (1)
• Goal: provide redundant, active-active L2 links to separate FP switcheswith active-active HSRP
• Challenge 1: depending on the path the packet AB takes, switch S3 will learn MAC A behind S1 or S2 (or MAC will be moving)
• Solution: introduce Emulated Switch S100 to represent devices behind VPCs: MAC A will appear behind S100 in S3 MAC address table. HSRP MAC is advertised with emulated switch as a source – taking advantage of VPC+ multipathing
44
S1 S2
S3
MAC A
MAC B
S100
S3# show mac address-table address 0000.0000.000a
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+-------------------
3000 0000.0000.000a dynamic 30 F F 100.0.0
S3# show fabricpath route switchid 100
1/100/0, number of next-hops: 2
via e1/1, [115/20], 1 day/s 05:56:40, isis_fabricpath-default
via e1/2, [115/20], 1 day/s 05:56:38, isis_fabricpath-default
S3# show fabricpath switch-id
SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED
----------+----------------+------------+-----------+--------------------
1 0000.0000.1001 Primary Confirmed Yes No
2 0000.0000.3002 Primary Confirmed Yes No
*3 0000.0000.3003 Primary Confirmed Yes No
100 0000.0000.1010 Primary Confirmed No Yes
Fabric PathVPC VPC+
• To enable VPC+ an Emulated Switch ID must be configured in VPC domain on both peers (must be the same on both peers and globally unique). ES represents ALL VPC+ channels of the domain
• Peer-link and VPC+ ports must be fabric-path capable
• Peer-link is FP interface(no STP, only FP vlans are carried, VPC check is no more ).VPC+ channels are CE
• VPC+ domain must be the root for CE STP, otherwise VPC+ channels will be blocked as L2GW inconsistent
• FP switches use same STP bridge ID but peer-switch is still recommended
S1 S2
S100
S1# show vpcvPC domain id : 2
vPC+ switch id : 100Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
vPC fabricpath status : peer is reachable through fabricpath
...
vPC role : primary
Number of vPCs configured : 0
...
Fabricpath load balancing : Disabled
Port Channel Limit : limit to 244
vpc domain 2fabricpath switch-id 100
Fabric Path
HSRP (and VRRP) in VPC+
• HSRP when enabled on VPC+ peers uses Emulated Switch ID as a source switch and thus benefits from VPC+ multipathing
• Control-plane-wise one peer will be active and other will be standby, but data-plane-wise both peers will be forwarding traffic (same as in VPC)
• FabricPath devices will have ECMP route to Emulated Switch
• CE devices will have HSRP VMAC pointing to a port-channel
• If only HSRP active-active is required VPC+ channels are optional
S1 S2
S3
S100
S3# show mac address-table vlan 100 address 0000.0c9f.f064VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
100 0000.0c9f.f064 dynamic 0 F F 100.0.65535
s3# show fabricpath route switchid 1001/100/0, number of next-hops: 2
via e1/1, [115/20], 1 day/s 05:56:40, isis_fabricpath-defaultvia e1/2, [115/20], 1 day/s 05:56:38, isis_fabricpath-default
CE1# show mac address-table vlan 100 address 0000.0c9f.f064VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 100 0000.0c9f.f064 dynamic 0 F F Po1
CE1
VPC+: Why, What and How (2)
• Challenge 2: flooded packets from A (with OSA of S100) might come to S3 from S1 or from S2, but RPF can only be 1 interface
• Solution: S1 and S2 advertise to S3 (via ISIS TLV) an affinity to single FTAG each, S3 will program RPF according to affinity. Multidestination traffic coming from VPC+ will be set to use FTAG 1 for VPC leg on S1 and FTAG 2 for VPC leg on S2
47
S1 S2
S3
MAC A
MAC B
S100
1/1 1/2
Affinity
FTAG1
Affinity
FTAG2
Use FTAG1 Use FTAG2
RPF
FTAG1,S100
RPF
FTAG2,S100
S3# show fabricpath route switchid 100FabricPath Unicast Route Table1/100/0, number of next-hops: 2
via Eth1/1, [115/40], 11 day/s 00:59:35, isis_fabricpath-defaultvia Eth1/2, [115/40], 11 day/s 01:03:27, isis_fabricpath-default
S3# show fabricpath isis database detail | i Affinity|Host|NumgHostname : S1 Length : 2
Affinity :Nickname: 100 Numgraphs: 1 Graph-id: 1
Hostname : S2 Length : 2Affinity :Nickname: 100 Numgraphs: 1 Graph-id: 2
S3# show l2 multicast trees
(ftag/2, topo/0, Switch-id 100), uptime: 1d01h, isisOutgoing interface list: (count: 1, '*' is the preferred interface)
* Interface Ethernet1/2, [admin distance/115] uptime: 1d01h, isis
(ftag/1, topo/0, Switch-id 100), uptime: 6d00h, isisOutgoing interface list: (count: 1, '*' is the preferred interface)
* Interface Ethernet1/1, [admin distance/115] uptime: 6d00h, isis
VPC+: Why, What and How (3)
• Challenge 3: multidestination packets from FP to CE need to be load-balanced too
• Solution: S1 and S2 will each be ‘designated forwarder’ for FTAG of their affinity: traffic for FTAG of affinity will be forwarded out of VPC and other FTAG traffic will be forwarded by peer
S1 S2
S3
MAC A
MAC B
S100
1/1 1/2
Affinity
FTAG1
Affinity
FTAG2
DF: FTAG1 DF: FTAG2
RPF
FTAG1,S100
RPF
FTAG2,S100
Po101
S1# show vpcvPC domain id : 100vPC+ switch id : 100...vPC Peer-link status---------------------------------------------------------------------1 Po1 up 2000-2001,3000-3001
vPC status-------------------------------------------------------------------------id Port Status Consistency Reason Active vlans vPC+ Attrib-- ---------- ------ ----------- ------ ------------ -----------101 Po101 up success success 10 DF: Yes
vPC status-------------------------------------------------------------------------id Port Status Consistency Reason Active vlans vPC+ Attrib-- ---------- ------ ----------- ------ ------------ -----------101 Po101 up success success 10 DF: Partial
vpc domain 100
fabricpath multicast load-balance
Fabric Path
VPC+: Prevention of Duplicate Packets
• How is packet received from VPC+ and flooded on S1 prevented from being flooded on S2 to same VPC+ again?
• N7K-F1 linecards:
Each VPC+ will have its own sub-switch ID. Mac addresses will be learned behind <es_id>.<subsw_id>.<lid>, for example 100.11.65535(emulated switch 100, sub-switch 11, LID 65535). S2 will recognize ES + SubSwitch tuple as its own port and will not flood the frame back to VPC
• N7K-F2, N7K-F3 linecards & N5K, N6K:
By default same as above, as below with ‘fabricpath multicast load-balance’
Each VPC+ peer will be forwarding only for 1 FTAG and traffic coming from other peer will have different FTAG. For example (previous slide) flooded packet coming from S1 will have FTAG1, but S2 will only flood FTAG2 packets out of the VPC
S1 S2
X
Required for FEX FP with N7K-F2
Fabric Path
VPC Failover
• VPC+ member link goes down• Traffic diverted over Peer-Link
• Peer-Link goes down (but Peer-Keepalive up)• Primary: No action
• Secondary: Bring down VPC+ channels
• Stop advertising reachability to Emulated Switch
• Dual active is much less likely than with normal VPC: if Peer-Link and Peer-Keepalive go down, but peer is reachable via FP – secondary will not become primary
S1 S2
S3
S100
S3# show fabricpath route switchid 100
1/100/0, number of next-hops: 1
via e1/1, [115/20], 1 day/s 07:14:24, isis_fabricpath-default
S1# show vpcvPC domain id : 2
vPC+ switch id : 100
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
vPC fabricpath status : peer is reachable through fabricpath
L2
L3
Fabric Path
Anycast HSRP
• Goal: provide N-gateway solution to increase redundancy and bandwidth
• Alternatives: 1. vPC/vPC+ provides 2 active gateways. Failure of a single gateway reduces available inter-vlan traffic by
half
2. GLBP allows more than 2 active gateways. Drawbacks:
• No ECMP load-balancing since a single virtual MAC is assigned to a single SwitchID
• Non-deterministic distribution of virtual MAC addresses (hard to troubleshoot)
• Solution: Anycast HSRP
Active Standby Listen Listen
All 4 devices actively
routing traffic for the HSRP
virtual MAC
L2
L3
Anycast HSRP
• The HSRP virtual MAC is bond to an Anycast SwitchID (ASID)
• ASID uses similar concept to vPC+ ES, where each Anycast gateway advertises the ASID via new Anycast HSRP Sub-TLV
• Each Anycast gateway will actively route traffic for the HSRP virtual MAC
feature interface-vlan
feature hsrp
interface Vlan100
ip address 10.1.100.1/24
hsrp version 2
hsrp 100
ip 10.1.100.254
interface Vlan101
ip address 10.1.101.1/24
hsrp version 2
hsrp 101
ip 10.1.101.254
hsrp anycast 1 ipv4
switch-id 1000
vlan 100-101
no shutdown
Configure HSRP under the
interface - HSRP version2
required
Configured the ASID for this
anycast bundle and
associate vlans
S1 S2 S3 S4
Code RequirementN7K
• 6.2(6)
N5K/N6K
• 6.0(2)N2(1) (SubTLV only)
• 7.0(0)N1(1)
4 Equal Cost
Routes to ASID
ASIDASIDASIDASID
Anycast HSRP
S202# show mac address-table dynamic
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 101 0000.0c9f.f065 dynamic 10 F F 1000.0.65535
* 100 0000.0c9f.f064 dynamic 10 F F 1000.0.65535
S202# show fabricpath isis database detail | i "LSPID|00-00|Nickname: 1000"
LSPID Seq Number Checksum Lifetime A/P/O/T
S1.00-00 0x00000100 0x815E 762 0/0/0/1
Nickname: 1000 Numgraphs: 2 Graph-id: 1, 2
S2.00-00 0x00000103 0xC618 776 0/0/0/1
Nickname: 1000 Numgraphs: 2 Graph-id: 1, 2
...
S202# show fabricpath route switchid 1000
...
1/1000/0, number of next-hops: 4
via Eth1/6, [115/40], 0 day/s 03:00:18, isis_fabricpath-default
via Eth1/7, [115/40], 0 day/s 03:02:55, isis_fabricpath-default
via Eth1/8, [115/40], 0 day/s 03:01:08, isis_fabricpath-default
via Eth1/9, [115/40], 0 day/s 03:03:45, isis_fabricpath-default
Each switch sends ISIS
TLVs advertising ASID
ECMP routes built toward
ASID to increase
redundancy and
bandwidth
HSRP Active Hellos are
sent out with a OSA of the
ASID and SA of the virtual
MAC
• FabricPath Overview
Benefits, Restrictions, and Configuration
• Key Concepts
Encapsulation, Trees, Topologies, STP
• Data Plane
Forwarding, Load-Balancing, MAC Learning
• vPC+
Challenges and Operation
• Troubleshooting
Verification steps, tools, and examples
Agenda
S202S201S102S101
S2S1
ES S100 ES S200
FP Vlans 100-199
A B C D
FabricPath: Configuration
install feature-set fabricpath
feature-set fabricpath
vlan 100-199
mode fabricpath
fabricpath switch-id 101
vpc domain 100
fabricpath switch-id 100
fabricpath multicast load-balance
! Fabricpath core ports
interface Ethernet6/4 - 5
switchport mode fabricpath
! Peer-link
interface port-channel1
switchport mode fabricpath
! vPCs are CE ports (mode access or mode trunk)
interface port-channel20
switchport
switchport mode trunk
vpc 20
Best practice to manually
configure switch-id
! S1
fabricpath domain default
root-priority 255
! S2
fabricpath domain default
root-priority 254
Configure roots for
FTAG 1 and 2
S202S201S102S101
S2S1
ES S100 ES S200
FP Vlans 100-199
A B C D
FabricPath: Health Check
S101# sh sys internal plugin info global | begin l2mp | head lines 5
Feature-set id: 2, name: l2mp
vdc: 1 state: PLUGIN_ENABLED_STATE
vdc: 2 state: PLUGIN_ENABLED_STATE
vdc: 3 state: PLUGIN_ENABLED_STATE
FabricPath plugin in good
state
Services running for URIB,
MRIB, DRAP, ISIS
CPU levels are reasonable
Memory below limits
S101# show system internal sysmgr service all | i 2rib|drap|fabric|PID
Name UUID PID SAP state Start count Tag Plugin ID
isis_fabricpath 0x41000243 6475 436 s0009 1 N/A 1
drap 0x0000024E 6476 448 s0009 1 N/A 1
m2rib 0x00000250 6435 449 s0009 1 N/A 1
u2rib 0x00000254 6474 452 s0009 1 N/A 1
S101# show processes cpu | i 2rib|drap|fabric|PID
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
6435 410 335 1 0.00% 0.00% 0.00% - m2rib
6474 170 735 0 0.00% 0.00% 0.00% - u2rib
6475 690 3764 0 0.00% 0.00% 0.00% - isis_fabricpath
6476 200 725 0 0.00% 0.00% 0.00% - drap
S101# show processes memory | i 2rib|drap|fabric|PID
PID MemAlloc MemLimit MemUsed StackBase/Ptr Process
6435 11149312 923422860 273965056 ffd8cb40/ffffffff m2rib
6474 3657728 564849190 262389760 ffbc5b80/ffffffff u2rib
6475 30515200 814058995 479059968 ff8eed50/ffffffff isis_fabricpath
6476 3067904 619628416 262160384 ffa58950/ffffffff drap
FabricPath: Health CheckS101# show fabricpath isis
System ID : 8478.ac0e.4743 IS-Type : L1 Fabric-Control SVI: Unknown
Process is up and running
Interfaces supported by Fabricpath IS-IS :
port-channel1
Ethernet6/27
Ethernet6/28
S101# show fabricpath topology vlan active
Topo-Description Topo-ID Active VLAN List
-------------------------------- --------- -------------------------
0 0 100-199
ISIS is running
system ID is accurate
Interface list matches configuration
Active Vlans match configuration
Interfaces in Up/Ready state
Adjacencies established
Adjacencies stable
S101# show fabricpath isis interface brief
Fabricpath IS-IS domain: default
Interface Type Idx State Circuit MTU Metric Priority Adjs/AdjsUp
--------------------------------------------------------------------------------
port-channel1 P2P 3 Up/Ready 0x01/L1 1500 40 64 1/1
Ethernet6/27 P2P 1 Up/Ready 0x01/L1 1500 40 64 1/1
Ethernet6/28 P2P 2 Up/Ready 0x01/L1 1500 40 64 1/1
S101# show fabricpath isis adjacency detail
Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:
System ID SNPA Level State Hold Time Interface
S102 N/A 1 UP 00:00:25 port-channel1
Up/Down transitions: 1, Last transition: 3w5d ago
Circuit Type: L1
Topo-id: 0, Forwarding-State: UP
S202S201S102S101
S2S1
ES S100 ES S200
FP Vlans 100-199
A B C D
FabricPath: Health Check
No growing errors on interfaces
All switches and ES are seen and in
confirmed state
S101# show fabricpath isis traffic port-channel 1
Fabricpath IS-IS domain: default
Fabricpath IS-IS Traffic for port-channel1:
PDU Received Sent RcvAuthErr OtherRcvErr ReTransmit
P2P-IIH 734 733 0 0 n/a
CSNP 2 1 0 0 n/a
PSNP 113 113 0 0 n/a
LSP 131 134 0 0 0
S101# show fabricpath switch-id
FABRICPATH SWITCH-ID TABLE
Legend: '*' - this system
'[E]' - local Emulated Switch-id
'[A]' - local Anycast Switch-id
Total Switch-ids: 10
=========================================================================
SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED/
ANYCAST
--------------+----------------+------------+-----------+----------------
1 8478.ac0e.4742 Primary Confirmed Yes No
2 8478.ac5b.2b42 Primary Confirmed Yes No
[E] 100 8478.ac0e.4743 Primary Confirmed No Yes
100 8478.ac5b.2b43 Primary Confirmed No Yes
* 101 8478.ac0e.4743 Primary Confirmed Yes No
102 8478.ac5b.2b43 Primary Confirmed Yes No
200 547f.eed6.70fc Primary Confirmed No Yes
200 547f.eedb.7e7c Primary Confirmed No Yes
201 547f.eed6.70fc Primary Confirmed Yes No
202 547f.eedb.7e7c Primary Confirmed Yes No
S202S201S102S101
S2S1
ES S100 ES S200
FP Vlans 100-199
A B C D
FabricPath: Unicast Example (MAC)
S202S201S102S101
S2S1
ES S100 ES S200
FP Vlans 100-199
A B C D
S101# show mac-address-table address-table vlan 100
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 100 0000.0000.000a dynamic 0 F F Po30
100 0000.0000.000d dynamic 0 F F 200.0.0
S101# show hardware mac address-table 6 vlan 100
FE | Valid| PI| BD | MAC | Index| Stat| SW | ... | SWID| SSWID| LID
| | | | | | ic | | ... | | |
---+------+---+------+---------------+-------+-----+-----+ ... |-----|------|-------
7 1 1 245 0000.0000.000a 0x00408 0 0x089 0x064 0x00b 0x00408
7 1 0 245 0000.0000.000d 0x00000 0 0x009 0x0c8 0x000 0x00000
7K
vPC30 vPC40
MACs are present in software
MAC table
Use Platform Dependent
commands to check hardware
MAC table
On S101, MAC D matches
software remote address (200.0.0)
MAC A has local SWID/SSWID
100.11 with LID 0x408
Hex SWID/SSWID
0xc8 0x00 = 200 0
0x64 0x0b = 64 11
S101# show system internal pixm info ltl 0x408
PC_TYPE PORT LTL RES_ID LTL_FLAG CB_FLAG MEMB_CNT
------------------------------------------------------------------------------
Normal Po30 0x0408 0x1600001d 0x00000000 0x00000002 1
7K
LID 0x408 maps to local Po30
FabricPath: Unicast Example (MAC)
MACs are present in software
MAC table
Use Platform Dependent
commands to check hardware
MAC table
On S202, MAC A matches
software remote address
(100.11.65535)
MAC A has local SWID/SSWID
200.0 with LID 0x15 (0x15 = 21)
S202S201S102S101
S2S1
ES S100 ES S200
FP Vlans 100-199
A B C D
vPC30 vPC40
S202# show mac address-table vlan 100
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 100 0000.0000.000a dynamic 0 F F 100.11.65535
* 100 0000.0000.000d dynamic 0 F F Po40
S202# show platform fwm info lif port-channel 40 | i local_id
Po40 pd: local_id 21 endnode_id 0 endnode_id_alloced 1 vif_id 0
5K
S202# show platform fwm info hw-stm | i HW|VLAN|_|---|000a|000d
HW STM Contents
dleft loc - bucket_type:line:bucket_number
misc - learn_type:ecc:valid:fcf
cdce format - ig:ul:switch_id:subswitch_id:end_node_id:pbp_idx:local_id
VLAN MAC Address Port loc misc cdce
------+----------------+--------------+--------+-------+--------------------
1.100 0000.0000.000d Po40 1:1111:0 1:0:1:0 2.0.c8.0.0.15 (e:0)
1.100 0000.0000.000a l2mp-nh 1:2918:0 1:0:1:0 2.0.64.b.ff.ff (e:0)
5K
LID 21 maps to local Po40
FabricPath: What command comes from where
MAC TableSwitch Table
I/O Module
SupervisorEngine
U2FIB
FabricPath IS-IS
U2RIB L2FM
MTM
DRAP
Other HW
Hardware Drivers
show fabricpath switch
show fabricpath isis switch
show fabricpath conflict all | link | switch | transitions
show fabricpath isis route
show mac address-table
slot <> show fabricpath unicast routes vdc
slot <> show hardware internal forwarding inst <> table <>
slot <> show hardware mac address-table
show fabricpath isis interface
show fabricpath isis adjacency
show fabricpath isis database
show fabricpath route
show platform fwm info l2mp route ftag <> switch <> hw
show platform fwm info hw-stm
7K
5K
7K
6K
5K 6K
FabricPath: Unicast Example (SWID)
Route for destination SWID present in
ISIS table and U2RIB
S101# show fabricpath isis route
Fabricpath IS-IS domain: default MT-0
Topology 0, Tree 0, Swid routing table
...
200, L1
via Ethernet6/27, metric 80
via Ethernet6/28, metric 80
S101# show fabricpath isis database detail
Fabricpath IS-IS domain: default LSP database
LSPID Seq Number Checksum Lifetime A/P/O/T
S201.00-00 0x00000006 0xF8A7 957 0/0/0/1
Hostname : S201 Length : 4
Capability : Device Id: 201 Base Topology
Affinity :
Nickname: 200 Numgraphs: 1 Graph-id: 1
Nickname :
Priority: 0 Nickname: 201 BcastPriority: 64
Priority: 0 Nickname: 200 BcastPriority: 0
S202.00-00 0x00000007 0x5F3B 884 0/0/0/1
Hostname : S202 Length : 4
Capability : Device Id: 202 Base Topology
Affinity :
Nickname: 200 Numgraphs: 1 Graph-id: 2
Nickname :
Priority: 0 Nickname: 202 BcastPriority: 64
Priority: 0 Nickname: 200 BcastPriority: 0
S101# show fabricpath route switchid 200
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
...
1/200/0, number of next-hops: 2
via Eth6/27, [115/80], 0 day/s 00:21:58,
isis_fabricpath-default
via Eth6/28, [115/80], 0 day/s 00:21:58,
isis_fabricpath-default
S202S201S102S101
S2S1
ES S100 ES S200
FP Vlans 100-199
A B C D
vPC30 vPC40
FabricPath: Unicast Example (SWID)
Use Platform Dependent commands to
verify route for destination SWID is
present in hardware
On N7K, first attach to appropriate
module via “attach module x”S202# show platform fwm info l2mp route ftag 1 swid 100
-------------------------------------------------------------------
l2mp_route[0x99f23ac]
route_type: 10 (0xa) merge_version: 1 (0x1)
iic interface: Eth1/7 (0x1a006000)
ftag: 1 (0x1) switchid: 100 (0x64)-> l2mp_nexthop[0x8944dc4]
num_paths: 2
nh[1]: Eth1/7 (0x1a006000)
nh[2]: Eth1/8 (0x1a007000)
5K
S202S201S102S101
S2S1
ES S100 ES S200
FP Vlans 100-199
A B C D
vPC30 vPC40
module-6# show fabricpath unicast routes vdc 3 ftag 1 switchid 200
Route in VDC 3
--------------------------------------------------------------------------------
FTAG | SwitchID | SubSwitchID | Loc/Rem | RPF | RPF Intf | Num Paths | Merge V
--------------------------------------------------------------------------------
0001 | 0200 | 0000 | Remote | Yes | Eth6/27 | 2 | 1
--------------------------------------------------------------------------------
...
PD Information for ECMP:
Common Info
--------------------------------
AMM key : 0x6000024
--------------------------------
Next Hop | Interface | LID
--------------------------------
0 | Eth6/27 | 0000006a
1 | Eth6/28 | 0000006b
7K
Two equal costs routes via
Eth6/27 and Eth6/28. RPF interface Eth6/27
Two equal costs routes via
Eth1/7 and Eth1/8. RPF interface Eth1/7
FabricPath: what comes from where
show fabricpath isis switch
show fabricpath mroute
show ip igmp snooping groups
show fabricpath isis topology summary
show fabricpath isis tree
show fabricpath isis database mgroup detail
show l2 multicast trees
MAC TableSwitch Table Other HW
I/O Module
SupervisorEngine
IGMP
Hardware Drivers
M2FIB
FabricPath IS-IS
MFDM
M2RIB
MTM
DRAP
show forwarding distribution l2 multicast [vlan <>] 7K
L2FM
S101# show fabricpath isis topology summary
FabricPath IS-IS Topology Summary
Fabricpath IS-IS domain: default
MT-0
Configured interfaces: port-channel1 Ethernet6/27 Ethernet6/28
Max number of trees: 2 Number of trees supported: 2
Tree id: 1, ftag: 1, root system: 8478.ac0e.4742, 1
Tree id: 2, ftag: 2 [transit-traffic-only], root system: 8478.ac5b.2b42, 2
Ftag Proxy Root: 8478.ac0e.4742
FabricPath: Multidestination (Flood)
S202S201S102S101
S2S1
ES S100 ES S200
FP Vlans 100-199
A B C D
vPC30 vPC40
Check the topology roots for
each FTAG
Map out the active links
How to read: on which
interface in given FTAG will this
switch accept multidestination
traffic from given switch
Example: accept traffic from
switch 100 on E6/19 in FTAG1
S101# show fabricpath isis trees
MT-0
Topology 0, Tree 1, Swid routing table
1, L1
via Ethernet6/27, metric 0
2, L1
via Ethernet6/27, metric 20
102, L1
via Ethernet6/27, metric 40
200, L1
via Ethernet6/27, metric 40
201, L1
via Ethernet6/27, metric 40
202, L1
via Ethernet6/27, metric 40
S1# show fabricpath isis trees
MT-0
Topology 0, Tree 1, Swid routing table
2, L1
via port-channel1, metric 20
100, L1
via Ethernet6/19, metric 40
101, L1
via Ethernet6/19, metric 40
102, L1
via Ethernet6/20, metric 40
200, L1
via Ethernet6/21, metric 40
201, L1
via Ethernet6/21, metric 40
202, L1
via Ethernet6/22, metric 40
Repeat on each
switch to map out
complete
forwarding tree
(FTAG 1)
S101# show fabricpath mroute vlan 100 flood
(vlan/100, *, *), Flood, uptime: 02:01:06, isis
Outgoing interface list: (count: 5)
Switch-id 1, uptime: 02:01:06, isis
Switch-id 2, uptime: 02:01:06, isis
Switch-id 102, uptime: 01:59:40, isis
Switch-id 201, uptime: 02:01:06, isis
Switch-id 202, uptime: 02:01:06, isis
FabricPath: Multidestination (Flood)
Flood entry – traffic that will be flooded to all active ports
(minus receiving port) in a Vlan
(remember about dynamic pruning)
Ignore multiple appearances of the same interface
(interface appears 1 per destination switch)
S101# show fabricpath mroute vlan 100 flood resolved
(ftag/2, vlan/100, *, *), Flood, uptime: 02:01:32, isis
Outgoing interface list: (count: 5)
Interface Ethernet6/28, Switch-id 1, uptime: 02:01:31, isis
Interface Ethernet6/28, Switch-id 2, uptime: 02:01:31, isis
Interface Ethernet6/28, Switch-id 102, uptime: 02:00:07, isis
Interface Ethernet6/28, Switch-id 201, uptime: 02:01:31, isis
Interface Ethernet6/28, Switch-id 202, uptime: 02:01:31, isis
(ftag/1, vlan/100, *, *), Flood, uptime: 02:01:32, isis
Outgoing interface list: (count: 5)
Interface Ethernet6/27, Switch-id 1, uptime: 02:01:31, isis
Interface Ethernet6/27, Switch-id 2, uptime: 02:01:31, isis
Interface Ethernet6/27, Switch-id 102, uptime: 02:00:07, isis
Interface Ethernet6/27, Switch-id 201, uptime: 02:01:31, isis
Interface Ethernet6/27, Switch-id 202, uptime: 02:01:31, isis
S202S201S102S101
S2S1
ES S100 ES S200
FP Vlans 100-199
A B C D
vPC30 vPC40
S202S201S102S101
S2S1
ES S100 ES S200
FP Vlans 100-199
A B C D
FabricPath: IP Multicast
vPC30 vPC40
*,G from local IGMP snooping
Local IGMP/snooping entries are
redistributed into FP
L2 multicast prune subtrees built
on each FP switch
S101 hashes multicast to FTAG 1
(remember vPC+ affinity)
Multicast Receiver
Multicast Sender
S202# show ip igmp snooping groups vlan 100
Type: S - Static, D - Dynamic, R - Router port, F - Fabricpath core port
Vlan Group Address Ver Type Port list
100 */* - RF Eth1/7
RF Eth1/8
100 239.1.1.1 v2 D Po40
S101# show fabricpath isis database mgroup detail | egrep "LSPID|Group|00-01"
LSPID Seq Number Checksum Lifetime A/P/O/T
S201.00-01 0x00000093 0xEA2C 1092 0/0/0/1
Group-Address : IP Multicast : Vlan : 100 Groups : 1
Group : 239.1.1.1 Sources : 0
S202.00-01 0x00000090 0xBD66 709 0/0/0/1
Group-Address : IP Multicast : Vlan : 100 Groups : 1
Group : 239.1.1.1 Sources : 0
S101# show fabricpath mroute vlan 100
(vlan/100, 0.0.0.0, 239.1.1.1), uptime: 20:35:57, isis
Outgoing interface list: (count: 2)
Switch-id 201, uptime: 20:35:57, isis
Switch-id 202, uptime: 20:35:57, isis
S101# show fabricpath mroute vlan 100 ftag 1
(ftag/1, vlan/100, 0.0.0.0, 239.1.1.1), uptime: 20:47:34, isis
Outgoing interface list: (count: 2)
Interface Ethernet6/27, Switch-id 201, uptime: 22:26:18, isis
Interface Ethernet6/27, Switch-id 202, uptime: 22:26:18, isis
Remember
RPF check
FabricPath: IP Multicast
S202S201S102S101
S2S1
ES S100 ES S200
FP Vlans 100-199
A B C D
vPC30 vPC40
vPC+ in partial status which
means multidestination traffic is
load-balanced between vPC peers
S201 has affinity for FTAG 1
S202 has affinity for FTAG 2
S201 will forward this frame
Multicast Receiver
Multicast Sender
S202# show vpc 40
vPC status
---------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans vPC+ Attrib
-- ---------- ------ ----------- ------ ------------ -----------
40 Po40 up success success 100-199 DF: Partial,
FP MAC:
200.0.0
S201# show fabricpath isis database detail S201.00-00 | sec Affinity
Affinity :
Nickname: 200 Numgraphs: 1 Graph-id: 1
S201# show fabricpath isis database detail S202.00-00 | sec Affinity
Affinity :
Nickname: 200 Numgraphs: 1 Graph-id: 2
QUIZ
Both S201 and S202 receive multicast
stream, who forwards out vPC 40? x
FabricPath: Hardware Multicast MAC
• Multicast MACs are stored differently from usual 0100.5exx.xxxx
F1
• Each mac appears twice: once per FTAG, use ‘show hard internal forwarding … table mac’ to find which is which
F2 module-6# show hardware mac address-table vlan <vlan> vdc <vdc> fe <fe>
FE | Valid| PI| BD | MAC | Index| Stat| SW | Modi| Age| ... | SWID| SSWID| LID
| | | | | | ic | | fied|Byte| ... | | |
---+------+---+------+---------------+-------+-----+-----+-----+----+ ... |-----|------|-------
7 1 1 245 0000.0000.000a 0x00408 0 0x009 1 199 ... 0x064 0x00b 0x00408
7 1 0 245 0000.0000.000d 0x00000 0 0x009 1 199 ... 0x0c8 0x000 0x00000
7 1 0 245 4180.0f01.0101 0x07fd8 1 0x000 0 0 ... 0x000 0x000 0x07fd8
7 1 0 245 4180.0f01.0101 0x07fda 1 0x000 0 0 ... 0x000 0x000 0x07fda
module-4# show hardware mac address-table vlan <vlan> vdc <vdc> fe <fe>
FE | Valid| PI| BD | MAC | Index|...| PV | RD| NN| UC|PI_E8| SWID| SSWID| LID
| | | | | |...| | | | | | | |
---+------+---+------+---------------+-------|...|----|---|---|---|-----|-----|------|-------
4 1 0 52 0100.ef01.0203 0x07ffb ... 0x00 0 0 0 0 0x000 0x000 0x07ffb
4 1 0 52 0100.ef04.0506 0x07ffb ... 0x00 0 0 0 0 0x000 0x000 0x07ffb
4 1 0 52 0100.ef01.0203 0x07ffb ... 0x00 0 0 0 0 0x000 0x000 0x07ffb
4 1 0 52 0100.ef04.0506 0x07ffb ... 0x00 0 0 0 0 0x000 0x000 0x07ffb
Looking back in time
• show fabricpath isis internal event-history adjacencyevents related to adjacencies (up/down/etc)
• show fabricpath isis internal event-history uribFP events related to URIB updates (for example to see whole history for given switch ID)
• show fabricpath isis internal event-history eventsOverall FP event history: DRAP interactions, switch additions, removals, SPF-related events
• show fabricpath isis internal event-history drapswitch ID, FTAG related events
Troubleshooting Tools: Pong
• Pong can be equated to L2Ping + L2TraceRoute
• Depends on IEEE 1588v2 HW support F-series, N5500, and N6000 all support PTP, but N5K/N6K at present doesn’t support pong
• Works by sending 2 types of packets: 1 packet to store timestamps at each hop and 2nd to collect stored timestamps
S101# pong destination-swid 2 destination-mac 8478.ac5b.2b42 vlan 100 details
Legend (*) - software delay(not hardware latency)
(#) - reverse path
(NA) - not available
--- ------------------------- --------------------------
Hop System-mac (switch-id) Switching time
(sec, nsec)
--- ------------------------- --------------------------
1 84-78-ac-0e-47-43 ( 101) 5588 353692400
2 84-78-ac-0e-47-42 ( 1) 5588 353692896
3 84-78-ac-0e-47-42 ( 1) 5588 353698488
4 84-78-ac-5b-2b-42 ( 2) 5588 415486312
5 84-78-ac-5b-2b-42 ( 2) 5588 930158536
6 84-78-ac-0e-47-42 ( 1) 5588 868372664
7 84-78-ac-0e-47-42 ( 1) 5588 868378248
8 84-78-ac-0e-47-43 ( 101) 5588 868378768
Round trip time: 0sec 14144 nsec
Send frame to SWID 2
(SysID of SWID 2 = 8478.ac5b.2b42)
* By default, Frame sent on VLAN 1. Be
sure to specify appropriate VLAN
Egress from SWID 101
Ingress SWID 1
Egress SWID 1
Etc..
MACs that can be reached:
- SysID or static
Not supported over ECMP on F2
Troubleshooting Tools: FPOAM
• FPOAM (Fabricpath Operations Administration and Management) is an effective tool set to monitor and diagnose data plane failures in FP networks.
• ping fabricpath
• traceroute fabricpath
• mtrace fabricpath
S202S101
ES S100 ES S200
202# mtrace fabricpath ftag 2 repeat 1
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'D' - Destination Unreachable, 'X' - unknown return code,
'V' - VLAN nonexistent, 'v' - VLAN in suspended state,
'm' - malformed request, 'C' - Cross Connect Error,
'U' - Unknown RBridge nickname, 'n' - Not AF,
'*' - Success, Optional Tlv incomplete,
'I' - Interface not in forwarding state,
'S' - Service Tag nonexistent, 's' - Service Tag in suspended state,
'c' - Corrupted Data/Test
Fabricpath mtrace for multicast ftag 2, vlan 1
Code SwitchId Interface State TotalTime
==================================================
! 201 Rcvd on Eth1/2 fwd 3ms
! 101 Rcvd on Eth1/2 fwd 4ms
! 102 Rcvd on Eth1/2 fwd 4ms
S102 S201
Troubleshooting Tools: FPOAM
S202S201S102S101
S2S1
ES S100 ES S200
FP Vlans 100-199
A B C D
202# show run fabricpath | section "oam profile 2"
fabricpath oam profile 2
vlan 100
flow forward
ether-type 0x800
ip source 100.1.1.20
ip destination 10.1.1.30
mac-address source 0000.1010.1010
mac-address destination 0000.3333.3333
protocol 1
202# traceroute fabricpath switch-id 1034 profile 2
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'D' - Destination Unreachable, 'X' - unknown return code,
'V' - VLAN nonexistent, 'v' - VLAN in suspended state,
'm' - malformed request, 'C' - Cross Connect Error,
'U' - Unknown RBridge nickname, 'n' - Not AF,
'*' - Success, Optional Tlv incomplete,
'I' - Interface not in forwarding state,
'S' - Service Tag nonexistent, 's' - Service Tag in suspended state,
'c' - Corrupted Data/Test
Sender handle: 14
Hop Code SwitchId Interface State TotalTime PathId
============================================================
1 ! 2 Rcvd on Eth6/2 fwd 3ms
2 ! 100 Rcvd on Eth1/1 fwd 4ms
• OAM Profiles can be used to replicate dataplane packet and follow the forwarding path
Troubleshooting Tools: Counters
S202(config)# ip access-list test-stats
S202(config-acl)# statistics per-entry
S202(config-acl)# permit ip host 10.1.100.101 host 10.1.100.201
S202(config-acl)# permit ip any any
S202(config-acl)# interface ethernet 1/7
S202(config-if)# ip port access-group test-stats in
S202(config-if)# end
S202S201S102S101
S2S1
ES S100 ES S200
FP Vlans 100-199
A B C D
vPC30 vPC40
S202# show ip access-lists test-stats
IPV4 ACL test-stats
statistics per-entry
10 permit ip 10.1.100.101/32 10.1.100.201/32 [match=0]
20 permit ip any any [match=0]
! Sent 5000 frames
S202# show ip access-lists test-stats
IPV4 ACL test-stats
statistics per-entry
10 permit ip 10.1.100.101/32 10.1.100.201/32 [match=5000]
20 permit ip any any [match=0]
Find the likely interface to receive packets
(note multidestination traffic might follow
different path sh fab isis trees)
Configure ACL with ‘statistics per-entry’
which explicitly matches traffic in question
Attach ACL to ingress FP port as a PACL
Check the counters
Run test traffic
Check the counters again
Compare
Troubleshooting Tools: CountersS1# attach module 6
module-6# show hardware internal dev-port-map
--------------------------------------------------------------
CARD_TYPE: 48 port 10G
FP port | PHYS | MAC_0 | L2LKP | L3LKP | QUEUE |SWICHF
...
19 4 4 4 4 4 0
20 4 4 4 4 4 0
21 5 5 5 5 5 0
22 5 5 5 5 5 0
...
module-6# test fabricpath unicast configure route-stats vdc 2 ftag 1 switchid 200 fe 5 table [mp | sw] commit
module-6# show fabricpath unicast route-stats vdc 2 ftag 1 switchid 200 fe 5
------------------------------------------
| VDC | FTAG | SwitchID | SubSwitchID |
-------------------------------------------
| 002 | 0001 | 0200 | 000 |
| FE | Adjacency | Statistics |
| 4 | Eth6/21| 0000000000 |
| 4 | Eth6/22| 0000000000 |
module-6# show fabricpath unicast route-stats vdc 2 ftag 1 switchid 200 fe 5
------------------------------------------
| VDC | FTAG | SwitchID | SubSwitchID |
-------------------------------------------
| 002 | 0001 | 0200 | 000 |
| FE | Adjacency | Statistics |
| 4 | Eth6/21| 0000000000 |
| 4 | Eth6/22| 0000000064 |
Find ingress interface & attach to
respective linecard
Find Ingress FE instance
Configure statistics (use FE+1)
Print statistics
Run traffic
Print statistics again – note statistics
are in HEX
Compare
Use MP table to get per next-hop stat
if there is >1 next-hop, else use SW
table
Troubleshooting Tools: Error/Drop Counters
• Usual datapath troubleshooting apply on N7K
• And on N5K/N6K
N5K# sh platform fwm info pif e1/5 | i stats|cdce
Eth1/5 pd: tx stats: bytes 304069130 frames 913992 discard 0 drop 0
Eth1/5 pd: rx stats: bytes 9647836468 frames 8319249 discard 0 drop 1650
Eth1/5 pd cdce_addr: switchid 30 sub-switchid 0, endnodeid 0
Eth1/5 pd cdce_addr: Mcast 0, locally-adm 1, OutOfOrder/don't learn 0
Eth1/5 pd cdce_addr: localid 5, pbp_idx 0
N5K# sh platform fwm info asic-errors 0
Printing non zero Carmel error registers:
DROP_SRC_VLAN_MBR: res0 = 495188 res1 = 0 [12]
DROP_CDCE_SW_TBL_RPF_MISS: res0 = 4 res1 = 0 [30]
DROP_SRC_FTAG_BITMAP_MBR: res0 = 5 res1 = 0 [31]
DROP_SRC_MASK_TO_NULL: res0 = 332912 res1 = 0 [44]
7k# show hardware internal errors module 6 | diff
... send 2000 transit packets using ping with timeout 0 ...
7k# show hardware internal errors module 6 | diff
< 1008 Self-forwarding check OSA drop 0000000287061579 3 –
> 1008 Self-forwarding check OSA drop 0000000287063630 3 -
< 2514 Ingress packets marked with drop_oth sent to IB 0000000002127119 4 –
> 2514 Ingress packets marked with drop_oth sent to IB 0000000002127173 4 -
< 50 smallcnt DSWID/DSSWID miss and DCE frame, def-gw disabled 0000000000000563 5-6 –
> 50 smallcnt DSWID/DSSWID miss and DCE frame, def-gw disabled 0000000000002563 5-6 -
show hardware internal errors
often produces lengthy outputs, use
diff to just see what has changed
between 2 timed samples
(with some test traffic in the middle)
PIF (physical interface) maintains RX/TX
and drop counters
Check if drops are non-zero & growing
(also check the ASIC number)
Use ASIC-errors command to get a
breakdown of drop reasons (and see if any
are growing in with test/ping traffic)
Troubleshooting Tools: ELAM
• Embedded Logic Analyzer Module (ELAM) is an engineering tool that is used to look inside Cisco ASICs.
• ELAM is architecture specific and therefore will have different capabilities and different CLI syntax across different forwarding engines (FE).
• It is possible to use ELAM as a capturing tool to validate:
1. Was the packet received
2. On which interface/VLAN did the packet arrive
3. What did the packet look like
4. How was the packet altered and where was it sent
• It is not intrusive
• It can be used at a very granular level to troubleshoot a single traffic flow which can be an invaluable tool to network administrators.
• When the going gets tough…
ELAM is NOT a supported feature.
It is a diagnostic tool designed for
internal use. Anything and
everything about it may change from
version to version without any notice
Troubleshooting Tool: ELAM Workflow
Identify the expected ingress
Forwarding Engine (FE)
Configure an ELAM trigger to capture specific
frame
Start the ELAM
After ELAM triggers, display and analyze the
data
Once triggered data can be displayed and analyzed
Typical ELAM challenges
Identifying the correct capture point and trigger
Understanding the captured data (for complex cases)
Troubleshooting Tools: ELAM
• Data Bus (DBUS) and Result Bus (RBUS)
The DBUS contains several platform specific internal fields along with the header information from a frame required to make the forwarding decision. We use the DBUS information to validate where the frame was received and basic data about the frame.
The RBUS will contain information about the forwarding decision to help determine if the frame was altered and where it was sent.
• Local Target Logic (LTL)
The LTL is an index used to represent a port or group of ports. The source LTL index and the destination LTL index tell us which port the frame was received and where it was sent.
• Basics to know before performing an ELAM
Troubleshooting Tools: ELAM Example
• Packet from host 10.1.100.101 <-> 10.1.100.201, expected ingress interface Eth6/19 on N7K-F2 linecard of S1
S202S201S102S101
S2S1
ES S100 ES S200
FP Vlans 100-199
A B C D
vPC30 vPC40
S1# attach module 6
Attaching to module 6 ...
module-6# show hardware internal dev-port-map
+-----------------------------------------------------------------------+
+----------------+++FRONT PANEL PORT TO ASIC INSTANCE MAP+++------------+
+-----------------------------------------------------------------------+
FP port | PHYS | MAC_0 | L2LKP | L3LKP | QUEUE |SWICHF
...
19 4 4 4 4 4 0
...
module-6# elam asic clipper instance 4
module-6(clipper-elam)# layer2
module-6(clipper-l2-elam)# trigger dbus ipv4 ingress if source-ipv4-
address 10.1.100.101 destination-ipv4-address 10.1.100.201
module-6(clipper-l2-elam)# trigger rbus ingress if trig
module-6(clipper-l2-elam)# start
module-6(clipper-l2-elam)# status
L2 DBUS Triggered
L2 RBUS Triggered
Linecard L2/L3 ASIC name
M-series Eureka/Lamira
F1 Orion
F2 Clipper
F3 Flanker
Eth6/19 is on FE instance 4
(code name clipper)
Configure a trigger specific to
this source/destination IP
Start the ELAM, send the
traffic and wait for it to trigger
Troubleshooting Tools: ELAM Example
S202S201S102S101
S2S1
ES S100 ES S200
FP Vlans 100-199
A B C D
vPC30 vPC40
module-6(clipper-l2-elam)# show dbus
<snip>
port-id : 0x2 last-ethertype : 0x800
vlan : 0x64 destination-index : 0x0
source-index : 0x62 bundle-port : 0x0
status-is-1q : 0x1 trill-encap : 0x0
mac-in-mac-valid : 0x1 dtag-ttl : 0x20
recirc-acos : 0x0 dtag-ftag : 0x1
source-ipv4-address: 10.1.100.101
destination-ipv4-address: 10.1.100.201
mim-destination-mac-address: 0200.c800.0000
mim-source-mac-address: 0200.640b.ffff
destination-mac-address 0000.0000.000d
source-mac-address: 0000.0000.000a
module-6(clipper-l2-elam)# show rbus
<snip>
di-ltl-index : 0x65 l3-multicast-di : 0x0
source-index : 0x62 vlan-id : 0x64
dtag=ftag : 0x1 dtag-ttl : 0x1f
mim-destination-mac-address: 0200.c800.0000
mim-source-mac-address: 0200.640b.ffff
Frame received on VLAN 100 (0x64) from a
source-index of 0x62 (next slide)
mac-in-mac valid (this is a FP frame)
dtag-TTL: fabricpath TTL of 32 (0x20)
ODA (0c8.00.0000) = 200.0.0
OSA (064.0b.ffff) = 100.11.65535
Frame transmitted on vlan 100 (0x64) to a destination
index of 0x65 (next slide)
dtag-TTL: fabricpath TTL decremented to 31 (0xf1)
Troubleshooting Tools: ELAM Example
• ELAM confirms that frame was received on Eth6/19, VLAN 100 with an OSA of 100.11.65535 and ODA of 200.0.0.
• ELAM also confirms that frame was forwarded out Eth6/22 on VLAN 100 with a decremented FP TTL
S202S201S102S101
S2S1
ES S100 ES S200
FP Vlans 100-199
A B C D
vPC30 vPC40
S1# show system internal pixm info ltl 0x62
Member info
------------------
Type LTL
---------------------------------
PHY_PORT Eth6/19
S1# show system internal pixm info ltl 0x65
Member info
------------------
Type LTL
---------------------------------
PHY_PORT Eth6/22
Get mapping of
source index to
physical port
Get mapping of
destination index to
physical port
Troubleshooting Tools: show tech
• show tech fabricpath isis
• show tech fabricpath switch-id
• show tech fabricpath topology
• Neither of these include FP routes, macs or comprehensive forwarding related data. Collect these separately:
• show tech l2fm detail
• show tech l2fm l2dbg
• show tech forwarding l2 unicast
• show tech forwarding l2 multicast
84
Troubleshooting Example: Broken HSRP
• Problem statement: HSRP active & standby do not ‘see’ each other in certain vlans. For example in vlan 1317 standby (S2) ‘sees’ the active (S1), but on active standby is unknown. A number of vlans are affected. This is new deployment.
• Initial assessment: possible reason for HSRP router not ‘seeing’ other router is HSRP hello packets not being received. In our case it is likely active router, not receiving hello packets from standby
• Quick debug on S1 confirms it only sends hellos in vlan 1317
• …and on S2 we see hellos being sent and received…
S4S3
S2S1
S1# debug hsrp engine packet hello interface vlan 1317
10:03:30 hsrp: Vlan1317[17/V4]: Hello out Active pri 100 ip 10.13.17.254
10:03:31 hsrp: Vlan1317[17/V4]: Hello out Active pri 100 ip 10.13.17.254
10:03:32 hsrp: Vlan1317[17/V4]: Hello out Active pri 100 ip 10.13.17.254
S2# debug hsrp engine packet hello interface vlan 1317
10:03:30 hsrp: Vlan1317[17/V4]: Hello in from 10.13.17.1 State Active pri 100 ip 10.13.17.254
10:03:30 hsrp: Vlan1317[17/V4]: Hello out Standby pri 50 ip 10.13.17.254
10:03:31 hsrp: Vlan1317[17/V4]: Hello in from 10.13.17.1 State Active pri 100 ip 10.13.17.254
10:03:31 hsrp: Vlan1317[17/V4]: Hello out Standby pri 50 ip 10.13.17.254
E1/1
Troubleshooting Example: Broken HSRP
• Are the HSRP frames from S2 to S1 getting lost?
• Findings so far:
• Working and Non-working packets may follow different paths
• Time to look at the Trees
S4S3
S2S1
S2# sh fabricpath load-balance multicast ftag-selected flow-type l2 dst-mac
0100.5e00.0002 src-mac 0000.0c00.0123 ether-type 800 vlan 1317 module 1
...
FTAG SELECTED IS : 2
S2# sh fabricpath isis topology summary
MT-0
Configured interfaces: port-
channel1 Ethernet1/1 Ethernet1/2
Number of trees: 2
Tree id: 1, ftag: 1, root system: 0000.0000.0002, 2
Tree id: 2, ftag: 2, root system: 0000.0000.0004, 4
S1# sh fabricpath load-balance multicast ftag-selected flow-type l2 dst-mac
0100.5e00.0002 src-mac 0000.0c07.ac11 ether-type 800 vlan 1317 module 1
...
FTAG SELECTED IS : 1
S2# show fabricpath isis trees
MT-0
Topology 0, Tree 1, Swid routing table
1, L1
via port-channel1, metric 20
...
Topology 0, Tree 2, Swid routing table
1, L1
via Ethernet1/1, metric 40
...
S1S2 FTAG 1 traffic uses Po1
(peer-link)
S1S2 FTAG 2 traffic uses E1/1
(goes through S4)
E1/1
Troubleshooting Example: Broken HSRP
• S4 is transit switch for HSRP S2 S1 traffic, hence we will not see packets in debug. We need to look at the data plane level if hello packet arrives/leaves.
• Options: SPAN, Counters, ELAM
• Let’s try hardware counters…
S4# show hardware internal errors module 1
...
|------------------------------------------------------------------------|
| Device:Orion Fwding Driver Role:L2 Mod: 1 |
| Last cleared @ Thu Apr 11 11:11:11 2011
| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|
Instance:0
ID Name Value Ports
-- ---- ----- -----
29 smallcnt Pkt dropped due to CBL 0000000000001227 1-2 -
2014 Ingress packets marked with drop_oth sent to IB 0000000000001227 1 –
S4# show hardware internal errors module 1 | diff
...wait some seconds...
S4# show hardware internal errors module 1 | diff
< 29 smallcnt Pkt dropped due to CBL 0000000000001229 1-2 –
> 29 smallcnt Pkt dropped due to CBL 0000000000001235 1-2 -
CBL drops grow at about the rate of
HSRP hellos. CBL stands for Color
Blocking logic (or Vlan Blocking
Logic). Essentially, hardware logic
defining whether given port/vlan is
blocking or forwarding packets.
E1/1
S4S3
S2S1
E1/1
S4# show fabricpath mroute vlan 1317
ERROR: Vlan 1317 does not exist
S4# show vlan id 1317
VLAN 1317 not found in current VLAN database
Root cause: Vlan missing from transit switch
All FP vlans must be defined on all FP
switches, otherwise there might be issues
similar to this for flooded traffic. ISIS will
prune off unnecessary flood traffic towards
tree branches that do not have ports behind
them.
Troubleshooting: Common Pitfalls
• All FP Vlans must be present on all FP switches
• else multicast trees might not be correct
• TCNs not propagated to required FP or CE switches. Configure STP domain where TCNs need to be propagated. Else, connectivity might be broken after re-convergence until MACs age out or are relearned
• At power up or reload, CE-side comes up faster than FP-side
• L2GW Inconsistency, ensure that FP switches have been configured with superior priority before connecting to CE switches.
CLI cheatsheet
• Interfaces in FP modeshow fabricpath isis interface [brief]
• ISIS adjacenciesshow fabricpath isis adjacency [detail]
• Root information for the treesshow fabricpath isis topology summary
• RPF information for the treesshow fabricpath isis trees
• OIFs for the treesshow fabricpath mroute
• Affinity to Ftagsshow fabricpath isis database detailshow system internal m2rib ftag
• Pongpong destination-swid <sw#> destination-mac <mac-address> vlan <vlan> count <#> … [detail]
Summary
• Core ConceptsKnown Unicast Best path with ECMP, Rest Tree-balanced
• Control PlaneISIS in the core, STP / IGMP snooping at CE
• Data PlaneMAC address table, SwitchID table, Tree table (RPF)
• TroubleshootingUnderstand what should be happening, verify what is happening, find a deviation, zoom in and repeat
90