advanced cisco catalyst 6500/6800 series...

Advanced Cisco Catalyst 6500/6800 Series Troubleshooting

BRKCRS-3148

Ravi U. Krishna ([email protected])

Customer Support Engineer, Cisco Services

© 2014 Cisco and/or its affiliates. All rights reserved. BRKCRS-3148 Cisco Public

Goal of the Session

Educate you commonly used commands and methods to troubleshoot Cisco Catalyst 6500 Virtual Switch (VSS) based on Supervisor 2T.

In the latter part of this session we shall introduce Catalyst Instant Access Solution and teach how to troubleshoot this device through some common troubleshooting commands.

For more in depth understanding and detailed troubleshooting of Cisco Catalyst 6500 VSS Switch, its recommended to have attended BRKCRS-3143 session prior to this.

3


Agenda

• Introduction

• Troubleshooting Virtual Switching System

• Introduction to Instant Access(IA) Solution

• Troubleshooting 6800ia Solution

• Conclusion

• Q and A

4

Introduction

5


What do I care about as an Network Admin..

• Is my VSS working fine ? Health check

• How can I check the path of a packet that is ingressing into my VSS

• What happens when one of my VSS supervisor engine crashes and how quickly can I recover from this downtime

6

Troubleshooting Virtual Switch System

7


Troubleshooting Virtual Switching System

• Architecture of 2T VSS

• Control plane health check

• L2/L3 Packet flow troubleshooting

• Dual Active detection

• VSS Supervisor Engine failover and recovery

Agenda

8


Virtual Switching System (VSS) Architecture

9

VSS Physical View VSS Logical View


Virtual Switching System (VSS) Architecture

Virtual Switch

Domain

Virtual Switch Link

Active Control Plane

Active Data Plane

Hot Standby Control Plane

Active Data Plane

Virtual Switch Primary Virtual Switch Secondary

10


Terminologies

• PFC: Policy Feature Card

• DFC: Distributed Forwarding Card

• FE: Forwarding Engine

• TCAM: Ternary Content Addressable Memory

• ACE: Access Control Entry

• ADJ: Adjacency information

• LIF: Logical Interface

• MSFC: Multilayer Switch Fabric Card

• FIRE: Fabric Interface and Replication Engine

• FIB: Forwarding Information Base

11

11


Architecture 2T VSS Internal Architecture

DBUS

PFC4

Traces # 1 to 26 RBUS EOBC

MSFC 5

Flash DRAM

CL1 TCAM

NET FLOW TCAM

L3/4 Engine

L2 Engine

ACE Counter

CL2 TCAM

40

Gbps

Fabric Interface

AND

Replication

Engine

Switch Fabric

26 x 40G Traces

2 x 1

Gbps

Central

Management

Processor

CPU

Port ASIC

FIB TCAM

ADJ TCAM

LIF Table

LIF Stats

RPF Table

LIF MAP

MSFC5 Complex contains single (combined SP/RP) CMP CPU FIRE ASIC that interfaces with fabric and bus and for multicast/SPAN replication

1GE / 10GE

Uplinks

Switch Fabric

LIF Table and Statistics

TCAMs for FIB/ADJ Layer 3/4 Forwarding Engine

Layer 2 Forwarding Engine

ACE Hardware Counters

Port ASIC to host external interface

12








Agenda

13


Control Plane health check VSS Specific Protocols (VSLP)

• Link Management Protocol (LMP)

– Used to track and reject Unidirectional Links, Exchange Chassis ID and other information between the two switches

• Role Resolution Protocol (RRP)

– Used to determine the compatible hardware and software versions to form the VSL. Also used to determine which Switch becomes Active and Hot Standby from control plane perspective

After the roles have been resolved through RRP, a Configuration Consistency Check is performed across VSL switches to ensure proper VSL operation. A failure in this check will result in reloading the hot-standby to operate in RPR mode.

14


Control Plane health check VSS Configuration and operation check

sup2T# show switch virtual

Switch mode : Virtual Switch

Virtual switch domain number : 100

Local switch number : 1

Local switch operational role: Virtual Switch Active

Peer switch number : 2

Peer switch operational role : Virtual Switch Standby

sup2T#

Unique number in your network between 1 and 255

Switch numbers to identify primary and secondary

Switch roles (active or standby)

15


Control Plane health check VSS Configuration and operation check

sup2T# show switch virtual link

VSL Status : UP

VSL Uptime : 22 hours, 43 minutes

VSL SCP Ping : Pass

VSL ICC Ping : Pass

VSL Control Link : Te1/2/4

VSL Encryption : Configured Mode - Off, Operational Mode – Off

sup2T#

Control Link carries EOBC and IBC control

Messages (SCP and ICC/IPC)

16


Control Plane health check VSLP Ping Check

Check if we can ping the two switches using VSLP protocol

sup2T# ping vslp output interface tenGigabitEthernet 1/2/4

Type escape sequence to abort.

Sending 5, 100-byte VSLP ping to peer-sup via output port 1/2/4, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 12/12/12 ms

sup2T#

Use – ‘show switch virtual link port-channel’

to determine the physical interfaces which are part of VSL

17


Control Plane health check VSS Configuration and operation check sup2T# show switch virtual slot-map Virtual Slot to Remote Switch/Physical Slot Mapping Table: Virtual Remote Physical Module Slot No Switch No Slot No Uptime ---------+-----------+----------+---------- 17 1 1 - 18 1 2 3d18h 19 1 3 3d18h 20 1 4 - 21 1 5 - <snip> 30 1 14 - 31 1 15 - 32 1 16 - 33 2 1 22:41:23 34 2 2 - 35 2 3 - 36 2 4 22:37:31 37 2 5 - <snip>

Gives mapping between the Switch number, Physical Slot number to its Virtual Slot number. It also shows how long the module is online in this configuration

Module in Switch 1 Slot 2 is up since 3 days and 18 hours

Module in Switch 2 Slot 1 is up since 22 hours, 41 mins and 23 secs

18


Control Plane health check

show switch virtual dual-active summary

show switch virtual link detail

show switch virtual redundancy

show switch virtual role detail

show switch virtual link port-channel

show vslp [lmp|rrp] [counters|fsm|summary]

debug vslp [lmp|rrp] packet*

debug switch virtual vsl-drop packets*

Other helpful commands Displays Dual Active capability

Displays info on LMP viz., counters, neighbors, hello timer, FSM info, etc

Displays peer switch redundancy info

Displays switch numbers and redundancy role

VSS Port-Channel info

Debug LMP/RRP hello’s between VSS

Debug dropped packets seen on VSL links

Displays RRP/LMP information

19






• Dual Active Detection


Agenda

20


Layer2 Packet flow troubleshooting Topology

DUT is the Device Under Test we are troubleshooting

DUT is a VSS Chassis with Supervisor 2T

Four TenGigabitEthernet L2 Etherchannel (Switch A DUT)

Four TenGigabitEthernet L2 Etherchannel (DUT Switch B)

SwitchA and SwitchB are 6500 standalone switch with Supervisor 2T

Po11

Ten1/4 Ten 1/1/1

Ten 1/1/2 Ten1/8

Ten 2/1/1 Ten1/5

Ten 2/1/2 Ten1/7

Po11

Switch A DUT Switch B

Po12

Ten 1/1/5 Ten1/3

Ten1/4 Ten 1/1/6

Ten1/7 Ten 2/1/5

Ten1/8 Ten 2/1/6

Po12

21


L2 Unicast Traffic

sup2T# show ip arp

Protocol Address Age (min) Hardware Addr Type Interface

Internet 192.168.10.1 - b414.8961.3780 ARPA Vlan10

Internet 192.168.10.2 31 0006.5bbc.81a2 ARPA Vlan10

Internet 192.168.10.3 32 0006.5bbc.7acb ARPA Vlan10

Traffic Configuration

22

Po11 Ten1/4 Ten 1/1/1

Ten 1/1/2 Ten1/8

Ten 2/1/1 Ten1/5

Ten 2/1/2 Ten1/7

Po11

Switch A sup2T Switch B

Po12 Ten 1/1/5 Ten1/3

Ten1/4 Ten 1/1/6

Ten1/7 Ten 2/1/5

Ten1/8 Ten 2/1/6

Po12

PC 1

192.168.10.2 PC 2

192.168.10.3

Vlan 10


L2 Unicast Traffic

sup2T# show mac address-table address 0006.5bbc.7acb

Legend: * - primary entry

age - seconds since last seen

<snip>

Displaying entries from Switch 1 DFC linecard [1]:

vlan mac address type learn age ports

----+----+---------------+-------+-----+---------+--------

* 10 0006.5bbc.7acb dynamic Yes 0 Po12



----+----+---------------+-------+-----+----------+-------

10 0006.5bbc.7acb dynamic Yes 110 Po12

Where are the MAC Addresses Learned? sup2T# show mac address-table address 0006.5bbc.81a2


age - seconds since last seen

<snip>



----+----+---------------+-------+-----+----------+-----

10 0006.5bbc.81a2 dynamic Yes 5 Po11



----+----+---------------+-------+-----+----------+-----

* 10 0006.5bbc.81a2 dynamic Yes 90 Po11

23

Host PC 1

Host PC 2


L2 Unicast Traffic Which Link in the EtherChannel Is Being Used?

24

SwitchA# show etherchannel load-balance module 1

EtherChannel Load-Balancing Configuration:

src-dst-ip vlan included

mpls label-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:

Non-IP: Source XOR Destination MAC address

IPv4: Source XOR Destination IP address

IPv6: Source XOR Destination IP address

MPLS: Label or IP

SwitchA# show etherhannel load-balance interface po11 ip 192.168.10.2 192.168.10.3

Computed RBH: 0x3

Would select Te1/8 of Po11

Mode is “src-dst-ip”. Only use src and dest IP as argument.

Link selected is Ten1/8 in Po11 of SwitchA for traffic sent to 192.168.10.3

Check load balancing configuration

Use ingress Module number in command in case per-module load-balancing is configured (SXH images and later)

Po11

Ten1/4 Ten 1/1/1

Ten 1/1/2 Ten1/8

Ten 2/1/1 Ten1/5

Ten 2/1/2 Ten1/7

Po11


Po12

Ten 1/1/5 Ten1/3

Ten1/4 Ten 1/1/6

Ten1/7 Ten 2/1/5

Ten1/8 Ten 2/1/6

Po12


L2 Unicast Traffic Network Path Verification: Result

Each packet flow can use different links in the bundles !

Po11

Ten1/4 Ten1/1/1

Ten1/1/2 Ten1/8

Ten2/1/1 Ten1/5

Ten2/1/2 Ten1/7

Po11


Po12 Ten1/1/5 Ten1/3

Ten1/4 Ten1/1/6

Ten1/7 Ten2/1/5

Ten1/8 Ten2/1/6

Po12

Po11

Ten1/4 Ten1/1/1

Ten1/1/2 Ten1/8

Ten2/1/1 Ten1/5

Ten2/1/2 Ten1/7

Po11


Po12 Ten1/1/5 Ten1/3

Ten1/4 Ten1/1/6

Ten1/7 Ten2/1/5

Ten1/8 Ten2/1/6

Po12

25


Layer3 Packet flow troubleshooting Topology

26

DUT is the Device Under Test we are troubleshooting (6500 VSS with Sup2T)

R1 and R2 are two 6500 standalone switches with supervisor 720 in them.

Four TenGigabitEthernet L2 Etherchannel Trunk (R1 DUT)

- Vlan 10, 20, 30 and 40 are assigned with 192.168.10.0 /24,

192.168.20.0 /24, 192.168.30.0 /24 and 192.168.40.0 /24 subnets respectively.

Four L3 Links (DUT R2)

- Four links are assigned with 172.16.10.0 /24, 172.16.20.0 /24,

172.16.30.0 /24 and 172.16.40.0 /24 subnets respectively.

Po11 Ten1/4 Ten 1/1/1

Ten 1/1/2 Ten1/8

Ten 2/1/1 Ten1/5

Ten 2/1/2 Ten1/7

Po11

R1 DUT R2

Ten1/3

Ten1/4

Ten1/7

Ten1/8

Ten 1/1/5

Ten 1/1/6

Ten 2/1/5

Ten 2/1/6 Host 1

100.100.100.1

Host 2

200.200.200.1

VLANS 10,20,30 and 40 L3 Links


L3 Unicast Traffic Different Switching Paths for L3 Traffic in Catalyst 6500

27

27

Process Switching Path

Software-based CEF Switching Path

Hardware-based CEF switching Path

DUT

Gives logical representation of different switching paths in Catalyst 6500.

Host1 Host2


Layer3 Packet flow troubleshooting Host 1 Host 2: Which L3 Next Hop / L2 Link from R1?

28

SW R1# show ip route 200.200.200.1

Routing entry for 200.200.200.1/32

Known via "ospf 100", distance 110, metric 3, type intra area

Last update from 192.168.40.1 on Vlan40, 00:10:12 ago

Routing Descriptor Blocks:

192.168.40.1, from 192.168.0.2, 00:10:12 ago, via Vlan40

Route metric is 3, traffic share count is 1

192.168.30.1, from 192.168.0.2, 00:10:12 ago, via Vlan30


* 192.168.20.1, from 192.168.0.2, 00:10:12 ago, via Vlan20


192.168.10.1, from 192.168.0.2, 00:10:12 ago, via Vlan10


R1# show ip cef exact-route 100.100.100.1 200.200.200.1

100.100.100.1 -> 200.200.200.1 => IP adj out of Vlan40, addr 192.168.40.1

R1# show mls cef exact-route 100.100.100.1 0 200.200.200.1 0

Interface: Vl10, Next Hop: 192.168.20.1, Vlan: 10, Destination Mac:

b414.8961.3780

R1# show etherchannel load-bal int port-ch 11 ip 100.100.100.1 200.200.200.1

Computed RBH: 0x7

Would select Te1/8 of Po11

Next hop used for HW based CEF

(HW forwarding path). Note: “0” is used for both src and dest L4 port numbers as test flow was ICMP echo

Check which link between R1 and DUT is chosen.

HW

Equal Cost Routes to the destination prefix

Next hop used for SW based CEF (SW forwarding data path)

Note: R1 is a Cat6500 with Sup720.

HW HW

HW HW

HW 28


Layer3 Packet flow troubleshooting Host 1 Host 2: Which L3 Next Hop / L2 Link from DUT?

Next hop used for HW based CEF

(HW forwarding path). Note: “0” is used for both src and dest L4 port numbers as test flow was ICMP echo

29

sup2T# show ip route 200.200.200.1 Routing entry for 200.200.200.1/32

Known via "ospf 100", distance 110, metric 2, type intra area

Last update from 172.16.20.2 on TenGigabitEthernet1/1/6, 00:36:01 ago


172.16.40.2, from 192.168.0.2, 00:36:01 ago, via TenGigabitEthernet2/1/6






* 172.16.10.2, from 192.168.0.2, 00:36:01 ago, via TenGigabitEthernet1/1/5


sup2T# show ip cef exact-route 100.100.100.1 200.200.200.1 100.100.100.1 -> 200.200.200.1 => IP adj out of TenGigabitEthernet1/1/6, addr 172.16.20.2

sup2T# show platform hardware cef exact-route 100.100.100.1 0 200.200.200.1 0 Interface: Te2/1/6, Next Hop: 172.16.40.2, ifnum: 0x12, Destination Mac: f866.f2d2.fa80

LIF: 0x20004013

SW

HW

Next hop used for SW based CEF (SW forwarding data path)

Equal Cost Routes to the destination prefix

29






• Dual Active Detection


Agenda

30


Dual Active Detection Possibility of VSL breaking

• In a VSS domain, one Switch is elected VSS Active and the other is elected VSS Standby after RRP negotiation.

• Since VSS is configured on a PortChannel interface with more than one physical interface bundled, the possibility of PortChannel going down is remote.

• However it’s a possibility… Such a situation is called dual-active scenario. VSS must detect this dual-active condition and take recovery action.

• There are three methods to detect such a condtion

– Using enhanced PAgP (ePAgP)

– Using Dual-Active Fast Hello Packets

– Using Instant Access

VSL

Active Standby

Active Active

All these are by default enabled. However we need to configure the physical interface(s) and Port-Channels to be part of these individual methods

31


Dual Active Detection Dual Active Detection Fast Hellos Troubleshooting

sup2T(config)# interface range Gig1/7/1, Gig2/7/1

sup2T(config-if-range)# dual-active fast-hello

%VSDA-SW1_SP-5-LINK_UP: Interface Gi1/7/1 is now dual-active detection capable

%VSDA-SW2_SPSTBY-5-LINK_UP: Interface Gi2/7/1 is now dual-active detection capable

sup2T# show switch virtual dual-active fast-hello Fast-hello dual-active detection enabled: Yes Fast-hello dual-active interfaces: Port Local State Peer Port Remote State -------------------------------------------------

Gi1/7/1 Link up Gi2/7/1 Link up

VSL

Gi1/7/1 Gi2/7/1

Fast Hello

Dual Active Trusted

ePAgP MEC

SW1 SW2

Fast Hello configuration requires an extra physical connection between both virtual switch nodes

Po101

All dual active detection protocols and methods can be implemented in parallel

R1

32


Dual Active Detection

sup2T(config)# interface Port-Channel 101

sup2T(config)# shutdown

sup2T(config)# switch virtual domain 100

sup2T(config)# dual-active detection pagp trust channel-group 101

sup2T(config)# interface Port-Channel 101

sup2T(config)# no shutdown

Dual Active Detection ePAgP Troubleshooting

sup2T# show switch virtual dual-active pagp PAgP dual-active detection enabled: Yes

PAgP dual-active version: 1.1

Channel group 101 dual-active detect capability

Dual-Active trusted group: Yes

Dual-Active Partner Partner Partner

Port Detective Capable Name Port Version

Gi1/2/3 Yes R1 Gi2/4 1.1

Gi1/6/2 Yes R1 Gi5/2 1.1

Gi2/9/36 Yes R1 Gi4/16 1.1

Ensure neighbor runs a SW version that supports ePAgP

Choose interfaces on each Switch and different modules for redundancy

Prior to implementing trust settings, network administrators should plan for a downtime to provision ePAgP based dual active configuration

Enabling or disabling dual-active trusted mode on Layer 2/Layer 3 MEC requires MEC to be in an administrative shutdown state.

33


Dual Active Detection

• Managing the VSS system during a dual-active condition becomes challenging when two individual systems in the same network tier contain a common network configuration.

• To recover from this condition the Old-Active Virtual Switch disables _all_ physical and logical interfaces.

• To minimize network instability, it is highly recommended to exclude only network management ports from both virtual-switch chassis.

Dual Active Recovery sup2T(config)# switch virtual domain 100

sup2T(config-vs-domain)# dual-active exclude interface Gig 1/5/2

sup2T(config-vs-domain)# dual-active exclude interface Gig 2/5/2

sup2T# show switch virtual dual-active summary

Pagp dual-active detection enabled: Yes

Bfd dual-active detection enabled: Yes

Interfaces excluded from shutdown in recovery mode:

Gi1/5/2 , Gi2/5/2

In dual-active recovery mode: No

Do NOT change the configuration when the Old Active goes to Recovery Mode.

Configuration must be saved MANUALLY and reloaded to bring the switch back up into the VSS

34








Agenda

35


VSS Supervisor Engine failover and recovery

• Standby Supervisor Crash

– Standby Switch and Chassis goes for a reload

– All modules in Standby Chassis are power-cycled

– Recovers by joining back the Active in SSO mode

• Active Supervisor Crash

– Standby Switch(SW2) takes over as New Active

– SW2 now handles all Control Plane traffic

– SW1 chassis reboots as new Standby and joins the VSS pair in SSO mode

Single supervisor in each chassis

VSL

SW1(Active) SW2(Standby)

VSL

SW1(Active) SW2(Standby) SW1(Standby) SW2(Active)

36


VSS Supervisor Engine failover and recovery QuadSup in SSO – sup2T running 15.1(1)SY1

ICA

ICS

ICA

ICS

VSL Link

Fast Hello

SW1 SW2

SW2 ICA / SSO Standby

SW2 ICS

SW1 ICA / SSO Active

SW1 ICS

What Happens

when SSO Active

Crashes ?

Which is the

switchover

target ?

ICA : In-Chassis Active

ICS : In-Chassis Standby

(Standby HOT)

37


VSS Supervisor Engine failover and recovery QuadSup in SSO – sup2T running 15.1(1)SY1

ICS

ICA

ICA

ICS

VSL Link

Fast Hello

SW1 SW2

SW2 ICA / SSO Active

SW2 ICS SW1 ICA / SSO Standby

SW1 ICS

ICA : In-Chassis Active

ICS : In-Chassis Standby

(RPR Warm /

ChassisSSO)

38



sup2T-qsup# show switch virtual redundancy | inc Proc|Software state

Switch 1 Slot 1 Processor Information :

Current Software state = ACTIVE


Current Software state = STANDBY HOT (CHASSIS)


Current Software state = STANDBY HOT (switchover target)


Current Software state = STANDBY HOT (CHASSIS)

Sup2T-qsup#

QuadSup in SSO – sup2T running 15.1(1)SY1

39

Switchover Target

(SSO Standby)

Switch 1 Slot 1 is the

Current SSO Active

InChassis Standby

InChassis Standby



sup2T-qsup# show module switch all | inc Switch|---|Supervisor

Switch Number: 1 Role: Virtual Switch Active

--- ----- -------------------------------------- ------------------ -----------

1 5 Supervisor Engine 2T 10GE w/ CTS (Acti VS-SUP2T-10G SAL1746G471

2 5 Supervisor Engine 2T 10GE w/ CTS (CSSO VS-SUP2T-10G SAL1731APWE

--- ---------------------------------- ------ ------------ ------------ -------

Switch Number: 2 Role: Virtual Switch Standby

--- ----- -------------------------------------- ------------------ -----------

1 5 Supervisor Engine 2T 10GE w/ CTS (Hot) VS-SUP2T-10G SAL1746G476

2 5 Supervisor Engine 2T 10GE w/ CTS (CSSO VS-SUP2T-10G SAL1731APX3

--- ---------------------------------- ------ ------------ ------------ -------

Sup2T-qsup#

QuadSup in SSO – sup2T running 15.1(1)SY1

40

Hot : Switchover target

Acti : Current Active

CSSO: InChassis Standby in RPR Warm/Chassis SSO



show redundancy switchover

show redundancy state domain in-chassis

show switch virtual redundancy

show switch virtual role detail

show switch virtual redundancy mismatch

show switch virtual slot-map

Other helpful commands Displays Uptime since last switchover

Displays Inchassis redundancy information

Displays overall redundancy state along with image name, uptime etc

Displays Local and remote switch nos

Config-sync mismatch details

Gives mapping between the Switch number, Physical Slot number and its Virtual Slot number. Uptime gives us information on which slot experienced switchover

41

Introduction to Instant Access(IA) Solution

43


IT Spends Most of Their Time in Repetitive Operational Actions for Access Switches

44

INTRODUCING 28% Monitoring,

troubleshooting

19%

Security

configurations

18%

Initial install,

configs, testing

14% Upgrading equipment


Catalyst Instant Access Evolution

45

STANDALONE

ACCESS

SWITCH

LACP /

PAGP

ACCESS

SWITCH

VSS

SiSi SiSi

ACCESS

SWITCH ACCESS

SWITCH

LACP /

PAGP

VSL

INSTANT ACCESS

INSTANT

ACCESS

CLIENT

INSTANT

ACCESS

CLIENT

VSL

SiSi SiSi


Traditional Campus with stacking at access

46

SiSi SiSi

SiSi SiSi

Building 1 Building 2 Building 3 Building 4

Core

SiSi SiSi SiSi SiSi SiSi SiSi

34 Total Devices Management (image and configuration)

48 Access Trunks/Port-Channels

4032 User Ports

46


VSS Campus with Stacking

47

Building 1 Building 2 Building 3 Building 4

Core

29 Total Devices for Image and Configuration Management

48 Access Trunks/Port-Channels

4032 User Ports


Catalyst Instant Access

48

Core

5 Total Devices for Image and Configuration Management

Automated Trunk Configuration

4032 User Ports


Catalyst Instant Access

49

Core

NO Routing Protocols or Spanning-Tree configuration

between Access and Distribution

NO Trunks to Configure from Access to Distribution

NO Configuration or Image Management at Access

Troubleshooting 6800ia Solution

50


Terminologies

• FEX – Fabric Extender

• IA Parent – Instant Access Parent / Controller Switch

• IA Client – Instant Access Client / Cat6k Remote Line Card

• SDP – Switch Discovery Protocol

• SRP – Switch Role Protocol

• SCP – Switch Configuration Protocol

• RSL – Remote Satellite Link (fabric link interconnecting IA Parent with IA Client)

• VIF – Virtual Interface (logical representation of FEX physical ports)

• RPF – Route Path Forwarding

• VNTAG – Virtual Native Tagging

51


Troubleshooting 6800ia solution Instant Access Components

IA Parent

52

FEX-Fabric IA Client

IA Parent

Supervisor 2T

X6904 -16x10G

Supervisor 2T

X6904 -16x10G

Virtual Switching System (VSS)

52


Troubleshooting 6800ia solution Instant Access Components

IA Parent

53

Supervisor 2T

X6904 -16x10G

Supervisor 2T

X6904 -16x10G

Catalyst 6880-X

Catalyst 6807-XL Catalyst 6500E

FEX-Fabric IA Client

53


Troubleshooting 6800ia solution

• Configuring Instant Access (FEX) and Discovery

– Topology

– Configuration CLI’s

– Troubleshooting CLI’s when FEX does not come online

• Unicast Packet Forwarding through FEX’es

– Topology

– Layer 2 unicast traffic CLI’s


• Multicast Packet Forwarding through FEX

– Topology

– IGMP / MCAST forwarding CLI’s

Agenda

54


Configuring Instant Access (FEX) and Discovery FEX Layer2 Topology

55

• VSS is Supervisor 2T based system operating in Virtual

Switching mode

• DUT1 and DUT2 are the Device Under Test we are

troubleshooting

• DUT1 and DUT2 are two 6800IA’s dual homed to VSS

Modules

• 2 x WS-C6504-E

• 2 x VS-SUP2T-10G

• 2 x WS-X6904-40G

• 2 x C6800IA-48TD

DUT-1 DUT-2

VSS


Configuring Instant Access (FEX) and Discovery

sup-2t# config t

sup-2t(config)# interface port-channel 100

sup-2t(config-if)# switchport

sup-2t(config-if)# switchport mode fex-fabric

sup-2t(config-if)# fex associate 110

sup-2t(config-if)# no shut

sup-2t(config-if)# exit

sup-2t(config)# interface range TenGig 1/3/8, TenGig 2/4/5

sup-2t(config-if)# switchport

sup-2t(config-if)# channel-group 100 mode on

sup-2t(config-if)# no shut

FEX Configuration commands

3. Associate it with a

fex-id 110

1. Create a Layer2

Port-channel 100

2. Configure its

mode as fex-fabric

4. Select the Fabric physical links

5. Bundle them into

Port-Channel in mode on

Repeat the configuration for configuring second FEX Client (with FEX ID 120)

56



sup2T# show fex

FEX FEX FEX FEX

Number Description State Model Serial

---------------------------------------------------------------------------

110 FEX0110 online C6800IA-48TD FOC1736W1A8

120 FEX0120 online C6800IA-48FPD FOC1736W197

sup2T#

Verify FEX is online

FEX id’s

110 and 120

FEX states:- Init, Connected, Registration, Image Download, Registered, Online, Offline

FEX model FEX Serial

Numbers

57

New IA Client releasing this week at CiscoLive 2014.

C6800IA-48FPDR



sup2T# show fex detail

FEX: 110 Description: FEX0110 state: online

FEX version: 15.0(2)EX4

Extender Model: C6800IA-48TD, Extender Serial: FOC1736W1A8

FCP ready: yes

Image Version Check: enforced

Fabric Portchannel Ports: 2

Fabric port for control traffic: Te2/4/5

Fabric interface state:

Po100 - Interface Up.

Te1/3/8 - Interface Up. state: bound


Contd..

Verify Individual FEX states in detail

Contd..

FEX: 120 Description: FEX0120 state: online

FEX version: 15.0(2)EX4

Extender Model: C6800IA-48FPD, Extender Serial: FOC1736W197

FCP ready: yes

Image Version Check: enforced

Fabric Portchannel Ports: 2

Fabric port for control traffic: Te1/3/5

Fabric interface state:

Po200 - Interface Up.



sup2T#

58



show fex <fex-id> detail

show switch virtual slot-map

test scp ping <vslot id of fex>

show module fex <fex-id>

show platform fex-debug fex <fex-id> sdp

show monitor event-trace fex clock <hh:mm>

show fex image bundle version

debug fex [error | sdp | srp | init] <fex-id> *

debug switch virtual fexmgr [error | event | packet] <fex-id> *

Other helpful commands Check in which state FEX is stuck

Get vslot id of fex and

Check if FEX Internal Network is Up

Check FEX diagnostics have passed

Check if SDP handshake was successful

Event trace utility to store all fex related events at different stages

Turn on common FEX debugs and restart FEX

Check the version and the stack member details

59



sup2T# show fex system platform usage

FEX id usage details

Fex-ids inuse: 110, 120

Fex-ids online: 110, 120

Total Used Free

----- ---- ----

12 2 10

FEX slot usage details

FEX-id Switch-id Vslot Pslot Status

------ -------- ----- ----- ------

110 3 50 1 In-use

120 4 51 2 In-use

120 4 52 1 In-use

Total Used Reserved Free

----- ---- -------- ----

21 3 0 18

FEX ports usage details

Contd..

Stacked FEX and Usage details FEX-id Switch-id Ports

------ -------- -----

110 3 48

120 4 96

Total Used Free

----- ---- ----

1008 144 864

Stack members usage details

FEX-id Switch-id Used Free

------ -------- ---- ----

110 3 1 4

120 4 2 3

VNTAG MGR Usage

-----------------------

Max unicast VIFs available 2048

Total unicast VIFs used 144

Max non-mdest VIFs available 1019

Total non-mdest VIFs used 2

Max mdest VIFs available 13309

Total mdest VIFs used 12

sup2T#

Total FEX ports supported is 1008 in this release (15.1(2)SY1)

Total FEX modules that can be stacked is 5

Total FEX ID’s configurable is 12 in this release (15.1(2)SY1)

Each FEX module will consume one vslot id

FEX 120 is a stacked fex with 2 switches bundled into one FEX id

60




– Topology




– Topology




– Topology


Agenda

61


Unicast Packet Forwarding through FEX’es Topology for Layer2 traffic

62

• VSS is Supervisor 2T based

system operating in Virtual

Switching mode

• FEX110 is dual homed to VSS

through fabric PortChannel 100

• FEX12 0 is dual homed to VSS

through fabric PortChannel 200

• One Host PC is directly

connected to sup2T uplink port

• Other two Host PCs are directly

connected to each of FEX’es

• All the hosts are on VLAN 10 with

subnet ip’s 10.10.10.0/24

10.10.10.2

10.10.10.3

10.10.10.4

VLA

N 1

0

Po100 Po200

FEX110 FEX120

VSS


FEX Interface Naming

interface GigabitEthernet 110/1/0/1

<Interface-type>/<fex-id>/<module>/<submode>/<port>

FEX-ID

101-199 Stack

Sub

Module FEX

Port

63


Unicast Packet Forwarding through FEX’es

sup2T# show ip arp


Internet 10.10.10.1 - 0007.7d75.7d80 ARPA Vlan10

Internet 10.10.10.2 0 0050.568b.6012 ARPA Vlan10

Internet 10.10.10.3 91 c8f9.f90c.2941 ARPA Vlan10

Internet 10.10.10.4 90 0024.c4a5.bcc1 ARPA Vlan10

sup2T#

Layer2 unicast traffic troubleshooting

64



sup2T# show mac address-table address 0024.c4a5.bcc1


<snip>

Displaying entries from active supervisor:


----+----+---------------+-------+-----+----------+-----------------------------

10 0024.c4a5.bcc1 dynamic Yes 230 Gi110/1/0/1

Displaying entries from DFC switch [1] linecard [3]:


----+----+---------------+-------+-----+----------+-----------------------------

* 10 0024.c4a5.bcc1 dynamic Yes 0 Gi110/1/0/1

Contd..


Contd..

Displaying entries from standby supervisor:


----+----+---------------+-------+-----+----------+-----------------------------

10 0024.c4a5.bcc1 dynamic Yes 225 Gi110/1/0/1



----+----+---------------+-------+-----+----------+-----------------------------

* 10 0024.c4a5.bcc1 dynamic Yes 230 Gi110/1/0/1

sup2T#

* Denotes the primary forwarding entry. This entry indicates the ingress DFC linecard on which this mac was first learned.

65



sup2T# show mac address-table address c8f9.f90c.2941


<snip>



----+----+---------------+-------+-----+----------+-----------------------------

10 c8f9.f90c.2941 dynamic Yes 45 Gi120/1/0/1



----+----+---------------+-------+-----+----------+-----------------------------

* 10 c8f9.f90c.2941 dynamic Yes 45 Gi120/1/0/1

Contd..


Contd..

Displaying entries from standby supervisor:


----+----+---------------+-------+-----+----------+-----------------------------





----+----+---------------+-------+-----+----------+-----------------------------


sup2T#

66



sup2T# show counters interface Gig110/1/0/1

Time since last clear

---------------------

Never

64 bit counters:

0. rxHCTotalPkts = 80764

1. txHCTotalPkts = 448759

2. rxHCUnicastPkts = 60545

3. txHCUnicastPkts = 0

4. rxHCMulticastPkts = 20219

5. txHCMulticastPkts = 448249

6. rxHCBroadcastPkts = 0

7. txHCBroadcastPkts = 510

8. rxHCOctets = 8626030

9. txHCOctets = 38547300

10. rxTxHCPkts64Octets = 372976

<snip>


sup2T# clear counters

sup2T# show counters interface Gig110/1/0/1 delta

Time since last clear

---------------------

00:00:07

64 bit counters:

0. rxHCTotalPkts = 1

1. txHCTotalPkts = 4

2. rxHCUnicastPkts = 1

3. txHCUnicastPkts = 0

4. rxHCMulticastPkts = 0

5. txHCMulticastPkts = 4

<snip>

67


Unicast Packet Forwarding through FEX’es Topology for Layer3 traffic

68

10.10.10.2

20.20.20.3 10.10.10.4

Po100

FEX110 FEX120

VSS

Gig120/1/0/1 Gig110/1/0/1

Gig1/2/1

Po200


Unicast Packet Forwarding through FEX’es Layer3 unicast traffic troubleshooting

sup2T# show ip arp 20.20.20.3


Internet 20.20.20.3 22 c8f9.f90c.2944 ARPA Vlan20

sup2T# show mac address-table address c8f9.f90c.2944

<snip>



---+----+-------------+----------+-------+-----+----------




----+----+---------------+-------+-------+------+-------------


sup2T# show ip arp 10.10.10.4


Internet 10.10.10.4 26 588d.093c.09c1 ARPA Vlan10

sup2T# show mac address-table address 588d.093c.09c1

<snip>



----+----+---------------+-------+-----+----------

10 588d.093c.09c1 dynamic Yes 0 Gi110/1/0/1



----+----+---------------+-------+-----+--------+-----------

* 10 588d.093c.09c1 dynamic Yes 5 Gi110/1/0/1

69



sup2T# show ip route 10.10.10.4


Known via "connected", distance 0, metric 0 (connected, via interface)


* directly connected, via Vlan10


sup2T#show ip route 20.20.20.3


Known via "connected", distance 0, metric 0 (connected, via interface)


* directly connected, via Vlan20


sup2T#

sup2T# show ip cef 10.10.10.4

10.10.10.4/32

attached to Vlan10

sup2T#show ip cef 20.20.20.3

20.20.20.3/32

attached to Vlan20

sup2T#show ip cef exact-route 20.20.20.3 10.10.10.4


sup2T#show ip cef exact-route 10.10.10.4 20.20.20.3


sup2T#

Use ‘exact-route’ command to get next hop adjacency of the packet between source and destination ip during CEF switching

70



sup2T# show platform hardware cef lookup 10.10.10.4

Codes: decap - Decapsulation, + - Push Label

Index Prefix Adjacency

50 10.10.10.4/32 Vl10 ,0024.c4a5.bcc1

sup2T#

sup2T# show platform hardware cef exact-route 20.20.20.3 10.10.10.4

Interface: Vl10, Next Hop: 10.10.10.4, ifnum: 0x9E, Destination Mac: 588d.093c.09c1 LIF: 0x2000000A

sup2T#

sup2T# show adjacency 10.10.10.4 detail

Protocol Interface Address

IP Vlan10 10.10.10.4(8)

0 packets, 0 bytes

epoch 0

sourced in sev-epoch 0

Encap length 14

0024C4A5BCC100077D757D800800

L2 destination address byte offset 0

L2 destination address byte length 6

Link-type after encap: ip

ARP

sup2T#

Rewrite(next hop)info. Read as

DMAC|SMAC|0800

71



sup2T# show platform hardware cef ip 10.10.10.4 detail

<snip>

Format:IPV4 (valid class vpn prefix)

M(50 ): 1 F 2FFF 255.255.255.255

V(50 ): 1 0 0 10.10.10.4

(A:163841, LS:0, NR:0, RI:0, DF:0 CP:0 DGTv:1, DGT:0)

sup2T# show platform hardware cef entry 50

Codes: decap - Decapsulation, + - Push Label

Index Prefix Adjacency

50 10.10.10.4/32 Vl10 ,0024.c4a5.bcc1

sup2T#

72



sup2T# show platform hardware cef adjacencies entry 163841 detail

Index: 163841 -- Valid entry (valid = 1) --

Adjacency fields:

___________________________________________________

|adj_stats = EN | fwd_stats = EN | trig = 0

|_________________|__________________|______________

|l3_enable = ON (classify as Layer3) | age = 3

|_________________|__________________|______________

|format = IP | rdt = OFF | ignr_emut = 0

|_________________|__________________|______________

|vpn = 0x3FFF | elif = 0xA | ri = 3

|_________________|__________________|______________

|top_sel = 0 | zone_enf = OFF | fltr_en = OFF

|_________________|__________________|______________

Contd..

RIT fields: The entry has a Layer2 Format

|add_shim_hdr= NO | rec_findex = N/A | rec_shim_op = N/A

|_________________|__________________|____________________

|rec_dti_type = N/A | rec_data = N/A

|____________________________________|____________________

|modify_smac = YES| modify_dmac = YES| egress_mcast = NO

|____________________________________|____________________

|ip_to_mac = NO

|_________________________________________________________

|dest_mac = 0024.c4a5.bcc1 | src_mac = 0007.7d75.7d80 |

|____________________________|_____________________________

Statistics: Packets = 0

Bytes = 0

73




– Topology




– Topology




– Topology


Agenda

74


Multicast packet forwarding through FEX Topology

75

10.10.10.2

FEX110 FEX120

VSS

Gig120/1/0/1 Gig110/1/0/1

Gig1/2/1 L3 Cloud

Vlan 20

Vlan 20 Vlan 10

Receiver 20.20.20.3 Receiver

172.16.25.2 Source


Multicast Packet Forwarding through FEX’es Layer3 multicast traffic troubleshooting

sup2T# show ip mroute 239.1.1.1

IP Multicast Routing Table

Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,

<snip>

(*, 239.1.1.1), 1w0d/00:02:51, RP 10.10.10.1, flags: SJC

Incoming interface: Null, RPF nbr 0.0.0.0

Outgoing interface list:

Vlan10, Forward/Sparse, 03:05:38/00:02:32


(172.16.25.2, 239.1.1.1), 03:28:03/00:02:46, flags: T

Incoming interface: Vlan20, RPF nbr 20.20.20.22

Outgoing interface list:


sup2T#

*,G entry

S,G entry

OIL

RPF Neighbor

Incoming interface

76



sup2T# show ip igmp groups 239.1.1.1

IGMP Connected Group Membership

Group Address Interface Uptime Expires Last Reporter Group Accounted

239.1.1.1 Vlan10 02:59:40 00:02:33 10.10.10.2

239.1.1.1 Vlan20 6d02h 00:02:23 20.20.20.3

sup2T# show mac address-table multicast igmp-snooping 239.1.1.1

vlan mac/ip address LTL ports

+----+-----------------------------------------+------+-----------------------

10 ( *,239.1.1.1) 0x6112 Router Po100 Gi110/1/0/1

20 ( *,239.1.1.1) 0x6116 Router Gi1/2/1 Po20 Po200 Gi120/1/0/1

sup2T#

Both vlan’s show as part of this igmp group (239.1.1.1)

Local Target Logic is used to determine the physical port that will forward this packet

77



sup2T# test platform software switch virtual vntag_mgr vif-map ltl 0x6112 detail

VIF INFO:

VIF# 3072

Type MULTI-DESTINATION VIF

LTL# 6112

OperStatus# 2

Number of Ports:1

List of Ports: Gi110/1/0/1,

sup2T#

Check which FEX ports will carry

this multicast traffic

Provides a mapping between

VIF and LTL index

78



sup2T# test platform software switch virtual vntag_mgr vif-map ltl 0x6116 detail

VIF INFO:

VIF# 3074

Type MULTI-DESTINATION VIF

LTL# 6116

OperStatus# 2

Number of Ports:1

List of Ports: Gi120/1/0/1,

sup2T#

79


Multicast Packet Forwarding through FEX’es

sup2T# show platform hardware multicast routing ip source 172.16.25.2 group 239.1.1.1 detail

IPv4 Multicast CEF Entries for VPN#0

(172.16.25.2, 239.1.1.1/32)

FIBAddr: 0x40 IOSVPN: 0 RpfType: SglRpfChk SrcRpf: Vl20

CPx: 0 s_star_pri: 1 non-rpf drop: 0

PIAdjPtr: 0x28002 Format: IP rdt: off elif: 0xC5409

fltr_en: off idx_sel/bndl_en: 0 dec_ttl: on mtu_idx: 2(1518)

PV: 1 rwtype: MCAST_L2_EXPS

met3: 0x0 met2: 0x8022

Packets: 13961 Bytes: 14351908

Contd..

Layer3 multicast traffic troubleshooting

Check for hardware programming.

Ensure packets/bytes are incrementing

PIAdjPtr provides statistics for ingress

LC carrying mcast traffic on same VLAN

80



sup2T-sw1-dfc3# show platform hardware multicast routing ip source 172.16.25.2 group 239.1.1.1 detail

IPv4 Multicast CEF Entries for VPN#0

<snip>

NPIAdjPtr: 0x18003 Format: IP rdt: on elif: 0xC5409

fltr_en: off idx_sel/bndl_en: 0 dec_ttl: off

PV: 0 rwtype: MCAST_L3_REWRITE

met3: 0x58 met2: 0x0 DestNdx: 0x7FF3

Packets: 13325 Bytes: 13698100

MET offset: 0x58

OIF AdjPtr Elif CR (Sw1)

+-------------+-----------+-----------+------------+

Vl10 0x800A 0xA 3/T2

sup2T-sw1-dfc3#

Check for correct hardware programming.

Ensure packets/bytes are incrementing

Check this on the egress module/dfc

NPIAdjPtr provides statistics for egress

LC carrying mcast traffic

81

Questions ?


Complete Your Online Session Evaluation

• Give us your feedback and you could win fabulous prizes. Winners announced daily.

• Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center.

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

83

https://www.ciscolive.com/online


Continue Your Education

• Demos in the Cisco Campus

• Walk-in Self-Paced Labs

• Table Topics

• Meet the Engineer 1:1 meetings

84

advanced cisco catalyst 6500/6800 series...

Documents