troubleshooting cisco catalyst 3850...

Troubleshooting Cisco Catalyst 3850 Series Switches BRKCRS-3146

Naoshad Mehta

Principal Engineer

Twitter: @Naoshad #convergedaccess #cat3850 #CLUS

© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public

Troubleshooting Cisco Catalyst 3850 Series Switches Session Overview and Objectives

3

Cisco is bringing together the best of wired and wireless networking

into “One Network” with Converged Access on the Catalyst 3850 Switch

In this session, learn about the capabilities of the 3850 switch and

troubleshoot common issues seen on the 3850 running the IOS-XE

Operating System. Learn about the switch architecture and troubleshooting

hardware, RTU Licensing, Boot-up Sequence, Memory and CPU utilization,

Stacking, High Availability, Forwarding features on the UADP ASIC, QoS and

Wireless.


Your Instructor today … Naoshad Mehta

Principal Engineer, Enterprise Switching Software

4

I’m a Principal Engineer with the Enterprise Switching Software team at Cisco. My current

focus is the adoption of Catalyst 3850 and Converged Access Architecture in the marketplace.

I’ve been with Cisco for 13+ years. My primary responsibility since 2010 was the delivery of

the Catalyst 3850 and CT5760 Wireless Controller. I have been intimately involved with the

design and implementation of almost every software aspect of the 3850 and I’m here to help

you learn more about the architecture and how to troubleshoot the 3850.

Prior to working on the 3850, I have worked on a wide spectrum of technologies (MPLS, Traffic

Engineering, L2VPN, EVCs, etc.), Products (Nexus 7K, 7600, 7500, 7200) and Operating

Systems (Classic IOS, NXOS and IOS-XE).


Agenda Architecture & Product Overview

Multicast Forwarding

Hardware Troubleshooting

Installer and Bootup

Licensing

Memory and CPU

Unicast Forwarding

CPU Punt Path

Stacking

High Availability

Wireless

QoS

Misc. Tools and Tricks

Summary


Glossary

6

A S Active Switch Standby Switch

3850 Switch

MC MA Wireless Mobility Agent

Wireless Mobility Controller

FED – Forwarding Engine Driver WCM – Wireless Controller Module

PDS – Packet Delivery Service UADP – Unified Access Data Plane ASIC

Reference slide that may not be presented in the session

Confusing command/output. Engineering team is committed to fix these.


Suggested Sessions and Reference Material

BRKCRS-2889 - Converged Access System Architecture - Diving into the 'One Network’

BRKCRS-2888 - Converged Access Design Options

BRKARC-3438 - Cisco Catalyst 3850 Series Switching Architecture

Converged Access White Paper – Wired/Wireless System Architecture, Design and Operation - bit.ly/1a4YRh2

7

http://t.co/zAlQEtp0Mq








Licensing

Memory and CPU

Unicast Forwarding

CPU Punt Path

Stacking

High Availability

Wireless

QoS


Summary


In this section, you will learn about …

9

Overview of the 3850

IOS-XE Evolution

IOS-XE Architecture


Catalyst 3850 Switch

Bui l t on C isco ’s Innova t i ve “UADP” ASIC

* Roadmap

Wireless CAPWAP Termination

Integrated Controller: Up to 50 APs and 40G per switch

Up to 2000 Clients per Stack

40 Gbps Uplink Bandwidth

Stackpower

Line Rate on All Ports

SGT/SGACL*

480 Gbps Stacking Bandwidth

Full POE+

FRU Fans, Power Supplies

Granular QoS/Flexible

NetFlow

Best-in-Class Wired Switch – with Integrated

Wireless Mobility functionality


IOS

IOS 12.2(52)SE

IOS XE Evolution

11

IOS XE 3.2.2(SE)

Management Interface

Module Drivers

Common Infrastructure / HA

I OS -X E

• Modern IOS to enable multi-core CPU

• Easy customer migration

• While maintaining IOS functionality and look and feel

• Allow hosted applications like Wireshark

Management Interface

Module Drivers

Linux Kernel

Common Infrastructure / HA

IOSd

Features Components

Hosted Apps

Features Components WCM

Kernel


Internal IPC

Avail

ab

ilit

y F

ram

ew

ork

Packet Delivery Service

Service

Location

Forwarding &

Feature Mgr (FFM)

System

Manager

Platform

Manager

Consolidated

Logging

Comet

Services

Licensing

Services

Interface

Manager

Libraries/

Utilities

Services

External

Transports

(TCP/SCTP/UDP)

Wireless Controller HA

Stack Manager (3K)

IOS XE Software Internals Overview

Kernel

IOS

d R

P/L

C

Features PD

Platform

Drivers

Low Level APIs

UADP ASIC

Drivers

Forwarding Engine Driver

12



Architecture & Product Overview

Agenda



Licensing

Memory and CPU

Unicast Forwarding

CPU Punt Path

Stacking

High Availability

Wireless

QoS


Summary



14

Front Panel LEDs

Rear Panel LEDs and Ports

Supported uplink modules


System LEDs Overview

15


System LEDs – Definitions

• SYST LED

Off = System off

Green = System operating normally

Blinking green = Running POST

Amber = System is malfunctioning

Blinking amber = Network module, power supply, or fan module is malfunctioning

• XPS LED

Off = No XPS cable installed or switch is in StackPower mode

Green = XPS connected and ready to provide backup power

Blinking green = XPS is connected but cannot provide backup power

Amber = XPS is in standby or a fault condition

Blinking amber = Power supply in the switch has failed and is being backed up by XPS

• ACTV LED

Off = Switch is not the active switch

Green = Switch is the active switch or is in standalone mode

Blinking green = Switch is in standby mode

Amber = An error has occurred in the data stack, possibly related to active member selection

• S-PWR LED

Off = StackPower cable not connected or switch is in standalone mode

Green = Switch is connected to an XPS or to 2 StackPower neighbors in a ring configuration

Blinking green = Switch is connected to only 1 StackPower neighbor in a ring configuration

Amber = Fault detected

Blinking amber = StackPower configuration is overbudget

16


System LEDs – Definitions (cont.) • STAT LED

Off = Rather than indicating link status, the port LEDs are indicating duplex, speed, stack, or PoE status

Green = Port LEDs are indicating link status

• DUPLX LED

Off = Rather than indicating duplex status, the port LEDs are indicating link, speed, stack, or PoE status

Green = Port LEDs are indicating duplex status

• SPEED LED

Off = Rather than indicating speed status, the port LEDs are indicating link, duplex, stack, or PoE status

Green = Port LEDs are indicating speed status

• STACK LED

Off = Rather than indicating stack status, the port LEDs are indicating link, duplex, speed, or PoE status

Green = Port LEDs are indicating stack status

• PoE LED

Off = Rather than indicating PoE status, the port LEDs are indicating link, duplex, speed, or stack status; None of the downlink ports have been denied power or are in a fault condition

Green = Port LEDs are indicating PoE status and none of the downlink ports have been denied power or are in a fault condition

Blinking amber = Port LEDs are indicating PoE status and at least one of the downlink ports has been denied power or is in a fault condition

• CONSOLE LED

Off = USB console is inactive

Green = USB console is active (RJ45 console is inactive)

17


Rear Panel Ports & Associated LEDs • CONSOLE SERIAL LED

Off = RJ45 console is inactive (USB console is active)

Green = RJ45 console is active (USB console is inactive)

• MGMT LED

Off = Link down

Green = Link is up with no activity

Blinking green = Link is up with activity

18


Network Modules

WS-C3850-NM-4-1G

• 4 x 1G

• SFP

• Supported on WS-C3850-24 & WS-C3850-48 Port

WS-C3850-NM-2-10G

• 4 x 1G OR 2 x 10G OR 2 x 1G + 1 x 10G

• SFP & SFP+

• Supported on WS-C3850-24 & WS-C3850-48 Port

WS-C3850-NM-4-10G

• Auto-sensing – All Combinations

• SFP & SFP+

• Supported on WS-C3850-48 only

19

Type of interface 4x1G 2x1G 2x10G 4x10G

1G GigabitEthernet 1 - 4 GigabitEthernet 1 - 2 &

TenGigabitEthernet 3 - 4

TenGigabitEthernet 1 – 4

10G - TenGigabitEthernet 3 - 4

TenGigabitEthernet 1 – 4


Network Modules – Troubleshooting

• All network modules are hot-pluggable, so they can easily be swapped between units to help narrow down or rule out hardware failures

• As expected, optical modules are also hot-pluggable and can be easily rotated through ports to home in on or rule out hardware failures

If link is down on a module, one often overlooked yet common cause is dirty or damaged fiber

• Each port has a bicolor (green/amber) LED that provides a visual indication of its status

Off = Link down

Solid green = Link up, no activity

Blinking green = Link up with activity

Solid amber = Link disabled

Blinking amber = Link is off due to a fault or because a limit set via software has been exceeded

20


Licensing





Memory and CPU

Unicast Forwarding

CPU Punt Path

Stacking

High Availability

Wireless

QoS


Summary



22

3850 Image naming convention

Packages in the image

Install vs. bundle boot

Password recovery


3850 Image Naming Convention

23

cat3k_caa-universalk9.SPA.03.02.01.SE.150-1.EX1.bin

cat3k = Platform Family

C = Converged

A = Access

A = Access Switch

Feature Set

Enabling/Disabling of

features controlled by

installed license

S = Digitally signed Image

P = Production Image

A = Key Version

IOS XE Version IOSd Version


Booting IOS XE Software

Install Boot (default mode)

• Packages are installed on flash

• Supports AP image pre-download

• No additional memory requirement

• Image must be installed in flash:

• boot flash:packages.conf

Bundle Boot

• Packages are expanded in RAM

• No support for AP image pre-

download

• Additional memory equal to the size

of image bundle required

• Image can be booted from flash:,

usbflash: or tftp:

• boot flash:cat3k_caa-

universalk9.SPA.03.02.01.SE.1

50-1.EX1.bin

24


IOS XE Packages

Package Name Contents

cat3k_caa-base Kernel distribution and Open Source Software

cat3k_caa-infra IOS XE infrastructure software – Installer, HA

manager, etc.

cat3k_caa-iosd-universalk9 IOSd Software

cat3k_caa-platform Platform software – FED, Stack Manager, etc.

cat3k_caa-drivers Platform Drivers and UADP libraries

cat3k_caa-wcm Wireless Controller Module software, AP images

and Wireless Web GUI 25


Common Bootup Issues

Switch does not auto-boot when rebooted after saving config

– Caused by CSCue76684

– Upgrade to software version 3.2.1SE or later

– Workaround 1: At the “switch:” prompt enter “boot flash:packages.conf”

– Workaround 2: Configure “boot system switch all flash:packages.conf” under the global config mode before saving config

Switch does not auto-boot when rebooted after configuring “boot system …”

– Root Cause: “boot config” is configured in startup-config

– Workaround: Remove “boot config …” from startup-config

– Fixed in software version 3.2.2SE

26


• Password recovery on 3850 does NOT follow the 3750 family procedure

• 3850 password recovery is as follows:

1. Power cycle switch and hold the Mode button (on the front top left) for a few seconds (officially 12) until the status LED gets amber, that will get you in Boot Loader prompt (Switch:)

1. Set the following variables

2. Boot the 3850

3850 Password recovery

Switch: SWITCH_IGNORE_STARTUP_CFG=1

Switch: SWITCH_DISABLE_PASSWORD_RECOVERY=0

Switch:

Switch: boot

Warning!

Console:9600 baud

8 data bits, No flow control

1 stop bit, No parity

27


4. Skip the initial configuration dialog and go to enable (no password required):

5. Copy startup-config back to running-config:


--- System Configuration Dialog ---

Would you like to enter the initial configuration dialog? [yes/no]: no

Would you like to terminate autoinstall? [yes]: yes

Press RETURN to get started!

Switch> enable

Switch#

Switch# copy startup-config running-config

28


6. Go to global configuration, and remove or change the password:

7. Go to exec mode and enable reading of startup-config

8. Disable password recovery if required


Switch# configure terminal

Switch(config)# no enable password

Switch(config)# no enable secret

Switch(config)# enable secret cisco

Switch(config)# end

Switch# write (copy running-config startup-config)

Switch# no system ignore startupconfig switch all

Switch# system disable password recovery switch all

29


Software Upgrade on 3850

30

Software upgrade in Installed Mode is done via the “software install …” command

Prerequisites for software installation:

The switch’s free memory must be greater than the size of the bundle being installed

The free space in flash: must be greater than the size of the bundle being installed

All switches must be running in installed mode

When installing a bundle from a local storage device, the device must exist on all switches performing

the installation operation

The packages in the bundle to be installed must have valid digital signatures

A failed installation might require a rollback using “software rollback” command or a

manual clean using “software clean” command.


Upgrade/Install a Bundle on flash

Switch# software install file flash:cat3k_caa-universalk9.SPA.03.02.01.SE.150-1.EX1.bin Preparing install operation ... [2]: Copying software from active switch 2 to switch 1 [2]: Finished copying software to switch 1 [1 2]: Starting install operation [1 2]: Expanding bundle flash:cat3k_caa-… [1 2]: Copying package files [1 2]: Package files copied [1 2]: Finished expanding bundle flash:cat3k_caa-… [1 2]: Verifying and copying expanded package files to flash: [1 2]: Verified and copied expanded package files to flash: [1 2]: Starting compatibility checks [1 2]: Finished compatibility checks [1 2]: Starting application pre-installation processing [1 2]: Finished application pre-installation processing [1]: Old files list: Removed cat3k_caa-base.SPA.03.02.00.SE.pkg Removed cat3k_caa-drivers.SPA.03.00.00.SE.pkg Removed cat3k_caa-infra.SPA.03.02.00.SE.pkg Removed cat3k_caa-iosd-universalk9.SPA.150-1.EX.pkg Removed cat3k_caa-platform.SPA.03.02.00.SE.pkg Removed cat3k_caa-wcm.SPA.03.02.00.SE.pkg

Preparation stage

Installing to Flash

Post Install Checks

Removing old files

31


Software Rollback Use the ‘software rollback’ command to revert to the previously installed package set (packages.conf.00-).

Switch# software rollback Preparing rollback operation ... [2]: Starting rollback operation [2]: Starting compatibility checks [2]: Finished compatibility checks [2]: Starting application pre-installation processing [2]: Finished application pre-installation processing [2]: Old files list: Removed cat3k_caa-base.SPA.03.02.01.SE.pkg Removed cat3k_caa-drivers.SPA.03.02.01.SE.pkg Removed cat3k_caa-infra.SPA.03.02.01.SE.pkg Removed cat3k_caa-iosd-universalk9.SPA.150-1.EX1.pkg Removed cat3k_caa-platform.SPA.03.02.01.SE.pkg Removed cat3k_caa-wcm.SPA.03.02.01.SE.pkg [2]: New files list: Added cat3k_caa-base.SPA.03.02.00.SE.pkg Added cat3k_caa-drivers.SPA.03.02.00.SE.pkg Added cat3k_caa-infra.SPA.03.02.00.SE.pkg Added cat3k_caa-iosd-universalk9.SSA.150-1.EX.pkg Added cat3k_caa-platform.SPA.03.02.00.SE.pkg Added cat3k_caa-wcm.SPA.03.02.00.SE.pkg [2]: Creating pending provisioning file [2]: Finished rolling back software changes. New software will load on reboot. [2]: Do you want to proceed with reload? [yes/no]: n Switch#

Removed newly installed image

Reverted to older image

32


Recover a Corrupted Install

33

Copy the image bundle to USB flash and bootup using the following command from

the Bootloader prompt:

boot usbflash0:cat3k_caa-universalk9.SPA.03.02.01.SE.150-1.EX1.bin

Copy the image bundle to USB flash and recover the switch by using the recovery

mechanism built into the switch from the Bootloader prompt:

emergency-install usbflash0:cat3k_caa-

universalk9.SPA.03.02.01.SE.150-1.EX1.bin


Memory and CPU


Licensing




Unicast Forwarding

CPU Punt Path

Stacking

High Availability

Wireless

QoS


Summary



35

Right To Use (RTU) Licensing on the 3850

How to resolve license mismatch in stack

RTU Licensing show commands

Wireless AP Licensing

Wireless License Behavior in stack


Right To Use (RTU) / Honor Based Licensing

Trust Based Licensing Model

Built in licenses, not tied to Unique Device Identifier

Three license levels – lanbase, ipbase and ipservices

Activated using CLI by accepting the End User License Agreement

Portable across devices

No Need to access cisco.com License Portal

36


License Mismatch

37

license right-to-use deactivate ipservices

license right-to-use activate ipbase acceptEULA

Reload switch

IP Base

IP Base

IP Base IP

Services

A

S


Licensing Show commands

38

Switch# show license right-to-use slot 1

Slot# License name Type Count Period left

----------------------------------------------------------

1 ipbase permanent N/A Lifetime

1 lanbase permanent N/A Lifetime

1 apcount adder 4 Lifetime

License Level on Reboot: ipservices

Switch# show license right-to-use mismatch

Slot# License Name Adder AP Count Base AP Count

---------------------------------------------------------------

3 ipservices 0 0


Wireless Licenses

50 AP Licenses can be applied on a 3850 stack

Licenses can be applied on individual switches or on the Active

Licenses can be applied in increments of 1

AP adder licenses are applied using the CLI: license right-to-use activate apcount 10 slot 1 acceptEULA

AP licenses are applied on the Mobility Controller

39


Wireless License Behavior

40

4 AP

License

S

A

A

No License. AP

unable to join.

Active loses power

S

Recommendation is to apply licenses on each switch based on the number of APs connected to the switch.

license right-to-use activate apcount 2 slot 1

license right-to-use activate apcount 2 slot 2


Licensing

CPU Punt Path

Memory and CPU





Unicast Forwarding

Stacking

High Availability

Wireless

QoS


Summary



42

3850 CPU Complex

Troubleshooting memory utilization

Troubleshooting high CPU utilization


CPU Complex

43

Cavium 6230

800 MHz, 4 core CPU

2MB L2 Cache

UADP 1

UADP 2

USB/RJ-45 Console 10/100/1000 RJ-45

Ethernet Mgmt

SGMII UART

PCIe

PCIe

4GB DDR3

w/ ECC

DDR3 - 1333

FPGA for

Stack Power

I2C

RTC

ACT II

FPGA for PHY,

LED, etc.

I2C

2GB Flash 64MB

Bootloader

Boot Bus


High Memory Utilization Frequently Asked Questions

44

Why should I be concerned about high memory utilization ? It is very important have enough free memory to support features and network convergence events that require

transient memory.

What are the usual symptoms of high memory usage ?

Memory utilization of process(es) keeps increasing

System runs out of buffers and software packet forwarding stops

Memory allocation failures are reported

System crashes after reporting out of memory

At what percentage level should I start troubleshooting ? It depends on the nature and level of feature config on the switch. It is very essential to find a baseline memory

usage during normal working conditions, and start troubleshooting when it goes above specific threshold.

E.g., Baseline memory usage 40%. Start troubleshooting when the memory goes above 60% and constantly keeps

increasing without adding any new configuration.


Switch1# show processes memory sorted

System memory : 3930916K total, 1118032K used, 2812884K free, 221968K kernel reserved

Lowest(b) : 2252987972

PID Text Data Stack Heap RSS Total Process

10623 56892 36452 92 5400 196116 336728 iosd

5534 8716 311168 92 4620 136908 562460 fed

10619 21976 555372 88 13980 102320 723240 wcm

6032 4 97708 116 91996 99044 116676 idope.py

12259 4 193244 236 38244 73672 299464 wnweb_paster.py

5536 660 163524 88 4332 55968 336496 stack-mgr

6057 3532 137308 88 2200 54200 311676 ffm

6076 112 160908 88 6764 44728 233548 cli_agent

6058 1232 287972 88 8112 38352 438040 eicored

Memory show commands

45

Total Memory IOS-XE Process


Memory show commands

46

Switch1# show processes memory detailed process iosd sorted

Processor Pool Total: 536870912 Used: 135242980 Free: 401627932

IOS Proce Pool Total: 16777216 Used: 9483360 Free: 7293856

PID TTY Allocated Freed Holding Getbufs Retbufs Process

0 0 168268072 31876024 126376204 0 0 *Init*

164 0 1534944 0 1558112 907264 0 NGWC DOT1X Proce

0 0 0 0 984492 0 0 *MallocLite*

1 0 657344 1544 678968 0 0 Chunk Manager

276 0 925564 297800 563696 0 0 os_info_p provid

39 0 415892 1856 376480 0 0 IPC Seat RX Cont

250 0 298204 464 320908 0 0 IPC LC Message H

IOS Tasks


Common Causes for high memory utilization

Common Cause Recommended Solution

Extensive Config Reduce configuration to supported scale

Excessive memory allocated to trace buffers Reset trace buffers to default sizes

DoS Attack/Punted traffic causing buffer

depletion

Identify packets and block them using an

ACL

Protocol flaps/re-convergence causing high

transient memory utilization

Identify reason for network instability

Memory Leak caused by software bug Open a Service Request

47


Troubleshooting Memory

Troubleshooting Steps Commands

Check memory usage on system show processes memory sorted

Check memory usage of a particular process show processes memory detailed process fed

Check memory usage of IOSd show processes memory detailed process iosd

Check allocators of memory within IOSd show memory detailed process iosd allocating-

process totals

Command Summary

48


High CPU Utilization Frequently Asked Questions

49

Why should I be concerned about high CPU utilization ? It is very important to protect the control plane for network stability, as resources (CPU, Memory and buffer) are

shared by control plane and data plane traffic (sent to CPU for further processing)

What are the usual symptoms of high CPU usage ?

Control plane instability e.g., OSPF flap

Reduced switching / forwarding performance

Slow response to Telnet / SSH

SNMP poll miss

At what percentage level should I start troubleshooting ? It depends on the nature and level of the traffic. It is very essential to find a baseline CPU usage during normal

working conditions, and start troubleshooting when it goes above a specific threshold.

E.g., Baseline CPU usage 25%. Start troubleshooting when the CPU usage is consistently at 50% or above.


Switch# show proc cpu sorted

Core 0: CPU utilization for five seconds: 96%; one minute: 7%; five minutes: 6%




PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

5533 120300 1608989 74 0.29 0.40 0.42 1088 fed

5535 44890 1401868 32 0.24 0.11 0.10 0 stack-mgr

10582 416280 5787047 71 34.25 0.57 0.62 34816 iosd

6201 111520 119850 930 0.15 0.15 0.15 0 cpumemd

5534 38430 3608873 10 0.10 0.10 0.10 0 platform_mgr

10578 115030 4737397 24 0.10 0.12 0.11 0 wcm

5455 1500 40856 36 0.05 0.05 0.05 1088 slproc

6183 5270 211347 24 0.05 0.02 0.04 0 obfld

6185 4320 110250 39 0.05 0.01 0.03 0 console_relay

6198 20900 186795 111 0.05 0.02 0.00 0 ffm

1 1700 1112 1528 0.00 0.09 1.43 0 init

2 0 138 0 0.00 0.00 0.00 0 kthreadd

3 10 1634 6 0.00 0.00 0.00 0 migration/0

4 0 3 0 0.00 0.00 0.00 0 sirq-high/0

Troubleshooting High CPU

50

Quad Core CPU

Platform Processes

IOS-XE Processes

137% across 4

cores

Identify the Culprit



51

Switch# show processes cpu detailed process iosd sorted





PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

(%) (%) (%)

10582 L 451160 6379641 70 34.25 0.71 0.60 34816 iosd

10582 L 0 10582 414060 6194757 0 24.00 0.60 0.50 34816 iosd

10582 L 3 11543 36980 180107 0 10.25 0.11 0.10 0 iosd.fastpath

10582 L 2 11544 120 4777 0 0.00 0.00 0.00 34816 iosd.aux

6 I 57680 5216 0 3.00 0.33 0.22 0 Check heaps

304 I 2200 1790 0 12.17 0.00 0.00 0 HTTP CORE

218 I 2370 14495 0 8.33 0.00 0.00 0 IP Input

211 I 190 214 0 0.33 0.00 0.00 0 RSMP Server

306 I 10 23 0 0.11 0.00 0.00 0 SEP NODE PROC

5 I 0 2 0 0.00 0.00 0.00 0 IPC ISSU Dispatch P

7 I 220 336 0 0.00 0.00 0.00 0 Pool Manager

3 I 0 1 0 0.00 0.00 0.00 0 HA-IDB-SYNC

High CPU caused

by HTTP traffic

Interrupt switched

traffic (eg. Wireless

Control)

Drill Down Deeper



Troubleshooting Steps Commands

Check CPU usage on IOS threads show process cpu detailed process iosd

[sorted]

Check CPU usage on platform dependent and

Nova threads

show process cpu detailed process {fed |

platform_mgr | stack-mgr | ha_mgr | eicored…}

Check traffic on the RX and TX CPU queues show platform punt client, show platform punt tx

Check details of CPU queues show platform punt statistics port-asic 0 cpuq 0

direction {rx | tx}

Command Summary

52


Stacking

Memory and CPU

Licensing

CPU Punt Path





Unicast Forwarding

High Availability

Wireless

QoS


Summary



54

3850 CPU Punt Path

Reasons for punting packets to CPU

Identifying and capturing packets punted to CPU


CPU Punt Path Architecture

55

IOSd WCM Punt Shim

Forwarding Engine Driver

Packet Handler

UADP ASIC

32 RX PDS Queues 8 TX PDS Queues

32 RX Queues 8 TX Queues

Processes

Control Packets

Processes

Wireless Control

Packets

Interfaces with

UADP ASIC and

Packet Delivery

Service (PDS)


Common Cause for Punting Traffic to CPU

Common Cause Recommended Solution

Same interface forwarding change design, use “no ip redirect”

ACL logging disable ACL logging

ACL deny causing switch to send ICMP

unreachable

no ip unreachables1

Forwarding/Feature exception (out of TCAM/adj

space)

reduce TCAM usage

resize TCAM region (TCAM2/3)

SW-supported feature disable the feature or reduce the amount of

traffic

IP packets with TTL<2 or options disable the offending traffic

Broadcast Storm Fix STP loop, disable traffic

Unexpected control/data traffic Control Plane Policing (CoPP), Deny ACL

Software Bug Open a Service Request

1. Should be configured on all the L3 interfaces of the switch.

56


Decoding CPU Queues

57

Switch# show platform punt client

tag buffer jumbo fallback packets received failures

alloc free bytes conv buf

65536 0/1024/1600 0/0 0/512 64845 64845 3371071 0 0

65544 0/ 96/1600 0/4 0/0 0 0 0 0 0

65545 0/ 96/1600 0/8 0/32 1947 1947 612588 0 0

65546 0/ 512/1600 0/32 0/512 13563 137795 24587306 0 0

65548 0/ 512/1600 0/32 0/256 10903 10903 650232 0 0

65551 0/ 512/1600 0/0 0/256 56 56 12088 0 0

65561 411/ 512/1600 0/0 0/128 557245 556834 39010862 0 0

65562 0/ 512/1600 0/16 0/256 0 0 0 0 0

CPU Queue Number

25 (65561-65536) Number of packets in

queue awaiting

processing

Size of Queue Size of each buffer


Displaying packets in the queue

show buffers detailed process iosd assigned packet | beg ng3k_rx25

Buffer information for ng3k_rx25 buffer at 0x35E98E8C

data_area 0x35E9932C, refcount 1, next 0x0, flags 0x80

linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1

if_input Vlan10, if_output 0x0 (None)

source: 10.32.111.83, destination: 10.33.21.219, id: 0x4BE0, ttl: 63,

TOS: 0 prot: 6, source port 51378, destination port 22

35E99382: 6400F124 F1C11410 9FE43A49 08004500 d.q$qA...d:I..E.

35E99392: 00984BE0 40003F06 56110A20 6F530A21 ..K`@.?.V.. oS.!

35E993A2: 15DBC8B2 0016588A DB9F6C34 421A5018 .[H2..X.[.l4B.P.

35E993B2: FFFF8666 000072A2 E1AB5431 78970F84 ...f..r"a+T1x...

58


CPU Punt Path

High Availability

Stacking

Memory and CPU

Licensing





Unicast Forwarding

Wireless

QoS


Summary



60

3850 Stacking Architecture

Stacking Show commands

Troubleshooting failure to form a stack


3850 StackWise-480 Overview

• 3850 StackWise-480 is a new generation of Catalyst 3850 stacking

240Gbps of bandwidth (120Gbps TX & 120Gbps RX per connector)

Similar to previous stacking implementations, ring redundancy is achieved via ring-wrap capabilities provided in hardware

NOT backward compatible with currently fielded stacking technologies, most notably StackWise Plus

61


3850 StackWise-480 Cables

• StackWise-480 currently supports 3 cables

STACK-T1-50CM = 0.5m cable

STACK-T1-1M = 1m cable

STACK-T1-3M = 3m cable

• All StackWise-480 cables include ACT II chips for counterfeit protection

62


• 6 rings in total

• 3 rings go East

• 3 rings go West

• Each ring is 40G

• Total Stack BW = 240G

• With Spatial Reuse = 480G

Stack Interface of UADP

Stack Interface of UADP ASIC

Assuming 4 x 24-port 3850 Switches

Packets are segmented/reassembled in HW (256 byte

segments)

Understanding the Stack Ring

Is math really an

opinion?

63


Destination Stripping Packet travels ½ the rings. Taken out of stack by destination

1 3

1 3

Assuming 4 x 24-port 3850 Switches

4 2

4 2

Spatial Reuse

64


Show commands

65

Switch# show switch detail

Switch/Stack Mac Address : 6400.f124.df80 - Local Mac Address

Mac persistency wait time: Indefinite

H/W Current

Switch# Role Mac Address Priority Version State

------------------------------------------------------------

*1 Active 6400.f124.df80 10 0 Ready

2 Standby 6400.f124.de80 1 0 Ready

Stack Port Status Neighbors

Switch# Port 1 Port 2 Port 1 Port 2

--------------------------------------------------------

1 OK OK 2 2

2 OK OK 1 1

Priority, followed by MAC

Address determines which

switch gets elected as Active


Show commands

66

Switch# show switch stack-ports summary

Sw#/Port# Port Status Neighbor Cable Length Link OK Link Active Sync OK #Changes to LinkOK In Loopback

---------------------------------------------------------------------------------------------------------------

1/1 OK 2 50cm Yes Yes Yes 0 No

1/2 OK 2 Unknown Yes Yes Yes 0 No



Cable with corrupted

EEPROM


Image Version Mismatch

If the switches are in version mismatch state, they will not stack

Debugging:

– Check if the switch version matches the active using show version command

– If they do not match, upgrade the switch to the Active’s version

Switch# show switch


---------------------------------------------------------------------------

*1 Active 6400.f125.1480 1 V01 Ready

2 Member 6400.f125.2680 1 0 V-Mismatch



67


Switch Stuck in Syncing State

If a switch is stuck in syncing state

Debugging:

– Run the command “sh switch” to see the states

– Open a Service Request with Cisco TAC

Switch# show switch


---------------------------------------------------------------------------

*1 Active 6400.f125.1480 1 V01 Ready

2 Standby 6400.f125.2680 1 V01 Ready

3 Member 6400.f125.2500 1 V01 Ready

4 Member 6400.f125.2480 1 0 Syncing

68


Stacking

Unicast Forwarding

CPU Punt Path

High Availability

Memory and CPU

Licensing





Wireless

QoS


Summary



70

3850 HA Architecture

Election of Active and Standby

Show commands for checking HA states

Troubleshooting


Catalyst 3750-X – StackWise-Plus - Hybrid control-plane processing

- N:1 stateless control-plane redundancy

- Distributed L2/L3 Forwarding Redundancy

- Stateless L3 protocol Redundancy

Catalyst 3850 – StackWise-480 - Centralized control-plane processing

- 1+1 Stateful redundancy (SSO)

- Distributed L2/L3 Forwarding Redundancy

- IOS HA Framework alignment for L3 protocol

HA Redundancy – Shift from 3750-X

71


HA SSO Architecture

72

Interfaces

L2

L3

QoS

Interfaces

L2

L3

QoS

Wireless

Wireless

Feature State is synced between Active and Standby Member in stack

Feature States are inactive on Standby Member

S

A


Route Processor Domain – a set of SW processes (e.g. IOSd, WCM) that

implement the centralized Active and Standby portions of the stack control plane

Line Card Domain – a set of SW processes (e.g. FED, Platform Manager) that

implement the distributed Line Card portions of the stack control plane

Infra Domain – Support SW for the RP and LC Domains

Active Switch – supports the Active RP Domain, a LC Domain and Infra Domain

Standby Switch – supports the Standby RP Domain, a LC Domain and Infra

Domain

Member Switch – supports a LC Domain and Infra Domain.

Election – assigning roles or functions within the stack

Software HA Processes– Roles and Definitions

73


RP Infra LC

RP Infra

Infra LC

Infra LC

S LC

• Active starts RP Domain

(IOSd, WCM, etc) locally

• Programs hardware on all LC Domains

• Traffic resumes once hardware is

programmed

• Starts 2min Timer to elect Standby in parallel

• Active elects Standby

• Standby starts RP Domain locally

• Starts Bulk Sync with Active RP

• Standby reaches “Standby Hot”

2min timer

A

Catalyst 3850 – HA State Machine

74


Switch# show switch

Switch/Stack Mac Address : 2037.06cf.0e80

H/W Current


------------------------------------------------------------

*1 Active 2037.06cf.0e80 10 PP Ready

2 Standby 2037.06cf.3380 8 PP Ready

3 Member 2037.06cf.1400 6 PP Ready

4 Member 2037.06cf.3000 4 PP Ready

* Indicates which member is providing the “stack Identity” (aka “stack MAC)

Active

Standby

Show switch with SSO

75


Show redundancy states

76

Switch# show redundancy states my state = 13 –ACTIVE

peer state = 8 -STANDBY HOT

Mode = Duplex

Unit ID = 2

Redundancy Mode (Operational) = SSO

Redundancy Mode (Configured) = SSO

Redundancy State = SSO

Manual Swact = enabled

Communications = Up

client count = 76

client_notification_TMR = 360000 milliseconds

keep_alive TMR = 9000 milliseconds

keep_alive count = 0

keep_alive threshold = 9

RF debug mask = 0

Terminal state for SSO. If “peer

state” is stuck in any other state

for more than 10 minutes, open

a service request with TAC

If Communication channel is not

Up, there might be a problem

with stack connectivity. Check

stack cable.


High Availability


Stacking

Unicast Forwarding

CPU Punt Path

Memory and CPU

Licensing




Wireless

QoS


Summary



78

Examine Hardware resources in use by each feature

Commands to debug Unicast L3 Forwarding on the 3850


Configuration and Show commands

No new configuration added for Unicast Features on 3850

Configuration, show commands compatible with 3750X

Additional Platform CLIs have been added

Refer to configuration guide and command line reference for full details

79


TCAM Utilization

80

Switch1# show platform tcam utilization asic all

CAM Utilization for ASIC# 0

Table Max Values Used Values

--------------------------------------------------------------------------

Unicast MAC addresses 32768/512 82/22

Directly or indirectly connected routes 32768/8192 7/89

IGMP and Multicast groups 8192/512 0/16

Security Access Control Entries 3072 173

QoS Access Control Entries 2816 52

Netflow ACEs 1024 15

Input Microflow policer ACEs 256 7

Output Microflow policer ACEs 256 7

Control Plane Entries 512 187

Policy Based Routing ACEs 1024 9

Tunnels 256 12

Input Security Associations 256 4

Output Security Associations and Policies 256 9


Unicast L3 Forwarding Show Commands

81

Switch1# show ip cef 72.163.4.161 internal

0.0.0.0/0, epoch 3, flags default route, RIB[S], refcount 6, per-destination sharing

sources: RIB, DRH

feature space:

Broker: linked, distributed at 1st priority

ifnums:

GigabitEthernet1/0/1(72): 172.25.33.1

path 32733FA4, path list 38B46BB8, share 1/1, type recursive, for IPv4

recursive via 172.25.33.1[IPv4:Default], fib 38B47A30, 1 terminal fib, v4:Default:172.25.33.1/32

path 32733F34, path list 36F8C660, share 1/1, type adjacency prefix, for IPv4

attached to GigabitEthernet1/0/1, adjacency IP adj out of GigabitEthernet1/0/1, addr 172.25.33.1

37BCEF40

output chain: IP adj out of GigabitEthernet1/0/1, addr 172.25.33.1 37BCEF40

Switch1#


Unicast L3 Forwarding Show Commands

82

Switch1# show adjacency gi1/0/1 172.25.33.1 detail

Protocol Interface Address

IP GigabitEthernet1/0/1 172.25.33.1(11)

0 packets, 0 bytes

epoch 0

sourced in sev-epoch 1

Encap length 14

0011922DA3416400F124DFE40800

L2 destination address byte offset 0

L2 destination address byte length 6

Link-type after encap: ip

ARP


Platform Show Commands

83

Switch1#show platform ip adj

IP Adj entries

dest if_name dst_mac si_hdl ri_hdl pd_flags token

---- ------- ------- ------ ------ -------- -----

172.25.33.55 Gi1/0/1 0050.56ba.0053 0x4f0580f0 0x4f051f20 0x51 0xdf

172.25.33.1 Gi1/0/1 0011.922d.a341 0x4f051210 0x4f058ff8 0x51 0xe6

Switch1#show platform abstraction print-resource-handle 0x4f058ff8 0

Handle:0x4f058ff8 Res-Type:ASIC_RSC_RI Res-Switch-Num:255 Asic-Num:255 Feature-

ID:AL_FID_L3_UNICAST_IPV4 Lkp-ftr-id:LKP_FEAT_INVALID ref_count:1

Hardware Indices/Handles:priv_ri/priv_si Handle:0x4f051210 index0:0x13 mtu_index/l3u_ri_index0:0x0

Features sharing this resource:55 (1)

Cookie length 56

0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x3 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x11 0x92

0x2d 0xa3 0x41 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0

0x0 0x0 0x0 0x0 0x0 0x0 0x0

==============================================================

Rewrite Index Handle

Destination MAC Address

Multiple Features can share RI


Wireless

Unicast Forwarding

High Availability


Stacking

CPU Punt Path

Memory and CPU

Licensing




QoS


Summary



85

Multicast display and configuration commands on the 3850

Platform show commands for IGMP Snooping


Configuration & Display

IP igmp snooping is enabled by default.

IPv6 mld snooping is enabled using‘ipv6 mld snooping’ command

No platform configuration commands.

Some of the IOS configuration and display commands are listed below.

Refer IOS Command Manual for a complete list of commands.

Configuration Commands:

ip igmp snooping ip igmp snooping querier ip igmp snooping vlan <num> mrouter interface <interface-name> ip igmp snooping vlan <num> static <x.x.x.x> interface <interface-name> ipv6 mld snooping ipv6 mld snooping vlan <num> mrouter interface <interface-name> ipv6 mld snooping vlan <num> static <x:x:x::x> interface <interface-name>

Display Commands:

show ip igmp snooping show ip igmp snooping mrouter show ip igmp snoop querier show ipv6 mld snooping address show ipv6 mld snooping mrouter show ipv6 mld snooping querier

86


Platform Display

Switch#show platform ip igmp snooping ? <1-4094> VLAN ID groups Display all IGMP multicast group information retry retry information switch Switch number vlan vlan information | Output modifiers <cr>

Display platform information for igmp snooping group entry Switch#show platform ip igmp snoop 1 group 226.1.1.1 L2 Multicast Group Information ============================== Vlan Id: 1 Group: 226.1.1.1 Member ports: Gi1/0/6 (36906207297929242) CAPWAP ports: Flags: Host Type Flags: 0 Failure Flags : 0 DI handler: 0x61c1b248 REP RI handler: 0x61c1b400 SI handler: 0x61c1b730 TCAM handler: 0x61c1b7d8 Group Key: 00000001E2010101000000000000000000000000

Receiver Ports for Group,

interface-name(interface-id)

If Entry is installed successfully in

asic, Failure Flags will be zero

Resource Handles for entry in ASIC, If it is

non-zero, Resource is allocated successfully

87


QoS


Wireless

Unicast Forwarding

High Availability

Stacking

CPU Punt Path

Memory and CPU

Licensing





Summary



89

Wireless Converged Access Components and Architecture

State Transitions a client goes through

Troubleshooting a wireless roam


• Can act as a Mobility Agent (MA) for terminating CAPWAP tunnels for locally connected APs …

• as well as a Mobility Controller (MC) for other Mobility Agent (MA) switches, in small deployments - MA/MC functionality works on a Stack of Catalyst 3850 Switches - MA/MC functionality runs on Stack Master - Stack Standby synchronizes some information (useful for intra-stack HA)

Best-in-Class Wired Switch – with Integrated

Wireless Mobility functionality

Converged Access –

Physical Entities – Catalyst 3850 Switch Stack

MC

MA

90



Traffic Flow and Roaming – Branch, L2 / L3 Roam (within SPG)

SPG

MC MA MA

1142-1

10.101.1.109

10.101.3.109

MA

PoA

PoP

L09-3850s-3# show wireless client summary

Number of Local Clients : 1

MAC Address AP Name WLAN State Protocol

--------------------------------------------------------------------------------

001e.65b7.7d1a L09-AP1142-1 2 UP 11n(5)

L09-3850s-3# show wcdb database all

Total Number of Wireless Clients = 1

Local Clients = 1

Mac Address VlanId IP Address Src If Auth Mob

-------------- ------ --------------- ------------------ -------- -------

001e.65b7.7d1a 2003 10.101.203.1 0x009350C0000000E4 RUN LOCAL

L09-3850s-3# show wireless client mac 001e.65b7.7d1a detail

Client MAC Address : 001e.65b7.7d1a

Mobility State : Local

Client States:

L2_AUTH

AUTHENTICATING

LEARN_IP

RUN

91




SPG

1142-1

10.101.3.109

MC MA MA

10.101.1.109

MA

PoA

PoP

Ca5

L09-3850s-3# show mac address dynamic | inc Ca

2003 001e.65b7.7d1a DYNAMIC Ca5

L09-3850s-3# show capwap summary

Name APName Type PhyPortIf Mode McastIf

------ -------------------------------- ---- --------- --------- -------

Ca1 - mob - unicast -

Ca2 - mob - unicast -

Ca5 L09-AP1142-1 data Gi1/0/7 multicast Ca4

Name SrcIP SrcPort DestIP DstPort DtlsEn MTU

------ --------------- ------- --------------- ------- ------ -----

Ca1 10.101.3.109 16667 10.101.1.109 16667 No 1464

Ca2 10.101.3.109 16667 10.101.2.109 16667 No 1464

Ca5 10.101.3.109 5247 10.101.3.98 31901 No 1449

L09-3850s-3# show ip dhcp snooping binding

MacAddress IpAddress Lease(sec) Type VLAN Interface

------------------ --------------- ---------- ------------- ---- --------------------

00:1E:65:b7:7d:1a 10.101.203.1 10617 dhcp-snooping 2003 Capwap5

92




SPG

1142-1

Ca5

3600-1

L09-3850-1# show wireless client summary


--------------------------------------------------------------------------------

001e.65b7.7d1a 3600-1 2 UP 11n(5)

L09-3850-1# show wcdb database all

Total Number of Wireless Clients = 1

Foreign Clients = 1

Mac Address VlanId IP Address Src If Auth Mob

-------------- ------ --------------- ------------------ -------- -------

001e.65b7.7d1a 2001 10.101.203.1 0x00C55A40000000A6 RUN FOREIGN

10.101.3.109

SPG

MC MA MA

10.101.1.109

MA

PoP

PoA

93


Wireless



Unicast Forwarding

High Availability

Stacking

CPU Punt Path

Memory and CPU

Licensing




QoS

Summary



95

QoS Features available on the 3850

MQC Configuration commands

Platform QoS commands


Modular QoS based CLI (MQC)

– Alignment with 4500E series (Sup6, Sup7)

– Class-based Queuing, Policing, Shaping, Marking

More Queues

– Up to 2P6Q3T queuing capabilities

– Standard 3750 provides 1P3Q3T

– Not limited to 2 queue-sets

– Flexible MQC Provisioning abstracts queuing hardware

Granular QoS control at the wireless edge

Tunnel termination allows customers to provide QoS treatment per SSIDs, per-Clients and common treatment of wired and wireless traffic throughout the network

Enhanced Bandwidth Management

Approximate Fair Drop (AFD) Bandwidth Management ensures fairness at Client, SSID and Radio levels for NRT traffic

Wireless Specific Interface Control

Policing capabilities Per-SSID, Per-Client upstream and downstream

AAA support for dynamic Client based QoS and Security policies

Per SSID Bandwidth Management

Wired Wireless

QoS – What’s New with Converged Access Policy-map PER-PORT-POLICING

Class VOIP

set dscp ef

police 128000 conform-action transmit

exceed-action drop

Class VIDEO

set dscp CS4


exceed-action drop

Class SIGNALING

set dscp cs3


exceed-action drop

Class TRANSACTIONAL-DATA

set dscp af21

Class class-default

set dscp default


Platform QoS CLI

97

Switch# show platform qos policy target GigabitEthernet 1/0/48

Input policy :

--------------

Not attached

Output policy :

--------------

POLICY: defportangn Num Classes:1 PLC Map Targets:0 Queue LBL Targets uplink:0 downlink:0

PMAP:0x6345d778 NextPMAP:0x585d5518 PrevPMAP:0x57b02b98

UP Mask: 0, Lookup Type:0

COS Mask: 0, dscp mask:0

Filter flags: 0, Action Flags:0x14, num_classmaps 1 policy_type: MARKING/POLICING

nfl_req_pending_cnt:0 pmap_qsize:0

CLASS: class-default

CMAP:0x124b42a0 Next:(nil) Prev:(nil)

Masks:- UP:0, CoS: 0, Dscp:0

Filter flags 0

Not Supported

Negate: NO Next:(nil) . . . .

Switch# sh platform qos policy hw_state target GigabitEthernet 1/0/48

Input policy : Not attached

Output policy : defportangn

H/W programming State: INSTALLED IN HW


Platform Wireless QoS CLI

98

Switch# sh wireless client username naoshad


--------------------------------------------------------------

2477.038f.24c4 AP-3600-1 4 UP 11n(5)

Switch# show platform qos wireless afd client mac 2477.038f.24c4

IF Type:CLIENT

ASIC: 0

Index: 2320

Afd Max Rate: 4194300


QoS

Summary

Wireless



Unicast Forwarding

High Availability

Stacking

CPU Punt Path

Memory and CPU

Licensing





Additional Troubleshooting Commands

100

Topic Command

Platform specific Feature information show tech-support platform <feature> (eg.

wireless, acl, fnf, etc.)

Trace Buffers for non-IOSd processes show mgmt-infra trace messages <component>

(eg. fed-punject-detail, stack-mgr-events, etc.)

Generate Live Core of a Process (internal

command)

resource process dump <process id obtained from

show process> [ switch <switch number> ]

Generate System Report (internal command) resource create_system_report

Identify memory leaks show memory debug leaks detailed process

<process name> summary


Core Dumps and System Reports

101

• System generates a fullcore, crashinfo and System Report when a process terminates abnormally

• A System Report is generated each time the switch is rebooted

• System Report contains a dump of all the trace buffers in the system

• When filing a TAC case, please attach the fullcore, crashinfo and System Report files (whatever is applicable) from the crashinfo: filesystem


Wireless



Unicast Forwarding

High Availability

Stacking

CPU Punt Path

Memory and CPU

Licensing




QoS

Summary


Summary

• Provided a High Level Architectural overview of features on the 3850

• Basic Troubleshooting functionality available on the 3850

• Do you have a better understanding of:

• 3850 as a platform

• Key differences between 3850 and 3750X

• Basic troubleshooting on the 3850

• Would you like to see:

• More/Less of any particular topic

• More topics

• Longer session

103


Maximize your Cisco Live experience with your

free Cisco Live 365 account. Download session

PDFs, view sessions on-demand and participate in

live activities throughout the year. Click the Enter

Cisco Live 365 button in your Cisco Live portal to

log in.

Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily.

Receive 20 Cisco Daily Challenge points for each session evaluation you complete.

Complete your session evaluation online now through either the mobile app or internet kiosk stations.

104

troubleshooting cisco catalyst 3850...

Documents