cisco support community expert series webcast · tuesday, march 17, 2015 demystifying unified...
TRANSCRIPT
Niles Pyelshak
Customer Support Engineer
March 17, 2015
Demystifying Unified Computing System (UCS) Interfaces for troubleshooting
Cisco Support Community
Expert Series Webcast
Upcoming Expert Series Webcast
April 7th, 2015
SourceFire IDS mode integration on ISR G2 and ISR 4K routers will be reviewed in details during the webcast along with a live demo.
SourceFire integration on the ISR G2
and ISR 4K routers with Kureli Sankar
http://bit.ly/webcast-april7
Finesse for UCCX, next generation agent desktop
and transitioning from CAD. Finesse general
capabilities, and how it fits into the UCCX
environment with Abhiram Kramadhati, Cisco
Technical Solutions Manager
Now through March 27th
Ask the Expert Events – Active
Join the discussion for these Ask The Expert Events:
http://bit.ly/knowledgesharing-mar17
Cisco Webex Meeting Server (CWMS):
Install/Configuration and Troubleshooting with Cisco
Experts: Amit Sing, Hari Haran S M, and Jyothi V.
https://supportforums.cisco.com/expert-corner/top-contributors
Participate in Live
Interactive
Technical Events
and much more
http://bit.ly/1jlI93B
Become an Event Top Contributor
Rate Content Now your ratings on documents, videos, and blogs count give points to the authors!!!
So, when you contribute and receive ratings you now get the points in your profile.
Help us to recognize the quality content in the community and make your searches easier. Rate content in the community.
https://supportforums.cisco.com/blog/154746
Encourage and acknowledge people who generously share
their time and expertise
Cisco Support Community Expert Series Webcast
• Today’s featured expert is Cisco Support Engineer, TAC Niles Pyelshak
• Ask your questions now in the Q&A window
Niles Pyelshak Customer Support Engineer
Cisco TAC, Server Virtualization
Tuesday, March 17, 2015
Demystifying Unified Computing
System (UCS) Interfaces for
troubleshooting CCIE#44608
Expert Series Webcast: Demystifying Unified Computing System (UCS) Interfaces for troubleshooting
Technical Expert Question Manager
Robert Burns
CCIE #37856
Now through March 25th
Ask the Expert Event following the Webcast
Join the discussion for these Ask The Expert Events:
http://bit.ly/knowledgesharing-mar17
Niles will be continuing the discussion in an Ask the Expert
event. So if you have more questions, please visit the
Knowledge Center on the Cisco Support Community
https://supportforums.cisco.com/discussion/12451761/ ask-expert-demystifying-unified-computing-system-ucs- interfaces-troubleshooting
If you would like a copy of the presentation slides, click the PDF file link in the chat box on the right or go to:
bit.ly/docslides-mar17
Thank You For Joining Us Today!
Submit Your Questions Now! Use the Q & A panel to submit your questions
and the panel of experts will respond.
Please take a moment to
complete the survey at
the end of the webcast
Niles Pyelshak
Customer Support Engineer
March 17, 2015
Cisco Support Community Deep Dive Expert Series Webcast
Demystifying Unified Computing System (UCS) Interfaces for troubleshooting
Polling Question 1
Do you have Unified Computing System(UCS) in your Data Center?
a. I have UCS-B series but not C-Series
b. I have UCS-C Series but not B-Series
c. I don’t have UCS and have never used it.
• What’s in a UCS Domain?
• End Host/Switch Mode
• UCS Unified Ports
• Chassis Connectivity
• Server Connectivity
• Traffic Flow
• UCS Interfaces
• HIF to NIF Pinning
• Fabric Failover
• Basic Network Troubleshooting
• Following the Path of a Packet
High Level Overview of UCS Life of a Packet and Troubleshooting
Fabric Interconnect (FI)
• 10 GbE unified fabric switch • Connects up to 160 servers
UCS Fabric Extender (FEX)
• 10 GbE unified remote linecard • Flexible bandwidth connectivity
Compute (Blade and Rack)
• x86 industry standard • Patented extended memory
UCS Virtual Interface Card (VIC)
• CNA adapters • Flexible server I/O
Unified Management
System Components: High-level Overview
UCS Fabric Interconnect
UCS Fabric Interconnect – UCS 6248
• 1RU
• 32 unified base ports and 1 expansion slot • Line rate – 960Gbps
UCS Fabric Interconnect – UCS 6296
• 2RU
• 48 unified base ports and 3 expansion slots • Line rate – 1920 Gbps
UCS Fabric Extender
UCS IO Module (IOM) – 2204 or 2208
• 4 or 8 10GbE fabric links (to Fabric Interconnect)
• 16 or 32 10GbE server links (to servers)
Nexus 2232PP or Nexus 2232TM
• 8 10GbE fabric links (to Fabric Interconnect)
• 32 10GbE server links (to servers)
UCS VIC Adapters
UCS VIC1240 plus Pass-through (PT) Expansion Card - Blades
• VIC1240: Up to 4 x 10 GbE
• PT: Expands VIC1240 up to 8 x 10GbE
• Up to 256 vPCIe
UCS VIC 1280 - Blades
• Up to 8 x 10GE ports • Up to 256 vPCIe
UCS VIC 1225 - Racks
• Up to 2 x 10GE ports • Up to 256 vPCIe
UCS Fabric Portfolio
2 3 4
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
CISCO UCS 6296UP
STAT
ID
CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
STAT
ID
Configuring Unified Ports
• Configured on a per FI basis
• Slider based configuration
• Reboot is required for the new port personality to take into affect
• Recommendation is to configure GEM card, therefore GEM is only needed to be rebooted
Unified Port Screen
Fabric Forwarding Mode of Operations
• End-host mode (EHM): Default mode
• No spanning-tree protocol (STP); no blocked ports
• Admin differentiates between server and network ports
• Using dynamic (or static) server to uplink pinning
• No MAC address learning except on the server ports; no unknown unicast flooding
• Fabric failover (FF) for Ethernet vNICs (not available in switch mode)
• Switch mode: User configurable
• Fabric Interconnects behave like regular ethernet switches
• STP parameters are lock
Fabric Forwarding Mode of Operations Mode of Operations
Putting the Components Together
6200
Fabric A
6200
Fabric B
B200 B250
CNA
I OM A CNA CNA
I OM
B
I OM
A
I OM
B
ETH 1 ETH 2
MGMT MGMT
Chassis 1 Chassis 20
Fabric Switch
Fabric Extenders
Uplink Ports
Compute Blades
Half / Full width
OOB Mgmt
Server Ports
Virtualized Adapters
Cluster
SAN A SAN B
• Configured on a per FI basis
• Slider based configuration
• Reboot is required for the new port personality to take into affect
• Recommendation is to configure GEM card, therefore GEM is only needed to be rebooted
Unified Port Screen
• Configured on a per FI basis
• Slider based configuration
• Reboot is required for the new port personality to take into affect
• Recommendation is to configure GEM card, therefore GEM is only needed to be rebooted
Unified Port Screen
Putting the Components Together
6200
Fabric A
6200
Fabric B
B200 B250
CNA
I OM A CNA CNA
I OM
B
I OM
A
I OM
B
ETH 1 ETH 2
MGMT MGMT
Chassis 1 Chassis 20
Fabric Switch
Fabric Extenders
Uplink Ports
Compute Blades
Half / Full width
OOB Mgmt
Server Ports
Virtualized Adapters
Cluster
SAN A SAN B
UCS Fabric Topologies Chassis Bandwidth Options
24
2x 4 Link 80 Gbps per Chassis
2x 8 Links 160 Gbps per Chassis
2x 2 Link 40 Gbps per Chassis
2x 1 Link 20 Gbps per Chassis
2208XP only
• A IOM (sometimes called ‘Fabric Extender’) provides
• A 1GE switch used for internal management (1GE per slot)
• A number of 10G-KR sever facing links (HIF)
• A number of Fabric links (NIF)
• NIC cards on the servers use those HIF ports for external connectivity
• Each IOM provides a separate dedicated IO channel for internal management connectivity
• There is no local switching on IOMs – traffic is always switched by the FIs
What are those IOMs?
Let’s go back in time a bit
26
Chassis Management
Controller
FLASH
EEPROM
DRAM
Control
IO
Chassis Signals
Switch
4 Physical ports to Fabric
Interconnect
8 internal backplane ports to blades
ASIC with some
funny name
Up to 20Gbps
per slot
Up to
80Gbps per
chassis
2104XP
Gen 1 IOM
So this is what we had with the 2104XP
27
2208
2208
220x-XP Architecture
Chassis Management
Controller
FLASH
EEPROM
DRAM
Control
IO
Chassis
Signals
Switch
Woodside ASIC
Internal backplane ports to blades
No Local Switching – ever!
Traffic goes up to FI
Fabric Ports to FI
2204
2204
Feature 2204-XP 2208-XP
ASIC Woodside Woodside
Fabric Ports (NIF)
4 8
Host Ports (HIF)
16 32
CoS 8 8
Latency ~ 500ns ~ 500ns
• These interfaces (show int brief – NXOS shell) are backplane traces
• Eth x/y/z nomenclature where
• x = chassis number
• y = is always 1
• z = host interface port number
Blade Northbound Ports
2
9
Server Connectivity
`
VIC
M81K
R
Palo ASIC Port 0 Port 1
21
04
- B
21
04
- A
B200 M
1/M
2
B 2
30M
1/M
2
x16 Gen1
IOM 2104 with M81KR in M1/M2 Blades
UCS 5100 backplane has 4 passive KR lanes to each half width server slot
BW to half width slot
= Dual 10Gb
IOM 2104 with M81KR in M1/M2 Blades
`
VIC
1280
Sereno ASIC Port 0 Port 1
22
08
- B
22
08
- A
B200 M
1/M
2
B 2
30M
1/M
2
x16 Gen1
IOM 2208 with VIC 1280 in M2 Blades
IOM 2208 has 4 KR lanes to each server slot
BW to half width slot
= Dual 4x10Gb
VIC 1280 has dual 4x10GB ports
IOM 2208 with VIC 1280 in M2 Blades
Traffic Flow
Inter-Fabric Traffic Example (1)
VM1
VM2
VNIC 0
6100 A 6100 B
VNIC 1
L2 Switching
ESX HOST
EHM EHM
Inter-Fabric Traffic Example (2)
VM
1
VM
2
VNIC 0
6100 A 6100 B
VNIC 1
L2 Switching
ESX HOST
EHM EHM
UCS Interfaces
UCS Interfaces
Uplink(border) port
Server(fabric) port
Network(NIF) port
Host(HIF) port
VIF VIF VIF
UCS Fabric
Interconnects
IO Modules
VIC Adapter
Virtual Circuit
Polling Question 1
What are the two types of VIFS?
a) DCE
b) VFC
c) Veth
d) HIF
Virtual Interfaces (VN-TAG)
• Virtual interfaces (VIFs) help distinguish between FC and Eth interfaces
• They also identify the origin server
• VIFs are instantiated on the FI and correspond to frame-level tags assigned to blade mezz cards
• A 6-byte tag (VN-Tag) is preprended by Palo and Menlo as traffic leaves the server to identify the interface
• VN-Tag associates frames to a VIF
• VIFs are ‘spawned off’ the server’s EthX/Y/Z interfaces
VN-Tag: Instantiation of Virtual Interfaces
4
0
Adapter
Switch
10GE
A
Eth 1/1
IOM A
6200-A
Physical Cable
Virtual Cable
(VN-Tag)
Abstracting the Logical Architecture
Blade
10GE
A
vEth 1
IOM A
6200-A
vFC
1
Service Profile
(Server)
Cable
vNIC
1
vEth
1
6200-A
vHBA
1
vFC
1
(Server)
Dynamic, Rapid
Provisioning
State abstraction
Location
Independence
Blade or Rack
Logical Physical
vHBA
1
vNIC
1
42
VN-Tag at the Adapter (Mezz Card) Level
• Ethernet and FC are muxed on the same physical links concept of virtual interfaces (vifs) to split Eth and FC
• Two types of VIFs: veth and vfc
• Veth for Ethernet and FCoE; vfc for FC traffic
• Each EthX/Y/Z or Po interface typically has multiple vifs attached to it to carry traffic to and from a server
• To find all vifs associated with a EthX/Y/Z or Po interface, do this:
VIFs
FCoE: a vfc bound to a veth
Another Way to Find VIFs:
4
6
HIF to NIF Pinning
IO Module HIF to NIF Pinning 2208XP – 1 Link
1-4
5-8
9-12
13-16
17-20
21-24
25-28
29-32
FEX Fabric Interconnect
Slot 1
Slot 2
Slot 3
Slot 4
Slot 5
Slot 6
Slot 7
Slot 8
One link
IO Module HIF to NIF Pinning 2208XP – 2 Link
FEX Fabric Interconnect
29-32
1-4
5-8
9-12
13-16
17-20
21-24
25-28
Slot 1
Slot 2
Slot 3
Slot 4
Slot 5
Slot 6
Slot 7
Slot 8
Two links
IO Module HIF to NIF Pinning 2208XP – 4 Link
FEX Fabric
Interconnect
29-32
1-4
5-8
9-12
13-16
17-20
21-24
25-28
Slot 1
Slot 2
Slot 3
Slot 4
Slot 5
Slot 6
Slot 7
Slot 8
Four links
IO Module HIF to NIF Pinning 2208XP – 8 Link
FEX
29-32
1-4
5-8
9-12
13-16
17-20
21-24
25-28
Fabric
Interconnect
Slot 1
Slot 2
Slot 3
Slot 4
Slot 5
Slot 6
Slot 7
Slot 8
Eight links
• What happens in a 4-link topology when you loose 1 link?
• Server interfaces pinned to that link go down*
• The remaining 3 links still pass traffic for the other blade servers
• To recover the failed servers’ vNICs, re-acknowledged of the chassis is required
• After a re-ack UCS falls back to 2 links with regards to blade to fabric port mapping
• That’s because the link count must be a power of 2!
IOM and Failover
* unless you enabled Fabric Failover
HIF to NIF Pinning Port Channel Modes
Port-channel pinning
54
Fabric Port Channeling
Fabric Failover
Polling Question 2
What is the preferred failover?
a) Hardware (Fabric) Failover
b) Software Failover (OS, Hypervisor)
• Fabric Failover is only applicable in EHM.
• NIC teaming software required to provide failover in Switch mode.
Fabric Failover
Recommendation:
End Host Mode
vNIC 1
10GE
10GE
vEth 1
OS / Hypervisor / VM
vEth 1
IOM IOM
PHY Adapter Cisco VIC – M81KR
Menlo – M71KR VIRT Adapter
6100-A 6100-B L1 L2
L1 L2
Physical Cable
Virtual
Cable
Fabric Failover
Fabric Failover
Fabric Failover
Following the Path of a Packet
• UCSM top level : show service-profile circuit server <chassis#>/<slot#>
• 2.0(1) release has an improved output
Following the Path of a Packet
Following the Path of a Packet
VIF
Virtual Circuit
(VIF 1236)
UCS Fabric
Interconnects
IO Modules
VIC Adapter
Pins out uplink Eth 1/9
Following the Path of a Packet
Following the Path of a Packet
Following the Path of a Packet
VIF
Virtual Circuit
(VIF 1236)
UCS Fabric
Interconnects
IO Modules
VIC Adapter
Interface 1/1/21
VNTAG 2
Pins out uplink Eth 1/9
Following the Path of a Packet
Following the Path of a Packet
VIF
Virtual Circuit
(VIF 1236)
UCS Fabric
Interconnects
IO Modules
VIC Adapter
Interface 1/1/21
(Pinned to 1/2)
VNTAG 2
Pins out uplink Eth 1/9
Server Port 1/2
Following the Path of a Packet
Following the Path of a Packet
VIF
Virtual Circuit
(VIF 1236)
UCS Fabric
Interconnects
IO Modules
VIC Adapter
VNTAG 2
Pins out uplink Eth 1/9
Designated Receiver Eth 1/9
Interface 1/1/21
(Pinned to 1/2)
Server Port 1/2
C-Series Integration into UCSM
CIMC
GE LOM PCIe Adapter
CPU Mem
OS or Hypervisor
C260M2, C460M2
C220M3, C240M3
C22M3, C24M3 all at CiCM
1.4(6) or higher
C200/C210/C250 NO
Mix of B & C Series is
supported (no B Series
required)
Nexus 2232 Nexus 2232
Mgmt Traffic
Data Traffic
C-Series UCSM Integration Single Wire Management with VIC1225 and UCSM 2.1
UCSM 2.2: Direct Connect with VIC1225 (no FEX)
75
CIMC
GE LOM
CPU Mem
OS or Hypervisor
C260M2, C460M2
C220M3, C240M3
C22M3, C24M3 all at CiCM
1.4(6) or higher
C200/C210/C250 NO
PCIe Adapter
Wrapping up
Polling Question 3
Are B-series servers required to integrate rack mount servers into UCSM?
a) Yes
b) No
• UCS is first and foremost a server, not a switch
• Gen 1 vs Gen 2 components
• 2208 and VIC 1280 allow port-channels
• VIC 1280 with 2208XP for maximum bandwidth
• End-host mode forwarding rules: dynamic pinning
Recap
• Preferred mode of operation should always be end-host mode
• Very much plug and play, scalability, L2 multipathing, fabric failover
• Switch mode: spanning-tree, practically no user configuration possible
• VIF, VN-TAG, HIF, NIFs, VETH, VFC, Border Interfaces
• Operational consistency: C-series integration
Fault Codes: http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ts/faults/reference/ErrMess/UCS_SEMs.html
Submit Your Questions Now! Use the Q & A panel to submit your questions and our expert will respond
Collaborate within our Social Media
Facebook- http://bit.ly/csc-facebook
Twitter- http://bit.ly/csc-twitter
You Tube http://bit.ly/csc-youtube
Google+ http://bit.ly/csc-googleplus
LinkedIn http://bit.ly/csc-linked-in
Instgram http://bit.ly/csc-instagram
Newsletter Subscription http://bit.ly/csc-newsletter
Learn About Upcoming Events
Cisco has support communities in other languages!
Spanish https://supportforums.cisco.com/community/spanish
Portuguese https://supportforums.cisco.com/community/portuguese
Japanese https://supportforums.cisco.com/community/csc-japan
Russian https://supportforums.cisco.com/community/russian
Chinese
http://www.csc-china.com.cn
If you speak Spanish, Portuguese, Japanese, Russian or Chinese we invite you to participate and collaborate in your language
More IT Training Videos and Technical Seminars on the Cisco Learning Network
View Upcoming Sessions Schedule
https://cisco.com/go/techseminars
Please take a moment to complete the survey
Thank you for Your Time!
Basic Network Troubleshooting
Check Vmware, Microsoft or UCS
configuration(VLANs, native etc..)
Do you see the VMs
MAC addresses?
Basic Network Troubleshooting: Through Isolation
Isolate two VMs(on
different hosts) or
two bare metal
installs on same the VLAN to the same
FI.
YES
NO
Check local firewall settings on VMs. Arp could be failing. Could
be duplicate IP or MAC issue.
Can you ping
between both VMs?
Isolate VMs to
other fabric and run 3 previous steps
YES
NO
YES
Check
configuration on
switch. Check
for VLANs being trunked over
interfaces, vPC,
STP. UCS disjoint setup.
NO Can you ping the VM
from the upstream L3 device?
Check arp table.
See if you have
duplicate IP or MAC. Check
broadcast
receiver UCS.
NO
YES
Pin VMs out different FI’s. Do you see the
VMs MAC addresses
on upstream switch?
YES
NO Upstream routing
issue.
Can you ping inter-VLAN
between two UCS VMs or
bare metal install?
Basic Network Troubleshooting: Identifying Duplicate IP Addresses
• Disable interface that is
intermittently dropping pings.
• Ping from a VM or L3 device that is in the same VLAN.
• Check to see if MAC address resolved during
ARP. If the MAC did resolve compare/contrast.
• You can disable adapter or shutdown the host. Use MAC address table to see if MAC address moves to other vethernets or interfaces on the upstream switch
• Follow the vethernet or ethernet interface
Basic Network Troubleshooting: Identifying Duplicate MAC Addresses
• Connect nxos : show platform fwm info mac <mac> <vlanID>
• A “Mac history” section was added in 2.0(1) release
• The “old_if_index” is helpful in cases were MAC is learned from different places same time (for example, duplication issue)
Basic Network Troubleshooting: MAC History
VN-Tag at the Adapter (Mezz Card) Level
CIMC
GE LOM PCIe Adapter
CPU Mem
OS or Hypervisor C200M2, C210M2,
C220M3, C240M3,
C250M2, C260M2 or
C460M2
Mix of B & C Series is
supported (no B Series
required)
Nexus 2232 Nexus 2232
2 LOM ports
exclusive CIMC connectivity
Adapter support:
Emulex CNA Qlogic CNA
Intel 10g NIC Broadcom 10g NIC
Cisco VIC
Mgmt Traffic
Data Traffic
C-Series UCSM Integration Before UCSM 2.1 or without VIC1225
SPAN
96
Let’s go back in time once more
The traces are there; we were just not using them …
Following the Path of a Packet
• Connect nxos : show platform software enm internal info vlandb id <vlanID>
UCS 6248 Hardware Diagram
98
10 Gig
12 Gig
12 Gig
10 Gig
Carmel 1
Unified Crossbar Fabric Sunnyvale
Carmel 2 Carmel 3
Carmel 4 Carmel 5 Carmel 6
Ca
rme
l
CP
U
Intel
Jasper Forest
PEX 8525
4 port PCIE
Sw itch
South
Bridge
12 Gig
12 Gig
Carmel 1
Carmel cpu
Sunny vale
Carmel 2
Mgmt
Console
Xcon1
Xcon2
DD
R3
x2
PCIE Dual Gig
PCIe x4 PCIe x4 PCIe x4
PCIe x8
PCIE Dual Gig
PCIE Dual Gig
0 1 0 1 N/C
0
1
Carmel 6
Memory
Flash
NVRAM
Serial
• No slot based pinning
• No invalid link count for NIF ports (no “power of 2” rule)
Port-Channel Pinning
9
9
Pinned
to Po
2200-IOM VIC1200
adaptor with
DCE links in
Port-Channel
Gen-1 adaptor
with single 10G
link
8 links, Discrète (default)
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
F
E X
8 links, Port-channel
F E
X
Fabric Interconnect
Fabric Interconnect
• Statically pinned to Individual fabric links, no dynamic re-pinning
• Failure of the Fabric port will bring the HIF down triggering a failover
• Statically pinned to logical link (port-channel)
• Failure of the Fabric port will not bring the HIF down no failover take place.
• Traffic re-hashed to available links
HIF to NIF Pinning Link Failure Connectivity from IOM to FI and Failover Behavior
Fabric Failover
Adapter
Switch
10GE
A
Eth 1/1
IOM A
6200-A
Physical Cable
Virtual Cable
(VN-Tag)
Abstracting the Logical Architecture
102
Blade
10GE
A
vEth 1
IOM A
6200-A
vFC
1
Service Profile
(Server)
Cable
vNIC
1
vEth
1
6200-A
vHBA
1
vFC
1
(Server)
Dynamic, Rapid
Provisioning
State abstraction
Location
Independence
Blade or Rack
Logical Physical
vHBA
1
vNIC
1