© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential (please do not distribute) 1
Unified Fabricaka FCOE
Dave Gibson Senior Systems Engineer Cisco Systems
2© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Legal Disclaimer
Many of the products and features described herein remain in varying stages of development and will be offered on a when-and-if-available basis. This roadmap is subject to change at the sole discretion of Cisco, and Cisco will have no liability for delay in the delivery or failure to deliver any of the products or features set forth in this document.
3© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Agenda
The Evolution of the Data Center
Introduction to FCoE
Standards Defined
Nexus and the Unified Fabric
Nexus 5000
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential (please do not distribute) 4
The Evolution of theData Center
5© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Data Center Access Layer Trends
Multi-Core CPU architectures allowing bigger and multiple workloads on the same machine
Server virtualization driving the need for more I/O bandwidth per server
Growing need for network storage driving the demand for higher network bandwidth to the server
Increasing adoption of Blades in data centers.
10G LOM on server Motherboard
6© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Next-Gen Switch Design Goals
•Enable Virtualization•Address increase in server processing power
•End-to-end data center architecture•Operational consistency across platforms
•Consolidate LAN & SAN infrasctucture•Standards based solution•Reduce total cost of ownership
•Build with superior performance in mind•Support low latency applications (e.g. HPC, clustered app’s)
•Scale to 40G and 100G in future•Increase feature velocity
7© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Cisco Nexus Family
Complete data center class switching portfolio
Consistent data center operating system across all platforms
Infrastructure scalability, transport flexibility and operational manageability
1K1K1K1KCisco Nexus 1000V
Cisco Nexus 1000V
2008
2008
x86x86
NX-OS Data Center Operating System
Data Center Network Manager
Nexus 2000
(Fabric Extender)
Nexus 2000
(Fabric Extender)
Nexus 5000
(Fixed Config Switch)
Nexus 5000
(Fixed Config Switch)
Nexus 7000
(Modular Switch Platform)
Nexus 7000
(Modular Switch Platform)
Nexus 1000V
(Virtual Switch)
Nexus 1000V
(Virtual Switch) Nexus 4000
(Blade Switch)
Nexus 4000
(Blade Switch)
9© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Parallel LAN/SAN Infrastructure
Inefficient use of Network Infrastructure
5+ connections per server – higher adapter and cabling costs
• Adds downstream port costs; cap-ex and op-ex
• Each connection adds additional points of failure in the fabric
Multiple switching modules in Blade Chassis
Longer lead time for server provisioning
Multiple fault domains – complex diagnostics
Management complexity
Before I/O Consolidation
Ethernet FC
LAN SAN BSAN A
Blade Chassis with I/O ModulesBlade Chassis
with I/O ModulesServer with
NICs and HBAsServer with
NICs and HBAs
10© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
I/O Consolidation
Data Center Bridging and FCoE
Ethernet Fibre Channel (FC)
Nexus 5000Nexus 5000
LAN SAN BSAN A
Reduction of server adapters
Simplification of access layer and cabling
Gateway free implementation – fits in installed base of existing LAN and SAN
Lower Total Cost of Ownership
Fewer Cables
Investment Protection (LANs and SANs)
Consistent Operational ModelBlade Chassis with Nexus 4000
Blade Chassis with Nexus 4000
Server with CNAsServer with CNAs
Nexus 5000Nexus 5000
11© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Adapter Evolution:Consolidation Network Adapter
12© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Operating System View
13© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Evolution of 10G Ethernet Physical MediaRole of Transport in Enabling these Technologies!
100Mb 1Gb 10Gb
UTP Cat 5 UTP Cat 5SFP Fiber
10Mb
UTP Cat 3
Mid 1980’s Mid 1990’s Early 2000’s Late 2000’s
X2SFP+ Cu (BER better than 10 )
SFP+ FiberCat 6/7
-18
Technology Cable Distance Power(each side)
Transceiver Latency (link)
SFP+ CUCopper
Twinax 7m ~0.1W ~0.1μs
SFP+ USRUltra short reach
MM OM2MM OM3
10m100m
1W ~0.1μs
SFP+ SRShort reach
MM 62.5 μmMM 50 μm
26-33m66-300m
1W ~0.1μs
SFP+ LRLong range
SMF G.652 10km 0.5W
10GBASE-T Cat6Cat6a/7Cat6a/7
55m100m30m
~8W~8W~4W
2.5μs2.5μs1.5μs
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential (please do not distribute) 14
Introduction to FCoE
15© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
What is Fibre Channel over Ethernet?
From a Fibre Channel standpoint it’sFC connectivity over a new type of cable called… an Ethernet cloud
From an Ethernet standpoints it’sYet another ULP (Upper Layer Protocol) to be transported
FCoE is an extension of Fibre Channelonto a Lossless Ethernet fabric
16© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Unified Fabric OverviewFibre Channel over Ethernet (FCoE)
04/19/23 16
• Mapping of FC Frames over Ethernet
• Enables FC to Run on a Lossless Ethernet Network
• Mapping of FC Frames over Ethernet
• Enables FC to Run on a Lossless Ethernet Network
• Fewer Cables• Both block I/O & Ethernet
traffic co-exist on same cable
• Fewer adapters needed• Overall less power• Interoperates with
existing SAN’s• Management SAN’s
remains constant
• No Gateway
• Fewer Cables• Both block I/O & Ethernet
traffic co-exist on same cable
• Fewer adapters needed• Overall less power• Interoperates with
existing SAN’s• Management SAN’s
remains constant
• No Gateway
FCoEFCoE BenefitsBenefits
Fibre Channel Traffic
Ethernet
17© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
FCoE EnablersE
ther
net
Hea
der
FC
oEH
eade
r
FC
Hea
der
FC Payload
CR
CE
OF
FC
S
Same as a physical FC frame
Control information: version, ordered sets (SOF, EOF)
Normal ethernet frame, ethertype = FCoE
10Gbps Ethernet
Lossless EthernetMatches the lossless behavior guaranteed in FC by B2B credits
Ethernet jumbo frames
18© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Unified I/OFibre Channel over Ethernet (FCoE)
FCoE is managed like FC at initiator, target, and switch level
Same Operational Model Same Operational Model
Same Techniques ofTraffic ManagementSame Techniques ofTraffic Management
Same Managementand Security ModelsSame Managementand Security Models
Easy to UnderstandEasy to Understand
Completely based on the FC model
Same host-to-switch and switch-to-switch behavior as FC
e.g. in order delivery, FSPF load balancing
WWNs, FC-IDs, hard/soft zoning, DNS, RSCN
FCoE isFibre ChannelFCoE isFibre Channel
19© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Ethernet
IP
Network Stack Comparison
TCP
iSCSI
FCIP
FCoE
FCP
FC
IP
TCP
FCPFCP
Ethernet Ethernet
SCSI
FC FC
SCSI SCSI SCSI SCSI
SCSI iSCSI FCIP FCoE FC
PHYSICAL WIRE
Less Overheadthan FCIP, iSCSI
20© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Destination MAC Address
Source MAC Address
(IEEE 802.1Q Tag)
ET = FCoE Ver Reserved
Reserved
Reserved
Reserved SOF
Encapsulated FC Frame (with CRC)
EOF Reserved
FCS
FCoE Frame Format
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential (please do not distribute) 22
FCoE StandardsDefined
23© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
A larger picture
IEEE 802•Evolution of Ethernet (10 GE, 40 GE, 100 GE, copper and fiber)
•Evolution of switching (Priority Flow Control, Enhanced Transmission, Congestion Management, Data Center Bridging eXchange)
INCITS/T11•Evolution of Fibre Channel (FC-BB-5)
•FCoE (Fibre Channel over Ethernet)
IETF•Layer 2 Multi-Path
•TRILL (Transparent Interconnection of Lots of Links)
24© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
DCE versus DCB
DCE is an old Cisco marketing term
Cisco is now using the term DCBThe term IEEE uses
Cisco supports the DCB standard activityBy implementing products that are DCB compliant
CIN-DCBX – Cisco, Intel, Nuova Data Center Bridging Exchange protocol, pre-standard
CEE-DCBX – Converged Enhanced Ethernet Data Center Bridging Exchange protocol, which is standards base
25© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
What’s FC-BB-5
FC-BB-5 covers the majority of the FC features, using Ethernet
From an Ethernet perspective, FC-BB-5 isEthernet control plane referred to as FIP (Fibre Channel over Ethernet Initiation Protocol)
discover and build virtual paths between end points
Ethernet data plane providing FCoE forwarding
including both FC control plane and FC data plane (FCF)
26© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
FC-BB-6
It is an active working group of T11 that will discuss the future of FCoE or FCoE v2.0
It is just started, 18 months to have a standardApproximate target spring 2011
You can track it on http://www.fcoe.com
27© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
The two protocols have:• Two different Ethertypes• Two different frame formats
Protocol Organization
FCoE itself … Is the data plane protocol
It is used to carry most of the FC frames and all the SCSI traffic
FIP (FCoE initiation protocol)
It is the control plane protocol
It is used to discover the FC entities connected to an Ethernet cloud
It is used to login to and logout from the FC fabric
28© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
What’s NOT FC-BB-5
FC-BB-5 doesn’t deal with how lossless is realized in Ethernet
no Priority Flow Control, Bandwidth Management, etc.
FC-BB-5 doesn’t deal with management functions
29© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Feature / Standard Standards Status
IEEE 802.1Qbb
Priority Flow Control (PFC)
Enable multiple traffic types to share a common Ethernet link without interfering with each other
PAR approved
1.0 published
IEEE 802.1Qaz
Bandwidth Management (ETS)
Enable consistent management of QoS at the network level by providing consistent scheduling
PAR approved
1.0 published
Data Center Bridging Exchange Protocol (DCBX)
Management protocol for enhanced Ethernet capabilities
This is part of IEEE 802.1Qaz
IEEE DCB standards statusDCB technologies allow Ethernet to be lossless and to manage bandwidth allocation of SAN and LAN flows
30© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
04/19/23 Nuova Systems Inc. 30
Data Center Ethernet: PFC & Bandwidth Management
CoS based Bandwidth ManagementCoS based Bandwidth Management
• Enables Intelligent sharing of bandwidth between traffic classes control of bandwidth• 802.1Qaz Enhanced Transmission
10 GE Realized Traffic Utilization
3G/s HPC Traffic3G/s
2G/s
3G/sStorage Traffic3G/s
3G/s
LAN Traffic4G/s
5G/s3G/s
t1 t2 t3
Offered Traffic
t1 t2 t3
3G/s 3G/s
3G/s 3G/s 3G/s
2G/s
3G/s 4G/s 6G/s
Priority Flow ControlPriority Flow Control
• Enables lossless behavior for each class of service• PAUSE sent per virtual lane when buffers limit exceeded
Transmit QueuesEthernet Link
Receive Buffers
ZeroZero ZeroZero
OneOne OneOne
TwoTwo TwoTwo
FiveFive FiveFive
FourFour FourFour
SixSix SixSix
SevenSeven SevenSeven
ThreeThree ThreeThreeSTOP PAUSE EightVirtualLanes
31© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
DCBX Overview
Auto-negotiation of capability and configuration
Priority Flow Control capability and associated CoS values
Allows one link peer to push config to other link peer
Link partners can choose supported features and willingness to accept
Discovers FCoE Capabilities
Responsible for Logical Link Up/Down signaling of Ethernet and FC
DCBX negotiation failures will result in: vfc not coming up
Per-priority-pause not enabled on CoS values with PFC configuration
http://download.intel.com/technology/eedc/dcb_cep_spec.pdfhttp://www.ieee802.org/1/files/public/docs2008/
32© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
FCoE control plane
33© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
FIP: FCoE Initialization Protocol
FCoE VLAN discovery
Automatic discovery of FCoE VLANs
Device discovery
ENodes discover VF_Port capable FCF-MACs for VN_Port to VF_Port Virtual Links
VE_Port capable FCF-MACs discover other VE_Port capable FCF-MACs for VE_Port to VE_Port Virtual Links
The protocol verifies the Lossless Ethernet network supports the required Max FCoE Size
Virtual Link instantiation
Builds on the existing Fibre Channel Login process, adding the Negotiation of the MAC address to use
Fabric Provided MAC Address (FPMA), or
Server Provided MAC Address (SPMA)
Virtual Links maintenance
Timer based
34© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Fabric Provided MAC Addresses
MAC address assigned for each FC_ID:
Consistent with the Fibre Channel model
Multiple FC-MAPs may be supported
One per SAN
No table needed for Encapsulation
Multiple MACs may be needed for NPIV
48 bits
FC-MAP(0E-FC-00)
FC-ID7.8.9
24 bits
24 bits
FC-MAP(0E-FC-00)
FC-ID7.8.9
Cisco Nexus 5000 uses FPMA
Server Provided MAC Addresses
Adapter uses burned-in or configured MAC address:
Consistent with the Ethernet model
FCF needs a table to map between MAC addresses and FC_IDs
48 bits
Burned in or Configured
MACAddress
35© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Initial Login Flow ladder
ENode FCoE Switch
VLANDiscovery
FLOGI/FDISC FLOGI/FDISC Accept
FC CommandFC Commandresponses
FIP:FCoEInitialization Protocol
FCOEProtocol
VLANDiscovery
FCFDiscovery
Solicitation FCFDiscoveryAdvertisement
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential (please do not distribute) 44
FCoE data plane
45© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
ENode: Simplified Model
ENode (FCoE Node): a Fibre Channel HBA implemented within an Ethernet NIC aka CNA (Converged Network Adapter)
FCoE LEP : The data forwarding component that handles FC frame encapsulation/decapsulation
FCoE Controller is the functional entity that performs the FIP and instantiates VN_Port/FCoE_LEP pairs.
Enetport
Enetport
FC Node
FCoE_LEP FCoE_LEP
FCoE_Controller FCoE_Controller
46© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
FCoE Switch: Simplified Model
FCF (Fibre Channel Forwarder), the forwarding entity inside an FCoE switch
Ethport
Ethport
Ethport
Ethport
Ethport
Ethport
Ethport
Ethport
Ethernet Bridge
FCport
FCport
FCport
FCport
FCF
FCoE_LEP
FCoE Switch
FCoE_Controller
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential (please do not distribute) 47
FCoE Network Topology
48© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
FCoE: Initial Deployment
SAN A SAN B10GE
Backbone
10GE
4/8 Gbps FC
VF_Ports
VN_Ports
Nexus 5000 (FCF)
49© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
FCoE: Adding Blade Servers
SAN A SAN B
10GEBackbone
10GE
4/8 Gbps FC
VF_Ports
VN_Ports
50© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
FCoE: Adding Native FCoE Storage
SAN B
10GE
4/8 Gbps FC
VF_Ports
VN_Ports
SAN A
VN_Ports
10GEBackbone
51© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
FCoE: Adding VE_ports
10GEBackbone
10GE
4/8 Gbps FC
VF_Ports
VE_Ports
SAN ASAN B
VN_Ports
52© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Nexus Topologies
53© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
The Unified Data Center Architecture
L2
L3
VM
VM
VM
VM
VMVM
VM VM
VM VM
A
Aggregation: Typical L3/L2 boundary. DC aggregation point for uplink and DC services offering key features: VPC, VDC, 10GE density and 1st point of migration to 40GE and 100GE
Access: Classic network layer providing non-blocking paths to servers & IP storage devices through VPC. It leverages Distributed Access Fabric Model (DAF) to centralize config & mgmt and ease horizontal cabling demands related to 1G and 10GE server environments
Virtual Access: A virtual layer of network intelligence offering access layer-like controls to extend traditional visibility, flexibility and mgmt into virtual server environments. Virtual network switches bring access layer switching capabilities to virtual servers without burden of topology control plane protocols. Virtual Adapters provide granular control over virtual and physical server IO resources
L3
PODPOD
Core: L3 boundary to the DC network. Functional point for route summarization, the injection of default routes and termination of segmented virtual transport networks
L2
vL2
Rack 1 Rack 2 Rack 3
VMVMVM
VMVMVM
VMVMVM
VMVMVM
VMVMVM
VMVMVM
VMVMVM
VMVMVM
VMVMVM
VMVMVM
VM VM
VMVM
VMVM
VM VM
VM VM
VMVM
VMVM
VM VM
VM VM
VMVM
B
Rack 1 Rack x
NEXUS 2000 NEXUS 1000v
NEXUS 7000 - VPC
NEXUS 5000
NEXUS 7000 - VPC
NEXUS 7000
Service Appliances
Catalyst 6500
Service Modules
Unified Compute System
54© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
CBS 3100| MDS 9100 Blade
Catalyst 49xxRack
Nexus 7000 End-of-Row
Nexus 5K|2KTop of Rack
1GbE,10GbE Server Access
MDS 9500Storage
Catalyst 6500End-of-Row
Storage
IP+MPLS WAN Agg Router
WAN
MDS 9500StorageServices
Gigabit Ethernet
10 Gigabit Ethernet
4, 8Gb Fibre Channel
UCS bladeorNexus 4K
10 Gigabit FCoE/DCE
Catalyst 650010GbE VSS AggDC Services
Nexus 700010GbE Core
Fitting the pieces together…
Nexus 700010GbE AggCatalyst 6500DC Services
DC AggregationSAN A/B
1GbE Server Access
Nexus 1000V VN-Link
DC Access
DC Core
90© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Cisco Nexus 5000 Architecture
91© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Hardware Architecture
92© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Virtual Output Queues
CrossbarFabric
Ingress Port
Packet Buffer
Egress Port
Packet BufferPacket Buffer
SchedulerEgress Queue
Q1
Q8
Q1
Q8
Egress Queue
VOQ 1
VOQ N
Egress Port
Packet BufferPacket Buffer
Egress Queue
Q1
Q8
VOQ 1
VOQ N
Q1
Q8
Q1
Q8
93© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Policy Enforcement
Frames evaluated by multistage engine searches occur in parallel results, and are evaluated in pipeline diagnostics, and control plane tap pipelines.
Multipath Expansion
VLAN MembershipCheck
Interface, VLAN, and MAC Binding
MAC and L3 Binding(IP & Fibre Channel)
Fibre Channel Zone Membership Check
Port ACLs
VLAN ACLs (ingress)
QoS ACLs (ingress)
Role Based ACLs (egress)
Control P
lane Redirect/S
nooping
Sw
itch Port A
nalyzer (SP
AN
) and Diagnostic S
ampling
failpass
pass fail
pass
pass
permit
permit
permit
permit
fail
fail
deny
deny
deny
policerdrop
To SupTo SPANsession
Parsed Packet
Collect Interface Configuration and
State
Virtual Interface Table (512)
Vlan Translation Table (4K)
Vlan Translation Table (4K)
Vlan State Table (1K)
Determine Destination
(ingress only)
Fibre Channel Switch Table (4K)
EthernetLearning
Policy EnforcementACL Search Engine
(2K)
MultipathExpansion
(ingress only)
Zoning Table(2K)
Zoning Table(2K)
RBACL Label Table(2K)
RBACL Label Table(2K)
Binding Table(2K)
Binding Table(2K)
Fibre Channel Multipath Table (1K)
PortChannelTable(16)
Multicast Vector Table (4K)Station Table
(16K)
Station Table(16K)
Editing Instructions &Virtual Output Queue List
96© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Default QoS Configuration
switch1# sh policy-map
Type qos policy-maps ==================== policy-map type qos default-in-policy class type qos class-fcoe set qos-group 1 class type qos class-default set qos-group 0
Type queuing policy-maps ======================== policy-map type queuing default-in-policy class type queuing class-fcoe bandwidth percent 50 class type queuing class-default bandwidth percent 50 policy-map type queuing default-out-policy class type queuing class-fcoe bandwidth percent 50 class type queuing class-default bandwidth percent 50
Type network-qos policy-maps =============================== policy-map type network-qos default-uf-policy class type network-qos class-fcoe pause no-drop mtu 2240 class type network-qos class-default mtu 1538
switch2# show class-map
Type qos class-maps ===================
class-map type qos class-fcoe match cos 3
class-map type qos class-default match any
Type queuing class-maps =======================
class-map type queuing class-fcoe match qos-group 1
class-map type queuing class-default match qos-group 0
Type network-qos class-maps ==============================
class-map type network-qos class-fcoe match qos-group 1
class-map type network-qos class-default match qos-group 0
• Qos is always on.
• Four default class of services defined when system boots up
• Two for control traffic. One for FCoE traffic and another one for Ethernet traffic
• Match CoS 3 for class-fcoe.
• Class-fcoe is no-drop with MTU 2240.
• Match any for class-default
• Class-fcoe and class-default get 50% of guaranteed bandwidth by default
99© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Nexus 5000 Software Features Set
Layer 2
802.1w (Rapid Spanning Tree), 802.1s (Multiple Spanning Tree), RPVST+, Root Guard, Uplink Guard, Bridge Assurance, PortFast, CDP, PVLANs, UDLD, LACP, IGMP Snooping, 802.1Q trunks, Port-Channel, SVI, SPAN, Jumbo Frames, NTP, Link State Tracking (LST)
Management/Security
Radius, Tacacs+, AAA, CallHome, SSHv1/V2, telnet, IPv4 & IPv6 mgmt, SNMP MiBs, Traps, EthAnalyzer (wireshark), RBAC, DCNM, RME support via Cisco Works, syslog, coredump, RMON, first-setup script, accounting log, checkpoint and configuration rollback
ACL/QOS
PACLs, VACLs, Session based ACLs, ACL based QOS, egress Bandwidth Limiting, 802.1p priority, strict priority scheduling, WRED, Tail Drop, Storm Control (broadcast, multicast), Egress Shaper
FCOE FIP Snooping Bridge, DCBXP, PFC (Priority Flow Control), 8 Virtual Lanes, ETS (Enhance Transmission Selection)
100© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Nexus 5000 and FC Connectivity
101© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Switch Mode
Nexus 5000 FC module can be ISL’ed to another FC switch (E_port) Zoning, DPVM, etc. are enforced on the Nexus 5000 Domain manager, FSPF, zone server, fabric login server, name
server run on Nexus 5000 Require a domain ID for every VSAN Interop mode considerations when connecting to non-Cisco FC
switches
Note: Nexus 5000 supports direct connectivity to FC initiator (server HBAs) and targets (storage arrays)
102© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
N-Port Virtualization (NPV) mode
Nexus 5000 FC module can work in NPV modeServer-facing ports are regular F portsUplinks toward SAN core fabric are NP ports
Nexus 5000 switches assign FCIDs to attached devices First byte in FCID received from core SAN switch
One VSAN per uplink on Nexus 5000 (will change in future)No trunking or channelling of NP ports
Zoning, DPVM, etc. are not enforced on the Nexus 5000 Domain manager, FSPF, zone server, fabric login server, name server
They do not run on Nexus 5000
No local switchingAll traffic routed via the core SAN switches
103© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
N-Port Virtualization (NPV): An Overview
Nexus 5000 to SAN Fabric A & BAssign FCIDs to servers – no domain to configure!
F-port
NPV-Core Switch (MDS or 3rd party switch with NPIV support)
NP-port
FC
VSAN 5
Can have multipleuplinks – one VSAN per uplinkTwo uplinks can be in the same VSANNo port channel or trunking
HostHost
Host
F-ports
N-ports
VSAN 10
Servers log in (FLOGI) locally
113© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Working withNexus 2148(Optional)
114© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Nexus 2000 Fabric ExtenderVirtual Chassis
115© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Nexus 2000 Fabric Extender1GE Connectivity
116© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00
Fabric ExtenderUplink Modes
Fabric Extender associates (pins) a server side (1GE) port with an uplink (10GE) port
Server ports are either individually pinned to specific uplinks (static pinning) or all interfaces pinned to a single logical port channel
Behaviour on FEX uplink failure depends on the configuration
Static Pinning – Server ports pinned to the specific uplink are brought down with the failure of the pinned uplink
Port Channel – Server traffic is shifted to remaining uplinks based on port channel hash
Static Pinning
Port Channel
Server Interface goes down
Server Interface stays active