brkarc 3454 - in-depth and personal with the cisco nexus 2000 fabric extender architectures features...

In-depth and personal with the Cisco Nexus 2000 Fabric Extender Architectures, Features, and Topologies

Pavan S Kaushik

Technical Marketing Engineer

BRKARC-3454

https://www.ciscolive.com/us2016/content/sessionDetail.do?SESSION_ID=5182

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Never heard of FEX before ??

No Problem !!

FEX Gurus in the house ??

No Problem !!

BRKARC-3454 3


Session Goals

• Brief Nexus 5000/6000/7000/9000 Parent platform introduction.

• Discuss Nexus 2000 platform architectures, supported parent switches, actual designs and best practices.

• Nexus 2000 topologies, features and implementation.

• vPC/FCoE/FC/FP/QoS/ACI Deep dive and troubleshooting will not be covered in this session.

BRKARC-3454 4


House Rules

• We have Nexus 2300 switches in the Room !! Do take a look.

• Keep the session interactive.

• Highly detail-oriented questions will be taken offline(ex. Optics/Model compatibility etc).

5BRKARC-3454

Agenda

• Introduction

• Parent switches overview

• Platform overview

• Features deep-dive and packet-walk

• Design and topologies

• Configuration and Provisioning

• Connectivity options and Optics

• Summary

Introduction Nexus 2000 / FEX


Modular SwitchIn-chassis Linecards

Cisco Nexus 2000 Fabric EXtenders

Parent Switch*With Remote Linecards (FEX)

* - Parent switch can be Fixed or Modular

BRKARC-3454 8


FEX: Virtualized Access Switching

• Flexible and simplified top-of-rack (ToR) designs

• Simplified Management, Centralized configuration

• Multiple Parent Switch support – N5K, N6K, N7K, N9K

• Unified Ports provide flexible LAN and SAN deployment

Extending the boundaries of Data Center Fabric

Virtualized Switch

. . .

BRKARC-3454 9


Nexus 2000 Solution

Low OPEXLow CAPEX

Lower Devices to ManageCentral Point of Management

Traditional ArchitectureAgnostic(ACI, FA)

Enhanced BuffersLow over subscription

High Port Density

Enhanced

Performance

Low TCO

Simplified

Operation

Nexus 2000 Architecture with Single Point of Management

Architecture

Flexibility

BRKARC-3454 10

Parent switches overview : Nexus 5000/6000/7000/9000


Fixed Switching Platforms Modular Switching Platforms

Application Centric Infrastructure (ACI)

Nexus 9000 Standalone

Nexus 2200/Nexus 2300

Nexus 6000 Nexus 7000

Nexus 5000/5500

Nexus 5600

Nexus 2000 - Parents Switches Family

Nexus 7700

Nexus 9000

BRKARC-3454 12


NEXUS 5600 SERIES

Nexus 5672UP7.0(1)N1(1)

Nexus 56128P7.0(2)N1(1)

Nexus 5624Q7.1(0)N1(1a)

Nexus 5648Q7.1(1)N1(1)

Nexus 5696Q7.0(4)N1(1)

Switch Fabric

Throughput1.44 Tbps 2.56 Tbps 1.92 Tbps 3.84 Tbps 7.68 Tbps

Port-to-Port Latency ~1.0 us

Layer 3 Capability Integrated Line-Rate

Switch Footprint 1RU 2RU 1RU 2RU 4RU

10 Gigabit Ethernet

Port 72 128 96 192 384

40 Gigabit Ethernet

Port6 8 24 48 96

Unified Ports 16 48 N/A N/A 48

1/10 Gigabit Ethernet

With FEX1152 1152 1152 1152 2304

Packet Buffer 25MB per 3x40G (or 12x10G)

FEX Support Yes* Yes* Yes* Yes* Yes*

SC

AL

ED

EN

SIT

YP

ER

FO

RM

AN

CE

BRKARC-3454 13

* Verify model-specific compatibility through DataSheets / ReleaseNotes / Hardware Installation Guides


Nexus 7000 Product Family Shipping up to 384x40G ports and 192x100G ports

192 x 10-Gbps Ports or


48 x 100-Gbps Ports



96 x 100-Gbps Ports



192 x 100-Gbps Ports768 x 10-Gbps Ports* or

192 x 40-Gbps Ports

384 x 10-Gbps Ports* or

96 x 40-Gbps Ports

336 x 10-Gbps Ports *

84 x 40-Gbps Ports

96 x 10-Gbps Ports *

24 x 40-Gbps Ports

Nexus 7000 Family F3 Modules

Nexus 7706Nexus 7710

Nexus 7718Nexus 7018Nexus 7010

Nexus 7009Nexus 7004

48x 10-Gbps Ports or

24x 40-Gbps Ports or

Nexus 7002

BRKARC-3454 14


Nexus 7000 / 7700 Line Card Modules

M1 1G/10G M2 10G / 40G / 100G

F1 10G F2 10GF2E 10G F3 40G

F2E 10GF3 10G / 40G / 100G

F3 closes the F/M feature gap!

M3 1G / 10G / 40G

BRKARC-3454 15


Nexus 7000/7700 Supervisor FEX support

Nexus 7000 SUP2 Nexus 7000 SUP2E Nexus 7700 SUP2E

CPU Quad-Core Xeon Dual Quad-Core Xeon Dual Quad-Core Xeon

Speed (GHz) 2.13 2.13 2.13

Memory (GB) 12 32 32

Max VDCs 4 (user) +1 (admin) 8 (user) +1 (admin) 8 (user) +1 (admin)

Max VLANs per FEX 2000 2000 2000

Max VLANs per FEX HIF 75 75 75

Max. no. of FEX/Ports 32 FEX /1536 Ports 64 FEX /3072 Ports 64 FEX /3072 Ports

BRKARC-3454 16


NEXUS 7000 Line Card FEX supportM2-24XP M2-06FQ F2-48XP F2E-48XP F3-12FQ F2E-48XP F3-48XP F3-24FQ

Chassis

Supported Nexus 7000 Nexus 7000 Nexus 7000 Nexus 7000 Nexus 7000 Nexus 7700 Nexus 7700 Nexus 7700

Ports and

Speed24x10GE 6x40GE 48x1/10GE 48x1/10GE 12x40GE 48X1/10GE 48x1/10GE 24x40GE

Interfaces SFP+ QSFP+ SFP/SFP+ SFP/SFP+ QSFP+ SFP/SFP+ SFP/SFP+ QSFP+

Max Fabric

Cards and BW

(GBps)

3

240

3

240

5

480

5

480

3

480

3

480

3

480

5

960

Performance

(Mpps) 120120 720 720 720 720 720 1440

vPC (Support) Yes Yes Yes Yes Yes Yes Yes Yes

FCoE, FP

SupportNo Yes Yes Yes Yes Yes Yes Yes

Max. Power (W) 795 795 450 450 500 450 450 650

FEX Support Yes* Yes* Yes* Yes* Yes* Yes* Yes* Yes*

BRKARC-3454 17

* Verify model-specific compatibility through DataSheets / Release Notes / Hardware Installation Guides


Cisco Nexus 9500 Platform SwitchesDensity in DC Optimized Footprint

21 R

U

8-Slot

Cisco Nexus® 9500

4-Slot

16

57.6 Tbps

512

1024

576

2304/2048

8

28.8 Tbps

256

512

288

1152/1024

4

14.4 Tbps

128

256

144

576/512

Nexus 9516

16-Slot


Payload Slots

Capacity

100G Density

50G Density

40G Density

10/25G Density

13 R

U

7 R

U

Fabric Bandwidth:

3.2 Tbps/ slot

Cloud Deployment Options:

ACI Multi-Site, Telemetry,

Cloud Scale

Multi-Generation

Investment Protection:

No Mid-plane, Power Supply Headroom

for 100/400G and integrated optics/

encryption

BRKARC-3454 18


Nexus 9500 Supervisor FEX support

Supervisor A Supervisor B

CPU 4 Core, 1.8GHz 6 Core, 2.2GHz

Memory (GB) 16 GB 24GB

Storage (SSD) 64 GB 256GB

Max. Power (W) 80 90

FEX Support 32 FEX /1536 Ports 32 FEX / 1536 Ports

VLANs per FEX 2000 2000

VLANs per FEX HIF 75 75

BRKARC-3454 19


NEXUS 9500 Line Card FEX support

X9636PQ X9536PQ X9564PX X9432PQ X9464PX

Chassis Supported Nexus 9504/9508 Nexus 9500 Nexus 9500 Nexus 9500 Nexus 9500

Ports and Speed 36x40GB 36x40GB 48x1/10GB,

4x40GB32x40GB

48x1/10GB,

4x40GB

Interfaces QSFP+ QSFP+SFP/SFP+

QSFPQSFP+

SFP/SFP+

QSFP+

Performance

(Mpps)120 720 720 720 720

vPC (Support) Yes Yes Yes Yes Yes

Buffer 36 MB 104 MB 104 MB 24 MB 12 MB

Max. Power (W) 795 450 450 310 500

FEX Support Yes* Yes* Yes* Yes* Yes*

BRKARC-3454 20



+

Nexus 9300 Fixed Series ACI Leaf and NX-OS Switches

9396 SFP+(48x10+12x40G)

9396 Copper (48x10+12x40G)

93128 Copper (96x10+8x40G)

Modular Uplink | 2HCY‘13

• Analytics

• Quad-core CPU

• 64 GB SSD

• 24 GB Memory

93120 Copper (96x10+6x40G)

9372(E) SFP+ (48x10+6x40G)

9332 QSFP+ (32x40G)

9372 (E) Copper (48x10+6x40G)

Integrated Uplink | CY’14-15

93180YC-EX (48x25G+6x100G)

93108TC-EX (48x10GT+6x100G)

32p QSFP Switch

28p 40G + 4p 100G

24p 40G + 6p 100G

NEW

Q2’CY16

Q4CY16

21


Nexus 9300 Switches FEX support

Nexus 9332PQ Nexus 9372PX Nexus 9396PX

Form Factor (RU) 1 1 2

Throughput(Tbps) 2.56 1.44 1.44

1/10GB Ports X 48 48

40 GB Ports 32 6 12

Latency (us) < 2 < 2 < 2

Buffer 25 MB 25 MB 50 MB

Max. Power (W) 650 650 650

Applications ACI Leaf ACI Leaf ACI Leaf

FEX Support Yes Yes Yes

BRKARC-3454 22



FEX Scalability

+ 16 -

64

FE

X

Cisco Parent Switch

Cisco Nexus® 2000 FEX

Supports 3000+ ports per single point of management

Nexus

5500

Nexus

5600

Nexus

5696

Nexus

7000

Nexus

7700

Nexus

9300

Nexus

9500

L3 16 24 32 64 64 16 32

L2 24 24 48 64 64 16 32

23

Platform Overview:Nexus 2000 / FEX


Nexus 2000 / FEX – Naming Convention

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

N2K-C2232PP-10GE

1–3: Cisco Nexus 2000 Series Identifier

4: Dash

5: Chassis

6: Series Number

7: Series Generation

8–9: Number of Ports

10: Host Physical Interface Type (P = Requires SFP+, T = Copper, UP = Unified Port, F = Includes FET)

11: Uplink Physical Interface Type (P = Requires SFP+, Q = QSFP, T = Copper, F = Includes FET, M = Module)

12: Dash

13–16: Host Connection Speed, Enhanced model or Bundle identifier (-BUN)

BRKARC-3454 25


History of Cisco Nexus 2000 SeriesSimplified Operations with Revolutionary Scale

N2K-C2248TP-E-1GE(Enhanced Buffers)

N2K-C2232PP-10GE

N2K-C2232TM-E-10GE

N2K-C2248PQ – 10GE(QSFP Uplink)

Blade FEX

SFP/SFP+ FEX

B22 Dell FEX1/10G FEX for Dell Blade servers

B22 FTS FEX1/10G FEX for Fujitsu Blade serversB22 HP FEX

1/10G FEX forHP Blade servers

1/10G FEX forIBM Blade servers

B22 IBM

N2K-C2348UPQ-1/10GE(40GE Uplink, Enhanced Buffers)

N2K-C2332TQ-1/10/40GE(40G Uplink, Enhanced Buffers)

BRKARC-3454 26

N2K-C2348TQ / TQ-E(40G Uplink, Enhanced Buffers)

Base-T FEX


Nexus 2000 100M/1GBASE-T

Nexus 2224TP Nexus 2248TP Nexus 2248TP-E


Network Interfaces 2 (SFP+) 4 (SFP+) 4 (SFP+)

Host Interfaces 24 48 48

Performance 88Gbps 176Gbps 176Gbps

FCoE Support No No No

Buffer 8 MB 8 MB 32 MB

Max. Power (W) 80 W 95 W 95 W

Oversubscription 1.2:1 1.2:1 1.2:1

Nexus Parent Switches Nexus 5000, 6000, 7000, 9000 Nexus 5000, 6000, 7000, 9000 Nexus 5000, 6000, 7000, 9000

BRKARC-3454 27


Nexus 2000 1/10GBASE-T

Nexus 2232TM Nexus 2232TM-E Nexus 2332TQ Nexus 2348TQ/E*

Form Factor (RU) 1 1 1 1

Network

Interfaces 8 (SFP+) 8 (SFP+) 4 (QSFP+) 6 (QSFP+)

Host Interfaces 32 32 32 48

Performance 560Gbps 560Gbps 1.2Tbps 1.2Tbps

FCoE Support No No No Yes (FCoE)

Buffer 8 MB 8 MB 32 MB 32 MB

Max. Power (W) 350 W 240 W 350 W 350 W

Oversubscription 4:1 4:1 2:1 2:1

Nexus Parent

Switches

Nexus 5000, 6000, 7000,

9000

Nexus 5000, 6000, 7000,

9000

Nexus 5000, 6000,

7000, 9000

Nexus 5000, 6000, 7000,

9000

* Some N2348TQ-E items are on the roadmap

BRKARC-3454 28


Nexus 2000 10G (SFP+)

Nexus 2232PP Nexus 2248PQ Nexus 2348UPQ


Network Interfaces 8 (SFP+) 4 (QSFP+) 6 (QSFP+)

Host Interfaces 32 48 48

Performance 560Gbps 960Gbps 1.2Tbps

FCoE Support Yes Yes Yes

Buffer 8 MB 32 MB N2H, 8 MB H2N 32 MB

Max. Power (W) 210 W 175 W 350 W

Oversubscription 4:1 3:1 2:1

Nexus Parent SwitchesNexus 5000, 6000, 7000, 9000

UCS FINexus 5000, 6000, 7000, 9000 Nexus 5000, 6000, 7000, 9000

BRKARC-3454 29


FA

N S

IDE

OF

FE

XP

OR

T S

IDE

OF

FE

XNexus 2348UPQ Overview

• 6x40G QSFP+ Ports• 48x1/10G SFP+ Ports

Choice of Port Side

Intake/Exhaust Airflow

Redundant 1+1

AC/DC Power Supplies Unit

Redundant N+1 Fan

Modules

• 48 Fixed 1/10G SFP+ Unified Capable Ports.

• Traditional Ethernet or FCoE

• 48 Unified Ports provide up to 16GFC

• 32M of Buffer with Lower Oversubscription ratio of 2:1

• Parent Switch – N5k, N6k, N7k, N9k.

BRKARC-3454 30


Nexus 2348UPQ Buffer Allocation

• Total 32MB shared by 6 NIF(Network Interface) and 48 HIF(Host Interface)

• Dedicated buffer pool and shared buffer pool Each of the NIF or HIF can manage all 32MB buffer

• Queue-limits can be used to control buffer-usage by bursty flows.

NIF 1-6x40GE

HIF 1-48

dedicated

shared32MB

BRKARC-3454 31


Nexus 2000 Internal ArchitectureNexus 2248PQ vs. Nexus 2348UPQ

QSFP QSFP QSFP QSFP

Uplink/BufferUplink/Buffer

Port 1 - 24(Ethernet/FCoE)

ASIC A

Port 25 - 48(Ethernet/FCoE)

ASIC B

4x 40Gbe QSFP+(4x10G-Mode only)

Port 1 - 48(Unified Ports: Ethernet/FCoE/FC)

QSFP QSFPQSFP QSFP QSFP QSFP

3rd Generation ASIC

6x 40Gbe QSFP+(40G or 4x10G-Mode)

BRKARC-3454 32

* For all N23XX : max. 8 x 10GE Uplink Port-Channel bundle


Cisco Nexus 2348UPQCompared with other Nexus 2000 1/10GE Fabric Extender

Model Nexus 2232PP Nexus 2248PQ Nexus 2348UPQ

Host Ports 32x SFP/SFP+ (1/10G) 48 x SFP/SFP+ (1/10G)48x SFP/SFP+ (1/10G)

(true 40G or 4x 10G-Mode)

Uplink Ports 8x 10GbE SFP+4x 40Gbe QSFP+(4x 10G-Mode only)

6x 40Gbe QSFP+*(true 40G or 4x 10G-Mode)

FCoE ✔ ✔ ✔

Buffer 157.5KB per port157.5KB per port

2x 16M Shared Buffer (Uplink)32MB Shared Buffer

BRKARC-3454 33

* For all N23XX : max. 8 x 10GE Uplink Port-Channel bundle


Fabric Extender 100M/1G/10G CopperNexus 2348TQ-E

48 port 3rd Generation Fabric Extender

• Cost optimized version of N2K-C2348TQ.

• Identical to N2K-C2348TQ in features.

• 48 x 100M/1G/10G host/downlink interfaces

• Large 32MB Buffer

• Supports FCoE for LAN/SAN convergence.

48 100/1000 RJ45 DownlinksBRKARC-3454 34

Nexus 23XX FEX Uplink support

Max. 6 x 40GE Uplinks

or

Max. 8 x 10GE Uplinks

(throught breakout cables)


Fabric Extender 100M/1GENexus 2248TP-E-GE Overview

• 48 ports 100M/1G-BaseT Enhanced Fabric Extender

• 48x 100/1000M host/downlink interfaces.

• 4x 10GE on network/uplink interfaces

• Upstream N5K supports various FEX (mix-and-match)

• 6 user-defined QoS queues (8 QoS queues)

• Design scenario: • High density 100M/1G access

• Cost effective 100M/1G solution

• Buffer optimized for specific Data Center workloads such as Big Data, Hadoop and Distributed Storage

48 100/1000 RJ45 Downlinks

Key differences from N2248TP:

• Optimized Buffers

• Enhanced Debug counters

BRKARC-3454 35


Nexus 2248TP-E Buffer Allocation

• Total 32MB shared by 4 NIF(Network Interface) and 48 HIF(Host Interface)

• Dedicated buffer pool and shared buffer pool. Size of each pool is configurable by software

• Current 128KB/port dedicated at NIF. Rest (around 26MB) is shared

• Each of the NIF or HIF can manage all 32MB buffer

NIF 1-4x10GE

HIF 1-48

dedicated

shared32MB

BRKARC-3454 36


Bigdata and Hadoop

Cisco Unified Fabric

NIF 1-4x10GE

HIF 1-48

dedicated

shared

N2248TP-E

Cloudera

certified !!

N2248-TP-E

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/white_paper_c11-690561.html

32MB Buffer

BRKARC-3454 37

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/white_paper_c11-690561.html


32 ports 1/10G Fabric Extender

• 32x 1/10GE SFP+ host interfaces

• 8x SFP network interfaces

Design Scenario

• 1/10G Server Connectivity (SFP+ ToR)

• Connectivity Flexibility

• Predictable low latency

Nexus 2232PP-10GE Fabric Extender

8 SFP+ Uplinks32 1G/10GE SFP+ Downlinks

BRKARC-3454 38


Nexus 2232TM-E

What is same as 2232TM?• Form Factor, ports, power supplies, etc.

• Cabling options and distance supported

• Port to port latency

What is different?• A newer and better 10GBase-T PHY (40nm)

• Better BER characteristics expected – qualification underway

• Reduction in power consumption

32 1G/10GT Downlinks

8 x 10GE SFP+ Uplinks

BRKARC-3454 39


• IEEE 802.3.an standard

• Structured Copper Cabling with RJ45 connectors

• Supports 10Gbps throughput for up to 100 meters or 330 feet

• Leverages Category 6, Category 6A, and Category 7 type cabling

Cable TransceiverLatency

PHY Power(each side)

DistanceTechnology

Cat6a/7Cat6a/7

~3ms~3ms

~6W~4W

100m30m

10GBASE-T – 65nm

Cat6a/7Cat6a/7

~2.5ms~2.5ms

~5W~3W

100m30m

10GBASE-T – 40nm

Nexus 2232TM-E 10GBASE-T

BRKARC-3454 40


48 ports 10/1G Fabric Extender

• 4x QSFP (16x10GE SFP+) on network interfaces

• Feature parity with Nexus 2232PP

• Additional uplink buffers (2x16MB)

Design Scenario

• High Density 10GE SFP+ ToR

• Connectivity Flexibility

• Predictable low latency

48 1G/10GE SFP+ Downlinks

4 QSFP+ Uplinks

Nexus 2248PQ-10GE Fabric Extender

BRKARC-3454 41


Nexus 2000 Overview

2348UPQ/2348TQ PPT

• Redundant 1+1 AC/DC Power Supplies

• Redundant N+1 Fan Modules

• Color coding for both power supplies and Fan for port side exhaust

and port side intake airflow

Port Side Exhaust

Port Side Intake

BRKARC-3454 42


Nexus 2000 Airflow

Port Side Exhaust(Hot Aisle-Server Rack Airflow Alignment)

Port Side Intake(Cold Aisle-Network Rack Airflow Alignment)

BRKARC-3454 43

Features deep-dive and Packet Walk


Nexus 5000/6000 and 2000Packet Forwarding Overview

Unified

Crossbar

Fabric

Ingress

UPCEgress

UPCNexus 2000

FEX ASIC

1. Frame received on N2K HIF port

1

2 3 4 5

6

2. Nexus 2000 appends VNTag and

forwards frame to fabric uplink

3. Nexus 5000 UPC performs ingress forwarding and

queuing

4. If required egress queuing and flow control

5. Nexus 5000 UPC appends destination VNTag and forwards frame on fabric link

6. VNTag stripped and

frame forwarded out on N2K HIF

port

Nexus 5000/6000 Nexus 2000Nexus 2000

Nexus 2000

FEX ASIC

BRKARC-3454 45


Nexus 2000 Fabric Extender (FEX) VNTag

Logical Interface (LIF) on the

ingress UPC is used to forward

the packet

N2K ASIC maps specific VNTag to

HIF interface

Packet is forwarded over fabric link using

a specific VNTag

• Nexus 2000 Fabric Extender operates as a remote line card and does not support local switching

• All forwarding is performed on the Nexus 5500/5600/6000 UPC VNTag is a Network Interface Virtualization (NIV) technology that ‘extends (Logical Interface = LIF) to the Nexus 2000 VIF referred to as a Host Interface (HIF)

• VNTag is added/removed to the packet between Fabric Extender and Nexus 5500/5600/6000/7000/9000

• VNTAG is a 6Bytes Virtual NIC Tag used for forwarding decisions

HIF

LIF

VNTAG Ethertype

source virtual interface

destination virtual interfaced p

l Frame

Payload

CRC[4]

VNTAG[6]

SA[6]

DA[6]

802.1Q[4]

BRKARC-3454 46


Nexus 2000 Port ChannelsNexus 2348/2248 Port Channels

• Nexus 2000 series FEX support local port channels

• A local port channel on the N2K is still seen as a single extended port

• Extended ports are each mapped to a specific VNTag

• HW hashing occurs on the N2K ASIC

• Number of ‘local’ port channels on each N2K is based on the local ASIC

• 22xx – Support up to 24 local port channels of up to 8 interfaces each as well as vPC (total of 2 x 8 = 16 ports)

• 23xx – Support up to 24 local port channels of up to 8 interfaces each as well as vPC (total of 2 x 8 = 16 ports)

1. Packet is received and

lookup forwards out a LIF (N2K)

interface

3. N2K ASIC hashes locally and transmits packet

on one HIF interface

VMKVM

2. Packet is forwarded over

fabric link using a specific VNTag for

the destination N2K LIF (port

channel)

BRKARC-3454 47


Pinning a HIF to a Port Channel

Port Channel

• All server ports are pinned To the

Port Channel

• Automatic load balancing

in case of a link failure

• Definite traffic flow path

visibility and dedicated

bandwidth

BRKARC-3454 48


FCoE and Unified Ports

• Fibre Channel over Ethernet (FCoE) allows encapsulation and transport of Fibre Channel traffic over a shared Ethernet network

• SAN “A” / “B” isolation is maintained across the network

• Unified Ports may be configured to support either native Fibre Channel or Ethernet

Nexus 2000 flexible definition of port function

Fibre

Channel

Traffic

Ethernet

F

C

Fibre Channel

Disk Array

Servers with Converged

Network Adapters (CNAs)

Nexus 5500 Ethernet and

Fibre Channel SwitchesSAN

A

SAN

B

FCoE Links

BRKARC-3454 49


Nexus 2000 SPAN

• FEX Host ports CAN be SPAN source

ports

(Except with F2 Line card in Nexus 7000

parent switch)

• FEX Fabric ports CAN be SPAN source

ports with all parent switches

• FEX Host ports CANNOT be SPAN

destination port with any parent switch

……

Sender Host

Receiver Host

Receiver Host

✔ ✖

✔

BRKARC-3454 50


Nexus 2000 QoSTuning the Port Buffers• Each Fabric Extender (FEX) has local port buffers (FEX leverages

a shared memory model)

• You can control the queue limit for a specified Fabric Extender for egress direction (from the network to the host)

• You can use a lower queue limit value on the Fabric Extender to prevent one blocked receiver from affecting traffic that is sent to other non-congested receivers ("head-of-line blocking”)

Gen 2 UPC

Unified Crossbar Fabric

Gen 2 UPC

Nexus 2000

FEX ASIC

40G Fabric

1G Sink

10G Source (NFS)

# Disabling the per port tail drop threshold

N5672UP-1(config)# system qos

N5672UP-1(config-sys-qos)# no fex queue-limit

N5672UP-1(config-sys-qos)#

# Tuning of the queue limit per FEX HIF port

N5672UP-1(config)# fex 100

N5672UP-1(config-fex)# hardware N2248T queue-limit 356000

N5672UP-1(config-fex)# hardware N2248T queue-limit rx/tx ?

<CR>

<2560-652800> Queue limit in bytes

10G

NF

S

BRKARC-3454 51


• Speed mismatch between 10G NAS and 1G server requires QoS tuning

• 32MB shared buffer to handle larger traffic bursts

• Hadoop, AVID are examples of bursty applications

• You can control the queue limit for a specified Fabric Extender for egress direction (from the network to the host)

N5672UP-1(config-fex)# hardware N2348UPQ queue-limit 4000000 rx

N5672UP-1(config-fex)# hardware N2348UPQ queue-limit 4000000 tx

N5672UP-1(config)#interface e110/1/1

N5672UP-1(config-if)# hardware N2348UPQ queue-limit 4096000

VM

#4

VM

#3

VM

#2

NAS

iSCSI

10G Attached Source (NAS Array)

1G Attached Server

Nexus 2348UPQ32MB Shared Buffer

Tune 2348UPQ to support a extremely large

burst (Hadoop, AVID, …)

10G

NF

S

BRKARC-3454 52


Nexus 2000 Offload

CDP

LLDP

LACP

DCBX

QoS

• Efficient utilization of Nexus parent switch CPU

• TCAM entries are installed on corresponding FEX

ASICs

• QoS offload is disabled by default


N5672UP-1(config-fex)# hardware n2348uPQ qos-policy-offload

BRKARC-3454 53


Nexus 2000 QoS Offload Supported

• N2224TP Fabric Extender 24x1G 2x10G SFP+ Module

• N2232PP Fabric Extender 32x10G SFP+ 8x10G SFP+ Module

• N2232TM Fabric Extender 32x10GBase-T 8x10G SFP+ Module

• N2248T Fabric Extender 48x1G 4x10G SFP+ Module

• N2248TP E Fabric Extender 48x1G 4x10G SFP+ Module

• N2248PQ Fabric Extender 48x10G SFP+ 16x10G SFP+ Module

• N2232TM-E Fabric Extender 32x10GBase-T 8x10G SFP+ Module

• N23xx Fabric Extender 48x10G SFP+ 6x40G QSFP Module

BRKARC-3454 54


Nexus 2000 STP Design Considerations

N5672UP-1#show spanning-tree int eth100/1/1 detail

Port 1025 (Ethernet100/1/1, vPC) of VLAN0190 is designated forwarding

Port path cost 1, Port priority 128, Port Identifier 128.1025

Snip->>>

The port type is edge by port type edge trunk configuration

Link type is point-to-point by default

Bpdu guard is enabled

Bpdu filter is enabled by default

BPDU: sent 11, received 0

• N2K HIF ports have BPDU Guard enabled by default (it is not possible to disable currently)

• If a BPDU is received port will transition to err-disable state

• Global BPDU Filter compliments BPDU Guard

• On link up port will send 11 BPDUs and then stop (in order to reduce CPU load)

EE E

1. X-Connected patch cable

3. BPDU Guard err-disables

edge port and prevents loop

2. BPDU Sent on Link-Up

4. BPDU are not sent once link is

up and active

BRKARC-3454 55


Nexus 2000 : Routing Protocol Neighborships

• Routing Protocol neighborship across FEX requires Priority queueing for Control Plane traffic at FEX HIF, among others.

• Currently this feature is only Supported on Nexus 7k starting 6.2(2), except for N2248PQ.

Note:

Nexus 5000/6000/9000 Series switches do NOT support this.

It is not recommended to connect high performance / critical servers to a FEX, rather use the Parent Port.

On the N7k-FEX, this option may be used for non-critical low bandwidth devices that need routing neighborships.

BRKARC-3454 56

HIF

Router

Routing-Protocol

Neighborship


FCoE DeploymentFEX as Unified TOR

Nexus 5K

FCEthernet

Converged

FCoE

LAN

FC/FCoE SAN

Nexus 2K

B22 Dell FEX1/10G FEX for Dell Blade servers

B22 HP FEX

1/10G FEX forHP Blade servers

1/10G FEX for FTS Blade servers

B22F FEX

B22 IBM FEX1/10G FEX for IBM Blade servers

• The fabric extenders (FEX) can be used at the ToR

• ToR FEX connect to the end or row switches which manages the FEX

• FEX work like external line cards to the end or row switches

• The EoR switches splits the Ethernet and FC traffic

BRKARC-3454 57


Nexus 7K - FEX FCoE Features Update

NXOS 7.3(0)N1(1) introduces support for:

• FCoE over FEX with F3 (N2K-C2348UPQ-10GE, B22HP, N2K-C2232PP-10GE)

• F3 FCoE support for FEX with physical port vPC and vPC+

BRKARC-3454 58


FC Ports on Nexus 2348UPQ UP FEX

• 6x40G QSFP+ Ports

• 4x10G SFP+ Ports Flexibility

• 48x1/10G SFP+ Ports

or 16G FC

• 48 Fixed 1/10G SFP+ Unified Capable Ports supporting Ethernet or

FCoE or FC

• Unified Ports provide up to 24x 16GFC ports or 48x 8GFC ports

• 32M of Buffer with Lower Oversubscription ratio of 2:1

• Parent Switch for Ethernet/FCoE: Nexus 5K, 6K, 7K and 9K

• Parent Switch for FC: Nexus 5600

• Shipped with Ethernet/FCoE support in July 2014

with 7.0(3)N1(1)

• Enabled FC support in March 2016 with 7.3(0)N1(1)

• Fibre Channel FEX supported number (ID) range is

from 100 to 130

• Support native Fibre Channel speed 2/4/8/16G

• Fibre Channel ports must be configured in groups of 4

ports and starts from beginning

First FEX Solution Designed for All Storage Connectivity

• Only F port is supported on FC-FEX

• NP Port type is not supported so NPV device cannot

connect to the FC FEX directly

• F-port trunking and F-port channel are not supported

on FC-FEX ports

• The default BB credits on each FC interface are 32;

and can be set to max 128

• Scale per parent switch: 24 FEX, MAX 96 FC ports

The FC ports on Nexus 2348UPQ are supported only on the Cisco Nexus 5600 switches.


To support 16G FC, the fabric mode on the parent Nexus 5600 should be in 40G mode.

Only odd-numbered HIF ports can be configured as the 16G FC.

When the odd-numbered HIF port is configured as 16G FC, the adjacent even-numbered ports must be changed to the out-of-service state. Those adjacent even-numbered ports next to it cannot be used.

When the speed is set to Auto, the 16G host port comes up at 8G speed. For the ports to come up at 16G speed, use the switchport speed 16000 command to explicitly set the speed to 16G.

Configure 16G FC Ports on UP FEX

stg-5672-16G-1(config-vsan-db)# interface fc121/1/11

stg-5672-16G-1(config-if)# switchport speed 16000

fc121/1/11: (error) Config not allowed - required to change the mode of the next port to out-of-service

stg-5672-16G-1(config-if)#

BRKARC-3454 60


Extending ACI Policy Based Forwarding into Existing Data Center Networks

N5K N2K N6K

AVS OVS

Extended ACI Fabric

AVSvSwitch

ACI Enabled

Remote N9K

• Extend Policy Based Forwarding

• Extend Visibility, Fault and Audit

• Automated Device Management for extended Fabric nodes

vSwitch HyperV

BRKARC-3454 61

Design and Topologies


Single Homed Nexus, Single Homed FEX Attached Host

Nexus 7000

Nexus 5000

Nexus 2000

Nexus 7000/5000 Port Channel

• Simple and Easy to Configure

End to end

• Traffic is automatically re-distributed across remaining NIFs upon failure

• Simple L2/L3 Flow visibility HIF/NIF to next hop nexus switch

• Single link to host: no redundancy at Server or FEX level.

UCS / Hosts

BRKARC-3454 63


Nexus 7000

Nexus 5000

Nexus 2000

Nexus 7000/5000 Virtual Port Channel

• Easy to Configure FEX

• Network Resiliency and High availability between N7k & N5k with vPC / double-sided vPC.

• Simple L2/L3 visibility from HIF-NIF to N5k

• Single link to host: no redundancy at Server or FEX level.

UCS / Hosts

Dual Homed Nexus, Single Homed FEX Attached Host

BRKARC-3454 64


Nexus 7000

Nexus 5000

Nexus 2000

Nexus 7000/5000/2000 Virtual Port Channel

• High Availability and network

resiliency between N7k & N5k& FEX

• No Blocking ports in STP when usingvPC.

• L2/L3 Flows can be based on primaryor secondary vPC switch.

UCS / Hosts

Dual Homed Nexus/FEX, Single Homed FEX Attached Host

BRKARC-3454 65


• Network Resiliency and Highavailability across network andcompute

• Increased Network Bandwidth

• Active NIC teaming can also be used for further options.

Nexus 7000

Nexus 5600

Nexus 2000


Active

Dual Homed Nexus/FEX, Dual Homed FEX AA-Host

UCS / Hosts

BRKARC-3454 66


Dual Homed Nexus/FEX, Dual Homed FEX AS-Host

Nexus 7000

Nexus 5000

Nexus 2000


ActiveStandby

• Network Resiliency and High

Availability across network and

compute

• Increased Network Bandwidth

• Active/Standby NIC teaming can

be dependent on CNA capabilities

UCS / Hosts

BRKARC-3454 67


Nexus 2000 (FEX) Straight-Through Deployment with VPC

• Port-channel connectivity from the server

• Two Nexus switches bundled into a vPC pair

• Suited for servers with Dual NIC and capable of running Port-Channel

Fex 100

Fabric Links

Fex 101

VPC

HIF HIF

S1 S2

68BRKARC-3454


Nexus 2000 (FEX)Active-Active Deployment with VPC

Fabric Links

HIFHIF

Fex 100Fex 101

S1 S2

• Fabric Extender connected to two Nexus

5X00 / 6000 or Nexus 7000/7700

• Suited for servers with Single NIC or

Dual NIC not having port-channel

capability.

• Scale implications of less FEX per

system and less VPC

Note : • This design is currently not supported on Nexus 9X00

69BRKARC-3454


• N7X00 can support up to 64 FEXs

• N7X00 supports max. 15 Active-Active FEX in 7.2(0)D1(1)

• N7X00 supports max. 32 Active-Active FEX in 7.2(0)D1(1)

• Straight-Through FEX and Active-Active FEX cannot exist on the same ASIC instance

• Layer 3 HIF ports are not supported with Active-Active FEX

• Active-Active FEX is not supported with vPC+

Nexus 2000 (FEX) Active-Active Scale & Limitations (N7X00)

BRKARC-3454 70


Nexus 2000 (FEX) - Enhanced VPC

Fabric Links

Fex 100 Fex 101

S1 S2• Port-channel connectivity to dual-homed

FEXs

• From the server perspective a single access

switch with port-channel support – each line

card supported by redundant supervisors

• Ideal design for a combination of single

NIC and Dual NIC servers with port-

channel capability

• Scale implications of less FEX per system

and less VPC

Note :

This design is currently not supported on N7000 / N7700 and N9X00

HIFHIF

71BRKARC-3454


Physical Port vPC

• vPC configuration on a physical Layer 2 port as opposed to a port-channel

• Front panel ports and FEX ports connected to F2/F2e/F3 only

• Improves scaling as separate PC interface not created for single-link VPC leg

• Key benefit: more than 1000 host facing VPCs with FEX

vPC domain

FEX101 FEX102

e101/1/1 e102/1/1Po1

Port-channel vPC

VPC1 VPC1Po1

vPC domain

FEX101 FEX102

e101/1/1 e102/1/1

Physical port vPC

VPC1 VPC1

interface e101/1/1

switchport

vpc 1

lacp mode active

72BRKARC-3454


Nexus 2000 (FEX) Active-Active (Unsupported)

Reference : Supported and Unsupported FEX Topologies: N5000/6000/7000/9000

http://www.cisco.com/c/en/us/support/docs/switches/nexus-2000-series-fabric-extenders/200363-Nexus-2000-Fabric-Extenders-Supported-Un.html

BRKARC-3454 73

http://www.cisco.com/c/en/us/support/docs/switches/nexus-2000-series-fabric-extenders/200363-Nexus-2000-Fabric-Extenders-Supported-Un.html


VXLAN

Hypervisor

VXLAN

Hypervisor

VXLAN

Hypervisor

VXLAN

Hypervisor

VXLAN Termination on TOR (Nexus 5600)Supported Combinations

vtep vtep vtep

✔ ✔✔ ✔

*Switch Local Port-Channel (non VPC) is supported- TOR to Hypervisor- TOR to FEX- FEX to Hypervisor- TOR to FEX to Hypervisor

vtep vtep vtep

BRKARC-3454 74


VXLAN

Hypervisor

VXLAN

Hypervisor

VXLAN

Hypervisor

VXLAN

Hypervisor


vtep vtep vtep vtep

✔ ✔✔ ✔

vtep vtep vtep

BRKARC-3454 75


VXLAN

Hypervisor

VXLAN

Hypervisor

VXLAN

Hypervisor

VXLAN

Hypervisor


vtep

vtep

vtep vtep

vtep vtep vtep

✔ ✔✔ ✔

*Switch Local Port-Channel (non VPC) is supported- TOR to Hypervisor- TOR to FEX- FEX to Hypervisor- TOR to FEX to Hypervisor

vtep vtep vtep

BRKARC-3454 76


VXLAN

Hypervisor

VXLAN

Hypervisor

VXLAN

Hypervisor

VXLAN

Hypervisor


vtep vtep vtep vtep

✔ ✔✔ ✔

BRKARC-3454 77


VXLAN

Hypervisor

VXLAN

Hypervisor

VXLAN

Hypervisor

VXLAN

Hypervisor

VXLAN Termination on TOR (Nexus 5600)Unupported Combinations

vtep vtep

vtep

vtep vtep

vtep vtep vtep

✖ ✖✖ ✖

*FEX connectivity with combination of Hypervisor (Host originated VXLAN) and Bare-Metal Serversis not supported

vtep vtep vtep

BRKARC-3454 78

Configuration and ProvisioningNexus 2000


N5672UP-1(config)# feature fex

N5672UP-1(config-if)# int port-channel100

N5672UP-1(config-if)# switchport mode fex-fabric

N5672UP-1(config-if)# fex associate 100

N5672UP-1(config-if)# vpc 100

N5672UP-1(config-if)# no shut

Configuration

Configuration

N5672UP-1(config)#show fex 100

FEX: 100 Description: FEX0100 state: Online

FEX version: 7.1(0)N1(1a) [Switch version: 7.1(0)N1(1a)]

Extender Serial: FOC1818R2M9

Extender Model: N2K-C2348UPQ, Part No: 73-15489-03

Pinning-mode: static Max-links: 1

Fabric port for control traffic: Eth2/5

FCoE Admin: false

FCoE Oper: true

FCoE FEX AA Configured: false

Fabric interface state:

Po100 - Interface Up. State: Active

Eth2/5 - Interface Up. State: Active

Eth2/6 - Interface Up. State: Active

Verification

BRKARC-3454 80


N5672UP-1(config)# slot 101

N5672UP-1(config-slot)# provision model N2K-C2348UPQ

Pre-Provisioning Configuration

N5672-1(config-slot)# show run int eth101/1/1

!Command: show running-config interface Ethernet103/1/1

!Time: Mon Mar 23 10:58:01 2015

version 7.1(0)N1(1a)

interface Ethernet101/1/1

N5672-1(config-if)# int eth101/1/1

N5672-1(config-if)# switchport access vlan 190

N5672-1(config-if)# no shut

Provision

Configuration & Verification

BRKARC-3454 81


N5672UP-1(config-if)# sh queuing interface eth100/1/1

Ethernet110/1/1 queuing information:

Input buffer allocation:

Qos-group: 0

frh: 2

drop-type: drop

cos: 0 1 2 3 4 5 6

xon xoff buffer-size

---------+---------+-----------

19200 24320 48640

Queueing:

queue qos-group cos priority bandwidth mtu

--------+------------+--------------+---------+---------+----

2 0 0 1 2 3 4 5 6 WRR 100 1600

Queue limit: 66560 bytes

Queue Statistics:

queue rx tx

------+---------------+---------------

2 24434 3066978

Port Statistics:

rx drop rx mcast drop rx error tx drop mux ovflow

---------------+---------------+---------------+---------------+--------------

0 0 0 0 InActive

Ingress queue limit(Configurable)

Egress queue limit(Configurable)

Egress queues:

CoS to queue mapping

Bandwidth allocation

MTU

Drop counters

Nexus 2248TP-E Enhanced Counters

BRKARC-3454 82


N5672UP-1(config-if)# show queuing interface ethernet 100/1/1

if_slot 32, ifidx 0x1f630000

Ethernet100/1/1 queuing information:

Input buffer allocation:

Qos-group: 0

frh: 2

drop-type: drop

cos: 0 1 2 3 4 5 6

xon xoff buffer-size

---------+---------+-----------

5120 15360 71680

Queueing:

queue qos-group cos priority bandwidth mtu

--------+------------+--------------+---------+---------+----

2 0 0 1 2 3 4 5 6 WRR 100 1600

Queue limit: 2088960 bytes

Queue Statistics:

queue rx tx

------+---------------+---------------

2 126 49

Port Statistics:

rx drop rx mcast drop rx error tx drop mux ovflow

---------------+---------------+---------------+---------------+--------------

0 0 0 0 InActive

Priority-flow-control enabled: no

Ingress queue limit

Egress queue limit

Egress queues:

CoS to queue mapping

Bandwidth allocation

MTU

Drop counters

Nexus 2348UPQ Enhanced Counters

BRKARC-3454 83


N5672UP-1(config-if)# show interface ethernet 100/1/1 flowcontrol

--------------------------------------------------------------------------------

Port Send FlowControl Receive FlowControl RxPause TxPause

admin oper admin oper

---------------------------------------------------------------------

Eth100/1/1 on off off off 0 0

Ethernet Flow Control (802.3x) Configuration

N5672UP-1(config-if)# interface ethernet 100/1/1

N5672UP-1(config-if)# flowcontrol send on/off

N5672UP-1(config-if)# flowcontrol receive on/off

• Why do I care to honor flow control?

BRKARC-3454 84


N5672UP-1(config)# feature fcoe (If FCoE not enabled)

FC license checked out successfully

fc_plugin extracted successfully

FC plugin loaded successfully

FCoE manager enabled successfully

FC enabled on all modules successfully

Enabled FCoE QoS policies successfully

N5672UP-1(config)# vlan 100

N5672UP-1(config)# fcoe vsan 100

N5672UP-1(config)# vsan database

N5672UP-1(config)# vsan 100


N5672UP-1(config)# fcoe

N5672UP-1(config)# interface vfc 1011

N5672UP-1(config)# bind interface ethernet 100/1/1

N5672UP-1(config)# switchport trunk allowed vsan 100

N5672UP-1(config)# no shut

FCoE configuration

• Is it not Fairly easy to configure FCoE?

BRKARC-3454 85


N5k-1#show fex 100 detail

FEX: 100 Description: FEX0100 state: Online

FEX version: 7.1(0)N1(1a) [Switch version: 7.1(0)N1(1a)]

FEX Interim version: 7.1(0)N1(1a)

Switch Interim version: 7.1(0)N1(1a)

Extender Serial: FOC1818R2M9

Extender Model: N2K-C2348UPQ, Part No: 73-15489-03

Snip->>>>>>

Logs:

03/18/2015 09:52:21.289192: Module register received

03/23/2015 09:18:18.516214: Module Offline

03/23/2015 09:18:18.561004: Deleting route to FEX

03/23/2015 09:18:18.568532: Module disconnected

03/23/2015 09:21:53.581415: Module register received

03/23/2015 09:21:53.583664: Registration response sent

03/23/2015 09:21:54.634249: create module inserted event.

03/23/2015 09:21:54.635044: Module Online Sequence

03/23/2015 09:22:03.006536: Module Online

BRKARC-3454 86

Configuration Verification

Connectivity OptionsNexus 2000


10G SFP+ Connectivity 40G QSFP+ Connectivity

Nexus 2300 Connectivity Options

Nexus Parent Switch Nexus Parent Switch


BRKARC-3454 93

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=ecUMKfb_CBKKAM&tbnid=pFDn-u0XqPoHRM:&ved=0CAUQjRw&url=http://10gfiber.com/shop/lclc-om3-10g-fiber-optic-cables/&ei=Jew4UtDcNKiciQLR9oGwAg&psig=AFQjCNEGmYreQ_bnZUSANf2pndhpkag3Fg&ust=1379548573739376

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=ecUMKfb_CBKKAM&tbnid=pFDn-u0XqPoHRM:&ved=0CAUQjRw&url=http://10gfiber.com/shop/lclc-om3-10g-fiber-optic-cables/&ei=Jew4UtDcNKiciQLR9oGwAg&psig=AFQjCNEGmYreQ_bnZUSANf2pndhpkag3Fg&ust=1379548573739376


Nexus 2300 Connectivity OptionsNetwork Interface (NIF)

• The Network Interface Fabric Ports can operate 10G/40G mode

• FET-40G, QSFP-Bidi, QSFP+, Active Twinax, including AoC cables can be used to connect to a parent switch

while operating in 40G mode

Host Interface (HIF)• SFP+ Host Interface will support SR, LR, GLC-T, GLC-SX-MM, AoC

• Supported Twinax: 1, 3, 5M (passive) and 7, 10M (active)

*Check the parent switch documentation for the latest supported cable options Twinax Passive cable is

Not supported on NIF

BRKARC-3454 94


OM3-MM Fiber Cable

For QSFP-40G-SR4 and

QSFP-40G-CSR4

12-Fiber MPO Connector

• MPO/MTP connector with 12

pins

• Use 4 fiber pairs

− 4 TX and 4 RX allows for 40 GE

− 4 unused fibers in the center

BRKARC-3454 95


QSFP-40G-SR4

40G-SR4 40G-SR4

40G-SR4

10G-SR

10G-SR

10G-SR

10G-SR

MPO-12 MPO-12

MPO-12 LC Breakout

Support 100M with OM3 MMF and 150M with OM4 MMF

BRKARC-3454 96


QSFP 40G Bidirectional Transceiver – 40G deployment with 10G cabling

Cisco’s innovation has created a new QSFP pluggable 40Gbit/s Bidirectional (BiDi) Transceiver that works with existing infrastructure: Multi-mode fiber

– Using the existing 10Gbit/s Multimode infrastructure

– QSFP-40G-SR-BD has dual LC Connector

– 40Gbit/s BiDi transceiver has two 20Gbit/s channels each transmitted and received simultaneously on two wavelengths

BRKARC-3454 97


QSFP-40G-CSR4 Supports 300M with OM3 MMF

and 400M with OM4 MMF

Compatible with Cisco QSFP-40G-SR4

Compatible with 10G SFP-SR

40G-CSR4 40G-SR4/CSR4

40G-CSR4

10G-SR

10G-SR

10G-SR

10G-SR

MPO-12 MPO-12

MPO-12 LC BreakoutBRKARC-3454 98


QSFP-40G-SR-BD

40G-BD 40G-BD

40G-BD

10G-SR

10G-SR

10G-SR

10G-SR

Supports 100m with OM3 with MMF and 150m with OM4 (OM4+) MMF

LC duplex connectors

Wavelength: ~850nm and ~900nm

Pull-Tab Color: grey

Multimode

LC Duplex

Multimode

LC Duplex

BRKARC-3454 99


Physical Connection Choices

QSFP-SR4/CSR4 QSFP-SR4/CSR4

1M, 3M, 5M Passive

7M, 10M Active

QSFP-SR4/CSR4

QSFP-4x10G

40G QSFP+ 40G QSFP+

QSFP-LR4 QSFP-LR4

40G QSFP+ 40G QSFP+

QSFP-SR-BD QSFP-SR-BD

40G QSFP+

40G QSFP+

40G QSFP+

40G QSFP+

40G QSFP+

40G QSFP+

10G SFP+

10G SFP+

BRKARC-3454 100


FET-40G (aka QSFP+ FET)

• Low-cost QSFP optical transceiver connecting FEX to Cisco Nexus 6004• Supported on Cisco Nexus 5500/N5600/6000/N7000/N9000 and Nexus 2248PQ-10G/Nexus 2300• Interoperable with FET-10G• Support for 100m distance with OM3

Cisco Nexus 5600/6000 Cisco Nexus 5600/6000

Cisco Nexus 2232PP and 2232TM-E 2232TM

Cisco Nexus 2248TP-E 2248TPCisco Nexus 2248PQ-10G/N2300

FET-40G

FET-40G

FET-10G

FET-40G

BRKARC-3454 101


Convert 40GE Interface to Four 10 GE Interfaces

Convert the QSFP Interfaces to Four 10 GE Interfaces When Connecting to a SFP+ Port or to the QSFP Interfaces That

Operate in Four 10 GE Mode, such as the QSFP Uplink of N2248PQ-10G or the Cisco Nexus® 5500 QSFP GEM.

Apply global CLI to change interface types

to 10 GE• Every three contiguous QSFP interfaces resides

on one Port ASIC.

• The port range specified in the CLI has to

include all ports on the ASIC

Power off the affected modules

• Every group of 12 QSFP interfaces is managed

as one module, even for the fixed interfaces

Power on the affected modules

1N6004(config)# interface breakout slot

1 port 1-6 map 10g-4x

N6004(config)# poweroff module ?

<1-8> Please enter module number

N6004(config)# poweroff module 1

N6004(config)# no poweroff module 1

N6004(config)#

2

3

BRKARC-3454 102


QSA-QSFP+ to SFP Adapter

• Convert one QSFP port to one SFP+ port.

• Provides 1G/10G transceiver support

• Long reach connection between for QSFP+ devices and SFP+ devices

• Following 1G/10G transceivers are qualified with QSA in release

• 10G-LR

• 1G GLC-T, 1G SX, LX

• Requires a mode change on QSFP+ port to 4x10G

• Only first 10G interface can be used. Remaining three 10G shown as “Link Not Connected”

QSFP+ Adapter

1G SFP or 10G SFP+

Transceiver

BRKARC-3454 103

SummaryNexus 2000


Understand FEX Architectures & Capabilities

Realize the benefits offered by a FEX design

Evaluate FEX for YOUR network requirements.

Key Takeaways

Lower CAPEX

OPEXEasy to Provision, manage, troubleshoot.

Lowest Cost option for connectivity

Unified

NetworkFlexible Ethernet, FCoE and FC solutions with FEX

Flexible and Scalable ToR design provides Large number of Ports,

with low number of management points.

Simplify your

Network

BRKARC-3454 105


Acronyms and Abbreviations

• HIF – Host Interface • NIF – Network Interface • CIF – CPU Interface • VNTag – Virtual NIC Tag • H2N – Host to Network• N2H – Network to Host• ASIC – Application-Specific Integrated Circuit• vPC – Virtual Port-Channel• FEX – Fabric Extender• VXLAN – Virtual Extensible LAN• ACI – Application Centric Infrastructure• APIC – Application Policy Infrastructure Controller• OTV – Overlay Transport Virtualization• STP – Spanning-Tree• LPM – Longest Prefix Match

• SoC – System on Chip / Switch on Chip• FC – Fiber Channel• FCoE – Fiber Channel over Ethernet• UP – Unified Port• UCF – Unified Crossbar Fabric• UPC – Unified Port Controller• IGMP – Internet Group Management Protocol • ALE – Application Leaf Engine• NFE – Network Forwarding Engine• TCAM – Ternary Content Addressable Memory• FIB – Forwarding Information Base• HPC – High-Performance Computing• Mroute – Multicast Route• LC – Line Card• EOBC – Ethernet Out-of-Band Channel• CPU – Central Processing Unit

BRKARC-3454 106


Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.

BRKARC-3454 107

CiscoLive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/us

http://ciscolive.com/us


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

BRKARC-3454 108

Please join us for the Service Provider Innovation Talk featuring:

Yvette Kanouff | Senior Vice President and General Manager, SP Business

Joe Cozzolino | Senior Vice President, Cisco Services

Thursday, July 14th, 2016

11:30 am - 12:30 pm, In the Oceanside A room

What to expect from this innovation talk

• Insights on market trends and forecasts

• Preview of key technologies and capabilities

• Innovative demonstrations of the latest and greatest products

• Better understanding of how Cisco can help you succeed

Register to attend the session live now or

watch the broadcast on cisco.com

Thank you

Appendix


Host to Network

1

10G MAC Rx 10G MAC Tx

2

VNTag

Forwarding

802.1q

RewriteVNTag

Rewrite

Shared Packet Buffer 32MB

37 write

37 read

3

Input port

Accounting

Flow Cntr

4

Read Mux Tx Arb

Output Queue Time Stamp

5

6

Svif Check

and

ACL Data

HIF

7

10G MAC Tx

VNTag Rewr 802.1q Rewr 8

9

HIF

BRKARC-3454 113

Reference Slides


Reference Slides1

Shared Packet Buffer 32MB

37 write

37 read

5

Input port

Accounting

Flow Cntr

4

3

Unicast FW

Multicast FW

HIF

10G MAC Rx 10G MAC Tx

VNTag

Forwarding

802.1q

Rewrite

VNTag

Rewrite

2

Read Mux Tx Arb

Output Queue

6

NIF

10G MAC Tx

VNTag Rewr 802.1q Rewr

8

7

9BRKARC-3454 114

Network toHost


Typical UP FEX Topologies

Active-Active FEX

VP

C

VSAN A VSAN B

Nexus

5600

Nexus

2348UPQ

• FC-HIF ports configuration must be the same in both the

primary and the secondary VPC switches.

• For the Active-Active FEX, the port-conversion CLI should

be configured on both the vPC peer switches.

FPC FPC

Straight-Through FEX

Nexus

5600

Nexus

2348UPQ

VSAN A VSAN B

VP

C

• The Active-Active FEX is supported, but only one FC FEX

can be associated with a parent switch in the pair.

switch1(config)# fex 101

switch1(config-fex)# fcoe

40G FCoE40GE

40G FCoE

10GE NIC + FC HBA

8/16G FC

10GE

10GE NIC + FC HBA

8/16G FC

10GE

10G FCoE CNA

BRKARC-3454 115

Reference Slides


To support 16G FC, the fabric mode on the parent Nexus 5600 should be in 40G mode.

Only odd-numbered HIF ports can be configured as the 16G FC.

When the odd-numbered HIF port is configured as 16G FC, the adjacent even-numbered ports must be changed to the out-of-service state. Those adjacent even-numbered ports next to it cannot be used.

When the speed is set to Auto, the 16G host port comes up at 8G speed. For the ports to come up at 16G speed, use the switchport speed 16000 command to explicitly set the speed to 16G.

Configuration of 16G FC Ports on UP FEX

stg-5672-16G-1(config-vsan-db)# interface fc121/1/11

stg-5672-16G-1(config-if)# switchport speed 16000

fc121/1/11: (error) Config not allowed - required to change the mode of the next port to out-of-service

stg-5672-16G-1(config-if)#

BRKARC-3454 116

Reference Slides


VFC Creation on 4-Port (HIF) VPC

N5600-1 N5600-2

HostCNA

vPC

vfc1 vfc2 vfc3 vfc4

Switch(config)# int port-channel 500

Switch(config-if)# fcoe multi-vfc enable the feature

N5600-1 N5600-2

FEX-1 FEX-2

HostCNA

vPC

vfc1 vfc2 vfc3 vfc4

Supported from Cisco NX-OS release 7.2(0)N1(1)

Straight-Through FEX

• Only supported on N5600/N6000 platforms (Bigsur ASIC)

• Only supported on Nexus 2300 FEX (Tiburon ASIC)

• Not supported for NPV mode

BRKARC-3454 117

Reference Slides