virtual machine fabric extension (vm-fex) bringing the virtual machines directly on the network...

Virtual Machine Fabric Extension (VM-FEX)Bringing the Virtual Machines Directly on the Network

BRKCOM-2005

Dan Hanson, Technical Marketing Manager, Data Center Group, CCIE #4482

Timothy Ma, Technical Marketing Engineer, Data Center Group

© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2005 Cisco Public 3

The Session will Cover

FEX Overview &History VM-FEX Introduction VM-FEX Operational Model VM-FEX General Baseline on UCS VM-FEX with VMware on UCS VM-FEX with Hyper-V on UCS VM-FEX with KVM on UCS VM-FEX General Details on Nexus 5500 Summary

FEX Overview & History

© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2005 Cisco Public

*IEEE 802.1BR Pre-Standard

Fabric Extender Evolution

FEX Architecture Consolidates network management FEX managed as line card of parent

switch Uses Pre-standard IEEE 802.1BR

IEEE 802.1BR*

Many applications require

multiple interfaces

One NetworkParent Switch to Top of Rack

Today

FEX

Network Administrator

5


Today

Adapter FEX Consolidates multiple 1Gb interface

into a single 10Gb interface Extends network into server Uses Pre-standard IEEE 802.1BR

One NetworkParent Switch to Adapter

IEEE 802.1BR *

Adapter FEX

Many applications require

multiple interfaces

FEX



IEEE 802.1BR *


6


Today

IEEE 802.1BR *

Adapter FEX

Hypervisor

One NetworkVirtual Same As Physical

VM-FEX Consolidates virtual and physical

network Each VM gets a dedicated port on

switch Uses Pre-standard IEEE 802.1BR

IEEE 802.1BR *IEEE 802.1BR *

VM network managed by

Serveradministrator

VM-FEX

FEX




7


Hypervisor

IEEE 802.1BR*

One NetworkParent Switch to Application

Single Point of ManagementFEX Architecture Consolidates network management FEX managed as line card of parent

switch

Adapter FEX Consolidates multiple 1Gb interface

into a single 10Gb interface Extends network into server

VM-FEX Consolidates virtual and physical

network Each VM gets a dedicated port on

switch

IEEE 802.1BR*IEEE 802.1BR*

Adapter FEXToday

Manage network all the way to the OS interface –

Physical and Virtual

FEX

VM FEX


* IEEE 802.1BR Pre-Standard


8


VNTAG mimics forwarding vectors inside a switch

D: Direction, P: Unicast/Multicast, L: Loop

Policy associated with the Virtual Interface NOT port

VLAN member ship, QoS, MTU, Rate limit etc

VNTAG Ether type

Destination Virtual Interface

Source Virtual Interfacever

P

R

ApplicationPayload

TCPIP

EthernetVNTAG

FEX architecture

Switch

FEX

LAN

Frame

VNTAGFrame

Key Architectural Component #1: VNTAG “Intra-Chassis” Bus Header

L

D


FEX Data ForwardingRevisiting Traditional Modular Switches (Example Catalyst 6500)

Constellation Bus had 32 byte header for fabric switching– Vast majority of modular switch vendors have an internal “Tag” for fabric

communications

Originally, Centralized forwarding ASICs– Line cards fed into these ASICs directly

When we needed higher performance – we added faster Switch Fabrics, and Distributed Forwarding Capabilities to system

What this really meant – adding more ASIC forwarding capacity to the system to minimize the number of devices a flow had to traverse

10


FEX Data ForwardingDecoupling the Modular Switch

Think the original C6k Satellite Program for VSL and RSL

The Constellation Bus now is smaller header – 6 Byte VNtag header– Core to FEX technology and being standardized as 802.1BR– This is NOT a 1:1 mapping to VEPA/802.1bg which is designed to offer an enhanced forwarding

mechanism between peer devices via a single upstream device

Keep the ASIC counts for high performance but put them on the Central controlling switch instead of all these line cards– Latency and bandwidth were more a function of the layers of ASICs to traverse in a tree – rather than the

location of these ASICs (the fiber/copper paths for a packet to propagate)

Add protocols for configuration and firmware management of these remote cards (Satellite Control Protocol, Satellite Discovery Protocol)– Allows us to get away from manual firmware code management per (remote) line-card

Move from Store-and-Forward behavior to Cut-Through switching to make latency actually better

11


Fabric Extension (FEX) ConceptVirtualising the Network Port

LAN LANSwitch port extended over Fabric Extender

Lo

gic

al S

wit

ch

Switch

Switch

Multi-tier architecture FEX architecture

Switch

FEX

Collapse network tiers, fewer network management points


FEX Technology for Unified I/O

Virtual Switch Ports, Cables, and NIC Ports

Mapping of Ethernet and FC Wires over Ethernet

Service Level enforcement Multiple data types (jumbo, lossless,

FC)

Individual link-states Fewer Cables

Multiple Ethernet traffic co-exist on same cable

Fewer adapters needed Overall less power Interoperates with existing Models

Management remains constant for system admins and LAN/SAN admins

Possible to take these links further upstream for aggregation

Individual Ethernets

DCB Ethernet

Individual Storage (iSCSI, NFS, FC)

Blade ManagementChannels (KVM, USB,CDROM, Adapters)

13


Key Architectural Component #2 : UCS VIC

256 PCIe devices

Devices can be vNICs or vHBAs

Each device has a corresponding switch interface

Bandwidth 2x4x10 Gb

Uses 4x10 Ether Channel, HW 40Gb Capable

vNICs/vHBAs NOT limited to 10Gb

PCIe Gen-2 x 16

Mezzanine and PCIe

256 PCIe devices

vFC

vHBAs

vNIC

vNIC

vNIC

vEth

vEth

vEth

Dual 4x10Gb

VM-FEX Introduction

15


1. When VMs move across physical ports—the network policy must follow Live Migration

2. Must view or apply network/security policy to locally switched traffic

3. Need to maintain segregation of duties while ensuring non-disruptive operations

PortProfile

Hypervisor

Hypervisor

Server Admin

Network Admin

SecurityAdmin

Server Virtualization Issues

16


Cisco Virtual Networking Options

Extend networking into hypervisor(Cisco Nexus 1000V Switch)

Cisco UCS VM-FEX

Server

Extend physical network to VMs(Cisco UCS VM-FEX)

Hypervisor Hypervisor

Cisco Nexus 1000V

Generic Adapter

Server

17


UCS VM-FEX Distributed Modular System Removing the Virtual Switching Infrastructure to a FEX

=

Distributed Modular System

VM-FEX: Single Virtual-Physical Access Layer Collapse virtual and physical switching into a single access layer VM-FEX is a Virtual Line Card to the parent switch Parent switch maintains all management & configuration Virtual and Physical traffic treated the same

LAN

N7000/C6500

MDS

SAN

Access LayerUCS 6100

1 160

UCS VIC UCS VIC

App

OS

App

OS

App

OSApp

OS

App

OS

App

OS

App

OS

App

OS

App

OSApp

OS

App

OS

App

OS

UCS IOM UCS IOM

+

UCS Fabric Interconnect Parent Switch

Cisco UCS VIC

UCS IOM-FEX

+ Dist r ibu ted Modu lar System

. . .


Extending FEX Architecture to the VMsCascading of Fabric Extenders

Lo

gic

al S

wit

ch

Virtualized Deployment

Switch

FEX

Hypervisor

vSwitch

App

OS

App

OS

App

OS

LAN

Lo

gic

al S

wit

ch

VM-FEX architecture

Switch

FEX

Hypervisor

LAN

App

OS

App

OS

App

OS

VM-FEX

Switch port extendedover cascaded Fabric Extenders to the Virtual Machine

Lo

gic

al S

wit

ch


Nexus 5000/2000 VM-FEX Distributed Modular System Removing the Virtual Switching Infrastructure to a FEX

=


VM-FEX: Single Virtual-Physical Access Layer Collapse virtual and physical switching into a single access layer VM-FEX is a Virtual Line Card to the parent switch Parent switch maintains all management & configuration Virtual and Physical traffic treated the same

LAN

N7000/C6500

MDS

SAN

Access LayerNexus 5500

1 160

UCS VIC UCS VIC

App

OS

App

OS

App

OSApp

OS

App

OS

App

OS

App

OS

App

OS

App

OSApp

OS

App

OS

App

OS

Nexus 2000 Nexus 2000

+

Nexus 5500 Parent Switch

Cisco UCS VIC

Nexus 2000 FEX

+ Dist r ibu ted Modu lar System

. . .


Nexus 5000 + Fabric Extender Single Access Layer

=


+

Nexus 5000 Parent Switch

Cisco Nexus® 2000 FEX

Over 6000 production customers Over 5 million Nexus 2000 ports deployed

Distributed Modular System Nexus 2000 FEX is a Virtual Line Card to the Nexus 5000 Nexus 5000 maintains all management & configuration No Spanning Tree between FEX & Nexus 5000

LAN

N7000/C6500

MDS

SAN

Access LayerN5000

1 12

N2232

Dist r ibu ted Modu lar System

N2232. . .


- Management complexity: each VEPA is an independent point of management

- Doesn’t support cascading Reflective Relay (used in basic VEPA)

- Vulnerable: ACLs based on source MAC (can be spoofed)

- Resource intensive: Hypervisor component consumes CPU cycles

- Inefficient bandwidth : separate copy of each Mcast and Bcast packets on the wire

- Ease of management: one switch manages all Port Extenders (adapters/switches/virtual interfaces)

- Supports cascading of Port Extenders (multi-tier, single point of management)

- Virtual Machine aware FEX

- Secure: ACLs based on VN-TAG

- Scalable: Mcast and Bcast replication performed in HW at line rate

- Efficient: no impact to server CPU

IEEE-802.1BR vs. IEEE802.1Qbg

VEPA based on IEEE 802.1QbgFEX based on IEEE 802.1BRSwitch

FEX

Log

ica

l Sw

itch

VM- FEX


Deployments of Cisco’s FEX Technology

Nexus 5000/5500/7000 + Nexus 2200

UCS 6100/6200 + IOM 2kB22H with Nexus 5500 (HP)

Server Rack

Rack Server

Rack Server

Rack Server

Rack Server

Rack Server

Rack Server

Rack Server

Rack Server

Rack Server

Rack Server

Rack Server

Rack Server

Rack FEX

FEX

Switch

FE

X

Chassis FEX

Switch / FI

Blade Server

Blade Server

Blade Server

Blade Server

Blade Server

Blade Server Chassis

Blade Server

Blade Server

Blade Server

1 2

UCS 6100/6200 + VIC 1 or 2Nexus 5500 + VIC P81E

Blade/Rack Server Adapter

OS

FEX

Adapter FEX

Switch / FI

3

1 2 3 4 n

1

Port 0

UCS 6100/6200 + VIC 1 or 2 + VM Mgmt LinkNexus 5500 + VIC P81E

vCenter/VMM

Man

agem

ent

Plan

e Integ

ration

UCS Manager

VM Host

Hypervisor

VM

VM

FEX

VM-FEX

VM

Switch / FI

RedHat KVM

4

1 2 n

23


VM-FEX Operations ModelPre-Boot Configuration

Hyp

ervi

sor

Hyp

ervi

sor

Step 1: Preboot– UCS defined PCIe devices and enumerations– Host discovers PCIe devices


VM-FEX Operational ModelDefining “Port Profiles” on the UCS or Nexus 5000

Step 1: PrebootUCS defined PCIe devices and enumerations

Host discovers PCIe devices

Step 2: Port ProfileFolder of Network Policy defined

Hyp

ervi

sor

Hyp

ervi

sor

Port ProfilesDefinition

WEB Apps

HR

DB

Compliance

VLAN WebVLAN HR

VLAN DBVLAN Comp

UCSM or Nexus 5500


VM-FEX Operational ModelPushing Port Profiles to the Hypervisor System



Step 2: Port ProfileFolder of Network Policy on UCS or Nexus 5500 defined

Step 3: Port Profile ExportPort Profile name list exported to virtualization manager

Hyp

ervi

sor

Hyp

ervi

sor

VLAN WebVLAN HR

VLAN DBVLAN Comp

Hypervisor Manager

UCSM or Nexus 5500 exports Port Profiles

UCSM or Nexus 5500


VM-FEX Operational ModelMapping of Port Profiles to VM Virtual Adapters



Step 2: Port ProfileFolder of Network Policy on UCS or Nexus 5500 defined

Step 3: Port Profile ExportPort Profile name list exported to virtualization manager

Step 4: VM DefinitionNamed Policy in VM

Hyp

ervi

sor

Hyp

ervi

sor

VLAN WebVLAN HR

VLAN DBVLAN Comp

Hypervisor Manager

Network Manager

VM

VMVMVM


VM-FEX Operational ModelSimplifying the Access Infrastructure

Unify the virtual and physical networkSame Port Profiles for various hypervisors and bare metal servers

Consistent functions, performance, management

Physical Network

Virtual Network

Hyp

ervi

sor

Hyp

ervi

sor

VMVMVMVMVMVMVMVM

VETH

VNIC


VM-FEX Operational ModelTraffic Forwarding

Removing performance dependencies from VM location

Offloading software switching functionalities from host CPU

More on this in upcoming slides

Physical Network

Hyp

ervi

sor

Hyp

ervi

sor

VMVMVMVMVMVMVMVM

VETH

VNIC

VM-FEX Operational Model


VM-FEX Modes of OperationVMware ESX

vSphere 5

Emulated Mode

vEth

vEth

dvNIC

dvNIC dvNIC

dvNIC

VMDirectPath

vSphere 5

vEth

vEth

Emulated Mode Each VM gets a dedicated PCIe

device Appears as distributed virtual switch

to hypervisor LiveMigration supported

High Performance Mode Co-exists with Standard mode

Bypasses Hypervisor layer

~30% improvement in I/O performance

Appears as distributed virtual switch to hypervisor

Currently supported with ESXi 5.0 +

LiveMigration supported


Dynamic VIC device

Config:Used by

PCI mgmt layer

Mgmt Bar:

Used by Vmkerne

l and PTS

Data Bar :

Vmxnet3 compliant Rings

and Register

s

Vmxnet3

OS PCI subsystem

Control Path

Emulation <-> PT transitions

Port Events

PCIe events

VMVmxnet3 Driver

Ethernet Device Driver

Cisco DVS Data Path

VMDirectPath: How is Works


UCS VM-FEX Modes of OperationWindows Hyper-V & Red Hat KVM with SR-IOV

33

Emulated Mode Hypervisor Bypass

Emulated Mode Each VM gets a dedicated PCIe

device Appears as a Virtual Function to

Guest OS LiveMigration supported

High Performance Mode Co-exists with Standard mode

Bypasses Hypervisor layer

~30% improvement in I/O performance

Appears as a Virtual Function to Guest OS

Currently supported through SR-IOV with Hyper-V 2012 & RHEL KVM 6.3

Live Migration supported

Hyper-V 2012

vEth

vEth

dvNIC

dvNIC

VF VF

PF

SvNIC

vEth

dvNIC

dvNIC

Hyper-V 2012

vEth

vEth

VF VF

SvNIC

vEth

PF


SR-IOV: How is Works

Hyper-V HostRoot Partition

Hyper-V Switch

Physical NIC

Virtual Machine

Virtual NICSwitchingVLAN Filtering

Data Copy VMBUS


Hyper-V Switch

SR-IOV Physical Function

Virtual MachineVirtual Function

SwitchingVLAN Filtering

Data Copy

Network I/O path without SRIOV Network I/O path with SRIOV

34


VM-FEX Operational ModelLive Migration with Hypervisor Bypass

vNIC

vNIC

vEthvEth vNIC

vSphere 4

vEth

Temporary transition from to standard I/O

• VM Sending TCP stream (1500MTU) • UCS B200 M2 blades with UCS VIC card

19:0

6:19

19:0

6:21

19:0

6:23

19:0

6:25

19:0

6:27

19:0

6:29

19:0

6:31

19:0

6:33

19:0

6:35

19:0

6:37

19:0

6:39

19:0

6:41

19:0

6:43

19:0

6:45

19:0

6:47

19:0

6:49

19:0

6:52

0

2500

5000

7500

10000

Time (secs)

Mb

ps

Live Migration to secondary host

1 sec silent period

Hypervisor

Hypervisor


VM-FEX Modes of OperationEnumeration vs. Hypervisor Bypass

VM-FEX Mode VMware Hyper-V KVM

Emulation Pass Through (PTS) Hyper-V Switch SR-IOV with MacVTap

Hypervisor Version ESX 4.0 U1 + Window Server 2012 RHLE 6.2 +

UCSM Version 1.4 + 2.1 2.1

VMotion / Live Migration Support Support Support

VM-FEX Mode VMware Hyper-V KVM

Hypervisor Bypass Hypervisor Bypass SR-IOV SR-IOV with PCI Passthrough

Hypervisor Version ESX 5.0 + Window Server 2012 RHEL 6.3

UCSM Version 1.4 + 2.1 2.1

VMotion / Live Migration Support Support N/A

3 4

1 3 4 5 6 7 8Chassis IO Module A

1 2

Server Ports

3 4

1 3 4 5 6 7 8

1 2

Server Ports

VN Tag @ 10Gbe

22Chassis IO Module B

Internal Connections

vfc0

2 3 4 5 6

0 1

vNIC1(s) vNIC2(s)

vfc1

2 3 4 5 6

VM

-FE

X

UCS 6x00 Physical Ports

Chassis IOM Ports


Chassis IOM Ports

VIC CPU

Virtual Interface Control Logic


HBA 0vHBA0

HBA 1vHBA1

veth10

1 1

87 21Fiber Channel Uplink Ports

8721Fiber Channel Uplink Ports

6565

veth1 veth2 veth3 veth4 veth1 veth2 veth3 veth4 veth100

Mgmt Uplink

0

Mgmt Uplink

CIMCKVM etc.

Cisco Adapter

UCS B or C Series Server

UCS Fabric Interconnect B (port profiles)UCS Fabric Interconnect A (port profiles)

ESX Kernel Module / Libvirt / HyperV Extendable Switch

UCS VM-FEX System ViewDeploying on a UCS B or C Series Infrastructure


UCS VM-FEX System ViewDeploying on a UCS B or C Series Infrastructure

3 4

1 3 4 5 6 7 8Chassis IO Module A

1 2

Server Ports

3 4

1 3 4 5 6 7 8

1 2

Server Ports

VN Tag @ 10Gbe

22Chassis IO Module B

ServiceConsoleKernel


vfc0

2 3 4 5 6

0 1

d-vNIC1vNIC1(s) d-vNIC2 vNIC2(s)d-vNIC3

vfc1

2 3 4 5 6

VM

-FE

X


Chassis IOM Ports


Chassis IOM Ports

VIC CPU



ESX Kernel Module / Libvirt / HyperV Extendable SwitchHBA 0vHBA0

HBA 1vHBA1d-vNIC4

veth10

1 1



6565


Mgmt Uplink

0

Mgmt Uplink

CIMCKVM etc.

UCS Fabric Interconnect B (port profiles)UCS Fabric Interconnect A (port profiles)

Cisco Adapter

UCS B or C Series Server


VM-FEX Scalability Number of VIF Supported per Hypervisor

39

Cisco UCS 6100 / 6200 Series

Hypervisor Half-Width Blade with Single VIC Full-Width Blade with Dual VIC

ESX 4.0 – 4.1 (DirectPath I/O) 56 (54 vNIC + 2 vHBA) 56 (54 vNIC + 2 vHBA)

ESXi 5.0 – 5.1 (DirectPath I/O) 116 (114 vNIC + 2 vHBA) 116 (114 vNIC + 2 vHBA)*

Windows 2012 (SR-IOV) 116 (114 vNIC + 2 vHBA) 232 (228 vNIC + 4 vHBA)

KVM 6.1 – 6.3 (SR-IOV) 116 (114 vNIC + 2 vHBA) 232 (228 vNIC + 4 vHBA)

* Additional VIC will NOT increase the total VIF count due to OS limitation

* Multiple VIC is Supported for full width blade and B200M3

Nexus 5500 Series

Hypervisor Adapter FEX VM-FEX

ESX 4.1 – ESXi 5.1 96 vNIC 96 vNIC

* Only one VIC (P81E/VIC1225) is Supported for each C series rack server


VM-FEX Advantage

40

Simpler Deployments– Unifying the virtual and physical network– Consistency in functionality, performance and management

Robustness – Programmability of the infrastructure – Troubleshooting, traffic engineering virtual and physical together

Performance – Near bare metal I/O performance – Improve jitter, latency, throughput and CPU utilization

Security – Centralized security policy enforcement at controller bridge – Visibility down to VM to VM traffic

VM-FEX General Baseline on UCS


UCS General Baseline #1: Dynamic vNICs Policy Setting a Dynamic Adapter Policy Up

Policies are to automatically provision dynamics on Servers

Dependent on the number of Fabric Interconnect to IO Module connections(# IOM to FI links * 63) - 2 for Gen 2 Hardware (62xx, 22xx and VIC12xx) (# IOM to FI links * 15) - 2 for Gen 1 Hardware (61xx, 21xx and Palo)


UCS General Baseline #2: Building Service ProfileAdding the Dynamic Policy and Static Adapters

2 Statics – 1 to each UCS Fabric

Change dynamic vNIC connection policy to setup dynamics


UCS General Baseline #2: Building Service ProfileStatic and Dynamic Adapter Policy

Adapter Policy Static vNIC Dynamic vNIC

VMware ESXi VMware VMwarePassThru

Window Hyper-V SR-IOV Windows

RedHat KVM SR-IOV Linux


UCS General Baseline #3: Building Port ProfilesCreating Folders of Network Access Attributes

Creating Port Profiles Includes:VLAN(s)Native and/or Tagging allowedQoS Weights and Flow RatesUpstream Ports to always use


UCS General Baseline #4: Building Port ProfilesEnhanced Options like VMDirectPath with VM-FEX

Selecting High Performance will only Impact VMware deployment today

No problem if selected and used on other hypervisors


UCS General Baseline #5: Communication with ManagerEstablishing Communication to Hypervisor Manager

Tool discussed later to simplify the whole integration process


UCS General Baseline #5: Communication with Manager

6100 6200

Hypervisor Manager DVS Hypervisor Manager DVS

ESXi 1 vCenter per UCS doman

1 DVS per vCenter

1 vCenter per UCS doman

4 DVS per vCenter

Hyper-V 1 MMC Instance 5 DVS per MMC Instance

1 MMC Instance 5 DVS per MMC Instance

KVM N/A 1 DVS per UCS Domain

N/A 1 DVS per UCS Domain

Port Profile per UCS Domain

512 512

Dynamic Ports per port profile

4096 4096

Dynamic Ports per DVS

4096 4096


UCS General Baseline #6: Publishing Port ProfilesExporting Port Profiles to these to Hypervisor Manager

Publish Port Profiles to Hypervisors and virtual switches within

VM-FEX Implementation with VMware on UCS

51


VMware VM-FEX: Infrastructure RequirementsVersions, Licenses, etc.

VMware VM-FEX is available B series, integrated and standalone C series

Each VIC card supports up to 116 Virtual Machine Interface (OS limitation)

Enterprise Plus License required (as is for any DVS) on Host

Standard License and above is required for vCenter

Hypervisor features are supported for both emulated and hypervisor bypass modevMotion, HA, DRS, Snapshot and Hot add/remove virtual device, Suspend/Resume

VMDirectPath (Hypervisor Bypass) with VM-FEX is supported with ESXi 5.0+

VM-FEX upgrade is supported from ESXi4.x to ESXi5.x with Customized ISO and VMware Update Manger


VM-FEX and VMware SR-IOV Comparison

53

VM-FEX is the hypervisor bypass solution with vMotion capability – VMware SR-IOV is incompatible with hypervisor features including vMotion, HA, DRS …

VM-FEX has the highest Virtual Machine interface per host – With UCSM 2.1 release, each ESXi host support up to 116 VM interface

– With ESX5.1, SR-IOV supports up to 64 VF with Emulex and 41VF with Intel adapter

VM-FEX is available on both UCS blade and rack severs – Blade Server, Integrated rack server with UCSM and Standalone rack server with Nexus 5500

– With ESX5.1, SR-IOV is only available on PCIe adapter and standalone rack server

VM-FEX enable centralized network management and policy enforcement – Network policy is configured as port profile in UCSM / N5K and push to vCenter as network label

– Clean separation between Network and Server responsibility

VM-FEX Configuration is fully automated – Easy VM-FEX tool VM-FEX provides inter VMs traffic visibility through network tool


VMware VM-FEX Configuration Workflow1. Configure Service Profile and adapter

VMW ESX

Server

vCenter

VMW ESX

Server

UCS DVS (PTS)

VM #1

VM #4

VM #3

VM #2

VM #5

VM #8

VM #7

VM #6

UCS exports Port Profiles to VC

UCS

Serveradministrator


2. Creating Port Profile and Cluster in UCSM

4. Install Cisco VEM software

bundle and plugin in vCenter

3. Configure USCM VMware Integration wizard

5. Add ESX host into DVS cluster in vCenter

6. Configure Virtual Machine setting to enable hypervisor bypass 7. Verify VM-FEX configuration

in both UCSM and vCenter


VMware VM-FEX Demo Topology

Server

Cisco UCSVM-FEX

VMware ESX 5.1

Server

Cisco UCSVM-FEX

Vware ESX 5.1

Cisco UCS Fabric Interconnect

Virtual Ethernet ports (vEth)


DirectPath I/O Active VMXNET 3 Adapter

VM-FEX NTTTCP Sender

vSwitch NTTTCP Sender

VM-FEX NTTTCP Receiver

vSwitch NTTTCP Receiver

55

VMware VM-FEX Demo

56


VMware VM-FEX Best Practice

57

Pre-provision the number of dynamic vNIC for future usage – Changing the quantity and the adapter policy require server reboot

Select “High Performance Mode” in Port Profile to enable hypervisor bypass

Utilize ESX Native VMXNET3 Driver – User configurable parameter including queue, interrupt, ring size through policy – Recommend to have Num (vCPU) = Num (TQ) = Num(RQ) to enable DirectPath I/O

Other consideration to deploy VM-FEX– ESX heap memory size : MTU size – ESX available interrupt vectors : Guest OS and adapter policy – Dedicated spreadsheet for VM-FEX calculation and sizing


Easy VM-FEX Tool

VM-FEX Implementation with Nexus 5K

59

UCS VM-FEX System View

8

1 3 4 5 6 322232 FEX A

1 2

2232 Server Ports

8

1 3 4 5 6 32

1 2

2232 Server Ports

VN Tag @ 10Gbe

222232 FEX B


vfc0

2 3 4 5 6

0 1

vNIC1(s) vNIC2(s)

vfc1

2 3 4 5 6

VM

-FE

X

Nexus 55xx Physical Ports

2232 Fabric Ports


2232 Fabric Ports

VIC CPU



ESX Kernel Pass Through ModuleHBA 0vHBA0

HBA 1vHBA1

veth10

1 1




Mgmt Uplink

0

Mgmt Uplink

CIMCKVM etc.

Cisco Adapter

UCS C Series Server

47 47

48 48

vPC Connections

Nexus 55xx A (port profiles) Nexus 55xx B (port profiles)

Nexus VM-FEX System ViewDeploying on a UCS C Series with Nexus 5500 Infrastructure


UCS VM-FEX System View

8

1 3 4 5 6 322232 FEX A

1 2

2232 Server Ports

8

1 3 4 5 6 32

1 2

2232 Server Ports

VN Tag @ 10Gbe

222232 FEX B


vfc0

2 3 4 5 6

0 1

vNIC1(s) vNIC2(s)

vfc1

2 3 4 5 6

VM

-FE

X


2232 Fabric Ports


2232 Fabric Ports

VIC CPU



HBA 0vHBA0

HBA 1vHBA1

veth10

1 1




Mgmt Uplink

0

Mgmt Uplink

CIMCKVM etc.

Cisco Adapter

UCS C Series Server

47 47

48 48

ServiceConsoleKernel

d-vNIC1 d-vNIC2 d-vNIC3 d-vNIC4

6565

Nexus 55xx A (port profiles) Nexus 55xx B (port profiles)

ESX Kernel Pass Through Module

vPC Connections (veth’s not a vPC at FCS)



Nexus 5548

Fabric Extenders

Port 1 Port 2

Nexus 5500 VM-FEX Demo Topology

Nexus 5548/C22 M3/ VIC1225/ESXi 5.1

Nexus 5548-A is pre-configured, focus on 5548-B with the same configuration in the demo

Uplink Redundancy – Each static vNIC configure as Active/Standby

• No need for OS teaming software

• Required for hypervisor uplink

Each dynamic vNIC attach to uplink in Round Robin fashion

vPC Doman and Peer Link is configured to synchronize veth numbering for the same VM

– Not used for the forwarding plane

ESXi 5.1

vNIC 1CH1

vNIC 2CH2dNIC 1 dNIC 2 dNIC 3

VM 1vNIC 0

VM 2vNIC 0

VM 3vNIC 0

C22 M3

VIC 1225

vPC Peer Link

vEth vEth vEth vEth vEth vEth

62


Nexus 5500Nexus 5500

0. Set Up Connection2. Install N5K Feature License 3. Configure Static Binding interface and enable VNTag on host interfaces4. Configure Port Profile 5. Configure DVS & Extension 10. Verify VM-FEX status

6. Download Extension and register plugin 8. Add Sever into DVS cluster 9. VM created and attach port profile

Cisco Adapter

1. VM vNICs provisioned and VNTag Mode enable 7. Install VEM on ESXi host

Nexus 5500 VM-FEX Configuration Workflow

UCS C-series CIMC


Serveradministrator

Serveradministrator

63

Nexus 5K VM-FEX Demo

64


Nexus 5500 VM-FEX Best Practice

65

VM-FEX supports single N5k topology, vPC Topology is recommended In vPC Topology, ensure both N5k have the same port-profile configuration In vPC Topology, need to configure the same SVS connection on both N5k but

only Primary switch has active connection to vCenter – When secondary switch takeover primary role, seamlessly activate the connection to vCenter

Enable “vethernet auto-create” feature – Automatically create vEth port for dynamic vNIC during server boot up

– Auto created vEth are numbered > = 32769

DirectPath I/O is active with “high-performance host-netio”

cmd in port profile

VM-FEX Implementation with Hyper-V on UCS

66


Hyper-V Scale ComparisonVmware vSphere 5 .1 Windows Server 2008 R2 Windows Server 2012

HW Logical Processor Support

160 LPs 64 LPs 320 LPs

Physical Memory Support 2 TB 1 TB 4 TB

Cluster Scale 32 Nodes up to 4000VMs 16 Nodes up to 1000 VMs 64 Nodes up to 4000 VMs

Virtual Machine Processor Support

Up to 64 VPs Up to 4 VPs Up to 64 VPs

VM Memory Up to 1 TB Up to 64 GB Up to 1 TB

Live Migration Concurrent vMotion 128 per datastore

Yes, one at a time Yes, with no limits. As many as hardware will allow.

Live Storage Migration Concurrent Storage vMotion 8 per datastore, 2 per host

No. Quick Storage Migration via SCVMM

Yes, with no limits. As many as hardware will allow.

Servers in a Cluster 32 16 64

VP:LP Ratio 8:1 8:1 for Server12:1 for Client (VDI)

No limits. As many as hardware will allow.

67


SR-IOV Overview


Hyper-V Switch

Physical NIC

Virtual Machine

Virtual NICSwitchingVLAN Filtering

Data Copy VMBUS


Hyper-V Switch

SR-IOV Physical Function

Virtual MachineVirtual Function

SwitchingVLAN Filtering

Data Copy

Network I/O path without SRIOV Network I/O path with SRIOV

68


Hyper-V Extensible Switch Architecture

Root Partition

Extension Miniport

Extension ProtocolHyper-V Switch

Physical NIC

Virtual Machine

Host NIC VM NIC

Virtual Machine

VM NIC

Filtering Extensions

Forwarding ExtensionWFP Extensions

Capture Extensions

VM-FEX Forwarding Extension

VF Driver VF Driver

PF Driver

Hyper-V extensible switch architecture is an open API model that enhance vSwtich feature

Three types of extension is defined by Hyper-V

– Capture Extension

– Filtering Extension (Window Filtering Platform)

– Forwarding Extension (VM-FEX)

Multiple extension is allowed

– Still need to verify with Vendor for compatibility

– Several extension is incompatible with WFP

Extension state is unique for each vSwitch

– Leverage SCVMM to centrally configure extension

Cisco also provides both PF and VF Drivers

69


PF Driver

SCVMM Management of Switch Extensions

UCS

VMM Service

SCVMM

UCS Manager API

Network and Security Policy

database

UCSM Provider Plugin

Virtualization

VM1 VM3VM2

Root Partition

VMM Agent

Capture ExtensionFiltering

ExtensionForwarding Extension

Physical NIC

SCVMM Server

VM-FEX Forwarding Extension

VF Driver

UCS VIC

Service Profile

70


Hyper-V VM-FEX: Infrastructure Requirements

VM-FEX is available for Hyper-V on UCS B series and Integrated C series– Standalone C series support is on the road map

Each VIC card supports up to 116 Virtual Machine Interface – Install additional VIC will double VM interface (B200M3 with 2 VIC -> 232 per host)

Windows Server 2012 is required for both Host and Guest OS (same level of Kernel) – Do Not Support Windows Server Core and Hyper-V standalone server

VM-FEX with Live Migration fully supported– Various options for share storage – Failover Cluster, SMB, share nothing storage

Full PowerShell library support for automation– PowerShell Commandlet for UCSM, Hyper-V and SCVMM


Hyper-V VM-FEX: Infrastructure Requirements Support Two Management Approaches

– Microsoft Management Console (MMS) for Standalone Hyper-V deployment – System Center Virtual Machine Manager (SCVMM SP1) for Integrated Hyper-V deployment

UCS Manager full Integration with Systems Center Virtual Machine Manager (SCVMM) 2012 SP1– Expect to release with UCSM 2.2 – UCS Provider Plugin includes VM-FEX switch forwarding extension– SCVMM use UCSM Provide Plugin to pull information from UCSM

Cisco provides both VIC drivers and VM-FEX switch forwarding extension– VIC Utility Tool is provided (MSI) – The same Windows Driver for both Physical Function (Host) and

Virtual Function (Guest)


Hyper-V VM-FEX: SCVMM Network Definition

73

L o g i c a l S w i t c h ( D V S )

FND: PUBLIC-SJC

VMN: WEB

VM1

VM2

vSwitch

VM4

VM5

vSwitch

WEB,Silver--VPP

HOST GROUP: SJ HOST GROUP: NYC

Uplink PP-SJC

Uplink PP-NYC

Gold-VPP Silver-PP Bronze-PP

VMND: WEB, VLAN: 55

VMND: WEB, VLAN: 155

FND: PUBLIC-NYC

FN: PUBLIC

WEBGold-VPP

Fabric Network (FN) – A network abstraction representing a logical network composed of network segments (VLANs) spanning across multiple sites

Fabric Network Definition (FND) – A network abstraction composed of site-specific network segments

VM Network Definition (VMND) – A sub-network abstraction composed of a single network segment (and an IP pool) at a specific site

VM Network (VMN) – A sub-network abstraction composed of network segments spanning across multiple sites. Used by a tenant’s VM

Uplink Port-Profile (UPP) – Carries a list of allowed FNDs for a pNIC

Virtual Port Profile (VPP) – Profile defining QoS/SLA characteristics for a vNIC

Logical Switch – Microsoft’s native DVS and define Live Migration Boundary

Live Migration Boundary


Hyper-V VM-FEX Configuration Workflow

Server Admin

Network Admin

UCS Provider

Plugin

Hyper-V Host

UCS VM-FEX Forwarding Extension

Fabric Interconnect

UCS Manager

NX-OS

Dynamic VM-FEX Veth link provisioning

Pull Fabric, VM Network and Port Profile Definitions

SCVMM

1. Configure Service Profile

2. Setup SCVMM and Create Port Profile

3. Install UCSM Provider Plugin

4. Configure SCVMM Switch Extension Manager

5. Configure SCVMM Logical Switch

6. Associate Native VM Network to External Provided VM Network

7. Assign Hyper-V Host to Logical Switch and attach port Classification

8. Verify VM-FEX Connectivity in UCSM

74


Hyper-V VM-FEX Demo Topology

Server

Cisco UCSVM-FEX

Microsoft Hyper-V

Server

NTTTCP/Server

Cisco UCSVM-FEX

Microsoft Hyper-V

Cisco UCS Fabric Interconnect



SR-IOV enabled adapter

NTTTCP/Client

75

Hyper-V VM-FEX Demo

76


Hyper-V VM-FEX Best Practice

77

Always utilize SCVMM to configure Hyper-V and Virtual Machine property Utilize NTTTCP as performance benchmark tool in Windows Platform

– NTTTCP is Microsoft internal testing tool – Latest version available – NTTTCP v5.28 with Windows Server 2012 (April 2013)

Optimized for 10GE interface

http://gallery.technet.microsoft.com/NTttcp-Version-528-Now-f8b12769

– Enable Receive Side Scaling (RSS) Use Powershell Command - Set-VMNetworkAdapter –VMName “Server” – IovQueuePairsRequested 4 Need to shutdown VM to apply RSS

– iSCSI boot is NOT support for PF as an overlay iSCSI vNIC

http://gallery.technet.microsoft.com/NTttcp-Version-528-Now-f8b12769

VM-FEX Implementation with KVM on UCS

78


RHEL KVM VM-FEX: Infrastructure Requirements VM-FEX is available for Hyper-V on UCS B series and Integrated C series

– Standalone C series support is on the road map

Each VIC card supports up to 116 Virtual Machine Interface – Install additional VIC will double VM interface (B200M3 with 2 VIC -> 232 per host)

UCS Manager 2.1 release is required to supported SR-IOV in KVM Install Red Hat as Virtualization Host

– RHEL 6.2 for VM-FEX emulation mode (SR-IOV with MacVTap)

– RHEL 6.3 for VM-FEX hypervisor bypass mode (SR-IOV with PCI passthrough)

– MacVTap Direct (Private) mode is no longer supported with UCSM release 2.1

Live migration feature only supported in emulation mode Guest Operating System RHEL 6.3 Required to support SR-IOV with PCI passthrough

– RHEL 6.3 inbox driver supports SR-IOV with PCI passthrough

Scripted nature of configuration at FCS– No current RHEV-M for RHEL KVM 6.x

Virtual Machine interface management via editing of VM domain XML file


VM-FEX with KVM Architecture

……

UCS Switch

Cis

co

VIC

Adapter Port

Switch Port

……

Use

rK

ern

el Macvtap Interface

Netdev Interface

Libvirt

Virsh

Netlink Socket

Application

virtio-net

eth1

Macvtap 1

vhost-net

KVM

Guest OS

Guest 2

Application

virtio-net

Guest OS

Guest 1

eth2

Macvtap 2

vhost-net

eth0 ethn

PF VF1 VF2 VFn

Port Profile1: Qos1, vlan1


Veth 1 Veth 2



Libvirt – Open source management tool is used for managing virtual machines provides a generic framework supports for a various virtualization

- A Virtual Machine in Libvirt is represented as a domain XML file and store under QEMU user space

- Virsh is GUI interface built on top of Libvirt API

MacvTap - Linux Macvtap driver provides a mechanism to connect a VM interface directly to a host physical device

- Libvirt uses macvtap to provide direct attach of VM NIC to host physical device

- VM-FEX bypass MacvTap Interface


VM-FEX on KVM Configuration Steps

1. Upgrade UCSM to 2.1+ firmware (Del Mar release support SR-IOV)

2. Configure Service Profile with static (PF) and dynamic (VF) adapter policy

3. Creating Port Profile and Port Profile Client (only single default DVS support)

4. Install VM OS with RHEL 6.3 to support SR-IOV

5. Modify Virtual Machine Domain XML file to enable VM-FEX function

6. Connect VM to Virtual Machine Manager (GUI interface of Virsh)

7. Verify VM-FEX configuration in both UCSM and RHEL host

81

KVM VM-FEX Demo

82


VM-FEX Customer Benefit

83

Trouble shooting & traffic engineering VM traffic holistically from the physical network

Programmability, ability to re-number VLANs without disruptive changes

Simplicity One infrastructure for virtual & physical resource provisioning,

management, monitoring and troubleshooting

Consistent features, performance and management for virtual & physical infrastructure

VMDirectPath and SR-IOV enabled near bare metal I/O performance

Line rate traffic to the virtual machine

Performance

Robustness


BRKCOM-2005Recommended Viewing

www.YouTube.com/ciscodatacenter

Playlist UCS Technical Videos http://www.youtube.com/ciscodatacenter#p/c/F04A2C6AA04DF055

Overview Cisco UCS Advantage http://www.youtube.com/watch?v=IW4zHXIjpPU

UCS Advantage Videos on YouTube

http://www.youtube.com/watch?v=IW4zHXIjpPU


Category Title URL

UCS server Service Profiles and Templates http://www.youtube.com/watch?v=JW-YtVN75R0

UCS server Organizations and Roles http://www.youtube.com/watch?v=tb-L0zv3If

UCS server Extended Memory Technology http://www.youtube.com/watch?v=kS3ehPRcVDo

UCS server Server Pre-Provisioning http://www.youtube.com/watch?v=o7BuEE3hNPE

UCS server BIOS Policies http://www.youtube.com/watch?v=Pr6EptC9JXQ

UCS server RAID Policies http://www.youtube.com/watch?v=Vcs56wjUWuI

UCS server Firmware Policies http://www.youtube.com/watch?v=vjj8Xz0NqI4

UCS server Server Pools and Qualification Policies http://www.youtube.com/watch?v=KTw7M3T-VOw

UCS server Maintenance Policies http://www.youtube.com/watch?v=QQTlm98NgTI

UCS server High Availability During Upgrades http://www.youtube.com/watch?v=57HXMGn88HA

UCS server Monitoring with BMC BPPM http://www.youtube.com/watch?v=mdoEZf7tM5E

UCS server Microsoft Hyper-V on UCS http://www.youtube.com/watch?v=G3x_YOYK-Fo


http://www.youtube.com/watch?v=JW-YtVN75R0

http://www.youtube.com/watch?v=tb-L0zv3If

http://www.youtube.com/watch?v=kS3ehPRcVDo

http://www.youtube.com/watch?v=o7BuEE3hNPE

http://www.youtube.com/watch?v=Pr6EptC9JXQ

http://www.youtube.com/watch?v=Vcs56wjUWuI

http://www.youtube.com/watch?v=vjj8Xz0NqI4

http://www.youtube.com/watch?v=KTw7M3T-VOw

http://www.youtube.com/watch?v=QQTlm98NgTI

http://www.youtube.com/watch?v=57HXMGn88HA

http://www.youtube.com/watch?v=mdoEZf7tM5E

http://www.youtube.com/watch?v=G3x_YOYK-Fo


BRKCOM-2005Recommended ViewingCategory Title URL

UCS I/O Adapter Templates http://www.youtube.com/watch?v=KpVEn3DhfOM

UCS I/O Network Interface Virtualization http://www.youtube.com/watch?v=njjbCEblxVc

UCS I/O Adapter Fabric Failover http://www.youtube.com/watch?v=tlu8RSq6T_M

UCS I/O Extend the Network to the Virtual Machine http://www.youtube.com/watch?v=Ylizxq18yxE

UCS I/O Traffic Analysis of All Servers http://www.youtube.com/watch?v=PHTdXy_8Zdg

UCS I/O Ethernet Switching Modes http://www.youtube.com/watch?v=roX8MRN66UM

UCS I/O Fibre Channel and Switch Modes http://www.youtube.com/watch?v=VSetsgOYYCo

UCS I/O FC Port Channels and Trunking http://www.youtube.com/watch?v=PpzKPguRTXc

http://www.youtube.com/watch?v=KpVEn3DhfOM

http://www.youtube.com/watch?v=njjbCEblxVc

http://www.youtube.com/watch?v=tlu8RSq6T_M

http://www.youtube.com/watch?v=Ylizxq18yxE

http://www.youtube.com/watch?v=PHTdXy_8Zdg

http://www.youtube.com/watch?v=roX8MRN66UM

http://www.youtube.com/watch?v=VSetsgOYYCo

http://www.youtube.com/watch?v=PpzKPguRTXc


Category Title URL

UCS Infrastructure

Lights-Out Management http://www.youtube.com/watch?v=QEO1d_1vTxs

UCS Infrastructure

Easy VM-FEX Deployment http://www.youtube.com/watch?v=0aAuj80cNvg

UCS Infrastructure

Server Power Grouping http://www.youtube.com/watch?v=EgoFe33YoD8

UCS Infrastructure

Blade and Rack-Mount Management http://www.youtube.com/watch?v=aOsx4YMiOho

UCS Infrastructure

Manager Platform Emulator http://www.youtube.com/watch?v=ZNNrs2e0wvk

UCS Infrastructure

Cisco Developer Network and Sandbox http://www.youtube.com/watch?v=Syhl6SAiwew


http://www.youtube.com/watch?v=QEO1d_1vTxs

http://www.youtube.com/watch?v=QEO1d_1vTxs

http://www.youtube.com/watch?v=0aAuj80cNvg

http://www.youtube.com/watch?v=EgoFe33YoD8

http://www.youtube.com/watch?v=aOsx4YMiOho

http://www.youtube.com/watch?v=ZNNrs2e0wvk

http://www.youtube.com/watch?v=Syhl6SAiwew


Maximize your Cisco Live experience with your free Cisco Live 365 account. Download session PDFs, view sessions on-demand and participate in live activities throughout the year. Click the Enter Cisco Live 365 button in your Cisco Live portal to log in.

Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily.

Receive 20 Cisco Daily Challenge points for each session evaluation you complete.

Complete your session evaluation online now through either the mobile app or internet kiosk stations.

88

virtual machine fabric extension (vm-fex) bringing the virtual machines directly on the network...

Documents

br ieee

cisco andor

br prestandard ieee

switch ieee

network brkcom

cisco public4

cisco public5

cisco public6