tech update – aci...• lessons learned – customer design & deployments • lessons learned...

Michael Petersen, CCIE #39836 Systems Engineer

September 2015

Hypervisor integration, lessons learned Tech Update – ACI

•  VMM/Hypervisor integration – VMware, Microsoft, OpenStack

•  Lessons learned – Customer Design & Deployments

•  Lessons learned – Building the fabric, preparing for transition

•  Tour of Demo lab – New tenant creation, new features, VMM integration & troubleshooting

Agenda:

September 2015

Cisco Data Center Strategy & Vision Defined by Applications. Driven by Policy. Delivered as a Service / Solution

BUSINESS OUTCOMES

Business Agility New Business Models Lower TCO

BUSINESS REQUIREMENTS

Compute Cloud Network

Policy

Policy

Policy

4 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

VMM/Hypervisor integration - Why do we need it? How does I work?


ACI – Policy Defined Networking Logical network provisioning of stateless hardware

5

Outside (Tenant VRF)

App DB Web

QoS

Filter

QoS

Service

QoS

Filter

ACI Fabric

Application Policy Infrastructure Controller

Non-Blocking Penalty Free Overlay

APIC

HYPERVISOR HYPERVISOR HYPERVISOR


Hypervisor Interaction with ACI Two modes of Operation

•  ACI Fabric as an IP-Ethernet Transport

•  Encapsulations manually allocated •  Separate Policy domains for Physical

and Virtual

VLAN 10 VLAN 10 VXLAN 10000

Non-Integrated Mode

•  ACI Fabric as a Policy Authority •  Encapsulations Normalized and

dynamically provisioned •  Integrated Policy domains across

Physical and Virtual

APP WEB DB

Integrated Mode

DB

6


vCenter DVS SCVMM

§  Relationship is formed between APIC and Virtual Machine Manager (VMM)

§  Multiple VMMs likely on a single ACI Fabric

§  Each VMM and associated Virtual hosts are grouped within APIC

§  Called VMM Domain

§  There is 1:1 relationship between a Virtual Switch and VMM Domain VMM Domain 1

Hypervisor Integration with ACI Control Channel - VMM Domains

vCenter AVS

VMM Domain 2 VMM Domain 3

7


VXLAN VNID = 5789

VXLAN VNID = 11348

NVGRE VSID = 7456

Any to Any

802.1Q VLAN 50

Normalized Encapsulation

Localized Encapsulation

IP Fabric Using VXLAN Tagging

Payload IP VXLAN VTEP

•  All traffic within the ACI Fabric is encapsulated with an extended VXLAN header •  External VLAN, VXLAN, NVGRE tags are mapped at ingress to an internal VXLAN

tag •  Forwarding is not limited to, nor constrained within, the encapsulation type or

encapsulation ‘overlay’ network •  External identifies are localized to the Leaf or Leaf port, allowing re-use and/or

translation if required

Payload

Payload

Payload

Payload

Payload

Eth IP VXLAN Outer

IP

IP NVGRE Outer IP

IP 802.1Q

Eth IP

Eth MAC

Normalization of Ingress Encapsulation

ACI Fabric – Integrated Overlay Data Path - Encapsulation Normalization

8


Hypervisor Integration with ACI VMM Domains & VLAN Encapsulation

EP

EP

EP EP

VMM Domain 1 4K EPGs

VMM Domain 2 4K EPGs

VLAN 5 VLAN 16

16M Virtual Networks

VNID 6032

§  VLAN ID only gives 4K EPGs (12 bits)

§  Scale by creating pockets of 4K EPGs

§  Map EPGs to VMM Domain based on scope of live migration

§  Place VM anywhere

§  Live migrate within VMM domain

9


Hypervisor Integration with ACI Endpoint Discovery

DVS Host

APIC

VMM

Control (vCenter API)

Data Path

§  Virtual Endpoints are discovered for reachability & policy purposes via 2 methods:

§  Control Plane Learning:

-  Out-of-Band Handshake: vCenter APIs

-  Inband Handshake: OpFlex-enabled Host (AVS, Hyper-V, etc.)

§  Data Path Learning: Distributed switch learning

§  LLDP used to resolve Virtual host ID to attached port on leaf node (non-OpFlex Hosts)

OpFlex Host

Control (OpFlex)

Data Path

10


APIC

OpFlex: AN OPEN, extensible policy protocol OPFLEX WAS DESIGNED TO OFFER:

Policies: •  Who can talk to whom

•  What about

•  Ops requirements

Abstract policies rather than device-specific configuration 1. Flexible, extensible definition of using XML / JSON 2.

Support for any device including virtual switches, physical switches, network services with strong interoperability across vendors

3.

Open, standardized API with an open source reference implementation 4.

OPFLEX PROXY

OPFLEX AGENT

OPFLEX AGENT

OPFLEX AGENT

HYPERVISOR SWITCH ADC FIREWALL

https://wiki.opendaylight.org/view/OpFlex:Opflex_Architecture 11


L/B

EPGAPP

EPG DB F/W

EPG WEB

Application Network Profile

VM VM VM

WEB PORT GROUP

APP PORT GROUP

DB PORT GROUP

Hypervisor Integration with ACI APIC

§  ACI Fabric implements policy on Virtual Networks by mapping Endpoints to EPGs

§  Endpoints in a Virtualized environment are represented as the vNICs

§  VMM applies network configuration by placement of vNICs into Port Groups or VM Networks

§  EPGs are exposed to the VMM as a 1:1 mapping to Port Groups or VM Networks

12


One L2 hop between Leaf and Hypervisor

vCenter 1.  Blade Switch sends LLDP* to

Leaf and ESX

2.  ESX & Leaf send parsed LLDP information to vCenter & APIC resp.

3.  APIC receives LLDP information from vCenter

4.  APIC identifies the leaf and ports where ESXi hosts are attached for the given DVS. APIC download policy to leaf and provision on

*Can use CDP or mix of CDP/LLDP

1 1

1 1

1 2

1 2 1 3 1 4


EPGs are Port-Groups – What does it look like?

14


VMM/Hypervisor integration - VMware vCenter integration (DVS)


VMWare Integration Three Different Options

+

Distributed Virtual Switch (DVS) vCenter + vShield Application Virtual Switch

(AVS)

•  Encapsulations: VLAN •  Installation: Native •  VM discovery: LLDP •  Software/Licenses:

vCenter with Enterprise+ License

•  Encapsulations: VLAN, VXLAN

•  Installation: Native •  VM discovery: LLDP •  Software/Licenses:

vCenter with Enterprise+ License, vShield Manager with vShield License

•  Encapsulations: VLAN, VXLAN

•  Installation: VIB through VUM or Console

•  VM discovery: OpFlex •  Software/Licenses:

vCenter with Enterprise+ License

16


ACI Hypervisor Integration – VMware DVS

ACI VMware vCenter

Tenant N/A EPG Port Group

Subnet N/A VMM Controller vCenter Datacenter VMM Domain Virtual Distributed Switch

Web

Policy

App

Policy

DB

Port Group – Web VLAN 100

Port Group – App VLAN 101

Port Group – DB VLAN 102

Virtual Distributed Switch

VMware vCenter Datacenter VMM

Domain


APIC Admin

VI/Server Admin Instantiate VMs, Assign to Port Groups

L/B

EPG APP

EPG DB

F/W

EPG WEB


Create Application Policy

Web Web Web App

HYPERVISOR HYPERVISOR

VIRTUAL DISTRIBUTED SWITCH

WEB PORT GROUP

APP PORT GROUP

DB PORT GROUP

vCenter Server / vShield

8

5

1

9 ACI Fabric

Automatically Map EPG To Port Groups

Push Policy

Create VDS 2

Cisco APIC and VMware vCenter Initial

Handshake

6

DB DB

7 Create Port Groups

ACI Hypervisor Integration – VMware DVS/vShield

APIC

3

Attach Hypervisor to VDS

4 Learn location of ESX Host through LLDP

18


ACI Hypervisor Integration – VMware DVS

Name of VMM Domain Type of vSwitch (DVS or AVS) Associated Attachable Entity Profile (AEP) VLAN Pool

vCenter Administrator Credentials

vCenter server information

19


VMM/Hypervisor integration - VMware vCenter integration (AVS)


Southbound OpFlex API

VM VM VM VM

N1KV VEM

vSphere

Hypervisor Manager

§  OpFlex Control protocol -  Control channel -  VM attach/detach, link state

notifications §  VEM extension to the fabric §  vSphere 5.0 and above §  BPDU Filter/BPDU Guard §  SPAN/ERSPAN §  Port level stats collection §  Remote Virtual Leaf Support

(future)

Application Virtual Switch (AVS) Integration Overview

21

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public TECACI--2009

Extending ACI to Existing Virtual & Physical Networks VLAN & VXLAN Extension

Layer 2

AVS

Layer 2

AVS AVS

AVS AVS

AVS

§  AVS supports OpFlex to integrate with APIC

§  Supports a Full multi-hop Layer 2 Network between Nexus 9k and AVS: Investment Protection

§  Layer 2 network is required to support OpFlex bootstrapping in this phase

OpFlex

OpFlex

22


Application Virtual Switch (AVS)

Hypervisor

VM VM EPG App

No Local Switching Mode

VM VM EPG Web

Punt to Leaf for all traffic

Hypervisor

VM VM EPG App

Local Switching Mode

VM VM EPG Web

Punt to Leaf for Inter-EPG traffic

No Local Switching Mode •  Policy enforcement in the iLeaf •  VXLAN encap only •  aka “FEX Enable Mode”

Local Switching Mode (recommended) •  Intra-EPG local switching •  Both VLAN and VXLAN encap •  aka “FEX Disable Mode”

23


APIC Admin

VI/Server Admin Instantiate VMs, Assign to Port Groups

L/B

EPG APP

EPG DB F/W

EPG WEB



Web Web Web App


Application Virtual Switch (AVS)

WEB PORT GROUP

APP PORT GROUP

DB PORT GROUP

vCenter Server

8

5

1

9 ACI Fabric

Automatically Map EPG To Port Groups

Push Policy

Create AVS VDS 2

Cisco APIC and VMware vCenter Initial

Handshake

6

DB DB

7 Create Port Groups

ACI Hypervisor Integration – AVS

APIC

3

Attach Hypervisor to VDS

4 Learn location of ESX Host through OpFlex

OpFlex Agent OpFlex Agent

24


ACI Hypervisor Integration – VMware AVS

Name of VMM Domain Type of vSwitch (DVS or AVS)

Associated Attachable Entity Profile (AEP) VXLAN Pool

vCenter Administrator Credentials

vCenter server information

Switching mode (FEX or Normal)

Multicast Pool

25


What is the ACI vCenter Plugin?

§  A VMware vCenter Web Client plugin (vSphere 5.5) for ACI

§  Empowers virtualization admins to define network connectivity independently of the networking team while sharing the same infrastructure Virtualization admin is able to configure network connectivity (subnets, port-groups) with tenant isolation

•  Focused on simplicity User does not need to understand the ACI Policy Model Follows the vCenter Web Client GUI standards No configuration of “in-depth” networking stuff – this is done through APIC by the networking expert

26


ACI-vCenter Interactions Current Model

Network Team

Virtualiza1on Team

vCenter

ACI Port Group

ESXi Hypervisors

APIC Cluster

27


ACI-vCenter Interactions vCenter Plugin Model

Network Team

Virtualiza1on Team

vCenter

ACI Plugin

ACI Port Group

ESXi Hypervisors

APIC Cluster

1

2

3

Virt. Admin

28


VMware vCenter Plugin View

29


VMware vCenter Plugin View

30


VMM/Hypervisor integration - MS SCVMM & Azure Pack integration


Microsoft Interaction with ACI Two modes of Operation

•  Policy Management: Through APIC •  Software / License: Windows Server with

HyperV, SCVMM •  VM Discovery: OpFlex •  Encapsulations: VLAN, NVGRE (Future) •  Plugin Installation: Manual

Integration with SCVMM

APIC

Integration with Azure Pack

APIC

•  Superset of SCVMM •  Policy Management: Through APIC or

through Azure Pack •  Software / License: Windows Server with

HyperV, SCVMM, Azure Pack (free) •  VM Discovery: OpFlex •  Encapsulations: VLAN, NVGRE (Future) •  Plugin Installation: Integrated

+

32


APIC Admin

SCVMM Admin Instantiate VMs, Assign to VM Networks

L/B

EPG APP

EPG DB F/W

EPG WEB



MSFT SCVMM

8

5

1

9 ACI Fabric

Automatically Map EPG To VM Networks

Push Policy

Create Virtual Switch

2

Cisco APIC and MSFT SCVMM Initial

Handshake

6

ACI Hypervisor Integration – MSFT SCVMM

APIC

3 Attach Hypervisor to Virtual Switch

4 Learn location of HyperV Host through OpFlex


OpFlex Agent

HYPERV VIRTUAL SWITCH

7 Create VM Networks

OpFlex Agent

WEB VM NETWORK

APP VM NETWORK

DB VM NETWORK

33

Web Web App App DB

34 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential ACI FABRIC

Microsoft System Center | R2 with Service Provider Foundation

Azure Pack Portal

Cisco ACI with Azure Pack Microsoft System Center/Azure Pack

Websites, Apps, Database, VMs, ACI Provider Portal Consumer

Self-Service Portal

Websites VMs SQL Service Bus

Policy Management: APIC / Azure Pack

VM Discovery: OpFlex

Encapsulation: VLAN,

Zero touch network provisioning

Service Insertion (Physical/Virtual)

ACI PROVIDER SERVICE

OpFlex Driver


Cisco ACI and Microsoft AzurePack Workflow

ACI Fabric

WAP User Network Admin

Instantiate VMs 1

2 Create/Attach to VM Networks

Push Network Profiles to the Cisco® APIC

Fabric Bring Up

2

1

Automatically Pushes Policy on Leaf Where VM Attaches

Fabric Tracks VM Start, Attach/Detach

Get VLAN Pools Allocated For Each EPG 3

WEB APP WEB APP DB

Server 1 Server 2

Windows Azure Pack Portal

Virtual Switch

3 Access Physical & Virtual Services

4 Publish Shared Services

Full Infrastructure Visibility, Telemetry 4

Policy Enforcement


Microsoft Azure Pack Integration Admin Experience

Add & Configure service providers for this deployment (APIC IP Address, Login Credentials, etc.)

Usage & Billing statistics per user and other admin functions

36


Microsoft Azure Pack Integration Tenant Experience

Services this account has access to Resources of ACI service currently created and

consumed by this tenant

Application Network Profiles are created through Azure Pack, and pushed to APIC using REST

APIs

37


APIC Admin (Basic Infrastructure)

Azure Pack Tenant

3

6

ACI Fabric

Push Network Profiles to APIC

Pull Policy on leaf where EP attaches

Indicate EP Attach to attached leaf when VM starts

1

2


ACI Azure Pack Integration

APIC

Get VLANs allocated for each EPG


7

Azure Pack \ SPF

SCVMM Plugin APIC Plugin OpFlex Agent OpFlex Agent OpFlex Agent

Instantiate VMs

5

1

4Create VM Networks

4

38

Web Web Web Web App App DB DB


VMM/Hypervisor integration - OpenStack integration


OpenStack definition!?

40


Initial Focus on Networking (Neutron)

OpenStack Components

41


ACI and OpenStack Integration

APIC Plugin for OpenStack

Group-Based Policy (optional)

Hypervisor: KVM with Open vSwitch

Zero Touch Provisioning

Service Insertion (Physical/ Virtual)

ACI FABRIC

OpenStack Controller Group-Based Policy

(optional) APIC ML2

Heat Horizon

HYPERVISOR OPEN

VSWITCH

OPFLEX AGENT OPEN

VSWITCH

OPFLEX AGENT

CLI


Tenant

Network Security Group

Security Group Rule

Network: external Router

Port Subnet

Core API L3 + External Net Extension

Sec Grp Extension

OpenStack Neutron Networking Model

43


Tenant

Bridge Domain Context (VRF)

Subject

App Profile Outside Network

Subnet

Endpoint Group

Contract

Cisco ACI Model

44


Cisco OpenStack ACI Model Neutron API Mapping

OpenStack ACI Tenant Tenant

No Equivalent Application Profile

Network EPG + Bridge Domain

Subnet Subnet

Security Group Handled by Host

Security Group Rule Handled by Host

Router L3 Context

Network:External L3 Outside

45


APIC Admin (Performs Steps 3)

OpenStack Tenant (Performs Steps 1,4) Instantiate VMs


Web Web Web Web App App 4

3

5 ACI Fabric

Automatically Push Network Profiles to APIC

Push Policy

Create Network, Subnet, Security Groups, Policy

NETWORK

ROUTING

SECURITY

1

2

DB DB


NOVA

NEUTRON

OPEN VIRTUAL SWITCH OPEN VIRTUAL SWITCH OPEN VIRTUAL SWITCH

ACI OpenStack Integration – Phase 1

APIC

46


Group-based Policy in OpenStack Approved for Juno Release

•  Messy mapping ACI to current OpenStack components –  Endpoint Groups (Ports + Security Groups) –  Contracts (Security Groups + Security Group Rules)

•  Goal : Introduce ACI model into OpenStack

•  Starting with Groups and Group based Policies

https://blueprints.launchpad.net/neutron/+spec/group-based-policy-abstraction

47


ACI OpenStack Integration – Phase 2

2

ACI Admin (manages physical

network, monitors tenant state)

L/B

EPG APP

EPG DB F/W

L/B

EPG WEB



3

5 ACI Fabric

Push Policy

OpenStack Tenant (Performs step 1,4) Instantiate VMs

Web Web Web Web App App 4

Create Application Network Profile

1

DB DB


NOVA

NEUTRON

Automatically Push Network Profiles to APIC

L/B

EPG APP

EPG DB F/W

L/B

EPG WEB


APIC


Coming Soon: OpFlex Integration for OVS

§  Local policy enforcement on each hypervisor

§  Floating IP / NAT support

§  APIC GUI integration / VMM Domain for OpenStack

§  Per host statistics

§  Service redirection

OpFlex Offers:

Hypervisor

vm4

Project 1 Project 2 Project 3

vm5 vm3

vm5 vm6

OpFlex Agent

OpFlex Proxy

V(X)LAN

OpenStack Controller Group-Based

Policy (optional) APIC ML2


Lessons learned Customer Designs & Deployments


Typical Stretched ACI fabric (1:1 L2-stretch DC)

DC-1 DC-2

WAN/Edge

ACI Fabric (3 controllers)

Compute Compute Edge/L4-7 Services Edge/L4-7 Services

Spines; 9336(Fixed), 9504 (1 SUP), Leafs 9372PX, Dedicated Leafs, POD´s


Lessons learned Building the fabric, preparing for transition

VLAN 10 VLAN 20 New Server Group APIC Cluster

Policy Automation w/ ACI

2-Stage Migration Policy automation for existing devices


Contracts are required for outside L2 connectivity

Bridge Domain: outside_vlan600

Tenant: ESXi VRF: VRF01

Node-101/eth1/1 Node-102/eth1/1

EPG: L2-OUT-EXT-BD

Contract = Allow Communication ANP: ESXi-hosts

vPC_to_UCS_a vlan-10

vPC_to_UCS_b vlan-10

EPG: ESXi-HOST-EPG 192.168.10.11 192.168.10.10

Outside


•  Functionally we are expanding the VLAN’s into ACI.

Connect Fabric to existing Network

VLAN 600 / Subnet 10.XX.XX.0/24

P P VM VM VM

ACI Fabric

EPG-ESXi-HOST = VLAN 600

Trunk (.1Q)


Tour – Demo Lab New tenant/App, EPG creation with VMM, Troubleshooting


Thank you!

tech update – aci...• lessons learned – customer design & deployments • lessons learned...

Documents