integration and interoperation ofd2zmdbbm9feqrf.cloudfront.net/2016/anz/pdf/brkaci-2001.pdf ·...

Integration and Interoperation of Existing Nexus Networks into an

ACI ArchitectureMike Herbert - Principal Engineer, Cisco

Introduction

We Are at the Cusp of a Major Shift

DIGITAL EXPERIENCESEFFICIENCY SIMPLICITY | SPEED

Adoption

Curve

IT as a Service IaaS | PaaS | SaaS | XaaS

Flexible Consumption Models

CONSOLIDATIONVIRTUALISATION

HYBRID

CLOUDS

2000 2010 2015 The Next 5+ Years

AUTOMATION

TRADITIONAL DATA

CENTRE

We are here

CLOUD DATA CENTRE

Efficiency

Application Oriented Policy = Operational Simplicity

Introducing: Application Centric Infrastructure (ACI)

Apps + Infrastructure

Physical + Virtual + ContainersOpen + Secure

On-Premises + Cloud

“Users”“Files”

ACI Fabric

Define Endpoint Groups

Any endpoints anywhere within the

fabric, virtual or physical

Enforce Ingress Fabric Rules

Hardware rules on each port, security in

depth, embedded QoS

Single Point of Orchestration

Different administrative groups

use same interface, high level

of object sharing

Application Policy Infrastructure

Controller (APIC)Create Contracts Between Endpoint

Groups

Port-level rules: drop, prioritise, push to

service chain; reusable templates

Service Graph

Single Pass Services

Security administrator defines

generic templates in APIC,

availed to contract creation

All TCP/UDP: Accept, Redirect

UDP/16384-32767: Prioritise

All Other: Drop

Policy Contract “Users → Files”

ACI is a network Fabric which provides a new communication abstraction model

“Users”“Files”

ACI Fabric

Logical Endpoint Groups by

Role

Heterogeneous clients, servers,

external clouds; fabric controls

communication

Every device is one hop away, microsecond

latency, no power or port availability

constraints, ease of scaling

Flexible Insertion

ACI Controller manages all

participating devices, change control

and audit capabilities

Unified Management and Visibility

Fabric Port Services

Hardware filtering and bridging; default

gateway; seamless service insertion,

“service farm” aggregation

Flat Hardware Accelerated

Network

Full abstraction, de-coupled from

VLANs and Dynamic Routing, low

latency, built-in QoS

Application Centric Infrastructure Fabric

Four Big Questions

8

“Is ACI a Closed System?”

“Do I need to replace all of my existing infrastructure to begin leveraging ACI?”

“Do I need to understand all my application flows to leverage ACI”

“Do I have to upgrade everything in my Data Centre”

ABSOLUTELY NOT !!!

Let’s see WHY and HOW …

ACI Policy Based Forwarding via an Integrated Overlay

ACI Fabric – An IP network with an Integrated OverlayVirtual and Physical

• Cisco’s ACI solution leverages an integrated VXLAN based overlay

• IP Network for Transport

• VXLAN based tunnel end points (VTEP)

• VTEP discovery via infrastructure routing

• Directory (Mapping) service for EID (host MAC and IP address) to VTEP lookup

PayloadIPVXLANVTEP

APIC

VTEP VTEP VTEP VTEP VTEP VTEP

vSwitc

hvSwitc

hVTEPVTEP

IP Transport

ACI FabricIETF VXLAN Group Based Policy

ACI VXLAN (VXLAN) header provides a tagging mechanism to identify properties associated with frames forwarded through an

ACI capable fabric. It is an extension of the Layer 2 LISP protocol (draft-smith-lisp-layer2-01) with the additional of policy group,

load and path metric, counter and ingress port and encapsulation information. The VXLAN header is not associated with a

specific L2 segment or L3 domain but provides a multi-function tagging mechanism used in ACI Application Defined Networking

enabled fabric.

Ethernet

HeaderPayload FCS

Outer

IP

Outer

UDPVXLAN

Outer

Ethernet

Inner

EthernetPayload

New

FCS

VXLAN Instance ID (VNID) M/LB/SPSource GroupFlags

Rsvd

8 Bytes

1 Byte

N L Rsvd I

N: The N bit is the nonce-present bitL: The L bit is the Locator-Status-Bits field enabled bitI: The I bit is the Instance ID bit, Indicates the presence of the VXLAN Network ID

(VNID) field. When set, it indicates that the VNID field is valid

IP

Header

Inner IP

Header

Flags/DR

E

ACI Leverages VXLANRFC Draft – Next IETF Meeting

• Forward based on destination IP Address for intra and inter subnet (Default Mode)

• Bridge semantics are preserved for intra subnet traffic (no TTL decrement, no MAC

header rewrite, etc.)

• Non-IP packets will be forwarded using MAC address. Fabric will learn MAC’s for non-IP

packets, IP address learning for all other packets

• Route if MAC is router-mac, otherwise bridge (standard L2/L3 behaviour)

IP Forwarding:

Forwarded using DIPi

address, HW learning of IP

address

10.1.3.11 10.6.3.210.1.3.35 10.6.3.17

MAC Forwarding:

Forwarded using DMAC

address, HW learning of

MAC address

Overlay Elements – Host Forwarding

10.1.1.10 10.1.3.1

1

10.6.3.2

Distributed Default Gateway

• ACI Fabric supports full layer 2 and layer 3 forwarding semantics, no changes required to applications or end point IP

stacks

• ACI Fabric provides optimal forwarding for layer 2 and layer 3

• Fabric provides a pervasive SVI which allows for a distributed default gateway

• Layer 2 and layer 3 traffic is directly forwarded to destination end point

• IP ARP/GARP packets are forwarded directly to target end point address contained within ARP/GARP header

(elimination of flooding)

10.1.3.35 10.1.1.1010.1.3.1

110.6.3.2

Directed ARP Forwarding

10.1.3.35

Location Independent ForwardingLayer 2 and Layer 3

10.1.3.1

1

10.6.3.2

Pervasive SVI

• Default Gateway can reside internal or external to the Fabric

• Pervasive SVI provides a distributed default gateway (anycast gateway)

• Subnet default gateway addresses are programmed in all Leaves with end points present for the specific Tenant

IP subnet

• Layer 2 and layer 3 traffic is directly forwarded to destination end point

• External Gateway is used when Fabric is configured to provide layer 2 transport only for a specific Tenant

10.1.3.35 10.1.1.1010.1.3.1

110.6.3.2

External Default Gateway

10.1.3.35

Where is my Default Gateway?

10.6.3.2

10.6.3.110.1.3.1

Overlay Elements - Mapping/Directory ProxyInline Hardware Mapping DB - 1,000,000+ hosts

10.1.3.11 fe80::462a:60ff:fef7:8e5e10.1.3.35 fe80::62c5:47ff:fe0a:5b1a

• The Forwarding Table on the Leaf Switch is divided between local (directly attached) and

global entries

• The Leaf global table is a cached portion of the full global table

• If an endpoint is not found in the local cache the packet is forwarded to the ‘default’

forwarding table in the spine switches (1,000,000+ entries in the spine forwarding table)

Local Station Table

contains addresses of

‘all’ hosts attached

directly to the iLeaf

10.1.3.11

10.1.3.35

Port 9

Leaf 3

Proxy A*

Global Station Table

contains a local cache

of the fabric endpoints

10.1.3.35 Leaf 3

10.1.3.11 Leaf 1Leaf 4

Leaf 6fe80::8e5efe80::5b1a

Proxy Station Table contains

addresses of ‘all’ hosts attached

to the fabric

Proxy Proxy Proxy Proxy

Proxy ScalingScaled based on number of Fabric NFE’s per chassis

Spine Proxy Total Host

Entries in the

Mapping DB

Network

Forwarding

Engines Per Fabric

9336 200K* 2 x NFE

9504 (6 fabrics) 300K 1

9508 (6 fabrics) 600K 2

9516 (6 fabrics) 1M+ 4

NFE

Fabric Module for Nexus 9504

NFE NFE


NFE NFE


NFE NFE

*9336 maintains a single copy of each host entry in the HW proxy DB, 950x maintains redundant copies sharded across Fabric NFE’s

Advantage of a Directory Based FabricBuilt-in Endpoint Tracker

Advantages Atomic Counters

• Detect fabric misrouting, debug & isolate application connectivity issues

• Per-application, per-EP, per-EPG real-time, comprehensive traffic counters

• Example:

• Configure atomic counters on all leafs to count packets EP1->EP2

• Any counts NOT on Leaf03 or Leaf06 highlight misrouted packets

• Drill-down to Leaf03, Leaf01 and check routing, forwarding entries

• Configure via policy in appropriate context

10.1.1.10 10.1.3.12 10.6.3.2 10.1.3.35

EP1

Leaf01 Leaf06

EP2

Troubleshooting Workflows (e.g. EP to EP)

• Webserver and Application servers are having issues.

• Used the tool and gave us logical topology and helped us in isolating the issue.

• We found issue is -LLDP neighbour is bridge and its port vlan 1 mismatches with the local port vlan Unspecified Go see all this working in the World of Solutions

Drag and Drop GUI – 11.2 Release

• SHOW RUN is back

• View the entire running configuration

• Specify filters to show discrete sections

APIC CLI

Integrating the Networks

ACI InfrastructureUnderstanding Networks and Groups

APIC

Outside

(Tenant

VRF)

App DBWeb

QoS

Filter

QoS

Service

QoS

Filter

Location for Endpoints that are ‘Inside’ the

Fabric are found via the Proxy Mapping DB

(Host Level Granularity)

Location for Endpoints that are

‘Outside’ the Fabric are found via

redistributed routes sourced from

the externally peered routers

(Network Level Granularity)

‘Outside’ EPG associated

with external network

policies (OSPF, BGP, …

peering)

Forwarding Policy for ‘inside’ EPG’s defined by associated

Bridge Domain network policies

A Tenant is a container for all

network, security,

troubleshooting and L4 – 7

service policies.

Pepsi-Tenant Coke-Tenant

Tenant

Tenant resources are isolated

from each other, allowing

management by different

administrators.


Private Network 1

Private Network 2

Private Network 1

Private Network 2

Private networks (also called

VRFs or contexts) are defined

within a tenant to allow

isolated and potentially

overlapping IP address space.

Private Networks


Private Network 1

Private Network 2

Private Network 1

Private Network 2

Within a private network, one

or more bridge domains must

be defined.

A bridge domain is a L2

forwarding construct within the

fabric, used to constrain

broadcast and multicast traffic.

Bridge Domain 1

Bridge Domain 2

Bridge Domain 3

Bridge Domain 4

Bridge Domain 1

Bridge Domain 2

Bridge Domain 3

Bridge Domain 4

Bridge Domain

EPG Definition

EP EP

EPGEPG

Application

Profile

EP EP

EPs are devices which attach to the network either virtually or physically, e.g:

• Virtual Machine

• Physical Server (running Bare Metal or Hypervisor)

• External Layer 2 device

• External Layer 3 device

• VLAN

• Subnet

• Firewall

• Load balancer

Virtual Port, Physical Ports, External L2 VLAN, External L3 subnet


Private Network 1

Private Network 2

Private Network 1

Private Network 2

EPGs exist within a single

bridge domain only – they do

not span bridge domains.

Bridge Domain 1

Bridge Domain 2

Bridge Domain 3

Bridge Domain 4

Bridge Domain 1

Bridge Domain 2

Bridge Domain 3

Bridge Domain 4

EPG

End Point Groups

EPG

EPGEPG

EPG

EPG

EPG

EPG EPG

EPG

Mapping the Configuration to the Packet

M/LB/SPFlagsFlags/DR

EVNID == BD/VRFSource Class ID == EPG

• ACI Fabric leverages an application centric policy

model

• VXLAN Source Group is used as a tag/label to

identify the specific end point for each application

function (EPG)

• Policy is enforced between an ingress or source

application tier (EPG) and an egress or destination

application tier (EPG)

• Policy can be enforced at source or destination

Coke-Tenant

Private Network 1

Private Network 2

Bridge Domain 1

Bridge Domain 2

Bridge Domain 3

Bridge Domain 4

EPG

EPG

EPG EPG

EPG

ACI is a NetworkAn Example - Connecting/Extending ACI via Layer 2

Layer 2

Layer 2

Layer 2

Extend L2 domain beyond ACI fabric - 2 options

1. Manually assign a port to a VLAN which in turn mapped to an EPG. This extend EPG beyond ACI fabric

(EPG == VLAN)

2. Create a L2 connection to outside network. Extend bridge domain beyond ACI fabric. Allow contract

between EPG inside ACI and EPG outside of ACI

Lets Look at the Links

FEX on ACI Support Matrix

9396PX 9396TX 93128TX 9372PX (Q1CY15) 9372TX 9500

N2K-C2248PQYes, with

breakout cable

N/A N/A

Yes, with breakout

cable

N/A

N/A

9500 not

planned as a

leaf for CY15

N2K-C2248TP-E

yes yes

N2K-C2248TP-1GE

N2K-C2232PP-10GE

N2K-C2232TM-E

2348UPQ (Q2CY15)Yes, with

breakout cable

Yes, with breakout

cable

For YourReference

FEX Attached via Switch Interface Profile For YourReference

VXLAN

VNID = 5789VXLAN

VNID = 11348

NVGRE

VSID = 7456

Any to Any

802.1Q

VLAN 50

Normalised

Encapsulation

Localised

Encapsulation

IP Fabric Using

VXLAN Tagging

PayloadIPVXLANVTEP

• All traffic within the ACI Fabric is encapsulated with an extended VXLAN header

• External VLAN, VXLAN, NVGRE tags are mapped at ingress to an internal VXLAN tag

• Forwarding is not limited to, nor constrained within, the encapsulation type or

encapsulation ‘overlay’ network

• External identifies are localised to the Leaf or Leaf port, allowing re-use and/or translation

if required

Payload

Payload

Payload

Payload

Payload

Eth

IPVXLAN

Outer

IP

IPNVGREOuter

IP

IP802.1Q

Eth

IP

Eth

MAC

Normalisation of Ingress

Encapsulation

ACI Fabric – Integrated OverlayData Path - Encapsulation Normalisation

Extend the EPGOption 1

VLAN 30

Layer 2

100.1.1.3 100.1.1.5

EPG

100.1.1.7100.1.1.99

• VLAN’s are localised to the leaf nodes

• The same subnet, bridge domain, EPG can be configured as a ‘different’ VLAN on each leaf

switch

• In 1HCY15 VLAN’s will be port local

100.1.1.3

BD

Existing

App

VLAN 20

Extend the EPGOption 1

Layer 2

VLAN 10

100.1.1.3 100.1.1.5

EPG

100.1.1.7100.1.1.99

• Single Policy Group (one extended EPG)

• Leverage vPC for interconnect (diagram shows a single port-channel which is an option)

• BPDU should be enabled on the interconnect ports on the ‘vPC’ domain

100.1.1.3

VLAN 30

VLAN 20

BD

Existing

App

VLAN 10 VLAN 10 VLAN 10

Assign Port to an EPG

• With VMM integration, port is assigned to EPG by APIC dynamically.

• In all other cases, such as connecting to switch, router, bare metal, port need to be assigned to EPG manually or use API

• Use “Static Binding” under EPG to assign port to EPG

• The example assigns traffic received on port eth1/32 with vlan tagging 100 to EPG VLAN 100

Assign Port to EPGVLAN Tagging Mode

• Tagged. Trunk mode

• Untagged. Access mode. Port can only be in one

EPG

• 802.1P Tag. Native VLAN.

• No Tagged and Untagged(for different port) config

for same EPG with current software

• Assign port eth1/1 with VLAN 100 tagged mode and port eth1/2 with VLAN 100 untagged mode to EPG WEB is not supported

• Use 802.1P Tag. Port eth1/1 vlan 100 tagged, eth1/2 vlan 100 902.1P Tag

• VLAN to EPG mapping is switch wide significant

C

Extend the Bridge DomainOption 2

Layer 2

100.1.1.3 100.1.1.5100.1.1.7100.1.1.99

• External EPG (policy between the L2 outside EPG and internal EPG)


• BPDU should be enabled on the interconnect ports on the ‘vPC’ domain

• L2 outside forces the same external VLAN << fewer operational errors

100.1.1.3

BD

Existing

App

EPG

InsideEPG

Outside

VLAN 30VLAN 10 VLAN 10 VLAN 10

VLAN 10

VLAN 20

L2 Outside Connection Configuration Example

• Step 1. Create L2 Outside

connection.

• Associate with BD.

• Specify VLAN ID to connect to

outside L2 network

• External Bridge Domain is a way

to specify the VLAN pool for

outside connection.

• It is NOT a Bridge Domain.


• Step 2. Specify leaf node

and interface providing

L2 outside connection


• Step 3. Create external EPG

under L2 outside connection

• Step 4. Create contract

between external EPG and

internal EPG

C

Loop Protection vPC & ACI

Layer 2

100.1.1.3 100.1.1.5100.1.1.7100.1.1.99

• External EPG (policy between the L2 outside EPG and internal EPG)


100.1.1.3

BD

Existing

App

EPG

InsideEPG

Outside

Switch to Leaf

vPC

Host vPC

vPC Behaviour – Standalone & ACI Differences

vSwitch vSwitch

No vPC Peer

Link

Required

Standard vPC ACI Based vPC

Orphan Port

‘No’ Orphan

Ports (Single

Homed Servers

‘not’ orphans)

Implicit Uplink

Tracking

Hardware Based

Recovery for server link

failures (no STP no vPC

state updates)

vPC in ACI Fabric• Traffic is both sourced and destined to the

anycast vPC VTEP address from remote

Leaves

• A hardware hash in the spine will determine

which of the two peers forwards a specific flow

downstream to the attached device (flow

hashing between the peers via spine

• In the event of a downlink failure on one of the

peers (all local member ports are down)

1. A bounce entry is created for the end

points reachable via the port channel

pointing to the peers VTEP

2. All MAC/IP to Leaf bindings for the specific

vPC are removed from the COOP

database and the spine proxy

• On failure of a peer the remaining Leaf

converts all vPC ports to non-VPC local ports

Host or Switch

VTEP VTEP

vPC Anycast

VTEP vPC Anycast

VTEP

Traffic within the Fabric is sent to

the vPC anycast address

ACI Fabric Loopback Protection

STP Loop

Detection

LLDP Loop

Detection

• Multiple Protection Mechanisms against external loops

• LLDP detects direct loopback cables between any two switches in the same fabric

• Mis-Cabling Protocol (MCP) is a new link level loopback packet that detects an external L2 forwarding loop

• MCP frame sent on all VLAN’s on all Ports

• If any switch detects MCP packet arriving on a port that originated from the same fabric the port is err-disabled

• External devices can leverage STP/BPDU

• MAC/IP move detection and learning throttling and err-disable

APIC

MCP Loop

Detection

(supported with

11.1 release)

ACI Interaction with STP

STP Root

Switch

Same L2 Outside

EPG

(e.g. VLAN 10)

• No STP running within ACI fabric

• BPDU frames are flooded between ports configured to be members of the same external L2 Outside (EPG)

• No Explicit Configuration required

• Hardware forwarding, no interaction with CPU on leaf or spine switches for standard BPDU frames

• Protects CPU against any L2 flood that is occurring externally

• External switches break any potential loop upon receiving the flooded BPDU frame fabric

• BPDU filter and BPDU guard can be enabled with interface policy

APIC

MCP Policy

• Disabled by default

• On Edge Ports combine with BPDU Guard

EP Move Based Loop Detection

• Enabled by default.

• Loop detection interval - default 60s – time frame during which the EP moves is calculated

• Loop detection multiplication factor - default 4 –the number of ep moves within the time frame defined above.

• Example : EP from port1 – port2 – port1 is considered 1 move. Number of EPs involved doesn’t matter.

• Action - default BD learn disable – This is the action taken when the EP move is detected. You can enable both the actions.

C

Managing Flooding Within the BD

Layer 2

100.1.1.3 100.1.1.5100.1.1.7100.1.1.99

• In a classical network traffic is flooded with the Bridge Domain (within the VLAN)

• You have more control in an ACI Fabric but need to understand what behaviour you want

100.1.1.3

BD

Multi

EPG

EPG

App 1EPG

Outside

VLAN 30VLAN 10 VLAN 10 VLAN 10

VLAN 10

EPG

App 2

VLAN 20

Managing Flooding Within the FabricARP Unicast

ARP Flooding Disabled

(Default)

• Disable ARP Flooding – ARP/GARP is

forwarded as a unicast packet within the

fabric based on the host forwarding DB

• On egress the ARP/GARP is forwarded as a

flooded frame (supports hosts reachable via

downstream L2 switches)

Firewall Configured as

the Default Gateway

ARP

Managing Flooding Within the FabricARP Flooding

ARP Flooding Enabled

• Enabling ARP Flooding – ARP/GARP is

flooded within the BD

• Commonly used when the default GW is

external to the Fabric

Firewall Configured as

the Default Gateway

ARP

Managing Flooding Within the FabricUnknown Unicast Proxy Lookup

Unknown Unicast

Lookup via Proxy

• Hosts (MAC, v4, v6) that are not known by a

specific ingress leaf switch are forwarded to

one of the proxies for lookup and inline

rewrite of VTEP address

• If the host is not known by any leaf in the

fabric it will be dropped at the proxy (allows

honeypot for scanning attacks)

Unknown

Unicast

Proxy

HW Proxy

Lookup

Managing Flooding Within the FabricUnknown Unicast Flooding

• Hosts (MAC, v4, v6) that are not known by a

specific ingress leaf switch are flooded to all

ports within the bridge domain

• Silent hosts can be installed as static entries

in the proxy (flooding not required for silent

hosts)

Unknown Unicast

Flooded

Unknown

Unicast

Unknown

Unicast

Flooded

Managing Flooding Within the FabricScoping Broadcasts to a Micro Segment

100.1.1.3 100.1.1.5100.1.1.7100.1.1.99 100.1.1.3

EPG

BEPG

AEPG

C100.1.1.72

Traffic Type 11.0(x) Behaviour 11.1(x) Behaviour

ARP Flood or Unicast Flood or Unicast

Unknown Unicast Flood or Leverage Proxy Lookup Flood or Leverage Proxy Lookup

Unknown IP Multicast Flood or OMF Flood or OMF

L2 MCAST, BCAST, Link Local Flood Flood within the BD, Flood within the EPG,

Disable Flooding within the BD/EPG

Managing Flooding Within the FabricMulti Destination Flooding (Supported with 11.1(x) – Q2CY15)

• Link Level Traffic is either

• Contained within the EPG

• Contained within the Bridge Domain

• Dropped

• Security Segmentation for Link Level Traffic

Link Level

BCAST

Manage

Flooding within

the BD

100.1.1.3

100.1.1.5

100.1.1.7100.1.1.99

100.1.1.72

100.1.1.52

EPG ‘A’

100.1.1.4

EPG ‘A’EPG ‘B’ EPG ‘B’

EPG ‘B’

Multi-Destination Protocol Traffic

• BGP

• IGMP

• PIM

• STP-BPDU (flooded within EPG)

• ARP/GARP (controlled by ARP Proxy)

• ND

• OSPF/OSPF (v6)

• EIGRP

• CDP

• LACP

• LLDP

• ISIS

• EPG/BD Level Broadcast Segmentation is not supported for the following network control protocols currently

• Traffic that may need to be processed by the network infrastructure (route peering from the fabric to the WAN router, port channel setup, etc.) is flooded within the Bridge Domain

Configure ACI Bridge Domain Settings

• Temporary Bridge Domain specific settings while we are using the HSRP gateways in the existing network.

• Select Forwarding to be “Custom” which allow

• Enable Flooding of L2 unknown unicast

• Enble ARP flooding

• Disable Unicast routing

Tenant “Red”

Context “Red”

Bridge Domain “10”

Subnet 10 EPG-10

Migrate Workloads

Existing Design

HSRP

Default GW

VLAN 10 / Subnet A

P PVM VM VM

APIC

EPG “10”

P PVM VM VM

APIC point of view, the policy model

VM’s will need to be connected to new Port

Group under APIC control (AVS or DVS).

Complete the MigrationChange BD settings back to normal for ACI mode

• Change BD settings back to default.• No Flooding

• Unicast Routing enabled.

Migrating Default Gateway to the ACI Fabric

Change GW MAC address. By default, All

fabric and all BD share same GW MAC

Enable Routing and ARP flooding

Extension and Connecting It’s a Network with any VLAN Anywhere

Anycast Default Gateway

10.10.10.8 10.20.20.3210.10.10.9

10.20.20.33

10.20.20.3110.10.10.6

Any IP - Anywhere

Application

Client

Subnet

10.20.20.0/24

Subnet10.10.10.0/24

Subnet

10.30.30.0/24

Subnet

10.40.40.0/24

Subnet

10.50.50.0/24

External Networks

(Outside)

Redirect to Pre-

configured FW

Redirect to Pre-

configured FW

Critical Users

(Outside)

Middle Ware

Servers

Web

ServersOracle

DB Contract

Redirect to dynamically

configured FW

NFS ContractRedirect to

dynamically

configured FW

Default Users

(Outside)

NFS

Servers

Subnet

10.20.20.0/24

Subnet10.10.10.0/24

Subnet

10.30.30.0/24

Permit TCP any

any Redirect to Pre-

configured FW

Policy can be added gradually starting with what you have today

Simple Policy During Migration - Any-to-Any Configuration

Contracts

Provided

Filter Contracts

Provided

Contracts

consumed

Filter

EPG “VLAN 10” VLAN10 Default ALL ALL Default

EPG “VLAN 20” VLAN20 Default ALL ALL


ALLVLAN 10

VLAN 20

VLAN 30

I want to have a very open configuration with VLAN10 talking to anything (1)

Create “Contract” ALL if it doesn’t exist yet

Use filter “common/default”

I want to have a very open configuration with VLAN10 talking to anything (2)

EPG VLAN 10 provides and consumes “ALL”

Extension and Connecting Dynamic Distributed ACL’s

Permit ACL is applied on

all ports between VLAN

10, 20 & 30

10.10.10.8 10.20.20.3210.10.10.9

10.20.20.33

10.20.20.3110.10.10.6

All Subnets are allowed to communicate with this policy applied

Later if I want to put an ACL between VLAN 10 and 20

ALLVLAN 10

VLAN 20

VLAN 30

Contracts Provided Filter Contracts Provided Contracts

consumed

Filter

EPG “VLAN 10” VLAN10 Default VLAN20 Port 80

EPG “VLAN 20” VLAN20 Default ALL ALL Default


Extension and Connecting Dynamic ACL’s

Dynamic ACL is applied

between all endpoints

only allowing port 80

10.10.10.8 10.20.20.3210.10.10.9

10.20.20.33

10.20.20.3110.10.10.6

Traffic is controlled between VLAN 10 & 20 to HTTP (port 80)

Fabric MigrationvPC & ACI Co-Existence

ACI Fabric Provides Anycast

Gateway, Access Control, Service

Insertion and Automation

10.10.10.8 10.20.20.32 10.10.10.9 10.20.20.33 10.20.20.1410.10.10.97

vPC Based Design

Single Mobility & Policy Domain

Anycast

Gateway with

full ACI policy

Fabric MigrationFabricPath & ACI Co-Existence

ACI Fabric Provides Anycast

Gateway, Access Control, Service

Insertion and Automation

10.10.10.8 10.20.20.32 10.10.10.9 10.20.20.33 10.20.20.1410.20.20.7 10.10.10.97

FabricPath Layer 2 Fabric


Anycast

Gateway

Integrating and Extending L4-7 Services

Automate Service Insertion Through APIC

APP DBWEBEXTERNAL

APIC Policy Model

Endpoint Group (EPG): Collection of similar End Points identifying a particular

Application Tier. Endpoint could represent VMs, VNICs , IP, DNS name etc

Application Profile: Collection of Endpoint Groups and the policies that define way

Endpoint group communicate with each other

Application profile

PolicyPolicyPolicy

ACI Service Insertion via Policy

• Automated and scalable L4-L7 service insertion

• Packet match on a redirection rule sends the packet into a services graph.

• Service Graph can be one or more service nodes pre-defined in a series.

• Service graph simplifies and scales service operations

Begin EndStage

1

FW

_A

DC

1

EPG

2

EPG

1Application

Admin

Service

Admin

ASA

5585

Netscaler

VPX

Chain

“FW_ADC 1”

Policy-based

Redirection

Stage

2

L4-L7 Service GraphsAutomation for Service Insertion and Configuration

Service Automation Through Device Package

Open Device

Package

Policy

Engine APIC provides extendable policy model through Device Package

Configuration

Model

Device Interface: REST/CLI

APIC Script Interface

Call Back Scripts

Event Engine

APIC– Policy Manager

Configuration

Model (XML File)

Call Back Script

Provider Administrator can upload a Device Package

Device Package contains XML fine defining Device Configuration

Device scripts translates APIC API callouts to device specific callouts

APIC

L4-7 Service Graph with “Unmanaged” Device11.2 Release

UI hides all other settings

related to package,

configuration parameters, and

connectivity when managed

mode is “un-checked”

Backbone

Extending ACI in to Current Data Centre’s Standard Architecture with Services

vSwitch

Services Chassis

vSwitch vSwitch

Services Chassis

Extension and Connecting Services Switch

Dynamic ACL is applied

between all endpoints

only allowing port 80

10.10.10.8 10.20.20.3210.10.10.9

10.20.20.33

10.20.20.3110.10.10.6

• Start with the picture we now understand


10.10.10.8 10.20.20.32

• Lets simplify things


10.10.10.8 10.20.20.32

• Start with just a few switches


10.10.10.8 10.20.20.32

• Let’s connect them the way we should, redundantly (the previous slides showed a simple interconnect just as an example


10.10.10.8 10.20.20.32

• Let’s use a small spine switch

• We are starting with a very small fabric


• Same picture as seen if the fabric was a services switch


• Attach the L4-7 Services

• They can be physical and/or virtual


vSwitch vSwitch vSwitch

• Add in the Core/WAN and the Servers and vSwitches


vSwitch vSwitch vSwitch

• Activate the services, leverage the services chaining and dynamic provisioning

• Leverage the fabric as the layer 3 gateway for all the other VLAN’s

ACI L4-L7 – Device Package Update For YourReference

Partner Product Availability

Cisco ASA 5585 model and ASAv Now

A10 Thunder Appliances Now

AVI Networks AVI Controller Now

Citrix NetScaler MPX, SDX, VPX, NetScaler 1000v Now

F5 BIG-IP

BIG-IQ

Now

Q4 CY15

Palo Alto Networks Panorama Now

Radware Alteon ADC Appliances

Defence Pro

Now

Q3 CY15

Cisco Sourcefire FirePower Q2 CY15

Fortinet Fortigate Appliances Q3 CY15

Check Point Security Gateway Appliances Q4 CY15

Intel Security - McAfee Network Security Platform Q4 CY15

Symantec Data Loss Prevention (DLP) 1H CY16

Veritas NetBackup 1HCY16

Integrating the Networks Layer 3

It’s Routing

Fabric MigrationMigrating the Connectivity to the Core

10.10.10.8 10.20.20.32 10.10.10.9 10.20.20.33 10.20.20.1410.10.10.97


WAN - Core

ACI 11.1 Routing Capabilities

• Supports BGP (iBGP and eBGP), OSPF (normal, stub, and nssa areas), EIGRP, and static routes

• Supports IPv4 and IPv6 (IPV6 for BGP, OSPF, and Static)

• ACI fabric acts as a transit network between different L3 domains. All protocols are redistributed into MP-BGP at border leaf.

• Supports inbound and outbound route filtering using import and export route control.

• Supports set options using route profiles and action rules (set tags, BGP communities, metrics, metric type)

• Supports policy enforcement between L3out EPGs (prefix based EPGs)

• Supports up to three transit points per VRF

ACI fabric has network security enabled by default

BGP RR

MP-BGP Peering

OSPF Adjacency

Route Redistribution

BGP RR

• Allow two L3out to exchange routes with dynamic routing protocol

• Contract between two external EPGs. 11.1 release supports permit all contract

External

EPG-1 External

EPG-2

Contract

Route Export and External EPG ConfigurationRemember you need to configure both

Transit routes advertisement

control. If selected this route

will be advertised to external

routes of this L3out

External EPG classifier for

security contract. If selected this

subnet will be programmed to HW

IP Prefixes to external EPG

mapping table

Can I Run Routing Protocol HereRouting Protocol Over vPC

• IP stack is enhanced to support protocol over vPC. Border leaf forward control traffic to its peer when the destination MAC is SVI MAC and destination IP is peer’s IP

VNI 5555

VLAN 10 VLAN 10

SVI SVI

OSPF

OSPF

SVI IP: 200.200.1.3 SVI IP: 200.200.1.4

Protocol packets

Primary IP:

200.200.1.1

Interconnecting ACI to WAN and DCI

OTV, LISP, BGP EVPN

Single Fabric Multi-Site (Stretched) Fabric

Site/Room ‘A’ Site/Room ‘B’

• Single Fabric + Multi-Site

• Single Operational Zone (VMM, Storage, FW/LB are all treated as if it is ‘one’ zone)

• e.g. Single vCentre with Synchronised Storage

• Interconnect between sites

• Direct Fibre (40G), DWDM (40G or multiple 10G), Pseudo Wire (10G or 40G)

Interconnect Leaf

Nodes

HYPERVISOR HYPERVISOR HYPERVISOR

10 msec. Round Trip

Integration of Stretched ACI Fabric with LISPOptimised Ingress Routing

• Provides inbound routing correction for the ACI endpoints via the Locator/ID Separation

Protocol Multi-Hop Across Subnet Mode (LISP MH ASM)

• https://www2.wwt.com/stretched-active-active-aci-fabric/white-paper-stretched-active-

active-aci-fabric/

• https://www2.wwt.com/video-aci-active-active-demo/

Fabric ‘A’ Fabric ‘B’

Integrating with OTV

BD extend via

OTV/VPLS with

multiple L2Out

• ACI fabrics can connect to any existing layer 2 DCI system, OTV, VPLS, VXLAN

with EVPN, …

• Same basic L2 connection as connecting to any other Nexus or Catalyst switch

• Improvements with 11.2 ACI release provide an automatic common default

gateway across both connected fabrics

• Prior to 11.2 manual configuration is required for default GW and ARP config

OTV

EVPN

OTV Design Requirements Prior to the 11.2 ACI Release• Unique BD MAC address - To enable ARP broadcasts to work properly when

interconnecting two Fabrics, it is required to make the associated Bridge Domain MAC addresses unique per fabric.

• BD settings – ARP Flooding must be Enabled. L2 Unknown Unicast can be Disabled.

• External Bridged Domain - An External Bridged Domain must be created in Fabric / Access Policies. This External Bridged Domain must be associated with an EPG that will be used with OTV. The External Bridge Domain is selected under the EPG configuration in the “Domains” section.

• BPDU-Filter enabled - In Fabric / Access Policies, the access policy used for the vPC to the OTV gateways must have BPDU Filter enabled. This is critical in order to provide high availability during failover scenarios and avoid lengthy periods of traffic loss during these periods.

Subnet 10.10.10.0/24


11.2 Release - Support for the same Pervasive GW IP & MAC on independent fabrics

BD extend via

OTV/VPLS with

multiple L2Out

10.10.10.5

DG = 10.10.10.1

10.10.10.34

DG = 10.10.10.110.10.10.57

DG = 10.10.10.1

10.10.10.39

DG = 10.10.10.1

• Support for the same default gateway address on independent fabrics with BD extended

(same pervasive default gateway)

• Same subnet exists in independent fabrics with layer 2 transport (OTV, VPLS, VXLAN

EVPN)

• Common gateway to allow attachment of device with hard coded DG at either site

• Replicates the current vPC HSRP localisation design

OTV

EVPN

Subnet 10.10.10.0/24

Fabric ‘A’

Integrating with OTV

BD extend via

OTV/VPLS with

multiple L2Out

10.10.10.5

DG = 10.10.10.1

10.10.10.34

DG = 10.10.10.1

• ACI fabrics can connect to any existing layer 2 DCI system, OTV, VPLS,

VXLAN with EVPN, …

• Same basic L2 connection as connecting to any other Nexus or Catalyst

switch

OTV

EVPN

10.10.10.8 10.20.20.32 10.10.10.9

Subnet/BD ‘A’

Layer 3

Site ‘A’ Site ‘B’

Multi-Site Leveraging IP Based EPG – 11.(2) Release

ACI Toolkit exports Contracts on both Fabrics

App Web AppWeb

1

Customer Creates L3Out to second fabric2

On detection of new endpoint on Fabric ‘A’ ACI

Toolkit will create a static IP == EPG binding on

Fabric ‘B’

3

On ingress on Fabric ‘B’ Border Leaf EPG S_Class

will be assigned and policy enforced locally4

IP == EPG classification on

the Border Leaf (Scaling is

limited to LST (24K) entries

Multiple EPG within the

same BD/Subnet

Multisite App

GUI

ACI Toolkit

Multisite App

GUI

ACI Toolkit

Control-

PlaneEVPN MP-BGP - RFC 7432

(draft-ietf-l2vpn-evpn)

Data-

Plane

Multi-Protocol Label Switching

(MPLS)draft-ietf-l2vpn-evpn

Provider Backbone Bridges

(PBB)

draft-ietf-l2vpn-pbb-evpn

Network Virtualisation Overlay

(NVO)draft-ietf-bess-evpn-overlay

EVPN over NVO Tunnels (e.g. VXLAN) for Data Centre Fabric encapsulations

Provides Layer 2 and Layer 3 Overlays over simple IP Networks

mBGP EVPN - Ethernet VPNStandards based

Multi-Fabric Scenarios Multi-Pod – 1HCY16

• Host Level Reachability Advertised between Fabrics via BGP

• Transit Network is IP Based

• Host Routes do not need to be advertised into transit network

• Policy Context is carried with packets as they traverse the transit IP Network

• Forwarding between multiple Fabrics is allowed (not limited to two Pods)


Web/AppDB

Web/App

mBGP - EVPN

Single APIC Cluster

Interfacing ACI to WAN/DCI Routing (11.3 Release)Extending GBP VXLAN to the PE

Web/AppDB

• Integration of WAN/DCI Services with ACI

• Leverage Nexus 7000, ASR 9000

DCI

OTV/VXLAN/VPLS

WAN

ACI Fabric 2

DC Site 2

Branch

PE

PE

MP-BGP EVPN with VXLAN

encapsulation to the N7K, ASR9K

Automated Provisioning via

OpFlex

Web/AppDB

Fabric ‘1’

Web/AppDB

Extending the FabricMulti-Pod + ASR9K and N7K Border Leaf (1HCY16)

Single APIC Domain

Web/App

. . .

Multiple

Pod’s

Single Logical Fabric allows sharing of resources across Pod’s

Fabric ‘N’

IP DC Core

High Speed DC Core Interconnect

supporting BiDir (no host routes, infra-

loopback only)WAN and Backbone Routers

Direct BGP peering with fabrics

(VXLAN from ASR9K and N7K to

spine) DC Core can be 2 to n’

switches in any topology

Deep Dive Breakout into ACI Multi-Site

Extending ACI to Multiple SitesDual Site Deployment Deep Dive

Santiago Freitas ([email protected]), Customer Solutions Architect

Patrice Bellagamba ([email protected]), Distinguished Systems Engineer

BRKACI-3503

Integrating ACI - Virtual Switching

Hypervisors & ContainersACI Provides Automated Integration for all workloads

Hardware

Operating System

Hypervisor

Virtual Machine

Operating

System

Bins / libs

App App

Virtual Machine

Operating

System

Bins / libs

App App

Hardware

Hypervisor

Virtual Machine

Operating

System

Bins / libs

App App

Virtual Machine

Operating

System

Bins / libs

App App

Hardware

Operating System

Container

Bins / libs

App App

Container

Bins / libs

App App

Type 1 Hypervisor Type 2 Hypervisor Linux Containers (LXC)

Containers share the OS kernel of the host and thus are lightweight.

However, each container must have the same OS kernel.

Containers are isolated, but share OS

and, where appropriate, libs / bins.

“Manage ACI Fabrics”

VMWare Integration5.1, 5.5, 6.x, vCentre Plugin and vRealise

Microsoft Interaction with ACI

Two modes of Operation

• Policy Management: Through APIC

• Software / License: Windows Server with HyperV, SCVMM

• VM Discovery: OpFlex

• Encapsulations: VLAN, NVGRE (Future)

• Plugin Installation: Manual

Integration with SCVMM

APIC

Integration with Azure Pack

APIC

• Superset of SCVMM

• Policy Management: Through APIC or through Azure Pack

• Software / License: Windows Server with HyperV, SCVMM, Azure Pack (free)

• VM Discovery: OpFlex

• Encapsulations: VLAN, NVGRE (Future)

• Plugin Installation: Integrated

+

Openstack and OVS Integration with ACIOpFlex Integration for OVS

Local policy enforcement on each

hypervisor

Floating IP / NAT support

APIC GUI integration / VMM

Domain for OpenStack

Per host statistics

Service redirection

OpFlex Offers:

Hypervisor

vm4

Project 1 Project 2 Project 3

vm5vm3

vm5

vm6

OpFlex Agent

OpFlex Proxy

V(X)LAN

OpenStack ControllerGroup-Based

Policy (optional)APIC ML2

Deep Dive Breakout into ACI Hypervisor & L4-7 Integration

Integration of Hypervisors and L4-7 Services into an ACI Fabric

BRKACI-2006

Futures, UCS, TrustSec, Remote Leaf

IP Core with a Hardware Directory Service

• Three data plane functions required within the ‘core’ of an ACI fabric

• Transit: IP forwarding of traffic between VTEP’s

• Multicast Root: Root for one of the 16 multicast forwarding topologies (used for optimisation of multicast load

balancing and forwarding)

• Proxy Lookup: Data Plane based directory for forwarding traffic based on mapping database of EID to VTEP bindings

• Not all functions are required on all ‘spine’ switches

Transit Node

Proxy Lookup

Cached EID

Entry

Proxy Lookup –

Unknown EID

Multicast Root

ACI Spine Proxy == LISP Proxy Tunnel Router+ Map-Server

http://tools.ietf.org/html/draft-moreno-lisp-datacenter-deployment-00

Extension of the ACI Overlay to Remote LeafACI Extended Overlay

Direct Attach Endpoints

Hypervisor Attached Endpoints (VLAN or

VXLAN)

vSwitch

ACI VXLAN Extended Overlay

Infrastructure VRF Extended

• ACI Policy overlay can be extended over existing IP networks

Full ACI VXLAN Switching Enabled

Leaf

L2 ‘or’ L3

VTEP VTEP VTEP VTEP

Lets Look at the Details

VTEP VTEP

Future: ACI & TrustSec Integration Statement of Direction

SGT EPG

ACI-Enabled DCSGTEPGTrustSec-Enabled

Network

Consistent Policy

• Cisco intends to integrate TrustSec into ACI to allow customers to address breach,

segmentation & compliance challenges by using context from TrustSec-enabled

networks in ACI data centre policies

• Enables customers to apply a cohesive security policy across the enterprise –

leveraging user roles and device type together with application context

• Group-based policy approaches of ACI and TrustSec vastly simplify security design,

operations, and compliance

Web/App

ACI and TrustSec Integration Enables Identity and Device based Context to Control Access to Data Centre Resources

Campus

Engineering

TrustSec SGT mapped to ACI EPG at the

Policy Domain Boundary

• Data Plane: Mapping of TrustSec to ACI EPG (GBP VXLAN) at TrustSec Border Router

• Control Plane: Communicate reachability between DC and WAN/Campus via MP-BGP EVPN

• Policy Plane: Management of Policy Groups and mappings (EPG) from ISE to APIC

Branch

EmployeeWeb App DB

Campus / Branch

ISE Policy Domain

Data Centre

APIC Policy Domain

TrustSec

Border Router

(ASR1K, ASR9K)

App1 DBWeb1

QoS

Filter

QoSServi

ce

QoS

Filter

BranchEnterprise-wide

Group Policy

ACI

Fabric

UCS Software Defined Server Architectural Operation

• UCS Manager is a Centralised Software Controller

• UCS has Programmable Hardware

• Deep Control of Server Components• NIC/HBA Creation and Cabling

• Network edge access, availability, security, and QoS

• BIOS and Local Storage Policy

• Board and Peripheral Firmware Management

• Various SLA Enforcement Policies

• Many Others…

• Compute and storage are covered within UCS

• Venn Diagram will how intersect with the periphery of ACI

Compute

Storage

Application

Services

Network

Direct Policy Sharing between UCSM and APIC - Sample

IP Enabled Data Centre

Network

AVS

Directory/Proxy

Service Nodes

AVS

Border Leaves

APIC Policy

ControllerACI Leaf

Nexus 9000

ACI Virtual Leaf

(AVS)vSwitch

ACI Services

extended in to any

existing IP

enabled Data

Centre

ACI Policy and Automation

Extended to Virtual Servers

via Cisco AVS

ACI Policy and Automation

Extended to Physical and

Existing Virtual Servers via

Cisco Nexus 9000

ACI Enabled L4-7

Virtual and Physical

Services (Support for

Existing and New

Services Instances)

Extending ACI Policy and Automation into the Existing Data Centre

ACI It’s an ‘IP’ Network

integration and interoperation ofd2zmdbbm9feqrf.cloudfront.net/2016/anz/pdf/brkaci-2001.pdf ·...

Documents