network management automation of cisco nexus...

Network Management Automation of Cisco Nexus Fabrics

Tom Nosella, Sr. Director, Technical Marketing

BRKDCT-2444

• Fabric Management Challenges

• The 4 Slide VXLAN Primer

• The Nexus Fabric Manager

• Building a Managed Fabric

• Connecting to the Fabric

• Expanding the Fabric

• Upgrading the Fabric

• Conclusion

Agenda

Fabric Management Challenges

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Datacenter Fabric Management Challenges

3. Rapid rollout of fabric infrastructure• Need to respond to needs of the business

• Both initial installation and fabric expansion

2. Minimize fabric downtime• Eliminate misconfigurations (high cause of downtime)

• Rapid recovery of fabric outages

1. Want to take advantage of new protocols and architectures• IT operations expertise requirements for fabric management proficiency

• New protocols and architectures come with complexity challenges

BRKDCT-2444 5


Many Approaches to Fabric Management

• CLI – switch-by-switch – most common• Unnecessarily inefficient and highly error prone

• Requires extensive knowledge of protocols and syntax

• Scripting – to CLI and/or API• Achieves some efficiency – requires devops expertise

• Geared mostly to static config snippets and software management

• Off-the-shelf management solution• Largely element management focused – switch-by-switch

• Some limited templating capabilities – still require CLI/protocol expertise

BRKDCT-2444 6


How Do We Achieve High Management Efficiency?

AUTOMATIONWhat is the right model for you?

CLI ACIScriptingElement

Management

CLI InteractionAutonomous System

FABRIC MANAGEMENT AUTOMATION

???

BRKDCT-2444 7


Desirable Traits of a Fabric Management System

Fabric awareness • Not just a switch is a switch – comprehension of topology and architecture

Workflow oriented• Closer alignment with application/business needs – less focus on CLI/protocols

Self managing, self configuring• System can build and maintain fabric configuration based on workflow outputs

Extendable (API) • Ability to tie system into higher level orchestration system

Full lifecycle management• Ongoing management services throughout all phases of fabric lifecycle

BRKDCT-2444 8


Creation Expansion

FaultsReporting

Connection

FABRICMANAGER

Fabric Management Lifecycle and Considerations

• New switch bring-up• Zero-touch experience

• Initial switch configuration

• Fabric layout discovery

• Infrastructure configuration

• Device discovery

• Single and dual-homed hosts

• Broadcast domains

• Gateway functions

• Adding switch (leaf or spine)• Zero-touch experience

• Dynamic configuration

• Cabling verification

• Broadcast domain expansion

• Fault management system

• Self-resolution

• External notifications

• Switch RMA process

• Task log – who, what, when

• Object-based history/logs

• Logical/physical performance

• Fabric inventory

BRKDCT-2444 9

The 4 Slide VXLAN Primer


What is Virtual Extensible LAN (VXLAN)?

• VXLAN provides a network with segmentation, IP mobility, scale, and stability

• Standards based network overlay technology

• Layer-2 and layer-3 over standard routed network

• Leverages layer-3 ECMP – all links forwarding

• Increased name space to 16 million identifiers (24 bit)

• Segmentation and multitenancy

• Integration of physical and virtual endpoints

Layer-3Network

BRKDCT-2444 11


VXLAN Important Terminology

• VXLAN Tunnel Endpoint (VTEP)

• Performs encapsulation/de-encapsulation

• Usually located in leaf layer

• Can be in hardware or software

• Virtual Network Identifier (VNI)

• Mapping of VLAN to VXLAN (eg VNI 5000 maps to VLAN 500)

• Multiple VLANs can map to same VNI

• Underlay Network

• The IP routed network upon which VXLAN is built

Layer-3Network

(Underlay)VN

I 5000

VLAN 500

VTEP

VTEP

BRKDCT-2444 12


Important VXLAN Services

• VTEP Discovery

• Join multicast group for discovery

Ethernet VPN (EVPN) control plane - leverages BGP

• End Device (MAC) Discovery

• Flood-and-learn – requires multicast

Ethernet VPN (EVPN) control plane – leverages BGP

• Handling Broadcast, Unknown, Multicast (BUM)

• Multicast

Ingress replication – in hardware

Layer-3Network

(Underlay)VN

I 5000

VLAN 500

VTEP

VTEP

MAC : 11:11:11:11:11:11

MAC : 22:22:22:22:22:22

BRKDCT-2444 13


VN

I 5000

Basic VXLAN Diagram

Layer-3 Network(Underlay)

VLAN 500

MAC1 : 11:11:11:11:11:11

MAC5 : 55:55:55:55:55:55

VTEP 1

MAC2 : 22:22:22:22:22:22 MAC3 : 33:33:33:33:33:33

VLAN 500 VLAN 400

VLAN 500

VLAN 400

MAC4 : 44:44:44:44:44:44

Interior GatewayProtocol (eg. OSPF)

MP-BGP/EVPN

MAC VNI NEXT HOP

MAC1 5000 VTEP1_IP

MAC2 5000 VTEP2_IP

MAC3 4000 VTEP2_IP

VTEP 2

VTEP 3

BRKDCT-2444 14

The Nexus Fabric Manager


Cisco Nexus Fabric Manager

Intelligent Fabric Lifecycle Management

• Fabric-wide focus – auto-configuration and management of fabric

• Initial support for Cisco Nexus 9000 Familyrunning stand-alone NX-OS mode

• Automation based on knowledge of underlying fabric architecture

• Designed to simplify fabric management through its various lifecycle phases

• Delivered via VXLAN-based architectureFabric Management Lifecycle

Creation Expansion

FaultsReporting

Connection

FABRICMANAGER

BRKDCT-2444 16


What Does the Nexus Fabric Manager Do ?

Fabric Level Abstraction - Point-and-Click Interface

• Interaction via a simplified request model

• Say what you need, not how to do it

• Simplified point-and-click interface

• Focus on high ease of use

• Simplified tiles view for quick access andefficient management of numerous objects

• Intelligent live, actionable, topology mapping facility

Tiles View

Topology View

BRKDCT-2444 17


Fabric Management Workflows

Sample fabric management workflows

1. Create a fabric

• NFM creates and manages HA-enabled fabric

2. Add a new switch to the fabric

• NFM discovers, adds, and configures new switch

3. Create a broadcast domain

• NFM creates and manages VLANs and VXLAN topology

Assign VNID from NFM managed pool

Assign VLAN from NFM managed pool

Establish VLANport membership

Map VLAN to VNID on target leafs

Attach VNID to VTEP

• Optimized for fabric management workflows

• Help network ops quickly support business needs

• Switch features managed based on workflows

Add to broadcast domain

Build a

broadcast

domain

BRKDCT-2444 18


Nexus Fabric Manager Architectural Overview

• Physical appliance containing management engine and web UI

• Can manage N9K switch it has IP connectivity to (NX-OS mode)

• Communication with switches via NX-OS API

• Required initial switch configuration

1. Preconfigure mgmt IP, gateway, username/password and import switch

2. Leverage Auto Fabric Provisioning (AFP) via NFM embedded POAP services (zero touch)

Point-and-Click

User Interface

Fabric-Aware

Control Engine

RE

ST

AP

I

FABRICMANAGER

Mg

mtN

etw

ork

Sw

itch

Po

ol

BRKDCT-2444 19


Ma

na

ge

me

nt N

etw

ork

Managed and Monitored Objects

• Switches and interfaces in a switch pool can be in managed or monitored mode

• Imported switch initially set as monitored

• Only operational state and stats monitored

• Switch software upgrades supported

• API access and SNMP traps enabled

• Can set perimeter interfaces to monitored

• No changes performed by fabric manager

• Can perform custom configs via switch CLI• Eg. Custom ‘funky’ BGP config (not currently

supported by NFM) to uplink to a core network

MONITORED

MONITORED

MANAGED

MANAGED

MANAGED MANAGED

MANAGED

monitoredinterfaces

FABRICMANAGER

BRKDCT-2444 20


Switch Interface Roles

Switchpool

?

Managed

Monitored

Foreign

Unknown – unknown device

Network Perimeter InterfacesConnected to outside switchpool (foreign)

Network Infrastructure InterfacesConnected to devices inside switchpool

Host-facing – connected to a host

Uplink – connected to router or L4/7

Managed

Monitore

d

Managed

Monitore

d

N/A Switch-facing – switch-to-switch linksN/A Peer-link – used for vPC

Switch-facing – connected to foreign switch

Known

Unknown

BRKDCT-2444 21


Two UI Views – Both Searchable and Actionable

Tiles ViewTopology View

BRKDCT-2444 22


Nexus Fabric Manager Tiles View

• Tiles are used to present information for numerous objects

• Efficient organization and quick retrieval of object details

Hyperlink to switch

interfaces tiles view

Hyperlink to candidate

vPC peer switch

Switch role

Switch model Multi-select

Switch

IP address

Switch name (CLI)

Switch name (NFM)

BRKDCT-2444 23


User Interface Object Search Capability

host11

Text searches

BRKDCT-2444 24

Building a Managed Fabric


Starting Leaf-Spine Topology

• All switches are new or with erased configurations – ie. greenfield only

• NFM will not erase switches, so importing a partially configured switch could likely cause problems

BRKDCT-2444 26


Let’s Describe Each Step

Architecture User Interface Command Line Interface

BRKDCT-2444 27


Main User Interface Window

Multi-Edit

Contextual Menu

User Menu

Faults

Function Tabs

Filter/Sort Bar

Admin Menu

Main

Window

BRKDCT-2444 28


Build a Fabric with the Nexus Fabric Manager

Rack the switches

Cable switches in leaf-spine topology

Boot switches with basic configuration or use Auto Fabric Provisioning

Discover switches via seed switch IP address within fabric manager

Select all switches and change to managed mode

Fabric is now managed

BRKDCT-2444 29


Racking and Cabling the Switches

• Switch cabling must resemble two-tier leaf-spine architecture• NFM will verify topology and alert if required

• NFM shuts down invalid links

• Candidate vPC peer links between leafs• Will discover and designate as peer candidates

• No vPC configuration added until user instructs NFM to build host-facing vPC

• Hosts can be configured in single or multi-homed connection arrangement to leaf switches

Improper Cabling

Candidate

vPC Links

Ma

na

ge

me

nt N

etw

ork

FABRICMANAGER

BRKDCT-2444 30


Ma

na

ge

me

nt N

etw

ork

Booting the Switches• Switches require a few pieces of

basic configuration to be imported by NFM• IP_addr, IP_gateway, username/password

• Nexus 9500 series: L3 interfaces to be enabled for auto fabric discovery mode (via CDP)

FABRICMANAGER

APF

1. Basic configured switch - CLI console• Only require above – skip remainder

2. Auto Fabric Provisioning (AFP)• Enables the NFM to bootstrap new switches

• Based on Power-On-Auto-Provisioning (POAP)

• Import switches by their serial number

• Assign leaf/spine and configuration in one step

BRKDCT-2444 31


The Switchpool

• The switchpool is the container object within which all manageable fabric objects reside –highest level fabric object

• As switches are added to the switchpool, they can be managed or monitored by the NFM

• NFM currently supports one switchpool

• Foreign devices (hosts, switches) are considered always outside the switchpool

SWITCHPOOL

BRKDCT-2444 32


Discovering and Importing a Fabric

• Four methods to import a fabric

• Basic configured switch – switch-by-switch – auto-discover turned off• Provide switch IP, import switch, discover neighbors, select neighbors and repeat –

switches in monitored mode

• Basic configured switch – auto-discover turned on• Provide seed switch IP, import switch, discover neighbors, and repeat until no supported

switches with same credentials found – switches in monitored mode

• Auto fabric provisioning (AFP) to monitored mode• Bootstrap switches and import as monitored mode switches

• Auto fabric provisioning (AFP) to managed mode• Bootstrap switches and import as managed mode switches

• Methods can be mixed to import fabric

BRKDCT-2444 33


BASIC Basic Configured Switch Import

IP address of switch

Serial number of switch

Must provide serial to activate

To access switch – can come from profile


Switch name at CLI – optional

Local to NFM – optional

Auto / leaf / spine

Will override switchpool defaults

• This method assumes switches already have IP address and username/password configured

Can set desired image – no auto upgrade

Required

Optional

BRKDCT-2444 34


Successful Basic Configured Switch Import

• Switches are in monitored mode by default• Unless imported via serial and set to managed

• Only configuration changes made to switches is viaSSH to switch to enable API access and set SNMP trap destination to NFMfeature nxapi

nxapi https port 443

nxapi use-vrf management

snmp-server host 172.31.160.88 traps version 2c agent_community udp-port 17015

snmp-server host 172.31.160.88 use-vrf management udp-port 17015

• NFM assigns unique SW# name to switchalong with actual switch name at CLI

• Must verify switch role to ensure discoveredrole is accurate

Monitored

Mode

(LEAF2)

(LEAF2)

BASIC

BRKDCT-2444 35


Ma

na

ge

me

nt N

etw

ork

Fabric Discovered and in Monitored Mode

MONITORED

MONITORED

MONITORED

MONITORED

MONITORED MONITORED

Monitored

Mode

FABRICMANAGER

CDP/LLDP CDP/LLDP

BRKDCT-2444 36


Auto Fabric Provisioning (AFP)

• Ability to pre-provision entire fabric and then simply turn switches on

• Pre-build switch objects in NFM using switch serial numbers

• Leverages NX-OS embedded POAP services

• Switches put into POAP mode by write erase and rebooting switch

• Switches continue POAP process until success or user interrupts

• Note: if0 must be configured with IP address and be reachable by booting switches and their DHCP requests – ie. same VLAN

• May see NFM fault as shown below if if0 not configured

BRKDCT-2444 37


Auto Fabric Provisioning (AFP)

• Creates switch object and associated AFP profile to bootstrap and import switch

Desired address for new switch

Serial number of new switch

Managed/Monitored



New name to assign to switch

Role to assign to switch

Profile to assign to switch

Image to be upgraded to automatically

NFM-local description

Required

Optional

BRKDCT-2444 38


Auto Fabric Provisioning (AFP) Process

• Can verify switch in POAP mode at switch console2016 Apr 17 14:55:14 switch %$ VDC-1 %$ %POAP-2-POAP_DHCP_DISCOVER_START: POAP DHCP Discover phase started

2016 Apr 17 14:55:22 switch %$ VDC-1 %$ %POAP-2-POAP_FAILURE: POAP DHCP discover phase failed

2016 Apr 17 14:55:29 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: USB Initializing Success

2016 Apr 17 14:55:29 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: USB disk not detected

2016 Apr 17 14:55:29 switch %$ VDC-1 %$ last message repeated 1 time

{repeats every 15 seconds}

• NFM will attempt to find the switch but will fail (this is normal)•

• Can verify AFP process is progressing by looking at switch console2016 Apr 17 15:51:28 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Using DHCP, information received over mgmt0 from 172.31.160.89

2016 Apr 17 15:51:28 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Assigned IP address: 172.31.160.34

2016 Apr 17 15:51:28 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Netmask: 255.255.255.128 Picked up from if0 config2016 Apr 17 15:51:28 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: DNS Server: 1.1.1.1 Assigned by default – 0K2016 Apr 17 15:51:28 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Default Gateway: 172.31.160.1 Picked up from if0 config2016 Apr 17 15:51:28 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Script Server: 172.31.160.88 Bootst2016 Apr 17 15:51:28 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Script Name: poap.py

2016 Apr 17 15:51:41 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: poap_dhcp_intf_ac_action_configuration_success: the script

download string is [copy tftp://172.31.160.88/poap.py bootflash:scripts/script.sh vrf management ]

. . .

2016 Apr 17 15:56:10 switch %$ VDC-1 %$ %POAP-2-POAP_SCRIPT_EXEC_SUCCESS: POAP script execution success

2016 Apr 17 15:56:11 switch %$ VDC-1 %$ %POAP-2-POAP_RELOAD_DEVICE: Reload device

Takes 5-7 minutes

BRKDCT-2444 39


• Moving switches to managed mode starts fabric configuration• Intra-fabric port channels (can be disabled)

• Underlay IP addressing

• Underlay interior gateway protocol (OSPF)

• Bi-directional Forward Detection (BFD)

• Multi-protocol BGP and route reflectors

• EVPN configuration

• VTEP and loopback interface creation

• Enable LLDP

Now For the Magic - Managed Mode

BRKDCT-2444 41


Sample Applied Configuration in Spine Switches204 new lines of CLI added – per switch

Enabling Features1. feature nxapi2. nv overlay evpn3. feature ospf4. feature bgp5. feature interface-vlan6. feature vn-segment-vlan-based 7. feature lacp8. feature lldp9. feature bfd10. feature nv overlay

Enable EVPN

Enable VXLAN

Enable discovery

Enable VXLAN

Creating Port-Channels1. interface port-channel500 2. description This interface has been created

by Nexus Fabric Manager at 172.31.160.41. This port-channel was auto-created betweensw1 and sw2

3. no switchport4. mtu 92165. bfd interval 50 min_rx 50 multiplier 3 6. no ip redirects 7. ip address 10.0.0.15/31 8. no ipv6 redirects 9. ip ospf network point-to-point 10. ip router ospf 100 area 0.0.0.0 11. ip ospf bfd

Automatically added by NFM

Jumbo MTU

Auto addressing P2P links

Routing Protocols1. router ospf 100 2. bfd3. router-id 10.0.0.1 4. redistribute static route-map local-into-ospf5.

6. router bgp 655357. router-id 10.0.0.1 8. neighbor 10.0.0.3 remote-as 65535 9. remote-as 65535 10. update-source loopback501 11. address-family l2vpn evpn12. send-community both 13. route-reflector-client14. neighbor 10.0.0.4 remote-as 65535 15. remote-as 65535 16. update-source loopback501 17. address-family l2vpn evpn18. send-community both 19. route-reflector-client 20. neighbor 10.0.0.5 remote-as 65535 21. remote-as 65535 22. update-source loopback501 23. address-family l2vpn evpn24. send-community both 25. route-reflector-client

iBGP process - Private AS

Distribute EVPN info

Connect to route reflector

BRKDCT-2444 42


Sample Applied Configuration in Leaf Switches215 new lines of CLI added – per switch

EVPN Configuration1. evpn2. vni 16777214 l2 3. rd auto 4. route-target import auto 5. route-target export auto

Special VNI for L3

VXLAN VTEP Interface1. interface nve1 2. no shutdown3. description Used by NFM for VXLAN termination4. source-interface loopback500 5. host-reachability protocol bgp6. member vni 16777214 associate-vrf

VXLAN VTEP interface

MP-BGP/EVPNSpecial VNI for L3

Loopbacks for VXLAN Underlay1. interface loopback500 2. description Used by NFM for VXLAN 3. termination (source-interface of nve1) 4. ip address 10.0.0.7/32 5. ip ospf network point-to-point 6. ip router ospf 100 area 0.0.0.0 7. ip ospf bfd8. interface loopback501 9. description Used by NFM for EVPN routing 10. ip address 10.0.0.4/32 11. ip ospf network point-to-point 12. ip router ospf 100 area 0.0.0.0 13. ip ospf bfd

Used for VTEP reachability via OSPF

Used for BGP reachability for EVPN

Underlay Routing Protocol1. router bgp 65535 2. router-id 10.0.0.4 3. neighbor 10.0.0.1 remote-as 65535 4. remote-as 65535 5. update-source loopback501 6. address-family l2vpn evpn7. send-community both 8. neighbor 10.0.0.2 remote-as 65535

9. remote-as 65535 10. update-source loopback501 11. address-family l2vpn evpn12. send-community both 13. vrf switchpool-default 14. address-family ipv4 unicast 15. advertise l2vpn evpn

One per spine

Default VRF for VXLAN routing

Underlay VRF1. vrf context underlay 2. address-family ipv4 unicast 3. vrf context switchpool-default 4. vni 16777214 5. rd auto 6. address-family ipv4 unicast 7. route-target both auto 8. route-target both auto evpn

Currently unused

Overlay routingfor switch pool

BRKDCT-2444 43


Process Summary . . .

Managed Mode

MonitoredMode

BASIC

Managed Mode

BRKDCT-2444 44


User Interface Topology Clustered Leaf

Switches (vPC) Spine Switch Leaf Switch

Discovered Host

Host Interface

Switch Interface

Port Channel

BRKDCT-2444 45


Switch and Interface Profiles

• Used to apply common feature configurations to groups of objects

• Can assign default profiles for certain objects - one per object• For all leaf, spine switches, for all host-facing, switch-facing, uplink interfaces

• Can be assigned to single switches, interfaces in object edit panel

• As new profile changes are made, CLI changes automatically pushed to switch(es)

BRKDCT-2444 46


Creating a Switch Profile

Default profiles dialog from switch pool settings panel

BRKDCT-2444 47


radius-server key 7 "ToIkLhPpG"

radius-server host 10.10.1.1 key 7 "ShMoMhTl" authentication accounting

aaa group server radius RadServer

server 10.10.1.1

Adding Extra CLI Configuration

• Can add CLI configuration snippets to switch profiles• Eg. Can be used to add a specific RADIUS/TACACS+ configuration

• No syntax validation, no automatic ‘no’ of commands if removed

• Object-specific profiles can be created with object-specific CLI snippets

BRKDCT-2444 48


MP-BGP

OSPF

Ma

na

ge

me

nt N

etw

ork

The Fabric is Now Managed

MANAGED

MANAGED

MANAGED

MANAGED

MANAGED MANAGED

VT

EP

VT

EP

VT

EP

VT

EP

FABRICMANAGER

CDP/LLDP CDP/LLDP

BRKDCT-2444 49

Connecting to the Fabric


Steps to Connecting Devices to the Fabric

Perimeter Device Discovery

Port Channels and host-facing vPCs

Broadcast Domains

Gateways

Virtual Routing and Forwarding (VRF)

BRKDCT-2444 51


Perimeter Device Discovery

• Foreign device - any device connected toperimeter interfaces on leaf switches

• Hosts, firewalls, other switches, etc

• To be discovered they must support and be advertising CDP and/or LLDP

• If not discovered, leaf switch interface must be manually assigned a role for the interface to be enabled – otherwise role remains undetermined and soft shutdown• Ie. active host with no agent will remain isolated

until role assigned to leaf switch perimeterinterface

• Neighbors tab shows all foreign devices attached to fabric

Foreign

Hypervisor

(vSwitch)

Foreign

host

Foreign

switch

BRKDCT-2444 52


Foreign Device Discovery• Five categories of foreign devices - Host, hypervisor, networking, switch, unknown

• NFM processes 'Platform ID' from CDP, and/or 'System description' from LLDP

• NFM recognizes ESXi as 'hypervisor', KVM or generic Linux as 'host', and Nexus 5k, 7k, 9k as 'switch'. All other devices will be considered 'unknown’

• If device speaks CDP and LLDP, information from both is used

• Foreign device and foreign device interface objects are created

• Foreign device objects are persistent even if connected switches are deleted

Foreign hypervisor object Foreign interface object Foreign interface object

BRKDCT-2444 53


Connecting a Server to the Fabric

• Hosts can be single or multi-homed

• Multi-homed servers may require Virtual Port Channel (vPC) to fabric

• Nexus Fabric Manager automatically identifies candidate vPC links as part of fabric discovery

• No configuration is pushed to switches until user action to build port channel or vPC

• Host-facing vPCs cannot use leaf-spine links for vPC peer (unlike ACI)

Switches Tab

Interfaces Tab

BRKDCT-2444 54


Creating a Port Channel (vPC)

filter

2

Filter and Select Host Interfaces

host13

1

filter

host13PC

BRKDCT-2444 55


Port Channel (vPC) Notes• Each port channel has two types of IDs

• NFM-based unique ID referred to as logical ID• Switch-to-switch logical IDs start at 2000 – two per port channel

• Host-to-switch logical IDs start at 1 – one per port channel or vPC

• Switch-based ID referred to as physical ID• Actual ID as shown within switch CLI configuration

• Can be different on switches at either end

• Two per port channel (4 per vPC – two on each switch)

• Nexus fabric manager assigns all physical port channel IDs from 500 , vPC Domain IDs from 1

• User can manually add port channels at CLI with IDs below 500

Port Channels

L_ID2 = po2001L_ID1 = po2000

P_ID1 = po500 P_ID2 = po500

L_ID1 = po1

P_ID1 = po502P_ID2 = po503

P_ID1 = po503P_ID1 = po502

BRKDCT-2444 56


Port Channel Example Logical ID (NFM)

Neighbor switchLogical ID (NFM)

Members (switch)

Physical ID (switch)

BRKDCT-2444 57


Building a Broadcast Domain

• Connect devices at layer 2 – fabric wide

• Can assign VLAN ID or let NFM assign• NFM always assigns VNID

• One or more broadcast domains can be assigned to an interface• Interfaces always in VLAN trunking mode

• NFM automatically builds required VXLAN configuration on all switches

• Two methods• Select switch interfaces and assign – method 1

• Create broadcast domain then add interfaces – method 2

Ma

na

ge

me

nt N

etw

ork

MANAGED

MANAGED

MANAGED

MANAGED

MANAGED MANAGED

VT

EP

VT

EP

VT

EP

VT

EP

FABRICMANAGER

BRKDCT-2444 58


3

Interfaces selected to

build broadcast domain

Creating a Broadcast Domain

filter

2

Filter and Select Host Interfaces

host13

1

filter

BD_10

10 Optional

BRKDCT-2444 59


Applied Configuration For New Broadcast Domain

Interface Configuration1. interface Ethernet1/12. switchport mode trunk3. switchport trunk allowed vlan 104. spanning-tree bpduguard enable

• Configuration built by Nexus Fabric Manager and pushed to leaf switches

• VNID assigned with fixed offset (20000 by default – configurable)

VXLAN VTEP Interface1. interface nve12. no shutdown3. description Used by NFM for VXLAN termination4. source-interface loopback5005. host-reachability protocol bgp6. member vni 200107. ingress-replication protocol bgp8. member vni 16777214 associate-vrf

Enabling ingress replication for BUM packets

VLAN/VNID Configuration1. vlan 1,10,3966-39672. vlan 103. vn-segment 200104. vlan 39675. vn-segment 16777214

New VLAN/VNI pair

EVPN Configuration1. evpn2. vni 20010 l23. rd auto4. route-target import auto5. route-target export auto6. vni 16777214 l27. rd auto8. route-target import auto9. route-target export auto

Enabling EVPN

BRKDCT-2444 60


Broadcast Domain Notes

• Nexus Fabric Manager can automatically assign VLAN ID and will start at VLAN 2 • VXLAN VNID set as VLAN+offset (in settings – default 20,000)

• Broadcast domain only active with members, just like switch

• When interface is added to broadcast domain, it is put into VLAN trunking mode with VLAN enabled• Can set native untagged VLAN through interface settings

• Must still add to broadcast domain to enable native VLAN

BroadcastDomains

Just setting native VLAN in interface settings

1. interface Ethernet1/12. switchport mode trunk3. switchport trunk native vlan 24. switchport trunk allowed vlan none5. spanning-tree bpduguard enable

1. interface Ethernet1/12. switchport mode trunk3. switchport trunk native vlan 24. switchport trunk allowed vlan 25. spanning-tree bpduguard enable

Also adding it to the broadcast domain

BRKDCT-2444 61


VT

EP

MANAGED

MANAGED

VT

EP

FABRICMANAGER

3 4

1 2

2Gateway IP: 20.20.20.1/24MAC: aa:aa:aa:aa:aa:aa

1Gateway IP: 10.10.10.1/24MAC: aa:aa:aa:aa:aa:aa

4Gateway IP: 20.20.20.1/24MAC: bb:bb:bb:bb:bb:bb

3Gateway IP: 10.10.10.1/24MAC: bb:bb:bb:bb:bb:bb

Building an IP (Anycast) Gateway

• The VXLAN architecture provides Anycast gateway function• Same IP gateway per broadcast domain on

each switch

• Common MAC address per switch for all broadcast domains - configurable

• Eliminates tromboning of traffic to reach gateway

• Routing can occurs between broadcast domain gateways on each switch

BRKDCT-2444 62


Interfaces selected to

build broadcast

domain

Creating an IP Anycast Gateway

10.10.10.1/24

BD_10

10 Optional

• By creating a gateway as part of broadcast domain creation, a VXLAN anycastgateway is also created• Can go back and edit

broadcast domain to add gateway

• Can add gateway to different overlay VRF

BRKDCT-2444 63


Applied Configuration For New Anycast Gateway

Broadcast Domain Switched Virtual Interface1. interface Vlan102. no shutdown3. vrf member switchpool-default4. bfd interval 50 min_rx 50 multiplier 35. no ip redirects6. ip address 10.10.10.1/247. no ipv6 redirects8. fabric forwarding mode anycast-gateway

• Anycast gateway MAC configurable – same for all broadcast domains

• Gateway automatically put into default overlay VRF (called underlay-l3)

• Can be added to new VRF in one step – covered in next section

Anycast Gateway MAC Address1. fabric forwarding anycast-gateway-mac CABB.D324.7D50

Configurable in settings

BRKDCT-2444 64


Anycast Gateway Notes

• Because VLANs automatically added to each leaf switch in fabric, so is the gateway function

• DHCP relay configuration can be added at CLI level on gateway interfaces

• ARP suppression not currently automatically enabled – will be enabled in future release

AnycastGateways

BRKDCT-2444 65


Building a VRF

• Virtual Routing and Forwarding (VRF) domain can be added to the fabric through UI

• Gateway objects can be added to VRFs

• Interfaces in L3 mode can be added to VRFs

• VRFs are added to all leaf switches

• Once anycast gateway added to VRF . . .

• Extra VNI and VLAN assigned to VRF (seen in CLI)• VNI used for layer 3 routing between networks

• VLAN not used (NX-OS requirement)

VLAN 3

VLAN 2

VLAN 3

VLAN 2

VLAN 2

VLAN 3

VLAN 3900 / VNI 23900

VLAN 3901 / VNI 23901

Gateways

Gateways Gateways

VRF 1

VRF 2

BRKDCT-2444 66


Default [Switch_Pool] VRF

• Default VRF is called switchpool-default in CLI (Actually an overlay VRF)

• VLAN 3967 / VNI 16777214 are reserved should user create an gateway in the default VRF

• Not used otherwise

Default [Switch_Pool] Overlay VRF SVI Configuration1. interface Vlan39672. no shutdown3. vrf member switchpool-default4. no ip redirects5. ip forward6. ipv6 address use-link-local-only7. no ipv6 redirects

Default [Switch_Pool] Overlay VLAN/VNI1. vlan 39672. vn-segment 16777214

Assigned by NFMAssigned by NFM

Default [Switch_Pool] Overlay VRF1. vrf context switchpool-default2. vni 167772143. rd auto4. address-family ipv4 unicast5. route-target both auto6. route-target both auto evpn

Technically the ‘overlay’ VRF

Default [Switch_Pool] Overlay VRF BGP Configuration1. router bgp 655352. . . .3. . . .4. vrf switchpool-default5. address-family ipv4 unicast6. advertise l2vpn evpn

Enable EVPN advertisements

BRKDCT-2444 67


Creating a VRF

New VRF Configuration1. vrf context Tenant_22. address-family ipv4 unicast

• When VRF added and has no members, only one line of configuration added

Tenant_2

BRKDCT-2444 68


Add Broadcast Domain Gateway to VRF1

Tenant_2

BRKDCT-2444 69


Applied Configuration for VRF + Gateway

New VLAN/VNID Mapping1. vlan 22. vn-segment 200023. vlan 39654. vn-segment 167772135. vlan 39676. vn-segment 16777214

Added for L3 routing in VRF

User broadcast domain

New VRF Configuration1. vrf context Tenant_22. vni 167772133. rd auto4. address-family ipv4 unicast5. route-target both auto6. route-target both auto evpn


New VRF SVIs1. interface Vlan22. no shutdown3. vrf member Tenant_24. bfd interval 50 min_rx 50 multiplier 35. no ip redirects6. ip address 10.10.10.1/247. no ipv6 redirects8. fabric forwarding mode anycast-gateway

9. interface Vlan396510. no shutdown11. vrf member Tenant_212. ip forward13. ipv6 address use-link-local-only


New VRF BGP configuration1. router bgp 655352. router-id 10.0.0.53. . . .4. . . .5. vrf Tenant_26. address-family ipv4 unicast7. advertise l2vpn evpn

New VRF Configuration1. interface nve12. no shutdown3. description Used by NFM for VXLAN termination4. source-interface loopback5005. host-reachability protocol bgp6. member vni 16777213 associate-vrf7. member vni 200028. ingress-replication protocol bgp9. member vni 16777214 associate-vrf

BRKDCT-2444 70


VRF Notes

• When adding broadcast domains, must be aware of VLANs assigned by system to accommodate anycast gateways in non-default VRFs

• No overlay routing protocol enabled in VRF• All anycast gateways in VRF will route locally between them

• Can manually add network advertisement statements to BGP/EVPN configuration in specific switch for given VRF

VRFs

1. router bgp 655352. . . .3. . . .4. vrf Tenant_25. address-family ipv4 unicast6. network 100.100.100.0/247. advertise l2vpn evpn

Advertise network to other leafs

BRKDCT-2444 71

Expanding the Fabric


Expanding the Fabric• Step 1 – rack and cable new

switches in leaf-spine topology

• Step 2a – power on and provide switches with IP addr, gateway, username/passwd

• Switches will pop up in UI as foreign and can be imported from there

• Step 2b – preconfigure new switch objects using auto fabric provisioning(AFP) and then simply turn on switches

• NFM will bootstrap switches via POAP, import them, and build their entire configuration

MP-BGP

OSPF

Ma

na

ge

me

nt N

etw

ork

MANAGED

MANAGED

VT

EP

VT

EP

VT

EP

FABRICMANAGER

CDP/LLDP CDP/LLDP

MANAGED

MANAGED

VT

EP

CDP/LLDP

MANAGED MANAGED

BRKDCT-2444 73

Upgrading the Fabric


Upgrading the Fabric

• Switches are added into ‘upgrade groups’• Image and upgrade policy are applied to groups

• Images are stored in Nexus Fabric Manager

• Edit groups to change image to upgrade to next release

BRKDCT-2444 75


Creating Upgrade Group

Backup_All

7.0(3)I2(2a) (nxos.7.0.3.I2.2a.bin)

BRKDCT-2444 76


Running an Upgrade Task

• Two backup strategies• Parallel and sequential

• Create ‘salt-and-pepper’ or ‘left-and-right’ upgrade groups

• Reuse group with new releases

BRKDCT-2444 77

Conclusion


Conclusion

• Fabric awareness in a management platform leads to automation

• The less reliance on CLI and protocol knowledge – the faster to results

• The new Nexus Fabric Manager delivers automation and simplification of fabric lifecycle management

• The north-bound API of the Nexus Fabric Manager lends itself to integration into higher level orchestration

• The Nexus Fabric Manager live demo is in Cisco Datacenter display

BRKDCT-2444 79


Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.

BRKDCT-2444 80

CiscoLive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/us

http://ciscolive.com/us


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

BRKDCT-2444 81

Thank you

network management automation of cisco nexus...

Documents