cisco ip fabric for media design guide - cisco - global ... ip fabric for media helps you migrate...

© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. of 37

White Paper

Cisco IP Fabric for Media

Design Guide

April 2018

Rahul Parameswaran (rparames)

CISCO SYSTEMS, INC.


Contents

Introduction .............................................................................................................................................................. 3

Frame accurate switching ....................................................................................................................................... 4

Nexus 9000 for IP fabric for Media ......................................................................................................................... 4

Spine and leaf design with network controller ...................................................................................................... 5 Why use a layer 3 spine and leaf design .............................................................................................................. 5 Non-blocking multicast in CLOS architecture ........................................................................................................ 5 Why Media Fabrics can’t rely on ECMP and PIM alone ........................................................................................ 6 Flow orchestration with Cisco DCNM media controller ......................................................................................... 6 Cisco DCNM media controller features ................................................................................................................. 6 Cisco DCNM media controller installation ............................................................................................................. 7 Fabric topology ..................................................................................................................................................... 7 Fabric configuration using Power-On Auto Provisioning (POAP) .......................................................................... 9 Fabric configuration using the command-line interface ....................................................................................... 12 Topology discovery ............................................................................................................................................. 14 Host discovery .................................................................................................................................................... 15 Host policy .......................................................................................................................................................... 16 Flow policy .......................................................................................................................................................... 17 Flow setup ........................................................................................................................................................... 18 Flow visibility and bandwidth tracking ................................................................................................................. 18 Flow statistics and analysis ................................................................................................................................. 20 Flow alias ............................................................................................................................................................ 20 Events and notification ........................................................................................................................................ 21 Precision Time Protocol ...................................................................................................................................... 22 Quality of Service (QoS) ..................................................................................................................................... 23 API integration with the broadcast controller ....................................................................................................... 23 Failure handling .................................................................................................................................................. 24 Cisco DCNM media controller connectivity options: fabric control ...................................................................... 26 Designing the control network ............................................................................................................................. 28 Cisco IP fabric for Media architecture ................................................................................................................. 29 Multi-Site and PIM Border ................................................................................................................................... 31

Guidelines and Limitations ................................................................................................................................... 32 Topology discovery ............................................................................................................................................. 32 Flow setup ........................................................................................................................................................... 32 Host policy .......................................................................................................................................................... 32 Flow policy .......................................................................................................................................................... 33 Design ................................................................................................................................................................. 33 Live and File based workflows on same fabric .................................................................................................... 33

Single switch with controller ................................................................................................................................ 34

Single switch without a controller ........................................................................................................................ 35

Conclusion ............................................................................................................................................................. 36

For more information ............................................................................................................................................. 36


Introduction

Today, the broadcast industry uses an SDI router and SDI cables to transport video and audio signals. The SDI

cables can carry only a single unidirectional signal. As a result, a large number of cables, frequently stretched over

long distances, are required, making it difficult and time-consuming to expand or change an SDI-based

infrastructure.

Cisco IP Fabric for Media helps you migrate from an SDI router to an IP-based infrastructure (Figure 1). In an IP-

based infrastructure, a single cable has the capacity to carry multiple bidirectional traffic flows and can support

different flow sizes without requiring changes to the physical infrastructure.

An IP-based infrastructure with Cisco Nexus 9000 Series switches:

● Supports various types and sizes of broadcasting equipment endpoints with port speeds up to 100 Gbps

● Supports the latest video technologies, including 4K and 8K ultra HD

● Allows for a deterministic network with zero packet loss, ultra low latency, and minimal jitter, and

● Supports the AES67 and SMPTE-2059-2 PTP profiles

Figure 1(a) SDI router

Figure 1(b) IP fabric

The Society of Motion Picture and Television Engineers (SMPTE) 2022-6 standard define the way that SDI is

encapsulated in an IP frame and SMPTE 2110 defines how video, audio and ancillary data is carried over IP.

Similarly, Audio Engineering Society (AES) 67 defines the way that audio is carried over IP. All these flows are

typically User Datagram Protocol (UDP) and IP multicast flows. A network built to carry these flows must help

provide zero-drop transport with guaranteed forwarding, low latency, and minimal jitter.


There are multiple design options that are available today based on a given use case

1. Spine and Leaf CLOS based architecture using Nexus 9000 series switches with Data Center Network

Manager (DCNM) Controller – Provides a flexible and scalable architecture that is suitable for studio

deployments.

2. A single switch, Nexus 9000 series with DCNM controller – Provides a design that is identical to an SDI router.

Simple architecture with DCNM controller offering security and flow visibility. Studio or OBVAN type

deployment.

3. A single switch, Nexus 9000 without a controller – Provides a design that is identical to an SDI router. Simple

architecture. Switch operates like any multicast router.

Frame accurate switching

When switching from one source to another, it is important to ensure the switch occurs at a frame boundary. The

ways to ensure frame accurate switching are as follows:

Network Timed Switching: Here the network is responsible to switch on a frame boundary. This is not possible in a

standard off the shelf switch as it requires FPGA to switch at the frame boundary.

Source Timed Switching: Source timed switching requires the source to modify the UDP port number when a

switch is triggered. At a precise time interval, the source is instructed to change the destination UDP port number

of the signal. The network switch is programmed with a rule to steer the signal to the receiver just before the

source switches their destination UDP port. While this works, it does not scale when the network involves multiple

devices (routers and switches) and limits the ability to switch flows across different sites.

Destination Timed Switching: The destination subscribes or joins the new flow before leaving the old flow and

switches at the frame boundary. Destination timed switching does not require special capabilities in the network

and can also scale across multiple network switches and sites.

Given the goal of the industry is to use commercial off-the-shelf switches, in general it has chosen to move forward

with destination timed switching and Cisco has adopted that model.

Nexus 9000 for IP fabric for Media

Nexus 9000 Series delivers proven high performance and density, low latency, and exceptional power efficiency in

a range of form factors. It also performs line rate multicast replication with minimal jitter. Each switch can operate

as a PTP boundary clock and supports SMPTE 2059-2 and AES67 profile.

Listed below are supported Nexus 9000 switches with their role

Table 1. Nexus 9000 switch port capacity and supported role

Part Number Description Supported Mode

N9K-C92160-YC 48x10/25G+4x100G(6x40G) Leaf with Controller/Standalone Switch with/without Controller

N9K-C93180YC-EX and FX 48x10G/25G+ 6x100G Leaf with Controller/Standalone Switch with/without Controller

N9K-C93108TC-EX and FX 48x1G/10G Base T + 6x100G Leaf with Controller/Standalone Switch with/without Controller

N9K-C9236C 36x100G/40G (Supports 4x10G,4x25G) Leaf or Spine with Controller/Standalone Switch with/without Controller

N9K-C9272Q 72x40G Port (Supports 4x10G on ports 36-71) Leaf or Spine with Controller/Standalone Switch with/without Controller


Part Number Description Supported Mode

9500 with N9K-X9636C-R 36x100G Spine with Controller/Standalone Switch with/without Controller

9500 with N9K-X9636Q-R 36x40G Spine with Controller/Standalone Switch with/without Controller

Spine and leaf design with network controller

Figure 2. Multiple-spine and multiple-leaf architecture with Cisco DCNM media controller

Why use a layer 3 spine and leaf design

Spine and leaf CLOS architecture has proven to be flexible and scalable and is widely deployed in modern data

center design. No matter where the receiver is connected, the path always involves a single hop through the spine,

thereby essentially guaranteeing deterministic latency.

Although a Layer 2 network design may seem simple, it has a very large failure domain. A misbehaving endpoint

could potentially storm the network with traffic that is propagated to all devices in the Layer 2 domain. Also, in a

Layer 2 network, traffic is always flooded to the multicast router or querier, which can cause excessive traffic to be

sent to the router or querier, even when there are no active receivers. This results in non-optimal and non-

deterministic use of bandwidth.

Layer 3 multicast networks contain the fault domain and forward traffic across the network only when there are

active receivers, thereby ensures optimal use of bandwidth. This also provides granular application of filtering

policy that can be applied to a specific port instead of all devices like in case of a layer 2 domain.

Non-blocking multicast in CLOS architecture

SDI routers are non-blocking in nature. A single Ethernet switch such as the Nexus 9000/9500 is also non-

blocking. A CLOS architecture provides flexibility and scalability, however, there are a few design considerations

that needs to be taken into consideration to ensure a CLOS architecture remains non-blocking.


In an ideal scenario, the sender leaf (First hop router) sends one copy of the flow to one of the SPINE switch. The

SPINE creates “N” copies, one for each receiver LEAF switch that has interested receivers for that flow. The

receiver leaf (Last hop router) creates “N” copies of the flow, one per local receiver connected on the leaf. At times,

especially when the system is at its peak capacity, you could encounter a scenario where sender leaf has

replicated a flow to a certain spine, but the receiver leaf cannot get traffic from that spine as its link bandwidth to

that spine is completely occupied by other flows. When this happens, the sender leaf must replicate the flow to

another spine. This results in the sender leaf using twice the bandwidth for a single flow.

To ensure the CLOS network remains non-blocking, a sender leaf must have enough bandwidth to replicate all of

its local senders to all spines. By following this guideline, the CLOS network can be non-blocking.

Bandwidth of all senders connected to a leaf must be equal to the bandwidth of the links going from that

leaf to each of the spine. Bandwidth of all receivers connected to a leaf must be equal to the aggregate

bandwidth of all links going to all spines from that leaf.

For example: A two spine design using N9k-C93180YC-EX, with 6x100G uplinks, 300G going to each spine

can support 300G of senders and 600G of receiver connected to the leaf.

In a broadcasting facility, most of the endpoints are unidirectional – camera, microphone, multiviewers etc. Also,

there are more number of receivers than senders (ratio is 4:1). Also, when a receiver no longer needs a flow, it

leaves the flows, freeing up the bandwidth. Hence, the network can be designed with placement of senders and

receivers such that the CLOS architecture becomes non-blocking.

Why Media Fabrics can’t rely on ECMP and PIM alone

In IT data centers, Equal-Cost Multipath (ECMP) is extremely efficient because most traffic is TCP based, with

millions of flows. A Cisco IP Fabric for Media network has fewer flows, but the flows use more bandwidth, and all

are UDP flows. In this scenario, ECMP may not always be efficient, resulting in hash collusion. Hence, an IP

network needs a controller to orchestrate flows based on bandwidth availability.

Flow orchestration with Cisco DCNM media controller

The Cisco Data Center Network Manager (DCNM) media controller provides a variety of features and functions.

One important function is to ensure that flow is set up between sources and receivers with guaranteed bandwidth

available for the flow. Every flow request sent either by an IGMP join message from a receiver or an API call from

the broadcast controller triggers the media controller to compute an available path in the network and to program

the path end to end. This process helps ensure efficient use of bandwidth. It also helps prevent hash collusion,

which could potentially occur with ECMP.

Cisco DCNM media controller features

The DCNM media controller is an extension to Cisco DCNM. It includes all the features available with DCNM plus

the following media controller features:

● Fabric configuration using Power-On Auto Provisioning (POAP) to help automate configuration

● Topology and host discovery to dynamically discover the topology and host connection

● Flow orchestration to help set up traffic flow with guaranteed bandwidth

● Flow policies, including bandwidth reservation and flow policing

● Host policies to secure the fabric by restricting senders and receivers

● Flow visibility and analytics, including end-to-end path visibility with bit rate set on a per-flow basis


● Events and notifications for operation and monitoring

● Northbound API integration with the broadcast controller

For more information about DCNM, see https://www.cisco.com/c/en/us/support/cloud-systems-management/data-

center-network-manager-10/model.html.

Cisco DCNM media controller installation

For the steps to install the DCNM media controller, see https://www.cisco.com/c/en/us/support/cloud-systems-

management/prime-data-center-network-manager/products-installation-guides-list.html.

The recommended approach is to set up the DCNM media controller in native high-availability mode.

After DCNM is installed, you must manually configure it to work in media-controller mode. You accomplish this by

connecting to the DCNM server through Secure Shell (SSH) and logging in with the username root.

[root@mc ~]# appmgr stop dcnm

[root@mc ~]# appmgr set-mode media-controller

With OVA installation, ensure DCNM is deployed using Large Configuration (four vCPUs and 12G RAM) and in

Thick Provisioned mode.

Fabric topology

The number and type of leaf and spine switches required in your IP fabric depend on the number and type of

endpoints in your broadcasting center.

Follow these steps to help determine the number of leaf switches you need:

Count the number of endpoints (cameras, microphones, gateway, production switchers etc.) in your broadcasting

center. For example, assume that your requirements are as follows:

● Number of 40-Gbps ports required: 40



The uplink bandwidth from a leaf switch to a spine switch must be equal to or greater than the bandwidth

provisioned to endpoints. Tables 1 and 2 list the supported switches and their capacities. Figure 3 shows the

network topology.

Table 2. Supported leaf switch

Leaf Switch Endpoint Capacity Uplink Capacity

Cisco Nexus 9236C Switch 25 x 40-Gbps endpoints 10 x 100-Gbps (1000-Gbps) uplinks

Cisco Nexus 9272Q Switch 36 x 40-Gbps endpoints 36 x 40-Gbps (1440-Gbps) uplinks

Cisco Nexus 92160YC-X Switch 40 x 10-Gbps endpoints 4 x 100-Gbps (400-Gbps) uplinks

Cisco Nexus 93180YC-EX Switch 48 x 10-Gbps endpoints 6 x 100-Gbps (600-Gbps) uplinks

Cisco Nexus 93108TC-EX Switch 48 x 10-Gbps endpoints 6 x 100-Gbps (600-Gbps) uplinks

https://www.cisco.com/c/en/us/support/cloud-systems-management/data-center-network-manager-10/model.html

https://www.cisco.com/c/en/us/support/cloud-systems-management/data-center-network-manager-10/model.html

https://www.cisco.com/c/en/us/support/cloud-systems-management/prime-data-center-network-manager/products-installation-guides-list.html

https://www.cisco.com/c/en/us/support/cloud-systems-management/prime-data-center-network-manager/products-installation-guides-list.html


Table 3. Supported spine switch

Spine Switch Number of Ports

Cisco Nexus 9236C Switch 36 x 100-Gbps ports

Cisco Nexus 9272Q Switch 72 x 40-Gbps ports

Cisco Nexus 9508 with N9K-X9732C-EX Line Card 256 x 40-Gbps ports

Cisco Nexus 9508 with N9K-X9636Q-R Line Card 288x 40-Gbps ports

Cisco 9508 with N9K-X9636C-R Line Card 288x 100-Gbps ports

● The 9236C can be used as a leaf switch for 40-Gbps endpoints. Each supports up to 25 x 40-Gbps

endpoints and requires 10 x 100-Gbps uplinks.

● The 93180YC-EX can be used as a leaf switch for 10-Gbps endpoints. Each supports up to 48 x 10-Gbps

endpoints and requires 6 x 100-Gbps uplinks.

● The 93108TC-EX can be used as a leaf switch for 1/10GBASE-T endpoints. Each supports up to 48 x

1/10GBASE-T endpoints with 6 x 100-Gbps uplinks.

● 40 x 40-Gbps endpoints would require 2 x 9236C leaf switches with 20 x 100-Gbps uplinks.

● 160 x 10-Gbps endpoints would require 4 x 93180-EX leaf switches with 24 x 100-Gbps uplinks.

● 70 x 1-Gbps endpoints would require 2 x 93108-EX leaf switches with 4 x 100-Gbps uplinks. (Not all uplinks

are used.)

● The total number of uplinks required is 48 x 100 Gbps.

● The 9500 with N9K-X9636C-R line card or a 9236C can be used as a SPINE.

● With 9236C, each switch supports up to 36 x 100-Gbps ports. Two spine switches with 24 x 100-Gbps ports

per spine can be used.

● With 9508 and N9K-X9636C-R line card, each line card supports 36x100G ports. Two line cards with a

single spine switch can be used.

Figure 3. Network topology with 9236c as spine


Figure 4. Network topology with 9508 as spine

Fabric configuration using Power-On Auto Provisioning (POAP)

POAP automates the process of upgrading software images and installing configuration files on Cisco Nexus

switches that are being deployed in the network. When a Cisco Nexus switch with the POAP feature boots and

does not find the startup configuration, the switch enters POAP mode, locates the DCNM Dynamic Host

Configuration Protocol (DHCP) server, and bootstraps itself with its interface IP address, gateway, and DCNM

Domain Name System (DNS) server IP addresses. It also obtains the IP address of the DCNM server to download

the configuration script that is run on the switch to download and install the appropriate software image and device

configuration file (Figure 5).

Figure 5. POAP process


The DCNM controller ships with configuration templates: the Professional Media Network (PMN) fabric spine

template and the PMN fabric leaf template. The POAP definition can be generated using these templates as the

baseline. Alternatively, you can generate a startup configuration for a switch and use it during POAP definition.

(The section “Fabric Configuration Using the Command-Line Interface” later in this document provides steps for

creating a configuration manually.)

When using POAP, you follow these steps:

● Create a DCHP scope for temporary IP assignment.

● Upload the switch image to the DCNM image repository.

● Generate the switch configuration using startup configuration or a template.

Figures 6 through 11 illustrate these steps.

Figure 6. DCNM > Configure > POAP

Figure 7. DCNM > Configure > POAP > DHCP Scope


Figure 8. DCNM > Configure > POAP > Images and Configurations

Figure 9. DCNM > Configure > POAP > POAP Definitions

Figure 10. DCNM > Configure > POAP > POAP Definitions


Figure 11. Generate configuration using a template: DCNM > Configure > POAP > POAP Definitions > POAP Wizard

Fabric configuration using the command-line interface

Fabric setup can be performed through the switch Command-Line Interface (CLI). The main components include

an Interior Gateway Protocol (IGP) such as Open Shortest Path First (OSPF) for unicast routing and PIM for

multicast routing. However, PIM is set up in passive mode because the DCNM media controller provisions the PIM

Outgoing Interface (OIF)) and does not use protocol negotiation. Using Nonblocking Multicast (NBM) mode in the

media controller, the switch is informed of the DCNM controller IP address and credentials. TCAM carving is used

for host security policies and flow statistics.

The configurations are shown here.

Setting up the NBM mode controller

! Enable the feature

feature nbm

feature nxapi

! Enable mode controller and specify controller IP

nbm mode controller

controller ip <dcnm_vip_address> vrf <vrf_name>

controller-credentials username admin password 0 <password>

! Verification

show nbm controller

VRF IP Role Status Online-

management 10.23.234.120 ACTIVE ONLINE 2016-12-04

14:43:00.151088


Configuring TCAM carving

! RACL TCAM for Host Policy and ing-l3-vlan-qos for flow statistics

! Requires a switch reload post configuration

! Only required on LEAF switches

hardware access-list tcam region ing-racl 512

hardware access-list tcam region ing-l3-vlan-qos 256

hardware access-list tcam region ing-nbm 1536

Configuring PIM and IGP (OSPF)

! PIM is enabled but in passive mode, IGP needed for any unicast traffic on

fabric

feature pim

feature ospf

router ospf 1

maximum-paths 36

! Maximum paths must be greater than number of uplinks between a single leaf and

spine layer. Default is 8 which is sufficient in most scenarios

interface etx/y

ip pim sparse-mode

ip pim passive

ip router ospf 1 area 0

Configuring the interfaces

! Fabric Port Configuration (links between leaf and spine)

interface Ethernet1/50

no switchport

ip address x.x.x.x/x

ip router ospf 1 area 0.0.0.0

ip pim sparse-mode

ip pim passive

no shutdown

! Host Port Configuration (link between switch and end host)




ip ospf passive-interface

ip pim sparse-mode

ip pim passive

ip igmp version 3

ip igmp immediate-leave


ip igmp suppress v3-gsq

no shutdown

! Host Port Configuration when using an SVI with port as a trunk or access port

interface Ethernet 1/2

switchport mode access

switchport access vlan 100

spanning-tree port type edge

interface vlan 100




ip pim sparse-mode

ip pim passive

vlan configuration 100

ip igmp snooping fast-leave

Topology discovery

The DCNM media controller automatically discovers the topology when fabric is brought up using POAP. If the

fabric is provisioned through the CLI, the switches need to be manually discovered by DCNM. Be sure that fabric

has been discovered after you set the default flow and host policy and before any endpoints are connected and

discovered on the fabric. Figures 12 and 13 show the steps required to discover the fabric.

Figure 12. DCNM > Inventory > Discover Switches > LAN Switches


Figure 13. DCNM > Media Controller > Topology

Host discovery

A host can be discovered in the following ways:

● When the host sends an Address Resolution Protocol (ARP) request for its default gateway: the switch

● When the sender host sends a multicast flow

● When a receiver host sends an IGMP join message

● When the host is manually added through an API

You can find host information by choosing Media Controller > Host.

The host is populated on the topology page, providing a visual representation showing where the host is

connected.

Only hosts that are discovered are displayed on the topology page.

Figure 14(a) DCNM > Media Controller > Topology


Figure 14(b) DCNM > Media Controller > Hosts

Host policy

Host policy is used to prevent or allow a sender to send traffic to certain multicast groups and a receiver to

subscribe to certain multicast groups. Default policy can use a whitelist or a blacklist model. Sender host policy is

implemented with an ingress access-list programmed on the sender leaf. Receiver host policy is implemented by

applying an IGMP filter on the receiver leaf.

The host policy page includes all policies created on DCNM through the GUI or API. It also indicates the devices to

which a given policy is applied. To see this information, click the “i” on the host policy page. (Figure 15).

Configuring the default host policy

Be sure to make any changes to the default host policy before any active host appears on the network. The default

host policy specifies whether to use a blacklist (allow all by default) or a whitelist (deny all by default) and should be

defined before hosts are added to the network.

Changes can be made to a host-specific policy at any time. However, no changes can be made to the default

policy anytime it is applied to any host. In order to make changes to the default policy, ports facing the hosts must

be shut down, the flows then time out and DCNM controller removes the policy associations on the switch.


Figure 15. DCNM > Media Controller > Policies > Host Policies

Flow policy

The flow policy is used to specify the flow properties, including the flow bandwidth and Differentiated Services

Code Point (DSCP) marking. The default policy is applied to flows that do not have a specific policy defined. The

flow policy helps ensure that sufficient bandwidth is provisioned on the network. It also helps ensure that the

sender does not send traffic at a rate higher than the rate provisioned using the policy. Excessive traffic is policed.

(Figure 16).

Configuring the default flow policy

Changes to the default flow policy must be made before any active flows exist in the system. The controller ships

with a default flow size of 3 Gbps and a DSCP of CS1.


Figure 16. DCNM > Media Controller > Policies > Flow Policies

Flow setup

Flow setup can be accomplished in two ways. A receiver, using Internet Group Management Protocol (IGMP) can

request for a flow, or the broadcast controller can setup a flow using an API. Details on the API and the API

integration is discussed in API Integration with the Broadcast Controller section.

Flow visibility and bandwidth tracking

One of the broadcast industry’s biggest concerns in moving to IP is maintaining the capability to track the flow path.

The DCNM media controller provides end-to-end flow visibility on a per-flow basis. The flow information can be

queried from the DCNM GUI or through an API.

One can view bandwidth utilization per link through the GUI or an API. Also, when the Bandwidth check box is

selected, the color of the links change based on their utilization. See Figures 17, 18, and 19.


Figure 17. DCNM > Media Controller > Topology > Multicast Group

Figure 18. DCNM > Media Controller > Topology and Double-Click Link


Figure 19. DCNM > Media Controller > Topology

Flow statistics and analysis

DCNM maintains a real-time per-flow rate monitor. It can provide the bit rate of every flow in the system. If a flow

exceeds the rate defined in the flow policy, the flow is policed, and the policed rate is also displayed. Flow

information is cached for a period of 24 hours. Flow information can be exported and stored for offline analysis

(Figure 20).

Figure 20. DCNM > Media Controller > Flow Status

Flow alias

Every flow on the fabric is either a video, audio or ancillary flow. For an operator to recognize a flow a more

meaningful name can now be associated with a multicast group. Flow Alias provides option to name a flow. The

flow alias name can be referenced throughout the DCNM GUI instead of the multicast group. (Figure 21).


Figure 21. DCNM > Media Controller > Flow Alias

Figure 22. DCNM Flow View with Flow Alias

Events and notification

The DCNM media controller logs events that can be subscribed to using Advanced Message Queuing Protocol

(AMQP). The events are also logged and can be viewed through the GUI. Every activity that occurs is logged: a

new sender coming online, a link failure, a switch reload, a new host policy pushed out, etc. (Figure 23).


Figure 23. DCNM > Media Controller > Events

Precision Time Protocol

Precision Time Protocol (PTP) is used to provide clock information to all endpoints to help ensure that they are

time synchronized. If you need PTP, you must enable the boundary clock function on all switches. The fabric does

not support multicast routing for PTP packets.

The active and passive grandmaster switches are connected to the leaf switch.

Two profiles are supported: 2059-2 and AES 67. Any profile can be enabled on any interface. The slave switches

connected to the grandmaster switch must be configured with the same PTP parameters (Figure 24).

Example

feature ptp

! ptp source IP can be any IP. If switch has a loopback, use the loopback IP as

PTP source

ptp source 1.1.1.1


ptp

ptp delay-request minimum interval smpte-2059-2 -2

ptp announce interval smpte-2059-2 0

ptp sync interval smpte-2059-2 -3


Figure 24. Grandmaster and passive clock connectivity

Quality of Service (QoS)

QoS provides differential treatment to traffic based on its DSCP markings. The suggested markings for traffic

traversing the fabric is as indicated in the table below (Table 4).

Table 4. QoS

Service Class (Flow/Control) DSCP Values Queue

PTP Time and Sync EF Control/Priority Level 1

Intercom Audio EF Priority Level 1

AES67, 2110 Audio AF41 Priority Level 2 or Bandwidth

2022-6, 2110 Video AF41 Priority Level 2 or Bandwidth

2110 Ancillary CS3 Bandwidth

Other/Unicast 0 Best Effort/Default

API integration with the broadcast controller

The broadcast controller integrates with the DCNM media controller through open REST APIs. The integration

helps ensure that the end operator experience remains unchanged while the solution moves from an SDI network

to an IP network. The broadcast controller remains synchronized with the DCNM network controller (Figure 25).


Figure 25. Integration of broadcast and DCNM media controllers

For a list of APIs, see https://developer.cisco.com/site/data-center-network-manager/#overview.

Failure handling

Most deployments implement redundancy by provisioning parallel networks and connecting endpoints to both

networks (for example, 2022-7). Any failure on one network will not affect traffic to endpoints because a copy of the

frame is always available on the redundant network (Figure 26).

Within a fabric, when a failure takes place, such as a link down or a spine failure, recovery mechanisms built into

the DCNM media controller will help ensure that traffic converges. However, a few seconds may be needed to

reprogram the flows on other available paths. Flows are not moved back to the original path after the original path

recovers. New flows use the recovered path. The DCNM media controller provides a list of flows affected by any

failure.

https://developer.cisco.com/site/data-center-network-manager/#overview


Figure 26. Redundant IP fabrics: Completely independent of each other

The following types of failures may occur:

● Link flap between a leaf and spine (fabric link)

● Spine reload or power down

● Leaf reload or power down

● Loss of connectivity between switch and DCNM media controller

● DCNM failure (server reload or power down)

Link failure

When a link fails between a leaf and spine, flows are moved to other available links. In scenarios where available

bandwidth is not sufficient to move all affected flows, the media controller will move certain flows (random). For the

flows that it was not able to service, an event will be generated notifying the user about the affected flows.

When bandwidth becomes available in the future, the flows will be programmed after an IGMP join message is

received from a receiver.

Spine failure

A spine reload causes all flows routed on that spine to fail. The media controller moves the affected flows to other

spine switches. In scenarios where available bandwidth is not sufficient to move all affected flows, the media

controller will move certain flows (random). For the flows that it was not able to service, an event will be generated

notifying the user about the affected flows.

When bandwidth becomes available in the future, the flows will be programmed after an IGMP join message is

received from a receiver.

Leaf failure

A Leaf failure will result in the loss of traffic to endpoints connected to that leaf.

After recovery, the endpoints will have to resend IGMP join messages.


Connectivity loss between Cisco DCNM and switch

The DCNM media controller is the control plane for the fabric. When connectivity is lost between the switch and the

controller, no new control plane activities are possible. New flows cannot be programmed, and flows cannot be

removed. Existing flows will continue to work. The switch will periodically retry to connect to DCNM.

Cisco DCNM failure

DCNM high availability is implemented using an active and cold standby pair. The databases of the active and

standby instances are synchronized. When the active instance fails, the standby instance detects the failure and

assumes the active role.

DCNM high-availability failover takes about three minutes to complete, during which time no new control-plane

activities can be performed (Future release will provide active, hot standby HA implementation).

Cisco DCNM media controller connectivity options: fabric control

The switches in the IP fabric can communicate with DCNM using Out-Of-Band (OOB) management, OOB front-

panel ports, or in-band management.

DCNM ships with two Network Interface Cards (NICs): Eth0 and Eth1. POAP works on Eth1, with the DHCP server

on DCNM configured to listen to DHCP messages on Eth1 only. Switches can use POAP using the management

port or a front-panel port. For POAP to work, the OOB management port or the front-panel port used for

communication with DCNM must be on the same VLAN as the DCNM Eth1 port.

Figures 27, 28, and 29 show the control options.

Figure 27. Using OOB management ports to communicate with Cisco DCNM media controller


Figure 28. Using OOB Front-Panel Ports to Communicate with Cisco DCNM media controller

Figure 29. Using Inband Ports to Communicate with Cisco DCNM media controller

The following CLI snippet shows the configuration using the OOB management port:

interface mgmt0

vrf member management

ip address 1.2.3.4/24

nbm mode controller

controller ip <ip address> vrf management

controller-credentials username admin password 7 ljw39655


The following CLI snippet shows the configuration using the OOB front-panel port:

! Define Fabric Control VRF

vrf context fabric_control

! Specify the VRF name under NBM controller configuration

nbm mode controller

controller ip <dcnm_ip> vrf fabric_control


description to_fabric_control_network

vrf member fabric_control


no shutdown

CLI snippet when using Inband

! Create a loopback interface on each switch

! Advertise the loopback into IGP

! Extend the IGP to DCNM Control Network

interface loopback 100

ip address 1.2.3.4/24

ip router ospf area 0

nbm mode controller

controller ip <ip address> vrf default

controller-credentials username admin password 7 ljw39655

Designing the control network

The control network can be divided into two segments. One is fabric control network, which includes the network

between the IP fabric and the DCNM media controller. The other is the endpoint control network, which enables the

broadcast controller to communicate with the endpoints and the DCNM media controller.

Figure 30 shows the logical network connectivity between the broadcast controller, endpoints, and DCNM.

The control network typically carries unicast control traffic between controllers and endpoints.


Figure 30. Control network

Cisco IP fabric for Media architecture

Figures 29 through 31 shows various deployment topologies using Cisco IP Fabric for Media.

Figure 29 describes a topology that supports a redundant IP network that can carry 2022-6/7 or 2110 flows.

Endpoints must be capable of supporting hitless merge.

Figure 30 shows the flexibility offered by a SPINE and LEAF design that can be used in a distributed studio

deployment model.


Figure 31 shows a possible deployment scenario to connect a remote site using the concept of a remote leaf. The

leaf is not configured any different from a leaf in the main facility. It is a part of the fabric.

Figure 31. Redundant IP networks

Figure 32. Distributed studios


Figure 33. Remote leaf

Multi-Site and PIM Border

To aid remote production and to interconnect various fabrics with each other or to connect a media fabric to an

external PIM network, the multi-site/PIM border feature has been introduced. This solution uses PIM SSM (source

specific multicast) to enable flow setup between remote senders and receivers. For a receiver to request flow from

a remote sender, it must send the IGMPV3 report, along with the sender and group address. Using the unicast

routing table, DCNM and the switch creates and propagates PIM (S,G) join towards the source. When the fabric

that receives the PIM (S,G) join, it programs the flows to go all the way from the sender to the receiver. DCNM

intelligently manages the inter site link bandwidth, ensuring reliable flow programming.

This solution requires designating a leaf as a border-leaf. The border-leaf is the leaf that is used to interconnect the

fabric to remote sites or a PIM router. The current software release supports a single border-leaf per site, however,

the border-leaf itself can have multiple layer 3 routed links towards the external network or sites.

The following CLI snippet shows the configuration of the Border Leaf:

! Specify the switch role under NBM controller configuration

nbm mode controller

switch-role border-leaf

!Enable PIM on the interface connecting remote site/external PIM router


description to_external_network

ip pim sparse-mode

ip router ospf 1 area 0


no shutdown


Figure 34. Multisite

Guidelines and Limitations

The following section presents guidelines and limitations for the solution described here.

Topology discovery

Because the topology can be discovered when the switches use POAP or when they are manually added to

DCNM, be sure to add only switches that are participating in the Cisco IP Fabric for Media and that are managed

by the DCNM media controller.

Flow setup

Flows can be set up using IGMP join messages and through an API call. The system supports IGMP Version 2

(IGMPv2) and IGMPv3. Also, a static OIF cannot be added through the CLI. Any static flow setup must be

implemented through an API call.

Note the following guidelines and limitations for flows set up using an API:

● DCNM notifies the broadcast controller and user about any unsuccessful API call that requires the

broadcast controller or user to retry the call. The notification is done via the AMQP bus.

● Static APIs are not supported when the receiver is connected through a Layer 2 port and Switch Virtual

Interface (SVI).

Host policy

The default host policy shipped with DCNM is a blacklist policy (allow all by default). Any changes to the default

policy must be made before any hosts are discovered on the system. The default policy cannot be edited after

hosts are active. This restriction does not apply to host-specific policies.

The receiver host policy applied to a host connected through a Layer 2 port and SVI applies to all the join

messages sent by all the hosts on that VLAN and cannot be applied to just a single receiver.


When there are multiple hosts behind a Layer 3 port or SVI and the default host policy is a whitelist, in certain

scenarios even when a host policy permits a host, ARP is the only mechanism through which the host is detected

and that specific host policy is programmed. Host detection through IGMP or multicast traffic may not work.

Do not modify any host policy from the CLI. You cannot apply any custom polices from CLI. Changes has to be

made using the DCNM media controller.

Flow policy

You must make any changes to the default policy before any flows are active on the fabric.

Flow policy cannot be modified while the flow is active. You can make modifications only when the flow ages out,

which typically occurs after the sender stops transmitting the flow (inactive flows time out after 180 seconds).

As a best practice, assume a flow size that is at least 5 percent greater than the flow bit rate. This configuration will

accommodate a certain amount of burst without causing the flow to be policed. For example, a 3-Gbps flow must

be provisioned as 3.15 Gbps or 3.3 Gbps.

Design

Always be sure that the uplink bandwidth from the leaf layer to the spine layer is equal to or greater than the

bandwidth to the endpoint. For example, when using a Cisco Nexus 93180YC-EX Switch, 48 x 10-Gbps

connectivity to the endpoints requires a minimum of 5 x 100-Gbps uplinks. You can use all 6 x 100-Gbps uplinks.

When possible, spread endpoint across different leaf switches to evenly distribute sources and receivers on all leaf

switches.

Limit the number of Spines to 2. In scenarios where a fixed SPINE like the 9326C is not sufficient, consider using

the Nexus 9508 as a SPINE.

Design redundant IP networks (2022-7). The networks should be independent, and each should have its own

instance of the DCNM media controller.

The bandwidth management algorithm does not account for unicast traffic on the network. Be sure that unicast

traffic, if any, occupies the minimum network bandwidth.

Live and File based workflows on same fabric

The benefit of a true IP Fabric is its ability to carry any IP Streams. Using QoS (Quality of Service) flows can be

prioritized according to business requirements. In an IP Fabric for Media, LIVE production workflows which are

multicast (2022-5, 2110) are of a higher priority than file based workflows, which are typically unicast. When

DCNM orchestrates multicast flows, it automatically puts the flows in high priority class, that way, as these flows

egress the switch, it is provided the highest priority. User has to define customer policies to ensure unicast flows

are placed in low priority queue. The configuration below ensures LIVE traffic (multicast) is prioritized over File

traffic (unicast).

! Create an ACL matching unicast traffic

ip access-list pmn-ucast

10 permit ip any 0.0.0.0 31.255.255.255

20 permit ip any 128.0.0.0 31.255.255.255

30 permit ip any 192.0.0.0 31.255.255.255

!Create an ACL matching multicast traffic


ip access-list pmn-mcast

10 permit ip any 224.0.0.0/4

policy-map type qos pmn-qos

class type qos pmn-ucast

set qos-group 0

class type qos pmn-mcast

set qos-group 7

class class-default

!Apply the service policy on all interfaces on all switches in the fabric

interface ethernet 1/1-54

service-policy type qos input pmn-qos

Single switch with controller

In certain deployments, a single switch provide sufficient ports and bandwidth and the endpoints are directly

connected to this switch. Any Nexus 9000 switch can be used with this type of deployment.

Controller offers features such as fabric configuration, flow visibility and analytics, security and north bound

integration with broadcast controller.

(Support for 9508 with R series line card will be available in a future release)

Setting up the NBM mode controller

! Enable the feature

feature nbm

feature nxapi

! Enable mode controller and specify controller IP

nbm mode controller

controller ip <dcnm_vip_address> vrf <vrf_name>

controller-credentials username admin password 0 <password>

! Verification

show nbm controller

VRF IP Role Status Online-

management 10.23.234.120 ACTIVE ONLINE 2016-12-04

14:43:00.151088





! Configuration below not required on R series Nexus 9500




Configuring interface






ip pim sparse-mode

ip pim passive

ip igmp version 3



no shutdown

Single switch without a controller

A single switch without controller can be deployed as well. There is no direct interaction between the broadcast

controller and the network. Broadcast controller communicate with the endpoints which using IGMP can subscribe

to any stream on the switch.

Below are the configurations required.




! Configuration not required on R series Nexus 9500




Configuring NBM mode flow

! 9500 with –EX series need the configuration below

! 9500 with R series line card is natively non-blocking and the CLI below is

optional


nbm mode flow

policy <policy-name>

bandwidth <flow-bandwidth>

ip group-range <ip-address> to <ip-address>

Configuring interface






ip pim sparse-mode

ip pim passive

ip igmp version 3



no shutdown

Conclusion

Cisco IP Fabric for Media provides the broadcast industry a path to migrate from SDI to an IP Based infrastructure

with multiple modes of deployment, be it a flexible Spine and leaf design or a single modular chassis or switch. The

fabric guarantees zero-drop multicast transport with minimal latency and jitter. The fabric is highly visible and

provides flow visibility and bit-rate statistics on a per-flow basis. Host polices help ensure that the fabric is secured

from any unauthorized endpoints. Integration with any broadcast controller through open REST APIs helps ensure

that any complexity in the network is abstracted and that the user has an unchanged operator experience.

For more information

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-

x/ip_fabric_for_media/solution/guide_703i45/b_Cisco_Nexus_9000_Series_IP_Fabric_for_Media_Solution_Guide_

703I45.html.

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/datasheet-c78-

735989.html.


736651.html.


738321.html.

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/ip_fabric_for_media/solution/guide_703i45/b_Cisco_Nexus_9000_Series_IP_Fabric_for_Media_Solution_Guide_703I45.html



https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/datasheet-c78-735989.html







Printed in USA C11-738605-02 04/18

cisco ip fabric for media design guide - cisco - global ... ip fabric for media helps you migrate...

Documents