dpdk/ovs hands on training - meetupfiles.meetup.com/19623913/dpdk_ovs_hands_on_training.pdf ·...

DPDK/OVS Hands On TrainingClayne Robison, Irene Liew and Sujata Tibrewala

Intel Corporation

Legal Notices and DisclaimersIntel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer.

No computer system can be absolutely secure.

Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance.

Intel, the Intel logo and others are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.

© 2016 Intel Corporation.

http://www.intel.com/performance

Intel Confidential - Subject to NDA restrictions Network Platforms Group

DisclaimersBy using this document, in addition to any agreements you have with Intel, you accept the terms set forth below.

You may not use or facilitate the use of this document in connection with any

infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.

A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.

Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.

The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.

Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm

3

* Other names and brands may be claimed as the property of others.

Goal of the training• Provide an overview of networking and need for fast packet processing

• Provide an overview understanding on OVS-DPDK use case in NFV

• Showcase DPDK acceleration capability in userspace packet processing

• Provide coding hands-on in DPDK application development in the NFV use case

4

agenda• Networking Overview

• Overview of DPDK

• Overview of OVS-DPDK

• Overview of the setup

• Hands-on exercise

5

Networking overview

Software Defined Networks/ Network Function Virtualisation

Network Functions

software.intel.com/networking

NFV Model


Current Model

SDN/NFV in action

Software Defined Networks/Network Function Virtualization Framework

OPNFV Brahmaputra Release

Need for Fast Packet Processing

Need for efficient Data Plane in NFV


Application Plane

Orchestration

Infrastructure Layer / Data Plane

Intel Architecture NFV/SDN Accelerators

VT-dSR-IOV

Virtual Machine Monitor(VMM)/Hypervisor

OpenStack

L2 VNF Applianc

e

L2 VNF Applianc

e

L3 VNF Applianc

e

Control Plane

OpenContrailOpen Daylight

ONOS

DPDKDPDK

DPDKVirtual NIC

VMDqNIC

Silicon

NICSilicon QAT

ChipsetAcceleration

Hyperscan

KVM XEN HYPER-V QEMU

Virtual NIC Virtual NIC

Microsoft azure

RDT

IA CPU

NICSilicon

Virtual Switch

Amazon EC2

L3 VNF Applianc

e

DPDK

Virtual NIC

Security VNF

Appliance

DPDK

Virtual NIC

DPDK

V

Fd.io Legopus Open vSwitch

POF OpenSwitch

BESS

DPDK

Virtual Switch

CloudStackOpen Shift Google Compute Engine

Security VNF

Appliance

DPDK

Virtual NIC

VMM/ Hypervisor

Networking Software Components

.


Packet processing

17software.intel.com/networking

Packet processing


Packet sanity -> CRC calculationVLAN tag present ? -> TPID

Ingress/Egress VLAN Filtering -> MAC + Port+ VLAN membership

SRC learning -> MAC + PORT

DEST MAC Learning -> MAC + port

QOS -> PCP

Packets per second


Frame Part Minimum Frame Size Maximum Frame Size

Inter Frame Gap (9.6 ms) 12 bytes 12 bytes

MAC Preamble (+ SFD) 8 bytes 8 bytes

MAC Destination Address 6 bytes 6 bytes

MAC Source Address 6 bytes 6 bytes

MAC Type (or length) 2 bytes 2 bytes

Payload (Network PDU) 46 bytes 1,500 bytes

Check Sequence (CRC) 4 bytes 4 bytes

Total Frame Physical Size 84 bytes 1, 538 bytes

Table 1. Maximum Frame Rate and Throughput Calculations For a 1-Gb/s Ethernet Link

[1,000,000,000 b/s / (84 B * 8 b/B)] == 1,488,096 f/s (maximum rate)

Packet rate/Cycle calculations


40gbps -> 80 mppsOr

One packet every 12.5 ns -> pipeline cycle time

If SRAM has 2 ns cycle time

More than 6 memory look ups cannot be done

Packet processing explosion VxLAN


Overview of DPDK

22

23

DPDK Overview

DPDK API

Traffic GensPktgen, T-Rex, Moongen, …

vSwitchOVS, Lagopus,

…

DPDK example

apps

AES-NI

VPPApplications

Event based program models

Threading Models

lthreads, …

VideoApps

EAL

MALLOC

MBUF

MEMPOOL

RING

TIMER

Core Libraries

KNI

POWER

IVSHMEM

Platform

LPM

Classification

ACL

Classify

e1000

ixgbe

bonding

af_pkt

i40e

fm10k

Packet Access PMD

ETHDEV

xenvirt

enic

ring

METER

SCHED

QoS

cxgbe

vmxnet3 virtio

PIPELINE

mlx4 memnic

others

HASH

Utilities

IP Frag

CMDLINEJOBSTAT

KVARGS

REORDER

TABLE

Legacy DPDK

Future accelerators

CryptoProgrammable

Classifier/Parser

HW

3rd

Party

GPU/FPGA

3rd

Party

SoC PMD

External mempool manager

SoC HW

SOC model

VNF Apps

DPDK Acceleration Enhancements

DPDK Framework

Network StackslibUNS, mTCP,

SeaStar, libuinet, TLDK, …

Compression

3rd

Party

HW/SW

IPSecDPI

Hyperscan

Proxy Apps, …

PMD 0 PMD 1 PMD 2 PMD 3

SR

-IOV

SR

-IOV

DPDK – Data Plane development KitDPDK Overview

• Runs in Linux user space in the Host and/or Guest

• Grabs cores and places a single pThread on each core

• Each core then becomes a run-to-completion thread

• DPDK uses SR-IOV to bypass the Linux Kernel to have direct access to the ports

• The PMD or (Poll Mode Driver) handles the access to the port hardware using SR-IOV Virtual Function interfaces one per PMD

• The applications use DPDK generic APIs to access and control the ports along with many utilities

24

Linux Kernel

Port 0 Port 1 Port 2 Port 3

DPDK Libraries and utilities

CoreX

DPDKApplication

CoreY

DPDKApplication

CoreY

DPDKApplication

CoreN

DPDKApplication

SR

-IOV

SR

-IOV


SR

-IOV

SR

-IOV


• Each application runs on a single core for best performance

• If applications need to share data then the application must provide the means to communicate

• DPDK does contain many types of ways to allow communication between cores lockless rings, semaphore, …

• The application needs to poll the ports it wishes to receive packets of data called mbufs, RTE_MBUF

• The Rx/Tx routines in DPDK manage multiple mbufs at a time for high performance

• The applications also process multiple mbufs at a time

25

Linux Kernel



CoreX

DPDKApplication

CoreY

DPDKApplication

CoreY

DPDKApplication

CoreN

DPDKApplication

SR

-IOV

SR

-IOV


SR

-IOV

SR

-IOV

DPDK – Data Plane development KitDPDK Features• DPDK contains a large number of tools and features for

application developers to use

• Things like:• Lockless rings• High performance Hash table design

• Contains multi-hash support

• LPM Longest Prefix Match (IPv4/v6)• vHost support• Packet Framework (pipeline, metering, ...)• IP Fragmentation• Hierarchical Scheduler using Service Level Agreements• Enhanced Memory Copy with AVX instructions

• Many different enhances per CPU arch as well

• KNI Kernel Network Interface• Power management support• Many more features all highly tuned code

26

Linux Kernel



CoreX

DPDKApplication

CoreY

DPDKApplication

CoreY

DPDKApplication

CoreN

DPDKApplication

SR

-IOV

SR

-IOV


SR

-IOV

SR

-IOV


• The primary Rx/Tx APIs is:

static inline uint16_t

rte_eth_rx_burst(uint8_t port_id, uint16_t queue_id,

struct rte_mbuf **rx_pkts,

const uint16_t nb_pkts);

static inline uint16_t

rte_eth_tx_burst(uint8_t port_id, uint16_t queue_id,

struct rte_mbuf **tx_pkts, uint16_t nb_pkts);

27

Linux Kernel



CoreX

DPDKApplication

CoreY

DPDKApplication

CoreY

DPDKApplication

CoreN

DPDKApplication

SR

-IOV

SR

-IOV


SR

-IOV

SR

-IOV

DPDK – Data Plane development KitDPDK simple startupint main(int argc, char ** argv) {

n = rte_eal_init( argc, argv);argc -= n; argv += n;application_args( argc, argv);

nb_ports = rte_eth_dev_count();

/* initialize and allocate port resources */…rte_eal_mp_remote_launch( lcore_main,

NULL, CALL_MASTER);

RTE_LCORE_FOREACH_SLAVE( lcore_id) {if ( rte_eal_wait_lcore( lcore_id) < 0)

return -1;}return 0;

}

28

Linux Kernel



CoreX

DPDKApplication

CoreY

DPDKApplication

CoreY

DPDKApplication

CoreN

DPDKApplication

SR

-IOV

SR

-IOV


SR

-IOV

SR

-IOV

DPDK – Data Plane development KitDPDK Overview Simple processing loopstruct rte_mbuf *pkts[32]; int pid = 0, qid = 0;

void lcore_main(void) {

/* Setup code for loop */

….

while( core_is_running ) {

n = rte_eth_rx_burst(pid, qid, pkts, 32);

if ( n ) {

n = process_pkts(pkts, n);

rte_eth_tx_burst(pid, qid, pkts, n);

}

}

}

29

Linux Kernel



CoreX

DPDKApplication

CoreY

DPDKApplication

CoreY

DPDKApplication

CoreN

DPDKApplication

SR

-IOV

SR

-IOV


SR

-IOV

SR

-IOV


• DPDK also runs in Virtual Machines (VM) or Guests running Linux

• These VM can access ports directly vi SR-IOV or using VirtIO (virtual I/O)

• DPDK does support multiple threads per core, but the thread must corperate using the core

• Many example applications are provided to show how to use most of the features in DPDK

• Using DPDK with your application in a run-to-completion design can increase your application performance

30

Linux Kernel



CoreX

DPDKApplication

CoreY

DPDKApplication

CoreY

DPDKApplication

CoreN

DPDKApplication

SR

-IOV

SR

-IOV


SR

-IOV

SR

-IOV


• All source code is provided and MIT licensed

• ~98% of the code is written in portable ANSI ‘C’

• Runs on IA32, IA64, MIPS, PowerPC, ARM, ...

• Supports NIC devices from many different vendors including Intel, Broadcom, EZChip, Chelsio, Mellanox, ENA, Cisco ENIC, Cavium, ThunderX, QLOGIC, NXP, ...

• DPDK is completely Open Source anyone can contribute to the project

• No fees of any kind to contribute or use DPDK

• DPDK: http://dpdk.org

• git clone http://dpdk.org/git/dpdk

31

Linux Kernel



CoreX

DPDKApplication

CoreY

DPDKApplication

CoreY

DPDKApplication

CoreN

DPDKApplication

SR

-IOV

SR

-IOV

http://dpdk.org/

http://dpdk.org/git/dpdk

Overview of OVS-DPDK

32

ovs-vswitchd

OVS Kernel Space Forwarding Plane

NIC NICNIC

Ne

tlink

DPDK

PMD Driver

User Space

User Space

Kernel Space

Kernel Space

PMD Driver

PMD Driver

acePMD

Driver

User Space

Kernel Space

PMD Driver

PMD Driver

OVS User Space Forwarding Plane

OVS with DPDK

ovs-vswitchd

Integrate latest DPDK library into OVS

• Main forwarding plane runs in userspace as separate threads of vswitchd

• DPDK PMD to communicate with physical network

• Available as a compile time option for standard OVS

• Enables support for new NICs, ISAs, performance optimizations, and features available in latest version of DPDK.

NIC NICNIC

DPDK

Native OVS

NIC NICNIC

OVS 2.5.0

DPDK Integration

Control Path

Open vSwitch (OVS) With DPDK ARCHITECTURE

ovs-vswitchd

dpif-netdev(User Space Forwarding)

Compute Node:

User Space

Controller

ovsdb OpenFlow

ovsdb server

ofproto

Compute Node:

Kernel space

Data Path

Control Path

VNF

virtio

Qemu

ne

tde

v

netdev_dpdk

DPDK librte_vhost

DPDK PMD

librte_eth

VNF

virtio

Qemu

dpif-netlink

(Kernel Space Forwarding –Control only)

netdev_linuxnetdev_vport

socketsoverlays

openvswitch.ko

Open vSwitch® with DPDK Processing Pipeline: OPENFLOW

Overview of the setup

36

OVS-DPDK VNF setup• Host-VM is the host machine that runs the OVS-

DPDK and hosts VNF-VM and Tenant-VM:

a. VNF-VM: VM that would run DPDK testpmdapplication – packet forwarding

b. Tenant-VM: VM that would run pktgen application as a traffic generator

• Tenant-VM generates packets to VNF-VM, which would then be forwarded by VNF-VM back to the Tenant-VM.

• Tenant-VM shows throughput in packets/second and Mbps.

• Host-VM creates OVS bridge (br0) and 4 x vhost-user ports with each 2 ports attached to VNF-VM and Tenant-VM

• By default the 1 PMD core thread is created in OVS-DPDK. All 4 vhost-user ports are sharing a core.

37

OVS-DPDK

VNF-VM(testpmd – forwarding)

Vhost-user0

Vhost-user1

br0

Tenant-VM(pktgen – traffic generator)

Vhost-user2

Vhost-user3

Host-VM

Legend:

Traffic flow

OVS bridge connection

Hands on exercise

38

Starting the exercise1. Login to Host-VM, VNF-VM and Tenant-VM using this credential:

User: user

Password: password

2. After login to the VMs, do the following: # sudo su

3. Tenant-VM shows the Pktgen application is running.

4. Testpmd application is located in VNF-VM. Go to /root/dpdk-16.07/app/testpmd directory:

# cd /root/dpdk-16.07/app/testpmd

5. The file we will be editing is iofwd.c in function static void pkt_burst_io_forward(struct fwd_stream *fs)

# vi iofwd.c

39

• This function forwards packets in I/O mode by forwarding packets “as-is”.

• It does not access to packets data.

• Coding instructions:

• Implement the code for receiving packets and transmitting packets

Tips:

• A data structure for packet burst is defined:

• Receive and transmit in burst of packets

• Use rte_eth_rx_burst and rte_eth_tx_burst function calls to implement the received and transmit

• rte_eth_* functions are define in:/root/dpdk-16.07/lib/librte_ether/rte_ethdev.h

40

static voidpkt_burst_io_forward(struct fwd_stream *fs){

struct rte_mbuf *pkts_burst[MAX_PKT_BURST];uint16_t nb_rx;uint16_t nb_tx;uint32_t retry;

/** Receive a burst of packets and forward them.*/

fs->rx_packets += nb_rx;

#ifdef RTE_TEST_PMD_RECORD_BURST_STATSfs->rx_burst_stats.pkt_burst_spread[nb_rx]++;

#endif

/* Transmit a burst of packets**/

/** Retry if necessary*/if (unlikely(nb_tx < nb_rx) && fs->retry_enabled) {

retry = 0;while (nb_tx < nb_rx && retry++ < burst_tx_retry_num) {

rte_delay_us(burst_tx_delay_time);nb_tx += rte_eth_tx_burst(fs->tx_port, fs->tx_queue,

&pkts_burst[nb_tx], nb_rx - nb_tx);}

}fs->tx_packets += nb_tx;

Starting the exercise – CONT.

pkt_burst_io_forward(struct fwd_stream *fs){


Tips in OVS-DPDK – Host-VM1. Check the vhost-user ports’ stats (e.g. RX, Tx, drop, error, etc)

in Host-VM:

• #/home/user/ovs/utilities/ovs-ofctl dump-ports br0

• Run /home/user/training/dump-ports.sh script to have the real-time stats

2. Check the traffic flows among the vhost-user ports in Host-VM:

• #/home/user/ovs/utilities/ovs-ofctl dump-flows br0

• Run /home/user/training/dump-flows.sh script to have the real-time stats

41

Every 1.0s: /root/ovs/utilities/ovs-ofctl dump-ports br0

OFPST_PORT reply (xid=0x2): 5 portsport 4: rx pkts=38467002, bytes=2308023618, drop=0, errs=0, frame=?, over=?, crc=?

tx pkts=526754, bytes=31630962, drop=37265166, errs=?, coll=?port 2: rx pkts=37775940, bytes=2266589652, drop=0, errs=0, frame=?, over=?, crc=?



tx pkts=520202, bytes=31242354, drop=37255585, errs=?, coll=?port LOCAL: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0

tx pkts=0, bytes=0, drop=0, errs=0, coll=0

Every 1.0s: /root/ovs/utilities/ovs-ofctl dump-flows br0

NXST_FLOW reply (xid=0x4):cookie=0x0, duration=2976.679s, table=0, n_packets=157232013, n_bytes=9433951236, idle_age=0, ip,in_port=2 actions=output:3cookie=0x0, duration=2976.677s, table=0, n_packets=158970348, n_bytes=9538224264, idle_age=0, ip,in_port=3 actions=output:2cookie=0x0, duration=2976.676s, table=0, n_packets=156882706, n_bytes=9412988304, idle_age=0, ip,in_port=1 actions=output:4cookie=0x0, duration=2976.674s, table=0, n_packets=159897515, n_bytes=9593854002, idle_age=0, ip,in_port=4 actions=output:1

Tips in Running Testpmd – VNF-VM1. Go to /root/dpdk-16.07/app/testpmd and compile

testpmd application: #make

2. Start testpmd application by running: #sh/root/run_testpmd.sh

3. Once testpmd application is running, do the following:testpmd> set fwd io

testpmd> start

4. To monitor the Tx and Rx traffic in the VNF-VM to check if packets are forwarded, type:

testpmd> show port stats all

42

Tips in Running PKtGEN – Tenant-VM1. To start traffic on Port0 and Port1, do:

Pktgen > start all

2. To start traffic on Port0 only, do:

Pktgen > start 0

3. To start traffic on Port1 only, do:

Pktgen > start 1

4. Other Pktgen options:

Pktgen> help

5. TotalRate shows packets per second and Mbits per second

43

Wokring testpmd

44

Tenant-VM VNF-VM

• Pktgen in Tenant-VM generates packets and transmits them to testpmd in VNF-VM via vhost-user ports running in OVS-DPDK (Host-VM).

• A working testpmd application receives the packets and forwards them back to Pktgen in Tenant-VM.

• Observed transmit (Tx) packets and receive (Rx) packets on the Tenant-VM and VNF-VM’s ports.

To run a functionaltestpmd

45

1. Go to /root/dpdk-16.07/app/test-pmd-working and compile testpmd application:

#export RTE_SDK=/root/dpdk-16.07

#export RTE_TARGET=x86_64-native-linuxapp-gcc

#make

2. Start testpmd application by running: #sh /root/run_testpmd.sh

3. Once testpmd application is running, do the following:testpmd> set fwd io

testpmd> start

BACKUP

46

Testpmd – i0fwd.c

47

static voidpkt_burst_io_forward(struct fwd_stream *fs){


/** Receive a burst of packets and forward them.*/

nb_rx = rte_eth_rx_burst(fs->rx_port, fs->rx_queue,pkts_burst, nb_pkt_per_burst);

if (unlikely(nb_rx == 0))return;

fs->rx_packets += nb_rx;/** Transmit a burst of packets*/

nb_tx = rte_eth_tx_burst(fs->tx_port, fs->tx_queue,pkts_burst, nb_rx);

/** Retry if necessary*/if (unlikely(nb_tx < nb_rx) && fs->retry_enabled) {

retry = 0;while (nb_tx < nb_rx && retry++ < burst_tx_retry_num) {

rte_delay_us(burst_tx_delay_time);nb_tx += rte_eth_tx_burst(fs->tx_port, fs->tx_queue,

&pkts_burst[nb_tx], nb_rx - nb_tx);}

}fs->tx_packets += nb_tx;

if (unlikely(nb_tx < nb_rx)) {fs->fwd_dropped += (nb_rx - nb_tx);do {

rte_pktmbuf_free(pkts_burst[nb_tx]);} while (++nb_tx < nb_rx);

}

dpdk/ovs hands on training - meetupfiles.meetup.com/19623913/dpdk_ovs_hands_on_training.pdf ·...

Documents