dpdk accelerated ostinato

http://ostinato.org/

Srivats P.

6WIND SPEED MATTERS The Challenge 2014 DPDK Design Contest

OSTINATO DPDK a elerated


What is Ostinato?

Open Source Cross Platform

Traffic Generator


What is DPDK?

Application

Libraries and user-space NIC drivers to boost packet processing performance


Together

DPDK Packet Processing

Power

OSTINATO Features and

Flexibility

1/10G line rate

feature rich

traffic generator

on commodity

hardware


Multi-core Processor

DPDK Programming Model

Core

Packet Processing Software

Engine

Core


Engine

Core

Launch and

Control of Engines

Core


Engine

DPDK Libraries + Environment Abstraction Layer (EAL)

Slaves Master

Run to completion model for engines Alternatively, they can collaborate in a pipeline model


Ostinato Controller -Agent Architecture

Agent (Drone)

Controller (Ostinato)

►Packet Generation ►Packet Capture ►Statistics

GUI/Python-Script ►Configuration ►Control ►Results

ProtoBuf based RPC

Protocols, Packet Length, Rates etc.


Drone Classes and Interfaces

Port Port Port Port

RpcServer

OstService

RPC interface Port Class

Interface

Ostinato

(Controller)

Drone

(Agent)


RPC Interface service OstService {

rpc checkVersion(VersionInfo) returns (VersionCompatibility);

rpc getPortIdList(Void) returns (PortIdList);

rpc getPortConfig(PortIdList) returns (PortConfigList);

rpc modifyPort(PortConfigList) returns (Ack);

rpc getStreamIdList(PortId) returns (StreamIdList);

rpc getStreamConfig(StreamIdList) returns (StreamConfigList);

rpc addStream(StreamIdList) returns (Ack);

rpc deleteStream(StreamIdList) returns (Ack);

rpc modifyStream(StreamConfigList) returns (Ack);

rpc startTransmit(PortIdList) returns (Ack);

rpc stopTransmit(PortIdList) returns (Ack);

rpc startCapture(PortIdList) returns (Ack);

rpc stopCapture(PortIdList) returns (Ack);

rpc getCaptureBuffer(PortId) returns (CaptureBuffer);

rpc getStats(PortIdList) returns (PortStatsList);

rpc clearStats(PortIdList) returns (Ack);

}


Port Class Interface (1/2) class AbstractPort {

public:

// Common functionality for all ports implemented by AbstractPort

int id();

const char* name();

bool modify(const OstProto::Port &port);

int streamCount();

StreamBase* streamAtIndex(int index);

StreamBase* stream(int streamId);

bool addStream(StreamBase *stream);

bool deleteStream(int streamId);

void updatePacketList();

(more)


Port Class Interface (2/2) (contd.)

// Pure virtual functions implemented by platform specific code

virtual OstProto::LinkState linkState() = 0;

virtual bool hasExclusiveControl() = 0;

virtual bool setExclusiveControl(bool exclusive) = 0;

virtual void clearPacketList() = 0;

virtual void setPacketListSize(quint64 size) = 0

virtual void loopNextPacketSet(qint64 size, qint64 repeats,

long repeatDelaySec, long repeatDelayNsec) = 0;

virtual bool appendToPacketList(long sec, long nsec,

const uchar *packet, int length) = 0;

virtual void setPacketListLoopMode(bool loop, quint64 secDelay,

quint64 nsecDelay) = 0;

virtual void startTransmit() = 0;

virtual void stopTransmit() = 0;

virtual bool isTransmitOn() = 0;

virtual void startCapture() = 0;

virtual void stopCapture() = 0;

virtual bool isCaptureOn() = 0;

virtual QIODevice* captureData() = 0;

void stats(PortStats *stats) = 0;

}


AbstractPort

PcapPort

LinuxPort BsdPort WinPcapPort

Port Class Diagram (UML)

DpdkPort

Uses libpcap for packet transmit

and capture


Key Point for transmitted packets

AbstractPort

computes the minimum set of unique packets required to be transmitted

pre-builds this minimum unique packet set

during Tx, this packet set is looped over and over

No packet buffers are allocated or built

during transmit


DpdkPort Engines (1/2)

Port Rx Ring Polling while (!stopRxPolling) {

rte_eth_rx_burst(rxPkts)

foreach pkt in rxPkts rte_pktmbuf_free(pkt)

}

Note: Ostinato, being a traffic generator, does not need to process Rx packets

Port Tx while (!stopTransmit) {

rte_eth_tx_burst(txPkts) // txPkts is pre-built

rte_delay_us(timeTillNextPktOrBurst)

}

Note: ensure rte_eth_tx_burst() does not free mbufs by bumping their refcnt via rte_pktmbuf_clone() during packet list pre-building time so we can keep transmitting the same mbuf again and again without realloc/rebuild


DpdkPort Engines (2/2)

Port Rx Capture while (!stopCapture) {

rte_eth_rx_burst(rxPkts)

foreach pkt in rxPkts

rte_memcpy(capBuf++, pkt)

rte_pktmbuf_free(pkt)

}

Port Statistics Rx/Tx while (!stopStatsPolling) {

rte_eth_stats_get()

sleep(1)

}


Engine - Core assignment strategies

DpdkPort engines to be assigned to cores

Always On (core assignment at init)

Drone core functionality

RPC, Protocol Builders, PacketList Builder etc.

Port Rx Ring Polling

Port Statistics Rx/Tx

On-Demand (core reserved at init or assigned on-demand)

Port Tx (of pre-built packets)

Port Rx Capture


Core assignment example strategy #1

Core Assignment

Master: Drone core functionality + DPDK port stats (one thread polls all ports)

Slave 0: Poll Rx Rings for all ports, Capture Rx for requested ports

Slave 1: Port 1 Tx

Slave 2: Port 2 Tx

…

Slave n: Port n Tx

Pros

Dedicated core to a port Tx

Optimal for port with only 1 Tx queue

Cons

2 cores are potentially under-utilized (Master, Slave 0)

Port Rx + Capture Rx may exceed core capacity leading to packet drops

Slave 1 – n cores may be under-utilized depending on Tx Rate (or if Tx off)

#ports = (#cores – 2)

Quad-core processor can support only 2 ports



Core Assignment


Slave 0: Poll Rx Rings for all ports

Slave 1: Port 1 Tx + Port 1 Capture Rx

Slave 2: Port 2 Tx + Port 2 Capture Rx

…

Slave n: Port n Tx + Port n Capture Rx

Pros

More Rx Capture Bandwidth for simultaneous capture on multiple ports

Optimal when packets transmitted from one port are captured on another

Cons


Port Tx rate may be affected by the multiplexed capture functionality

#ports = (#cores – 2)

Quad-core processor can support only 2 ports



Core Assignment


Slave 0: Poll Rx Rings for all ports, Capture Rx for requested ports

Slave 1: Port Tx Q1 for all ports

Slave 2: Port Tx Q2 for all ports

…

Slave n: Port Tx Qn for all ports

Pros

Multiple cores (equal to no. of supported Tx queues) dedicated to a port Tx

Optimal for ports with multiple Tx queues

All ports supported

Cons


Port Rx + Capture Rx may exceed core capacity leading to packet drops

Slave 1 – n may be under/over-utilized depending on supported Tx queues and Tx rates


The “core” problem

Engine - Core assignment strategy will determine Max number of supported ports

Max packet transmit/capture rate on a port

No one strategy fits all use cases

Optimal solution requires knowledge of system configuration and capacity

use case

Implement multiple strategies and allow

end-user to override the default strategy


DpdkPort Performance

Unfortunately, we don’t have access to DPDK supported hardware NICs and a server-grade multi-core processor

Current development is being done on emulated 82545EM NICs on a VirtualBox VM running on a quad-core laptop

On this development platform, for 64-byte packets, DpdkPort generates a max of ~190Kpps against PcapPort’s ~15Kpps


DPDK Potential Improvements

DPDK Build Environment

Not straight-forward to integrate into existing applications having their own build environment

A traffic generator doesn’t need active Rx Ring polling

If capture is not enabled on port, can PMD/NIC automatically empty rx ring and free mbuf? Note: Rx stats still need to be maintained and collected

Nano-second resolution Delay API

rte_delay_ns()


Source Code

Repo URL: https://code.google.com/r/pstavirs-dpdk/

Caveats

Code is work in progress (aka alpha quality)

Not everything discussed here is implemented as of today (Aug 1, 2014)

https://code.google.com/r/pstavirs-dpdk/




Build Instructions

DPDK

Download DPDK - http://dpdk.org/download

Follow build and verification steps from http://dpdk.org/doc/quick-start

Drone hg clone https://code.google.com/r/pstavirs-dpdk/ ostinato

Export environment variables RTE_SDK and RTE_TARGET

Follow build steps from http://ostinato.org/wiki/BuildingFromSource

http://dpdk.org/download

http://dpdk.org/doc/quick-start






http://ostinato.org/wiki/BuildingFromSource

http://ostinato.org/wiki/BuildingFromSource


Ostinato DpdkPort Screenshot


Support

For programmers who know their “stuff”

Contact the author Srivats P. <[email protected]>

Everyone else

Please wait till the code is ready for “General Availability” (aka beta quality)

mailto:[email protected]


That's all folks! Want to help with DPDK-Ostinato?

dpdk accelerated ostinato

Technology