Here Now - an Open Source Project Near

You

The Linaro LNG Open Data Plane Initiative

Mike Christofferson, Enea

In conjunction with Ola Liljedahl, Arm

FOUNDED

1968

TEN OFFICES

IN NORTH

AMERICA,

EUROPE AND

ASIA

REVENUE

~70 M

USD

NO. OF EMPLOYEES

426

Increasing data traffic in communication devices

require new and innovative software solutions to

handle bandwidth, performance, and power

requirements, as well as scalable systems

management and availability solutions

A robust product portfolio

Enea operating systems software is heavily

used in wireless Infrastructure (Macro, small

cell), gateway, etc. Enea Solutions run in

more than 50% of the world’s 8.2M radio

base stations.

Enea provides a commercial Linux

distribution, built by Yocto, with focus on real-

time

Proven, mature middleware solutions for over

10 years – High Availability, Systems

Management, and real-time database

Global presence, global development, and

headquartered in Stockholm, Sweden

Enea - Powering Distributed, Connected Systems

0

20

40

60

80

100

120

140

160

1995 2000 2005 2010 2015 2020 2025 2030

• Super-linear growth in: – Number of users

– Number of connected devices

– Number of over the top (OTT) applications and protocols

– Number of (standard) protocols (RFC’s)

– Bandwidth usage

– Power consumption of network infrastructure

What Is Happening

Logos and trademarks are used for illustration only and remain the property of their respective owners.

• Increasing and varying QoS requirements – Realtime (e.g. VoIP, video conferencing, gaming)

– Streaming (e.g. music, video)

– Messaging (e.g. IM, Ajax, M2M/IoT)

– Bulk (web data, file sharing, OTA updates)

• More functionality and services implemented in the network – Web caching – Content delivery (CDN) – Intrusion detection and prevention (IDS/IPS) – User-specific service level agreements (SLA)

Increasing Diversity and Functionality

Consequences

• Need flexibility of software and programmable hardware – Trend towards software-realised networking - function defined by

software

• Need familiar programming environments with robust tools – For TimeToMarket-driven development of new protocols and

services

• Need portability – Move functionality and applications between hardware platforms

optimised for different power/performance/cost points

• Need high abstraction – To enable innovation in efficient implementations

– E.g. OpenGL/OpenMAX

• Need efficient support for virtualization – Decouple functionality (SW) from capacity (HW)

– Dynamic partitioning of common HW for different functions

– Simple, robust and incremental deployment of new services

Solution – HW/OS

Develop and deploy networking applications on general purpose processors/architectures • Increasingly ARM and x86

• Users and partners are drawn to the big ecosystems around these architectures

Networking applications running in Linux user space Develop, debug and deploy using standard Linux tools

• Robust user space access to networking HW resources

Linux enhanced to provide bare metal-like environment Bare Metal Linux

Avoid TLB misses, interrupts, context switches, system calls, thread migration

Direct HW access from user space

Applications run isolated in user space on dedicated cores, unaffected by the Linux kernel and other applications

Optional real-time support

As needed by some wireless subsystems (<10μs interrupt response time)

What Is Open Data Plane? • ODP is an open source cross-platform framework

for data plane applications • Common API for application portability • Multiple implementations tuned to different

platforms for performance • Result: Easy app portability and performance

Application Environment

• Applications run in Linux user space with essentially zero system overhead

Open Data Plane Overview

Open Data Plane: The Time has Come • Networking silicon vendors have evolved data plane SDKs for years

– No cross-industry group has sanctioned any common interface on diverse silicon

• The Linaro Networking Working Group - a consortium of 12 networking stakeholders surveyed the open source landscape – Consensus: No ideal “one-size fits all”, implementation for diverse

hardware/software approaches

• A truly open source & open contribution & cross-platform data plane interface, driven by a cross section of stakeholders, is needed

• Based on the OpenGL model: A software API at a higher level of abstraction, that could offer flexibility of implementations underneath that suit diverse needs. – The Linaro non-profit open source software engineering organization is launching

just such a collaboration…

• So Linaro created…OpenDataPlane(ODP).org with charter contributors…

Open Data Plane API

• Standardized data plane API to enable Linux-based networking applications across any architecture – Open support for ARM, Intel, MIPS & PowerPC !

• Structured to enable future innovation – Lightweight abstraction preserves performance without

prescribing lower –level processing structure

– Access and management of HW accelerators

– Supports optional schedulers to provision easy management and traffic load balancing

• Proprietary SDKs sit underneath for OEM/operator software platform simplification (e.g. Supports DPDK on x86, USDPAA on QorIQ, etc)

9

Enabling an efficient, truly cross-platform standardized data plane processing model

Application and services

portability across a choice

of hardware platforms

ODP Foundational Principles

Event Machine – Work-driven many core data plane processing

SoC Abstraction – Portable API’s for access to HW/SoC resources

Bare Metal Linux (a.k.a. Bear Metal Linux)

– Minimal overhead and deterministic execution in Linux user space

EM

SoC

A

BM

L

Application

ODP Foundational Principles (2)

• A data plane/networking API and runtime – Loosely based on the NSN Event Machine

– Event/work-driven and polled programming models

– Portable API’s for accelerators and offloads

– Runs in user space under Bare Metal Linux for best performance and determinism

• Common API, optimised implementations – Separately owned and maintained API (e.g. OpenGL)

– Generic portable reference implementation from Linaro

– HW-optimised (possibly proprietary) implementations from networking SoC vendors

– Linaro maintains ODP for x86/DPDK

API and Concepts

• ODP loosely based on Event Machine, originally developed by NSN

– Generic framework for scalable multi-core programming, not limited to packet processing

• Event based abstraction and programming model for handling IO

– Supports packet flows, physical and virtual network interfaces, accelerators, SW endpoints, etc.

– Events represent different types of data: packets, timers, baseband data, HW notifications, SW messages...

• Supports scheduler based programming model (both HW & SW)

– Scheduling of IO events using different algorithms and knowledge of work in progress

– Implicit synchronisation and mutual exclusion between threads

• Supports different IO load balancing approaches – Chose best configuration for traffic profile, latency/throughput

requirements, and HW characteristics without changing the application

• Proven, already half a dozen HW-specific EM implementations

EM Basic Concepts

Queue groups

Queues with events

Scheduler

Cores/threads associated with queue groups

Idle core

Queues can be dynamically created and added to and removed from queue groups

Cores can be dynamically added to and removed from queue groups

Event handlers associated with queues

IP-fwd

NAT

GTP-U

DPI

RoHC

Work Scheduling

• Actual scheduling algorithms implementation dependent

• Scheduler can enforce ordering/mutual exclusion – Parallel, parallel/ordered and atomic queues

– Application doesn’t need software mutex for protecting per-flow state

• Logical flows/queues mapped to hardware queues (if available)

‘Pull’ work, on demand scheduling

Clusters/Cores/Threads Logical flows/queues thousands to millions

Flows/ QoS classes

WRR

SP

Scheduling algorithms WRR - Weighted Round Robin SP - Strict Priority

Work

Scheduler

Processing packet from flow A

Processing packet from flow B

Idle (power-gated) because of low load

Dynamic vs. Static Load Balancing

• Networking SoC’s have hardware suitable for dynamic load balancing – Queues associated with producer – Queue and buffer mgmt in hardware

• Server NICs designed for termination – Static load sharing (based on hashing) – Queues associated with consumer – Queue and buffer management in

software/shared memory

• Static increases average and worst case latency and buffer space – OK for ~8 cores… but not many-core ready

(some networking SoC’s already have 30+ cores/HW-threads)

• Static makes core elasticity very difficult (per-core state with application level seamless/lossless handovers) – Limits opportunity for power scaling

0

20

40

60

80

100

120

140

1 2 4 8 16 32 64 128

Ave

rage

late

ncy

/uS

Number of cores

Static load-sharing

Dynamic load-balancing

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1

0% 20% 40% 60% 80% 100%

Po

we

r

% Load

Static Load Sharing with typical DVFS

Dynamic Load Balancing Power gating + DVFS

Elasticity and Multi-core Load Balancing

Issues – Traffic load and pattern varies over time

– Industry trend is to use more cores and more power-efficient cores

– To hit the sweet spot PPA (Power/Performance/Area)

– Enabled by inherent parallelism in networking

– Ideally use as many (or few) cores as traffic load and SLA’s require and use them efficiently

ODP Supports hardware scheduler and dynamic load balancing

Cores can be added and removed

No fixed allocation of cores for specific application

Enables power or clock gating of idle cores

Cores can share load dynamically

Increased throughput,

Decreased packet latencies,

Increased core utilization

Scalable and Elastic Timer Support

Issues

• Many protocols need timers, often several timers per flow/connection – Millions of flows in core network means millions of timers

• Timers, like packets, are associated with flows/connections – Need mutual exclusion of flow context when processing a timer event

ODP

ODP schedules timers together with packets Timers and packets can be synchronized and load balanced together

Power and Performance Management

Issues • Traffic load varies

– Daily variation and intermittent bursts

• Use as many or as few cores needed to meet bandwidth and QoS requirements – Add and remove worker threads/cores – Adjust clock frequency of active cores – Power or clock gate inactive cores

ODP Supports power/performance management

Provides API for observing queue lengths Idle worker threads may yield to OS for background tasks or power down core Application can monitor traffic load and quickly react to increasing load

vSwitch Integration

Issues • Efficient and robust integration with software or hardware-accelerated

vSwitch

• No loss of performance for virtualised networking applications using the dataplane API

ODP ODP’s queue-based I/O hides actual device implementation

A queue may represent an actual network interface, a vSwitch port, a pipeline of further processing stages (e.g. for encryption or encapsulation) etc.

Allows for HW to copy packets between application and vSwitch No shared memory between application and vSwitch

Openness and Cross-platform

ODP provides:

Support for multiple architectures and platforms (e.g., ARM, x86, and MIPS)

Open source and an open collaboration Not controlled by any single company

Anyone may join in

Reference implementations are open source

Based on the Event Machine which currently is implemented on a number of different HW targets (using ARM/MIPS/PPC/x86 processors) Proven cross-platform support

Status Core API Definitions

API Component Description Status

BUFFER Shared memory, buffer pools, buffer types and access functions

Preliminary done, but still work in progress

CLASSIFICATION Ingress packet classification Preliminary work underway,

CRYPTO Algorithmic and protocol offload for crypto, hashing, RNG Proposal being implemented

IPC Inter-process communication control plane/data plane TBD

PACKET I/O Network interface abstraction Done

QUEUE Buffer queue management Done

TIMER Protocol timers, periodic ticks Done

SCHEDULER Ingress scheduling and distribution to threads/cores Done

Version 0.2. of the API spec available now Version 1.0 by year end 2014

Status Implementations

Platform Description Status

linux-generic Generic, portable reference implementation, uses Linux facilities (e.g. NetMap, crypto)

Implements BUFFER/CRYPTO/PACKET-IO/QUEUE/TIMER/SCHEDULER

linux-dpdk Implementation for x86 using DPDK as the acceleration layer.

Just started

linux-keystone2 HW-accelerated implementation for TI Keystone2 Tracking linux-generic

linux-qoriq HW-accelerated implementation for FSL DPAA In progress

Other implementations outside LNG also in progress...

• Cisco will demonstrate real app running on multiple HW-implementations of ODP – Usage of API’s

– Usage of HW acceleration through ODP API’s (e.g. ordered and atomic scheduling, crypto)

– Portability

• NSN has had early influence on general architecture and APIs

• Huawei is promoting ODP in public presentations and expressing their support in meetings

• Ericsson’s new influence in this project

Demos at Linaro Connect USA in Sep’ 14 Contributions and Interest from Major TEMs

The ODP API Specification can be influenced by anyone in the open community

What’s next?

• Adding more members to the ODP team – Several companies in discussions, e.g. Aricent, Juniper & others

downloading and commenting on ODP

• Developing NFV PoCs with ARM ecosystem partners, building on ODP – Hardening and optimizing the performance of ODP implementations

• Developing liaisons with OpenDayLight, Open Networking Foundation, NFV Working Group – Additionally, a new open source initiative to integrate open NFV

building blocks with ODP

• Evangelizing ODP in the broad community

Thanks for Attending

For more…. Visit us in booth #201 Go to opendataplane.org linaro.org/projects/networking