fast packet processing in the cloud

Slide title

70 pt

CAPITALS

Slide subtitle

minimum 30 pt

Fast packet processing in the cloud

Dániel Géhberger Ericsson Research

Slide title

44 pt

Text and bullet level 1

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Characters for Embedded font: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´¶·¸¹º»¼½ÀÁÂÃÄÅÆÇÈËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿĀāĂăąĆćĊċČĎďĐđĒĖėĘęĚěĞğĠġĢģĪīĮįİıĶķĹĺĻļĽľŁłŃńŅņŇňŌŐőŒœŔŕŖŗŘřŚśŞşŠšŢţŤťŪūŮůŰűŲųŴŵŶŷŸŹźŻżŽžƒȘșˆˇ˘˙˚˛˜˝ẀẁẃẄẅỲỳ–—‘’‚“”„†‡•…‰‹›⁄€™ĀĀĂĂĄĄĆĆĊĊČČĎĎĐĐĒĒĖĖĘĘĚĚĞĞĠĠĢĢĪĪĮĮİĶĶĹĹĻĻĽĽŃŃŅŅŇŇŌŌŐŐŔŔŖŖŘŘŚŚŞŞŢŢŤŤŪŪŮŮŰŰŲŲŴŴŶŶŹŹŻŻȘș−≤≥fifl

ΆΈΉΊΌΎΏΐΑΒΓΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΪΫΆΈΉΊΰαβγδεζηθικλνξορςΣΤΥΦΧΨΩΪΫΌΎΏ

ЁЂЃЄЅІЇЈЉЊЋЌЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯЁЂЃЄЅІЇЈЉЊЋЌЎЏѢѢѲѲѴѴҐҐәǽẀẁẂẃẄẅỲỳ№

Do not add objects or text in the

footer area

Ericsson Internal | 2016-04-14 |

› Motivation

› Service chains

› Hardware related topics, acceleration

› Virtualization basics

› Software performance and acceleration

› Packet Processing Execution Environment

› Conclusion

Outline

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


› High bandwidth

– Up to 10Gbps but a few 100Mbps should be generally available

› Low latency

– 1 ms 5G radio latency

– Optimal network deployment for low end-to-end latency

– Remote control use-cases

› High spectrum (NX) radio technology, massive MIMO, beamforming

– https://www.youtube.com/watch?v=Sqcv_a9GGUk

› Network slicing

– Create separated networks in seconds

› Fully cloud based deployment from the antenna site

– Network Function Virtualization and service chaining

› Support for IoT (the smart hype)

Motivation? 5G!

https://www.youtube.com/watch?v=Sqcv_a9GGUk

https://www.youtube.com/watch?v=Sqcv_a9GGUk

http://www.google.se/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwiYh_qKlJ3MAhUDVywKHYoeCNgQjRwIBw&url=http://www.cnet.com/news/mobile-industry-dips-its-toes-in-5g-waters-for-next-gen-networks/&psig=AFQjCNEgymDEzk-MRXdq6ZVLXgB6Iu2JwQ&ust=1461239664091184

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


› Splitting up legacy network components

– Core network examples: per user charging, rate limitation

– Radio example: PDCP functionality

› IPSec termination and GTP de-tunneling

› Sequence numbering, buffering

› Header compression

› Air-crypto and transmission towards the antenna site

› Possibly per user service chains

› Dynamic configuration

› 3rd party components

› Optimization for performance

– Hardware capabilities and acceleration, software technologies

– Placement: physical location, noisy neighbors

Service chains

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


› Throughput: 10/25/40/100 Gbps, dual port

› Interconnect

– (RJ45+CAT7 for 10Gbps) / SFP+ / QSFP+

– Latency!

› PCIe counts

– PCIe v2, 16 lanes: 64Gbps

– PCIe v3, 16 lanes: 128Gbps

› Offloading

– Checksum calculation

– TCP/UDP reassembling

– Tunneling

– RDMA over Ethernet

– Receive Side Scaling (RSS)

Modern Ethernet NICS

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


› 1 CPU core can handle ~10Gbps in the best (and simple) case

› Distribute the incoming packets between cores

› RSS hash: IP addresses, transport type, transport ports

– Assign the packet to an RX queue (descriptor ring)

– Default: # of queues = # of CPU cores, interrupts

› Sometimes it cannot work: encrypted IPSec flow

› We have TX queues as well

› SR-IOV uses these as well

NIC Queues

NIC Hash

RX queue 1

RX queue 2

RX queue n

…

CPU core 1

CPU core 2

CPU core n

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


› More than one CPU sockets

› Dedicated memory and PCIe

– Slower access from other sockets

– Where to schedule the packet processing tasks?

– Use the local memory

NON-Uniform memory access NUMA

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


› “Clouds will be built with off-the-shelf hardware elements”

› Partially true, but…

– Amazon, Google, Facebook and the other big players do not really follow this

– They can ask Intel for custom designs and build their own servers

› Hardware acceleration

– (GPU)

– Cryptography

– Packet re-ordering

– Using ARM and Network Processing Units instead of Intel

– Smart NICs

Commodity hardware?

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


› Open vSwitch data path in the NIC

Smart NIC example Netronome

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


› Intel – Ericsson joint datacenter hw/sw development

› Today exists only in a limited version, research topic

› Separated sleds for compute, networking, storage, (memory)

– Optical backplane interconnect

– Smart NICs

› Software defined hardware

– “Give me physical computer with 3 cores, 5GB memory, and 1TB disk”

– CPU slicing

– New type of OS?

– Latencies?

– Virtualization?

Rack Scale architecture Hyperscale datacenter

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


› Linux container – Common kernel

– 1 application

› Docker – Packaging!

› Data to container

Virtualization Technologies

› Library OS – Lightweight kernel

› Only kernel space

– Focus on 1 application

› Can be a JVM

– Well-known hypervisors again!

› Standard VM – Well-known hypervisors

› KVM, Xen, ESX, etc.

– Full kernel

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


Hypervisors

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


Packet forwarding with OVS 2.0

• Figure: mega packet per second!

• 14.88 millions of 64 byte packets per

second on 10G interface

• A packet arrives every ~67.2 ns

• 3.5 GHz 1 cycle = 0.28 ns

• 240 cycles for everyting for line-rate

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


› Simple ping RTT measurement with ~10000 samples

› Not tuned for low latency

Latency

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


› Kernel overhead – System calls

– Context switching on blocking I/O

– Data copying from kernel to user space

– Interrupt handling in kernel

Default packet processing with Linux

User space

Kernel space

NIC

App

Driver

RX/TX

queues

Socket Ring

buffers

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


› Fast packet processing solution from Intel

› Not only for Intel NICs any more

– Other vendors, virtual interfaces

› Set of libraries for applications

– Switch, VNF, etc.

› Polling packets

– Uses a dedicated CPU for 100%

› Direct user-space access

› Lockless buffer rings

› Batch packet handling

› Huge memory pages

› Pre-allocated buffers

DPDK Intel Data Plane Development Kit

User space

Kernel space

NIC

App DPDK Ring

buffers

UIO driver

RX/TX

queues

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


OVS 2.0 with DPDK and KVM

› The price:

– DPDK has to support the NIC

– DPDK aware switch, hypervisor and

application

– 3 CPU cores with a VM

› 1 in the switch for NIC VM

› 1 in the VM

› 1 in the switch for VM NIC

› (So the figure is not completely fair)

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


Power consumption

0

5

10

15

20

25

0 500 1000 1500

kp

ps /

W

Packet size

BananaPI (ARM) Atom Xeon

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


› ER Flow Switch

– 12 Mpps for packet forwarding through a VM or container

– JIT linking

› Packet Processing Execution Environment

– JIT linking, custom callbacks in the switch

› By extending OpenFlow

– Heavy PPs in VMs/containers

– Specific PPs on NPU, ARM, etc.

switching to Ericsson Research

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt





footer area


› 5G networks will be cloud based

– High bandwidth, low latency

› Acceleration is required

– NIC extra features

– Dedicated hardware acceleration

– User space fast packet processing

Conclusion

fast packet processing in the cloud

Documents