FLAREOpen Deeply Programmable Network

Node Architecture

Aki NakaoThe University of Tokyo

2012/10/181

Increase in Data Center Access Traffic

2

http://www.internetevolution.com/author.asp?section id=715&doc id=175123.

http://www.internetevolution.com/author.asp?section id=715&doc id=175123.

World-Wide Mobile Traffic Volume

3Ericsson Japan 「Traffic and Market Report」(2012/6)

Rapid Increase in Mobile “Data” Traffic

Top-10 Spamming BotNetsBotNet Size # of Spams

Grum 600K 40B

Bobax 100K 27B

Pushdo 1.5M 19B

Rustock 2M 17B

Bagle 500K 14B

Mega-D 50K 11B

Maazben 300K 2.5B

Xarvester 60K 2.5B

Donbot 100K 800M

http://www.techrepublic.com/4

How can we resolve newly observed, constantly arising problems in the current Internet?

5

In-Network Services:Cache, Traffic Off-Loading,

In-Network DPI, Network Layer Bot Detection

Multiple Flexibly & Deeply Programmable Networks

6

cloud1

cloud2

ContentOriented

NW

CacheOriented

NW

FutureNW

SensorProcessing

ＮＷ

CloudAccess

ＮＷ

ID/LocNW

LegacyNW

TV Broadcast

NW

Slice 1 Slice 2 Slice N

“Slices”  accommodate  a  variety  of  in-­‐network  servicesSlice  =  a  set  of  resources  reserved  across  mul�ple  network  domains  

Objective: Enabling future network architecture that can be quickly and flexibly programmed to adapt to diverse demands

Smartphones

OpenFlow

7

OpenFlow Controller

Fixed Data Plane Fixed Control Plane

(OpenFlow API)

Flow Pattern Match Actions

•Complex packet processing •Non-IP protocols handling•New classification rules based on more than 12 tuples•Proprietary API definition / dynamic reprogramming of APIs•Arbitrary combination of actions

Physical PortsAlthough OpenFlow enables flexible control of flows, we want more:

OpenFlow Switch

For most of us, OpenFlow/API is at the right level of abstraction

8

However, for some of us, •We should be able to extend API for complex actions•OpenFlow may be forcedly used in an inefficient manner, e.g.,

•copying L7 bits to MAC for control based on those bits...•parsing tuples in a non-standard way to implement VXLAN etc.

SDN and DPN

9

• Control-Plane Programmability • Route Control• Access Control• Network Management

• Data-Plane Programmability• Packet Data Processing

•Cache•Transcode•DPI

• Handling New Protocols• IPvN (N>6)•New Layer2•Content Centric Network (CCN)

• Meta-Control-Plane Programmability•Defining new proprietary APIs

Target Scope of OpenFlow

SDN

OpenFlowwith external processors

Irrelevant withOpenFlow

(+processors)

Scope of Deeply

ProgrammableNetwork(DPN)

DeeperProgrammabilityOut of scope

of OpenFlow

DPN as a super set of SDN

10

Programmability for Data Plane

Programmabilityfor Control Planevia a given API

Programmabilityfor defining an API

for C/D planes

DPN SDN

Making fully programmable network nodes?

11

Programmable Data PlaneProgrammable Control Plane

Physical Ports

•Achieve both programmability and performance at the same time•Instantly upgrade/downgrade switching logics•Enable network virtualization•Make a slice fully programmable (data-plane, control-plane)

Challenges:

Controller

Programmable Node

12

FLARE Open Deeply Programmable Network Node Architecture

FLARE Architecture

Sliver N

Sliver 2

Sliver 1

Packet Slicer

NodeManager

ControllerFLARE Central

Physical Ports13

Slow PathFast Path

Virtual Ports..

FullyProgrammable

FLARE Switch Implementation• Mini 1U / 1U / 2U Form Factor (only 200W)

• A combination of resource containers on many-core processor (fast path)＋x86 processor (slow path)

• 4x10Gbps (20Gbps Non-blocking）, 2x10G+8x1G Planned

• Up to 15 slow-path slivers can be instantiated

• Linux programmability at slow/fast-path slivers and packet slicer

• Parallel programming for high performance at fast-path

• OpenFlow switch logic and API can be programmed

14

FLARE Slivering/Slicing

Slicer Fast Path

SlicerController Slow Path

Fast Path

Slow Path

Physical Network Interfaces

Node Manager

Fast Path

Slow Path

FLARE Central

Create/Remove/Access Sliver

Virtual Network Interfaces Virtual Network Interfaces Virtual Network Interfaces

15

Developer Programs

Sliver=Virtual Machine Environment for Programming (C/D Planes)

Slicer

Slicer Sliver

Fast Path

SlicerContoller Slow Path

Fast Path

Slow Path

Fast Path

Slow Path

Physical Network Interfaces

Sliver1 Sliver2 SliverN

16

PacketSID1

Packet

Packet

SID-1

PacketSID1Packet

Per-Slice Switching

SID

Packet

Arbitrary Offset, Arbitrary Length

PacketSID

General Slicing

17

Extracting Slice ID

Internal Packet Format

Multiple Parts

Multiple Protocols/Controls Coexistence

18

Sliver N

Sliver 2

Sliver 1

NodeManager

ControllerFLARE Central

FLARE Switch

Physical Ports

Fast PathSlow Path

Virtual Ports..

OpenFlow

ExMAC Switch

Ethernet Switch

Packet Slicer

Control Plane VersioningChange according to flows, time, etc

19

Sliver N

Sliver 2

Sliver 1

NodeManager

ControllerFLARE Central

FLARE Switch

Physical Ports

Fast PathSlow Path

Virtual Ports..

OpenFlow v1.0

OpenFlow v2.0

Packet Slicer

OpenFlow v1.1

20

FLARE Controller

Programming Model

21

OFSwitch

FromIO(xgbe1)

ToIO(xgbe2)

FromIO(xgbe2)

ToIO(xgbe1)

Fast Path

Slow Pathofprotocoldpctl NOX

Multi-Threaded Modular Programminge.g., Click Software Modular Router (multi-threaded)

•Arbitrary switch logic(s) can be implemented in fast-path, slow-path and slicer sliver (e.g., OpenFlow switch logic can be modified/refined)

Simple Port Forwarding

22

0

2.5

5.0

7.5

10.0

1  core 2  cores 3  cores

7.2

5.0

2.63.6

2.51.3

64B128B256B512B1024B1514B

512B(Gbps)

NULLFromGxIO(xgbe1)

ToGxIO(xgbe2)

Forwarding Performance

23

Ethernet Switch

0

2.5

5.0

7.5

10.0

1  core 2  cores 3  cores 4  cores 5  cores

9.99.99.99.99.9 9.99.99.9

7.8

4.2

pkt_size=512Bpkt_size=1514B

(Gbps)

EtherSwitch

FromGxIO(xgbe1)

ToGxIO(xgbe2)

FromGxIO(xgbe2)

ToGxIO(xgbe1)

Switching Performance

OpenFlow Switch

OFSwitch

FromIO(xgbe1)

ToIO(xgbe2)

FromIO(xgbe2)

ToIO(xgbe1)

Data Plane

Control Planeofprotocoldpctl NOX

0

2.5

5.0

7.5

10.0

1  core 2  cores 3  cores 4  cores 5  cores

9.99.99.9

7.9

4.2

6.7

5.13.8

2.71.5

pkt_size=512Bpkt_size=1514B

(Gbps)

Switching Performance24

How deep programmability do we want?

• Control plane programmability only?

• Data plane too (cache, transcode, DPI)?

• Can we define a new L2 protocol?

25

Several questions to ask:

example1Redefining L2 protocol

26

A Case in Data Center Network

• Limitation in MAC address space• Conflict of MAC addresses in VM migration

• Limitation in VID (802.1Q) space• The number of tenants increases in IaaS

Data Center Network depends heavily on L2leading to solutions such as EUI-64 and VXLAN

27

Mac Address Extension

http://en.wikipedia.org/wiki/MAC_address28

Why not just extend MAC ID spacethrough deep programmability

enabled by FLARE Switch?

29

Slice 3

Slice 2

Slice 1

FLARE Switch

Extended MAC Address (96-bit MAC)

Extended MAC Switching

(Click Implementation)

Extended MAC (96bit) Switching

End System 1 End System 2

IP Slice VExDMAC ExSMAC T

IPExDMAC ExSMAC T

Slice 1

Slice 1

HyperVisor

30

Extended MAC Address (96-bit MAC)

IPExDMAC ExSMAC T

Slice 1

Extended MAC Switching

(Gbps)

Switching Performance

0

2.5

5.0

7.5

10.0

1  core 2  cores 3  cores 4  cores 5  cores

9.99.99.99.99.7 9.99.98.7

6.1

3.2

pkt_size=512Bpkt_size=1514B

ExMACSwitch

FromGxIO(xgbe1)

ToGxIO(xgbe2)

FromGxIO(xgbe2)

ToGxIO(xgbe1)

32

ExDMAC ExSMAC

TYPE IP

Extended DMAC Extended SMAC Type IP Datagram

GuestVM

Extended MAC1

GuestVM

Extended MAC2

GuestVM

Extended MAC1

GuestVM

Extended MAC2WAN

Migration

FLARE FLARE

33

VM Migration With Extended MAC

example2Trailer Slicing

34

Light Weight Network Virtualization(As Application of FLARE to NV)

• Tunneling (Good for Fixed Networks)

• VLAN （Limited to Ethernet/ 12bits)

• MPLS (Originally Traffic Engineering)

Essentially just need globally unique Slice ID35

Slice = A Set of Slivers

36

FLARE Controller

FLARE SwitchFLARE AP

SID

Packet

Arbitrary Offset, Arbitrary Length

PacketSID

General Slicing (Supported in DarkFlow Version)

37

Extracting Slice ID

Internal Packet Format

Multiple Parts

38

Packet SID

Slicer-slice identifies Slice ID(SID)Globally Unique SIDs Embedded in Trailer

Trailer Slicing

Network VirtualizationThrough Slice ID Embedded in Packet Trailers

39

Packet SID

Packet SID

Packet SID

Globally UniqueSlice ID

IsolatedNetwork

NameSpace

Header Trailer

•SID visible at all the layers (if layered architecture is adopted at all)

•No need to create complex tunnels (Convenient for mobile terminals/handsets)

Extended MAC (96bit) Switch

IP Slice V

IP Slice VExDMAC ExSMAC T

TSMACDMAC

IPExDMA ExSMAC T IPTSMACDMACIPTSMACDMACIPExDMA ExSMAC T

Slice 3

Slice 2

Slice 1

FLARE Switch

Extended MAC Address (96-bit MAC)

End System 1 End System 2MAC Addresss

Slice 2Slice 2 Slice 1Slice 1

Slice 1Slice 2

HyperVisor

Extended MAC Address MAC Addresss

Extended MACSwitching

EthernetSwitch

mux/demuxmux/demux

Network NameSpace Full Virtualization

Slice 3

Slice 2

Slice 1

HyperVisor

FLARE Switch

Ethernet Switching

Terminal1 Terminal2

IP Slice V

IP Slice VDMAC SMAC T

TSMACDMAC

Resource Containers

in two different containers

Ethernet Switching

IPDMAC SMAC T IPTSMACDMAC

Resource Containers

Slice 1

Slice 2

Slice 2Slice 1

mux/demuxmux/demux

IPDMAC SMAC T IPTSMACDMAC

Slice 2Slice 1 41

Beauty of Trailer Slicing

Slice 3

Slice 2

Slice 1

FLARE Switch

Ethernet Switching

Terminal1 Terminal2

IP

IPDMAC SMAC T

TSMACDMAC

Ethernet Switching

Slice V

Slice V

We can insert existing ethernet switch(es)! Slice 1

Slice 2

HyperVisor

42

43

slice 1 slice 2

eth0

veth0 veth1

wlan0

veth2 veth3

eth0 eth1 eth0 eth1

slicer classifier

Slicer Slice

FLARE Wireless LAN AP

Android Handsetsslice 2slice 1

Single SSIDIP Slice VIP Slice VDMAC SMAC T TSMACDMAC

HandSet Slicing

44

• Slice Per Device • Slice per OS (Android OS Virtualization)• Slice per Application

45

Slice Selector (IPv4/IPv6)

OpenFlow vs. FLARE DifferencesOpenFlow FLARE

Protocol Internet Protocol Arbitrary Frame(even non-standard)

Control PlaneProgrammability O Multiple APIs

Multiple OpenFlow SWs

Data PlaneProgrammability(Packet Process)

External O

Data PlaneProgrammability(New Protocol)

External O

46

OpenFlow vs. FLARE Differences

OpenFlow FLAREProgramming

ModelFlow Pattern Match

ActionLinux Container (LXC)Multi-Threaded Click

ProcessingEnvironment Not Integrated Integrated

Slicing Programmability Flow Matching Deep Packet Inspection

47

Conclusion

• Deeply Programmable Network (DPN) is considered extension to Software Defined Network（SDN) and an important topic to explore

• A network node architecture FLARE pursues more flexible programmability yet reasonable performance

48