1

Aaron Gember, Chaithan Prakash, Raajay Viswanathan, Robert Grandl,

Junaid Khalid, Sourav Das, Aditya Akella

OpenNF

2

SDN + software NFs• NFs examine/modify packets at layers 3-7

• Software NFs are replacing physical appliances• SDN applications (PLayer, SIMPLE, Stratos, etc.)

steer flows through NFs

Web ServerHomeUsers

CachingProxy

IntrusionPrevention

Firewall

Enables new applications that control the packet processing happening across instances of an NF

3

• Not moving flows => bottleneck persists• Naively moving flows => incorrect NF behavior

Example: scaling & load balancing

Firewall CachingProxy

IntrusionPrevention

Web ServerHomeUsers

Requires a control plane that enables management of both internal NF state and network forwarding state

4

Challenges

1. Dealing with race conditions– Packets may arrive while state is being moved,

causing state updates to be lost or re-ordered2. Giving applications flexibility– May need to move state at different granularities

3. Supporting many NFs with minimal changes– Undesirable to force NFs to conform to certain

state structures or allocation/access strategies

5

OpenNF

OpenNF Controller SDN Controller

Control Application

Northbound API

Southbound API

6

Outline

• Overview• Requirements• Design– Southbound API (addresses NF diversity)– Northbound API (addresses race conditions)

• Evaluation

7

Requirements

• Move flow-specific NF state at various granularities

• Copy and combine, or share, NFstate pertaining to multiple flows

• Support key guarantees (no loss, order preserved) when needed

• Track when/how state is updated

8

Existing approaches

• Control over routing (PLayer, SIMPLE, Stratos)• Virtual machine replication– Unneeded state => incorrect actions– Cannot combine => limited rebalancing

• Split/Merge and Pico/Replication– Address specific problems => limited suitability– Require NFs to create/access state in specific

ways => significant NF changes

9

• State created or updated by an NF applies to either a single flow or a collection of flows

• Classify state based on scope• Flow provides a natural way for reasoning about

which state to move, copy, or share

NF state taxonomy

Connection

Connection

TcpAnalyzer

HttpAnalyzer

TcpAnalyzer

HttpAnalyzer

Per-flow state

ConnCount

Multi-flow state

All-flows stateStatistics

10

API to export/import state

• Three simple functions: get, put, delete– Version for each scope (per-, multi-, all-flows)– Filter defined over packet header fields

• NFs responsible for– Identifying and providing all state matching a filter– Combining provided state with existing state

No need to expose internal state organization No changes to conform to a specific allocation strategy

11

API to observe/prevent updates

• Problem: need to prevent (e.g., during move) or observe (e.g., to trigger copy) state updates

• Solution: event abstraction– Functions: enableEvents and disableEvents– Instruct NF to raise an event and process, buffer,

or drop packets matching a filter

Only need to change an NF’s receive packet function

12

Move operation

OpenNF Controller

Control Application

move (port=80,Inst1,Inst2)

getPerflow(port=80)

[Chunk1] putPerflow(Chunk1)delPerflow(port=80)[Chunk2] putPerflow(Chunk2)

forward(port=80,Inst2)

SDN Controller

Inst2Inst1

13

Packet arrivals during move

• Packets may arrive during a move operation

• Fix: suspend traffic flow and buffer packets– May last 100s of ms => connection timeouts– Packets in-transit when buffering starts are dropped

111

Inst2 ismissingupdates

Inst2Inst1

move(yellow,Inst1 ,Inst2 )

Loss-free: All state updates due to packet processing should be reflected in the transferred state, and all

packets the switch receives should be processed

Use events for loss-free move

• enableEvents(blue,drop) on Inst1;

• get/delete on Inst1; put on Inst2

• Buffer events at controller• Flush packets in

events to Inst2

• Update forwarding

14

S Inst2Inst1

AS

S S,S+AS+A

S+A

S,S+A,A

15

Re-ordering of packets

Order-preserving: All packets should be processed in the order they were forwarded

to the NF instances by the switch

Controller Switch Inst2

Flush buffer

Request forwarding update

Inst1

S+A

S+AA

AD1

D2D1

D1

D1

S+AA

D2

D1

• Flush packets in events to Inst2

• enableEvents(blue,buffer) on Inst2

• Forwarding update: send to Inst1 & controller• Wait for packet from

switch (remember last)• Forwarding update:

send to Inst2

• Wait for event for last packet from Inst2

• Release buffer of packets on Inst2

Order-preserving move

16

S

SS,S+A

S+A

S,S+A,AA

AAD1

S,S+A,A,D1

17

Copy and share operations

• Used when multiple instances need to access a particular piece of state

• Copy – no or eventual consistency– Issue once, periodically, based on events, etc.

• Share – strong or strict consistency– All packets reaching NF instances trigger an event– Packets in events are released one at a time– State is copied between packets

Example app: Load balanced network monitoring

movePrefix(prefix,oldInst,newInst): copy(oldInst,newInst,{nw_src:prefix},multi) move(oldInst,newInst,{nw_src:prefix},per,LF+OP) while (true): sleep(60) copy(oldInst,newInst,{nw_src:prefix},multi) copy(newInst,oldInst,{nw_src:prefix},multi)

scan.brovulnerable.broweird.bro

Example app: Selectively invoking advanced remote processing

enhanceProcessing(flowid,locInst): move(locInst,cloudInst,flowid,per,LF)

scan.brovulnerable.broweird.bro

scan.brovulnerable.broweird.brodetect-MHR.bro!

20

Implementation

• OpenNF Controller (≈3.8K lines of Java)– Written atop Floodlight

• Shared NF library (≈2.6K lines of C)• Modified NFs (3-8% increase in code)– Bro (intrusion detection)– PRADS (service/asset detection)– iptables (firewall and NAT)– Squid (caching proxy)

End-to-end benefits

• Load balanced monitoring with Bro IDS– Load: 10K pkts/sec cloud trace– After 180 sec: move HTTP flows (489) to new Bro

• OpenNF: 260ms to move (optimized, loss-free)– Log entries equivalent to using one instance

• VM replication: 3889 incorrect log entries• Forwarding control only: scale down delayed

by > 1500 seconds

22

Southbound API call processing

Serialization/deserialization costs dominate

Cost grows with state complexity

Efficiency with guarantees• State: 500 flows in PRADS; Load: 1000 pkts/s• Move

• Copy – 176ms• Share – 7ms (or more) for every packet

23

194 pktsdropped!

130 pktsbufferedat dstInst

230 pktsin events

Guarantees come at a cost!

24

Controller performance

Improve scalability with P2P state transfers

25

• Systematic engineered APIsimplemented by NFs and used by control applications

• Enables rich control of the packet processing happening across instances of an NF

• Provides key guarantees andrequires minimal NF modifications

Conclusion

http://agember.com/go/opennf