Thoughts and ideas on distributed security

Dropletdroplet@kernelchina.org

Topics• None-CP design• CP-based design• RTC (run-to-completion) • Event-driven• Parallel vs pipeline • Message and RPC• Locking and IPC• Resource management• Session affinity• Reorder

P1 IOC1

IOC2

IOC3

SPU1

SPU2

SPU3

P2P3

hash1

hash2

hash3

None-CP design

None-CP design highlight

• Intelligence on IOC– IOC distributes packet by hashing

• Simple load balance– Performance depends on traffic pattern

• Need resource synchronization between SPUs– Static allocation on SPUs (Inefficient)– Dynamic allocation on SPUs (Need central

resource management point or resource synchronization between SPUs)

P1 IOC1

IOC2

IOC3

SPU1

SPU2

SPU3

P2P3

CP-based design

CPfirst path

fast path

CP-based design highlight

• Intelligence on CP• CP is responsible for– Session distribution and load balance– Resource management

• CP is the bottle neck?

RTC (run-to-completion)

• FT is RTC (run-to-completion)– It's a big loop, no yield, no scheduling– Watchdog will be triggered if a FT hold CPU 20 seconds

on Australia

• Methods for FT break and follow up– Timer events (used by session scan, HA cold sync etc…)– User events (used by session SZ etc…)

• Timer interrupt would interrupt FT every 10 milliseconds?

Event-driven

• FT is event-driven– FT is polling DPQ– Run callback registered for different DPQ– Locking on queues– Locking between events

Parallel vs pipeline

• All threads run in parallel• For dedicate path, different stage connected

by queues, different threads run in pipeline

ParallelQueue1

Thread1

Queue2Thread2

Queue3Thread3

Multi threads on different queues

Queue4

Thread4

Thread5

Thread6

Multi threads on the same queue

PipelineQueue1 Queue2 Queue3

Thread1 Thread1 Thread1 Thread1

Same thread on different stages

Queue1 Queue2 Queue3Thread1 Thread2 Thread3 Thread4

Different threads on different stages

Parallel vs pipeline• Small and simple tasks– Pros

• Cache friendly

– Cons• Latency• Complicate design and programming

• Big and complicate tasks– Pros

• Easy design and programming

– Cons• Unpredictable runtime

Message and RPC

• Message between IOC/CP/SPU– Kind of RPC– Reliable?– Serialization?– Message type/Message size– Message driven state machine

Locking and IPC

• Spinlock and spinlock enhancement– Spinlock• Busy waiting (no sleep, no timeout)• Get lock in random order• Same priority for all lockers

– Lock order– Lock reentrance

• IPC between FTs– User event

Spinlock enhancement• Read/Write lock– Concurrent read, exclusive write

• Serialization lock– Assign a ticket to locker

• Sequence lock– Writer can interrupt reader

• TryLock– Timeout on a period of time

• RCU– Copy on read, very complicate

Resource management

• Central management (Resource on CP)– Session/resource handled by all threads– Session and resource are handled by different threads– Session CP/Resource CP

• Sparse management (Resource on SPUs)– Static allocation with dynamic allocation

• Resource allocation– Allocate one resource per request– Allocate batch of resources per request

session request

CP threads

resource request

SPU threads

CP session threads

session request

resource request

CP resource threads

Central management

session/resource request

CP threads

Need CPU power?

Waste if there is no resource

request!

Memory issue!One SPU is

special!

Sparse managementSPU1/Resource bundle1 SPU2/Resource bundle2

SPU3/Resource bundle3

Local allocation Local allocation

Local allocation

Remote allocation

Remote allocation Remote allocation

Sparse management

• NUMA?• Remote allocation performance is not predictable • Static resource partition may not fit with SPU session

capacity• Complicate remote allocation mechanism

Resource allocation

Resource repository

Resource client

Request Reply

Allocate one resource per request

Resource repository

Resource client

Allocate batch of resources per request

Request Reply

Local allocation

Resource allocation• Allocate one resource per request– Cons

• Too many messages between resource client and resource repository

– Pros• Simple

• Allocate batch of resources per request– Cons

• Need resource management function on resource client and resource repository

– Pros• Reduce messages between resource client and resource repository

Session affinity

• Typical case– The packets for one session can be handled by any

threads– The threads can handle packets for any session

• Session affinity– One thread handle packets for the session exclusively– Different threads can handle different sessions

simultaneously– Serialized state machine

Session affinity

LBTAssign seq and queue

DPQ

FT

FT

Queue and return

Queue and return

Session queue

FTProcessing packet

FTLBT

Session queue

Serialization by sequence

Serialization by locking and event

Session affinity

• Serialization by sequence– Cons

• Waste

– Pros• LBT simple

• Serialization by locking and event– Cons

• LBT complex

– Pros• Efficient

Reorder

DPQ

FT

FT

FT

Out of order processing

LBTIn order

Tag sequence

POTIn order

Untag sequence

• Global ordering: Tag sequence on receiving • Session based ordering: Tag sequence on matching session • LBT and POT is single thread

Summary

• CP is necessary• Real time/ Multi core programming model• Locking is complicate• Resource management is difficult• Application needs to be multi thread aware• Order is a mandatory requirement

THANK YOU