ebpf and xdp in suricata reloaded€¦ · test data using a test pcap of 445mo. real trafﬁc but...

eBPF and XDP in Suricata reloaded

É. Leblond/P. Manev

Stamus Networks

Nov. 1, 2019

É. Leblond/P. Manev (Stamus Networks) eBPF and XDP in Suricata reloaded Nov. 1, 2019 1 / 49

What, why them again? it’s not Vancouver here

Vancouver was just the beginning of the trip

1 ProblemPacket loss impactElephant flow

2 Going bypassAF_PACKET bypass via eBPFXDP support

3 Offloading bypass in hardware

4 Experimentation at LRZ

5 Future

6 Conclusion

5 Future

6 Conclusion

Impact of loosing packets

MethodologyUse a sample trafficModify the pcap file to have specified random packet lossDo it 3 times par packet lossGet graph out of that

Test dataUsing a test pcap of 445Mo.Real traffic but lot of malicious behaviorsTraffic is a bit old

Alert loss by packet loss

Some numbers10% missed alerts with 3% packets loss50% missed alerts with 25% packets loss

The case of file extraction

Some numbers10% failed file extraction with 0.4% packets loss50% failed file extraction with 5.5% packets loss

5 Future

6 Conclusion

The elephant flow problem (1/2)

Ring buffer overrunLimited sized ring bufferOverrun cause packets lossthat cause streaming malfunction

Ring size increaseWork aroundUse memoryFail for non burst

Dequeue at NQueue at speed N+M

5 Future

6 Conclusion

5 Future

6 Conclusion

Suricata eBPF bypass architecture

Bypass

eBPF bypassSuricata specialized filterFlow tables for IPv4 and IPv6Bypass function add entry to flow table

Flow handlingAt timeout of flow, fetch the corresponding entries in the tableCompare counters, remove entries if no update or updatecounters

5 Future

6 Conclusion

eXtreme Data Path

Reaching bare metal performanceAnswer to high performance need

DDoS fightCustom protocol implementation

Run userspace codeWhen Linux network stack do too much

MotivationAvoid cost of skb creation"Kill" DPDK

Universal solution and APIsAvoid non Linux application on Linux

XDP explained

A recent Linux kernel feature

Run a eBPF code the earliest possiblein the driverin the cardbefore the regular kernel path

Act on dataDrop packet (eXtreme Drop Performance)Transmit to kernelRewrite and transmit packet to kernelRedirect to another interfaceCPU load balance

Implementation in Suricata

Similar to eBPF filterSame logic for bypassOnly verdict logic is different

But annoying differenceeBPF code does the parsingNeed to bind to an interface

Suricata XDP architecture

AF_PACKET IPS mode

IPS and bypassCan’t drop packet at bypass stageNeed to forward from one iface to another

AF_PACKET IPS mode

IPS and bypassCan’t drop packet at bypass stageNeed to forward from one iface to another

XDP and IPS mode: bypass

Use TX_REDIRECTDirect copy from interface to interface

Results

Direct NIC to NIC transferSkip all kernel taskWire speed copyIf eBPF code is fast enough

Obtained performanceTODO: Ask Brad Woodberg to update his tests

TLS bypass

Bypass counter shows a lot of TLS is not capture bypassedProblem with short session that are already in buffer at bypasstimeBypass is done too late

XDP bypass of TLS

Conditional XDP bypass of encrypted traffic

# i f ENCRYPTED_TLS_BYPASS/∗ Packet to or from por t 443 ∗ /i f ( ( dpor t == __constant_ntohs (443 ) ) | |

( spo r t == __constant_ntohs ( 4 4 3 ) ) ) {__u8 ∗app_data ;/∗ Let ’ s jump to data ∗ /nh_of f += s i z e o f ( s t r u c t i phd r ) + s i z e o f ( s t r u c t tcphdr ) ;/∗ Please eBPF v e r i f i e r and V i c t o r w i th defens ive code ∗ /i f ( data_end > data + nh_of f + 4) {

app_data = data + nh_of f ;/∗ Drop a p p l i c a t i o n data f o r t l s 1.2 ∗ /i f ( app_data [ 0 ] == 0x17 &&

app_data [ 1 ] == 0x3 && app_data [ 2 ] == 0x3 ) {r e t u r n XDP_DROP;

5 Future

6 Conclusion

HW offload in Netronome (quoting Simon Horman)

The Netronome CX SmartNICs feature a network flow processor(NFP, or more commonly NPU).BPF programs is JITed to the instruction set of the NFP, which isanalogous to JITing programs to the host instruction set, sayx86_64 or aarch64.The JITed program is then loaded onto the NFP where it runsnaively.

Suricata XDP HW architecture

RSS queue assignement in eBPF code

Programmable RSS load balancingNetronome cards allow you to set target RSS queues in eBPFLet’s fix the wrong thread problem (https://redmine.openinfosecfoundation.org/issues/2725)I was ready to suffer but a few lines later

/∗ IP−pa i r s + p ro toco l (UDP/TCP/ ICMP) h i t same CPU ∗ /__u32 xdp_hash = tup le . s rc + tup l e . ds t ;xdp_hash = SuperFastHash ( ( char ∗)&xdp_hash , 4 ,

INITVAL + iph−>p ro toco l ) ;c tx−>rx_queue_index = xdp_hash % RSS_QUEUE_NUMBERS;

RSS queue assignement in eBPF code

Programmable RSS load balancingNetronome cards allow you to set target RSS queues in eBPFLet’s fix the wrong thread problem (https://redmine.openinfosecfoundation.org/issues/2725)I was ready to suffer but a few lines later

/∗ IP−pa i r s + p ro toco l (UDP/TCP/ ICMP) h i t same CPU ∗ /__u32 xdp_hash = tup le . s rc + tup l e . ds t ;xdp_hash = SuperFastHash ( ( char ∗)&xdp_hash , 4 ,

INITVAL + iph−>p ro toco l ) ;c tx−>rx_queue_index = xdp_hash % RSS_QUEUE_NUMBERS;

5 Future

6 Conclusion

This talk...

Is only possible with community and vendor help !Thank you!

LRZNetronomeNapatechAccolade

The effort achieved major break through at 1/2/3am...Cpl PhDs-wise time effort worth so far

7.5 million ppsNasty network

Asyn/public/private IPsOne way trafficNot complete 3 way handshakesElephant flowsGo ahead make my day..

Close to 40Gbps university trafficXDP in HW / XDP in SW

Traffic

Just RTFM and start tuning

https://suricata.readthedocs.io/en/suricata-5.0.0/capture-hardware/ebpf-xdp.html#hardware-bypass-with-netronome

The truthIn fact not, we did write the code and the doc while doing the test

Just RTFM and start tuning

https://suricata.readthedocs.io/en/suricata-5.0.0/capture-hardware/ebpf-xdp.html#hardware-bypass-with-netronome

The truthIn fact not, we did write the code and the doc while doing the test

Increase flow tables size for IPv4 and IPv6

Htop with XDP driver mode on Mellanox

Htop with XDP hardware mode on Netronome

perf top with XDP driver mode on Mellanox

perf top with XDP hardware mode on Netronome

bpftool (starting to be standard packaged in Eoan)Quentin Monnet’s twitter doc https://twitter.com/qeole/status/1103688642701217794?lang=en

Brendan Gregg http://www.brendangregg.com/blog/2019-07-15/bpf-performance-tools-book.html

perf top/stat/recordHW Datasheets (CPU/NIC/etc)Dedicated HW for a long period of time (BIOS access is helpfultoo)Advanced knowledge of the setupChange one variable at a time and run a test

Netronome ethtool stats

Compile bpftool

bpftool and jq

Bfpftool show program

The good news

32 cores are dealing with 35Gbps University traffic with fullETPRO ( 50k sigs)Basically 1 core per 1 Gbps for a CPUIntel(R) Xeon(R) Platinum 8176 CPU @ 2.10GHz (28 core/socket)32 cores/32 RSS/32 suricata worker threads/AFPv3Netronome 40Gbps Agilio/eBPF enabled firmwareXDP offer huge flexibility via eBPFpinned maps save state cross Suricata restarts (major boost forIPS as well)

It’s not all roses (just yet)

SuricataSome counters seem off (including in Suricata)Suricata’s config/setup on higher speeds becomes more andmore complexAt higher speeds log writing seems to get a lot of log file lockcontention - directly affecting drops

OSThough to profile and pinpoint/ investigate the end-to-endpath/reason (tooling)Needs more investigation of the performance ‘hot spots‘

It’s not all roses (just yet)

XDPRSS/threads need to be power of 2 (32/64...)MTU 3000B , "one frame cannot exceed a page"

Test/QA can benefit fromDefined/repeatable test framework/log formats etcTroubleshooting and pin pointing hot spots requires common effortCold War style environment - switch off chats/phones/satellitedishes/TV and Radio sets and go underground

LLC Load misses

Busy thread

Some top level sigs

Worker threads still pegged

5 Future

6 Conclusion

Upcoming

Full test and configuration data release (2020)Napatech NIC testing and added to the mix (just released newdriver/code)Doc update for SuricataSEPTun III planning around Feb 2020 (cc @MichalPurzynski)

Future dev work

AF_XDPAF_XDP capture is now in Linux vanillaeBPF code send data to userspaceKernel work is skipped

Some missing featuresTimestamp are missing

XDP explained: AF_XDP

5 Future

6 Conclusion

Conclusion

Suricata, eBPF and XDPAvailable in Suricata 5, need Linux 4.16Network card bypass for Netronome

More informationSuricata doc: http://suricata.readthedocs.io/en/latest/capture-hardware/ebpf-xdp.html

XDP whitepaper:https://www.stamus-networks.com/2019/07/16/whitepaper-introduction-to-ebpf-and-xdp-support-in-suricata/

Questions ?

Thanks toJesper Dangaard BrouerAlexei StarovoitovDaniel Borkmann

Contact UsTwitter: @pevma @regiteric

ebpf and xdp in suricata reloaded€¦ · test data using a test pcap of 445mo. real trafﬁc but...

Documents

ground trafﬁc surveillance system for air trafﬁc control

the project life + manev€¦ · "the project life manev:...

trafﬁc - gambling insider

logging with netfilter and ulogd2...logging with netﬁlter...

bnb reserves cvetan manev

simulation-based trafﬁc assignment

bicycle trafﬁc signals · march 2018 march 2018 bicycle...

rail-system operator telecommunications · efﬁciently...

life+ manev: evaluation of manure management systems in...

encrypted network traffic hunting threats that use ·...

my candidature, by borislav manev

impact of mobility-on-demand on trafﬁc congestion ... ·...

text detection and recognition on trafﬁc panels from...

hierarchical trafﬁc grooming in wdm networks€¦ ·...

ГЛАВА 1. ВЪВЕДЕНИЕ В...

ФРАКТАЛНИ...

self-organized trafﬁc control

explosive handling safety review training · 2020. 1....

suricata idps and nftables: the mixed mode...suricata idps...

traffic performance score for measuring the …2 literature...