technical university of crete packet pre-filtering for network intrusion detection ioannis sourdis,...
TRANSCRIPT
Technical University of Crete
Packet Pre-filtering for Network Intrusion Detection
Ioannis Sourdis, Vasilis Dimopoulos,
Dionisios Pnevmatikatos and Stamatis Vassiliadis
CE, TU Delft, the NetherlandsECE, TU Crete, Greece
ICS-FORTH, Greece
5.12.2006 © Ioannis Sourdis 2
TU Crete
Introduction
Intrusion Detection System (IDS):
Packet Classification (Header Matching)
Payload Scan (Pattern Matching)
Related work on Pattern matching & Packet Classification
Internet
Where From?Where to?
What Content?
Packets
Is this Packet “suspicious”?
IDS
Is that enough for the core of a high-speed IDS???
5.12.2006 © Ioannis Sourdis 3
TU Crete
Motivation
IDS rulesets contain thousands of rules (SNORT IDS: 3,000-4,000)
IDS rule syntax becomes more complicated:
Significant Cost
Performance limitation
The high-speed Pattern matching techniques are required but are not adequate to accommodate the new IDS syntax features
HEADER
P1
payload
depth
HEADER
P1 P2min Distance
offset
within
5.12.2006 © Ioannis Sourdis 4
TU Crete
What do we need?
Need for different levels of processing packets
1st Level: use a simplified version of the rules It’s a Fact: Not ALL rules will match every single packet Exclude the majority of the rules
2nd Level: use advanced attack descriptions for the remaining rules
Reduce the cost and maintain high performance
Incoming Packets
Candidate rules IDs 2nd Level
Complete Match Enginefor the reported rules
(HW or SW)
Matching Rules IDs1st Level
Packet Pre-filtering
5.12.2006 © Ioannis Sourdis 5
TU Crete
How to accomplish this?
We are looking for a fast-inexpensive way to exclude/ “filter-out” for each incoming packet most of the IDS rules + point out a small subset of possibly matching rules?
Partially match each rule of the set
Considering only header info per rule groups of hundreds of rules. Many rules may have the same header description!
Suggestion: header + a small portion of payload pattern in the filter
Rule #1
Rule #2
Rule #3
.
.
.
.
.
.
.
Rule #N
Rule #N+1
.
.
Rule #M
IDS ruleset
Possibly matching
rules
5.12.2006 © Ioannis Sourdis 6
TU Crete
Packet Pre-filtering
How: Put a small FIRST part of each rule in the pre-filtering Match (if any) the header description of the rule (Source and
Destination Address, protocol) Match a prefix of the first payload pattern of the rule (constant
number of bytes)
IF for a packet, this part of the rule matches the rule needs to be fully matched
ELSE the rule is excluded/filtered-out
Packet Pre-filtering
Incoming Packets
Candidate rules IDs
Complete Match Engine
for the reported rules(HW or SW)
Matching Rules IDs
5.12.2006 © Ioannis Sourdis 7
TU Crete
Example
IDS Ruleset:1. Rule(1): header: H(1) payload: P(1)+[pattern_suffix]2. Rule(2): header: H(1) payload: P(2)+[pattern_suffix]…N. Rule(N): header: H(1) payload: P(N)+[pattern_suffix]
P(1)…P(N): prefixes of payload patterns (i.e. 4-10 characters long)
The pre-filtering won’t be efficient if incoming packets look like this:
H(1) P(1) … P(2) … P(3) … P(N) …
Candidate: Rule(1) Rule(3) ……..Rule(N)Rule(2)
5.12.2006 © Ioannis Sourdis 8
TU Crete
Example
IDS Ruleset:1. Rule(1): header: H(1) payload: P(1)+[pattern_suffix]2. Rule(2): header: H(1) payload: P(2)+[pattern_suffix]…N. Rule(N): header: H(1) payload: P(N)+[pattern_suffix]
P(1)…P(N): prefixes of payload patterns (i.e. 4-10 characters long)
The pre-filtering will be efficient if incoming packets look like this:
The less rules activated per single packet the most efficient the pre-filtering
H(1) P(1) … P(2) …
Candidate: Rule(1) Rule(2)
5.12.2006 © Ioannis Sourdis 9
TU Crete
HW Implementation
Field extractor: Extract header and payload
Payload Matching: Prefix of the 1st payload
pattern (i.e. 2-10 chars) Implementation DCAM Regular Expressions may also
be used (matching again Reg. Expr. prefixes)
Field Extractor
Header
Payload
Incoming Packets
Partial PayloadMatching Rule 0
Rule 1
Rule 2
Rule N
Priority Encoder
Header Matching
Bitmask
Candidate rules IDs
Header matching: Src/Dest IP+Port, Protocol Implementation simple
comparators
Bitmask, each bit corresponds to a rulePriority Encoder: Pipelined, encodes/outputs every SET bit of the bitmask.
5.12.2006 © Ioannis Sourdis 10
TU Crete
2nd Level: Complete Match Engine
Packet Pre-filtering
Incoming Packets
Candidate rules IDs
Complete Match Engine
for the reported rules(HW or SW)
Matching Rules IDs
HW or SW? SW assign in multiple threads to match the candidate rulesNetwork processor multiple processing engines...Fully customized HW system
We propose (not excluding other solutions):Reconfigurable Hardware: HW performance, flexibility, reconfiguration to update the ruleset
5.12.2006 © Ioannis Sourdis 11
TU Crete
Reconfigurable IDS core Organization
Pre-filtering points out the rules to be fully matchedSpecialized Engines: For each candidate rule:
A PE is reserved A firmware is transferred to the PE PE released EoP or rule mismatch
Coprocessors (Static patterns & Regular expression matching) perform payload scanPEs select the coprocessor info and decide whether a rule matches or not
Pre-Filtering Specialized Engines
PE PE PE PE
PE PE PE PE
PE PE PE PE
FirmwareMemory
Coprocessors
OUTPUT: MATCHING rule ID
Header Matching
Partial PayloadMatch
Priority Encoder Possible Match
Rules
Static Patterns
Regular Expressions
INCOMINGPackets
MATCHING rule ID
Output I/F
MATCHING rule ID
5.12.2006 © Ioannis Sourdis 12
TU Crete
What if (Candidate rules > PEs) ?
What does (Candidate rules>PEs) mean?
A single packet partially matches more than x rules (e.g. x=32 or 64)
Can such a packet be a normal packet?
What happens when (Candidate rules>PEs)?
In order to guarantee performance, the packet is reported, Admin policies determine the next step (i.e. drop)
Pre-Filtering Specialized Engines
PE PE PE PE
PE PE PE PE
PE PE PE PE
FirmwareMemory
Coprocessors
OUTPUT: MATCHING rule ID
Header Matching
Partial PayloadMatch
Priority Encoder
Possible Match Rules
Static Patterns
Regular Expressions
INCOMINGPackets
MATCHING rule ID
Output I/F
MATCHING rule ID
5.12.2006 © Ioannis Sourdis 13
TU Crete
Experimental Results
Defcon11 traces9 trace files~10 millions packets4.6 million packets have payloadpayload length:
Mean 698 bytes Max 1460 bytes
Milions of packets per trace
0
0.5
1
1.5
2
eth0
znb0
znb1
znb2
znb3
znb4
znb5
znb6
znb7
Millions
Header Only With payload
SNORT v2.43,191 rules
2,271 rules with payload description
920 only header
5.12.2006 © Ioannis Sourdis 14
TU Crete
Simulation Results
Pre-Filtering setup: Header matching Scr/dest IP+Port, Protocol Payload Pattern match 2-10 chars prefix match
For prefix>2 chars: Average Candidate rules per packet=[1,3] per traceOverall average: 1.8 rules per packet Only header match ~45 rules per packet
Average Candidate Rules After Pre-filtering
0
2
4
6
8
10
12
eth0
znb0
znb1
znb2
znb3
znb4
znb5
znb6
znb7
4 bytes6 bytes
8 bytes10 bytes
2 bytes
5.12.2006 © Ioannis Sourdis 15
TU Crete
Simulation Results
Payload prefix match= 2 chars: max 63 candidate rules per packetsPayload prefix match>=4 chars: max 32 candidate rules per packets
What does this mean:Max number of rules for further processing1% or 32 out of 3,200 rulesThe Max degree of parallelism needed (processing engines, threads etc.)
Maximum Candidate Rules After Pre-filtering
0
8
16
24
32
eth0
znb0
znb1
znb2
znb3
znb4
znb5
znb6
znb7
4 bytes
6 bytes
8 bytes
10 bytes
5.12.2006 © Ioannis Sourdis 16
TU Crete
Implementation Results
Field Extractor
Header
Payload
Incoming Packets
Partial PayloadMatching Rule 0
Rule 1
Rule 2
Rule N
Priority Encoder
Header Matching
Bitmask
Candidate rules IDs
0 2K 4K 6K 8K 10K 12K 14K 16K
8-bit
32-bit
Dat
apat
h w
idth
Slices
field extractor Pattern matching header matching Priority Encoder Control
Packet Prefiltering Area Cost
Datapath 8 bits/cycle:Virtex2: 2.7 GbpsVirtex4: 4 GbpsArea 11K slices (medium-small FPGA)
Datapath 32 bits/cycle:Virtex2: 9.7 GbpsVirtex4: 14 GbpsArea 15K slices (medium-small FPGA)
Priority encoder takes most of the area
5.12.2006 © Ioannis Sourdis 17
TU Crete
Conclusions
Packet Pre-filtering Points out a small rule subset per incoming packet
for further processing Offloads an IDS core Allows to utilize more sophisticated attack
descriptions on the 2nd phase of the system (for a few rules)
We include in the filter (per rule) Header matching (Source/Destination Address &
Protocol) 4-8 characters payload pattern match
5.12.2006 © Ioannis Sourdis 18
TU Crete
Conclusions
Performance:
99% of the IDS rules per incoming packet do not need further processing (in Defcon11 traces), without loosing detection precision.
Throughput: 2.7-10 Gbps (Virtex2) or 4-14 Gbps (Virtex4), 8 or 32 bits datapaths
Requirements:
Lightweight system, requires 10-15K slides, can fit in a medium-sized FPGA
Can be integrated in both HW or SW based systems