XFA Faster Signature Matching With Extended AutomataAuthor Randy Smith Cristian Estan and Somesh JhaPublisher IEEE Symposium on Security and Privacy 2008Presenter Yu-Hao TsengDate 20140115

1

Outlinebull Introductionbull Technical overviewbull Build XFAs from Regexbull Experimental Results

2

Introductionbull In this paper their primary goal is to improve the time and

space efficiency of signature matching in network intrusion detection systems (NIDS)

bull To achieve their goal they introduce extended finite automata (XFAs) which augment traditional FSAs with a finite scratch memory used to remember various types of information relevant to the progress of signature matching

3

Technical overviewbull For NIDS signatures REs overlap or subsume each otherbull Matching progress interleavedbull Many distinct combination of reachable states

bull Two signaturesbull where all and are distinct stringsbull which consists of all strings of length n

4

Technical overview (Cont)bull where all and are distinct strings

5

Technical overview (Cont)bull where all and are distinct stringsbull use a single bit of scratch memory

6

Technical overview (Cont)bull which consists of all strings of length n

bull use a counter

7

Technical overview (Cont)bull XFA = DFAs+ auxiliary variables1048708

bull Changes shape of automatabull Tames state space explosion

8

Build XFAs from Regex

bull Annotating regular expressionsbull Compiling to an XFAbull From parse trees to NXFAsbull From NXFAs to XFAs

bull ε ndasheliminationbull determinizing transitionsbull data determinization

bull Finding efficient implementations

9

Build XFAs from Regex (Cont)

bull Annotate Signaturebull New operators change parse tree and add domain values 1048708

bull Parallel concatenation ( ) adds a bit1048708bull Breaks up RE into string-like componentsbull Set a bit when the left operand acceptsbull Test the bit when the right operand accepts

bull ex abcd =gt abcd

10

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull XFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ QtimesΣrarrQ is the transition functionbull D is the finite set of values in the data domainbull QtimesΣtimesDrarrD is the per transition update function which defines how the

data value is updated on every transitionbull (q0d0) is the initial configuration which consists of an initial state q0 and

an initial data value d0bull F QtimesD is the set of accepting configurationssube

11

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull NXFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ Qtimes(Σ ε )timesQ is the nondeterministic relation describing the allowed sube cup

transitionsbull D is the finite set of values in the data domainbull δ rarr is the nondeterministic update function (or update relation) which

defines how the data value is updated on every transitionbull QD0 QtimesD is the set of initial configurations of the NXFAsubebull F QtimesD is the set of accepting configurationssube

12

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

13

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

14

ａ

ｂ

sum

1

[ a]

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

15

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

16

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

17

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Outlinebull Introductionbull Technical overviewbull Build XFAs from Regexbull Experimental Results

2

Introductionbull In this paper their primary goal is to improve the time and

space efficiency of signature matching in network intrusion detection systems (NIDS)

bull To achieve their goal they introduce extended finite automata (XFAs) which augment traditional FSAs with a finite scratch memory used to remember various types of information relevant to the progress of signature matching

3

Technical overviewbull For NIDS signatures REs overlap or subsume each otherbull Matching progress interleavedbull Many distinct combination of reachable states

bull Two signaturesbull where all and are distinct stringsbull which consists of all strings of length n

4

Technical overview (Cont)bull where all and are distinct strings

5

Technical overview (Cont)bull where all and are distinct stringsbull use a single bit of scratch memory

6

Technical overview (Cont)bull which consists of all strings of length n

bull use a counter

7

Technical overview (Cont)bull XFA = DFAs+ auxiliary variables1048708

bull Changes shape of automatabull Tames state space explosion

8

Build XFAs from Regex

bull Annotating regular expressionsbull Compiling to an XFAbull From parse trees to NXFAsbull From NXFAs to XFAs

bull ε ndasheliminationbull determinizing transitionsbull data determinization

bull Finding efficient implementations

9

Build XFAs from Regex (Cont)

bull Annotate Signaturebull New operators change parse tree and add domain values 1048708

bull Parallel concatenation ( ) adds a bit1048708bull Breaks up RE into string-like componentsbull Set a bit when the left operand acceptsbull Test the bit when the right operand accepts

bull ex abcd =gt abcd

10

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull XFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ QtimesΣrarrQ is the transition functionbull D is the finite set of values in the data domainbull QtimesΣtimesDrarrD is the per transition update function which defines how the

data value is updated on every transitionbull (q0d0) is the initial configuration which consists of an initial state q0 and

an initial data value d0bull F QtimesD is the set of accepting configurationssube

11

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull NXFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ Qtimes(Σ ε )timesQ is the nondeterministic relation describing the allowed sube cup

transitionsbull D is the finite set of values in the data domainbull δ rarr is the nondeterministic update function (or update relation) which

defines how the data value is updated on every transitionbull QD0 QtimesD is the set of initial configurations of the NXFAsubebull F QtimesD is the set of accepting configurationssube

12

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

13

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

14

ａ

ｂ

sum

1

[ a]

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

15

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

16

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

17

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Introductionbull In this paper their primary goal is to improve the time and

space efficiency of signature matching in network intrusion detection systems (NIDS)

bull To achieve their goal they introduce extended finite automata (XFAs) which augment traditional FSAs with a finite scratch memory used to remember various types of information relevant to the progress of signature matching

3

Technical overviewbull For NIDS signatures REs overlap or subsume each otherbull Matching progress interleavedbull Many distinct combination of reachable states

bull Two signaturesbull where all and are distinct stringsbull which consists of all strings of length n

4

Technical overview (Cont)bull where all and are distinct strings

5

Technical overview (Cont)bull where all and are distinct stringsbull use a single bit of scratch memory

6

Technical overview (Cont)bull which consists of all strings of length n

bull use a counter

7

Technical overview (Cont)bull XFA = DFAs+ auxiliary variables1048708

bull Changes shape of automatabull Tames state space explosion

8

Build XFAs from Regex

bull Annotating regular expressionsbull Compiling to an XFAbull From parse trees to NXFAsbull From NXFAs to XFAs

bull ε ndasheliminationbull determinizing transitionsbull data determinization

bull Finding efficient implementations

9

Build XFAs from Regex (Cont)

bull Annotate Signaturebull New operators change parse tree and add domain values 1048708

bull Parallel concatenation ( ) adds a bit1048708bull Breaks up RE into string-like componentsbull Set a bit when the left operand acceptsbull Test the bit when the right operand accepts

bull ex abcd =gt abcd

10

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull XFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ QtimesΣrarrQ is the transition functionbull D is the finite set of values in the data domainbull QtimesΣtimesDrarrD is the per transition update function which defines how the

data value is updated on every transitionbull (q0d0) is the initial configuration which consists of an initial state q0 and

an initial data value d0bull F QtimesD is the set of accepting configurationssube

11

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull NXFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ Qtimes(Σ ε )timesQ is the nondeterministic relation describing the allowed sube cup

transitionsbull D is the finite set of values in the data domainbull δ rarr is the nondeterministic update function (or update relation) which

defines how the data value is updated on every transitionbull QD0 QtimesD is the set of initial configurations of the NXFAsubebull F QtimesD is the set of accepting configurationssube

12

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

13

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

14

ａ

ｂ

sum

1

[ a]

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

15

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

16

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

17

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Technical overviewbull For NIDS signatures REs overlap or subsume each otherbull Matching progress interleavedbull Many distinct combination of reachable states

bull Two signaturesbull where all and are distinct stringsbull which consists of all strings of length n

4

Technical overview (Cont)bull where all and are distinct strings

5

Technical overview (Cont)bull where all and are distinct stringsbull use a single bit of scratch memory

6

Technical overview (Cont)bull which consists of all strings of length n

bull use a counter

7

Technical overview (Cont)bull XFA = DFAs+ auxiliary variables1048708

bull Changes shape of automatabull Tames state space explosion

8

Build XFAs from Regex

bull Annotating regular expressionsbull Compiling to an XFAbull From parse trees to NXFAsbull From NXFAs to XFAs

bull ε ndasheliminationbull determinizing transitionsbull data determinization

bull Finding efficient implementations

9

Build XFAs from Regex (Cont)

bull Annotate Signaturebull New operators change parse tree and add domain values 1048708

bull Parallel concatenation ( ) adds a bit1048708bull Breaks up RE into string-like componentsbull Set a bit when the left operand acceptsbull Test the bit when the right operand accepts

bull ex abcd =gt abcd

10

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull XFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ QtimesΣrarrQ is the transition functionbull D is the finite set of values in the data domainbull QtimesΣtimesDrarrD is the per transition update function which defines how the

data value is updated on every transitionbull (q0d0) is the initial configuration which consists of an initial state q0 and

an initial data value d0bull F QtimesD is the set of accepting configurationssube

11

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull NXFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ Qtimes(Σ ε )timesQ is the nondeterministic relation describing the allowed sube cup

transitionsbull D is the finite set of values in the data domainbull δ rarr is the nondeterministic update function (or update relation) which

defines how the data value is updated on every transitionbull QD0 QtimesD is the set of initial configurations of the NXFAsubebull F QtimesD is the set of accepting configurationssube

12

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

13

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

14

ａ

ｂ

sum

1

[ a]

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

15

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

16

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

17

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Technical overview (Cont)bull where all and are distinct strings

5

Technical overview (Cont)bull where all and are distinct stringsbull use a single bit of scratch memory

6

Technical overview (Cont)bull which consists of all strings of length n

bull use a counter

7

Technical overview (Cont)bull XFA = DFAs+ auxiliary variables1048708

bull Changes shape of automatabull Tames state space explosion

8

Build XFAs from Regex

bull Annotating regular expressionsbull Compiling to an XFAbull From parse trees to NXFAsbull From NXFAs to XFAs

bull ε ndasheliminationbull determinizing transitionsbull data determinization

bull Finding efficient implementations

9

Build XFAs from Regex (Cont)

bull Annotate Signaturebull New operators change parse tree and add domain values 1048708

bull Parallel concatenation ( ) adds a bit1048708bull Breaks up RE into string-like componentsbull Set a bit when the left operand acceptsbull Test the bit when the right operand accepts

bull ex abcd =gt abcd

10

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull XFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ QtimesΣrarrQ is the transition functionbull D is the finite set of values in the data domainbull QtimesΣtimesDrarrD is the per transition update function which defines how the

data value is updated on every transitionbull (q0d0) is the initial configuration which consists of an initial state q0 and

an initial data value d0bull F QtimesD is the set of accepting configurationssube

11

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull NXFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ Qtimes(Σ ε )timesQ is the nondeterministic relation describing the allowed sube cup

transitionsbull D is the finite set of values in the data domainbull δ rarr is the nondeterministic update function (or update relation) which

defines how the data value is updated on every transitionbull QD0 QtimesD is the set of initial configurations of the NXFAsubebull F QtimesD is the set of accepting configurationssube

12

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

13

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

14

ａ

ｂ

sum

1

[ a]

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

15

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

16

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

17

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Technical overview (Cont)bull where all and are distinct stringsbull use a single bit of scratch memory

6

Technical overview (Cont)bull which consists of all strings of length n

bull use a counter

7

Technical overview (Cont)bull XFA = DFAs+ auxiliary variables1048708

bull Changes shape of automatabull Tames state space explosion

8

Build XFAs from Regex

bull Annotating regular expressionsbull Compiling to an XFAbull From parse trees to NXFAsbull From NXFAs to XFAs

bull ε ndasheliminationbull determinizing transitionsbull data determinization

bull Finding efficient implementations

9

Build XFAs from Regex (Cont)

bull Annotate Signaturebull New operators change parse tree and add domain values 1048708

bull Parallel concatenation ( ) adds a bit1048708bull Breaks up RE into string-like componentsbull Set a bit when the left operand acceptsbull Test the bit when the right operand accepts

bull ex abcd =gt abcd

10

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull XFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ QtimesΣrarrQ is the transition functionbull D is the finite set of values in the data domainbull QtimesΣtimesDrarrD is the per transition update function which defines how the

data value is updated on every transitionbull (q0d0) is the initial configuration which consists of an initial state q0 and

an initial data value d0bull F QtimesD is the set of accepting configurationssube

11

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull NXFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ Qtimes(Σ ε )timesQ is the nondeterministic relation describing the allowed sube cup

transitionsbull D is the finite set of values in the data domainbull δ rarr is the nondeterministic update function (or update relation) which

defines how the data value is updated on every transitionbull QD0 QtimesD is the set of initial configurations of the NXFAsubebull F QtimesD is the set of accepting configurationssube

12

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

13

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

14

ａ

ｂ

sum

1

[ a]

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

15

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

16

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

17

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Technical overview (Cont)bull which consists of all strings of length n

bull use a counter

7

Technical overview (Cont)bull XFA = DFAs+ auxiliary variables1048708

bull Changes shape of automatabull Tames state space explosion

8

Build XFAs from Regex

bull Annotating regular expressionsbull Compiling to an XFAbull From parse trees to NXFAsbull From NXFAs to XFAs

bull ε ndasheliminationbull determinizing transitionsbull data determinization

bull Finding efficient implementations

9

Build XFAs from Regex (Cont)

bull Annotate Signaturebull New operators change parse tree and add domain values 1048708

bull Parallel concatenation ( ) adds a bit1048708bull Breaks up RE into string-like componentsbull Set a bit when the left operand acceptsbull Test the bit when the right operand accepts

bull ex abcd =gt abcd

10

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull XFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ QtimesΣrarrQ is the transition functionbull D is the finite set of values in the data domainbull QtimesΣtimesDrarrD is the per transition update function which defines how the

data value is updated on every transitionbull (q0d0) is the initial configuration which consists of an initial state q0 and

an initial data value d0bull F QtimesD is the set of accepting configurationssube

11

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull NXFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ Qtimes(Σ ε )timesQ is the nondeterministic relation describing the allowed sube cup

transitionsbull D is the finite set of values in the data domainbull δ rarr is the nondeterministic update function (or update relation) which

defines how the data value is updated on every transitionbull QD0 QtimesD is the set of initial configurations of the NXFAsubebull F QtimesD is the set of accepting configurationssube

12

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

13

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

14

ａ

ｂ

sum

1

[ a]

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

15

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

16

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

17

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Technical overview (Cont)bull XFA = DFAs+ auxiliary variables1048708

bull Changes shape of automatabull Tames state space explosion

8

Build XFAs from Regex

bull Annotating regular expressionsbull Compiling to an XFAbull From parse trees to NXFAsbull From NXFAs to XFAs

bull ε ndasheliminationbull determinizing transitionsbull data determinization

bull Finding efficient implementations

9

Build XFAs from Regex (Cont)

bull Annotate Signaturebull New operators change parse tree and add domain values 1048708

bull Parallel concatenation ( ) adds a bit1048708bull Breaks up RE into string-like componentsbull Set a bit when the left operand acceptsbull Test the bit when the right operand accepts

bull ex abcd =gt abcd

10

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull XFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ QtimesΣrarrQ is the transition functionbull D is the finite set of values in the data domainbull QtimesΣtimesDrarrD is the per transition update function which defines how the

data value is updated on every transitionbull (q0d0) is the initial configuration which consists of an initial state q0 and

an initial data value d0bull F QtimesD is the set of accepting configurationssube

11

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull NXFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ Qtimes(Σ ε )timesQ is the nondeterministic relation describing the allowed sube cup

transitionsbull D is the finite set of values in the data domainbull δ rarr is the nondeterministic update function (or update relation) which

defines how the data value is updated on every transitionbull QD0 QtimesD is the set of initial configurations of the NXFAsubebull F QtimesD is the set of accepting configurationssube

12

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

13

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

14

ａ

ｂ

sum

1

[ a]

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

15

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

16

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

17

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex

bull Annotating regular expressionsbull Compiling to an XFAbull From parse trees to NXFAsbull From NXFAs to XFAs

bull ε ndasheliminationbull determinizing transitionsbull data determinization

bull Finding efficient implementations

9

Build XFAs from Regex (Cont)

bull Annotate Signaturebull New operators change parse tree and add domain values 1048708

bull Parallel concatenation ( ) adds a bit1048708bull Breaks up RE into string-like componentsbull Set a bit when the left operand acceptsbull Test the bit when the right operand accepts

bull ex abcd =gt abcd

10

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull XFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ QtimesΣrarrQ is the transition functionbull D is the finite set of values in the data domainbull QtimesΣtimesDrarrD is the per transition update function which defines how the

data value is updated on every transitionbull (q0d0) is the initial configuration which consists of an initial state q0 and

an initial data value d0bull F QtimesD is the set of accepting configurationssube

11

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull NXFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ Qtimes(Σ ε )timesQ is the nondeterministic relation describing the allowed sube cup

transitionsbull D is the finite set of values in the data domainbull δ rarr is the nondeterministic update function (or update relation) which

defines how the data value is updated on every transitionbull QD0 QtimesD is the set of initial configurations of the NXFAsubebull F QtimesD is the set of accepting configurationssube

12

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

13

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

14

ａ

ｂ

sum

1

[ a]

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

15

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

16

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

17

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Annotate Signaturebull New operators change parse tree and add domain values 1048708

bull Parallel concatenation ( ) adds a bit1048708bull Breaks up RE into string-like componentsbull Set a bit when the left operand acceptsbull Test the bit when the right operand accepts

bull ex abcd =gt abcd

10

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull XFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ QtimesΣrarrQ is the transition functionbull D is the finite set of values in the data domainbull QtimesΣtimesDrarrD is the per transition update function which defines how the

data value is updated on every transitionbull (q0d0) is the initial configuration which consists of an initial state q0 and

an initial data value d0bull F QtimesD is the set of accepting configurationssube

11

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull NXFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ Qtimes(Σ ε )timesQ is the nondeterministic relation describing the allowed sube cup

transitionsbull D is the finite set of values in the data domainbull δ rarr is the nondeterministic update function (or update relation) which

defines how the data value is updated on every transitionbull QD0 QtimesD is the set of initial configurations of the NXFAsubebull F QtimesD is the set of accepting configurationssube

12

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

13

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

14

ａ

ｂ

sum

1

[ a]

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

15

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

16

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

17

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull XFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ QtimesΣrarrQ is the transition functionbull D is the finite set of values in the data domainbull QtimesΣtimesDrarrD is the per transition update function which defines how the

data value is updated on every transitionbull (q0d0) is the initial configuration which consists of an initial state q0 and

an initial data value d0bull F QtimesD is the set of accepting configurationssube

11

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull NXFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ Qtimes(Σ ε )timesQ is the nondeterministic relation describing the allowed sube cup

transitionsbull D is the finite set of values in the data domainbull δ rarr is the nondeterministic update function (or update relation) which

defines how the data value is updated on every transitionbull QD0 QtimesD is the set of initial configurations of the NXFAsubebull F QtimesD is the set of accepting configurationssube

12

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

13

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

14

ａ

ｂ

sum

1

[ a]

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

15

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

16

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

17

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)bull Compile to XFAbull Definitions

bull NXFA is a 7-tuple (Q D Σ δ ( )F)bull Q is the set of statesbull Σ is the set of inputs (input alphabet)bull δ Qtimes(Σ ε )timesQ is the nondeterministic relation describing the allowed sube cup

transitionsbull D is the finite set of values in the data domainbull δ rarr is the nondeterministic update function (or update relation) which

defines how the data value is updated on every transitionbull QD0 QtimesD is the set of initial configurations of the NXFAsubebull F QtimesD is the set of accepting configurationssube

12

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

13

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

14

ａ

ｂ

sum

1

[ a]

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

15

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

16

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

17

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

13

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

14

ａ

ｂ

sum

1

[ a]

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

15

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

16

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

17

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

14

ａ

ｂ

sum

1

[ a]

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

15

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

16

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

17

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)bull Compile to XFAbull From parse trees to NXFAs

bull Ex ab[^a]1 =gt ab[^a]1

15

1

2

0

3

4

bit = 0cnt = 0 sum

a

bsum

[^a]

ε

ε

ε

cnt++

if (bit == 1 ampamp cnt = 1) accept()

bit = 1

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

16

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

17

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

16

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

17

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

17

120576

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)bull Compile to XFAbull Alg 1 ε -elimination for NXFAs

18

120576

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo

A C

Frsquo

19

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C

A C

Frsquo

A C

A C

20

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 1) (C 0) (C 2)

δlsquo A C A C A C A B C

A C

Frsquo

A B C

A C

21

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0)Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C

A C

Frsquo

A B C

A B C

22

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C

A C

Frsquo

A C

A B C

23

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo

A C

A B C

24

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 2 determinizing transitions for NXFAs

Qrsquo A C A B C (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 0) (C 1) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (B 1) (A 0) (C 0) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (B 1) (C 2) (C 2) (C 1) (A 0) (A 0) (A 0) (C 0) (C 0) (C 0) (C 2) (C 1)

Drsquo (A 0) (B 0) (C 0) (C 2)

δlsquo A C A C A C A B C A B C A B C A B C A C A B C A C

A C

Frsquo (A C (C 1)) (A B C (C 1))

A C

A B C

25

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

26

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5

Drsquo 3 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5)

G G

35 35

27

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

G H

35 3 4 5

28

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5Drsquo 3 5 3 4 5

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5)

H H

3 4 5 3 4 5

29

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5 7

30

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7)

H G

3 4 5 3 5

31

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 7 3 5 6

32

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 7 3 4 5

33

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G G

3 5 6 3 5

34

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

35

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Alg 3 data determinization for NXFAs

Q G H 3 5 3 5 3 5 7 3 5 6 3 5 6 3 5 3 5 3 4 5 3 5 7 3 4 5 3 5 6 3 4 5 3 4 5 3 4 5 3 4 5 3 5 7 3 4 5 3 5

Drsquo 3 5 3 4 5 3 5 7 3 5 6

δlsquo G G G H H H H G H G

( ) (G 3 5)

Frsquo (G 3 5 6)

QD (G 3 5) (H 3 4 5) (G 3 5 7) (G 3 5 6)

G H

3 5 6 3 4 5

36

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Build XFAs from Regex (Cont)

bull Compile to XFAbull Finding efficient implementations

37

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Experimental Resultsbull 1450 Regular expressions extracted from Snort HTTPbull Characteristics of combined XFA1048708bull 41994 total states =gt 42 MBbull 195 bits (~25 bytes) of aux memorybull Instruction memory 35 MB

38

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Experimental Results (Cont)

39

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Experimental Results (Cont)

40

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion

Conclusionbull DFAs for regular expressions often blow up when combined

bull XFA = DFAs+ auxiliary variables1048708bull Changes shape of automatabull Tames state space explosion

bull Result compared to other feasible approaches reduce both time and space

41

• XFA Faster Signature Matching With Extended Automata
• Outline
• Introduction
• Technical overview
• Technical overview (Cont)
• Technical overview (Cont) (2)
• Technical overview (Cont) (3)
• Technical overview (Cont) (4)
• Build XFAs from Regex
• Build XFAs from Regex (Cont)
• Build XFAs from Regex (Cont) (2)
• Build XFAs from Regex (Cont) (3)
• Build XFAs from Regex (Cont) (4)
• Build XFAs from Regex (Cont) (5)
• Build XFAs from Regex (Cont) (6)
• Build XFAs from Regex (Cont) (7)
• Build XFAs from Regex (Cont) (8)
• Build XFAs from Regex (Cont) (9)
• Build XFAs from Regex (Cont) (10)
• Build XFAs from Regex (Cont) (11)
• Build XFAs from Regex (Cont) (12)
• Build XFAs from Regex (Cont) (13)
• Build XFAs from Regex (Cont) (14)
• Build XFAs from Regex (Cont) (15)
• Build XFAs from Regex (Cont) (16)
• Build XFAs from Regex (Cont) (17)
• Build XFAs from Regex (Cont) (18)
• Build XFAs from Regex (Cont) (19)
• Build XFAs from Regex (Cont) (20)
• Build XFAs from Regex (Cont) (21)
• Build XFAs from Regex (Cont) (22)
• Build XFAs from Regex (Cont) (23)
• Build XFAs from Regex (Cont) (24)
• Build XFAs from Regex (Cont) (25)
• Build XFAs from Regex (Cont) (26)
• Build XFAs from Regex (Cont) (27)
• Build XFAs from Regex (Cont) (28)
• Experimental Results
• Experimental Results (Cont)
• Experimental Results (Cont) (2)
• Conclusion