MITACS-PINTSPrediction In Interacting Systems

Project Leader : Michael Kouriztin

Network Security

Search and Rescue

DefenceInvestingEnvironmental

Monitoring

Fraud Detection

Nonlinear Filtering

Modeling

Observing

Countering Espionage in Cyber-Warfare: Detecting Stealthy Portscans

Jarett Hailes

Surrey Kim

Michael Kouritzin

Wei Sun

5th MITACS IT-Theme MeetingOctober 19, 2003

OutlineOutline

Problem of detecting stealthy port scansProblem of detecting stealthy port scans

SimulationsSimulations

Clustering model & Filtering equationClustering model & Filtering equation

Computer workable approximationComputer workable approximation

Port scanning: method for discovering network vulnerabilities

Reconnaissance stage of a hacker attacks.

“Probes” target network via sending packets

Port ScanningPort Scanning

Stealthy TechniquesStealthy Techniques

Slow Scans : to obscure the attack, an attacker could do the scan very slowly.

Multiple source scans : : using multiple sourcesusing multiple sources Idle scanning : bouncing scans from dumb "zombie"

host.

Spoofed Source IP : sending large number of packets with only one as the real source.

Current SolutionsCurrent Solutions

Existing solutions are : Existing solutions are :

Prone to false alarms and miss detectsProne to false alarms and miss detects

Easily foiled by new scanning techniquesEasily foiled by new scanning techniques

Insufficient information (black and white Insufficient information (black and white solutions)solutions)

Cause for unacceptable downtimeCause for unacceptable downtime

Extensive human management requiredExtensive human management required

End goalEnd goal : to probe 30 ports on 10 hosts. : to probe 30 ports on 10 hosts.

Scanning TechniqueScanning Technique: Half-open SYN Scan : Half-open SYN Scan and tand to obscure the attack : may use multiple computers (i.e. source IP may use multiple computers (i.e. source IP

addresses).addresses). may use may use dumb "zombie" host to bounce

scans. slows down scan rateslows down scan rate sends 300 packets in random ordersends 300 packets in random order

Example Example

Detection ProblemDetection Problem

To detect whether or not there is a port To detect whether or not there is a port scanner present.scanner present.

Via Filtering and Bayesian Model selectionVia Filtering and Bayesian Model selection

Only SYN packets are consideredOnly SYN packets are considered(i.e. No packet flag information used yet)(i.e. No packet flag information used yet)

Assume the traffic rates for target hostsAssume the traffic rates for target hosts

Portscan Detector ResultsPortscan Detector Results

0.000.050.100.150.200.250.300.350.400.450.50

%

0 1 2 3 4 5 6 7 8 9 10

Number of PortScans Detected

Number of PortScans Detected for 10 PortScans Simlation (20 runs)

Portscan Detector ResultsPortscan Detector Results

0.00

0.05

0.10

0.15

0.20

0.25

0.30

%

0 1 2 3 4 5 6 7 8 9 10

Number of False Positives

Number of False Positives for 10 PortScan Simulation (20 runs)

Traffic SummaryTraffic Summary

Signal to Noise Ratio

0

100000

200000

300000

400000

500000

600000

700000

800000

900,000

1,000,000

Nu

mb

er

of

Pa

ck

ets

Normal NetworkTraffic Packets : 923,424

Port ScannerPackets : 428

Challenges and Future WorkChallenges and Future Work

Enormous State Space Enormous State Space ::

Localization Localization : :

IP spoofingIP spoofing : Stealthy hacker scans all : Stealthy hacker scans all ports certain number of times, decreasing ports certain number of times, decreasing scan rate and using to reduce suspicionscan rate and using to reduce suspicion

Clustering ModelClustering ModelModel packet traffic as Model packet traffic as marked point processmarked point process with with marks, i.e. packet headers – (Destination, Source, marks, i.e. packet headers – (Destination, Source, Flags), in Flags), in Network traffic mixture of two types Network traffic mixture of two types Normal traffic rate: Normal traffic rate: Malicious & stealthy traffic rate: Malicious & stealthy traffic rate: depends on all previous scans depends on all previous scans Hacker can have stealthy strategy – e.g. scan Hacker can have stealthy strategy – e.g. scan

network host port over so many daysnetwork host port over so many daysWhich packets are due to port scansWhich packets are due to port scans??

)(u),(

tu

S

),( tu

Filtering ApproachFiltering ApproachNew Nonlinear Filtering ApproachNew Nonlinear Filtering Approach Provides probabilistic informationProvides probabilistic information Other bwOther bw Choose acceptable ratio of miss detect to false alarmChoose acceptable ratio of miss detect to false alarm Asymptotically optimal Asymptotically optimal

Normal TrafficNormal Traffic

Poisson measurePoisson measure – randomly distributes points – randomly distributes points across marks, rates, timeacross marks, rates, time

Number of points in disjoint regions independentNumber of points in disjoint regions independent Desired expected number of points everywhereDesired expected number of points everywhere

Normal Traffic Normal Traffic = = Observation noiseObservation noise that must be that must be “filtered out’’“filtered out’’

itvASVU iiitvA],0[],0[),,(1

1111]),0[],,0[,(

)()(1),(],0[),0[

1)](,0[1 dsddutAYtA

u

Port ScanningPort Scanning

Buried in this noise is the Buried in this noise is the signalsignal = count = count of Port Scan packets at various marksof Port Scan packets at various marks

Port Scan signal or cluster:Port Scan signal or cluster:

ObservationObservation = observed traffic: = observed traffic:

)()(1),(],0[),0[

2)],(,0[ dsddutAtA

u s

),(),(),( 1 tAtAYtAY

Simulation ExampleSimulation Example

End goalEnd goal : to probe 30 ports on 10 target : to probe 30 ports on 10 target hosts.hosts.

Normal Traffic Rates :Normal Traffic Rates :

Cluster dependent scanning rate :Cluster dependent scanning rate :

HostHost 11 22 33 44 55 66 77 88 99 1010

1.00 1.00 0.001 0.001 0.005 0.005 1.01 1.01 1.01 1.01 2.0 2.0 2.0 2.0 0.02 0.02 0.01 0.01 0.02 0.02 )(u

Bayesian Model SelectionBayesian Model Selection

Detecting whether or not there is anomalous Detecting whether or not there is anomalous traffic on observed computer system.traffic on observed computer system.

Bayes factorBayes factor satisfies satisfies

Nonlinear FilteringNonlinear Filtering

GoalGoal: Approximate: ApproximateIdeaIdea:: Choose that does not depend on Choose that does not depend on Then, calculations are simple Then, calculations are simple

Reference probability measure methodReference probability measure method There is artificial probability Q where There is artificial probability Q where is is

Poisson measure with intensity Poisson measure with intensity P(A) = L(t) Q(A) for events A occuring by t; L P(A) = L(t) Q(A) for events A occuring by t; L

is martingaleis martingale

)),(|(),( tssYAPtA t

)(u

Filtering EquationFiltering EquationUnnormalized conditional port scan distributionUnnormalized conditional port scan distribution

Then, we approximateThen, we approximate

Real-world conditional probability satisfiesReal-world conditional probability satisfies

fttftf ),1(/),(),(

)),(|)()),(((),( tssYtLtfEtf Q

)()()(

)()),(

)(

),()((

)()),,()()((()0,(),(

],0[

],0[

dsduYuu

usf

u

uf

dsdusuufftf

tM u

tM

(1)

Workable Approximation (I)Workable Approximation (I)

Under general conditions Under general conditions and after modest workand after modest work we find and prove:we find and prove:

NNN as and

In probability on pathspace for each fixed observation Y, i.e. in quenched sense.

Here ),(

),()(

tf

tff

t

NtN

t

N

Workable Approximation (II)Workable Approximation (II)

;,...,1 , ;1

NNk

Nk

d

k

Nk dkCySC

N

Nd

NN LJ ,...,0

, allfor )(mean )(Let NiN

iN dijCjC

Equation (1) is still unworkable so we let

NC1NC2

NdN

C. . .

S

Ex: Suppose S is 1-dimensional

NK

0

1

. . .

Number of Packets in Each Cell

Workable Approximation (III)Workable Approximation (III)Substituting into (1) and approximating counting measures on S with counting measures on with at most LN particles, one finds

Here

jC Nf

)(

1)(

Nd

kNky 1}{

Workable Approximation (IV)Workable Approximation (IV)

,for

NNiDi Ny

iN JjjjK

and , )))((()))((()0( 00 jCPljCln NN

NN

Nj

processesPoisson t independen be , ,,NJj

Nj

Nj XX

• We also discretize amplitude to yield Markov chain

approximation

• Suppose is sequence satisfying

• Let

Nl Nl

Workable Approximation (V)Workable Approximation (V)Our Markov chain solves Our Markov chain solves

The approximation is given by:The approximation is given by:

)),()()()(

),(

)()()(()),()()(

)()(

)()(),(()0()(

)(],0[

)(

0

,],0[

0

,

dsduYsnuu

jKu

dssnduuXdsduYuu

snu

dssndujKuXntn

N

uejtS

uyN

S

tN

j

N

jtS

N

j

S

tN

jN

N

j

N

j

N

j

N

N

NJj

NN

NjN

t jKfl

tnf )(

)()(

NNj Jjn ,