testing ids

Testing IDS

Overview

• Introduction• Measurable IDS characteristics• Challenges of IDS testing• Measuring IDS performances• Test data sets

2/108

3/108

Introduction

• Despite enormous investment in IDS technology, no comprehensive and scientifically rigorous methodology is available to test IDS.

• Quantitative IDS performance measurement results are essential in order to compare different systems.

4/108

Introduction

• Quantitative results are needed by:– Acquisition managers – to improve the

process of system selection.– Security analysts – to know the likelihood that

the alerts produced by IDS are caused by real attacks that are in progress.

– Researchers and developers – to understand the strengths and weaknesses of IDS in order to focus research efforts on improving systems and measuring their progress.

5/108

Measurable IDS characteristics

• Coverage• Probability of false alarms• Probability of detection• Resistance to attacks directed at the IDS• Ability to handle high bandwidth traffic• Ability to correlate events• Ability to detect new attacks

6/108

Measurable IDS characteristics

• Ability to identify an attack• Ability to determine attack success• Capacity verification (NIDS).

7/108

Coverage

• Determines which attacks an IDS can detect under ideal conditions.

• For misuse (signature based) systems– Counting the number of signatures and

mapping them to a standard naming scheme.• For anomaly detection systems

– Determining which attacks out of the set of all known attacks could be detected by a particular methodology.

8/108

Coverage

• The problem with determining coverage of an IDS lies in the fact that various researchers characterize the attacks by different numbers of parameters.

9/108

Coverage• These characterizations may take into

account the particular goal of the attack (DoS, penetration, scanning, etc.), the software, protocol and/or OS against which it is targeted, the victim type, the data to be collected in order to obtain the evidence of the attack, the use or not of IDS evasion techniques, etc.

• Combinations of these parameters are also possible.

10/108

Coverage

• The consequence of these differences are coarse granularity attack definitions and finer granularity attack definitions.

• Because of the disparity in granularity, it is difficult to determine attack coverage of an IDS precisely.

11/108

Coverage

• CVE is an attempt to alleviate this problem.

• But the CVE approach does not work either, if multiple attacks are used to exploit the same vulnerability using different approach (for example to evade IDS systems).

12/108

Coverage• Determining the importance of different

attack types is also a problem when determining coverage.

• Different environments may assign different costs and importance to detecting different types of attacks.

13/108

Coverage• Example:

– An e-commerce site may not be interested in surveillance attacks, but may be very interested in detecting DDoS attacks.

– A military site may be especially interested in detecting surveillance attacks in order to prevent more serious attacks by acting in their early phases.

14/108

Coverage

• Another problem with coverage is in determining which attacks to cover regarding system updates.– Example:

• It is worthless to test IDS coverage of the attacks against the defended system in which the measures against these attacks have already been applied (patching, hardening, etc.)

15/108

Probability of false alarms

• Suppose that we have N IDS decisions, of which:– In TP cases: intrusion – alarm.– In TN cases: no intrusion – no alarm.– In FP cases: no intrusion – alarm.– In FN cases: intrusion – no alarm.

16/108


• Total intrusions: TP+FN• Total no-intrusions: FP+TN• N=TP+FN+FP+TN• Base-rate – the probability of an

attack: N

FNTPIP

17/108


• Events: Alarm A, Intrusion I– The following rates are defined:

• True positive rate TPR

• True negative rate TNR

IAPFNTP

TPTPR

IAPTNFP

TNTNR

18/108


• False positive rate FPR

• False negative rate FNR

IAPTNFP

FPFPR

IAPFNTP

FNFNR

19/108

Probability of false alarms• This measure determines the rate of false

positives produced by an IDS in a given environment during a particular time frame.

20/108

Probability of false alarms• Typical causes of false positives:

– Weak signatures (alert on all traffic to a specific port, search for the occurrence of a common word such as ”help” in the first 100 bytes of SNMP or other TCP connections, alert on common violations of the TCP protocol, etc.)

– Normal network monitoring and maintenance traffic.

21/108

Probability of false alarms• Difficulties regarding measuring of false

alarm rate (1)– An IDS may have a different false positive

rate in different network environments, and “standard network” does not exist.

– It is difficult to determine aspects of network traffic or host activity that will cause false alarms.

22/108

Probability of false alarms• Difficulties regarding measuring of false

alarm rate (2)– Consequence: it is difficult to guarantee that it

is possible to produce the same number and type of false alarms in a test network as in a real network.

– IDS can be configured in many ways and it is difficult to determine which configuration of an IDS should be used for a particular false positive test.

23/108

Probability of detection

• This measurement determines the rate of attacks detected correctly by an IDS in a given environment during a particular time frame.

• Difficulties in measuring probability of detection (1)– The success of an IDS is largely dependent

upon the set of the attacks used during the test.

24/108

Probability of detection• Difficulties in measuring probability of

detection (2)– The probability of detection varies with the

false positive rate – the same configuration of the IDS must be used for testing for false positives and hit rates.

25/108

Probability of detection• Difficulties in measuring probability of

detection (3)– A NIDS can be evaded by using the stealthy

versions of attacks (fragmenting packets, using data encoding, using unusual TCP flags, enciphering attack packets, spreading attacks over multiple network sessions, launching attacks from multiple sources, etc.)

– This reduces the probability of detection, even though the same attack would be detected if no stealthy version would be applied.

26/108

Resistance to attacks against IDS• This measurement demonstrates how

resistant an IDS is to an attacker’s attempt to disrupt the correct operation of the IDS.

• Some typical attacks against IDS (1)– Sending a large amount of non-attack traffic

with volume exceeding the IDS processing capability – this causes dropping packets by the IDS.

27/108

Resistance to attacks against IDS• Some typical attacks against IDS (2)

– Sending to the IDS non-attack packets that are specially crafted to trigger many signatures within the IDS – the human operator is overwhelmed with false positives, or an automated analysis tools crashes.

28/108

Resistance to attacks against IDS• Some typical attacks against IDS (3)

– Sending to the IDS a large number of attack packets intended to distract the human operator, while the attacker launches a real attack hidden among these “false attacks”.

– Sending to the IDS packets containing data that exploit a vulnerability within the very IDS processing algorithms. Such vulnerabilities may be consequence of coding errors.

29/108

Ability to handle high bandwidth traffic• This measurement demonstrates how well

an IDS will function when presented with a large volume of traffic.

• Most NIDS start to drop packets as the traffic volume increases – false negatives.

• At certain threshold, most IDS will stop detecting any attacks.

30/108

Ability to correlate events• This measurement demonstrates how well

an IDS correlates attack events.• These events may be gathered from IDS,

routers, firewalls, application logs, etc.• One of the primary goals of event

correlation is to identify penetration attacks.

• Currently, IDS have limited capabilities in this area.

31/108

Ability to detect new attacks• This measurement demonstrates how well

an IDS can detect attacks that have not occurred before.

• Signature-only based systems will have 0 score here.

• Anomaly-based systems may be suitable for this type of measurement. However, they in general produce more false alarms than the signature-based systems.

32/108

Ability to identify an attack• This measurement demonstrates how well

an IDS can identify the attack that it has detected.

• Each attack should be labelled with a common name or vulnerability name, or by assigning the attack to a category.

33/108

Ability to determine attack success

• This measurement demonstrates if the IDS can determine the success of attacks from remote sites that give the attacker higher-level privileges on the attacked system.

• Many remote privilege-gaining attacks (probes) fail and do not damage the attacked system.

• Many IDS do not distinguish between unsuccessful and successful attacks.

34/108

Ability to determine attack success

• For the same attack, some IDS can detect the evidence of damage and some IDS detect only the signature of attack actions.

• The ability to determine the attack success is essential for the analysis of attack correlation and the attack scenario.

• Measuring this capability requires the information about both successful and unsuccessful attacks.

35/108

Capacity verification for NIDS• The NIDS demand higher-level protocol

awareness than other network devices (switches, routers, etc.)

• NIDS inspect more deeply the network packets than the other devices do.

• Therefore, it is important to measure the ability of a NIDS to capture, process and perform at the same level of accuracy under a given network load as it does on a quiescent network.

36/108

Capacity verification for NIDS• There exists a standardized capacity

benchmarking methodology for NIDS (e.g. CISCO has its own methodology).

• The NIDS customers can use the standardized capacity test results for each metric and a profile of their networks to determine if the NIDS is capable of inspecting their traffic.

37/108

Challenges of IDS testing

• The following problems (at least) make IDS testing a challenging task:– Collecting attack scripts and victim software is

difficult.– Requirements for testing signature-based and

anomaly-based IDS are different.– Requirements for testing host-based and

network-based IDS are different.– Using background traffic in IDS testing is not

standardized.

38/108

Challenges of IDS testing• Collecting attack scripts and victim

software (1)– It is difficult and expensive to collect a large

number of attack scripts.– The attack scripts are available in various

repositories, but it takes time to find relevant scripts to a particular testing environment.

39/108

Challenges of IDS testing• Collecting attack scripts and victim

software (2)– Once an adequate script is identified, it takes

approx. one person-week to review the code, test the exploit, determine where the attack leaves evidence, automate the attack and integrate it into a testing environment.

40/108

Challenges of IDS testing• Different requirements for testing signature-

based and anomaly-based IDS (1)– Most commercial systems are signature-based.– Many research systems are anomaly based.

41/108


based and anomaly-based IDS (2)– An ideal IDS testing methodology would be

applicable to both signature-based and anomaly-based systems.

– This is important because the research anomaly-based systems should be compared to the commercial signature-based systems.

42/108


based and anomaly-based IDS (3)– The problems with creating a single test to

cover both type of systems (1)• Anomaly based systems with learning require

normal traffic for training that does not include attacks.

• Anomaly based systems with learning may learn behaviour of the testing methodology and perform well without detecting real attacks at all.

43/108



cover both type of systems (2)• This may happen when all the attacks in a test are

launched from a particular user, IP address, subnet, or MAC address.

44/108



cover both type of systems (3)• Anomaly-based systems with learning can also

learn subtle characteristics difficult to predetermine (packet window size, ports, typing speed, command set used, TCP flags, connection duration, etc.) – artificially perform well in the test environment.

45/108



cover both type of systems (4)• The performance of a signature based system in a

test will, to a large degree, depend on the set of attacks used in the test.

• Then the decision about which attacks to include in a test may be in favour of a particular IDS – not objective.

46/108

Challenges of IDS testing• Different requirements for testing host-

based and network-based IDS (1)– Testing host-based IDS presents some

difficulties not present when testing network-based IDS (1)• Network-based IDS can be tested off-line by

creating a log file containing TCP traffic and replaying that traffic to IDS – this is convenient, because there is no need to test all the IDS at the same time.

• Repeatability of the test is easy to achieve.

47/108



difficulties not present when testing network-based IDS (2)• Host-based IDS use a variety of system inputs in

order to determine whether or not a system is under attack.

• This set of inputs is not the same for all IDS.

48/108



difficulties not present when testing network-based IDS (3)• Host-based IDS monitor a host, not a single data

feed.• Then it is difficult to replay activity from log files.• Since it is difficult to test a host-based IDS off-line,

an on-line test should be performed.• Consequence: problems of repeatability.

49/108

Challenges of IDS testing• Using Background traffic in IDS testing (1)

– Four approaches:• Testing using no background traffic/logs• Testing using real traffic/logs• Testing using sanitized traffic/logs• Testing using simulated traffic/logs.

– It is not clear which approach is the most effective for testing IDS.

– Each of the four approaches has unique advantages and disadvantages.

50/108


– Testing using no background traffic/logs (1)• This testing may be used as a reference condition.• An IDS is set up on a host/network on which there

is no activity.• Then, computer attacks are launched on this

host/network to determine whether or not the IDS can detect them.

• This technique can determine the probability of detection (hit rate) under no load, but it cannot determine the false positive rate.

51/108


– Testing using no background traffic/logs (2)• Useful for verifying that an IDS has signatures for a

set of attacks and that the IDS can properly label each attack.

• Often much less costly than other approaches.• Drawback: tests using this technique are based on

the assumption that an IDS ability to detect an attack is the same regardless of the background activity.

52/108


– Testing using no background traffic/logs (3)• At low levels of background activity, that

assumption is probably true.• At high levels of background activity, the

assumption is often false since the IDS performances degrade at high traffic intensities.

53/108


– Testing using real traffic/logs (1)• The attacks are injected into a stream of real

background activity.• Very effective for determining the hit rate of an IDS

given a particular level of background activity.• Background activity is real – contains all the

anomalies and subtleties – realistic hit rates.• Enables comparison of IDS hit rates at different

levels of activity.

54/108


– Testing using real traffic/logs (2)• Drawbacks (1)

– Repeatable test using real traffic is problematic – it is difficult to store and replay large amounts of real traffic at rates higher than 100 Mb/s (currently). Possible solution: parallelization – packet sequencing problems.

– The experiments of this kind usually use a small number of victim machines, set up only to be attacked during the test. Some anomaly detection IDS can then artificially elevate their performances during the test.

55/108



– The real background activity used may contain anomalies unique to the network, which favour one IDS over another. Example: a test network may heavily use a particular protocol that was processed more deeply by a particular IDS.

– The major problem with testing using real background traffic/logs: it is very difficult to determine false positive rates correctly, because it is virtually impossible to guarantee the identification of all the attacks that naturally occur in the background activity.

56/108



– It is difficult to publicly distribute the test, since there are privacy concerns related to the use of real background activity.

– Replay may damage the timings – timestamps should also be kept.

57/108


– Testing using sanitized traffic/logs (1)• Sanitizing – removing sensitive information from

real data.• The goal – to overcome the privacy problems of

using, analyzing, and distributing real background activity.

• Example: TCP packet headers may be cleansed, and packet payloads may be hashed.

• Real background activity is pre-recorded and then all the sensitive data are removed.

58/108


– Testing using sanitized traffic/logs (2)• Then, attack data are injected within the sanitized

data stream:– By replaying the sanitized data and running attacks

concurrently, or– By separately creating attack data and then inserting

these into the sanitized data.

• Advantages:– Test data are freely distributable– The test is repeatable.

59/108


– Testing using sanitized traffic/logs (3)• Disadvantages (1)

– Sanitization attempts may end up removing much of the content of the background activity – very unrealistic environment.

– The major problem: Sanitization attempts may fail – accidental release of sensitive data. It is infeasible for a human to verify the sanitization of a large volume of data.

– The injected attacks do not interact realistically with the sanitized background activity. Example: an injected buffer overflow attack may cause a web server to crash, but background activity still requests the web server.

60/108


– Testing using sanitized traffic/logs (4)• Disadvantages (2)

– When sanitizing real traffic, it may be difficult to remove the attacks that existed in the data stream – this causes problems with the false positive rate testing.

– Sanitizing data may remove information needed to detect attacks.

61/108


– Testing using simulated traffic/logs (1)• The most common approach to testing IDS.• A test bed network with hosts and network

infrastructure is created.• Background traffic is generated on this network, as

well as the attacks.• The test bed network includes victims of interest

with background traffic generated by means of complex traffic generators that model the actual network traffic statistics.

62/108


– Testing using simulated traffic/logs (2)• There is also a possibility to employ simpler traffic

generators to create a small number of packet types at a high rate.

• Network traffic and host audit logs can be recorded in such a test bed network for later playback.

• There is also possibility to perform evaluations in real time.

63/108


– Testing using simulated traffic/logs (3)• Advantages:

– Data can be distributed freely – they do not contain any private or sensitive information.

– There is a guarantee that the background activity does not contain any unknown attacks.

– IDS testing using simulated traffic is easily repeatable.

64/108


– Testing using simulated traffic/logs (4)• Disadvantages:

– It is very costly and difficult to create a simulation.– It is difficult to simulate a high bandwidth environment –

resource constraints.– Different traffic is needed for different types of networks –

academic, e-commerce, military, etc.

65/108

Measuring IDS performances• In order to compare different IDS, measures

of their performances are needed.• Of all the measurable characteristics

mentioned before, the true positive rate and the false positive rate are the most important for comparing IDS.

• The true positive rate and the false positive rate are included in various sublimation metrics for comparing IDS.

66/108

Measuring IDS performances• It is important to determine the probability

of intrusion, if an alert has been generated.

• This gives rise to a Bayesian probabilistic measure for characterising IDS performances.

• We need the total probability of an alert in order to determine the probability of intrusion given the alert.

67/108

Measuring IDS performances

TP FN

TN FP

I

I

I, I mutually exclusive A=(IA)(IA)

IAPIPIAPIPAP

A

A

Total probability of an alert

68/108


FNTP

TPTPRIAP

TNFP

FPFPRIAP

N

FNTPIP

IPIP 1

The base rate:

The true positive rate The false positive rate

69/108


• A performance measure: Bayesian detection rate:

• The greater the detection rate, the better the IDS, but…

)I|A(P)I(P)I|A(P)I(P

)I|A(P)I(P)A|I(P

71/108

Measuring IDS performances• Conclusion:

– If the base rate is low, the false alarm rate must be extremely low.

• Example:– KDD cup data set without filtering has a very

high base rate – no base rate fallacy.– What is good for the military, it is sometimes

very bad for a non-military environment.

72/108

Measuring IDS performances• Another performance measure: ROC

– Receiver Operating Characteristic– Used widely in systems for detection of

signals in noise (radars, etc.)– TPR vs. FPR curve– An ideal system has TPR=1 and FPR=0.

73/108

Measuring IDS performances• Example of a ROC curve:

% TPR

% FPR

IDS1

IDS2

74/108

Measuring IDS performances• The use of ROC curves for assessing IDS

has suffered criticism (McHugh, 2000):– Normally, an IDS would be characterised by a

single point in the coordinates FPR-TPR (However, if a parameter of an IDS is varied, the ROC curve is obtained, instead of a single point).

75/108

Measuring IDS performances• Example – ROC of the IDS with the

relabeling algorithm in which DB index and centroid diameters are implemented.

• The parameter: DeltaDB – varied between 0.2 and 0.45.

76/108


77/108

Test data sets

• For testing using simulated traffic/logs, a source of simulated traffic in which attacks are injected is needed.

• A widely used simulated traffic data set is the KDD cup ’99 data set.

78/108

KDD cup 1999

• It corresponds to testing IDS carried out by MIT Lincoln Laboratory in 1998 and 1999.

• In 1999, the KDD organized a contest in data mining and the data base used was that generated by Lincoln Laboratory.

79/108

KDD cup 1999

• KDD (SIGKDD) – ACM special interest group on knowledge discovery and data mining.

• The purpose of the KDD CUP ’99 contest was to classify the given data in order to differentiate attack records from the normal traffic records.

80/108

KDD cup 1999

• The KDD Cup 1999 Data (1)– Various intrusions simulated in a military air-

base network environment – 9 weeks of raw tcpdump data for a LAN simulating a typical U.S. Air Force LAN.

– 4,900,000 data instances – vectors of extracted feature values from connection records.

81/108

KDD cup 1999• The KDD Cup 1999 Data (2)

– Data were split into 2 parts:• The raw training data (4Gb of compressed binary

tcpdump – 7 weeks of network traffic – approx. 5 million connection records).

• Test data – 2 weeks – approx. 2 million connection records.

– A connection:• A sequence of TCP packets starting and ending at

some well defined time instants, between which data flow to and from a source IP address to a target IP address under a well defined protocol.

82/108

KDD cup 1999

• The KDD Cup 1999 Data (3)– Each connection is labelled as either normal

or as an attack, with exactly one specified attack type.

– Each connection record consists of about 100 bytes.

83/108

KDD cup 1999

• Four categories of simulated attacks– DoS – denial of service (e.g. Syn flood). – R2L – unauthorized access from a remote

machine (e.g. guessing password).– U2R – unauthorized access to superuser or

root functions (e.g. various “buffer overflow” attacks).

– Probing – surveillance and other probing for vulnerabilities (e.g. port scanning).

84/108

KDD cup 1999

• The test data do not have the same probability distribution as the training data.

• They include specific attack types not in the training data.

• This made the data mining task more realistic – the distribution of real data and types of possible attacks are normally not known during the training of the learning system.

85/108

KDD cup 1999

• The training data set contains 22 training attack types (1)– back DoS – buffer_overflow u2r – ftp_write r2l – guess_passwd r2l – imap r2l – ipsweep probe

86/108

KDD cup 1999

• The training data set attack types (2)– land dos– loadmodule u2r – multihop r2l – neptune dos – nmap probe – perl u2r

87/108

KDD cup 1999

• The training data set attack types (3)– phf r2l – pod dos– portsweep probe – rootkit u2r – satan probe – smurf dos

88/108

KDD cup 1999

• The training data set attack types (4)– spy r2l – teardrop dos – warezclient r2l – warezmaster r2l.

• The test data set contains 14 additional attack types.

89/108

KDD cup 1999• 41 higher level traffic features were

defined in order to help distinguishing normal connections from attacks.

• These features are divided into 3 categories:– Basic features of individual connections.– Content features within a connection

suggested by domain knowledge.– Traffic features computed using a 2-second

time window.

90/108

KDD cup 1999• Basic features of individual connections

(host-based traffic features):– Connection records were sorted by

destination host.– Features were constructed using a window of

100 connections to the same host instead of a time window.

– This is useful since some probing attacks scan the hosts (or ports) using a long time interval.

91/108

KDD cup 1999• Content features within a connection

suggested by domain knowledge:– These features look for suspicious behaviour

in the data portions, such as the number of failed login attempts.

92/108

KDD cup 1999• Traffic features computed using a two-

second time window (time based traffic features):– The same host features examine only the

connections in the past two seconds that have the same destination host as the current connection, and calculate statistics related to protocol behaviour, service, etc.

– The same service features examine only the connections in the past two seconds that have the same service as the current connection.

93/108

KDD cup 1999

• Basic features of individual connections (1)Feature

nameDescription Type

duration Length (in sec) of the connection Continuous

protocol_type Type of the protocol, e.g. tcp, udp, etc. Discrete

service Network service on the destination, e.g. http, telnet, etc.

Discrete

src_bytes Number of data bytes from source to destination

Continuous

dst_bytes Number of data bytes from destination to source

Continuous

94/108

KDD cup 1999

• Basic features of individual connections (2)

Feature name Description Type

flag Normal or error status of the connection

Discrete

land 1 if connection is from/to the same host/port; 0 otherwise

Discrete

wrong_fragment Number of “wrong” fragments Continuous

urgent Number of urgent packets Continuous

95/108

KDD cup 1999

• Content features (1)Feature name Description Type

hot Number of “hot” indicators Continuous

num_failed_logins Number of failed login attempts Continuous

logged_in 1 if successfully logged in; 0 otherwise

Discrete

num_compromised Number of “compromised” conditions

Continuous

root_shell 1 if root shell is obtained; 0 otherwise

Discrete

96/108

KDD cup 1999• Content features (2)Feature name Description Type

su_attempted 1 if “su root” command attempted; 0 otherwise

Discrete

num_root Number of “root” accesses Continuous

num_file_creations Number of file creation operations Continuous

num_shells Number of shell prompts Continuous

num_access_files Number of operations on access control files

Continuous

97/108

KDD cup 1999

• Content features (3)


num_outbound_cmds Number of outbound commands in an ftp session

Continuous

is_hot_login 1 if the login belongs to the “hot” list; 0 otherwise

Discrete

is_guest_login 1 if the login is a “guest” login; 0 otherwise

Discrete

98/108

KDD cup 1999

• Time-based traffic features (1)Feature name Description Type

count Number of connections to the same host as the current connection in the past 2 seconds

Continuous

The following features refer to so called “same host” connections

serror_rate % of connections that have “SYN” errors

Continuous

rerror_rate % of connections that have “REJ” errors

Continuous

99/108

KDD cup 1999

• Time-based traffic features (2)Feature name Description Type

“same host” connections (cont.)

same_srv_rate % of connections to the same service

Continuous

diff_srv_rate % of connections to different services

Continuous

srv_count Number of connections to the same service as the current connection in the past 2 seconds

Continuous

100/108

KDD cup 1999

• Time-based traffic features (3)


The following features refer to so called “same service” connections

srv_serror_rate % of connections that have “SYN” errors

Continuous

srv_rerror_rate % of connections that have “REJ” errors

Continuous

srv_diff_host_rate % of connections to different hosts

Continuous

101/108

KDD cup 1999• Selecting the right set of system features

is a critical step when formulating the classification tasks (in this case – intrusion detection algorithm).

102/108

KDD cup 1999• The 41 features were obtained by means

of the following process:– Frequent sequential patterns (frequent

episodes) from the network audit data were identified.

– These patterns were used as guidelines to select and construct temporal statistical features.

103/108

• Weaknesses of the KDD Cup data set:– Simulated data must be similar to real data –

there is no proof that KDD cup data are similar to real data.

– No anomalous packets that appear in real data.– No failure modes.– Synthetic attacks are not distributed realistically

in the background normal data.– Simulated TCP traffic is not diverse enough

(only 9 types of TCP traffic in KDD cup data set).

KDD cup 1999

104/108

• Stide (Sequence Time Delay Embedding) data set – collections of system calls– Instead of high-level features used in the KDD

CUP ’99 database, low level features are used in order to identify potential intrusions – sequences of system calls.

– In the training phase, Stide builds a database of all unique, contiguous system call sequences of a predetermined fixed length occurring in the traces.

Stide

105/108

• During testing, Stide compares sequences in the new traces to those in the database, and reports an anomaly measure indicating how much the new traces differ from the normal training data.

• 13726 traces of normal data were collected at the Computer Science Department, University of New Mexico.

Stide

106/108

• PESIM 2005 dataset – Fraunhofer Institute Berlin, Germany– Goal: to overcome the problem with the KDD

Cup 1999 dataset.– A combination of 5 servers in a virtual machine

environment (2 Windows, 2 Linux and 1 Solaris).

– HTTP, FTP, and SMTP services.– To achieve realistic traffic characteristics, news

sites were mirrored on the HTTP servers.– File sharing facilities offered on FTP servers.

PESIM 2005

107/108

• SMTP traffic injected artificially:– 70% mails from personal communication and

mailing lists.– 30% spam mails received by 5 individuals.

• Normal data pre-processed in the following way:– Random selection of 1000 TCP connections

from each protocol.– Attachments removed from the TCP traffic.

PESIM 2005

108/108

• Attacks against the simulated services generated by penetration testing tools.– Multiple instances of 27 different attacks were

launched against the HTTP, FTP, and SMTP services.

– The origin of the major part of the attacks is from the Metasploit environment.

– Some of the attacks were taken from the Bugtraq and Packet Storm Security lists.

• The problem: not publicly available!

PESIM 2005

testing ids

Documents

ids coverage

ids systems

ids technology

weaknesses of ids

surveillance attacks

different types of attacks

serious attacks

ddos attacks