CISC 879 - Machine Learning for Solving Systems Problems

Presented by: SatyajeetDept of Computer & Information Sciences

University of Delaware

Automatic Analysis of Malware Behavior using Machine LearningAuthor’s: Konrad Rieck, Philipp Trinius, Carsten Willems, and

Thosten Holz

CISC 879 - Machine Learning for Solving Systems Problems

Abstract & Introduction

• Malware - • Poses major threat to security of computer systems. • Very diverse – viruses, internet worms, trojan horses,• Amount of malware – millions of hosts infected

• Obfuscation and polymorphism impede detection at file level

• Dynamic analysis helps characterizing and defending.

CISC 879 - Machine Learning for Solving Systems Problems

Abstract & Introduction Contd..

• Framework for automatic analysis of malware behavior using Machine learning• Framework allows automatic analysis of novel

classes of malware with similar behavior – Clustering.

• Assigning unknown classes of malware to these discovered classes – Classification.

• An incremental approach based on both for behavior based analysis.

CISC 879 - Machine Learning for Solving Systems Problems

Automatic analysis of Malware Behavior

• Framework steps and procedure• Executing and monitoring malware binaries in

sandbox environment. Report generated on system calls and their arguments.

• Sequential reports are embedded in a vector space where each dimension is associated with a behavioral pattern.

• ML techniques then applied to the embedded reports to identify and classify malware.

• Incremental analysis progress by alternating between clustering and classification.

CISC 879 - Machine Learning for Solving Systems Problems

Report representation• Can be textual or XML

• Human readable and suitable for computation of general statistics

• But not efficient for automatic analysis

• Hence MIST (Malware Instr. Set)• Inspired from instr. set used in process design.

CISC 879 - Machine Learning for Solving Systems Problems

MIST

• Category of system calls• Operation - Reflects a particular system call• Arguments as argblocks.

CISC 879 - Machine Learning for Solving Systems Problems

Sandbox and MIST representation

CISC 879 - Machine Learning for Solving Systems Problems

Representation• These sequential reports identify typical behavior of

malware – Changing registry keys, modifying system files.

• But still not suitable for efficient analysis techniques. Hence the need to embed behavior reports in vector space – Using instruction q-grams.

• This embedding enables expressing the similarity of behavior geometrically – Calculating distance.

CISC 879 - Machine Learning for Solving Systems Problems

Clustering and Classification• Reports are embedded in vector space – Process

ready for applying ML techniques• Clustering of behavior – where classes of similar

behavior malware are identified.• Classification of behavior – which allows to assign

malware to known classes of behavior.• What allows us to do this? • Malware binaries are a family of similar variants

with similar behavior patterns !

CISC 879 - Machine Learning for Solving Systems Problems

Contd..

CISC 879 - Machine Learning for Solving Systems Problems

Algorithms

• Prototype extraction• Iterative algorithm• Extracts small set of prototypes from set of reports. First

one chosen at random.

• Clustering using Prototypes• Prototypes at beginning are individual clusters• Algorithm determines and merges nearest pairs of

clusters

• Classification using Prototypes• Allows to learn to discriminate between classes of

malware.

CISC 879 - Machine Learning for Solving Systems Problems

Algorithms Contd..• For each report algorithm determines the nearest

prototype of clusters in training data, if within radius then assigns to cluster

• Else rejects and holds back for later incremental analysis.

• Incremental analysis• Reports to be analyzed are received from source.• Initially classified using prototypes of known clusters• Thereby variants of known malware are identified for

further analysis.• Prototypes extracted from remaining reports and

clustered again.

CISC 879 - Machine Learning for Solving Systems Problems

Experiments and Results

CISC 879 - Machine Learning for Solving Systems Problems

Evaluating components• Prototype extraction

• Evaluated using Precision, Recall and Compression. • Precision – 0.99 when corpus compressed by 2.9 % & 7%

• Clustering• Evaluated using F-measure• F-measure for experiments – MIST 1 = 0.93 and MIST 2 =

0.95 better than previous related work 0.881

• Classification• F-measure for experiments – MIST 1= 0.96 and MIST 2 =

0.99

CISC 879 - Machine Learning for Solving Systems Problems

Experiments and Results Contd..

CISC 879 - Machine Learning for Solving Systems Problems

Experiments and Results Contd..

CISC 879 - Machine Learning for Solving Systems Problems

Conclusion

• A new framework introduced which overcomes several previous deficiencies.

• The framework is learning based• Framework can be implemented in practice

• Steps – Collection of malware, a study in sandbox environment, embed observed behavior in vector space, apply learning algorithms – clustering and classification.

• This process is efficient and learns automatically after initial setup and run.

CISC 879 - Machine Learning for Solving Systems Problems

Thank you !