annotated control flow graph for metamorphic malware detection › b156 › 64267e99d5a... ·...

14
c The British Computer Society 2014. All rights reserved. For Permissions, please email: [email protected] doi:10.1093/comjnl/bxu148 Annotated Control Flow Graph for Metamorphic Malware Detection Shahid Alam 1, Issa Traore 2 and Ibrahim Sogukpinar 3 1 Department of Computer Science, University of Victoria, Victoria, BC, Canada 2 Department of Electrical and Computer Engineering, University of Victoria, Victoria, BC, Canada 3 Department of Computer Engineering, Gebze Institute of Technology, Gebze, Kocaeli, Turkey Corresponding author: [email protected] Metamorphism is a technique that mutates the binary code using different obfuscations and never keeps the same sequence of opcodes in the memory. This stealth technique provides the capability to a malware for evading detection by simple signature-based (such as instruction sequences, byte sequences and string signatures) anti-malware programs. In this paper, we present a new scheme named Annotated Control Flow Graph (ACFG) to efficiently detect such kinds of malware. ACFG is built by annotating CFG of a binary program and is used for graph and pattern matching to analyse and detect metamorphic malware. We also optimize the runtime of malware detection through parallelization and ACFG reduction, maintaining the same accuracy (without ACFG reduction) for malware detection. ACFG proposed in this paper: (i) captures the control flow semantics of a program; (ii) provides a faster matching of ACFGs and can handle malware with smaller CFGs, compared with other such techniques, without compromising the accuracy; (iii) contains more information and hence provides more accuracy than a CFG. Experimental evaluation of the proposed scheme using an existing dataset yields malware detection rate of 98.9% and false positive rate of 4.5%. Keywords: annotated control flow graph; static binary analysis; malware detection; optimizations Received 24 August 2014; revised 7 November 2014 Handling editor: Steven Furnell 1. INTRODUCTION End point security is often the last defense against a security threat. An end point can be a desktop, server, laptop, kiosk or mobile device that connects to a network (Internet). Recent statistics by the International Telecommunications Union [1] show that the number of Internet users (i.e. people connecting to the Internet using these end points) in the world have increased from 20% in 2006 to 40% (almost 2.7 billion in total) in 2013. A study carried out by Symantec about the impacts of cybercrime reports, that worldwide losses due to malware attacks and phishing between July 2011 and July 2012 were $110 billion [2]. Because of the financial and other gains attached with the growing malware industry, there is a need to automate the process of malware analysis and provide real- time malware detection. In the early days, malware writers were hobbyists, however professionals have now become part of this group, either for the financial gains [3] attached to it, or for political and military reasons. One of the basic techniques used by a malware writer is obfuscation [4]. Such a technique obscures a code to make it difficult to understand, analyze and detect malware embedded in the code. Initial obfuscators were simple and detectable by sim- ple signature-based detectors. These signature-based detectors work on simple signatures such as byte sequences, instruc- tion sequences and string signatures (pattern of a malware that uniquely identifies it). However, they lack information about the semantics or behavior of the malicious program. To counter these detectors the obfuscation techniques evolved, producing metamorphic malware that use stealthy mutation techniques [48]. Likewise, to address effectively the chal- lenges posed by metamorphic malware, we need to develop new methods and techniques to analyze the behavior of a program and make a better detection decision with few false positives. Section D: Security in Computer Systems and Networks The Computer Journal, 2014 The Computer Journal Advance Access published December 15, 2014 at Qatar University on December 16, 2014 http://comjnl.oxfordjournals.org/ Downloaded from

Upload: others

Post on 03-Jul-2020

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Annotated Control Flow Graph for Metamorphic Malware Detection › b156 › 64267e99d5a... · detector to detect metamorphic malware. In [13], the authors use an intermediate language

c© The British Computer Society 2014. All rights reserved.For Permissions, please email: [email protected]

doi:10.1093/comjnl/bxu148

Annotated Control Flow Graph forMetamorphic Malware Detection

Shahid Alam1∗, Issa Traore2 and Ibrahim Sogukpinar3

1Department of Computer Science, University of Victoria, Victoria, BC, Canada2Department of Electrical and Computer Engineering, University of Victoria, Victoria, BC, Canada

3Department of Computer Engineering, Gebze Institute of Technology, Gebze, Kocaeli, Turkey∗Corresponding author: [email protected]

Metamorphism is a technique that mutates the binary code using different obfuscations and neverkeeps the same sequence of opcodes in the memory. This stealth technique provides the capabilityto a malware for evading detection by simple signature-based (such as instruction sequences, bytesequences and string signatures) anti-malware programs. In this paper, we present a new schemenamed Annotated Control Flow Graph (ACFG) to efficiently detect such kinds of malware. ACFG isbuilt by annotating CFG of a binary program and is used for graph and pattern matching to analyseand detect metamorphic malware. We also optimize the runtime of malware detection throughparallelization and ACFG reduction, maintaining the same accuracy (without ACFG reduction)for malware detection. ACFG proposed in this paper: (i) captures the control flow semanticsof a program; (ii) provides a faster matching of ACFGs and can handle malware with smallerCFGs, compared with other such techniques, without compromising the accuracy; (iii) containsmore information and hence provides more accuracy than a CFG. Experimental evaluation of theproposed scheme using an existing dataset yields malware detection rate of 98.9% and false positive

rate of 4.5%.

Keywords: annotated control flow graph; static binary analysis; malware detection; optimizations

Received 24 August 2014; revised 7 November 2014Handling editor: Steven Furnell

1. INTRODUCTION

End point security is often the last defense against a securitythreat. An end point can be a desktop, server, laptop, kioskor mobile device that connects to a network (Internet). Recentstatistics by the International Telecommunications Union [1]show that the number of Internet users (i.e. people connectingto the Internet using these end points) in the world haveincreased from 20% in 2006 to 40% (almost 2.7 billion intotal) in 2013. A study carried out by Symantec about theimpacts of cybercrime reports, that worldwide losses due tomalware attacks and phishing between July 2011 and July 2012were $110 billion [2]. Because of the financial and other gainsattached with the growing malware industry, there is a needto automate the process of malware analysis and provide real-time malware detection.

In the early days, malware writers were hobbyists, howeverprofessionals have now become part of this group, either forthe financial gains [3] attached to it, or for political and military

reasons. One of the basic techniques used by a malware writeris obfuscation [4]. Such a technique obscures a code to make itdifficult to understand, analyze and detect malware embeddedin the code.

Initial obfuscators were simple and detectable by sim-ple signature-based detectors. These signature-based detectorswork on simple signatures such as byte sequences, instruc-tion sequences and string signatures (pattern of a malwarethat uniquely identifies it). However, they lack informationabout the semantics or behavior of the malicious program. Tocounter these detectors the obfuscation techniques evolved,producing metamorphic malware that use stealthy mutationtechniques [4–8]. Likewise, to address effectively the chal-lenges posed by metamorphic malware, we need to developnew methods and techniques to analyze the behavior of aprogram and make a better detection decision with few falsepositives.

Section D: Security in Computer Systems and NetworksThe Computer Journal, 2014

The Computer Journal Advance Access published December 15, 2014 at Q

atar University on D

ecember 16, 2014

http://comjnl.oxfordjournals.org/

Dow

nloaded from

Page 2: Annotated Control Flow Graph for Metamorphic Malware Detection › b156 › 64267e99d5a... · detector to detect metamorphic malware. In [13], the authors use an intermediate language

2 S. Alam et al.

The main goal of this paper is to extract behavioral andstructural information from a program to detect the presenceof malware in the program. Control flow analysis (CFA)[9, 10] is one of the techniques used in compilers for programanalysis and optimization. The CFA of a program is expressedas a control flow graph (CFG). The CFG of a programrepresents all the paths that can be taken during the programexecution. Current techniques [11–20] that use CFG formalware detection are compute intensive (others have poordetection rate) and cannot handle malware with smaller CFGs.

It is difficult to write a new metamorphic malware [21]and in general malware writers reuse old malware. To hidedetection the malware writers change the obfuscations (syntax)more than the behavior (semantic) of such a new metamorphicmalware. If an unknown metamorphic malware uses all orsome of the same class of behaviors as are used by the trainingdataset (set of old metamorphic malware) then it is possible todetect these types of malware.

On the assumption and the motivation described above,we propose a new technique named Annotated ControlFlow Graph (ACFG) that can enhance the detection ofmetamorphic malware and can handle malware with smallerCFGs. We also optimize the runtime of malware detectionthrough parallelization and ACFG reduction, making thecomparison (matching with other ACFGs) faster, whilemaintaining the same accuracy (without ACFG reduction) formalware detection, than other techniques that use CFG formalware detection. The annotations in an ACFG provide moreinformation, and hence can provide more accuracy than a CFG.

ACFG1 proposed in this paper: (i) captures the controlflow semantics of a program; (ii) provides a faster matchingof ACFGs and can handle malware with smaller CFGs, incomparison with other such techniques, without compromisingthe accuracy; (iii) contains more information and henceprovides more accuracy than a CFG.

The rest of the paper is structured as follows. InSection 2, we discuss related research efforts that use CFG formetamorphic malware detection. In Section 3, we describe indetail the patterns used for CFG annotation, illustrate how abinary program can be translated to an ACFG and introduceour malware detection approach. In Section 4, we show howparallelization and ACFG reduction reduces the runtime of amalware detector. In Section 5, we evaluate the performance ofACFG using an existing malware dataset and compare it withother such techniques. We finally conclude in Section 6.

2. RELATED WORKS

This section discusses the previous research efforts that useCFG for metamorphic malware detection. We cover only

1The work presented in this paper is an expansion of the previouslypublished work in [22, 23].

academic research efforts that claim or will extend theirdetector to detect metamorphic malware.

In [13], the authors use an intermediate language calledCFGO-IL to simplify transformation of a program in thex86 assembly language to a CFG. After translating a binaryprogram to CFGO-IL, the program is optimized to makeits structure simpler. The optimizations also remove variousmalware obfuscations from the program. These optimizationsinclude dead code elimination, removal of unreachablebranches, constant folding and removal of fake conditionalbranches inserted by malware. The authors developed aprototype malware detection tool using CFGO-IL that takeadvantage of the optimizations and the simplicity of thelanguage. The size of a CFGO-IL program tends to increasecompared with the original assembly program and hence theCFG of the program. As mentioned in the paper [13], thetechnique proposed is not able to discriminate malware froma benign program if the size of the CFG is less than 100 nodes.ACFG can handle such smaller CFGs.

The method described in [14] uses model-checking to detectmetamorphic malware. Model-checking techniques check if agiven model meets a given specification. The paper [14] useda pushdown system to build a model that takes into accountthe behavior of the stack. They use IDA Pro [24] (a closesource disassembler) to build a CFG of a binary program. ThisCFG contains information about the register and the memorylocation values at each control point of the program, and istranslated into a pushdown system. The pushdown systemstores the control points and the stack of the program.

Model-checking is time consuming and it can sometimes runout of memory as was the case with the previous technique [25]by the same authors. The times reported range from a fewseconds (for a program consisting of 10 instructions) to over250 s (for a program consisting of 10 000 instructions). Real-life applications are much larger than the samples tested.

In [15], the authors compare the instructions of a basicblock in CFG of the original malware, with that of itsvariants. The comparison is performed using the longestcommon subsequence [26]. Unlike, other techniques they donot compare the shape of two CFGs, and hence is more timeefficient, but, as the preliminary results show, produce a greaternumber of false positives. This work is still in progress andtherefore the complete results are not available.

The technique described in [16] is the closest techniqueto the work presented in this paper. After disassemblinga program they perform normalization on the disassembledprogram. They then build an interprocedural CFG of thenormalized disassembled program. For malware detection,CFG of a benign program is checked to find if it contains asubgraph that is isomorphic to CFG of a malware program.The results reported in the paper are preliminary and areincomplete.

The method presented in [11, 12] uses CFG for visualizingthe control structure and representing the semantic aspects of a

Section D: Security in Computer Systems and NetworksThe Computer Journal, 2014

at Qatar U

niversity on Decem

ber 16, 2014http://com

jnl.oxfordjournals.org/D

ownloaded from

Page 3: Annotated Control Flow Graph for Metamorphic Malware Detection › b156 › 64267e99d5a... · detector to detect metamorphic malware. In [13], the authors use an intermediate language

Annotated Control Flow Graph for Metamorphic Malware Detection 3

program. They extended the CFG with the extracted API callsto have more information about the executable program. Thisextended CFG is called the API-CFG.

Their system consists of three components: PE (portableexecutable) file disassembler, API-CFG generator and classifi-cation module. They built the API-CFG as follows. First, theydisassemble a PE file using a third party disassembler. Thenthe unnecessary instructions that are not required for buildingthe CFG are removed from the disassembled instructions. Theinstructions kept are: jumps, procedure calls, API calls and tar-get of jump instructions.

A feature vector is generated using the API-CFG, which is asparse graph and can be represented by a sparse matrix. Theystore all the nonzero entries in a feature vector. An algorithmis given in the paper for converting the API-CFG to a featurevector.

Different classifiers (Decision Stump, SMO, Naive Bayes,Random Tree, Lazy K-Star and Random Forest) are used toprocess the data consisting of the feature vectors of the PE files,and then decide if a PE file is a malware or not.

The system implemented is dependent on a third partyclose source disassembler. The disassembler used cannotdisassemble more than one file at a time, so they used ascript to automate disassembling a set of files at once. Theproposed system cannot be used as a real-time malwaredetector. Furthermore, the proposed techniques cannot be usedto detect metamorphic malware, but as mentioned in the paperthis option will be explored in the future.

3. MALWARE ANALYSIS AND DETECTION USINGACFG

Before describing the technique proposed, we first define anACFG as follows:

Definition 1. A basic block is a maximal sequence ofinstructions with a single entry and a single exit point. Thereis no instruction after the first instruction that is the targetinstruction of a jump instruction, and only the last instructioncan jump to a different block. Instructions that can start a basicblock include the following. The first instruction, target of abranch or a function call, and a fall through instruction, i.e.an instruction following a branch, a function call or a returninstruction. Instructions that can end a basic block include thefollowing. A conditional or unconditional branch, a functioncall and a return instruction.

Definition 2. Control flow edge is an edge that representsa control flow between basic blocks. A control flow edge fromblock a to block b is denoted e = (a, b).

Definition 3. A CFG is a directed graph G = (V, E),

where V is the set of basic blocks and E is the set of control

flow edges. The CFG of a program represents all the paths thatcan be taken during the program execution.

Definition 4. An Annotated Control Flow Graph (ACFG)

is a CFG such that each statement of the CFG is assigned aMAIL Pattern.

3.1. Patterns for Annotation

We have recently proposed a new intermediate language namedMAIL (Malware Analysis Intermediate Language) [23]. Thereare 21 Patterns in the MAIL language. They are used to buildan ACFG of a program that can be used for graph and patternmatching during malware detection. In this section we describein detail these patterns as follows:ASSIGN: An assignment statement, e.g. EAX=EAX+ECX;ASSIGN_CONSTANT: An assignment statement including aconstant, e.g. EAX=EAX+0x01;CONTROL: A control statement where the target of the jumpis unknown, e.g. if (ZF == 1) JMP [EAX+ECX+0x10];CONTROL_CONSTANT: A control statement where the tar-get of the jump is known. For example, if (ZF == 1) JMP0x400567;CALL: A call statement where the target of the call isunknown, e.g. CALL EBX;CALL_CONSTANT: A call statement where the target of thecall is known, e.g. CALL 0x603248;FLAG: A statement including a flag, e.g. CF = 1;FLAG_STACK: A statement including flag register withstack, e.g. EFLAGS = [SP=SP–0x1];HALT: A halt statement, e.g. HALT;JUMP: A jump statement where the target of the jump isunknown, e.g. JMP [EAX+ECX+0x10];JUMP_CONSTANT: A jump statement where the target ofthe jump is known, e.g. JMP 0x680376JUMP_STACK: A return jump, e.g. JMP [SP=SP-0x8]LIBCALL: A library call, e.g. compare(EAX, ECX);LIBCALL_CONSTANT: A library call including a constant,e.g. compare(EAX, 0x10);LOCK: A lock statement, e.g. lock;STACK: A stack statement, e.g. EAX = [SP=SP–0x1];STACK_CONSTANT: A stack statement including a con-stant, e.g. [SP=SP+0x1] = 0x432516;TEST: A test statement, e.g. EAX and ECX;TEST_CONSTANT: A test statement including a constant,e.g. EAX and 0x10;UNKNOWN: Any unknown assembly instruction that cannotbe translated.NOTDEFINED: The default pattern, e.g. all the new state-ments when created are assigned this default value.where EAX, EBX, ECX are the general purpose registers, ZFand CF are the zero and carry flags, respectively, and SP is thestack pointer.

Section D: Security in Computer Systems and NetworksThe Computer Journal, 2014

at Qatar U

niversity on Decem

ber 16, 2014http://com

jnl.oxfordjournals.org/D

ownloaded from

Page 4: Annotated Control Flow Graph for Metamorphic Malware Detection › b156 › 64267e99d5a... · detector to detect metamorphic malware. In [13], the authors use an intermediate language

4 S. Alam et al.

3.2. Approach overview

Almost all malware use binaries to infiltrate a computersystem. Binary analysis is the process of automaticallyanalyzing the structure and behavior of a binary program. Weuse binary analysis for malware detection.

A binary program is first disassembled (for this paper, weassume that the binary is unpacked) and translated to a MAILprogram. We then build a CFG of the MAIL program andannotate it with patterns described above. This annotated CFG(ACFG) becomes part of the signature of the program and ismatched against a database of known malware samples to seeif the program contains malware or not. This approach is veryuseful in detecting known malware but may not be able todetect unknown malware.

To detect unknown malware, after a program sample istranslated to MAIL, an ACFG for each function in the programis built. Instead of using one large ACFG as a signature, wedivide a program into smaller ACFGs, with one ACFG perfunction. A program signature is then represented by the setof corresponding (smaller) ACFGs. A program that containspart of the control flow of a training malware sample, isclassified as malware, i.e. if a percentage (compared with somepredefined threshold computed empirically) of the numberof ACFGs involved in a malware signature match with thesignature of a program then the program is classified asmalware.

3.3. ACFG construction

To translate a binary program, we first disassemble the binaryprogram into an assembly program, and then translate thisassembly into an ACFG. We use a sample program, actuallya malware sample, to illustrate the steps involved in thetranslation as shown in Fig. 1. This example shows how anassembly program is translated into an ACFG.

The binary analysis of the function shown in Fig. 1 identifiesfive blocks in this function labeled 18–25. There are two sets ofcolumns separated by→. The first set of columns lists the hexdump of the function and the second set of columns lists thecorresponding translated MAIL statements with annotations.

The example shows the data embedded inside the codesection in block 27. This block is used to store, load andprocess data by the function as pointed out by the underlinedaddresses in the blocks. There are five instructions that changethe control flow of the program as indicated by the arcs in thefigure. There are two back edges 20 ← 22 and 19 ← 23,which indicate the presence of loops in the program. The jumpin block 24 indicates a jump out of the function.

Each MAIL statement in a block in the ACFG is annotatedwith a type also called a pattern. There are a total of 21patterns as explained in Section 3.1 that are used to build theACFG. For example, an assignment statement with a constantvalue and an assignment statement without a constant value are

two different patterns. Jump statements can have up to threepatterns and so on.

3.4. Subgraph matching

Before explaining the subgraph matching technique usedin this paper for malware detection, we first define graphisomorphism [27] as follows:

Let G = (VG, EG) and H = (VH , EH ) be any two graphs,where VG , VH and EG , EH are the sets of vertices and edgesof the graphs, respectively.

Definition 5. A vertex bijection (one-to-one mapping)

denoted as fV = VG → VH and an edge bijection denotedas fE = EG → EH are consistent if for every edge e ∈ EG

fV maps the endpoints of e to the endpoints of edge fE (e).

Definition 6. G and H are isomorphic graphs if thereexists a vertex bijection fV and an edge bijection fE that areconsistent. This relationship is denoted as G ∼= H.

An example of isomorphism is shown in Fig. 2. The edgesof graphs G and H1 are not consistent, e.g. edge {00, 10} ingraph G is not mapped to any edges in graph H1, thereforegraphs G and H1 are not isomorphic. Whereas the edges ofgraphs G and H2 are consistent, therefore graphs G and H2 areisomorphic.

In our malware detection approach, graph matching isdefined in terms of subgraph isomorphism. Given the inputof two graphs, subgraph isomorphism determines if one ofthe graphs contains a subgraph that is isomorphic to the othergraph. Generally, subgraph isomorphism is an NP-Complete(NP refers to nondeterministic polynomial time) problem [28].An ACFG of a program is usually a sparse graph, therefore itis possible to compute the isomorphism of two ACFGs in areasonable amount of time.

Based on the definition of graph isomorphism presentedabove, we formulate our ACFG matching approach as follows:

Let P = (VP , EP ) denote an ACFG of the program andM = (VM , EM ) denote an ACFG of the malware, whereVP , VM and EP , EM are the sets of vertices and edges of thegraphs, respectively. Let Psg = (Vsg, Esg) where Vsg ⊆ VP

and Esg ⊆ EP (i.e. Psg is a subgraph of P). If Psg ∼= M , thenP and M are considered as matching graphs.

After the binary analysis performed in Section 3.3 we obtaina set of ACFGs (each corresponding to a separate function) ofa program as shown in Fig. 1. To detect if a program containsa malware we compare the ACFGs of the program with theACFGs of known malware samples from our training database.If a percentage of the ACFGs of the program, greater than apredefined threshold, match one or several of the ACFGs of amalware sample (from the database) then the program will beclassified as malware. As part of the machine learning process,

Section D: Security in Computer Systems and NetworksThe Computer Journal, 2014

at Qatar U

niversity on Decem

ber 16, 2014http://com

jnl.oxfordjournals.org/D

ownloaded from

Page 5: Annotated Control Flow Graph for Metamorphic Malware Detection › b156 › 64267e99d5a... · detector to detect metamorphic malware. In [13], the authors use an intermediate language

Annotated Control Flow Graph for Metamorphic Malware Detection 5

FIGURE 1. Disassembly and translation to ACFG of one of the functions of a malware sample. The arcs are used to represent the control flowof the program and show the transfer of control from one block to another block.

Section D: Security in Computer Systems and NetworksThe Computer Journal, 2014

at Qatar U

niversity on Decem

ber 16, 2014http://com

jnl.oxfordjournals.org/D

ownloaded from

Page 6: Annotated Control Flow Graph for Metamorphic Malware Detection › b156 › 64267e99d5a... · detector to detect metamorphic malware. In [13], the authors use an intermediate language

6 S. Alam et al.

(a) (b) (c)

FIGURE 2. Example of graph isomorphism. Graphs G and H2 are isomorphic but not G and H1. (a) Graph G, (b) Graph H1, and (c) Graph H2.

(b)

(a)

FIGURE 3. An example of subgraph matching. The graph in (a) ismatched as a subgraph of the graph in (b). (a) A malware sampleand (b) the malware embedded (egg-shaped nodes) inside a benignprogram.

the threshold is computed empirically; by running the detectoron the training malware samples with different values ofthresholds; and picking a threshold with the best detection rate.An example of malware detection using subgraph matching isshown in Fig. 3.

3.5. Pattern matching

Very small graphs when matched against a large graph canproduce a false positive. Likewise to alleviate the impact ofsmall graphs on detection accuracy, we integrate a PatternMatching sub-component within the Subgraph Matchingcomponent. If an ACFG of a malware sample matches with anACFG of a program (i.e. the two ACFGs are isomorphic), thenwe further use the annotations/patterns, present in the ACFG,to match each statement in the matching nodes of the twoACFGs. A successful match requires all the statements in thematching nodes to have the same (exact) annotations, althoughthere could be differences in the corresponding statementblocks.

An example of Pattern Matching of two isomorphic ACFGsis shown in Fig. 4. One of the ACFGs of a malware sample,shown in Fig. 4a, is isomorphic to a subgraph of one of theACFGs of a benign program, shown in Fig. 4b. Consideringthese two ACFGs as a match for malware detection willproduce a wrong result, a false positive. The statements inthe benign program do not match with the statements in themalware sample. To reduce this false positive we have twooptions: (i) we can match each statement exactly with eachother or (ii) assign patterns to these statements for matching.Option (i) will not be able to detect unknown malware samplesand is time consuming, so we use option and (ii) in ourapproach, which in addition to reducing false positives has thepotential of detecting unknown malware samples.

For a successful pattern matching we require all thestatements in the matching blocks to have the same patterns. InFig. 4, only the statements in block 0 satisfy this requirement.The statements in all the other blocks do not satisfy thisrequirement, therefore these ACFGs fail the pattern matching.

4. RUNTIME OPTIMIZATION

The Subgraph Matching component matches the ACFGagainst all the malware sample graphs. As the number ofnodes in the graph increases the Subgraph Matching runtimeincreases. The runtime also increases with the increase in thenumber of malware samples but provide some options for

Section D: Security in Computer Systems and NetworksThe Computer Journal, 2014

at Qatar U

niversity on Decem

ber 16, 2014http://com

jnl.oxfordjournals.org/D

ownloaded from

Page 7: Annotated Control Flow Graph for Metamorphic Malware Detection › b156 › 64267e99d5a... · detector to detect metamorphic malware. In [13], the authors use an intermediate language

Annotated Control Flow Graph for Metamorphic Malware Detection 7

(b)(a)

FIGURE 4. Example of pattern matching of two isomorphic ACFGs. The ACFG in (a) is isomorphic to the subgraph (blocks 0–3) of the ACFGin (b).

optimization. We use two techniques to improve the runtime,namely, parallelization and ACFG reduction, described in thefollowing.

4.1. Parallelization

Figure 5 shows an example of matching ACFGs for malwaredetection as explained in Section 3.2. The Malware programhas 4 ACFGs (functions) represented as {m1, m2, m3, m4} andthe benign program has nine ACFGs (functions) represented as{b1, b2, b3, b4, b5, b6, b7, b8, b9}. Each ACFG of the malwareprogram is matched with ACFGs of the benign program. Theexample has the following successful matches: m1 → b5,m2 → b4 and m4 → b9. The three ACFGs of the malwareprogram successfully match with the three ACFGs out of thetotal nine ACFGs of the benign program, i.e. over 25% match.Using a threshold of 25% the benign program is detected asmalware.

The example discussed above, is matching one benignsample with one malware sample. Similarly, each testingsample is matched with all the training malware samples in the

FIGURE 5. Matching of ACFGs for malware detection. ACFGs atthe top belongs to a malware program and ACFGs at the bottombelongs to a benign program. A successful match is shown througha bold line with arrow on both sides.

dataset. Based on these matchings, there are two opportunitiesto parallelize the Subgraph Matching component.

1. Each ACFG matching in a sample is independent of theother matchings, these matchings can be performed inparallel.

2. Each sample can be processed independently of theother samples, the processing of the samples can beperformed in parallel.

Section D: Security in Computer Systems and NetworksThe Computer Journal, 2014

at Qatar U

niversity on Decem

ber 16, 2014http://com

jnl.oxfordjournals.org/D

ownloaded from

Page 8: Annotated Control Flow Graph for Metamorphic Malware Detection › b156 › 64267e99d5a... · detector to detect metamorphic malware. In [13], the authors use an intermediate language

8 S. Alam et al.

Multicore processors are becoming popular nowadays. Allthe current desktops, laptops and even the energy efficientsmall mobile devices contain a multicore processor. Intel hasrecently announced its Single-Chip Cloud Computer [29], aprocessor with 48 cores on a chip. Keeping in view this ubiq-uitousness of multicores in the host machines (also called theend points), the two opportunities listed above and to optimizethe runtime, we decided to use Windows threads to parallelizethe Subgraph Matching component. The following paragraphsgive an overview of our thread implementation approach.

We create two sets of threads. One for each sample undertest in the dataset, called the global threads, and the otherfor each ACFG matching in the sample, called the localthreads, i.e. each sample runs in its own thread and this threadcreates children threads, one for each ACFG in the sample.A local thread manager (LTM) is implemented to synchronizebetween the set of threads local to a sample. The LTM countsthe number of successful matches and sends a signal to allother local threads to stop if the count crosses the threshold,i.e. the sample is malware and hence there is no need forfurther ACFG matching. A global thread manager (GTM) isimplemented to synchronize between the global threads createdfor each sample. The GTM and LTM count and keep thetotal number of threads in the system to MAX_THREAD.The value of MAX_THREAD is computed using Equation (1).Threads share and synchronize information (such as threadcount etc.) through a shared variable. To decrease the wait timeand improve the runtime of a thread for next matching, otherinformation such as ACFGs are not shared among threads, andare copied to the memory space of each thread.

To evaluate the effectiveness of our parallelization approachdescribed above, we carried out an experiment using a differentnumber of threads ranging from 2 to 250. Another reasonfor this experiment was to develop an empirical model tocompute the number of threads to be used in the SubgraphMatching component. We used 250 malware samples and 30benign applications each with a different size of ACFG. Theexperiment was run on machines with 2 and 4 Cores. Details ofthe machines used and the results of this experiment are shownin Table 1. The percentage of CPU utilization on average wasthree times more with threads than without threads. This is alsoconfirmed by the improvement in the runtime.

As we increased the number of threads the reduction inruntime also increased up to a certain number (8 in 2 Coresand 64 in 4 Cores) of threads, after which the reduction inthe runtime started decreasing. As we increased the numberof benign samples from 30 to 1387 (the last row in Table 1)the runtime only reduced approximately by two times. Thisshows that increasing the number of samples will not effectthe results of this empirical study. Beside other factors such asthe number of CPUs/Cores and the memory available to theapplication, the other reasons for this reduction in the runtimeare the fact that more time was spent on thread managementand the increased sharing of information among threads.

TABLE 1. Runtime optimization by parallelizing the SubgraphMatching component using different number of threads.

2 Cores 4 Cores

Number of Runtime reduced Number of Runtime reducedthreads by (times) threads by (times)

2 3.20 4 3.984 5.92 8 4.118 7.20 32 5.9516 5.72 64 7.7632 5.64 128 7.53250 5.41 250 6.378 14.87 64 14.53

Machines used:Intel Core i5 CPU M 430 (2 Cores) @ 2.27 GHz with 4 GB of RAMand Windows 8 Professional installedIntel Core 2 Quad CPU Q6700 (4 Cores) @ 2.67 GHz with 4 GB ofRAM and Windows 7 Professional installed.

Based on this experiment, we developed Equation (1) tocompute the maximum number of threads to be used bythe Subgraph Matching component. We also give an option tothe user to choose the maximum number of threads used by thetool for the Subgraph Matching component.

Threads = (NC)3 (1)

where NC = Number of CPUs/Cores.

4.2. ACFG reduction

One of the advantages of using ACFG is that it containsannotations for malware detection. Our detection method usesboth subgraph matching and pattern matching techniques formetamorphic malware detection. Even if we reduce the numberof blocks in an ACFG (it is possible for an ACFG of somebinaries to be reduced to a very small number of blocks) westill get a good detection rate because of the combination ofthe two techniques.

To reduce the number of blocks (nodes) in an ACFG forruntime optimization, we carried out ACFG reduction, alsocalled ACFG shrinking. We reduce the number of blocks in anACFG by merging them together. Two blocks are merged onlyif the merging does not change the control flow of a program.

Given two blocks A and B in an ACFG, if all the paths thatreach node B pass through block A, and all the children of Aare reachable through B, then A and B will be merged.

Figure 6 shows an example of ACFG, from the datasetdescribed in this paper, before and after shrinking. Theshrinking does not change the shape of a graph. As we can see

Section D: Security in Computer Systems and NetworksThe Computer Journal, 2014

at Qatar U

niversity on Decem

ber 16, 2014http://com

jnl.oxfordjournals.org/D

ownloaded from

Page 9: Annotated Control Flow Graph for Metamorphic Malware Detection › b156 › 64267e99d5a... · detector to detect metamorphic malware. In [13], the authors use an intermediate language

Annotated Control Flow Graph for Metamorphic Malware Detection 9

(a)

(b)

FIGURE 6. ACFG of one of the functions of one of the samples of the MWOR class of malware, before and after shrinking. The ACFG has beenreduced from 92 nodes to 47 nodes. (a) Before shrinking and (b) after shrinking.

in the figure, the shape of the graphs before and after shrinkingare the same. More of these examples are available at [30].

We were able to substantially reduce the number of nodesper ACFG (in total a 90.6% reduction), as shown in Table 4.This reduced the runtime on average by 2.7 times (for a smallerdataset) and 100 times (for a larger dataset), and still achieveda detection rate of 98.9% with a false positive rate of 4.5% asis latter shown in Table 6.

5. EXPERIMENTS

We conducted an experiment to evaluate the performance ofour malware detection technique. The evaluation was carriedout using the prototype implementation of our detector namedMARD (for Metamorphic malware Analysis and Real-timeDetection) [22]. MARD fully automates the malware analysisand detection process, without any manual intervention dur-ing a complete run. We present, in this section, the evaluationmetrics, experimental settings, obtained results and a compari-son of ACFG with other such techniques.

5.1. Performance metrics

Before evaluating the performance of our malware detectiontechnique, we first define the following performance metrics:

True positive (TP) is the number of malware that areclassified as malware. True negative (TN) is the number ofbenign programs that are classified as benign. False positive(FP) is the number of benign programs that are classified asmalware. False negative (FN) is the number of malware thatare classified as benign.

Precision is the fraction of detected malware samples thatare correctly detected. Accuracy is the fraction of samples,including malware and benign, that are correctly detected aseither malware or benign. These two metrics are defined asfollows:

Precision = TP

TP+ FPAccuracy = TP+ TN

P + N

where P and N are the total number of malware and benignprograms, respectively. Now we define the mean maximumprecision (MMP) and mean maximum accuracy (MMA) for

Section D: Security in Computer Systems and NetworksThe Computer Journal, 2014

at Qatar U

niversity on Decem

ber 16, 2014http://com

jnl.oxfordjournals.org/D

ownloaded from

Page 10: Annotated Control Flow Graph for Metamorphic Malware Detection › b156 › 64267e99d5a... · detector to detect metamorphic malware. In [13], the authors use an intermediate language

10 S. Alam et al.

n-fold cross-validation as follows:

MMP = 1

n

n∑

i=1

Precisioni (2)

MMA = 1

n

n∑

i=1

Accuracyi (3)

We also use two other metrics, TP rate (TPR) and FP rate(FPR). The TPR, also called detection rate (DR), correspondsto the percentage of samples correctly recognized as malwareout of the total malware dataset. The FPR metric correspondsto the percentage of samples incorrectly recognized as malwareout of the total benign dataset. These two metrics are definedas follows:

TPR = TP

P(4)

FPR = FP

N(5)

5.2. Dataset

The dataset used for the experiments consists of total 3350sample Windows and Linux programs. Out of the 3350programs, 1020 are metamorphic malware samples collectedfrom three different resources [31–33], and the other 2330are benign programs. The 1020 malware samples belongto the following three metamorphic family of viruses: NextGeneration Virus Generation Kit (NGVCK) [34], SecondGeneration Virus Generator (G2) [35] and Metamorphic Worm(MWOR) generated by metamorphic generator [31]. NGVCKand MWOR family of viruses are further divided into two andseven Classes, respectively. This Class distribution is shown inTable 2.

The dataset distribution based on the number of ACFGsfor each sample, and size of the ACFG after normalizationand shrinking is shown in Tables 3 and 4, respectively. Thenormalizations carried out are removal of NOP (no operation)and other junk instructions.

The dataset contains a variety of programs with ACFGsranging from simple to complex for testing. As shown inTable 3, the number of ACFGs per malware sample rangesfrom 2 to 1272 and the number of ACFGs per benign programranges from 0 to 1148. Some of the Windows dynamic linklibraries that were used in the experiment do not have code butonly data (i.e. they cannot be executed) and as a result theyhave 0 node ACFGs. The sizes of these ACFGs are shown inTable 4. The size of the ACFGs of the malware samples rangefrom 1 to 301 nodes, and the size of the ACFGs of the benignprograms range from 1 to 521 nodes.

This variety of ACFGs and malware classes in the samplesprovides a good testing platform for the proposed malwaredetection framework.

TABLE 2. Class distribution of the 1020 metamorphic malwaresamples.

Number ofClass samples Comments

NGVCK_1 70 Generated by NGVCK with simple setof obfuscations, such as dead codeinsertion and instruction reorderingetc.

NGVCK_2 200 Generated by NGVCK with complexset of obfuscations, such as indirectjump (e.g. push followed by a retinstruction) to one of the datasections etc.

G2 50 Generated by G2MWOR_1 100 Generated by MWOR with a padding

ratio of 0.5MWOR_2 100 Generated by MWOR with a padding

ratio of 1.0MWOR_3 100 Generated by MWOR with a padding

ratio of 1.5MWOR_4 100 Generated by MWOR with a padding

ratio of 2.0MWOR_5 100 Generated by MWOR with a padding

ratio of 2.5MWOR_6 100 Generated by MWOR with a padding

ratio of 3.0MWOR_7 100 Generated by MWOR with a padding

ratio of 4.0

MWOR uses two morphing techniques: dead code insertion andequivalent instruction substitution. Padding ratio is the ratio of thenumber of dead code instructions to the core instructions of themalware. Padding ratio of 0.5 means that the malware has half asmuch dead code instructions as core instructions [31].

5.3. Empirical study

In this section, we discuss the experiments, and present theperformance results obtained. The experiments were run onthe following machine: Intel Core i5 CPU M 430 (2 Cores)@ 2.27 GHz with 4 GB RAM, operating system Windows 8professional.

We carried out two experiments: one with a smaller datasetusing 10-fold cross validation and the other with a largerdataset using 5-fold cross validation.

To find the runtime improvement by ACFG reduction, boththese experiments were carried out with and without ACFGreduction. We were able to complete all the runs (10 times)with and without ACFG reduction for the smaller dataset. Wecompleted all the runs (five times) with ACFG reduction withthe larger dataset but completed only one run (took over 34 hto complete) without ACFG reduction. This is one of the otherreasons to use a smaller dataset with a larger dataset, to getaccurate results for runtime improvement. In the following two

Section D: Security in Computer Systems and NetworksThe Computer Journal, 2014

at Qatar U

niversity on Decem

ber 16, 2014http://com

jnl.oxfordjournals.org/D

ownloaded from

Page 11: Annotated Control Flow Graph for Metamorphic Malware Detection › b156 › 64267e99d5a... · detector to detect metamorphic malware. In [13], the authors use an intermediate language

Annotated Control Flow Graph for Metamorphic Malware Detection 11

TABLE 3. Dataset distribution based on the number of ACFGs foreach program sample.

Malware samples (1020) Benign program samples (2330)

Number of Number of Number of Number ofACFGs samples ACFGs samples

2 250 0–50 12775–32 204 51–100 31733–57 222 101–199 31058–84 133 201–399 28285–133 105 400–598 111140–249 94 606–987 30133–1272 12 1001–1148 3

TABLE 4. Dataset distribution based on the size (number of nodes)for each ACFG after normalization and shrinking.

Malware samples (1020) Benign program samples (2330)

Number of Number of Number of Number ofnodes ACFGs nodes ACFGs

1–10 60 284 1–10 242 24011–20 1652 11–20 346621–39 2177 21–39 108641–69 288 40–69 36870–96 207 70–99 61104–183 254 100–194 60221–301 2 245–521 15

Total number of nodes before ACFG shrinking = 4 908 422. Totalnumber of nodes after ACFG shrinking = 462 974 (90.6% reduced).Runtime on average reduced by 2.7 times (for smaller dataset) and100 times (for larger dataset).

sections, we present and describe these two experiments withthe results.

5.3.1. Experiment with smaller dataset using 10-fold crossvalidation

Out of 3350 Windows programs we selected randomly 1351Windows programs. Out of these 1351 programs, 250 aremetamorphic malware samples and the other 1101 are benignprograms.

The 10-fold cross validation was conducted by selecting 25malware samples out of the 250 malware to train our detector.The remaining 225 malware samples along with the 1101benign programs were then used to test the detector. Thesetwo steps were repeated 10 times and each time a differentset of 25 malware samples were selected for training and theremaining samples for testing. The overall performance resultswere obtained by averaging the results obtained in the 10different runs.

TABLE 5. Malware detection results for smaller dataset.

Trainingset size DR (%) FPR (%) MMP MMA

25 94 3.1 0.86 0.96125 99.6 4 0.85 0.97

On average it took MARD 15.2037 s with ACFG shrinking and40.4288 s without ACFG shrinking to complete the malwareanalysis and detection for 1351 samples including 25 trainingmalware samples. This time excludes time for the training. MARDachieved the same values for all the other performance metrics (DR,FPR, MMP and MMA) with and without ACFG shrinking.

TABLE 6. Malware detection results for larger dataset.

Trainingset size DR (%) FPR (%) MMP MMA

204 97 4.3 0.91 0.96510 98.9 4.5 0.91 0.97

On average it took MARD 946.5824 s with ACFG shrinking andover 125 400 s (over 34 h) without ACFG shrinking to completethe malware analysis and detection for 3350 samples including 204training malware samples. This time excludes time for the training.Because of the time constraints we did not perform 5-fold crossvalidation without ACFG shrinking. The time (over 34 h) reportedis just for one run of the experiment without ACFG shrinking.

Using a threshold value of 25%, we conducted furtherevaluation by increasing the size of the training set from25 samples to 125 malware samples (50% of the malwaresamples). The obtained results are listed in Table 5. The DRimproved from 94% when the size of the training set is 25–99.6% when we used a training dataset of 125 samples.

5.3.2. Experiment with larger dataset using 5-fold crossvalidation

The 5-fold cross validation was conducted by selecting 204malware samples out of the 1020 malware to train our detector.The remaining 816 malware samples along with the 2330benign programs in our dataset were then used to test thedetector. These two steps were repeated five times and eachtime a different set of 204 malware samples were selected fortraining and the remaining samples for testing. The overallperformance results were obtained by averaging the resultsobtained in the five different runs.

Using a threshold value of 25%, we conducted furtherevaluation by increasing the size of the training set from204 samples to 510 malware samples (50% of the malwaresamples). The obtained results are listed in Table 6. The DRimproved from 97% when the size of the training set was 204–98.9% when we used a training dataset of 510 samples.

Section D: Security in Computer Systems and NetworksThe Computer Journal, 2014

at Qatar U

niversity on Decem

ber 16, 2014http://com

jnl.oxfordjournals.org/D

ownloaded from

Page 12: Annotated Control Flow Graph for Metamorphic Malware Detection › b156 › 64267e99d5a... · detector to detect metamorphic malware. In [13], the authors use an intermediate language

12 S. Alam et al.

TABLE 7. Summary and comparison with ACFG of the metamorphic malware analysis and detection systems discussed in Section 2 and others.

Data set sizeSystem Analysis type DR (%) FPR (%) benign/malware Real-time Platform

ACFG Static 98.9 4.5 2330/1020 ✓ Win & Linux 64API-CFG [11, 12] Static 97.53 1.97 2140/2305 ✗ Win 32Call-Gram [36] Static 98.4 2.7 3234/3256 ✗ Win 32CFGO-IL [13] Static 100 12 100/100 ✗ Win 32Chi-squared [37] Static ∼98 ∼2 40/200 ✗ Win & Linux 32Code-graph [38] Static 91 0 300/100 ✗ Win 32DTA [39] Dynamic 100 3 56/ 42 ✗ Win XP 64Opcode-HMM-Wong [40] Static ∼90 ∼2 40/200 ✗ Win & Linux 32Opcode-HMM-Austin [41] Static 93.5 0.5 102/77 ✗ Win & Linux 32Opcode-SD [42] Static ∼98 ∼0.5 40/800 ✗ Linux 32Opcode-graph [33] Static 100 1 41/200 ✗ Win & Linux 32Model-checking [14] Static 100 1 8/200 ✗ Win 32MSA [43] Static 91 52 150/1209 ✗ Win 32VSA-1 [44] Static 100 0 25/30 ✗ Win 32VSA-2 [45] Dynamic 98 2.9 385/826 ✗ Win XP 64

Real-time here means the detection is fully automatic and finishes in a reasonable amount of time as shown in Tables 5 and 6. Some of the reasonswhy other techniques does not provide real-time detection are mentioned in Section 2. Moreover, none of the other techniques compute or mentiontheir runtime for malware detection. The perfect results should be validated with more number of samples than tested in the paper. The values forOpcode-Graph are not directly mentioned in the paper. We compute these values by picking a threshold of 0.5 for the similarity score in the paper.

5.4. Comparison with others

Table 7 gives a comparison of ACFG with the researchefforts of detecting the metamorphic malware discussed inSection 2 and others, that do not use CFG and claim todetect metamorphic malware. None of the prototype systemsimplemented can be used as a real-time detector. Furthermorea few systems that claim a perfect detection rate were validatedusing small datasets.

Out of all the research efforts, API-CFG, Call-Gramand VSA-2 show impressive results. API-CFG does not yetsupport detection of metamorphic malware, VSA-2 is usinga controlled environment for detection, and Call-Gram isnot fully automated and their performance overheads arenot mentioned in the paper. The dataset used by VSA-2 iscomparatively smaller than the other two. Most of the othertechniques, such as Chi-Squared, Opcode-SD, Opcode-Graphand Opcode-Histogram show good results, and some of themmay have the potential to be used in a real-time detector byimproving their implementation.

Table 7 reports the best DR results achieved by thesedetectors. Out of the 15 systems, ACFG clearly shows superiorresults (the perfect results in Table 7 should be validatedwith more samples) and, unlike others is fully automatic,supports metamorphic malware detection for 64 bit Windows(PE binaries) and Linux (ELF binaries) platforms and has thepotential to be used as a real-time detector.

We have also compared our malware detection with fourcommercial antimalware programs, including AVG AntiVirus,

Microsoft Security Essentials, McAfee and Kaspersky. Weused G2 and NGVCK malware families (320 out of the1020 malware samples, Table 2) for testing these antimalwareprograms. Out of the 320 malware samples, MicrosoftSecurity Essentials detected 20 (6.25%), McAfee detected 268(83.75%), Kaspersky detected 268 (83.75%) and AVG detected274 (85.63%). The combined results of these antimalwareprograms tagged all the 320 samples as malware, however,none of the individual programs achieved a DR over 86%,whereas ACFG achieved a much higher DR.

6. CONCLUSION

In this paper, we have presented a new technique named ACFG,and shown through experimental evaluation its effectiveness inmetamorphic malware analysis and detection and the ability tohandle malware with smaller CFGs. We have also optimizedthe runtime of malware detection through parallelizationand ACFG reduction, that makes the comparison (matchingwith other ACFGs) faster, while keeping the same accuracy(without ACFG reduction) for malware detection, thanother techniques that use CFG for malware detection. Theannotations in an ACFG provide more information, and hencecan provide more accuracy than a CFG.

Currently, we are carrying out further research into usingsimilar techniques for web malware analysis and detection.Since metamorphic malware is a challenging test case, webelieve that our proposed technique can also be effectively

Section D: Security in Computer Systems and NetworksThe Computer Journal, 2014

at Qatar U

niversity on Decem

ber 16, 2014http://com

jnl.oxfordjournals.org/D

ownloaded from

Page 13: Annotated Control Flow Graph for Metamorphic Malware Detection › b156 › 64267e99d5a... · detector to detect metamorphic malware. In [13], the authors use an intermediate language

Annotated Control Flow Graph for Metamorphic Malware Detection 13

used for the detection of other malware and would like tocarry out such experiments in the future. Our future workwill also consist of strengthening our existing algorithmsby investigating and incorporating more powerful patternrecognition techniques.

REFERENCES

[1] International Telecommunications Union (ITU) (2011) TheWorld in 2011: ICT Facts and Figures. http://www.itu.int/ITU-D/ict/facts/2011/material/ICTFactsFigures2011.pdf

[2] Symantec, C. (2012) Norton Cybercrime Report.http://now-static.norton.com/now/en/pu/images/Promotions/2012/cybercrimeReport/2012_Norton_Cybercrime_Report_Master_FINAL_050912.pdf

[3] Bauer, J., Eeten, M. and Wu, Y. (2008) ITU Study on theFinancial Aspects of Network Security: Malware and Spam.http://www.itu.int/ITUD/cyb/cybersecurity/docs/itu-study-financial-aspects-of-malware-and-spam.pdf

[4] Collberg, C., Thomborson, C. and Low, D. (1998) Manufac-turing Cheap, Resilient, and Stealthy Opaque Constructs. Proc.25th ACM SIGPLAN-SIGACT Sym. Principles of ProgrammingLanguages, POPL’98, New York, NY, USA, January, pp. 184–196. ACM.

[5] OKane, P., Sezer, S. and McLaughlin, K. (2011) Obfuscation:the hidden malware. IEEE Secur. Priv., 9, 41–47.

[6] Linn, C. and Debray, S. (2003) Obfuscation of Executable Codeto Improve Resistance to Static Disassembly. ACM CCS, NewYork, NY, USA, October, pp. 290–299. ACM.

[7] Kuzurin, N., Shokurov, A., Varnovsky, N. and Zakharov, V.(2007) On the Concept of Software Obfuscation in ComputerSecurity. Proc. 10th Int. Conf. Information Security, ISC’07,Berlin, Heidelberg, October, pp. 281–298. Springer.

[8] Borello, J.-M. and Mé, L. (2008) Code obfuscation techniquesfor metamorphic viruses. J. Comput.Virol., 4, 211–220.

[9] Aho, A.V., Lam, M.S., Sethi, R. and Ullman, J.D. (2006)Compilers: Principles, Techniques, and Tools (2nd edn).Addison-Wesley Longman Publishing Co., Inc., Boston, MA,USA.

[10] Muchnick, S.S. (1997) Advanced Compiler Design andImplementation. Morgan Kaufmann Publishers, Inc., SanFrancisco, CA, USA.

[11] Eskandari, M. and Hashemi, S. (2012) A graph mining approachfor detecting unknown malwares. J. Vis. Lang. Comput., 23,154–162.

[12] Eskandari, M. and Hashemi, S. (2012) ECFGM: Enrichedcontrol flow graph miner for unknown vicious infected codedetection. J. Comput. Virol., 8, 99–108.

[13] Anju, S.S., Harmya, P., Jagadeesh, N. and Darsana, R. (2010)Malware Detection Using Assembly Code and Control FlowGraph Optimization. A2CWiC, 2010, New York, NY, USA,September, p. 65. ACM.

[14] Song, F. and Touili, T. (2012) Efficient malware detection usingmodel-checking. Form. Methods, 7436, 418–433.

[15] Vinod, P., Laxmi, V., Gaur, M.S., Kumar, G.P. and Chundawat,Y.S. (2009) Static CFG Analyzer for Metamorphic Malware

Code. Proc. 2nd Int. Conf. Security of Information andNetworks, SIN’09, New York, NY, USA, October, pp. 225–228.ACM SIGSAC.

[16] Bruschi, D., Martignoni, L. and Monga, M. (2006) DetectingSelf-Mutating Malware Using Control-Flow Graph Matching.DIMVA, 2006, July, pp. 129–143. Springer, Berlin, Heidelberg.

[17] Cesare, S. and Xiang, Y. (2011) Malware Variant DetectionUsing Similarity Search over Sets of Control Flow Graphs.TrustCom, 2011, Changsha, China, November, pp. 181–189.IEEE Computer Society.

[18] Guo, H., Pang, J., Zhang, Y., Yue, F. and Zhao, R. (2010)Hero: A Novel Malware Detection Framework based on BinaryTranslation. ICIS, 2010, Xiamen, China, October, pp. 411–415.IEEE.

[19] Kirda, E., Kruegel, C., Banks, G., Vigna, G. and Kemmerer,R.A. (2006) Behavior-Based Spyware Detection. Proc. 15thConf. USENIX Security Symposium, vol. 15, USENIX-SS’06,Berkeley, CA, USA, April 19. USENIX Association.

[20] Flake, H. (2004) Structural Comparison of Executable Objects.In Flegel, U. and Michael, M. (eds), DIMVA, LNI 46, Dortmund,Germany, July, pp. 161–173. GI.

[21] Szor, P. (2005) The Art of Computer Virus Research and Defense.Addison-Wesley Professional, Boston, MA, USA.

[22] Alam, S., Horspool, R.N. and Traore, I. (2014) MARD: AFramework for Metamorphic Malware Analysis and Real-TimeDetection. Advanced Information Networking and Applications,Research Track—Security and Privacy, AINA’14, Washington,DC, USA, May, pp. 480–489. IEEE Computer Society.

[23] Alam, S., Horspool, R.N. and Traore, I. (2013) MAIL: MalwareAnalysis Intermediate Language—A Step Towards Automatingand Optimizing Malware Detection. Security of Information andNetworks, SIN’13, New York, NY, USA, November, pp. 233–240. ACM SIGSAC.

[24] Eagle, C. (2008) The IDA Pro Book: The Unofficial Guide tothe World’s Most Popular Disassembler. No Starch Press, SanFrancisco, CA, USA.

[25] Song, F. and Touili, T. (2012) Pushdown model checking formalware detection. In Flanagan, C. and Konig, B. (eds), Toolsand Algorithms for the Construction and Analysis of Systems,Lecture Notes in Computer Science 7214. Springer, Berlin,Heidelberg, Germany.

[26] Cormen, T.H., Leiserson, C.E., Rivest, R.L. and Stein, C.(2009) Introduction to Algorithms (3rd edn). The MIT Press,Cambridge, MA, USA.

[27] Gross, J.L. and Yellen, J. (2005) Graph Theory and ItsApplications, Second Edition (Discrete Mathematics and ItsApplications). Chapman & Hall/CRC, Boca Raton, FL, USA.

[28] Cook, S.A. (1971) The Complexity of Theorem-ProvingProcedures. Proc. 3rd Annual ACM Symposium on Theory ofComputing, STOC’71, New York, NY, USA, May, pp. 151–158.ACM.

[29] Mattson, T.G., et al. (2010) The 48-Core SCC Processor: TheProgrammer’s View. SC’10, Washington, DC, USA, November,pp. 1–11. IEEE Computer Society.

[30] Alam, S. (2013). Examples of cfgs before and after shrinking.http://web.uvic.ca/∼salam/PhD/cfgs.html

Section D: Security in Computer Systems and NetworksThe Computer Journal, 2014

at Qatar U

niversity on Decem

ber 16, 2014http://com

jnl.oxfordjournals.org/D

ownloaded from

Page 14: Annotated Control Flow Graph for Metamorphic Malware Detection › b156 › 64267e99d5a... · detector to detect metamorphic malware. In [13], the authors use an intermediate language

14 S. Alam et al.

[31] Madenur Sridhara, S. and Stamp, M. (2013) Metamorphic wormthat carries its own morphing engine. J. Comput. Virol. HackingTech., 9, 49–58.

[32] Rad, B., Masrom, M. and Ibrahim, S. (2012) OpcodesHistogram for Classifying Metamorphic Portable ExecutablesMalware. ICEEE, Lodz, Poland, September, pp. 209–213.IEEE.

[33] Runwal, N., Low, R.M. and Stamp, M. (2012) Opcode GraphSimilarity and Metamorphic Detection. J. Comput. Virol., 8,37–52.

[34] NGVCK. Next generation virus generation kit, http://vxheaven.org/vx.php?id=tn02 (accessed November 7, 2014).

[35] G2. Second generation virus generator, http://vxheaven.org/vx.php?id=tg00 (accessed November 7, 2014).

[36] Faruki, P., Laxmi, V., Gaur, M.S. and Vinod, P. (2012) MiningControl Flow Graph as API Call-Grams to Detect PortableExecutable Malware. Security of Information and Networks,SIN’12, New York, NY, USA, October, pp. 130–137. ACMSIGSAC.

[37] Toderici, A. and Stamp, M. (2012) Chi-squared distanceand metamorphic virus detection. J. Comput. Virol., 9,1–14.

[38] Lee, J., Jeong, K. and Lee, H. (2010) Detecting MetamorphicMalwares Using Code Graphs. SAC, 2010, New York, NY, USA,March, pp. 1970–1977. ACM.

[39] Yin, H. and Song, D. (2013) Privacy-Breaching BehaviorAnalysis. Automatic Malware Analysis. Springer Briefs inComputer Science, pp. 27–42. Springer, New York, NY, USA.

[40] Wong, W. and Stamp, M. (2006) Hunting for metamorphicengines. J. Comput. Virol., 2, 211–229.

[41] Austin, T.H., Filiol, E., Josse, S. and Stamp, M. (2013) ExploringHidden Markov Models for Virus Analysis: A SemanticApproach. 2013 46th Hawaii International Conference onSystem Sciences (HICSS), Hawaii, USA, January, pp. 5039–5048. IEEE.

[42] Shanmugam, G., Low, R.M. and Stamp, M. (2013) Simplesubstitution distance and metamorphic detection. J. Comput.Virol. Hacking Tech., 9, 159–170.

[43] Vinod, P., Laxmi, V., Gaur, M. and Chauhan, G. (2012)MOMENTUM: Metamorphic Malware Exploration TechniquesUsing MSA Signatures. IIT, Abu Dhabi, UAE, March, pp. 232–237. IEEE.

[44] Leder, F., Steinbock, B. and Martini, P. (2009) Classification andDetection of Metamorphic Malware Using Value Set Analysis.MALWARE, 2009, Montreal, QC, Canada, October, pp. 39–46.IEEE.

[45] Ghiasi, M., Sami, A. and Salehi, Z. (2012) Dynamic MalwareDetection Using Registers Values Set Analysis. InformationSecurity and Cryptology, Tabriz, Iran, September, pp. 54–59.IEEE.

Section D: Security in Computer Systems and NetworksThe Computer Journal, 2014

at Qatar U

niversity on Decem

ber 16, 2014http://com

jnl.oxfordjournals.org/D

ownloaded from