automatic reverse engineering of malware emulators

Automatic Reverse Engineering of Malware Emulators

Sharif, M., Lanzi, A., Giffin, J., and Lee, W. Georgia Institute of Technology, S&P 2009.

Presented by WANG Zhi

Outline

Emulation Obfuscation Emulator Model Abstract Variable Binding Identifying Candidate VPC Identifying Emulation Behavior Extracting Syntax and Semantics

Emulation Obfuscation

The term emulation generally expresses the support of a binary instruction set architecture (ISA) that is different from that supported by CPU.

Jave(JVM), Javascript, Perl, Python…

Emulation Obfuscation

Using emulation for obfuscation

Emulation Obfuscation

Without knowledge of the bytecode language, even the location of bytecodes in the obfuscated program.

The key challenges in analyzing emulation obfuscation are how to extract the bytecode trace and the syntax and semantics of the bytecode instructions from the emulated program trace.

Emulator Model

A decode-dispatch based emulator

Emulator Model

The decode phase fetches opcode.

The dispatch phase selects appropriate handling routine.

The execution phase perform the dispatched handling routine

Reverse Engineering of Emulation

In this paper, they developed algorithms to extract the syntax and semantics of unknown bytecode based upon the execution behavior of the decode-dispatch emulator.

Abstract Variable Binding

The compiler translate source code into low-level CPU instructions while the high-level variables are assigned to memory location or registers.

In the x86 instruction trace, there are no high-level language variables.

To identify variables in the execution trace of the emulator is the first work.

Abstract Variable Binding

In this paper, memory locations are used to represent high-level variables, which are called abstract variables..

In the x86 architecture, memory read and write operations should use register indirect addressing.

Source Code: instruction = bytecode[VPC] Pseudo-x86 Instructions:

eax <= [VPC]instruction <= [eax]

Abstract Variable Binding

The first instruction loads the value of the abstract variable VPC to eax. In other word , binds eax to VPC, and this binding is propagated in the next instruction.

It use a forward and backward data-flow analysis to identify the abstract variables and their propogation.

Forward Binding

Forward binding identifies abstract variable from memory read operations.

Forward Binding

Forward binding propagates abstract variable bindings when operations compute an update to its previous value.

The outputs is a mapping describing bindings from memory read operations to abstract variables.

Backward Binding

Backward binding operates in the reverse order of forward bindings. It determines abstract variables from memory write operations

Identifying Dependent Abstract Variables

It identify dependencies among abstract variables by tracking the data flow from one abstract variable to another.

Lifetime Abstract Variables

It use memory locations as abstract variables, but the same memory location may be used for different variables at different points of execution.

The abstract variables on the stack or heap have shorter lifetime than those in the static data region

The Lifetime depends on the allocation and deallocation operations.

Identifying Candidate VPCs

VPC is a virtual program counter, like program counter or instruction pointer register.

The emulator fetches bytecode instructions from memory address specified by VPC

Identifying Candidate VPCs

It clusters all read operations to n clusters using a simple similarity metric that check whether they have common abstract variables.

The fetching behavior’s read operations should be contained within one of the n cluster because VPC is the common abstract variable.

Identifying Emulation Behavior

Decode-dispatch emulators have fundamental execution properties: a main loop with a bytecode fetch through the VPC, decoding of the opcode within the bytecode, dispatch to an opcode handler, and a change to the VPC value.

Identifying Emulation Behavior

They use a standard loop detection methods to detect the emulator’s main loop.

They analyze the abstract variable propagation to find decoding, dispatching and execution of bytecode behaviors.

Identifying Emulation Behavior

They use multi-level dynamic tainting to analyze the data flow.

There are four taint label: fetch, decode, dispatch and execute.

Identifying Emulation Behavior

If a abstract variable is read from memory, they mark the instruction taint label fetch.

Once the analyzer detects a dispatch-like behavior which is a control-flow transfer instruction and its target address is a taint variable, it will mark the instruction taint label dispatch, and mark the target of the contol-transfer as a probable execute routine

Extracting Syntax and Semantics

Once the analyzer identifies the emulation behavior, it reverse engineers each iteration of the emulator loop to extract the syntax and semantics of the bytecode instruction executed on that iteration.

Extracting Syntax and Semantics

The syntactic information shows how bytecode instructions are parsed into opcodes and operands.

The semantic information consisting of native instructions that carry out the actions of the bytecode instructions.

Extracting Syntax and Semantics

To identify the opcode part, they use taint analysis to determine which portion of the bytecode instruction was used for dispatch behavior.

The execution routine invoked by emulator for the opcode encodes the semantics.

Extracting Syntax and Semantics

By determining how the parse the bytecode and by locating control-flow transfer opcodes, a CFG for the bytecode can be constructed.

The CFG structure provides a foundation for subsequent malware analysis.

Thank you