1

Securing Untrusted Code via Compiler-Agnostic Binary Rewriting

Richard Wartell, Vishwath Mohan,

Dr. Kevin Hamlen, Dr. Zhiqiang LinThe University of Texas at Dallas

Supported in part by NSF, AFOSR, and DARPA

2

Software Fault Isolation (SFI)• Automatically rewrite binaries to make them safer

• [Wahbe, Lucco, Anderson, Graham, SOSP 1993]

Untrustedcode Rewriter Safe

code

3

Software Fault Isolation (SFI)• trusted & untrusted modules in

common address space• Example #1: web browser plug-ins• Example #2: trusted system libraries inside

untrusted application

• Goal: protect trusted modules from untrusted ones• confine untrusted module behaviors

• Example: Untrusted modules must obey trusted module interfaces• Blocks ROP attacks [Shacham, CCS 2007]

eMule.exe

kernel32.dll

user.dll

Trusted

Untrusted

4

Inlined Reference Monitors (IRMs)• SFI foundation supports higher-level

policies [Abadi, Budiu, Erlingsson, and Ligatti. CCS 2005]

• Example: IRMs [Schneider, ISS 2000]• Enforces powerful policies:

• program-specific (no other programs affected)• light-weight enforcement (minimize context

switches)• Statefulness

• Example: Adobe Reader may access the network (to check for updates) and may read my confidential files, but may not access the network after reading my confidential files.

reader.exe

kernel32.dll

user.dll

Trusted

UntrustedIRM

5

A Brief History of SFI

1995 2000 2005 2010

Wah

be1

PittSFI

eld

3

CFI /

SMAC

2

XFI4

NaCl5

1: [Wahbe, Lucco, Anderson, and Graham. SOSP 1993] 2: [Abadi, Budiu, Erlingsson, and Ligatti. CCS 2005] 3: [McCamant and Morrisett. USENIX 2006] 4: [Erlingsson, Abadi, Vrable, Budiu, and Necula. SOSDI 2006] 5: [Yee, Sehr, Dardyk, Chen, Muth, Ormandy, Okasaka, Narula, and Fullagar. S&P 2009]

6

A Brief History of SFI

1995 2000 2005 2010

Wah

be1

RISC o

nly

PittSFI

eld

3

Spe

cial

GCC

CFI /

SMAC

2

Needs

PDB

XFI

4Nee

ds P

DB

NaCl5

Speci

al G

CC

All prior works require explicit code-producer cooperation

1: [Wahbe, Lucco, Anderson, and Graham. SOSP 1993] 2: [Abadi, Budiu, Erlingsson, and Ligatti. CCS 2005] 3: [McCamant and Morrisett. USENIX 2006] 4: [Erlingsson, Abadi, Vrable, Budiu, and Necula. SOSDI 2006] 5: [Yee, Sehr, Dardyk, Chen, Muth, Ormandy, Okasaka, Narula, and Fullagar. S&P 2009]

7

Reins: REwriting and IN-lining System• Main Discovery:  means of enforcing SFI for near arbitrary

COTS binaries• no source code or debug info (assumed unavailable)• no disassembly listing• compiler-agnostic• real COTS binary features

• interleaved code and data• computed control-flows• dynamic linking• event-driven callbacks• multithreading

• Low overhead (~2%)• Formal machine-verification of policy enforcement

8

Binary Rewriting w/o metadata• Relocation information, debug tables and symbol stores not always available• Reverse engineering concerns

• Perfect static disassembly without metadata is provably undecidable• Best disassemblers (IDA Pro) make many mistakes

Program Instruction Count

IDA Pro Errors

mfc42.dll 355906 1216

mplayerc.exe 830407 474

vmware.exe 364421 183

9

Infeasibility of Perfect Disassembly

• Disassemble this hex sequence• Undecidable problem

FF E0 5B 5D C3 0F 88 52 0F 84 EC 8B

Valid Disassembly

FF E0 jmp eax

5B pop ebx

5D pop ebp

C3 retn

0F 88 52 0F 84 EC

jcc

8B … mov

Valid Disassembly

FF E0 jmp eax

5B pop ebx

5D pop ebp

C3 retn

0F db (1)

88 52 0F 84 EC

mov

8B … mov

Valid Disassembly

FF E0 jmp eax

5B pop ebx

5D pop ebp

C3 retn

0F 88 db (2)

52 push edx

0F 84 EC8B …

jcc

10

Original BinaryReins Binary

Separating Code from Data

HeaderIAT

.data.text

Original Memory Layout

Rewritten HeaderIAT

.data.told (NX bit set)

Rewritten Memory Layout

.tnew (NW bit set)

Denotes a section that is modified during static rewriting

High Memory

Low Memory

kernel32.dll user32.dll

user32.dll kernel32.dll

11

De-Shingling Disassembly

Hex Path 1 Path 2 Path 3 Path 4

FF jmp eax

E0 loopne

5B pop

5D L1: pop

C3 retn

0F jcc

88 mov

B0 mov

50

FF N/A

FF

8B L2: mov

Byte Sequence: FF E0 5B 5D C3 0F 88 B0 50 FF FF 8B

Disassembled Invalid

IncludedDisassembly

jmp eax

pop

L1: pop

retn

jcc

L2: mov

loopne

jmp L1

mov

jmp L2

12

Aligning Instructions

Original Binary

0x68900F mov eax, 0x6891D8

0x689015 add eax, 1

0x68901B call eax

… …

0x6891D9 push ebx

0x6891DA mov ebx, [esp+4]

Rewritten Binary

0x78900F nop

0x789010 mov eax, 0x6891d8

0x789016 add eax, 1

0x78901C nop (x4)

0x789020 nop (x8)

0x789028 and eax, 0x0FFFFFF0

0x78902E call eax

0x789030 …

0x7892E0 push ebx

0x7892E1 mov ebx, [esp+4]

0x7892E5 …

• Chunk instructions to 16 byte boundaries with targets at the beginning, and calls at the end [McCamant and Morrisett. USENIX 2006]

Alignment nops

Injected Instructions

13

Rewritten Binary

Preserving Good Flows

Original Binary

0x68900F mov eax, 0x6891D8

0x689015 add eax, 1

0x68901B call eax

… …

0x6891D9 push ebx

0x6891DA mov ebx, [esp+4]

• Turn original code section into a dynamic lookup table .told 0x6891D9 0xF4 loc_7892F0

.tnew 0x78900F nop

0x789010 mov eax, 0x6891d8

0x789016 add eax, 1

0x78901C nop (x4)

0x789020 cmp 0xF4, [eax]

0x789023 cmovz eax, [eax+1]

0x789027 nop

0x789028 and eax, 0x0FFFFFF0

0x78902E call eax

0x789030 …

0x7892F0 push ebx

0x7892F1 mov ebx, [esp+4]

0x7892F5 …Alignment nops

Injected Instructions

14

Preserving Good Inter-module Flows

jmp [IAT:CreateWindow]

Original Code Rewritten Code

CreateWindow

jmp [IAT:CreateWindow]

CreateWindow

• IAT data section locked non-writable

15

Computed Inter-module Flows

• computed jumps to trusted modules• dynamic linking (DLLs)• callbacks (event-driven programming)

trusted library

intermediarylibrary

(trusted)

rewrittencode

caller

callback stub

callback_ret

callback

return trampoline

16

Results

gzip

vpr

mcf

gap

bzip2

twolf

mes

a

art

equa

ke

gcc

g++

jar

objco

py

size

strin

gs

as

ar

whets

tone

linpa

ck

pi_cc

s5

md5

-8%

-4%

0%

4%

8%

12%

16%

17

IRM Synthesis

• Enforced policies on Eureka email client (>1.6MB code):• Disallow creation of .exe, .msi, or .bat files• Disallow execution of Windows explorer as an external process• Disallow opening more than 100 SMTP connections

• Malware policies:• Disallow creation of .exe, .msi, or .bat files

• Successfully stopped virus propagation for real world malware samples

Policy-adherantbinary

PolicyRewriterBinary

18

TCB

Formal Verification

• Formal verification of rewritten binaries• 1500 SLOC of 80-column OCaml code• no shared code between verifier and rewiter• median verification time:  0.4 ms/KB code

• Allows rewriter to remain completely untrusted!• rewriting deployable as an untrusted service

Policy-adherantbinary

PolicyRewriterBinary

Verifier

19

Compatibility Limitations

• COM objects• Runtime code generation (JIT)• Undocumented OS callbacks

20

Conclusion• Reins finally opens the door to full-scale COTS native SFI for

massively complex, real-world applications without source.• no source code, debug info, or disassembly (assumed unavailable)• compiler-agnostic• real COTS binary features

• interleaved code and data, computed control-flows, dynamic linking, event-driven callbacks, multithreading

• automated synthesis of monitor from policy specification• automated machine-verification• low runtime overhead (~2.4%)• successfully tested on real commercial applications (>3MB code)

• Practical Applications:• safe reuse of untrusted commercial software in security-critical

environments• rewriting on demand: rewriter deployable as an untrusted third-party

service due to separate verifier

21

References• R. Wahbe, S. Lucco, T. E. Anderson, and S. L. Graham. Efficient software-based

fault isolation. In Proc. ACM Sym. Operating Systems Principles, pages 203–216, 1993.

• F. B. Schneider. Enforceable security policies. ACM Trans. Information and Systems Security, 3(1):30–50, 2000.

• M. Abadi, M. Budiu, U. Erlingsson, and J. Ligatti. Control-flow integrity. In ACM Conference on Computer and Communications Security, pages 340-353, 2005.

• S. McCamant and G. Morrisett. Evaluating SFI for a CISC architecture. In Proc. USENIX Security Sym., 2006.

• Ú. Erlingsson, M. Abadi, M. Vrable, M. Budiu, and G. C. Necula. XFI: Software guards for system address spaces. In Proc. Sym. Operating Systems Design and Implementation, pages 75–88, 2006.

• H. Shacham. The geometry of innocent flesh on the bone: Return-into-libc without function calls (on the x86). In Proc. ACM Conf. Computer and Communications Security, pages 552–561, 2007.

• B. Yee, D. Sehr, G. Dardyk, J. B. Chen, R. Muth, T. Ormandy, S. Okasaka, N. Narula, and N. Fullagar. Native Client: A sandbox for portable, untrusted x86 native code. In Proc. IEEE Sym. Security and Privacy, pages 79–93, 2009.

22

Advantage over VMs• no air gap

• IRM has controlled but direct access to system resources and other processes

• no semantic gap• no dynamic instruction interpretation or translation

• better performance• fewer context switches• light-weight VM logic essentially in-lined into code

• formal verification• few VMs have been formally verified• each change to VM (e.g., to enforce new policy) requires re-

verification of VM