The Aftermath of a Fuzz Run:What to do with all those crashes?

David Moore

David Moore Bio

NeXT, Apple, Weblogic, BEA Systems, Azul Systems

Google, Twitter, Netflix, Optimizely, Card, kernel, 

ruby, php, cpio

Founder/CEO

Talk Outline1> Introduce/Review Memory Corruption Bugs

2> A Post Fuzz Run Workflow

3> Real World Examples

Section 1a:Introduce / Review 

Memory Corruption Bugs

InvalidReads/Writes

Stack vs HeapCorruption

int main (int argc, char **argv) { char buf[8]; strcpy(buf, argv[1]);}

./a.out AAAAAAAAAAAA

Use After Free

char* x = (char*)malloc(4);...

free(x);…

printf(x)   // uaf

Other Memory Bugs

Section 1b:What is Exploitability?

Reprogramming with input data 

not code

Reprogramming with existing code in the 

process

Does “exploitability” 

matter?

ExploitableBy Whom?

GoogleProject Zero

NSA

hhhhh??

Many modern exploits are bug 

chains

Surprisingly Exploitable

CAres / Chrome OS Remote Code 

Execution

Triggered by a trailing escaped dot:

www.foo.com\.

Section 1c:Memory Corruption 

Mitigations

Stack Canaries

DEPData Execution

Prevention

ASLRAddress Space Layout

Randomization

Section 2:A Post Fuzz Run Workflow

2a> Minimize crash corpus 2b> Use Memory Corruption Tools 2c> Determine Exploitability  or   Find the Root Cause

Whyminimize?

Minimize the Corpus of Crashes

Minimize each crashing case individually

fdupes

Section 2b:Memory Corruption 

Analysis Tools

All Bets are Off

Address Sanitizer

ASANfsanitizeaddress

Valgrind(memcheck)

Exploitable

Section 2c:Determine

Exploitability /Find the Root Cause

Disable ASLRecho 0 | sudo tee

/proc/sys/kernel/randomize_va_space

Identify critical memory

locations

gdbgcc g O0 target.c

./target AAAA

0x41414141

rr

rrproject.org

It is OK and normal to:

Feel lost / frustratedTake a lot of time

Feel like your wheels are spinningGet sick of staring at hex

OneMore Thing:

Once the bugs are fixed -Fuzz the target again

Section 3: Real World Examples

PHP:Low invalid read

Ruby:Heap Buffer

Overflow

Netflix Dynomite: Invalid Write

Netflix Dynomite:● Running in production ~2 

years● 1000 Customer facing nodes● 1 Million ops/sec peak load

References:● RPI  Modern Binary Exploitation  

GitHub: rpisec/mbe● Hacking: The Art of Exploitation  Jon 

Erickson● Project Zero Blog  What is Good 

Memory Corruption?● Sean Heelan’s Blog  Tracking Down 

Heap Overflows with rr

Thank You!David Moore@grajagandev

dave@fuzzstati0n.com

Slide 1Slide 2Slide 3Slide 4Slide 5Slide 6Slide 7Slide 8Slide 9Slide 10Slide 11Slide 12Slide 13Slide 14Slide 15Slide 16Slide 17Slide 18Slide 19Slide 20Slide 21Slide 23Slide 24Slide 25Slide 26Slide 27Slide 28Slide 29Slide 30Slide 31Slide 32Slide 33Slide 34Slide 35Slide 36Slide 37Slide 38Slide 39Slide 40Slide 41Slide 42Slide 43Slide 44Slide 45Slide 46Slide 47Slide 48Slide 49Slide 50Slide 51Slide 52Slide 53Slide 54Slide 55Slide 56Slide 57Slide 58Slide 59Slide 60Slide 61Slide 62Slide 63Slide 64Slide 65Slide 66Slide 67Slide 68Slide 69Slide 70Slide 71Slide 72Slide 73Slide 74Slide 75Slide 76Slide 77Slide 78Slide 79