fuzzing & software security testing
TRANSCRIPT
VEYSEL HATAŞ
09.05.2014
BILKENT UNIVERSITY COMPUTER ENGINEERING DEPARTMENT
FUZZING & SOFTWARE SECURITY TESTING
PROLOGUE
• 0x00401001 what_is_fuzzing ( )
• 0x00401002 history ( )
• 0x00401003 fuzzing_tools ( )
• 0x00401004 fuzzing_phases ( )
• 0x00401005 fuzing_framework ( )
• 0x00401006 fuzzing_methodes ( )
• 0x00401007 questions ( )
push ebp
mov ebp, esp
call what_is_fuzzing
WHAT IS FUZZING?
«Fuzz testing or Fuzzing is a Black Box software testing technique, which basically consists in finding implementation bugs using malformed/semi-malformed data injection in an automated fashion.»
«The purpose of fuzzing relies on the assumption that there are bugs within every program, which are waiting to be discovered. Therefore, a systematical approach should find them sooner or later.» -- http://www.owasp.org/index.php/Fuzzing
«Fuzzing is the process of sending intentionally invalid data to a product in the hopes of triggering an error condition or fault. These error conditions can lead to exploitable vulnerabilities.» HD Moore
“The degree to which a system or component can function correctly in the presence of invalid inputs or stressful environmental conditions.” IEEE
HISTORY
• Professor Barton Miller
• Developed fuzz testing with his students at the University of Wisconsin-Madison in
1988/89
• GOAL: improve UNIX applications
• Millers fuzzer was pretty basic: It sent random strings of data to the application
• 1999 brought PROTOS from University of Oulu
• 2004 Browser Fuzzing (Fuzzed HTML to find browser bugs)
• 2004 File Format Fuzzing (MS04-028 Buffer Overun ~ Remote Code Execution)
• 2006 ActiveX Fuzzing
• 2007 Man-in-the-middle Fuzzing (ProxyFuzz)
SOME FUZZING TOOLS
• Peach Fuzz
• BFF
• Sulley
• Spike
• TAOF
• COMRaider
SOME HELPFUL DEFINITIONS
• BUG
• The Harvard Mark II (1947) Real BUG !
• Security BUGs (Memory Corruptions, Overflow, Injections, Memory Leaks)
• BUG Hunting
• Source Code Analysis (White Box)
• Binary Analysis
• Static Analysis (Reverse Engineering)
• Dynamic Analysis (Fuzzing, Debugging/Tracing, Taint Analysis)
• In Memory Fuzzing
«In memory fuzzing is a technique that allows the analyst to bypass parsers; network -related limitations such as max connections, buit in IDS or flooding protection; encrypted or unknown (poorly documented) protocol in order to fuzz the actual underlying assembly routines that are potentially vulnerable.»
CORELAN TEAM
FUZZING PHASES
1. IDENTIFY THE TARGET
• Applications (players, readers, web apps…)
• Platforms (Linux, Windows, OSX)
• Devices (arm, phones, modems ..)
• …
2. IDENTIFY THE INPUTS
• File Formats (swf, pdf, png, jpeg, m3u …)
• Network Protocols (ftp, http, arp, ssl …)
• Reg keys, env variables, headers …
FUZZING PHASES
3. GENERATE FUZZ DATA
• Mutation
• Generation
4. EXECUTE FUZZ DATA
• Semi-valid input to the target application
5. MONITOR FUZZ DATA
THE PROCESS OF FUZZING FRAMEWORK
Identify Targets & Platform
Identify Input vectors
Generate Fuzz Data
• minimizing
Execute Fuzzed Data
Monitor for Exceptions
Determine Exploitability
• Unique crash logs
CERT - FUZZING FRAMEWORK
• Basic Fuzzing Framework (BFF)
• Linux and Mac OS X platforms
• Failure Observation Engine (FOE)
• Windows platform
BFF - BASIC FUZZING FRAMEWORK
• The CERT Basic Fuzzing Framework (BFF) is a software testing tool that finds defects in
applications. BFF performs mutational fuzzing on software that consumes file input.
• The BFF automatically collects test cases that cause software to crash in unique ways, as
well as debugging information associated with the crashes.
• The goal of BFF is to minimize the effort required for software vendors and security
researchers to efficiently discover and analyze security vulnerabilities found via fuzzing .
BFF - BASIC FUZZING FRAMEWORK
Features that are available in BFF
• Uniqueness determination is handled through intelligent backtrace analysis.
• Automated test-case minimization reduces the effort required to analyze results. This is
achieved by distilling the test case to the minimal changes to the input data required to
induce a specific crash.
• Online machine learning is applied to fuzzing parameter and input file selection to
improve the efficacy of the campaign.
• Crash severity/exploitability triage is provided.
BFF - BASIC FUZZING FRAMEWORK
ZZUF
It is a transparent application input fuzzer, whose purpose is to find bugs in applications by
corrupting their user-contributed data, changing the random-bits in the input. Zzuf is one of
the most common fuzzers available for bit flipping fuzzing.
CERT Triage Tools
The CERT Triage Tools include a GNU Debugger (GDB) extension called "exploitable" that
classifies Linux application bugs by severity and a wrapper script for batch execution.
BFF - TOOLS
BFF minimizes crashers (tools/minimizer_plot.py)
• Each crashing test case will have a number of bytes that have been modified from the
seed file.
• When a crasher is minimized, a test case is generated with a minimal number of bytes
that have changed from the seed file.
BFF - TOOLS
BFF crash analyzer (tools/bff_stats.py)
• Generates a concise summary of the fuzz run results so far, including how many times
each unique crash was seen, the first seed number it was seen at, the most recent seed
number it was seen, and the bitwise and byte wise Hamming Distance from the original
seed file for the minimized test case.
BFF - TOOLS
BFF crash analyzer (tools/create_crasher_script.py)
• It will generate a shell script that in turn can be used to regenerate all the test cases for a
given crash id.
BFF - BASIC FUZZING FRAMEWORK
Analyzing Results
• When the fuzzing run encounters a crash, BFF will analyze the details of the crash
• Capturing
• Stderr output
• Gdb output
• Valgrind output
• Callgrind output
BFF - BASIC FUZZING FRAMEWORK
Analyzing Results
• crashers: Contains a subdir for each uniquely-crashing test case and its analyzed results
• seeds: Contains the original seed files as well as logs specific to that seedfile
• uniquelog.txt: A log file that tracks the unique crashers found during the run
• gdb output
• the memory map
• signal information
• backtrace, registers
• disassembly
• CERT Triage Tools
BFF - BASIC FUZZING FRAMEWORK
DEMO 1
FUZZING METHODS
• Generation-based or Random-based (Smart Fuzzing)
The fuzzers generate input data for applications in a random way . This type of data
generation is very quick to implement but also useless in most cases.
• Easy, you do not need to know any protocol or format.
• Mutation-based (Dumb Fuzzing)
Mutation-based fuzzers generate data by analyzing an existing set of data provided by
the user and mutating some fields inside these data.
• Very hard, but explores every detail of protocol
• Proxy-based
• Specification-based
• Evolutionary-based
GENERATION BASED FUZZING
• Generational fuzzers are capable of building the data being sent based on a data model
provided by the fuzzer creator. Sometimes this is as simple as sending random bytes.
This can be done much smarter by knowing good data values and combining them in
interesting ways.
• Test cases are generated from some description of the format.
MUTATION BASED (DUMB FUZZING)
• Mutation-based fuzzers mutate existing data samples to create test data while generation-
based fuzzers define new test data based on models of the input.
• Mutational fuzzing is the act of taking well-formed input data and corrupting it in various
ways, looking for cases that cause crashes.
• Change Random Byte
• Byte Flipping
Charlie Miller code
numwrites = random.randrange (math.ceil ( ( float ( len (buf)) / FuzzFactor)))+1
for j in range (numwrites ) :
rbyte = random.randrange (256)
rn = random.randrange (len (buf) )
buf [rn] = "%c" %(rbyte);
CHANGE RANDOM BYTE
BYTE/DWORD FLIPPING
• FiveBelow is the dummy file fuzzer. It uses byte slides methode.
BYTE/DWORD FLIPPING
• FiveBelow is the dummy file fuzzer. It uses byte slides methode.
• ZDI-11-252
• Apple QuickTime PICT Image PnSize Opcode Remote Code Execution Vulnerability
BYTE/DWORD FLIPPING (ZDI-11-252)
DEMO 2
APPLE QUICKTIME PICT IMAGE PNSIZE OPCODE
REMOTE CODE EXECUTION
BYTE/DWORD FLIPPING (ZDI-11-252)
FILE FORMAT FUZZING
PDFuzzer (pdf header format fuzzer by Jeremy Brown )
PEACH FUZZER
• Peach is a SmartFuzzer that is capable of performing both generation and mutation based
fuzzing.
• Peach requires the creation of Peach Pit files that define the structure, type information,
and relationships in the data to be fuzzed. It additionally allows for the configuration of a
fuzzing run including selecting a data transport (Publisher), logging interface, etc.
PEACHPIT FILE
• DataModel
A DataModel defines the structure of a data block by specifying additional child elements
such as Number, Blob, or String.
• StateModel
The StateModel recreates the basic state machine logic needed to test a protocol. The
state model defines how to send and receive data to the fuzzing target.
PEACHPIT FILE
• Publishers (Network Publishers & Custom Publishers )
Publishers are the I/O interfaces used by Peach to send and receive data.
• Agent and Monitor
DEMO 3
PEACH FARM (PARALLEL FUZZING)
Peach Fuzzer Farm allows to build scalable
and distributed fuzzing harnesses.
• 88 Cores of fuzz
• 100,000 files per format (doc, xls, ppt, pdf)
• 5 Targets (Office 2010/2007, OpenOffice, PDF viewers)
• 4 File formats
• >250 GB of logs
• One crash about every six seconds
PEACH FARM (SAMPLE RESULTS)
• 942 Unique Major Hashes
• 440 in Office
• 396 in Open Office
• 68 in Ghostview
• 32 in Foxit
• 5 in Acrobat
• ! exploitable
• 150 EXPLOITABLE (15%)
• 188 PROBABLY_EXPLOITABLE (19%)
• 16 PROBABLY_NOT_EXPLOITABLE (1.6%)
• 588 UNKNOWN (62%)
ACTIVEX FUZZING
COMRaider is a tool designed to fuzz COM Object Interfaces.
ACTIVEX FUZZING
Target: CommuniCrypt Mail 1.16 (ANSMTP.dll/AOSMTP.dll)
Methode: AddAttachments
ACTIVEX FUZZING
Target: CommuniCrypt Mail 1.16 (ANSMTP.dll/AOSMTP.dll)
Methode: AddAttachments
EPILOGUE
• « Fuzzing will not always yield bugs that will allow you to execute code. Fuzzing does not
promise to make your dreams come true. Fuzzing is an art and a software programmer's
nightmare. Fuzzing will open your eyes to see that it is no longer enough to know the
code backwards and forward, inside and outside, layer by layer, line by line, bit by bit. »
Jeremy Brown
mov esp, ebp
pop ebp
ret
THANK YOU FOR LISTENING
• udacity: CS258 (Software Testing)
• http://fuzzing.org/
• Open Source Fuzzing Tools
• Fuzzing: Brute Force Vulnerability Discovery
• www.corelan.be