1
Minestrone: Tes-ng the SOUP Azzedine Benameur, Nathan Evans, Ma?hew Elder
MINESTRONE: Tes-ng the SOUP, CSET 13
Agenda
• Overview • MINESTRONE: – Architecture – Sta-c and dynamic detec-on technologies
– I/O Redirec-on – External Replica Monitoring
• Test and evalua-on: – Architecture – Test suite – Results
• Closing Thoughts
2
MINESTRONE: Tes-ng the SOUP, CSET 13
Overview
• IARPA STONESOUP Program: “Securely Taking On New Executable SoLware of Uncertain Provenance”
– Develop and demonstrate technology that provides comprehensive, automated techniques that allow end users to safely execute new soLware of uncertain provenance
– Addressing 8 “weakness” classes across 3 target language classes
• Team: Columbia University (PI: Angelos Keromy-s) with Stanford University, George Mason University (GMU), and Symantec
• 4-‐year project, 3 phases 3
MINESTRONE: Tes-ng the SOUP, CSET 13
Overview: NSA Source Code Analysis Tool Evalua-on
• hYp://www.iarpa.gov/stonesoup_Merced_DHSAWGbrief.pdf • Evaluated suite of tools (Coverity Prevent, FindBugs, For-fy SCA, GrammaTech CodeSonar, Klocwork Insight, Ounce Labs Ounce, PMD) against both C/C++ and Java vulnerability test cases in different CWE (Common Weakness Enumera-on) classes
4
MINESTRONE: Tes-ng the SOUP, CSET 13
Overview: STONESOUP Program Targets
• Target classes of vulnerabili-es (including example CWE numbers): – Number handling (e.g., integer overflow/underflow, sign conversion: #190, #191 – Resource drains (e.g., failure to release memory, structures, devices: #400, #404) – Tainted data/input valida-on errors (#78, #134) – Error handling (e.g., unhandled excep-ons/error status codes: #248, #252) – SQL injec-on / command injec-on (#78, #89) – Concurrency handling (e.g., race condi-ons, thread safety: #362, #366) – Buffer overflows/underflows/out of bounds accesses/memory safety (#121, #122) – Null pointer errors (#476)
• Target language classes: – Type-‐safe languages (Java, C#) – Type-‐unsafe languages (C, C++) – Binaries (x86, Windows or Linux)
5
MINESTRONE: Tes-ng the SOUP, CSET 13
MINESTRONE
• Architecture
• Sta-c and dynamic detec-on technologies
• Replica Diversifica-on
• I/O Redirec-on
• External Replica Monitoring
6
MINESTRONE: Tes-ng the SOUP, CSET 13
UnknownSoftware
Lightweight Containers
Lightweight Containers
ISR + defensive instrumentation
ISR + defensive instrumentation
UnknownSoftware
KLEEcontinuoussymbolicexecution
Runtime
Offline/parallel
Repli
cate
d run
time
Remove/optimizeunneeded defenses
Path explorationpreference & control flow
information
MINESTRONESystem Composer
Symbiotes
AnomalyDetection
RaceAvoidance
RaceDetection
Informationflow trackingoptimization
Attackdetection
I/O & statereplication
Symbiotes AnomalyDetection
Deployedapplication
(N instances)
Backendanalysis
(M << N instances)
Instrumentedreplicas
(P < N instances)
ResourceExhaustionDetection
ResourceExhaustionDetection
KLEEprophylactic
analysis
Minestrone Overview: System Architecture
7
MINESTRONE: Tes-ng the SOUP, CSET 13
MINESTRONE
• Architecture
• Sta-c and dynamic detec-on technologies
• I/O Redirec-on
• External Replica Monitoring
8
MINESTRONE: Tes-ng the SOUP, CSET 13
Detec-on technologies: Pin-‐based tools (theRing)
• REASSURE – Self-‐contained Mechanism for Healing SoLware Using Rescue Points – Detects program crashes and gracefully recovers
• ISR: Instruc-on Set Randomiza-on – Applica-on binary is randomized – Shared libraries can be also randomized
• DFT – Data flow tracking – High performance
9
MINESTRONE: Tes-ng the SOUP, CSET 13
Detec-on technologies: con-nued
• KLEE – Symbolic Execu-on – Fine-‐Grained detec-on
• Dyboc – Source to source transforma-on – Moving stack buffers to heap – Custom version of malloc(): pmalloc()
• Valgrind (baseline) – State of the art – Memcheck
10
MINESTRONE: Tes-ng the SOUP, CSET 13
MINESTRONE
• Architecture for confinement
• Sta-c and dynamic detec-on technologies
• I/O Redirec-on
• External Replica Monitoring
11
MINESTRONE: Tes-ng the SOUP, CSET 13
I/O Redirec-on: Network/Shared Memory/X11
• Paired-‐library: – Interpose_writer: writes to file from no-‐sec environment
– Interpose_reader: read from file in all replicas
12
MINESTRONE: Tes-ng the SOUP, CSET 13
MINESTRONE
• Architecture for confinement
• Sta-c and dynamic detec-on technologies
• I/O Redirec-on
• External Replica Monitoring
13
MINESTRONE: Tes-ng the SOUP, CSET 13
External Replica Monitoring
14
• OpenVZ allows easy replica monitoring – CPU from /proc/vz/vestat
– Memory using bean counters /proc/user_beancounters
– Network from /vz/root/$replica_id/sys/class/net/venet0/sta-s-cs/tx_bytes
• Overhead comparison: – Confinement between containers – Fair scheduling
MINESTRONE: Tes-ng the SOUP, CSET 13
Test and Evalua-on
• Architecture
• Test suite
• Results
15
MINESTRONE: Tes-ng the SOUP, CSET 13
Test and Evalua-on Process, cont’d.
• MITRE developed tes-ng framework and API • We/Symantec developed the interface to interact with the test harness
16
MINESTRONE: Tes-ng the SOUP, CSET 13
Test and Evalua-on
• Architecture
• Test suite
• Results
17
MINESTRONE: Tes-ng the SOUP, CSET 13
Test and Evalua-on: Test suite
• Vulnerability Classes: – Null pointer, 113 hYp://samate.nist.gov/SRD/testsuites/stonesoup/stonesoup-‐c-‐np.zip – Buffer Over/underflow, 231 hYp://samate.nist.gov/SRD/testsuites/stonesoup/stonesoup-‐c-‐mc.zip
• Input source: – Environment variable – Command line arguments – File – Network – Shared Memory – Clipboard
18
MINESTRONE: Tes-ng the SOUP, CSET 13
Test and Evalua-on
• Architecture
• Test suite
• Results
19
MINESTRONE: Tes-ng the SOUP, CSET 13
Test and Evalua-on: Results
20
MINESTRONE: Tes-ng the SOUP, CSET 13
Test and Evalua-on: Results
21
MINESTRONE: Tes-ng the SOUP, CSET 13
Test and Evalua-on: Results
22
MINESTRONE: Tes-ng the SOUP, CSET 13
Test and Evalua-on: Results
23
MINESTRONE: Tes-ng the SOUP, CSET 13
Test and Evalua-on: Results
24
MINESTRONE: Tes-ng the SOUP, CSET 13 25
Closing Thoughts
MINESTRONE: Tes-ng the SOUP, CSET 13
Lessons learned
• Symbolic execu-on limita-ons: – Limited model
– Very slow when it works (observed a 2700X overhead) • Wri-ng test suite from scratch is tricky: – Stack not always ini-alized to 0 – Provide the vulnerability loca-on to establish the ground truth
• I/O Redirec-on/replay is not a solved problem: – Many implementa-on available, ioapps, Jockey – Can you build it ? Do they work ?
• Enterprise products are not the silver bullet: – Single mul--‐purpose tools don’t outperform single purpose tailed tools
26
Thank you!
Copyright © 2011 Symantec Corpora-on. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corpora-on or its affiliates in the U.S. and other countries. Other names may be trademarks of their respec-ve owners. This document is provided for informa-onal purposes only and is not intended as adver-sing. All warran-es rela-ng to the informa-on in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The informa-on in this document is subject to change without no-ce.
27
Azzedine Benameur [email protected] Nathan Evans [email protected] MaYhew Elder [email protected]