real world software assurance test suite€¦ · test suite - cwes for c programs weakness type...

Real World Software Assurance Test Suite:

STONESOUPCharles Oliveira/SAMATE

<[email protected]>

Guest Researcher at Software and Systems Division, IT Laboratory

NIST

Outline- Introduction

- STONESOUP program

- Test suite

- Test case sample

- TEXAS usage

- Documents and reports

2

Introduction - SOUP

libssl libxml

libpq ...

Open source libs

Java/Spring

PHP/Zend

C++/Boost ...

Frameworks

Apache

Postgres

Drivers

...

Standalone apps

Application

Is this Software Of Unknown Provenance

(SOUP) safe?

3rd party software

3

STONESOUP program

Securely Taking On New Executable

Software Of Uncertain Provenance

(STONESOUP) http://www.iarpa.gov/index.php/research-programs/stonesoup

4

STONESOUP programThe goal of STONESOUP program was to eliminate the effects of vulnerabilities in

software applications by:

- extending the scope and capability of approaches for analysis, confinement, and

diversification;

- addressing a wide range of security vulnerabilities within the same framework;

- integrating approaches to leverage the strengths and weaknesses of each;

- adding no more than 10% running time slowdown.

5

STONESOUP programPhase 1

Neutralize 75% of vulnerabilities of 2 weakness types in 10k SLOC programs

Phase 2

Neutralize 80%+ of vulnerabilities of 4 weakness types in 100k SLOC programs

Phase 3

Neutralize 90%+ of vulnerabilities of 6 weakness types in 500k SLOC programs

Phase 3 performers were those that made significant

progress in Phase 2 as measured by the program metrics.

The three teams and the names of their developmental

tools are:

Kestrel Institute - VIBRANCE (video)

Columbia University - Minestrone

Grammatech - PEASOUP

6

STONESOUP program - PerformersSTONESOUP performers neutralize vulnerabilities in:

7

STONESOUP program - Test & Evaluation System- Test & Evaluation eXecution and Analysis System (TEXAS) was designed

and developed to test Performer technology

- Developed by STONESOUP team

- Command Line Interface (CLI) to run and evaluate tests cases

- Communication API to interact to Performer’s tools

8

Test suite - Base programs

637477

637

638636

637

480476

637

478

380JTree

380GNU Tree

160

479

380

GNU Grep

478

9

Number of test cases per base program in red circles

Test suite - CWEs for C programsWeakness type CWEs (56)

Concurrency handling(765) 363 367 412 414 543 609 663 764 765 820 821 833 831 828 479

Injection(701) 078 088 089

Number handling(725) 190 191 194 195 196 197 369 682 839

Resource drains(733) 400 459 674 774 789 834 835 401 771 773 775

Memory corruption(965) 120 124 126 127 129 134 170 415 416 590 761 785 805 806 822 824 843

Null pointer(693) 476

10



Injection(701) 078 088 089

Number handling(725) 190 191 194 195 196 197 369 682 839

Resource drains(733) 400 459 674 774 789 834 835 401 771 773 775



11

CWE-363: Race Condition Enabling Link Following (2.8)CWE-367: Time-of-check Time-of-use (TOCTOU) Race Condition (2.8)CWE-412: Unrestricted Externally Accessible Lock (2.8)CWE-414: Missing Lock Check (2.8)CWE-543: Use of Singleton Pattern Without Synchronization in a Multithreaded Context (2.8)CWE-609: Double-Checked Locking (2.8)CWE-663: Use of a Non-reentrant Function in a Concurrent Context (2.8)CWE-764: Multiple Locks of a Critical Resource (2.8)CWE-765: Multiple Unlocks of a Critical Resource (2.8)CWE-820: Missing Synchronization (2.8)CWE-821: Incorrect Synchronization (2.8)CWE-833: Deadlock (2.8)CWE-831: Signal Handler Function Associated with Multiple Signals (2.8)CWE-828: Signal Handler with Functionality that is not Asynchronous-Safe (2.8)CWE-479: Signal Handler Use of a Non-reentrant Function (2.8)



Injection(701) 078 088 089

Number handling(725) 190 191 194 195 196 197 369 682 839

Resource drains(733) 400 459 674 774 789 834 835 401 771 773 775



12

CWE-078: OS Command Injection (2.8)CWE-088: Argument Injection or Modification (2.8)CWE-089: SQL Injection (2.8)



Injection(701) 078 088 089

Number handling(725) 190 191 194 195 196 197 369 682 839

Resource drains(733) 400 459 674 774 789 834 835 401 771 773 775



13

CWE-190: Integer Overflow or Wraparound (2.8)CWE-191: Integer Underflow (Wrap or Wraparound) (2.8)CWE-194: Unexpected Sign Extension (2.8)CWE-195: Signed to Unsigned Conversion Error (2.8)CWE-196: Unsigned to Signed Conversion Error (2.8)CWE-197: Numeric Truncation Error (2.8)CWE-369: Divide By Zero (2.8)CWE-682: Incorrect Calculation (2.8)CWE-839: Numeric Range Comparison Without Minimum Check (2.8)



Injection(701) 078 088 089

Number handling(725) 190 191 194 195 196 197 369 682 839

Resource drains(733) 400 459 674 774 789 834 835 401 771 773 775



14

CWE-400: Resource Exhaustion (2.8)CWE-459: Incomplete Cleanup (2.8)CWE-674: Uncontrolled Recursion (2.8)CWE-774: Allocation of File Descriptors or Handles Without Limits or Throttling (2.8)CWE-789: Uncontrolled Memory Allocation (2.8)CWE-834: Excessive Iteration (2.8)CWE-835: Infinite Loop (2.8)CWE-401: Memory Leak (2.8)CWE-771: Missing Reference to Active Allocated Resource (2.8)CWE-773: Missing Reference to Active File Descriptor or Handle (2.8)CWE-775: Missing Release of File Descriptor or Handle after Effective Lifetime (2.8)



Injection(701) 078 088 089

Number handling(725) 190 191 194 195 196 197 369 682 839

Resource drains(733) 400 459 674 774 789 834 835 401 771 773 775



15

CWE-120: Classic Buffer Overflow (2.8)CWE-124: Buffer Underflow (2.8)CWE-126: Buffer Over-read (2.8)CWE-127: Buffer Under-read (2.8)CWE-129: Improper Validation of Array IndexCWE-134: Uncontrolled Format String (2.8)CWE-170: Improper Null Termination (2.8)CWE-415: Double Free (2.8)CWE-416: Use After Free (2.8)

CWE-590: Free of Memory not on the Heap (2.8)CWE-761: Free of Pointer not at Start of Buffer (2.8)CWE-785: Use of Path Manipulation Function without Maximum-sized Buffer (2.8)CWE-805: Buffer Access with Incorrect Length Value (2.8)CWE-806: Buffer Access Using Size of Source Buffer (2.8)CWE-822: Untrusted Pointer Dereference (2.8)CWE-824: Access of Uninitialized Pointer (2.8)CWE-843: Type Confusion (2.8)



Injection(701) 078 088 089

Number handling(725) 190 191 194 195 196 197 369 682 839

Resource drains(733) 400 459 674 774 789 834 835 401 771 773 775



16

CWE-476: NULL Pointer Dereference (2.8)

Test suite - CWEs for Java programsWeakness type CWEs (50)


Injection(526) 078 088 089 564

Number handling(532) 190 191 194 195 196 197 369 839

Resource drains(532) 400 459 674 774 789 834 835

Error handling(532) 209 248 252 253 390 391 460 584

Tainted data(498) 023 036 041 606

17



Injection(526) 078 088 089 564

Number handling(532) 190 191 194 195 196 197 369 839

Resource drains(532) 400 459 674 774 789 834 835

Error handling(532) 209 248 252 253 390 391 460 584

Tainted data(498) 023 036 041 606

18

CWE-363: Race Condition Enabling Link Following (2.8)CWE-367: Time-of-check Time-of-use (TOCTOU) Race Condition (2.8)CWE-412: Unrestricted Externally Accessible Lock (2.8)CWE-414: Missing Lock Check (2.8)CWE-543: Use of Singleton Pattern Without Synchronization in a Multithreaded Context (2.8)CWE-609: Double-Checked Locking (2.8)CWE-663: Use of a Non-reentrant Function in a Concurrent Context (2.8)CWE-764: Multiple Locks of a Critical Resource (2.8)CWE-765: Multiple Unlocks of a Critical Resource (2.8)CWE-820: Missing Synchronization (2.8)CWE-821: Incorrect Synchronization (2.8)CWE-833: Deadlock (2.8)CWE-832: Unlock of a Resource that is not Locked (2.8)CWE-567: Unsynchronized Access to Shared Data in a Multithreaded Context (2.8)CWE-572: Call to Thread run() instead of start() (2.8)



Injection(526) 078 088 089 564

Number handling(532) 190 191 194 195 196 197 369 839

Resource drains(532) 400 459 674 774 789 834 835

Error handling(532) 209 248 252 253 390 391 460 584

Tainted data(498) 023 036 041 606

19

CWE-078: OS Command Injection (2.8)CWE-088: Argument Injection or Modification (2.8)CWE-089: SQL Injection (2.8)CWE-564: SQL Injection: Hibernate (2.8)



Injection(526) 078 088 089 564

Number handling(532) 190 191 194 195 196 197 369 839

Resource drains(532) 400 459 674 774 789 834 835

Error handling(532) 209 248 252 253 390 391 460 584

Tainted data(498) 023 036 041 606

20

CWE-190: Integer Overflow or Wraparound (2.8)CWE-191: Integer Underflow (Wrap or Wraparound) (2.8)CWE-194: Unexpected Sign Extension (2.8)CWE-195: Signed to Unsigned Conversion Error (2.8)CWE-196: Unsigned to Signed Conversion Error (2.8)CWE-197: Numeric Truncation Error (2.8)CWE-369: Divide By Zero (2.8)CWE-839: Numeric Range Comparison Without Minimum Check (2.8)



Injection(526) 078 088 089 564

Number handling(532) 190 191 194 195 196 197 369 839

Resource drains(532) 400 459 674 774 789 834 835

Error handling(532) 209 248 252 253 390 391 460 584

Tainted data(498) 023 036 041 606

21

CWE-400: Resource Exhaustion (2.8)CWE-459: Incomplete Cleanup (2.8)CWE-674: Uncontrolled Recursion (2.8)CWE-774: Allocation of File Descriptors or Handles Without Limits or Throttling (2.8)CWE-789: Uncontrolled Memory Allocation (2.8)CWE-834: Excessive Iteration (2.8)CWE-835: Infinite Loop (2.8)



Injection(526) 078 088 089 564

Number handling(532) 190 191 194 195 196 197 369 839

Resource drains(532) 400 459 674 774 789 834 835

Error handling(532) 209 248 252 253 390 391 460 584

Tainted data(498) 023 036 041 606

22

CWE-209: Information Exposure Through an Error Message (2.8)CWE-248: Uncaught Exception (2.8)CWE-252: Unchecked Return Value (2.8)CWE-253: Incorrect Check of Function Return Value (2.8)CWE-390: Detection of Error Condition Without Action (2.8)CWE-391: Unchecked Error Condition (2.8)CWE-460: Improper Cleanup on Thrown Exception (2.8)CWE-584: Return Inside Finally Block (2.8)



Injection(526) 078 088 089 564

Number handling(532) 190 191 194 195 196 197 369 839

Resource drains(532) 400 459 674 774 789 834 835

Error handling(532) 209 248 252 253 390 391 460 584

Tainted data(498) 023 036 041 606

23

CWE-023: Relative Path Traversal (2.8)CWE-036: Absolute Path Traversal (2.8)CWE-041: Improper Resolution of Path Equivalence (2.8)CWE-606: Unchecked Input for Loop Condition (2.8)

Test suite - Base programs- Total of 7770 test cases which generates ~240GB compressed!!!

- The STONESOUP Test and Evaluation team (T&E) used 277 independent virtual

machines simultaneously on Amazon Web Services between April and December

2014 for performers to run the test cases.

- The NIST VM is 22GB and contains test cases patched from the base program

- The strategy was to patch the test cases, distributing ‘.diff’ files instead of whole

copies of each base program

24

Test suite - Virtual Machine (VMware)- Download (11x2GB) at http://samate.nist.gov/SARD/testsuite.php#standalone

- OS: Ubuntu 12.04

- CPU: 4 VCPU recommended

- Memory: 4GB (8GB recommended)

- Storage: 59GB Total / 41GB Used / 16GB Available

- Inside NIST_TT_VM folder there is a document with login and password for the VM

- Important directories:

- /opt/stonesoup: contains the entire NIST STONESOUP package including scripts and documents

- /opt/share: contains a TEXAS installation, test cases (diffs), base programs all their dependencies

- Performers’ tools are not in the VM

25

http://samate.nist.gov/SARD/testsuite.php#standalone

Test case sample (from the virtual machine)- Pick a test case in /opt/share/testcases/

- Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01

26



C or Java

27



CWE-476: NULL Pointer Dereference

28



29

Algorithmic variant: refined CWEs mapped to a code snippet previously defined by T&E team



Base program:

CMUD Coffee MUD CTRE GNU Tree FFMP FFMpeg GIMP Gimp

GREP GNU Grep OSSL OpenSSL PSQL Postgres SUBV Apache Subversion

WIRE Wireshark ELAS Elastic Search JMET Apache JMeter JENA Apache Jena

JTRE Java Tree LENY ApacheLenya LUCE Apache

Lucene POIX Apache POI

30



Injection point: represent specific locations in the base program that are guaranteed

to be executed given the defined I/O pairs. Identifiers reference different injection points in each base program.

31



Taint source:

01 ENVIRONMENT_VARIABLE

02 FILE_CONTENTS

03 SOCKET

04 SHARED_MEMORY

32



Data type:

01 ARRAY 05 STRUCT

02 SIMPLE 06 TYPEDEF

03 VOID_POINTER 07 UNION

04 HEAP_POINTER

33



Data flow:

01 ADDRESS_ALIAS_1 11 BASIC

05 ADDRESS_AS_CONSTANT 12 VAR_ARG_LIST

06 ADDRESS_AS_FUNCTION_RETURN_VALUE 17 BUFFER_ADDRESS_POINTER

10 INDEX_ALIAS_50 18 JAVA_GENERICS

34



Control flow:

01 INTERCLASS_1 18 POINTER_TO_FUNCTION

08 INTERFILE_1 19 RECURSIVE

12 INTERPROCEDURAL_1 22 MACROS

16 INTERRUPT 26 FUNCT_INVOC_OVERLOAD

35



Unique increment:increment in case of multiple test cases are sharing the same

parameters aforementioned.

36



- Browsing the test case

- install/: this test case installation files

- scripts/: specific scripts to manage running process

- src/: the entire base program + files seeded with intentional weaknesses

- testData/: input data which will [and won’t] trigger the seeded weakness

- testOutput/: matching output data for each input data

- C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01.xml: TEXAS “makefile”

- C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01.yaml: benign and exploiting inputs

37

Stage 1: standard compilation

Stage 2: compilation with performer

technology

TEXAS usage

I/O Pairs

Analysis/Compilation

Execution

Scoring

The source code or binary of a program is scanned looking for

CWE code patterns and applying diversification techniques to

harden the resulting binary. The output of the Analysis phase is a

binary executable.

The Execution step is run for each I/O, and involves actually

invoking the binary created in the Analyze step with known

inputs. Performer technology may also monitor the execution of

the binary to look for execution patterns indicative of an attack in

progress or software vulnerability.

Scoring executed immediately after the Execution step and looks at

the environment for the known outputs defined in the metadata

for the given I/O pair that was executed.

38

Analysis/Compilation

Execution

Scoring

Documents & ReportsMain STONESOUP documents provided at SARD website:

- Test and evaluation phase 3 final report

- Performers’ reports

- Weaknesses documentation

- Test cases creation guide

- TEXAS user guides

Visit: http://samate.nist.gov/SARD/around.php

39

http://samate.nist.gov/SARD/around.php

Questions?

Charles Oliveira/SAMATE

[[email protected]]

40