large scale generation of complex and faulty php test cases

Large Scale Generation of Complex and Faulty PHP

Test Cases

Bertrand STIVALET

Elizabeth FONG

ICST 2016Chicago, IL, USAApril 15th, 2016

http://samate.nist.gov

Authors

Bertrand STIVALETNational Institute of Standards and Technology

[email protected]

2

Elizabeth FONGNational Institute of Standards and [email protected]

“"If debugging is the process of removing software bugs, then

programming must be the process of putting them in"

E. Dijkstra

3

NIST - SAMATE - SARD

◎ NIST - National Institute of Standards and Technology○ Part of the US Department Of Commerce○ Promote U.S. Innovation and Industrial Competitiveness

◎ SAMATE - Software Assurance Metrics And Tool Evaluation○ Improve Software Assurance by:

● developing materials, specifications, and methods ● testing tools and techniques and measure their effectiveness

4

◎ SARD - Software Assurance Reference Dataset○ Provide database of known security flaws○ C/C++, JAVA, PHP, C#○ 148,903 Test cases / 665,481 Files

Outline

5

1. Software Testing

Outline

6

1. Software Testing

StaticApplicationSecurityTesting

Safe Code

Outline

7

Safe CodeSafe CodeTest Cases

1. Software Testing


2. Design of Test Cases

Safe Code

Outline

8


1. Software Testing


Test Cases Generator

2. Design of Test Cases3. PHP Vulnerability


Safe Code

Outline

9


1. Software Testing



2. Design of Test Cases3. PHP Vulnerability


4. Live Demo

1.Software TestingIntroduction to Static Analysis

10

Static Analysis

◎ Automated analysis of large software

◎ Defect detection and remediation

◎ Use different approaches:

11

○ Syntax checking○ Heuristics ○ Formal methods

Static Analysis

Buggy SourceCode

Compilation

12

Buggy Software



◎ Use different approaches

Static Analysis

Buggy SourceCode

13

Bug Report

Static Analysis

Remediation




Static Analysis

Fixed Source Code

14

SecureSoftware

Compilation




Static Analysis

Buggy SourceCode

15

Bug Report

Static Analysis




?

Static Analysis Testing

16

Static AnalysisSafe Code

Bug Report

True Negative


17


Safe Code

Bug Report

Bug ReportStatic Analysis

True Negative

False Positive


18


Safe Code

Bug Report

Bug ReportStatic Analysis

True Negative

False PositiveNOISE


19


Safe Code

Bug Report

Bug Report

Static AnalysisBuggy Code

Bug Report

Static Analysis

True Negative

True Positive

False PositiveNOISE


20


Safe Code

Bug Report

Bug Report


Buggy Code

Bug Report

Bug Report

Static Analysis

Static Analysis

True Negative

True Positive

False Positive

False Negative

NOISE


21


Safe Code

Bug Report

Bug Report


Buggy Code

Bug Report

Bug Report

Static Analysis

Static Analysis

True Negative

True Positive

False Positive

False NegativeMISSED

DEFECT

NOISE

◎ Improves software assurance

◎ Saves time and money

◎ Takes customized rule sets

◎ False positive (noise)

◎ False negative (missed defects)

◎ Limited scope

Pros and Cons

22

2.Design of Test CasesTest cases features

23

Test Cases Design

◎ Cover the most vulnerabilities possible◎ Various complexities◎ Statistically significant◎ Ground truth◎ Paired safe and flawed test cases ◎ Representative of production code

24

PHP Test Case Example

25


26

INPUT


27

INPUT

FILTERING


28

INPUT

FILTERING

SINK

3.PHP Vulnerability Test Cases GeneratorOverview of the Test Cases generator

29


30

Safe CodeSafe CodeBuggy Code

Safe CodeSafe Code

Conditional Loops Functions Classes Multiple Files

Complexities: choose none, one, or combine several

InputTemplates

FilteringTemplates

SinkTemplates

Selected Input

Selected Filtering

Selected Sink

File Structure: Input + Filtering + Sink

Safe Code

Test Cases Design

◎ Various complexities◎ Statistically significant◎ Ground truth◎ Paired safe and flawed test cases◎ Cover the more vulnerabilities possible ◎ Representative of production code

31

32

Vulnerabilities covered

◎ Vulnerabilities based on OWASP Top 10 2013 [ #safe / #unsafe ]

○ Injection [ 20912 / 5920 ]○ Broken Authentication and Session Management○ Cross Site Scripting (XSS) [ 5728 / 4352 ]○ Insecure Direct Object References [ 400 / 80 ]○ Security Misconfiguration [ 5 / 3 ]○ Sensitive Data Exposure [ 5 / 7 ]○ Missing Function Level Access Control ○ Cross-Site Request Forgery (CSRF)○ Using Known Vulnerable Component○ Unvalidated Redirects and Forwards [ 2208 / 2592 ]

4.Live DemoGenerating Test Cases to Testing

33

Live Demo

34

<?PHP Vulnerability Test Case Generator?>

Safe CodeSafe CodePHPTest

Cases

RIPS - Metrics

35

Missed defects- present : 912- found : 312*

Recall = 312 / 912 = 34.2%

* considering all findings are True positives

RIPS - True Positive

36

CWE_89__GET__no_sanitizing__multiple_select-interpretation.php

INPUT

FILTERING

SINK

SQL Injection :Userinput reaches sensitive sink.

Report

RIPS - False Positive

37

CWE_89__object-directGet__CAST-func_settype_float__multiple_AS-sprintf_%u.php

INPUT

FILTERING

SINK

SQL Injection :Userinput returned by function getinput() reaches sensitive sink.

Report

Conclusion

◎ Tools need evaluation!

◎ Test cases need improvement

◎ PHP Vulnerability Test Suite Generator:

○ Automated generation○ Modular and expandable○ Customizable with options○ 42 000 PHP test cases generated

38

Conclusion

◎ Tool is available on Github:

https://github.com/stivalet/PHP-Vuln-test-suite-generator

◎ Test cases are hosted in the SARD:

https://samate.nist.gov/SARD/view.php?tsID=103

◎ Project is already used by researchers:○ M. K. Gupta, et al, “Security Vulnerabilities in Web Applications", JCSSE 2015

○ M. K. Gupta, et al, "XSSDM: Towards Detection and Mitigation of Cross-Site Scripting Vulnerabilities in Web Applications", ICACCI 2015

○ SATE VI - Static Analysis Tool Exposition, NIST 201639

Thanks!Any questions?Find us at:

http://[email protected]

Twitter: @B_Stivalet

40

large scale generation of complex and faulty php test cases

Documents