cybersecurity ai [tensorfuzz] debugging neural networks with coverage-guided fuzzing ·...
TRANSCRIPT
![Page 1: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/1.jpg)
[TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing
Authors: Augustus Odena, Ian Goodfellow
Presentor: Tahseen ShababFacilitators: Susan Shu, Serena McDonnell
Date: 26th August, 2019
Cybersecurity AI
![Page 2: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/2.jpg)
Tahseen ShababPresenter
CEO, Bibu Labs
Susan Shu Serena McDonnellFacilitator
Data Scientist, Bell
Facilitator
Senior Data Scientist, Delphia
Speakers
![Page 3: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/3.jpg)
![Page 4: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/4.jpg)
We
Prof Hassan KhanChief Scientist, Bibu Labs
Prof. Kate Larson Prof. Larry SmithAdvisor - AI, Bibu Labs Advisor - Strategy, Bibu Labs
We Are Growing!
![Page 5: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/5.jpg)
Feb, 2019
$1.4 B Acquisition
![Page 6: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/6.jpg)
![Page 7: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/7.jpg)
July, 2019
![Page 8: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/8.jpg)
Cylance Hack: Enable Dynamic Debugging
Cylance Antivirus
Verbose Logging
Score: {
-1000: Most Malicious+1000: Most Benign
}
Dynamic Debugging Enabled
![Page 9: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/9.jpg)
Cylance Hack: Reverse Engineer Model
7000 Feature Vectors Neural Network
Post ProcessingAdded Filter
White/Black List
![Page 10: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/10.jpg)
Cylance Hack: Exploit Model Bias
● Researchers found bias in the model○ A small set of features that have significant effect on outcome
● “Added Filter” uses Clusters with specific names to Whitelist files,
one being a famous game
● Researchers added strings from games executable to real
malicious file
● Game Over!
![Page 11: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/11.jpg)
Have We Seen This Before?
![Page 12: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/12.jpg)
Lawd & Meek (2005) and Wittel & WU (2004)
● Attacks against statistical spam filters
○ Add good words
○ Words the filter consider indicative of non-spam to spam
● Append words which appear often in ham emails and rarely in spam
to a spam email
● Spam Filter Fooled!
![Page 13: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/13.jpg)
Why Are These Hard To Spot?
![Page 14: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/14.jpg)
● Traditional Software○ Devs directly specify logic of
system
● ML System○ NN learns rules automatically○ Developers can indirectly modify
decision logic by manipulating■ Training data■ Feature selection■ Models architecture
○ NN’s underlying rules are mostly unknown to developers!https://arxiv.org/pdf/1705.06640.pdf
Source of Blind Spots
![Page 15: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/15.jpg)
Adversarial Attacks
![Page 16: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/16.jpg)
Adaptive Nature of Hackers
● Hackers Take Path of Least Resistance ● If a Patch is deployed, Hackers will take the path of least resistance
Vulnerability 1
Vulnerability 2
Vulnerability 3
![Page 17: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/17.jpg)
Data Distribution Actively Manipulated
![Page 18: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/18.jpg)
● Hackers strategically insert
attack data
● Model trains periodically
● Decision boundary is altered
Data Poisoning
secml.github.io
![Page 19: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/19.jpg)
![Page 20: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/20.jpg)
![Page 21: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/21.jpg)
● Add Noise
● Classifier Misclassifies Object
● Model learns differently than
humans
Attack: Induce Specific Output
“Explaining and Harnessing Adversarial Examples”, Ian Goodfellow
![Page 22: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/22.jpg)
Submit queries, observe response
● Training Data
● Architecture
● Optimization Procedures
Attack: Expose Model Attributes
"Towards Reverse Engineering Black Box Neural Networks”, Seong Oh
![Page 23: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/23.jpg)
Taxonomy of Attacks Against ML Systems
Axis Attack Properties
Influence Causative - influences training
and test data
Exploratory - Influences test data
Security Violation
Confidentiality - goal is to uncover
training data
Integrity - goal is false negatives
(FNs)
Availability - goal is false positives (FPs)
Specificity Targeted - influence prediction of
particular test instances
Indiscriminate - influence predictions of all test instances
Adversarial Machine Learning - Joseph, Nelson, Rubinstein and Tygar, 2019
![Page 24: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/24.jpg)
Exploratory Attacks Against Trained Classifier
● Attacker doesn’t have access to training data
● Most known detection techniques are susceptible to blind spots
● How difficult is it for adversary to discover blind spots that is most
advantageous to them?
![Page 25: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/25.jpg)
How Can We Find these Blind Spots?
https://www.theemotionmachine.com/listen-to-family-and-friends-how-to-protect-yourself-from-blind-spots/
![Page 26: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/26.jpg)
● Check erroneous corner cases● Input: Unlabeled test input● Objective: Generate test data
to:○ Activate large number of neurons○ Force DNNs to behave differently
● Joint Optimization Problem: Maximize
○ Differential behaviour○ Neuron coverage
DeepXplore: White Box Testing
![Page 27: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/27.jpg)
● Perform gradient guided local search
○ Starting: seed input○ Find new inputs that maximize
desired goal
● Similar to backpropagation, but:
○ Inputs: Variable○ Weights: Constant
DeepXplore: Example
![Page 28: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/28.jpg)
● Bayesian Neural Network● Adding dropout before every
weight layer approximation of gaussian process
○ Both training and test
● Dropout during test○ Different output for same input
■ [4,5,1,2,3,6]○ Equivalent to MC sampling○ High Variance = High uncertainty
Bayesian NN: Modelling Uncertainty
https://www.cs.ox.ac.uk/people/yarin.gal/website/blog_2248.html
![Page 29: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/29.jpg)
TensorFuzz
![Page 30: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/30.jpg)
TensorFuzz
● Open Source Tool● Discovers errors which occur only for rare inputs (Blind Spots)● Key Techniques:
○ Coverage Guided Fuzzing○ Property Based Testing○ Approximate Nearest Neighbor
![Page 31: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/31.jpg)
TensorFuzz
● Open Source Tool● Discovers errors which occur only for rare inputs (Blind Spots)● Key Techniques:
○ Coverage Guided Fuzzing○ Property Based Testing○ Approximate Nearest Neighbor
![Page 32: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/32.jpg)
● Instrument Program for coverage
○ Add instructions to code allowing fuzzer to detect code paths
● Feed Random Inputs into program
● Continue to mutate inputs that exercised new part of the program
○ Genetic Algorithm
● Identify bugs
Coverage Guided Fuzzing (AFL)
![Page 33: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/33.jpg)
● Aids the discovery of subtle fault conditions in the underlying code
● Security vulnerabilities are often associated with unexpected or incorrect state transitions
AFL: Branch Edge Coverage
AFL Documentation
![Page 34: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/34.jpg)
● Identifies potentially interesting control flow changes, ○ Ex. A block of code being
executed twice when it was normally hit only once
AFL Documentation
AFL: Hit Count
![Page 35: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/35.jpg)
● Sequential bit flips with varying lengths and stepovers,
● Sequential addition and subtraction of small integers,
● Sequential insertion of known interesting integers (0, 1, INT_MAX, etc)
AFL: Mutation Strategy
![Page 36: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/36.jpg)
TensorFuzz
● Open Source Tool● Discovers errors which occur only for rare inputs (Blind Spots)● Key Techniques:
○ Coverage Guided Fuzzing○ Property Based Testing○ Approximate Nearest Neighbor
![Page 37: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/37.jpg)
● Verifies a function or program
abides by a property
● Properties check for useful
characteristics that must be seen
in output
Property Based Testing
https://medium.com/criteo-labs/introduction-to-property-based-testing-f5236229d237
![Page 38: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/38.jpg)
● Cover the scope of all possible inputs○ Does not restrict the generated
inputs ● Shrink the input in case of failure
○ On failure, the framework tries to reduce the input to a smaller input
● Reproducible and replayable○ Each time it runs a property test,
a seed is produced in order to be able to re-run the test again on the same datasets
Advantage
https://medium.com/criteo-labs/introduction-to-property-based-testing-f5236229d237
![Page 39: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/39.jpg)
TensorFuzz
● Open Source Tool● Discovers errors which occur only for rare inputs (Blind Spots)● Key Techniques:
○ Coverage Guided Fuzzing○ Property Based Testing○ Approximate Nearest Neighbor
![Page 40: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/40.jpg)
Approximate Nearest Neighbor
http://web.stanford.edu/class/cs369g/files/lectures/lec16.pdf
● Nearest Neighbor○ Given points p1,p2,...,pn, and
query point q, find closest point to q among p1,...,pn
● Approximate Nearest Neighbor○ Condition is relaxed○ Fin pi so that
■ d(q,pi) <=c.min d(q,pj)
![Page 41: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/41.jpg)
TensorFuzz
● Open Source Tool● Discovers errors which occur only for rare inputs (Blind Spots)● Key Techniques:
○ Coverage Guided Fuzzing○ Property Based Testing○ Approximate Nearest Neighbor
![Page 42: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/42.jpg)
Sadly, CGF Tools Don’t Work For Neural Networks
![Page 43: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/43.jpg)
● Coverage Metrics○ Lines of Code Executed○ Which branches have been taken
Traditional Software Workflow
https://arxiv.org/pdf/1705.06640.pdf
![Page 44: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/44.jpg)
● Software implementation may contain many branching statements
○ Based on architecture○ Mostly independent of input
● Different inputs will often execute ○ same lines of code ○ same branches,
● But will produce interesting variations in behaviour
Neural Network Workflow
https://arxiv.org/pdf/1705.06640.pdf
![Page 45: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/45.jpg)
How Does TensorFuzz Work?
![Page 46: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/46.jpg)
Let's Dive In!
Dio, Holy Diver
![Page 47: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/47.jpg)
TensorFuzz1. We interact with a
TensorFlow Graph instead of instrumented Computer Program
![Page 48: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/48.jpg)
2. Valid neural network inputs are fed instead of big array of bytes.
Ex. For, if inputs are sequences of character, only allow characters that are in vocabulary extracted from the training set
TensorFuzz
![Page 49: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/49.jpg)
3. Input Chooser intelligently chooses elements from input corpus.
Following heuristics is used:
: Probability of choosing corpus element ck at time t
tk: Time when ck was added to the corpus
Intuition: Recently sampled inputs are more likely to yield useful new coverage when mutated, but advantage decays over time.
TensorFuzz
![Page 50: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/50.jpg)
4. Mutator modifies input in a controlled manner
For text input, mutation occurs in accordance to following policy:
Uniformly at random perform one of following operations:
- Delete, Add, Subtract - Random character at
random location
TensorFuzz
![Page 51: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/51.jpg)
Diving Deeper
5. Mutated inputs are fed to Neural Network. The following are extracted from NN
- Set of coverage arrays- Enables computation
of coverage- Set of metadata arrays
- Fed as input to objective function
![Page 52: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/52.jpg)
5.a Objective Function
- Desired Outcome- Ex. Error, crash
Outputted Metadata arrays is fed into Objective function, and inputs causing system to reach goal of objective function are flagged
TensorFuzz
![Page 53: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/53.jpg)
5.b Coverage Analyzer
Core part of product
Reading arrays from TensorFlow runtime, turning them into python objects representing coverage, checking whether that coverage is new
TensorFuzz
![Page 54: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/54.jpg)
Desired Properties of Coverage Analyzer
● Check if Neural Network is in new state○ Enables detection of misbehaviour
● Check has to be fast● Should work with many different computation graphs
○ Remove Manual Intervention as much as possible● Exercising all of the coverage should be hard
○ Or else we won’t cover much of possible behaviours
![Page 55: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/55.jpg)
Use Fast Approximate Nearest Neighbour
● Determine if two sets of NN activations are meaningfully different from each other
● Provides a coverage metric producing useful results for neural network○ Even if underlying software implementation of the neural network does not make
use of many data-dependent branches
![Page 56: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/56.jpg)
Intuition: Coverage Analyzer
Activation
Activation
Activation
ActivationCurrent Input
Old Input
Delta DeltaDelta
New Coverage Reached If Distance Sufficiently Large
![Page 57: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/57.jpg)
● On New Activation Vectora. Use Approximate nearest
neighbors Algorithmb. Look up nearest neighbourc. Check distance between
current and nearest neighbour in Euclidean distance
d. Add input to corpus if distance is greater than Lhttps://medium.com/@erikhallstrm/backpropa
gation-from-the-beginning-77356edf427d
Coverage Analyzer: Details
![Page 58: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/58.jpg)
● Note: Often, good results are achieved only by looking at logits or layer before logits
https://medium.com/@erikhallstrm/backpropagation-from-the-beginning-77356edf427d
Coverage Analyzer: Details
![Page 59: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/59.jpg)
6. Mutated input is:
- Add to corpus if- New coverage is achieved
- Added to list of test cases if- Objective function is satisfied
TensorFuzz
![Page 60: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/60.jpg)
Break
![Page 61: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/61.jpg)
https://www.bandt.com.au/media/facebook-manipulated-users-feeds-experiment
Experiments
![Page 62: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/62.jpg)
Experiment: Finding NaNs
● NaNs consistently cause trouble for researchers and practitioners, but
they are hard to track them down
● A bad loss function is “fault injected” into a neural network
● TesnorFuzz could find NaNs substantially faster than a baseline
random search
![Page 63: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/63.jpg)
● Left: Coverage overtime for 10 different random restarts
● Right: An example of a random image that causes neural network to NaN
Experiment: Finding NaNs
![Page 64: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/64.jpg)
Experiment: Quantization Errors
● We often want to quantize neural networks● How to test for accuracy? ● We can look at differences in test sets, but often few show up● Instead, we can fuzz for inputs that surface differences
![Page 65: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/65.jpg)
● Left: Coverage overtime for 10 different random restarts. Note that 3 runs fail
● Right: An example of an image correctly classified by the original neural network but incorrectly classified by the quantized network
Experiment: Quantization Errors
![Page 66: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/66.jpg)
Discussion
![Page 67: Cybersecurity AI [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing · 2020-05-11 · [TensorFuzz] Debugging Neural Networks with Coverage-Guided Fuzzing Authors:](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f139da58594495abc07ae2f/html5/thumbnails/67.jpg)
Discussion Points
● How do we embed security testing into the ML Solution development lifecycle?
● Can explainable inference help to detect blind spots?● Can we use multiple classifiers in parallel to reduce the implications of an
attack on a specific model?