soc2016 - the investigation labyrinth
TRANSCRIPT
TheInvestigation Labyrinth
Chris SandersSecurity Onion Con 2016
Chris Sanders (@chrissanders88)
Find Evil @ FireEye Founder @ Rural Tech
Fund PhD Researcher GSE # 64 BBQ Pit Master Author:
Practical Packet Analysis Applied NSM
Agenda Era of Analysis DFIR Cognitive Revolution Researching the Investigation Process Data, Data, and more Data
The economics of NSM are not in our favor – how can we study the investigation process
to make it more efficient?
Economics of Security“If you want to understand the world of nature, master physics. If you want to understand the world of man, master economics.” - Taufiq Rashid
High Demand
for Security Expertise
Low Supply of Security
Practitioners
Expertise
Services
Software
Evolution of NSM “The profession
[security] is so nascent that the how-tos have not been fully realized even by the people who have the knowledge.”
Every thought-based profession goes through a cognitive crisis and revolution. Ours is coming.
Analysis Era
Collecton Era
Detection Era
Symptoms of a Cognitive Crisis1. Demand for expertise greatly
outweights supply2. Most information cannot be trusted or
validated3. Inability to mobilize and tackle big
systemic issues
The Cognitive Revolution in DFIR1. Understand the processes
used to perform investigations and draw conclusions
2. Develop repeatable methods and techniques for performing investigations
3. Build and advocate training that teaches analysts how to think about investigations, not just how to use tools.
Investigations as Mental Labyrinths The investigation is
the core construct of information security.
At a high level, an investigation is a series of decisions that begets other decisions.
Defenders don’t always know if they’ve taken the correct path.
Navigating the Labyrinth
Alert
OSINT
Reputation
File Hash
Sandbox Behaviors
AV Detections
(VT)
Imphash More File Hashes
Friendly Host
Network PCAP
Host
Windows Logs
Security Log
System Log
App LogRegistry
File SystemHostile
Host NetworkPCAP
Flow
Studying the Investigation Process
Goal: Increase Accuracy Decrease Time
How do you study something human thought?
Challenges: Creating unique
investigation scenarios takes time
There is no universal set of tooling
A Scenario-Based Approach to Investigation Analysis
Create a tool-agnostic investigation simulator
Make it portable and self contained Seed it with investigation scenarios
where one variable can be addressed at a time
Allow it to log investigator actions and output a log of decisions being made
Additional Data Sources
Case StudyAnalyzing the Flow of
the Investigation
The Compromise1. Victim visits friendly
website2. Redirect to EK landing page3. Download flash exploit4. Exploit is successful and
ransomware file downloads5. Ransomware installs and
executes6. Ransomware begins C2
communication
What data did analysts look at first?
72%16%
12%
Observed
PCAP FlowOSINT
Data Suggests: Analysts prefer a higher context data set…
…even if other data sets are available …even if lower context data sets can lead to a resolution.
Analysts don’t fully understand their own techniques
49%28%
23%
Reported
PCAP FlowOSINT
Did the first move affect analysis speed?
Data Suggests: While PCAP provides richer context, it may slow down the
investigation if that’s where you start Starting with a lower context data source can increase
speed when working with higher context data
PCAP Flow OSINT
16
10 9
PCAP Flow OSINT
27
13 13
Avg Time to Close Weighted Time to Close
What happens when Bro data replaces PCAP?
46%
25%
29%
Observed (Bro)
Bro Flow OSINT
72%
16%
12%
Observed (PCAP)
PCAP Flow OSINT
What happens when Bro data replaces PCAP?
PCAP Flow OSINT
16
10 9
Avg Time to Close (PCAP)
Bro Flow OSINT
10 10 11
Avg Time to Close (Bro)
Data Suggests: Better organization of high context data
sources can yield improvements in analysts performance
What data sources were viewed most and least frequently?
Data Suggests: Network data is used more frequently than host data…
…even when host data can be used exclusively to resolve. …even when easy access is provided to host sources.
Revisting data is more prevalent on higher context data sources
PCAP Flo
wOSIN
T
Host FS
OS Log
s
Memory
Data Sources Viewed Data Sources Revisited
PCAP84%
Flow11%
OSINT5%
How many steps were taken to make a disposition judgement?
Data Suggests: At some point, the number of data sources you
investigate impacts the speed of the investigation Understanding where data exists and when to use it can
impact analysis speed
6-10 11-15 16-20 21-250
5
10
15
6
129
3
Number of Steps
6-10 11-15 16-20 21-2505
1015202530
9 12 14
24
Avg Time to Close
Did analysts investigate friendly or hostile systems first?
9%
91%
Observed
Friendly Hostile
Data Suggests: Analysts are more compelled to investigate unknown external
threats than internal systems Analysts don’t fully understand their own techniques
41%59%
Friendly
Friendly Hostile
Do analysts seek to prove or disprove the alert?
Data Suggests: Analysts are almost always seek to prove an alert...
...despite the fact that disproving it is usually faster.
Prove vs. Disprove
Prove88%
Disprove12%
Prove Disprove05
101520
19
8
Avg Time to Close
Case StudyWhat separates
novice and expert analysts?
Mapping the Investigation Sample:
Novice and expert analysts Methodology:
30 case studies Stimulated recall interviews Focus on individual investigations of
varying types Perform key phrase anlayis – analyze
results
Key Phrase Mapping Dual Process Theory
Intuition: Implicit, inconscious, fast Reflection: Explicit, controlled, slow
IntuitionExperimentation
RestructuringImaginationIncubation
MetacognitionEvaluation
Goal SettingMaking Plans
ReflectionAnalytically
Viewing DataRule-Based Reasoning
Considering Alternatives
Results
Expe
rimen
tation
Restruc
turing
Imag
inatio
n
Incub
ation
Evalu
ation
Goal S
etting
Making
Plan
s
Novices Experts
Intuition Metacognition Reflection
