why cybersecurity needs big data
TRANSCRIPT
Why Cybersecurity Needs Big Data & Intro to Apache Metron
James Sirota, Director Security Solutions
March 2017
Michael Schiebel, Cybersecurity Strategy
2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Michael Schiebel, Cybersecurity Strategy
www.linkedin.com/in/michaelschiebel/
James SirotaDirector Security Solutions
www.linkedin.com/in/jsirota/
Anna YongCybersecurity Product
Marketing
www.linkedin.com/in/4everfusion/
3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Why Cybersecurity Needs Big Data Intro to Apache Metron How big data experts can help IT security teams Case Study: Accelerating Investigation of a Phishing Attack
4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Why Cybersecurity Needs Big Data
5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Digital World Generates Big Data That Security Teams Need to Process
6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Existing Cyber Security Solutions Don’t Scale to the Challenge
82% of breaches happened in minutes
8 months: Average time an advanced security breach goes unnoticed
70%-80% of breaches are first detected by a 3rd party.
2016 Verizon Data Breach Investigations Report
Current security tools installed in the data center can’t handle volume of data & threats from everywhere
7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Cybersecurity JourneySingle View into Security Risk
Free data from security tools
Correlate and discover threats
Operational efficiency and governance
Predictive insights using machine learning
Single unified view of enterprise risk & security posture.
Innovate Renovate
Single Holistic View
HistoricalRecords
OPEXReduction
SecurityTool
Ingest
DigitalProtection
FraudPrevention
PublicData
Capture
A C T I V EA R C H I V E
DATAD I S C OV E RY
P R E D I C T I V EA N A LY T I C S
CyberSecurity
MachineData
RiskModeling
8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Metron at Capital One
CapOne uses HDF to ingest log data into their cyber security data lake and uses Apache Metron to detect threats that cannot be detected by traditional cyber security tools
https://youtu.be/Nffx8SKn7l4?t=1h37m50s
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Introduction to Apache Metron
10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Metron Journey
Jan 2016OpenSOC renamed Metron
Dec 2016Accepted
into Apache Incubation
Oct. 2015Hortonworks, Mantech, B23 press
release
Sept 2014First release of OpenSOC
Beta by CiscoApril 2014
OpenSOC in production
June 2014OpenSOC
Community Edition
July 2015Cisco stops supporting OpenSOC
March 2016
First Apache Release
11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Data Services and Integration Layer
Search andDashboarding
Portal
Security Data Vault
CommunityAnalytical
Models
Provisioning,Management
and Monitoring
ModulesReal-time ProcessingCyber Security Engine
TelemetryParsers Enrichment
ThreatIntel
AlertTriage
Indexersand
Writers
Cyber SecurityStream Processing Pipeline
Apache Metron: Incubating Project
Telemetry Ingest Buffer
TelemetryData Collectors
Real-timeEnrich / ThreatIntel Streams
PerformanceNetwork
IngestProbes
/ OtherMachine Generated Logs(AD, App / Web Server,
firewall, VPN, etc.)
Security Endpoint Devices (Fireye, Palo Alto,
BlueCoat, etc.)
Network Data(PCAP, Netflow, Bro, etc.)
IDS(Suricata, Snort, etc.)
Threat Intelligence Feeds(Soltra, OpenTaxi,third-party feeds)
TelemetryData Sources
12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
How Big Data Experts Can Help Security Teams
13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
A Day in the Life of An Analyst:
• Too many disparate tools • Too many alerts to process • Too much noise• How to connect the dots of
the relevant data points together?
14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Problem Posed (For Existing Tools)Security
InformationManagementSystem (SIEM)
• I am prohibitively expensive• I have vendor lock-in• I can’t deal with big data• I am not open• I am not extensible enough
LegacyPoint Tools
• I was built for 1995• I am super specialized• I don’t scale horizontally• I have a proprietary format• You need a PhD to operate me
BehavioralAnalytics
Tools (UEBA)
• I have a limited # of models• I am not trained on YOUR data• I am built by a small startup
15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Problem Posed (For Bad Guys)AdvancedPersistent
Threat
ScriptKiddie
• I am very unique in a way I do things• I live on your network for about 300 days• I know what I am after and I look for it, slowly• Your rules will not detect me, I am too smart• I impersonate a legitimate user, but I don’t act like one
Apache Metron can take everything that is known about me and check for it in real time
Apache Metron can model historical behavior of whoever I am impersonating and flag me as I try to deviate
• My techniques are predictable and known• My attack vectors are also known• I fumble around a lot• I set off a large number of alerts• You are not the only person I’ve attacked• I brag about what I did or will do
Repeatable Patterns Unique Patterns
16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Case Study: Accelerate Investigation of a Phishing Attack
17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
The “Threat Story” the Workflow Told….
18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
The Challenges faced by the SOC Analyst to Create this Story…
Challenge• The analyst had to jump from the SIEM to
more than 7 different tools that took up valuable time.
• It took more than 24 hours across 2 SOC shifts to investigate, determine scope, remediate and do further forensics/investigation.
• Half of my time was spending getting the context needed for me to create the story
• The threat was detected too late. Instead of detecting the incident on 4/9, the threat should have been detected on 3/20 when the attacker spoofed Sonja’s email address
Need• Want a Centralized View of my data so I don’t
have to jump around and learn other tools Eliminate manual tasks to investigate a case
• Need to discover bad stuff quicker
• Need the System to create the context for me in real-time
• The current static rules in the SIEM didn’t detect the threat. Need smart analytics based on:
• User Sonja hasn’t used corp gmail in the last 3 months
• User Sonja can’t login from Ireland and Southern Cali at the same time
19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Old School vs. New School Security ControlsEmail
Security Rules
Firewall Rules IDS Rules Sandbox
Rules DLP RulesOld School ->(1-1)
New School ->(1-*) Email
Classifier Alerts TriageMalware
Family Classifier
Network Behavior Classifier
UEBA System
20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Metron Resources
http://hortonworks.com/apache/metron/
https://metron.incubator.apache.org/
https://www.meetup.com/futureofdata-london/events/237165504/
21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Questions?