big data analytics to enhance security คุณอนพัทย์...
TRANSCRIPT
Big Data Analytics to Enhance Security Anapat Pipatkitibodee
Technical Manager
The First NIDA Business Analytics and Data Sciences Contest/Conference วันที่ 1-2 กันยายน 2559 ณ อาคารนวมินทราธิราช สถาบันบัณฑิตพัฒนบริหารศาสตร ์
https://businessanalyticsnida.wordpress.com https://www.facebook.com/BusinessAnalyticsNIDA/
ใช้ Big Data มาเพิ่มความปลอดภัยได้อย่างไร Big Data Analytics Security Trends
Example Security Attack Integrated Security Analytics with Open Source
How to apply?
นวมินทราธิราช 3001 วันที่ 1 กันยายน 2559 16.30-17.00 น.
Big Data Analytics to
Enhance Security
Anapat Pipatkitibodee
Technical Manager
STelligence Company Limited
anapat.p@stelligence
Agenda
• Big Data Analytics
• Security Trends
• Example Security Attacks
• Integrated Security Analytics with Open Source
• How to Apply ?
Big Data Analytics
Everyone is Claiming Big Data
Traditional vs Big Data
Drivers of Big Data
• About 80% of the world’s data are semi-structured or unstructured.
Open Source Tools in Big Data
• Hadoop ecosystem
• NoSQL database
Apache Hadoop Stack
Reference:
Hadoop Essentials
by Swizec Teller
https://whatsthebigdata.com/2016/02/08/big-data-landscape-2016/
Big Data Analytics
• The process of examining large data
sets containing a variety of data types
i.e., big data.
• Big Data analytics enables
organizations to analyze a mix of
structured, semi-structured, and
unstructured data in search of
valuable information and insights.
Security Trends
Data Analytics for Intrusion Detection
• 1st generation: Intrusion detection systems
• 2nd generation: Security information and
event management (SIEM)
Limitation of Traditional SIEMs
Storing and retaining a large quantity of data was not economically
feasible.
Normalization & datastore schema reduces data
Traditional tools did not leverage Big Data technologies.
Closed platform with limited customization & integration options
Security Trend from Y2015 to Y2016
Fireeye M-Trends Report 2016
Security Trend from Y2015 to Y2016
• Threats are hard to investigate
Fireeye M-Trends Report 2016
All Data is Security Relevant = Big Data
Servers
Storage
Desktops Email Web
Transaction Records
Network Flows
DHCP/ DNS
Hypervisor Custom
Apps
Physical Access
Badges
Threat Intelligence
Mobile
CMDB
Intrusion
Detection
Firewall
Data Loss
Prevention
Anti-
Malware
Vulnerability
Scans
Traditional
Authentication
Data Analytics for Intrusion Detection
• 1st generation: Intrusion detection systems
• 2nd generation: Security information and
event management (SIEM)
• 3rd generation: Big Data analytics in
security (Next generation SIEM)
Example Security Attacks
Advanced Persistent Threats
• Advanced
• The attack can cope with traditional security solutions
• In many cases is based on Zero-day vulnerabilities
• Persistent
• Attack has a specific goal
• Remain on the system as long as the attack goal is not met.
• Threat
• Collect and steal information-Confidentiality.
• Make the victim's system unavailable-Availability.
• Modify the victim's system data-Integrity.
Example of Advanced Threat Activities
HTTP (web) session to
command & control
server
Remote control,
Steal data,
Persist in company,
Rent as botnet
WEB
Conduct Business
Create additional environment
Gain Access to system Transaction
.pdf executes & unpacks malware
overwriting and running “allowed” programs
Svchost.exe Calc.exe
Attacker hacks website
Steals .pdf files Web
Portal .pdf
Attacker creates
malware, embed in
.pdf,
Emails
to the target MAIL
Read email, open
attachment
Threat intelligence
Auth - User Roles
Host Activity/Security
Network Activity/Security
Link Events Together
Threat intelligence
Auth - User Roles, Corp Context
Host Activity/Security
Network Activity/Security
WEB
Conduct Business
Create additional environment
Gain Access to system Transaction
.pdf Svchost.exe Calc.exe
Events that
contain link to file
Proxy log
C2
communication
to blacklist
How was
process
started?
What created the
program/process
?
Process
making C2
traffic
Web
Portal .pdf
Correlated Security Log
Aug 08 06:09:13 acmesep01.acmetech.com Aug 09 06:17:24 SymantecServer acmesep01: Virus found,Computer
name: ACME-002,Source: Real Time Scan,Risk name: Hackertool.rootkit,Occurrences: 1,C:/Documents and
Settings/smithe/Local Settings/Temp/evil.tmp,"""",Actual action: Quarantined,Requested action: Cleaned, time:
2009-01-23 03:19:12,Inserted: 2009-01-23 03:20:12,End: 2009-01-23 03:19:12,Domain: Default,Group: My
Company\ACME Remote,Server: acmesep01,User: smithe,Source computer: ,Source IP: 10.11.36.20
Aug 08 08:26:54 snort.acmetech.com {TCP} 10.11.36.20:5072 -> 10.11.36.26:443 itsec snort[18774]:
[1:100000:3] [Classification: Potential Corporate Privacy Violation] Credit Card Number Detected in Clear
Text [Priority: 2]:
20130806041221.000000Caption=ACME-2975EB\Administrator Description=Built-in account for
administering the computer/domainDomain=ACME-2975EB InstallDate=NULLLocalAccount = IP: 10.11.36.20
TrueName=Administrator SID =S-1-5-21-1715567821-926492609-725345543 500SIDType=1
Status=Degradedwmi_ type=UserAccounts
Sources
All three occurring within a 24-hour period
Source IP
Data Loss
Default Admin Account
Malware Found
Time Range
Intrusion Detection
Endpoint Security
Windows Authentication
Source IP
Source IP
Incident Analysis & Investigation
Search historically - back in time Watch for new evidence
Related
evidence
from other
security
devices
Integrated Security Analytics with
Open Source
SQRRL Solution
https://sqrrl.com/
Anomaly detection in Visualizing
https://sqrrl.com/
Prelert Behavioral Analytics
for the Elastic Stack
http://info.prelert.com/
Prelert Behavioral Analytics
for the Elastic Stack
http://info.prelert.com/
How to Apply ?
Determining Data That Can Be Collected
Threat
intelligence
Auth - User Roles
Service
Host
Network
Network Security Through Data Analysisby Michael S
CollinsPublished by O'Reilly Media, Inc., 2014
• Third-party Threat Intel • Open source blacklist • Internal threat intelligence
• Firewall • IDS / IPS • Web Proxy • Vulnerability scanners
• VPNs • Netflow
• TCP Collector
• OS logs • Patching • File Integrity
• Endpoint (AV/IPS/FW) • Malware detection • Logins, Logouts log
• Active Directory • LDAP • AAA, SSO
• Application logs • Audit log
• Service / Process
Option 1 : Replace All Solution
• Data sent to new Big Data
Analytic Platform
• Big Data Analytic Platform
• Static Visualizations /
Reports
• Threat detection, alerts,
workflow, compliance
• Incident
investigations/forensics
• Non-security use cases
Big Data Analytic Platform
Raw data
Alerts Static
Visualizations
Forensics / Search
Interface
Option 2 : Big Data to Traditional SIEM
• Data sent to both system
• Big Data Analytic Platform
• Incident
investigations/forensics
• Non-security use cases
• Traditional SIEM
• Static Visualizations /
Reports
• Threat detection, alerts,
workflow, compliance
Big Data Analytic
Platform
Raw data
Forensics / Search
Interface
SIEM
Alerts Static
Visualizations
Co
nn
ecto
rs
Factors for evaluating
Big Data Security Analytics Platforms
Factors for Evaluating Open Source
• Scalable data ingestion HDFS
• Unified data management platform Cassandra / Accumulo
• Support for multiple data types Ready to Customized
• Real time Spark / Strom
• Security analytic tools No
• Compliance reporting No
• Easy to deploy and manage Manage many 3rd Party
• Flexible search, report and create
new correlation rule
No
