smart data cloud security alliance

8/12/2019 Smart Data Cloud Security Alliance

1/49

Big Data, Smart

Dumb Data and

Intelligence

Fred Wilmot,

Global Security Practic


2/49

Agenda

Simple Big Data

A Child Becomes a Teenager

Can Data Science Solve Business Problems

Visibility and Insight

Visibility and Insight, Context and Analytics

A Use Case for Smart Data

2


3/49

What is Big Data Really?

* Mike LoukidesOReilly Radar

When the size of the data its

becomes part of the problem

3


4/49

Big Data, Get Used To IT

4


5/49

Its Not Just About the Bigness

Data

Diversity

Data

VolumeTraditTools

Desiredto Answ

5


6/49

Big Data Comes from Machine

6

Volume | Velocity | Variety | Variability

W

Email

ClickstreaTelephony, IVR

Sensors, Telemat

Servers, Security Device

Machinedatais fastest growing, mostcomplex, most valuable area of big data


7/49

Distributed File System

(semi-structured)

Key/Value, Columnar or

Other (semi-structured)

Relational Database

(highly structured)

Map / Reduce

TeraDataGreenplum

CassandraCouchDBMongoDB

Redis

Big Data Technologies

SQL &

Map / Reduce

NoSQL

Tem

Hadoop

RDBMS

Sharding

HDFS Storage +

Map / Reduce

Rea

7

Unstru


8/49

A Child Becom


9/49

Cute Kid to Frustrating Teenage

9

Data Science is more than storing data in HDFS, NoSQ

cloud offering. its getting value, insight and analysis


10/49

Traughs

10


11/49

Data Science and Statistics

11

The sexy job in the next 10 years will be

statisticians

Hal Varian, Chief Economist at Google

Data scientists are the next-generation analytics

professional; they are responsible for turningthe data into insight. They extract meaning

from Big Data to help the business.


12/49

Big Data Roles and Their Challen

12

DData ScientistData Analyst Data Architect

Conduct analytics onstructured data with

traditional tools

Guide designs,set standards, and

manage developers

Extract meaning frombig data to help the

business

Build scbased o

platform

Accessing and using

data in Hadoop

Keeping pace with

evolving ecosystem

Low-level tools and low

productivity

Dev foc

and low


13/49

What About the Business Proble

13

Data

Scientarchilystoper

Security professionals need ALL thes


14/49

Required - A Broader Look at Securit

14

Increasing Threat Complexity

Cloud

SecurityData

Mobile

Security Data

Industrial Control Systems data (SCADA)

The Internet of Things

Increas

ingDataAmounts

Traditional IT

Security

Data

The


15/49

Big Data Analysis Demands

Data Integration

Advanced AnalyticsAlgorithmic Engine

Knowledge Management

Collaboration Platform

Data Driven Thinking

Supporting Context and insight

15


16/49

Big Data Security Requiremen

Now, the hard part for Business:

Data Experts to train expert systems

do not grow on trees

Capacity and scale for consumption

costs money, for unproven analytics

Adding responsibility to an under-

resourced workforce limits impact

How do I visualize insight?

16

You need the w


17/49


18/49

Delivering Visibility but What about

18

Big Data + Analytics = NOT ENO


19/49

Visibility with Context

Big Data + Analytics + Contextual Insight = Sm

19

Product

Analytics

Customer

Analytics

Se

An

Understand the relationship of

product feature effectivenessto trend line of product

success

Native mobile app feature

adoption and engagementby social class using all

handsets

Top malic

domains vcompany a

campaign,

user cla


20/49

Show Me

Lets Look at Low Value High Volume D

20


21/49

A DNS Bot net Example

21


22/49

22

Dynamic DNS Fast Flux


23/49

Bot net Behavior

Common Challenges

Better awareness of ecosystems (how large? Geographic understanding

C2 servers, signatures and attribution)

take down services (identify and degrade hostile botnets prior to an atta

enforcement, law enforcement/agreements to stop attacks.)

proactive ISP assistance (ASNs, router/flow data)

Full view for geographic perspective, what controls, IPs, protocols

more visibility into global actors - capabilities, weapons of choice, etc.

Sharing threat intelligence built into multiple vendor products across oth

Real-time and proactive DDoS forecasting, behavioral modeling with his

Deep Technical Expertise

23

h ld b ?


24/49

What Would Scooby Do?

How do I know its bad? Where do I start? Thats a lot of dat

We have some data analysis needs to be met DNS data (50,000 records/sec for the entire day) Proxy Data (200,000 unique URL requests for entire day)

Network Flow data, maybe deep packet inspection (easily in excess

We also need to add some context? Watchlist activity/Threat intelligence

System type/Application

Frequency of communication

Locality of request

Risk model

24

S l i


25/49

DNS Analytics

25


26/49

A l ti C t t


27/49

Analytics + Context

27

All Heuristic Bot Net DNS run

through Threat Intelligence

watchlists for matches

Th t I t lli


28/49

Threat Intelligence

Common Challenges

Sharing threat intelligence across multiple vendors

Collaborating on Internal Threat Intelligence across the business

Adding context to real-time analysis in an automated fashion for dispara

Associating threat profiles based on locality, frequency, affinity and com

business risk

Curating Threat Intelligence over time

Automation, and iteration as part of a mature Security Operations mode

28

E t l Th t I t lli I t


29/49

External Threat Intelligence Integr

import httplib, urllib

params = urllib.urlencode({'apikey':'f6dbdee2dc8c6118933b90178657877cc2cede3023ce0eba4xxxxxxxxxxxxxxx', 'ip': 46.229'ipq'})

headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "application

conn = httplib.HTTPConnection("us.api.ipviking.com:80")

conn.request("POST", "/api/", params, headers)

response = conn.getresponse()

print response.status, response.reasondata = response.read()

print(data)

conn.close()

29

I i ht F th Wi !


30/49

Insight For the Win!

30

Server DNS associations with othe

systems performing their Function

Reducing the TTL for these DNS entries will help prevent

attack from Syria now that we know what causes that

Lookup Table of Largest DNS Request by

Country, and geolocation


31/49

Augment Big Data with Smart Data


32/49

Augment Big Data with Smart Data

Extend context with lookups and external data sources.

32

CMDB

Reference

Lookups

Messag

e

Stores

LDAP, AD Watch

Correlate across multiple data sources and data sets using indexes

Drive Insight through Collaborat


33/49

Drive Insight through Collaborat

33

1. Start with Security Ops model

2. Big Data + Internal context data

3. Add Threat Intelligence data

4. Collaboration/Communication

4. Automation

5. Iteration

* Excerpt - sng o a a ases or reamng e wor nayss

i li , i l l , i I ,

, I

Cyber Operations Model


34/49

Cyber Operations Model

34

Compliance

Reporting and

Audit

Forensics and

Root Cause

Analysis

Secure Coding &

Development

Monitor

Security

Technologies

Incident

Response

Process

Incident

Handling Se

Fraud and

Theft Analysis

Behavioral

Analysis

Vulnerability

Remediation

Malware

Analysis

Threat

IntelligenceCyber Network

Defense

Cyber Network

Offense

Collaboration

Iteration

Automation

REACTIVE PROACTIVE

I d i l D E l i


35/49

MACHINEDATA

Industrial Data Explosion

The NEXT WAVE

I

D

IN

STRUCTURED

DATA

M di l D i D i i B tt P ti t


36/49

TrDe

ChCr

Medical Devices Driving Better Patient

Device

Manufactured

Shippedto Physician

Prescribedto patient

Returnedto iRhythm

P

B

PP

S

A


37/49

37

Enhance Efficiencies

and Reduce Costs

Develop Deep

Understanding of

Building

Analyze BSensors to

Energy Co

Operational InteLeads to More EBetter PerformiBuildings

Capture Energy,

Environmental, and

Operational data

A t D t f


38/49

Telemetry

Vehicles Acceleration, Braking,

Battery Charge and Location

Aggregate Data from

Vehicles Remotely

Shape Next-gen Electric Vehicles

Optimize Charging InfrastructureE

of charging and

charging locations

Frequency

Manageimpact

on the power gridthe

into customers

driving habits

Insights


39/49

Thank You!

Fred@Splunk.

Further Reading


40/49

Further Reading

OpenDNSDynamic DNS Fast Flux

DNS

Botnetshttp://www.elsevierdirect.com/companions/9781597491358/

NS.pdf

http://www.syssec-project.eu/media/page-media/3/dietrich-ec2nd11

40

Hybridization
http://www.elsevierdirect.com/companions/9781597491358/casestudies/DNS.pdfhttp://www.elsevierdirect.com/companions/9781597491358/casestudies/DNS.pdfhttp://www.syssec-project.eu/media/page-media/3/dietrich-ec2nd11.pdfhttp://www.syssec-project.eu/media/page-media/3/dietrich-ec2nd11.pdfhttp://www.syssec-project.eu/media/page-media/3/dietrich-ec2nd11.pdfhttp://www.syssec-project.eu/media/page-media/3/dietrich-ec2nd11.pdfhttp://www.syssec-project.eu/media/page-media/3/dietrich-ec2nd11.pdfhttp://www.syssec-project.eu/media/page-media/3/dietrich-ec2nd11.pdfhttp://www.syssec-project.eu/media/page-media/3/dietrich-ec2nd11.pdfhttp://www.syssec-project.eu/media/page-media/3/dietrich-ec2nd11.pdfhttp://www.elsevierdirect.com/companions/9781597491358/casestudies/DNS.pdfhttp://www.elsevierdirect.com/companions/9781597491358/casestudies/DNS.pdfhttp://www.elsevierdirect.com/companions/9781597491358/casestudies/DNS.pdf


41/49

Hybridization

41

ERP CRMWebLogsTimeSeries Files Social

Relation

Enter

DW

Real-time

HDFS

HBaseMapReduce

Hive

Data Services (REST, WS)

Pig

ETL

Internal apps,

customer-facingapps, mobile apps Analys(SAS, S

Tabl

Mobile Methodology


42/49

Web Ap

gy

Network

Client Application

Static Analysis

Dynamic Analysis

Static

Dynam

A Mobile App RE into json


43/49

A Mobile App RE into .json

{"tags": {"UTIL": ["Lcom/jumptap/adtag/actions/BrowserAdAction;","Lcom/inmobi/androidsdk/ai/container/IMWebView$2;", "Lcom/flurry/an

"Lcom/jumptap/adtag/media/JTMediaPlayer;","Lcom/jumptap/adtag/media/JtVideoAdView$3;","Lcom/jumptap/adtag/media/JtVideoAdView$4;","Lcom/inmobi/androidsdk/IMAdInterstitial$1$1;","Lcom/jumptap/adtag/media/JtVideoAdView;","Lcom/rovio/ka3d/GLSurfaceView$DefaultContextFactory;","Lcom/millennialmedia/android/MillennialMediaView;","Lcom/jumptap/adtag/utils/JtAdFetcher;", "Lcom/jumptap/adtag/utils/JtA

"Lcom/burstly/lib/component/networkcomponent/burstly/ormma/Ormmaoller;", "Lcom/inmobi/androidsdk/ai/container/IMWebView$TimeOut;","Lcom/burstly/lib/component/networkcomponent/burstly/ormma/Ormmaller;", "Lcom/jumptap/adtag/utils/JtSettingsParameters;", "Lcom/flurry/an"Lcom/inmobi/androidsdk/ai/controller/JSAssetController;","Lcom/jumptap/adtag/JtAdView;","Lcom/millennialmedia/android/BasicMMAdListener;", "Lcom/google/ads"Lcom/millennialmedia/android/HandShake$AdTypeHandShake ;

Strings and Androguard


44/49

Strings and Androguard

Splunk Shows Malicious Apps by Be


45/49

Splunk Shows Malicious Apps by Be

Lookup Using Key Value Persistent Cac


46/49

p g y

46

Download and install Redis

Download and install Redis Python module Import Redis module in Python and populate

key value DB

Import Redis module in lookup function

given to Splunk to lookup a value given a key

Redis i

source

value s

Redis Lookup


47/49

p

47

###CHANGE PATH According to your REDIS install ######

sys.path.append(/Library/Python/2.6//redis-2.4.5-py.egg)

import redis

def main()

#Connect to redisChange for your distribution

pool = redis.ConnectionPool(host=localhost,port=6379,db=0)

redp = redis.Redis(connection_pool=pool)

Redis Lookup (cont.)


48/49

48

def lookup(redp, mykey):

try:return redp.get(mykey)

except:

return

Combine Persistent Cache with External L


49/49

49

For data that is relatively static

First see if the data is in the persistent cache

If not, look it up in the external source such as a dat

web service

If results come back, add results to the persistent ca

return results

For data that changes often, you will need to create yo

retention policies

smart data cloud security alliance

Documents