security analytics and big data: what you need to know

David MonahanResearch Director

EMA

Security Analytics and Big Data: What You Need to

Know

Sameer Nori

Senior Product Marketing

Manager

MapR

Nick Amato

Director Technical

Marketing

MapR

© 2015 MapR Technologies 2

Today’s Presenters

Slide 2

David Monahan, Research Director, Risk & Security Management, EMA

David has over 15 years of IT security experience and has organized and managed both physical and

information security programs, including Security and Network Operations (SOCs and NOCs) for

organizations ranging from Fortune 100 companies to local government and small public and private

companies.

Sameer Nori, Senior Product Marketing Manager, MapR Technologies

Sameer has over ten years of experience in the technology industry in marketing, pre-sales, and

consulting, with domain experience in business intelligence, analytics, and big data.

Nick Amato, Director, Technical Marketing, MapR Technologies

Nick works with the MapR ecosystem and technology partners to identify new opportunities where the

MapR platform can bring value to customers. His areas of focus include third-party integrations with BI

tools, benchmarking, architecture, and enabling scalable data platforms.

http://www.google.com/imgres?imgurl=http://nycapitalregion.acp-international.com/images/Citrix_logo.JPG&imgrefurl=http://nycapitalregion.acp-international.com/ACP-NY_Meeting_Archive.htm&h=245&w=607&sz=30&tbnid=2X29M7XfBVbVxM:&tbnh=55&tbnw=136&prev=/images?q=citrix+logo&zoom=1&q=citrix+logo&usg=__RNMRmug1-l0Sgn50b97jlcVzBo4=&sa=X&ei=26N2Td2_CfO10QGmpfHpBg&ved=0CEUQ9QEwBg

http://www.google.com/imgres?imgurl=http://nycapitalregion.acp-international.com/images/Citrix_logo.JPG&imgrefurl=http://nycapitalregion.acp-international.com/ACP-NY_Meeting_Archive.htm&h=245&w=607&sz=30&tbnid=2X29M7XfBVbVxM:&tbnh=55&tbnw=136&prev=/images?q=citrix+logo&zoom=1&q=citrix+logo&usg=__RNMRmug1-l0Sgn50b97jlcVzBo4=&sa=X&ei=26N2Td2_CfO10QGmpfHpBg&ved=0CEUQ9QEwBg


Logistics for Today’s Webinar

Slide 3 © 2015 Enterprise

A PDF of the PowerPoint

presentation will be available

An archived version of the event

recording will be available at

www.enterprisemanagement.com

• Log questions in the Q&A panel located on

the lower right corner of your screen

• Questions will be addressed during the

Q&A session of the event

Questions

Event recording

Event presentation

http://www.enterprisemanagement.com/

David Monahan

Research Director, Security and Risk Management

Enterprise Management Associates

http://www.enterprisemanagement.com

@SecurityMonahan

The Convergence of

Security Analytics and Big Data

April 27, 2015

http://www.enterprisemanagement.com/


Threats Come From Everywhere

• Hacking: The mentality has changed

• Data breaches affect every industry

• Organizations are being attacked from all sides

– External threats

– Insider threats

• All information is up for grabs



Identifying Threats is Harder Than Ever


EMA research identified several troubling statistics about

identifying and responding to threats:

of organizations were between “Highly Doubtful”

and only “Somewhat Confident” that they could

detect an important security issue before it had a

significant impact.

of organizations believe they

are consistently successful in

in correlating security data to

business impact.

of organizations said they

were unable to stop exploits

because of outdated or

insufficient threat intelligence.

69% 22%

60%

41%

28%

33%

29%

TOO DIFFICULT SEPARATING LEGITIMATEFROM MALICIOUS ACTIVITY

TOO DIFFICULT PRIORITIZINGREMEDIATION ACTIVITIES

INABILITY TO REPORT MEANINGFULINFORMATION TO STAKEHOLDERS

INSUFFICIENT TOOLING TOSUPPORT SECURITY DUTIES

Top frustrations with IT Security Practices:


The Problem Requires Better Data and Better Tools

• Data volumes are too high

– EMA research identified that 45% of organizations are collecting

more than 40GB/day of logs

– Nearly 16% are collecting over 500GB/day of logs

• Data correlation and normalization is not sufficient

– Organizations are fielding 100:1 high priority and greater alerts per

person in security

• Operations, Analysts, and Responders need better context

and Higher Fidelity (Ponemon Study)

– Actionable Intelligence within 60 seconds reduced breach resolution

costs by an average of 40%



The Problem Requires Better Data and Better Tools (cont’d)

• Persistent threats and their complexity is expanding rapidly

– Criminal organizations are creating new and better attacks

• [Gameover] Zeus (Botnet and data theft)

• Crypto-Locker/Wall, CTB-Locker (data theft)

• Dexter, POSLogr, BlackPOS (Point of Sale Terminal malware)

– The Nations states show criminals virtually anything is possible

• StuxNet malware (Supervisory Control and Data Acquisition (SCADA)

malware)

• Direct Memory Access Video RAM malware

• TAO- Micro processor embedded malware (network sniffing, key logging,

data collection, remote access, etc.)

• “nls_933w.dll”- Hard drive Firmware embedded malware (anything)



The Problem Requires Better Data and Better Tools (cont’d)

• EMA Research has identified key issues with current tools

Most Significant Frustrations with IT Security Technologies


38%

36%

35%

LACK OF INTEGRATION/INTEROPERABILITY

TOOLS UNABLE TO RECOGNIZE EMERGING THREATS/ATTACKS

VENDORS ARE SLOW TO RESPOND TO EMERGING THREATS OR ATTACKS


SIEM Limitations


SIEM technology provides real-time analysis of security alerts generated by network hardware and applications.

This is limited “analysis” based primarily upon

correlation and normalization of alerts.SIEM only understands deltas for those things inside of

its defined rules or policies

SIEM understands network information and log entries

to correlate events at a network level and identify

system/application alerts.

SIEM does not understand human, system, and

application specific activity and patterns (behaviors) to

determine how some activities raise the threat level.

Post notification SIEM often requires manual investigation*.

* EMA research found 55% of organizations said they still conduct

manual incident investigations


SIEM Limitations (cont’d)

What features is your organization not getting from SIEM tools that it is looking for in Security Analytics technology/products?


65%

53%

51%

ADVANCED AUTOMATED RESPONSE CAPABILITIES

INCREASED ABILITY TO EASILY AGGREGATE AND CROSS

ANALYZE DATA FROM NON-SECURITY SOURCES (IENETFLOW, WEB ACCESS LOGS)

ENHANCED DATA VISUALIZATION


Poll Question #1

Have you heard of Security Analytics or

Security Intelligence as a solution?

A. Have not heard of it

B. Believe they are the same as SIEM

C. Deployed a security analytics solution

D. Considering security analytics in the next 6-12 months



Moving to Security Analytics


Security Analytics Improvements

Better context and fidelity Reduce false positives

Reduce alert volumes Provide better prioritization

Accelerate Incident Response

of organizations using Security Analytics have

seen a reduction in false positives or an

improvement in actionable alerts since they

implemented a Security Analytics technology.

of organizations that use

Security Analytics said that the

tool produced expected or

greater than expected value.

90% 95%


Why Security Analytics

Which of the following are your organization’s views or reasons why it needs/uses capabilities for advanced analytics or security data management for IT/information

security?


53%

46%

43%

36%

IMPROVES DEFENSE AGAINST TARGETED THREATS

INCREASES OPERATIONAL EFFICIENCIES DEMONSTRATING HIGHER

SECURITY EFFECTIVENESS TO THE BUSINESS

IMPROVES PRODUCTIVITY/EFFICIENCY OF IT SECURITY EFFORTS

IMPROVES STRATEGIC DECISION MAKING


Why Hadoop for Security Analytics

• We need tools that can handle more data and a wider variety of data.

– When asked if they would collect more data or a wider variety of data if they

could, 66% of organizations said they would. (Only 10% said they would not.)

– EMA Research - 57% of organizations said that they expect the greatest

improvements in security through data analysis to come from innovations from

IT security technologies and their vendors.

– For true fidelity we need to be able to combine ALL information relevant to

data management.

• User, system, application, network packet/netflow, infrastructure logging, HR records,

endpoint, et. al.

• EMA Research - 32% of organizations indicated they wanted to be able to analyze

unstructured data for use in security.



Benefits of Hadoop for Security Analytics

• Purpose-built for processing large amounts of data

• Designed for unstructured data analysis

• Business Analytics can be applied to security use cases

• Increased ROI from a tool that supports both Business Intelligence and Security Operations


47%

36%

35%

35%

MACHINE LEARNING TOOLS

FRAUD MANAGEMENT OR DETECTION SYSTEM

BUSINESS INTELLIGENCE (BI) PLATFORM

ENTERPRISE DATA WAREHOUSES

Which of the following non-traditional data sources are currently NOT included/supported by your

organizations current SIEM or log management system?

© 2015 MapR Technologies 17© 2015 MapR Technologies

Security Log Analytics on MapR


Zions Bank: Security Analytics and Fraud DetectionCost effective security analytics and fraud detection on one platform

• Fraud Operations and Security Analytics team at Zions maintains data stores, builds

statistical models to detect fraud, and then uses these models to data mine and

evaluate suspicious activity

“We initially got into centralizing all of our data from an information security perspective. We then saw

that we could use this same environment to help with fraud detection”Michael Fowkes - SVP Fraud Operations and Security Analytics

• Existing technology infrastructure could not scale

• Timeliness of reports degraded over the last several years

• Chose MapR and cut storage costs by 50%

• Querying time reduced from 24 hours to 30 min on 1.2 PB of data

• Leverage MapR scale for increased model accuracy and deeper insights

OBJECTIVES

CHALLENGES

SOLUTION

Business Impact


Zions Bank with MapR – Faster Operations at Lower Costs

Web Server

Data

Transactional

Data

3rd Party Real Time

Fraud Detection

Reporting and

Batch Analytics

Deeper Analysis with

Machine Learning

PRD and Dev on

MapR

N

F

S

Technical Benefits

High availability

Multi-tenancy

Snapshots

Performance

Business Benefits

Unified platform for data

Lower operating costs

Operational guarantees

Faster model development


Solutionary: Managed Security Services ProviderThreat detection on real-time streaming data via platform as a service

• To address their growing customer base by processing trillions of messages (petabyte)

per year while continuing to provide reliable security services

• To improve data analytics by leveraging newer, more granular unstructured data

sources

”MapR has taken Apache Hadoop to a new level of performance and manageability. It integrates into

our systems seamlessly to help us boost the speed and capacity of data analytics for our clients.”

- Dave Caplinger, Director of Architecture, Solutionary

• Expanding existing database solution to meet demand was cost prohibitive

• The existing technology could not process unstructured data at scale

• Replaced RDBMS with MapR Enterprise Database Edition to scale Reduced time

needed to investigate security events for relevance and impact

• Improved data analytics, enabling new services and security analytics

• 2x faster performance compared to competing solutions

OBJECTIVES

CHALLENGES

SOLUTION

Business Impact

Leader in Magic Quadrant


Why MapR for Security Analytics

Business

• Large scale and deep analytics on security data to reduce risk

• Early detection of advanced persistent threats and unknown threats

• React fast on any abnormal or malicious activity from internal and external actors

• Avoid fines, lawsuits, loss of business and negative PR

Technical

• Build a data vault for security event logs from multiple sources

• With more data to scrutinize, get insights into anomalous behavior and close loop with other security solutions

• Platform that enables analysis of both historical data as well as real-time analysis of large volumes of security data

Operations

• Fast ingestion of large volume of data and perform deep analytics

• Easy integration with existing IT ecosystem

• Low overhead to maintain system

• Early detection of threats and closed loop feedback with existing security solutions


The MapR Advantage

• Scale Reliability Across the Enterprise– Advanced multi-tenancy

– Business continuity – HA, DR

• Speed– 2-7x faster than other Hadoop distributions

– Ultra-fast data ingest (100M data points per sec)

– NFS & R/W file system

• Real-time & Self-Service Data Exploration– On-the-fly SQL without up-front schema

– Fast lookups and queries

Best Hadoop Platform for Security Log Analytics

Security

Streaming

NoSQL & Search

Provisioning &

coordination

ML, Graph

W orkflow & Data Governance

Batch

SQL

INTEGRATED

COMMERCIAL

ENGINES

TOOLSCOMPUTE

ENGINES

Batch

Interactive

Real-time

Online

Others

Management

Operations

Governance

Audits

Security

MapR-FS MapR-DB

MapR Data Platform


Poll Question #2

Do you use Hadoop for Security Analytics?

A. No, didn’t know it could be used for Security Analytics.

B. Yes, it's been 6 months or less.

C. Yes, it’s been deployed for 12 months or more.

D. No, but considering it in the next 6-12 months.


What’s in the Quick Start Solution

6 nodes of

MapR software

2 week

engagement

3 Hadoop

Professional

Certifications


Quick Start Service Engagement

Engagement includes:

1. Identification of data sources, transformations and reporting engines

2. Access and use of the solution template including source code

3. Training on customizing the solution template to the organization’s requirement

4. Deployment architecture document that enables a production deployment plan for the specific solution

SOLUTION

TEMPLATE

KNOWLEDGE

TRANSFERDEPLOYMENT

ARCHITECTURE


Components of the Solution Template

• Data Workflows

– Read/collect input data

– Handle bulk load and streaming use cases

• Parsers and Enrichment

– Process input data (filtering and deriving additional data as needed)

– Storing in one or more data types or formats

• Machine learning

– Clustering analysis

– Reservoir sampling analysis

INTEGRATED

COMMERCIAL

ENGINES

TOOLSCOMPUTE

ENGINES

MapR Data Platform


The Power of the Open Source Community

APACHE HADOOP AND OSS ECOSYSTEM

Security

YARN

Spark Streaming

Storm

StreamingNoSQL & Search

Juju

Provisioning &

Coordination

Sahara

ML, Graph

Mahout

MLLib

GraphX

EXECUTION ENGINES DATA GOVERNANCE AND OPERATIONS

Workflow & Data

Governance

Pig

Cascading

Spark

Batch

MapReduce v1 & v2

Tez

HBase

Solr

Hive

Impala

Spark SQL

Drill

SQL

Sentry Oozie ZooKeeperSqoop

Flume

Data Integration& Access

HttpFS

Hue

Data PlatformMapR-FS MapR-DB

Manag

em

ent


MapR: Best Solution for Customer Success

Premier

InvestorsHigh Growth

2X Growth In Direct Customers

90%Subscription Licenses

Software Margins

140%

Dollar-based Net Expansion

700+ Customers

2X Growth In Annual

Subscriptions ( ACV)

Best Product

Apache Open

Source


Security Log Analytics Template

MapR-FS

MapR-DB


Resources

https://www.mapr.com/solutions/quickstart/hadoop

-security-log-analytics-quick-start

– Research Report: The Evolution of Data Driven

Security

– Solution Brief: Jump-Start Security Log Analytics

https://www.mapr.com/solutions/quickstart/hadoop-security-log-analytics-quick-start


Free on-demand

Hadoop training leading to certification

Start becoming an expert now

mapr.com/training

50MIn Free Training


Q & A

@mapr maprtech

[email protected]

Engage with us!

MapR

maprtech

mapr-technologies