filar seymour oreilly_bot_story_

Security + Design * Data Science: A Bot Story

Bobby Filar & Rich Seymour

O’Reilly Security 2017

October 31, 2017

About Us

• Data Scientist

• Background in NLP

• @filar

• Data Scientist

• Background in HPC

• @rseymour

Bobby Filar Rich Seymour

• Endpoint Protection Platform

• ML/Domain Expertise

• @EndgameInc

• endgame.com

Endgame

State of Security Software

Detect Review Analyze Identify Notify Collect Validate React

Dwell Time Containment Time

Reduce Dwell Time using domain knowledge + ML

State of Security Software

Detect Review Analyze Identify Notify Collect Validate React

Dwell Time Containment Time

Reduce Dwell Time using domain knowledge + ML

• User and Entity Behavior Analytics (UEBA)

• Detection/Prevention

• Network Monitoring

State of Security Software

Detect Review Analyze Identify Notify Collect Validate React

Dwell Time Containment Time

Containment Time depends on real-time data availability and intuitivenessof the user interface

Opportunities to

Expand ML

Solutions

Reducing Containment Time Via ML is

Difficult!

• How do you optimize a human?

• How do you handle a diverse user base?

• Where do you find quality data?

• How can we optimize workflows?

User-Centric Design Study

Design Process

• Discovery• Understanding our users,

confirming/disproving biases, capturing

organizational workflows

• Concepting• Creating design requirements solutions

• Prototyping and User Testing• Feature creation and taking it back into the

‘wild’ for testing

Avoid Our Bias

Find What Users Need

Why Do This?

Objective Capture team dynamics and worker roles within security organization to identify challenges common across security teams

User Group Team Type Environment Collection Method

A Traditional SOC Day-to-day use User interviews

B Novice Training Team Mock Scenario Side-by-side

monitoring,

Retrospective & User

interviews

C Internal Red vs. Blue Mock Scenario Mirrored Scenario as

User Group B

D Traditional SOC & Consulting

group

Day-to-day use User testing

How our customers see it…

Insufficient Resources

• Onboarding & training

new hires & retention

• Limited time to review

alerts and incidents

Lack of easy-to-use

automated tools

• Difficult for non-

programmers to use

• Easy for programmers

to mess up!

Security platforms are just

difficult to use!

• Forces conformity

• Requires level of expertise

to extract value

Let’s meet the users

Tier 1

AnalystTier 3

Analyst

Forensic

Hunter

SOC

Manager

”Pretty much any alert or call here

goes through me first. It can be a

lot to handle."

Tier 1

Analyst

"I’m doing the hard work here,

also I could really use a Tier 2."

Tier 3

Analyst

"Don’t call me unless the server

room is on fire, I’m working on my

new radare2 theme."

Forensic

Hunter

"I have a meeting with the CISO,

then 3 one-on-ones and a report

due, no time for a quote."

SOC

Manager

Findings: Day in Life of a Tier I

Data

Deluge

Lack of

Context

Repetitive

Processes

Searching not

Analyzing

Lack of

Expertise

Lack of

Time

Findings: Pain Points

• Lacks context• Is it actually bad?• Is it anywhere else?• Did it talk to the network?

• Lacks connectivity• Is this alert tied to any others?

• Pivot on single IOC• Hash• Filename• IP address

Alert Type: Suspicious Binary

Alert Created: Feb 11, 2017

Severity: High

Confidence: 73%

File Path: C:\Temp\malware.exe

File Size: 45700

MD5: 5d41402abc4b2a76b9719d911017c592

File Created: Feb 11, 2017

What do you do when there are 100s of these each day?

Diverse Data Sources

Process File Users Network Registry

Solution

Make a Bot!

Hello, how can I help you?

You Can Start Making a Bot Today!

• Emphasize chatOps withHubot, Chattie, etc

• If you can leverage outside APIs!

• Define what you want to automate

• The bot transforms text into functions with arguments

search(process=‘fish’)

garbage input

usable output

scrubscrubscrub

But…

Solution

• An intelligent assistant can assist in the automation of tasks

• Guides user through complex tasks• Mimics human conversation• Natural Language Understanding

determines user intent

• Imagined an assistant that provides ability to:

• Ask questions• Execute workflows• Educate users• Recommend next steps

Hello, how can I help you?

Found the right tools (talk to us after for how)

spaCy

CRFsuite

From Research to Reality

Building an Intelligent Assistant From Scratch

Requirement #1

Eliminate query syntax via natural language

Natural Language vs. Query Language

Query LanguageSELECT * FROM TABLE process_event WHERE process_name == “odinaff.exe”;

Natural Language vs. Query Language

Query LanguageSELECT * FROM TABLE process_event WHERE process_name == “odinaff.exe”;

Natural LanguageSearch process event data for process odinaff.exe

Natural Language vs. Query Language

Query LanguageSELECT * FROM TABLE process_event WHERE process_name == “odinaff.exe”;

Natural LanguageSearch process event data for process odinaff.exe

RealityDid odinaff.exe run on any endpoints?

Interaction Types

Turn-based Conversation

User: Search processes

Artemis: Okay. Please provide a hash or filename

User: odinaff.exe

Artemis: Got it. Which endpoints would you like to target?

User: Windows 10 machines.

Artemis: Okay. Would you like to launch this search?

User: Yes

Artemis: Searching process data for odinaff.exe on

Windows10 endpoints.

Interaction Types

Turn-based Conversation

User: Search processes

Artemis: Okay. Please provide a hash or filename

User: odinaff.exe

Artemis: Got it. Which endpoints would you like to target?

User: Windows 10 machines.

Artemis: Okay. Would you like to launch this search?

User: Yes

Artemis: Searching process data for odinaff.exe on

Windows10 endpoints.

Resolve

Confirm

Handle

Interaction Types

Goal-oriented Conversation

User: Show me process event data for odinaff.exe

on all Windows 10 endpoints

Artemis: Are you sure?

User: Yes!

Interaction Types

API-Driven Investigations

curl 'api/v1/event_search' -H "Content-Type: application/json" -H

'authorization: <api_key> --data-binary '{"intent":"search_process",

"parameters": {

"process_name":"odinaff.exe",

"filepath": "C:\\Temp\\*.exe"

}

}'

Back to the Users!Requirement #2

Provide Context-Driven Alert Triage

•Understand what a user is currently viewing

• Ingest metadata in current view

•Map nested fields to common nomenclature

•Access indicators by saying “this <field>”

ContextSharing

Endpoint data

Endpoint data

Process data

Endpoint data

Process data

User data

“Search process data for this MD5 on all endpoints”

“Was this domain requested on any other endpoint?”

“Has user admin logged into this endpoint?”

Requirement #3

Educate Users on Platform Features

• Assist in feature discovery

• Reduce need to consult docs

• Recommend best practices

• Focus analysts in current view• Avoid copy/paste operations

WhisperText

Did it run anywhere else?

Where else is the file?

Has anyone else seen it in the wild?

Has it sent any data out of my

network?

Requirement #4

Recommend Next Steps

Short-term Memory

• Avoid defeated users

• Disambiguate user input

• Eliminate repetitive tasks

• Guide users through recommended actions using playbooks

search_network

search_network

search_networ

ksearch_proces

s

unknown

search_networ

k

unknown

unknown

“It looks like you’re trying to…”

“Would you like to run that

same query on this data?”

Repetition

Avoid Frustration

Long-Term MemoryCurrent User

Location Action

Alert List Accessed

Malware Alert

Malware Alert Open Artemis???

… …

Long-Term MemoryCurrent User

Location Action

Alert List Accessed

Malware Alert

Malware Alert Open Artemis???

… …

Location Action

Process Inject. Search process

Results view Kill process

Endpoint List Search DNS

Results View Search process

Results View Kill Process

Most Common Actions

Across Organization

Alert List Accessed

Malware Alert

Malware Alert Search Process

Results View Search Network

Results View Grab Memory

Long-Term MemoryCurrent User

Location Action

Alert List Accessed

Malware Alert

Malware Alert Open Artemis???

… …

Location Action

Process Inject. Search process

Results view Kill process

Endpoint List Search DNS

Results View Search process

Results View Kill Process

Most Common Actions

Across Organization

85% Of analysts perform

this sequence

Alert List Accessed

Malware Alert

Malware Alert Search Process

Results View Search Network

Results View Grab Memory

Long-Term MemoryCurrent User

Location Action

Alert List Accessed

Malware Alert

Malware Alert Open Artemis???

… …

Location Action

Process Inject. Search process

Results view Kill process

Endpoint List Search DNS

Results View Search process

Results View Kill Process

Most Common Actions

Across Organization

85% Of analysts perform

this sequence“We recommend the

following actions to triage

this alert”

Alert List Accessed

Malware Alert

Malware Alert Search Process

Results View Search Network

Results View Grab Memory

Requirement #5

Expedite Focused Collection

Structured Workflows & Automation

Domain expertise + Workflow *Automation• Use playbooks

• Endgame provides some built-in• Alert Remediation

• Quick Responses

• Multi-step analysis

• Orgs should be empowered to add their own

Make Security Software Work for You!

PowerShell Misuse Playbook

Traditional Investigation

1.Narrow scope to limited endpoints

2.Understand adversary TTPs

3.Gather events from limited endpoints

4.Analyze events from for signs of TTPs

5.Discover suspicious activity

6.Decode obfuscated commands

7.Pinpoint PowerShell activity

8.Expand scope to next set of endpoints

9.Repeat…

Endgame Artemis

“Find powershell activity”

Automatically discovers and

analyzes malicious activity across

your endpoints in minutes

Back to the Users!

“I want easy access to response actions”

“I want better insight into what my team is working on”

“I want to share new IOCs with my teammates”

“I still want an option for a structured query language”

How our customers NOW see it…

Better Use of Resources

• Helps with onboarding

& training new hires

• Decreases time to

review alerts and

incidents

Automation via NLP

• Easy for non-

programmers to use

• Exposes response

actions via API

Platform that works for you

• Encourages

personalization

• Reduces level of expertise

to extract value

ClosingThoughts

• Data science can be applied to more than just good/bad

• User-centric studies help capture diverse team environments

• Intelligent assistants can help alleviate rigid interfaces

Questions?Bobby Filar

@filar

bfilar@endgame.com

Rich Seymour@rseymour

rseymour @endgame.com