© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Randy Young, Splunk

Scott Pack, Adobe

November 29, 2016

SAC309

You Can’t Protect

What You Can’t SeeAWS Security Monitoring & Compliance Validation

What to expect from the session

•Learn how to automate data collection for security

monitoring and validate compliance for large numbers of

AWS accounts.

•Learn how Splunk & the Splunk App for AWS can enable

you to managing your AWS environment.

Presenters

• Scott Pack• Security Engineer @ Adobe

• SLC, UT

• 2 Year AWS User

• 4 Year Splunker

• Proudly DQd at 3 Pinewood Derbies

• Randy Young• Principal Product Manager @ Splunk

• Bezerkly, CA

• 8 Year AWS User

• 3 ½ Years a Splunker

• Proud Dubs Season Ticket Holder

R

The background

Digital Marketing

~55k physical hosts across 30 sites

Collection of ~20 admin teams.

• Different tech stacks, but mostly *nix

Monitoring Toolset:

• Netflow, FPC, IDS, Network Transaction

S

Security monitoring5

Security Engineering:

• Build & Maintain Monitoring Toolset

• Define (w/ SOC) “Security Notables”

• Work with Internal Audit to gauge compliance

Security Operations:

• Event Analysis

• “Hunting”

• Investigation

• Incident Response

S

What is Splunk?

Platform for Machine Data

Correlation &

EnrichmentField

Extraction

Reporting & Alerting

Data Collection &

Field Extraction

Multiple use cases across one platform

R

What can Splunk do for your AWS environment?

7

Splunk App for AWSEC2

EMR

Amazon

Kinesis

Route 53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Amazon

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

R

Shift to the cloud8

Lots of accounts … > 200

Dozens of teams, thousands of instances

Missing data to:

• Detect/respond to incidents

• Making assurances to Compliance

We received a mandate: Fix this

• Get whatever visibility you can

• Minimize risk of operations impact

• Be cost sensitive

S

AWS security incidents9

1. Infrastructure ImpactBaddie impacts the infrastructure as

an external user (DDOS)

2. Host CompromiseBaddie has some control of a host.

(Command Injection)

3. Account CompromiseBaddie interacts as an authenticated

AWS user. (Account Takeover)

S

Initiative goals

Identify & collect security relevant data

Analysis the same as on-premises

Data -> Splunk ES -> SOC

Minimize operations impact

Limit IAM users

No risk to services

Quick setup

10

S

Data sources

S

AWS native sources11/30/201612

CloudTrailAPI Usage &

Logging

VPC Flow LogsVirtual Interface

Connectivity

AWS ConfigAccount Configuration &

Inventory

ELB Access

LogsLoad Balancer

Logging

Trusted AdvisorSecurity Practice Checks

Identity & Access

ManagementCredential Report

R

Data examples13

CloudTrail

VPC Flow Logs

ELB Access Logs

Config Credential Report

R

Cross-account authentication14

IAM users• Use API Keys directly

Roles• AWS Security Token Service

• Can be “assumed” by a specified principal• Authenticate to an aggregation account user

• Assume the cross-account role

• Retrieve temporary access keys

• Make calls with temporary keys

Tutorial: Delegating Access using IAM Roles - http://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html

Shon Sha re:Invent 2014 - https://www.youtube.com/watch?v=0zJuULHFS6A

S

A few more AWS services15

S3 –

File/Object

Storage

Lambda – Code

without

Instances

Amazon

Kinesis – Data

Streaming

CloudWatch

Logs

SNS –

Notification

Service

DynamoDB –

NoSQL Database

S

Collection plumbing: S3

S3 Buckets:• ELB (1 per region)

• Permit PutObject from ELB IAM Roles

• Config

• Permit PutObject from config.amazonaws.com

• Config Parsed

• CloudTrail

• Permit PutObject from cloudtrail.amazonaws.com

• Trusted Advisor Results

• Permit PutObject from Lambda execution IAM role

11/30/201616

AWS ELB Account IDs for Log Delivery: http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-access-logs.html#attach-bucket-policy

S

Collection plumbing: VPC flows

Amazon Kinesis stream:

• 1 per region

CloudWatch log destinations

• 1 per region

• Directs to region-local Amazon Kinesis stream

17

S

18

Aggregation

18

CloudTrailVPC Flow

LogsConfig

ELB Access

Logs

Trusted

AdvisorIAM

Amazon

S3

Per Region

CloudWatch

Per Region

CloudWatch

Destination

Monitored

Account

Aggregation

Account

S

Registration

S

20

CloudFormation

Resources:

Config Role

FlowLogs Role

SecEng Role

SNS

Notification

Role

’s

Don

e!

Inputs:

Description

Jira Queue

Registration

LambdaRegistration

DynamoDB

Monitoring registration

S

Registration through web UI11/30/201621

S

22Scheduled delivery

enforcement

Distributor Handler

Config

STS

Config

Handler

IAM

Credential Report

STS

Distributor

CloudWatch CloudWatch

Scheduled retrieval & storage

S

Dashboards & analysis

S

Splunk apps & add-ons

• Input Methods: S3

• Input Sourcetypes: CloudTrail, VPC Flows, ELB Access Logs

• Parsing Handler: GZIPMessageHandler

11/30/201624

Aggregation reduces amount of Splunk inputs: 26 Total Inputs

• S3: 14

• Amazon Kinesis Inputs: 10

• Additional Logging: 2

Currently running on a dedicated Heavy Forwarder.

• If needed, split regions to different forwarders.

S

Sourcetypes, lookups, and other fun25

Sourcetypes: Cheated off the Splunk App for AWS.

• Set JSON KV format and check line-breaks

Use HTTP Event Collector FOR DynamoDB Registrations

• Scheduled lookup-generating search

• Auto lookups on each sourcetype

Tagging into Enterprise Security data models

• ELB Access Logs & VPC Flow Logs right out of the box

S

Onboarding dashboard26

S

Account overview

S

Compliance checks

Inspect Config + Credential Reports

+ Bunches more

Query per Standard/Compliance Requirement

S

Resource lookup

S

Example ES correlation rules30

• Console logins from outside org IP space

• Flows to/from threat actors

• Instance increase by X% within 24-hours

• AMI sharing to non-org AWS account

• URI/user agent web application attacks

• Multiple service API denies for 1 API key within X mins

• (Nimbostratus – Andres Riancho, BlackHat 2014)

S

Things that can go wrong:

S

Splunk hints32

Amazon Kinesis Modular Input*

• Can chew up memory.

• /opt/splunk/etc/apps/kinesis_ta/bin

java_args = [ JAVA_EXECUTABLE, "-classpath",CLASSPATH,"-

Xms512m","-Xmx512m",

"-

Dsplunk.securetransport.protocol="+SECURE_TRANSPORT,JAVA_MAIN

_CLASS]

Config snapshots are jsonormous

• Use Lambda to split out the resources.

* You can now use the Splunk TA for Kinesis InputsS

AWS hints

ELB permission granularity restrictions

• ModifyAttributes

Keep an eye on capacity. Watch:

• DynamoDB read capacity

• Amazon Kinesis shard usage

AWS internal actions

• Auto Scaling

• EMR

S

Where we’re at right now

• 57 AWS accounts currently enrolled

• ~3 TB/day

• Haven’t broken any accounts yet!

• Finding more data sources• Config Rules

• Amazon Inspector

• Automating our AWS security policy audit

• Written a handful of Splunk Enterprise correlation rules

• Actioned by SOC

• Automated Jira ticketing for remediation

11/30/201634

S

Make machine data accessible,

actionable and valuable to everyone.

35R

Splunk and AWS – Customer value

36

“Customers love the agility of AWS together with the end-to-end

visibility of Splunk.” Andy Jassy, AWS CEO

R

Operational Intelligence Security Intelligence- Etc.

AWS data leveraged across multiple use cases

Financial Intelligence

R

Operations Intelligence- What is my EBS footprint and posture

across all my accounts and all my

regions?

- Who started/stopped/restarted what

instances and when?

- What EC2 instances are underutilized

and perhaps overprovisioned?

- What is the traffic volume into my VPC

and where is it originating from?

- Why are certain resources unreachable

from certain subnets/VPCs?

- List resources with missing or non-

conforming tags?

- Etc.

Security Intelligence- Who added that rule in the security

group that protects our application

servers?

- Where is the blocked traffic into that

VPC coming from?

- What was the activity trail of a

particular user before and after that

incident?

- Alert me when a user imports key

pairs or when a security group

allows all ports

- What instances are provisioned

outside of a VPC, by whom and

when?

- What security groups are defined but

not attached to ay resource?

- Etc.

- Etc.

Sample use cases for AWS dataFinancial Intelligence

- How many instances are you

running?

- What Reserved Instances have you

purchased in the past?

- What is your Reserved Instance

utilization?

- How much are you paying per

account?

- How much are you using per service

across all accounts?

- How many Reserved Instances

should I buy based on usage?

- Is this account within budget this

month, and how have they tracked in

the last year?

- Etc.

R

Now you have all this data… what do you do with it?

HR Director: Good afternoon…

You: (smile nervously)

HR Director: Joe was let go today. Can you close his

account. I want to get an email if his account does anything

strange this weekend.

You: (nod) And create an alert.

R

sourcetype=aws:cloudtrail userIdentity.userName=joe|table _time event* user*

Save as alert > Email action

R

Now you have all this data… what do you do with it?

CFO: Good Afternoon…

You: (smile nervously)

CFO: Our production account’s spending is on track, but I need YOU to cut our development account spend by 1/3.

You: No problem!

R

AWS tag-based instance auto start/stop

43

Weekends

Non-Working Hours

1. Create IAM user ‘robot’

2. Install AWS CLI on splunk host

3. Define tag: PowerSave=LongRun/

RareRun/Normal on each instances

4. Create splunk alert

• CRON, run in morning/night

• SPL to search instances by tag

• Alert action to call AWS CLI to

batch start/stop instances

And save 40%

Development cost!

R

Now you have all this data… what do you do with it?

Developer: I am going to cut out early.

By the way, I ran a script and created a bunch of

untagged EC2 instances.

Can you help me find them?

Have a great weekend!

You: What the #*$%!

R

Tag AWS resource properly

Find untagged EC2 instances

• sourcetype=aws:description source="*:ec2_instances" NOT "tags.Name"=*| table

region id instance_type ip_address key_name

Define a naming conventions for EC2 instance and enforce it

• DLA_Jove_testEC2Cmd. D: Dev, L: Linux, A: AWS project

• <Role><OS><Project>_<Owner><Note>

• sourcetype=aws:description source="*:ec2_instances" (NOT "tags.Name"=*) OR

("tags.Name"=* tags.Name!=Q* tags.Name!=D* tags.Name!=P* tags.Name!=U*)

R

Just use the “Name” tag

4

6

R

48

Splunk app for AWS

demo

R

Splunk runs on and with AWS

SOC2 Type II Certified

Cloud Services Apps

Splunk Add-on for AWS

Splunk App for AWS

Specific

Integrations

Config, CloudTrail, CloudWatch,

VPC Flow Logs, Lambda: AWS IoT,

Amazon Kinesis: AWS

CloudFormation

Splunk Core + Enterprise

Security & ITSI available

Enterprise on AWS

For small IT teams, starts $3/day

Software

Apps and Integrations

As a Service on AWS

Delivery Models

For small IT teams, starts $75/mo

R

Launched: Splunk Light w/ app for AWSMultiple use cases across one platform

Splunk Light AMI on AWS Marketplace

Free 20GB License

6 Month Term = $6,000 Value

Bundled with App for AWS

Go To: https://aws.amazon.com/marketplace/ & Search “Splunk Light”

Demos available at AWS Re:Invent Booth #206

Thank you!

51

Contact:

scottjpack@gmail.com

github.com/scottjpack

Twitter: @scottjpack

Contact:randall.young@gmail.com

Twitter: @drandallyoung

Remember to complete

your evaluations!