© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Mike Dixon, Sr. Consultant, AWS

June 21, 2016

NextGen Security Operations on AWSUnlimited logging, scalable analysis, and pluggable

security tools without the price tag

Common challenges in the public sector Budget constraints Security slowing down mission Need for scalable (and low-cost) real-time log analysis Scaling uniformly across environment types

− Multiple locations/regions− On premises or cloud− Classified/unclassified

Implementing agile approaches within organizations with cultural or political challenges

Journey to an “all in” cloud infrastructure

What is agility?

Why be agile? − Cost reduction / pay for only what you use− Speed of deployment− Scalability− Reduced Risk

Hybrid Design Network Services

− Project segmentation− Layered firewall design

Central Services− VDI (on-prem and remote

access)− Software development

repositories− Central authentication and

security− Core services: DNS, DHCP,

NTP− Big Data (HADOOP/SPARK)− Desktop deployment automation

Hybrid Design: View from the Cloud Region Selection

− Geographic requirements (us-east, us-west, etc)

− Commercial vs. AWS GovCloud VPC design

− Internal/support services− External access− Special: Disconnected VPCs

Availability Zones− Inter-region failover

Routing− 10G DirectConnect− Inter-VPC connectivity− VPN Gateways to on-prem− Internet gateways, Out of band management

Developer Services Requirements

− Isolation Internal: Intranet access only External: Internet accessible,

facilitates collaboration with partners

− Availability Elastic Load Balancing (ELB) for

automatic, multi-AZ availability of instances

− Performance ElastiCache for fast, managed,

in-memory cache− Database

Amazon RDS: managed, scalable database service

Used for GITLab service failover No-downtime upgrades

Isolated Web Hosting High risk web presence!

Requirements− Isolated environment

“Special” disconnected VPC

− Minimizing Cost/Risk Gen1 Architecture: Dynamic Content

Web/App/DB Architecture

Route53, EC2, Internet GW Gen2 Architecture: Static Archive

Static Web Hosting with Route53, S3, and CloudFront

“All-In” Architecture

Cloud-first approach to infrastructure needs

Emphasizes DevOps and security

Reliability, scalability

NextGen Security Operations:Key Concepts Infrastructure Automation

− Rapid deployment / tear down− Automated security features

Location Independence− Flexibility across regions / domains (commercial, AWS GovCloud)− Integration with existing / legacy environments

Aggregated data sources− Real time, scalable analytics (making sense of collected data)

API driven incident response− Automated detection and response to security threats

Centralized Management / Transparency

Traditional security architecture Network traffic and logs captured

by taps and analyzed with on-premises tools

Limitations− Limited interfaces to manage live

datao Real time analysis with multiple tools

− Expensive to scaleo Log retention capacityo Processing powero Big data accesso Big data backup

Cloud-enabled security architecture All the benefits of the cloud

− Elasticity, Scalability− Ease of use− Flexibility− Cost efficiency− Reliability− Breadth of services

Security Automation Transparency

− Logging, auditing, metrics = Lots of data! Without the cost of licensing, on-prem storage, maintenance, facilities…

Cloud-enabled security architecture

Logging & auditing Infrastructure

− AWS CloudTrail− AWS Config

Network− VPC Flow Logs− ELB Access Logs

Application− Logstash, Elasticsearch, Kibana− Amazon CloudWatch logs integration

Flow Logs Example

CloudTrail Log Event

Logging: processing and forwarding Logstash

− Collects, processes, and forwards application events and log messages to a final destination in a customizable format

− Does not store data 3-part architecture

o Inputs – Configurable input plug-ins that are compatible with raw socket or packet communication, file tailing, and several message bus clients

o Filters – Filters that process, modify, and annotate the event data, and then parse and transform the data into useful formats

o Outputs – Ability to move the events to a variety of external services including Elasticsearch, Amazon Simple Storage Service (Amazon S3), local files, and several message bus implementations

Logstash configurationSample Apache Log:

input { file { /var/log/apache/access.log } }

filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] }}

output { elasticsearch { hosts => [“ELASTICSEARCHCLUSTER:9200"] } s3 { bucket => [“LOGS/infrastructure/websitelogs/”] } }

{ "message" => "127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] \"GET /xampp/status.php HTTP/1.1\" 200 3891 \"http://cadenza/xampp/navi.php\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\"", "@timestamp" => "2013-12-11T08:01:45.000Z", "@version" => "1", "host" => "cadenza", "clientip" => "127.0.0.1", "ident" => "-", "auth" => "-", "timestamp" => "11/Dec/2013:00:01:45 -0800", "verb" => "GET", "request" => "/xampp/status.php", "httpversion" => "1.1", "response" => "200", "bytes" => "3891", "referrer" => "\"http://cadenza/xampp/navi.php\"", "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\""}

127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"

1TB Log StorageS3 = ~$30/month (us-east-1) Glacier = ~$7/month (us-east-1) https://calculator.s3.amazonaws.com

Logging: search & analytics Elasticsearch

− Open-source search and analytics engine for log analytics, real-time application monitoring, and click-stream analytics

− Search-server based on Lucene that provides a distributed, multitenant-capable full-text search engine with a web interface and schema-free JSON

− Uses the Kibana web interface as its visualization plug-in

Amazon Elasticsearch Service (Amazon ES)− Managed service that makes it easy to deploy, operate,

and scale Elasticsearch clusters in AWS − Cluster scaling and self-healing− Region and Availability Zone (AZ) replication− Data durability, monitoring

Logging: log storage & retention Amazon Simple Storage Service (S3)

− Accessible through web services protocols (e.g., REST, SOAP)− Durable, low-cost, available, secure, scalable, integrated, easy to use− Integrates with AWS Key Management Service (AWS KMS)

Use case for logging− Centralized storage− Access control

o S3 bucket policies & security (WORM, MFA delete, versioning)o Cross-account access

− Lifecycle policies (Example)o Transition to the Standard - Infrequent Access storage class 180 days after creation dateo Archive to the GLACIER storage class 365 days after the object's creation dateo Permanently delete 2,562 days (7 years) after the object's creation date

Analytics− S3 data readily available− EMR (Spark/Hadoop) clusters look at S3 bucket− Dedicated Amazon ES to research project point to S3 bucket

Logging: content visualization Kibana

− Open source data visualization plug-in for Elasticsearch− Provides visualization capabilities on top of the content

indexed on an Elasticsearch cluster− Gives shape to data− Understands large volumes of data− Creates bar charts, line or scatter plots, histograms, pie

charts, and maps Provides sophisticated analytics

− Analyzes data intelligently, performs mathematical transformations

− Slices and dices the data to custom requirements

Logging: content visualization (Kibana)

Security automation: infrastructure as code

Takeaways: for your next workload on AWS Design for Agility Use available services for the “heavy lifting”

Security, logging, analytics Architect for location independence

Region requirements, compliance – GovCloud vs. commercial Consider points of isolation and integration

Segmentation with VPCs, NACLs, security groups, firewall VPC Peering, connectivity to on premises

Aggregate data sources (infrastructure, network, application logs) Make use of data (analysis, visualization) Automate (security as code – DevSecOps!)

Demo code

Additional info and demo code from today’s session:

stratussolutions.com/aws

Thank you!