Splunk for Visibility into AWSJim Nichols, Principal Engineer - Cloud Architect and Splunk Champion at EnerNOC

EnerNOC at a Glance

• A leading provider of SaaS-based Energy Intelligence Software (EIS) and related solutions

• Market leader in demand response (DR)

• Global company (over 1,300 employees in countries across North America, APAC, Europe) with HQ in Boston, MA

EnerNOC’s Energy Intelligence SoftwareFor enterprises: platform and solutions focus on the 3 drivers of energy expense

How you buy it

Budgets and ProcurementUtility Bill Management (UBM)

How much you use

Visibility and ReportingFacility Optimization

Project Tracking

When you use it

Demand ResponseDemand Management

Increase Use of Enterprise Energy Intelligence Software

More than 70,000 sites and devices stream data into EnerNOC’s energy intelligence software platform

Over 6,000 companies globally rely on EnerNOC to drive energy savings

EnerNOC Is Transforming Energy Management Across Industries

Increase Use of Enterprise Energy Intelligence SoftwareDemonstrated expertise trusted by the largest companies in the world:

Utility and Grid Operator PartnershipsEnerNOC has Extensive Expertise Working With Utilities and Grid Operators Globally

We also operate in wholesale markets:

The Case for Demand ResponseBalancing supply and demand on the electricity grid is difficult and expensive. End users that provide a balancing resource are compensated for the service.

Annual Electricity Demand As a Percent of Available Capacity

50%

100%

Winter Spring Summer Fall

75%

25%

90%

EnerNOC EIS Platform

Data Streaming and Processing at Scale

Cloud Hosted Web & Application

Servers

EnerNOC Hosted Web & Application

Servers

4000/1000 Users Daily/Hourly 100 Countries

EnerNOC Database Servers

Cloud Hosted Database Servers

32TB Persisted

About JimMultiple DevOps Roles @ EnerNOC since 2007

• Performance (site reliability), Architecture, Operations, Development…

• Charted with establishing Cloud Operations team in 2014

#1 Goal – Make sure production is up, running fast, and reliable

Brought Splunk in to look at web logs in March 2012• Now have ~150+ indexes, 100GB/day, 1000’s of forwarders

EnerNOC’s Splunk Champion

Energy Intelligence in the CloudGreat success with fast data on-premises: Oracle, Stored Procs, 3-tier Java

Strong business requirements for Big Data and Analytics, Rapid Development

Major shift in strategy 18-24 months ago: All-in with AWS• All new development in AWS• Migrating technology of acquisitions into AWS• Moving on premises workloads to the cloud

Efficiently made the pivot• Start of 2015: 30% in AWS, 70% in Data Center(s)• Start of 2016: 80% in AWS, 20% in Data Center(s)• Goal 2017: 95% AWS

Our AWS Environment

• AWS Usage

• Diverse Technology: MongoDB and Cassandra

12-25 AWS Accounts

All public AWS regions

1000+ EC2 Instances

150+ Engineers in 7+ Countries in AWS 90+ VPCs

Monitoring the TransitionBootstrap Cloud Operations team:

• Basic Operations • Architecture and Design• Security• Operational Best Practices

High degree of parallel activity

“Long” time Splunk customer (5+ years)• DevOps• Alerting and Monitoring• Fast and real-time data processing• Insight into many disparate systems in once place

Opportunity for Splunk

Leverage our mature Splunk practice in use for production DevOps

Bring data from all accounts to a single,

secure location

Enrich the application log data with AWS specific

data (CloudWatch)

Add additional layer of security on top of what AWS provides

Application specific dashboards with AWS data

Centralized alerting on AWS specific metrics

Splunk App for AWS

Splunk App for AWS v3, and then v4

Splunk IAM user in all accounts with limited read-only permissions

AWS CloudTrail in all regions

Setup each account, then inputs• Just pulled in CloudTrail, billing and a few s3 buckets (ELB logs, other data)• Since v4, pull in AWS Config, CloudWatch, S3 ELB

Start slowly, add accounts and inputs over time

Splunk in Action

• Started of looking at the dashboards, get a feel for how the environment looks

• Enable included alerts, or created own based off searches from dashboard panels

IAM Examples

IAM Unauthorized and Detailed Activity• Monitor for Unauthorized IAM Actions• For malicious activity• Non-conformant activity• Build tool errors

IAM Access Key Create/Update/Delete• Initially to avoid production outages and monitor key creation• Alert now allows us to enforce best practice of using IAM Instance Roles

Operational ExamplesDevelop insight into the AWS environment:

• What changes do we typically see?• Who are the active users/keys/IAM roles?

Troubleshooting:• Build Errors• Capacity Limits• AWS system errors

Next Steps

• Get in touch – jnichols@enernoc.com, splunk-admins@enernoc.com, or LinkedIn

• Boston Splunk Users Group

Securing your data on AWS

$6.53M 56% 70%

Your data and IP are your most valuable assets

https://www.csid.com/resources/stats/data-breaches/

Increase in theft of hard intellectual property

http://www.pwc.com/gx/en/issues/cyber-security/information-security-survey.html

Of consumers indicated they’d avoid businesses

following a security breach

https://www.csid.com/resources/stats/data-breaches/

Average cost of adata breach

In June 2015, IDC released a report which found that most customers can be more secure in AWS than their on-premises environment. How?

AWS can be more secure than your existing environment

Automating logging and monitoring

Simplifying resource access

Making it easy to encrypt properly

Enforcing strong authentication

AWS Foundation Services

Compute Storage Database Networking

AWS Global Infrastructure

Regions

Availability ZonesEdge Locations

Identity & Access Control

NetworkSecurity

Customer applications & contentYou get to define your controls IN the Cloud

AWS takes care of the security OF the Cloud

YouInventory & Config

Data Encryption

AWS and you share responsibility for security

Key AWS Certifications and Assurance Programs

The AWS infrastructure is protected by extensive network and security monitoring systems:

• Network access is monitored by AWS security managers daily

• AWS CloudTrail lets you monitor and record all API calls

• Amazon Inspector automatically assesses applications for vulnerabilities

Constantly monitored

The AWS infrastructure footprint protects your data from costly downtime

• 33 Availability Zones in 12 regions for multi-synchronous geographic redundancy

• Retain control of where your data resides for compliance with regulatory requirements

• Mitigate the risk of DDoS attacks using services like AutoScaling, Route 53

Highly available

AWS enables you to improve your security using many of your existing tools and practices

• Integrate your existing Active Directory

• Use dedicated connections as a secure, low-latency extension of your data center

• Provide and manage your own encryption keys if you choose

Integrated with your existing resources

Security Requires Visibility:Splunk App for AWSPraveen RangnathSr. Dir. of Cloud Product Marketing

Make machine data accessible,usable and valuable to everyone.

Proven at 11,000+ Customers in 110+ CountriesMore Than 80 of the Fortune 100

Splunk’s AWS Credentials

• AWS Advanced Technology Partner• AWS Big Data competency• AWS Security competency• AWS MSP Technology Provider• AWS Marketplace Partner (BYOL provider)• AWS IoT Launch partner for IoT analytics• AWS Security by Design Program Partner• 1st partner with published Blueprints for AWS Lambda• 1st partner to pass SaaS extension for Well Architected framework

EC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

RedshiftSNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

Customer Requirement: End-to-End AWS Visibility

Splunk App for AWS

Splunk App for AWS: The DataAWS CloudTrail

• Service that delivers logs of admin activity on AWS infrastructure

• Examples:• Start/Stop/Create instance• Change of User roles/rights• Modification of Network Configuration

• Delivers log files to customers; no UI, display, analysis, search

AWS Config• Provides resource inventory• Provides configuration history & change

information• Enables security & governance

Amazon Cloudwatch Metrics• IP traffic information to/from VPC network interfaces• Data stored and accessible from Amazon

CloudWatch LogsAmazon Cloudwatch VPC Flow Logs

• IP traffic information to/from VPC network interfaces• Data stored and accessible from Amazon

CloudWatch LogsAWS Access Logs

• Elastic Load Balancing (ELB)• Cloudfront CDN• S3

AWS Billing• Current Month via CloudWatch metrics• Monthly Detailed Billing

Splunk App for AWS: What Customers Gain

In-depth visibility into all activity in an AWS account

Real-time monitoring of AWS resource changes

Full audit-trail reporting across multiple AWS accounts and regions

Ability to prove both real-time and historical

compliance

Visual representation of the AWS Topology and resource relationships

Monitoring of VPC traffic utilization for

security insights

Dashboards showing historical and real-time

views/comparisons

Real-time visibility into AWS billing

Splunk App for AWS - Overview• The overview page shows you on

one screen information about:• Configuration changes• Compute• Storage• Billing• ELB• Cloudfront• Security

• Notable CloudTrail Activity is highlighted on the map.

• Drill down on any event and gain detailed information.

AWS Topology

• Topology view gives you a holistic view of your current or historical AWS deployment using AWS Config

• Maps out relationships between all the components, giving you a clear view into the environment

• Clickable layers adds additional visual queues for high CPU or network traffic

• Snapshot feature allows for topology to be saved for future use

AWS Usage

• In one glance, instantly see your EC2 usage and EBS Volume data info via CloudWatch metrics

• Click through dashboards for details on individual EC2 instances and EBS Volumes

• Drill down into raw search for even more detailed views on your instances.

VPC Flow Data - Security

• Utilizes VPC Flow Logs from CloudWatch for Security Analysis

• Drill down into rejected vs. accepted traffic

• View top Source Country and City information

• See top source / destination and IP addresses and ports

AWS Billing & Capacity Planning

• Utilizes Billing logs from CloudWatch for Month-to-Date billing and End-of-Month projections

• Detailed Historical Billing Dashboard available using Monthly AWS Detailed billing reports

• Capacity Planner gives additional clarity on AWS On-Demand instance spending

Index Untapped Data: Any Source, Type, Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Messaging

TelecomsOnline

Shopping Cart

Web Clickstream

s

Databases

Energy Meters

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security, Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

CloudTrail

Config

Lambda

EC2

True Goal: Complete Infrastructure Visibility

Portfolio of Cloud Solutions

100% Uptime SLASOC2 Type II certified

ISO 27001 certified

Apps for Cloud Services

AWS, ServiceNow, Akamai, SFDC …

AWS Integrations

AWS Lambda: First partner blueprintAWS IoT: Featured analytics platform

ES released as software and SaaS

Enterprise Security

ITSI released as software and SaaS

IT Service Intelligence

Easy to Get Started for FREE

Splunk App for AWS