amazon elasticsearch service security deep dive - aws online tech talks

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Amazon Elasticsearch Service

Security Deep Dive

November 9, 2017

Jon HandlerAWS Principal Solutions Architect

[email protected] or @_searchgeek

mailto:[email protected]

Get started at https://aws.amazon.com/elasticsearch-service/

Introduction

• Amazon Elasticsearch Service supported public

endpoints. Access control used IAM and/or IP-based

policies

• October 17, we added VPC support

• You can still use IAM for fine-grained access, but VPC

support enhances security and simplifies communication

with Amazon ES

https://aws.amazon.com/elasticsearch-service/


Data

Analysis

InsightsActions

EvolutionYour

Busines

s

What is Elasticsearch all about? Let’s start with

the data

• Derive insights from

high-volume,

unstructured or semi-

structured data

• at scale

• securely

• and cost-effectively

• without diverting focus

from your mission

Your data drives your

business



Some uses for your data

IT operations

monitoring

Security information and

event management

Application monitoring

and intelligence

IoT/TSDB monitoring

and data analysis

Search for your

application

Ad targeting



How Amazon ES helps

Log Analytics

Analyze un-structured and semi-structured logs generated

by websites, mobile devices, servers, sensors, and more for

a wide variety of applications such as digital marketing,

operational intelligence, fraud detection, ad tech, gaming,

and IoT.

Full Text Search

Provide a highly performant, rich search and navigation

experience over a diverse set of documents with support for

features including text matching, faceting, filtering, fuzzy

search, auto complete, and highlighting.

Distributed Search

Engine

Power search for your application with

an easy to use, highly performant

JSON document-oriented platform that

can store and retrieve billions of

documents, with integrated replication

across Availability Zones.

Real-Time

Application Monitoring

Capture activity logs across your

customer-facing applications and

websites by indexing data for analysis

in near real-time (less than one

second), visualize it, and perform

statistical aggregations to identify root

cause and fix issues.

Click-Stream

Analytics

Deliver real-time metrics on digital

content and enable authors and

marketers to connect with their

customers. Stream billions of small

messages into Elasticsearch where you

can aggregate, filter, and process the

data to provide content performance

dashboards.



Scalability AWS IntegrationSecurity &

Monitoring

Availability &

Data Protection

Ease of use &

TCO

Dev Platform &

Tools

• Fully-managed –

automated failure

management,

software patching

and maintenance

• Fast deployments

• Support for

search templates

• Zone awareness

to automatically

replicate data

across two

Availability Zones

• Automatic failure

recovery without

service disruption

• Automated

backups

• Authentication via

Amazon IAM

• Index level

access control

• Auditing via

Amazon

CloudTrail

• Monitoring and

Alerting via

Amazon

CloudWatch

• Seamless

scalability without

availability issues

• Wide variety of

instance types

and storage

options

• Up to 100 node

cluster support

• Programmatic

scale-up/ scale-

down support

based on

CloudWatch

metrics

• Data ingestion -

Amazon Kinesis

Firehose, Amazon

IOT, Amazon

CloudWatch Logs

• Amazon

CloudFormation

support

• Auditing and

Monitoring -

Amazon CloudTrail,

Amazon

CloudWatch

• Amazon IAM based

authentication

• Elasticsearch

open-source

APIs, supports

versions through

ES 5.5

• Logstash support

data for ingestion

and

transformation

• Kibana support

for visualization

• Support for 11

plugins covering

areas such as

extended Unicode

support, Phonetic

analysis etc.

Amazon ES benefits



FT – Financial Times

P R O B L E M• Which stories do our readers

care about? What’s hot?

• Required a custom clickstream analytics solution.

• Need a solution that delivers analytics in real-time.

• Did not have a team to manage analytics infrastructure.

S O L U T I O N• Streaming user data to

Elasticsearch Service for analysis. Created their own custom dashboards for editors and journalists – Lantern.

• Lantern - ”shines a light” on reader activity for the editors and journalists at the FT.

• Critical tool for making editorial decisions. Daily editorial meetings start by looking at Lantern dashboard.

B E N E F I T S• Reliability - Lantern is used

throughout the day by journalists and editors. Relying on Amazon to manage their systems for maximum uptime.

• Cost savings - Able to easily tune their cluster to meet their specifics needs without much management overhead


Security Model

for Amazon ES


Service architecture

AWS SDK

AWS CLI

AWS CloudFormation

Elasticsearch

data nodesElasticsearch

master nodes

Elastic Load

Balancing

AWS IAM

CloudWatchCloudTrail

Amazon Elasticsearch Service domain

Internet or

your VPC

AWS Console



AWS Security Responsibilities

• Creation of a service VPC that allows limited access to

the instances in your domain

• Application of security patches on the instances

• DDOS protection for the DNS name associated with the

domain via Route53

• Hides ports 9200 and 9300

• Built on top of AWS secure networking



• Decide on your authentication strategy

• IAM

• VPC

• Identify key roles and build IAM policies or create

networking infrastructure to support their interaction with

Amazon ES

• Apply and maintain IAM policies and security groups to

AWS resources

Customer Security Responsibilities


Using IAM for Authentication


Access via the internet



Use a template or write your own

We strongly recommend against using an “open access” policy



IAM policy application and resolution

role

Elastic IP

address

role

policy

policy

policy

Users have roles

Or policies

Instances have roles

with policies and IPsAmazon ES domains

have policies

• IAM authenticates based on all applicable identification and all policies

are in play

User-based policies Resource-based policy


Access Policy Application & Resolution

• Deny ALWAYS wins over competing policy types

• If you do not explicitly state a policy, deny is default

Allowed in a

resource-based

policy

Denied in a

resource-based

policy

Neither allowed

nor denied in a

resource-based

policy

Allowed in an

identity-based policy

Allow Deny Allow

Denied in an


Deny Deny Deny

Neither allowed nor

denied in an


Allow Deny Deny


Policy skeleton

{

"Version": "2012-10-17",

"Statement": [ {

"Effect":...

"Principal": ...

"Action": [...],

"Resource": ...,

"Condition": ...

} ]

}

• Effect: Allow or Deny

• Principal: AWS account ID

• Action• HTTP verbs

• Service actions

• Resource: Amazon ES

domain/index

• Condition: IP Address



Baseline IP-based access

Amazon ESIAM

Elastic IP

address

• Accessing from a known IP address

• The domain’s policy controls the Actions that are allowed/denied

• Access is anonymous

Policy



• IP-based control, resource-based policy. All users, all

Actions, all indexes

{"Sid": "","Effect": "Allow","Principal": { "AWS": "*” },"Action": "es:*","Resource": "arn:aws:es:us-east-1:12345678910:domain/test/*”"Condition": {

"IpAddress": {"aws:SourceIp": [”1.2.3.4"]

} }}

Access policy for IP-based access



Identity-based access

Amazon ESIAM

• Accessing with an identity, authenticated with

access/secret key

• Policy resolution as described

• Access is via signed request

PolicyAWS SigV4

signing

Lambda

AWS SigV4

signing



• Requests must be signed. User-name-1 can run all

actions against all indices

{"Sid": "","Effect": "Allow","Principal": {

"AWS": ["arn:aws:iam::12345678910:user/user-name-1”

]},"Action": "es:*","Resource": "arn:aws:es:us-east-1:12345678910:domain/test/*”

}

Access policy



Application access

Application

Amazon ES - search

Amazon ES - monitor

User

Amazon RDSUpdater

Kibana

Proxy

Administrator

DevOps/IT



Create roles for access to Amazon ESAdministrator IT/DevOps Application Updater Proxy

Type User-based User-based Resource-based Resource-based Resource-based

Actions • es:CreateElasticsearchDomain

• es:Describe*

• es:DeleteElasticsearchDomain

• es:ListDomainNames

• es:AddTags

• es:ListTags

• es:RemoveTags

• es:Update*

• es:ESHttpGet

• es:ESHttpPut

• es:ESHttpDelete

• es:ESHttpPost

• es:Describe*

• es:ListDomainNames

• es:AddTags

• es:ListTags

• es:RemoveTags

• es:Update*

es:ESHttpGet es:ESHttpPost es:ESHttpGet

Resources Amazon ES search

Amazon ES monitor

Amazon ES search

Amazon ES monitor

Amazon ES search Amazon ES search Amazon ES Monitor

IPs No No EIP for the application

instance

EIP for the updater

instance

EIP for the proxy



in your VPC


Amazon ES architecture with VPC

Availability Zone A

Availability Zone B

VPC subnet

security group

VPC subnet

security group


Data Master

Data

Master

IAM

IAM



Set up for VPC access

• Select your VPC

• Select a subnet with

sufficient IP space and

ENIs for 3x your data

instances

• Select a security group

to apply to the Amazon

ES ENIs



Simple VPC access

Availability Zone A

Subnet A

security group


Data

Master

Application

Instance(s)

Amazon RDS

Internet

gateway

Amazon Route 53



Simple VPC access

• Internet gateway provides access for application users,

search, and monitoring traffic within the subnet

• Security group has normal inbound/outbound rules

• Because the IPs are within the security group, SigV4

signing is not required



Application search within VPC

Availability Zone A

Availability Zone B

VPC subnet

security group

VPC subnet

security group


Data Master

Data

Master

IAMApplication

Application IAM

Internet

gateway



Application search within the VPC

• With Zone Awareness enabled, the domain is in 2

subnets

• IAM provides additional security for IP-based or signed

requests



Logging infrastructure in your VPC

security group

ELB

Logstash Indexers

Amazon Elasticsearch

Service Domain

Internet

gateway



Logging infrastructure in your VPC

• Logstash colocated with the infrastructure you are

monitoring

• Use an ELB across an autoscaled group of indexers to

batch and forward to Amazon Elasticsearch Service

• Use a reverse proxy in the VPC to forward Kibana traffic

to Amazon ES



Conclusions

• Amazon Elasticsearch Service now supports access

control using VPC security groups

• With IAM, you can further narrow access for particular

users

• All traffic remains within your VPC


Find out more:


AWS Centralized Logging:

https://aws.amazon.com/answers/logging/centralized-logging/

Elasticsearch at the AWS Database Blog:

https://aws.amazon.com/blogs/database/category/elasticsearch/

Or ask your Solutions Architect!

Amazon

Elasticsearch

Service


https://aws.amazon.com/answers/logging/centralized-logging/

https://aws.amazon.com/blogs/database/category/elasticsearch/

amazon elasticsearch service security deep dive - aws online tech talks

Documents