aws re:invent 2016: securing container-based applications (con402)

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

December 2016

CON402

Securing

Container-Based Applications

Henrik Johansson, Security Solutions Architect, AWS

Michael Capicotto, Solutions Architect, AWS

What to Expect from the Session

OS & Container Security

• Intro to container security and best practices

• Container lifecycle security (including vulnerability analysis)

• Managing secrets

Infrastructure security

• Enforce governance with Amazon ECS

• Using containers to automate security deployments

Intro to Container Security

Container vs VM isolation

Isolation

• Implemented via Linux namespaces

• Weaker than VM isolation

• Containers run on single kernel

• Containers share files, sockets, memory areas,

devices, etc.

Secure by default

pid namespace

nmt namespace

net namespace

uts namespace

user namespace

pivot_root

uid/gid drop

cap drop

all cgroups

selinux

apparmor

seccomp

Out-of-the-box default settings and profiles

Granular controls to customize settings

No SSH to containers

No system users

Immutable infrastructure

Secure

by

default

Defense through segmentation

• Run containers on top of virtual instances

• Keep instance and container up to date

• IAM roles for tasks to restrict host permissions

• Segregate containers

• Role/customer

• Risk/trust/exposure

Reduce attack surface area

• Container attack surface much smaller than VM

• Service, not system

• Emulate drivers

• VENOM attack / CVE-2015-3456

(http://venom.crowdstrike.com/)

• Best practice: run small images

• Only contain static binary

• Harden the cluster instance

• NIST, SANS, etc.

• Use many but smaller instances to limit blast radius

Best practices

• Limit memory, CPU allocation and resource limits

• Set filesystems to be read-only

• Limit container networking

• Remove setuid/setgid binaries from images

• Set containers to run as non-root user

• Leverage Linux kernel security features

Setting resource limits in ECS

Configured in the ECS task definition

• CPU

• Minimum number of CPU units to reserve for container

• RAM

• The number of MiB of memory reserved for container

• Ulimits

• Reduce for example filesize locks and memlocks allowed

"ulimits": [

{

"name": "core"|"cpu"|"data"|"fsize"|"locks"|"memlock"|"msgqueue"|"nice"|"nprock"…

"softLimit": integer,

"hardLimit": integer

}

...

]

• Configured in the ECS task definition

• Setting root file system to be read-only

• Maps to Docker ReadOnlyRootfs and --read-only

• Set volumes to be read-only

Read-only filesystem access

"readonlyRootFileSystem": true

"mountPoints": [

{

"sourceVolume": "string",

"containerPath": "string",

"readOnly": true

}

]

Securing container networking

• Containers should only open ports it needs

• Governance on task definition

• Pre deploy assessment

• Know expected traffic patterns

• North/South, East/West

• Disable inter-container communication

• Set Docker option flag --icc=false and --iptables=true

• Use --link to connect containers

Securing container networking -

• Detects threats from external and internal networks

• Attacks such as DDOS and XSS on containers

• Protects your container network from L3 to L7

• Detects application threats

• Has built-in security policies for 30+ applications and

protocols

• Prevents unauthorized connections between containers

• Monitors all ‘east-west’ container traffic

• Integrate logs with SIEM/Log aggregator

Remove setuid/setgid binaries

• Most apps don’t need access to setuid/setgid binaries

• Remove to avoid privilege escalation attacks

• Example Debian “defanged ” image Dockerfile:

FROM debian:wheezy

RUN find / -perm +6000 –type f -exec chmod a-s {} \; || true

Set a non-root user

Multiple methods

• Create the USER in Dockerfile

• Change to the user via USER command or sudo/gosu/su

• Set USER in Docker command or ECS task def

Don’t use root as default!

Seccomp

• Secure computing mode (Seccomp) is a Linux kernel

feature

• Supported from Docker 1.10

• Docker has default profile limiting many system calls (44

system calls out of 300+)

• Can also be customized for more blocks

• Can provide custom seccomp profile to containers via

Docker security options

SELinux

• Implementation of Mandatory Access Control (MAC)

• Default SELinux policy for Docker designed to protect host

from containers and containers from each other• Uses tag to restrict usage

• Containers assigned default process type: svirt_lxc_net_t

and files accessible to container: svirt_sandbox_file_t.

• Policy enforces only read/execute container permissions to

/usr directory

• Assigns unique MCS category number to prevent cross

container access to files or resources

Security options in ECS

Security options in ECS

ECS_SELINUX_CAPABLE=true

Container Lifecycle Security

Lifecycle risks

• Are my developers

introducing new

vulnerabilities

during

development?

• Are my container

images in my

repository

vulnerable?

• Do I have a runtime

defence if containers are

compromised?

• How do I ensure by

Docker platform is

secure over time?

Mitigation approaches

• Do image

vulnerability analysis

in the CI/CD pipeline

• Only allow

developers to use

“approved” images

• Configuration

governance

• Regularly analyse

images in your

private repository

• Only allow compliant

images to run in

production

• Have runtime defence

in place

• Regular checks of

Docker platform

against industry

benchmarks

Own your repo!

Secure Host via Docker Bench

• Recommendations provided by Center for Internet

Security’s Benchmark for Docker Engine

• Best practice: use Docker Bench to validate host

against recommendations

• www.dockerbench.com

Run DockerBench via EC2 Run Cmd

Output report to

CloudWatch logs

Create CloudWatch alarm

on non-compliance

Automating Docker Bench

Invoke Lambda on schedule

Scan running containers for vulnerabilities in dev, test, staging, and

production.

• Public CVE vulnerabilities

• Some application specific vulnerabilities which may or may not have

CVE

• Scan the host file system, not just containers

• Can to tell whether the host is safe or not to load application

containers.

Visualize services, containers, and network behavior easily in staging

and production

Protects running containers against violations, threats, and

vulnerabilities

Continuous integration

CI/CD pipeline / Container lifecycle

CodeCommit

Source Build CodePipeline

Jenkins

1. Build Docker image

2. Check image for vulnerabilities

3. Push image to ECR

4. Update ECS service via CloudFormation

Run

Live scan

Managing Secrets

Bake into image

Bake into image

Environment variables

• Suggested by 12-factor apps

• Environment variables can be seen in

too many places

• linked containers, ECS API calls, docker

inspect

• Can’t be deleted

S3-based secrets storage

• Secrets stored in S3 bucket

• Accessed via IAM roles for EC2

• Enforce encryption at rest and flight via IAM

policies and KMS

• Use VPC endpoint for S3 to lock down access

from certain VPCs

https://blogs.aws.amazon.com/security/post/Tx2B3QUWAA7KOU/

Third-party secrets management solutions

• Vault from HashiCorp

• https://www.vaultproject.io/

• Keywhiz from Square

• https://square.github.io/keywhiz/

• Secrets accessed via leases

• Good for Dynamic Secrets generation

Temporary credentials for RDS databases

Vault for RDS credentials

Enforce Governance with ECS

ECS primer

ECS cluster

…Task 2

Task N

Task 1

Task definition 1 Amazon ECR

Docker images

Enforcing governance… is simple!

Task definition controls

• Which Docker images to use (app_latest)

• CPU & memory allocation

• Container links

• Host-to-container port mappings

• **NEW** IAM roles for tasks

IAM roles for tasks

Benefits

• Simplify usage of AWS SDKs in containers

• Credential isolation between tasks

• Authorization per task

• Auditability in CloudTrail with taskArn

IAM roles for tasks - example

Web UI

Container

Cluster Instance

Data Insights

Container

ECS Task 2ECS Task 1

IAM role

The old way…

Amazon

DynamoDBAmazon S3

Undesired

permissionUndesired

permission

IAM roles for tasks - example

Web UI

Container

Cluster Instance

Data Insights

Container

ECS Task 2ECS Task 1

IAM Role 1

The new way!

IAM Role 2

Amazon

DynamoDBAmazon S3

IAM roles for tasks – explained

1. ECS agent periodically queries ECS control plane

2. Control plane generates ID token

Auto-rotated

3. ECS agent:

Constructs HTTP URL for each container

Sets AWS_CREDENTIALS_ENDPOINT in HostConfig

4. AWS SDK extracts URL

IAM roles for tasks - demo

Automate Security Deployments

Governance during continuous integration

Docker image

Developers Security Engineers Ops EngineersAmazon ECR

Task definition

Governance during continuous deploymentDevelopers Ops Engineers Security Engineers

App Team

Developers Ops Engineers Security Engineers

InfoSec Team

Docker Images

Web Application Host-Based Intrusion Detection

Securely merge…

…and deploy

Governance during continuous deployment

App Team InfoSec Team

Task Definition Docker Images Task Definition

Deploying security functionality - example

Rate limiting proxy

Web Application

Container

Cluster instance

Reverse-Proxy

Container

Outbound

network

access

ECS Task

Deploying security functionality - demo

Why should I care?

• Remove accidental conflicts

• Make security processes continuous and automatic

• Encapsulate software artifacts and implement controls

one level up

• Control changes to this framework via IAM

Not just containers!

Amazon Machine

Images (AMIs)

Docker Images

OS Packages

Amazon EC2

Container Service

AWS

CloudFormation

AWS CodeDeploy

Amazon Machine

Images (AMIs)

Docker Images

OS Packages

Amazon EC2

Container Service

AWS

CloudFormation

AWS CodeDeploy

Software Artifacts Deployment Services

Not just containers!

Not just containers!

CloudFormation

TemplateTask Definition

Application Specification

File (AppSpec.yml)

…applies to any

deployment

process.

Thank you!

Remember to complete

your evaluations!