improving infrastructure governance on aws - aws june 2016 webinar series

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Chris Munns, Business Development Manager - DevOpsJune 2016

DevOps on AWS:Improving Infrastructure Governance on AWS

https://secure.flickr.com/photos/mgifford/4525333972

Why are we here today?

Why are we here today?

Using cloud based infrastructure changes how to think about governing our infrastructure: Infrastructure can be provisioned in seconds.. and go

away just as quickly! Development teams expect a higher level of flexibility and

self control in interfacing with their infrastructure needs Being API driven means that the way people provision and

manage infrastructure in the cloud has changed compared to on-premises

Why are we here today?

That doesn’t mean that our basic governance needs change: We still need to have some ability to drive best

practices/patterns in our organizations We need to make sure that we’re able to audit

and track changes to our infrastructure for both regulation and security purposes

We need to make sure that we understand how resources are related and integrated

What can we do?

There are a few areas to focus on that can help us accomplish both the freedom to rapidly provision, manage, and update our infrastructure while meeting our governance needs: Policy as Code Infrastructure standardization (via code!) Self service environments Logging/Auditing/Reacting to infrastructure change

Policy as Code builds off of infrastructure as code practices by

allowing organizations to codify infrastructure and system configurations

allowing them to monitor and enforce compliance dynamically and at scale.

Infrastructure as Code is a practice in which infrastructure is provisioned

and managed using code and software development techniques,

such as version control and continuous integration.

Infrastructure as Code “levels”

AWS Resources

Operating System and Host Configuration

Application Configuration

Infrastructure as Code “levels”

AWS Resources

Operating System and Host Configuration

Application Configuration

allOfThis == $Code

Browse and launch

AWS ConfigAWS CloudTrail

Use and modify

Users Admin

Putting the AWS Management services together

AWS Service Catalog

Provision with Tags

API calls Configuration checks and reactions to change

Troubleshoot and Audit

Create templates of your infrastructure

CloudFormation provisions AWS resources based on dependency needs

Version control/replicate/update templates like code

Integrates with development, CI/CD, management tools

AWS CloudFormatio

n

Template CloudFormation Stack

JSON formatted fileParameter definitionResource creation

Configuration actions

Configured AWS resourcesComprehensive service support

Service event awareCustomizable

FrameworkStack creationStack updates

Error detection and rollback

CloudFormation – Components & Technology

Template File Defining Stack

The entire infrastructure can be represented in an AWS

CloudFormation template.

Many Stacks & Environments from One Template

Template File Defining Stack

The entire infrastructure can be represented in an AWS

CloudFormation template.

Use the version control system of your choice to store and track changes to this template

Many Stacks & Environments from One Template

GitPerforce

SVN…

Template File Defining Stack

GitPerforce

SVN…

Dev

Test

Prod

The entire infrastructure can be represented in an AWS

CloudFormation template.

Use the version control system of your choice to store and track changes to this template

Build out multiple environments, such as for Development, Test, Production and even DR using the same template

Many Stacks & Environments from One Template

CloudFormation example use cases:

Have “full stack” templates that can be used to stand up common application patterns inside your organization such as a 3-tier application template that: uses Lambda custom resources to look up appropriate VPC

information (VPC ID, Subnets, etc) based on tags creates an Elastic Beanstalk environment that supports Multi-AZ,

AutoScaling, CloudWatch Metrics, and Elastic Load Balancing contains security controls such as AWS Identity and Access

Management (IAM) roles, profiles, and policies, and Security Groups allows the user to specify the language of their application allows a user to specify which database they want (SQL or NoSQL)

and then creates the appropriate resource

Using Parameters and Conditionals are two key ways to make a single template much more dynamic:"Parameters" : { "Database": { "Type" : "String", "Default" : "RDS", "AllowedValues" : ["RDS", "DynamoDB", "None"], "Description" : "Database to create. Select None if using an existing database.” } },

"Conditions" : { "CreateRDS" : {"Fn::Equals" : [{"Ref" : "Database"}, "RDS"]}, "CreateDynamoDB" : {"Fn::Equals" : [{"Ref" : "Database"}, "DynamoDB"]}, "CreateNone" : {"Fn::Equals" : [{"Ref" : "Database"}, "None"]} },

“Resources” : { ”RDSdb01" : { "Condition" : " CreateRDS ", "Type" : "AWS::RDS::Instance",

https://secure.flickr.com/photos/wscullin/3770015991

Now that we have these templates, what can we do to help simplify our developers’ lives further and increase our infrastructure standardization?

Customized catalogs of products

Manage products centrally

Personalized, self-service portal

Integrate with existing systemsAWS Service Catalog

What is AWS Service Catalog?

AWS Service Catalog allows organizations to create and manage catalogs of IT services. It enables users to quickly deploy the approved IT services they need in a self-service manner.

Organizations Developers

ControlStandardization

Governance

AgilitySelf-service

Time to market

Creates portfolio

Adds constraints and grant access

1

4

5

AdministratorPortfolio

Users

Browse Products

6Launch ProductsAWS CloudFormation template

Creates product3Authors template2

ProductX

ProductY

ProductZ

7Deploys stacks

EventsEvents

88

Service Catalog

Create custom services

and grant access

Use a personalized

portal to find and launch services

Service Catalog use cases:

You can remove the need for developers to understand how all AWS services work. Treat infrastructure provisioning like buying components from a retail site: provide standardized Service Catalog products around

common internal application frameworks/architectural patterns provide common application component products such as

databases, queues, caches, worker tiers, etc build logging, monitoring, metrics into these stacks leverage service discovery tools when possible build in the same best practices across development, staging,

production environments with these provided products

We’ve helped solve some of our developer’s access and standardization issues, but how can we now go about auditing changes to our infrastructure?

https://www.flickr.com/photos/atoach/7623237104

AWS CloudTrail

Records AWS API calls for your account

Delivers log files of API calls to S3

Delivery typically within 15 minutes of API call

Logs contain detailed information

Log files can be encrypted and have their integrity verified by you

AWS CloudTrail

CloudTrail can help you achieve many tasks Security analysis Track changes to AWS resources, for

example VPC security groups and NACLs Compliance – log and understand AWS API

call history Prove that you did not:

Use the wrong region Use services you don’t want

Troubleshoot operational issues – quickly identify the most recent changes to your environment

AWS CloudTrail logs can be delivered cross-account

CloudTrail can help you achieve many tasks Accounts can send their trails to a central

account Central account can then do analytics Central account can:

Redistribute the trails Grant access to the trails Filter and reformat Trails (to meet

privacy requirements)

CloudTrail

CloudTrail – Amazon CloudWatch Logs Integration

Continuous recording

Inventory of AWS resources

New & deleted resources

Configuration change & compliance notifications

AWS Config

Relationships

Bi-directional map of dependencies automatically assigned

Change to a resource propagates to create Configuration Items for related resources

Example: Security Group sg-10dk8ej and EC2 instance i-123a3d9 are “associated with” each other

Configuration Item

All configuration attributes

Normalized

Point in time

Captured on configuration change

Component Description Contains

Metadata Information about this configuration item

Version ID, Configuration item ID, Time when the configuration item was captured, State ID indicating the ordering of the configuration items of a resource, MD5Hash, etc.

Common Attributes Resource attributes Resource ID, tags, Resource type. Amazon Resource Name (ARN)Availability Zone, etc.

Relationships How the resource is related to other resources associated with the account

EBS volume vol-1234567 is attached to an EC2 instance i-a1b2c3d4

Current Configuration Information returned through a call to the Describe or List API of the resource

e.g. for EBS VolumeState of DeleteOnTermination flagType of volume. For example, gp2, io1, or standard

Related Events The AWS CloudTrail events that are related to the current configuration of the resource

AWS CloudTrail event ID

Configuration Item

Sample Configuration Item "configurationItemVersion": "1.0", "configurationItemCaptureTime": "2014…", "configurationStateID": “….", "configurationItemStatus": "OK", "resourceId": "vol-ce676ccc", "arn": "arn:aws:us-west-………", "accountId": "12345678910", "availibilityZone": "us-west-2b", "resourceType": "AWS::EC2::Volume", "resourceCreationTime": "2014-02..", "tags": {}, "relationships": [ { "resourceId": "i-344c463d", "resourceType": "AWS::EC2::Instance", "name": "Attached to Instance" } ],

"relatedEvents": [ "06c12a39-eb35-11de-ae07-db69edbb1e4", ],

Metadata

Common Attributes

Relationships

Related Events

Sample Configuration Item"configuration": { "volumeId": "vol-ce676ccc", "size": 1, "snapshotId": "", "availabilityZone": "us-west-2b", "state": "in-use", "createTime": "2014-02-……", "attachments": [ { "volumeId": "vol-ce676ccc", "instanceId": "i-344c463d", "device": "/dev/sdf", "state": "attached", "attachTime": "2014-03-", "deleteOnTermination": false } ], "tags": [ { "tagName": "environment", "tagValue": "PROD"

Configuration

Having this data is one thing, but how can we react to change?

https://www.flickr.com/photos/livenature/204420128/

CloudTrail – Amazon CloudWatch Logs Integration

CloudTrail – Amazon CloudWatch Logs Integration

Trigger a CloudWatch Alarm based on API call!

Check configuration changes

Pre-built rules provided by AWS

Custom rules using AWS Lambda

Continuous assessment

Dashboard Compliance visualization Identify offending changes

GitHub repo: Community sourced rules

AWS Config Rules

Normalize

RecordChanging Resources

AWS Config & Config RulesDeliver

Stream

Snapshot (ex. 2014-11-05)AWS Config

APIs

Store

History

Rules

Config Rules

Config Rules

Config Rules GitHub repository

AWS Management Tools Partners

FIN, ACKWe’ve seen a quick run through today of the ways you can improve your governance on AWS: Treat your infrastructure and host configuration as code! This lends itself to being able to use services like Service

Catalog to enable self-service in your organization Track, trend, and alert on CloudTrail API logs to keep on top

of access to your AWS resources Use Config and Config Rules to understand the relationship

between resources and react to policy violations Putting all this together is what gives you Policy as Code!

Browse and launch

AWS ConfigAWS CloudTrail

Use and modify

Users Admin

Putting the AWS Management services together

AWS Service Catalog

Provision with Tags

API calls Configuration checks and reactions to change

Troubleshoot and Audit

But wait, there’s more!

Resources to learn more: More on DevOps: https://aws.amazon.com/devops/ AWS Management Services: https://aws.amazon.com/products/management/ AWS CloudFormation

https://aws.amazon.com/cloudformation/ AWS Service Catalog

https://aws.amazon.com/servicecatalog/ AWS CloudTrail

https://aws.amazon.com/cloudtrail/ AWS Config / Config Rules

https://aws.amazon.com/config/ GitHub repo: https://github.com/awslabs/aws-config-rules

Thank you!

Happy Deploying!