aws re:invent 2016: infrastructure continuous delivery using aws cloudformation (dev313)

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Dominic Divakaruni, Anil Kumar - AWS CloudFormation

11/30/2016

Infrastructure Continuous Delivery using

AWS CloudFormation

What to expect from this session

• We’ll show you how to:

• Architect your infrastructure using AWS CloudFormation

• Use AWS CloudFormation to set up AWS CodePipeline

pipelines

• Continuously deliver changes to stacks as you make

changes to your templates

• Demo

Lets look at release

processes

https://www.flickr.com/photos/jurvetson/5201796697/

• Integration

tests with

other systems

• Load testing

• UI tests

• Penetration

testing

Release processes have four major phases

Source Build Test Production

• Check-in

source code

such as .java

files.

• Peer review

new code

• Compile code

• Unit tests

• Style checkers

• Code metrics

• Create

container

images

• Deployment

to production

environments

Release processes levels

Source Build Test Production

Continuous integration

Continuous delivery

Continuous deployment

Continuous deployment

Release Processes levels

Source Build Test Production

Continuous integration

Our focus today

Continuous delivery

What about continuous delivery tools and

processes for infrastructure?

What do we need for infrastructure continuous

delivery?

• A way to treat infrastructure as code.

• Tools to manage the workflow that creates and updates

infrastructure resources.

• Tools to properly test and inspect your changes for

defects and potential issues

What do we need for infrastructure continuous

delivery?

Infrastructure as code

A practice in which infrastructure

is provisioned and managed using

code and software development

techniques, such as version

control and continuous

integration.

Workflow

Build, test, and deploy your code

every time there is a code

change, based on the release

process models you define,

enabling you to rapidly and

reliably deliver changes.

Infrastructure as Code

We need

Workflow

AWS CodePipelineAWS CloudFormation

AWS CloudFormation

• Create templates of your infrastructure

• Version control /code review /update

templates like code

• CloudFormation provisions AWS resources

based on dependency needs

• Integrates with development, CI/CD,

management tools

• No additional charge to use

Author templates in JSON or YAML

Use Change Sets to preview your changes

Continuous delivery workflows for stacks

Support for AWS Serverless App Model

Enable cross-stack references with Exports

Key new features

YAML

Author CloudFormation templates in JSON or YAML.

Syntax enhancements

JSON

YAML with enhancements

! function short form

Fn::Sub function

substitutes variables

Preview the set of actions CloudFormation will take on your

behalf before you create or update stacks.

CloudFormation Change Sets

Change Sets show you what resources will be created, updated

or replaced. This ensures that only expected operations are

executed.

Cross Stack References (Exports)

Network Stack

Outputs:VPCDescription: reference VPCValue: !Ref VPCExport:

Name: ProdVPC

App Stack

Resources:myTargetGroup:Type: AWS::ELBV2::TargetGroupProperties:

VpcId: Fn::ImportValue: ProdVPC

• Allows you to share information between independent stacks.

• Export a stacks output values. Other stacks in the same account and region

can import the exported values.

Nested stacks

Application

Resources:NetworkResources:

Type: AWS::CloudFormation::Stack

Network Resources

Resources: MyVPC

Type: AWS::EC2::VPC

ECS Service

Resources:MyService:

Type: AWS::ECS::Service

• Create a stack composed of multiple

templates.

• Compose and re-use templates with

frequently used resources

Considerations for Exports and Nested Stacks

Nested Stacks Cross Stack References

Recommended

uses cases

Advantages

Considerations

• Template reuse

• Use multiple templates but

manage as single stack

• Sharing common resources

• Allows for independent stacks based

on resource lifecycle or ownership.

• Convenient management.

One stack manages all

resources and nested stacks.

• Creation order and

dependencies are managed

• Separation of concern

• Share databases and VPCs

• Lets you limit blast radius with

safeguards

• Updates and rollbacks have

a wide surface area

• Reusing templates that have

custom resource names

• Replacing updates requires

changes to the importing stacks to

execute.

• Does not manage creation order

Use case

Let’s examine a sample application

Deconstruct the application into the necessary AWS resources

Create CloudFormation templates based your management needs

Model your continuous delivery pipeline

Continuously deliver infrastrucure changes as you iterate on your architecture

Use CloudFormation to model, provision and manage changes to your pipeline

Microservices application based on Amazon

ECS

Two interconnecting microservices deployed as ECS

services (website-service and product-service).

The application runs on a highly available ECS cluster

deployed across multiple availability zones with auto

scaling

Available at github.com/awslabs/ecs-refarch-cloudformation

Reference architecture

Public Subnet

Private Subnet

Availability Zone

Internet

Gateway

Public Subnet

Private Subnet

Availability Zone

Application

Load Balancer

NAT GatewayNAT Gateway

ECS Cluster

CloudWatch Logs (Container Logs)

ECS Host ECS Host ECS Host ECS HostAuto Scaling Group

github.com/awslabs/ecs-refarch-cloudformation

Decompose into AWS resource types

NAT Gateway

Elastic IP

Default Public Route

Public Subnet 1Private Subnet 1

Default PrivateRoute

PrivateRouteTable

NAT Gateway

Elastic IP

Public Subnet 2Private Subnet 2

Default PrivateRoute

PrivateRouteTable

Ava

ilab

ilit

y Zo

ne

1A

vaila

bil

ity

Zon

e 2

VPCInternetGateway

Public Route Table

Load BalancerSecurity Group

ECS Host Security Group

Application Load Balancer

Load BalancerListener

Load BalancerDefault TargetGroup

ECS Cluster

Auto ScalingGroup

Auto ScalingLaunch Configuration

ECS (IAM) Role

IAM Instance Profile

ECS Service

ECS Task Definition

CloudWatchLog Group

TargetGroup

Listener Rule

Service Role

ECS Service

ECS Task Definition

CloudWatchLog Group

TargetGroup

Listener Rule

Service Role

Network SecurityLoad

BalancingFront End

ServiceECS

ClusterBack EndService

Build CloudFormation templates based on this

logical grouping

Template Description

Network VPC, AZs, subnets, routing, NAT and internet gateways

Security groups Security groups for the application

Load balancers ALBs that are deployed to the public subnets

ECS cluster ECS cluster deployed to private subnets

Back end service ECS service and task definition for the back end app

Front end service ECS service and task definition for the webpage

Setup your template to flow configuration to

each other

NetworkTemplate

SecurityTemplate

Load BalancingTemplate

Front End svcTemplate

ECS ClusterTemplate

Back End SvcTemplate

Outputs

Load BalancerListener

Load BalancerDNS Name

ECS Cluster

Load BalancerSecurity Group

ECS Host Security Group

VPC

Public Subnets

PrivateSubnets

Network

Security

Load Balancing

ECS Cluster

Front End

Back End

with nested stacks

Use these templates to build your stacks

with cross-stack references

Network

Security

Load

Balancing

ECS Cluster

Front End

Back End

Parent Template Microservices

Stack

Nested

templates

Templates Individual Stacks

Applying continuous delivery

Applying continuous delivery for your

infrastructure

Continuous delivery service for fast and reliable

application and infrastructure updates

Builds, tests and deploys your code each time there is a

code change.

Built in actions for AWS CloudFormationAWS

CodePipeline

How does this align with release phases?

Source Test Deploy

Source stage for

CloudFormation

templates can be

AWS CodeCommit,

S3 or GitHub

Use CloudFormation

Change Sets to verify

deployments prior to

execution

Create, update or

delete Stacks or

Change Sets.

Model your pipelines

Iterate more often on your application

and infrastructure code

Launch new versions in Dev and

promote to prod

Manage your network resources

separately per its own cadence.

Maintain separate, mirror sandbox and

production network environments.

Production

VPC, Security Groups,

Load Balancing

Sandbox

VPC, Security Groups,

Load Balancing

Production

ECS Cluster, Application

Front & Back Ends

Dev

ECS Cluster, Application

Front & Back Ends

Application PipelineNetwork Resources Pipeline

Pipeline for network resources

Source repo

Networking resources for

Sanbox/Dev environments

Individual stacks. Ordered to

account for dependencies

Change Sets to preview changes to prod

Manual approval before you

changes are applied to prod

Apply Changes to Prod

1

2

3

4

5

Pipeline for your application

Pipeline triggered as soon as new

versions are posted

Run your tests and clean up

your dev environment when

done, so you aren’t charged

for the instances you don’t

use.

Review to ensure resource modification

or replacement is what you expect

Continuously deliver changes to Prod

1

2

3

4

Create and manage your pipeline using

CloudFormation

Pipeline Artifact Store

S3 BucketPipeline Notifications

SNS Email NotificationsPipeline IAM Roles

CloudFormation template to setup your pipeline

Could be provisioned

in a separate stack

with IAM resources –

with cross-stack refs

Create and manage your pipeline using

CloudFormation

Choose ‘deploy’ action with CloudFormation

as the provider

CloudFormation has enabled several action modes

– REPLACE_ON_FAILURE creates a new stack if one

doesn’t exist, updates it if it does or replaces it if its

in a failed state

You can use template configuration files or

specify parameter overrides within the template

that defines your pipeline

Stage

Action

Action

configuration

Name of your CloudFormation template

Template walkthrough – model your pipeline

DEMO!

FIN, ACK

We’ve seen how to compose and continuously deliver your

infrastructure as code on our software release process:

• Different ways to decompose your infrastructure into

templates and stacks

• Create and provision your continuous delivery pipeline

for your infrastructure

• Deliver changes to your environments with speed and

quality.

re:Invent 2016 sessions on Continuous Delivery:

• DEV201 - DevOps on AWS: Accelerating Software Delivery with the AWS Developer

Tools

• CON302 - Development Workflow with Docker and Amazon ECS

• DEV403 - DevOps on AWS: Advanced Continuous Delivery Techniques

Resources to learn more:

• Continuous delivery: https://aws.amazon.com/devops/continuous-delivery/

• Continuous delivery for CloudFormation stacks -

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-

delivery-codepipeline.html

• CodePipeline - https://aws.amazon.com/documentation/codepipeline/

But wait, there’s more!

Thank you!

Remember to complete

your evaluations!