managing aws infrastructure using cloudformation

AWS User Group Norway

http://meetup.com/AWS-User-Group-Norway/

Managing AWS infrastructure using CloudFormation by Anton Babenko

23.11.2015Oslo, Norway

About the group

Anders BjørnestadAnton Babenko

Arne Solheim Olaf Skaug

“This is a group for people interested in Amazon Web Services. Anyone can participate, ranging from AWS evangelists to the curious. The main focus of the group is to build up a community around AWS with socializing and talks on topics like innovations, development and possibilities. Having trouble with a stack? Maybe you'll meet someone with a solution or an approach that you haven't already tried.”

AWS Certified SysOps,Sr. Software Eng at Telenor Digital linkedin.com/in/antonbabenko anton@antonbabenko.com

AWS Certified Architect and Developer, Sr. Consultant at Webstep linkedin.com/in/abjoerne anders.bjornestad@webstep.no

Organizers:

Cloud Architect at Nordcloud Ltd

linkedin.com/in/olafskaug olaf.skaug@nordcloud.com

CTO at Nordeca Insight

linkedin.com/in/arnesolheim arne.solheim@nordeca.com

Today’s meeting1) News from AWS

2) Anton Babenko - Managing AWS infrastructure using CloudFormation

3) Arne Solheim - CloudfrontPizza and drinks sponsored by:

23NOV2015

Agenda:Basics of CloudFormation

Use cases

Best practices

Challenges and limitations

AWS CloudFormation

“... an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.”

https://aws.amazon.com/cloudformation/

{ "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "JSON string", "Metadata" : { template metadata }, "Parameters" : { set of parameters }, "Mappings" : { set of mappings }, "Conditions" : { set of conditions }, "Resources" : { set of resources - REQUIRED }, "Outputs" : { set of outputs }}

Template anatomy

● Collection of AWS resources

● Supports:

○ Create

○ Update

○ Describe

○ Delete

Stack

Manage resources

Bootstrapping of instances

Application deployment

Use cases

Manage AWS resources

Initial bootstrapping of instances

Use Ansible, Puppet, Chef or AWS OpsWorks for more real things

Application deployment

Consider using AWS alternatives also (CodeDeploy, Elastic Beanstalk, ECS)

Use cases

Use generator tool (JSON is not for humans)

Python: https://github.com/cloudtools/troposphere

Ruby: https://github.com/sparkleformation/sparkle_formation

Ruby: https://github.com/tongueroo/lono

Scala: https://github.com/MonsantoCo/cloudformation-template-generator

Manage AWS resources

Use generator tools - user-data escaping hell

Ruby example 2:

:UserData => base64(interpolate(file('userdata.sh')))

Ruby example 1 (erb template):

"UserData": { "Fn::Base64": { "Fn::Join": [ "", [ <%= user_data('userdata.sh.erb') %> ] ] }

Cloudformation JSON example 1:

"UserData": { "Fn::Base64": { "Fn::Join": [ "", ["#!/bin/bash -ex","\n", "yum update -y","\n", "# here is my long shell script. Imagine how long it can be :)"] ] }}

Cloudformation JSON example 2:

"UserData": "here is long & base64 encoded string. Imagine how long it can be :)"

Use Cloudformer to create templates based on existing AWS resources

CloudFormer: https://aws.amazon.com/developertools/6460180344805680

I have created AWS resources manually

CloudFormation Designer view of CloudFormer stack

Template: https://s3.amazonaws.com/cloudformation-templates-eu-west-1/CloudFormer.template

3 ways to organize stacks

Fit small deployments

Small size limit (50kb when load from local file; 450kb when load from S3)

1.Stand-alone stacks

Easy execution - aws-cli

Allow to put reusable parts into separate stacks

One fails = all fails = all rollback

Pass parameters to the internal stacks through parent

Reference to resources in parent: { "Fn::GetAtt" : [ "myVPCStack", "Outputs.VPCId" ] }

2.Nested stacks

Manual execution one after another

Pass outputs as inputs to downstream template

Pass parameters to each stack directly

Independent failure = independent rollback

Can’t use WaitContitions with other stacks

3.Pipelined stacks

Continuous Integration: Infrastructure & Application

Infrastructure:● Parametrize everything● Validate templates (AWS CLI)● Version templates● Probably run on separate AWS account first (CI)

Application deployment:● EC2 cloud-init + new application version = new deploy

CloudFormation challenges

● Can’t import already created resources without deleting them first● Not all AWS resources/features/services are supported by CloudFormation (eg, EC2 keypairs)● No officially supported CloudFormation generator available● No way to see what kind of changes are going to be applied● Failed state… what to do ?

○ Do not update resources created by CloudFormation manually

CloudFormation limitation

● JSON format is not very human-friendly● No iteration and limited conditional support (and/or/not/equal)● Limited ability to adjust stack based on dynamic conditions ● Managing dependencies between templates

○ Tieing together inputs/outputs is not directly supported

Summary

● Use JSON generators● Keep templates maintainable and single purposed● Probably start with stand-alone stack and iterate● Decide how you can handle failures (rollback just one stack or all)● Integrate with CI similar way how you do with your application● Use CloudFormation for very primitive application deployments

Available templates: https://aws.amazon.com/cloudformation/aws-cloudformation-templates/

CloudFormation Masterclass: http://www.slideshare.net/AmazonWebServices/aws-cloudformation-masterclass

(DVO304) AWS CloudFormation Best Practices: http://www.slideshare.net/AmazonWebServices/dvo304-aws-cloudformation-best-practices

Links

Questions ?

Thank you!

See you at DevOps Norway meetup 14th of December 2015:

Manage AWS infrastructure (as code) using Terraform

http://www.meetup.com/DevOps-Norway/events/226820193/