managing aws infrastructure using cloudformation
TRANSCRIPT
AWS User Group Norway
http://meetup.com/AWS-User-Group-Norway/
Managing AWS infrastructure using CloudFormation by Anton Babenko
23.11.2015Oslo, Norway
About the group
Anders BjørnestadAnton Babenko
Arne Solheim Olaf Skaug
“This is a group for people interested in Amazon Web Services. Anyone can participate, ranging from AWS evangelists to the curious. The main focus of the group is to build up a community around AWS with socializing and talks on topics like innovations, development and possibilities. Having trouble with a stack? Maybe you'll meet someone with a solution or an approach that you haven't already tried.”
AWS Certified SysOps,Sr. Software Eng at Telenor Digital linkedin.com/in/antonbabenko [email protected]
AWS Certified Architect and Developer, Sr. Consultant at Webstep linkedin.com/in/abjoerne [email protected]
Organizers:
Cloud Architect at Nordcloud Ltd
linkedin.com/in/olafskaug [email protected]
CTO at Nordeca Insight
linkedin.com/in/arnesolheim [email protected]
Today’s meeting1) News from AWS
2) Anton Babenko - Managing AWS infrastructure using CloudFormation
3) Arne Solheim - CloudfrontPizza and drinks sponsored by:
23NOV2015
Agenda:Basics of CloudFormation
Use cases
Best practices
Challenges and limitations
AWS CloudFormation
“... an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.”
https://aws.amazon.com/cloudformation/
{ "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "JSON string", "Metadata" : { template metadata }, "Parameters" : { set of parameters }, "Mappings" : { set of mappings }, "Conditions" : { set of conditions }, "Resources" : { set of resources - REQUIRED }, "Outputs" : { set of outputs }}
Template anatomy
● Collection of AWS resources
● Supports:
○ Create
○ Update
○ Describe
○ Delete
Stack
Manage resources
Bootstrapping of instances
Application deployment
Use cases
Manage AWS resources
Initial bootstrapping of instances
Use Ansible, Puppet, Chef or AWS OpsWorks for more real things
Application deployment
Consider using AWS alternatives also (CodeDeploy, Elastic Beanstalk, ECS)
Use cases
Use generator tool (JSON is not for humans)
Python: https://github.com/cloudtools/troposphere
Ruby: https://github.com/sparkleformation/sparkle_formation
Ruby: https://github.com/tongueroo/lono
Scala: https://github.com/MonsantoCo/cloudformation-template-generator
Manage AWS resources
Use generator tools - user-data escaping hell
Ruby example 2:
:UserData => base64(interpolate(file('userdata.sh')))
Ruby example 1 (erb template):
"UserData": { "Fn::Base64": { "Fn::Join": [ "", [ <%= user_data('userdata.sh.erb') %> ] ] }
Cloudformation JSON example 1:
"UserData": { "Fn::Base64": { "Fn::Join": [ "", ["#!/bin/bash -ex","\n", "yum update -y","\n", "# here is my long shell script. Imagine how long it can be :)"] ] }}
Cloudformation JSON example 2:
"UserData": "here is long & base64 encoded string. Imagine how long it can be :)"
Use Cloudformer to create templates based on existing AWS resources
CloudFormer: https://aws.amazon.com/developertools/6460180344805680
I have created AWS resources manually
CloudFormation Designer view of CloudFormer stack
Template: https://s3.amazonaws.com/cloudformation-templates-eu-west-1/CloudFormer.template
3 ways to organize stacks
Fit small deployments
Small size limit (50kb when load from local file; 450kb when load from S3)
1.Stand-alone stacks
Easy execution - aws-cli
Allow to put reusable parts into separate stacks
One fails = all fails = all rollback
Pass parameters to the internal stacks through parent
Reference to resources in parent: { "Fn::GetAtt" : [ "myVPCStack", "Outputs.VPCId" ] }
2.Nested stacks
Manual execution one after another
Pass outputs as inputs to downstream template
Pass parameters to each stack directly
Independent failure = independent rollback
Can’t use WaitContitions with other stacks
3.Pipelined stacks
Continuous Integration: Infrastructure & Application
Infrastructure:● Parametrize everything● Validate templates (AWS CLI)● Version templates● Probably run on separate AWS account first (CI)
Application deployment:● EC2 cloud-init + new application version = new deploy
CloudFormation challenges
● Can’t import already created resources without deleting them first● Not all AWS resources/features/services are supported by CloudFormation (eg, EC2 keypairs)● No officially supported CloudFormation generator available● No way to see what kind of changes are going to be applied● Failed state… what to do ?
○ Do not update resources created by CloudFormation manually
CloudFormation limitation
● JSON format is not very human-friendly● No iteration and limited conditional support (and/or/not/equal)● Limited ability to adjust stack based on dynamic conditions ● Managing dependencies between templates
○ Tieing together inputs/outputs is not directly supported
Summary
● Use JSON generators● Keep templates maintainable and single purposed● Probably start with stand-alone stack and iterate● Decide how you can handle failures (rollback just one stack or all)● Integrate with CI similar way how you do with your application● Use CloudFormation for very primitive application deployments
Available templates: https://aws.amazon.com/cloudformation/aws-cloudformation-templates/
CloudFormation Masterclass: http://www.slideshare.net/AmazonWebServices/aws-cloudformation-masterclass
(DVO304) AWS CloudFormation Best Practices: http://www.slideshare.net/AmazonWebServices/dvo304-aws-cloudformation-best-practices
Links
Questions ?
Thank you!
See you at DevOps Norway meetup 14th of December 2015:
Manage AWS infrastructure (as code) using Terraform
http://www.meetup.com/DevOps-Norway/events/226820193/