aws presents: infrastructure as code on aws - chefconf 2015

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Scott McDonald, Sr. Consultant

4/2/2015

Infrastructure as Code on AWSExample of Using CloudFormation to Deploy Chef Server 12 and Automatically Bootstrap Clients in AWS

Infrastructure as Code is….

• A technical domain revolving around building and managing infrastructure programmatically.

• A way to enable the reconstruction of the business from nothing but a source code repository, an application data backup, and bare metal resources.

• Your primary constraint should be the amount of time it takes to restore your application data.

AWS CloudFormation: Infrastructure as Code

AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion

First released in 2010

Amazon CloudFormation

• Infrastructure as Code

• Integrates with version control

• JSON formatted documents

• Templates for repeatable infrastructure

• Stacks of resources

• Supports AWS resource typesAWS CloudFormation

AWS CloudFormation: Infrastructure as Code

Document, version control, and share your applications and infrastructure as a JSON document

Provision app and other AWS resources (VPC, DynamoDB, RDS< EC2, Security Groups,) from a template

Repeatable, reliable deployments for test/dev/prod in any AWS Region

Resource Property Types

• AutoScaling• CloudFront• CloudWatch• DynamoDB• EC2• Elastic Beanstalk

• Elastic Load Balancer• IAM• RDS• S3• SNS/SQS

AWS CloudFormation: Application stack example (continue)

Template File Defining Stack

GitSubversionMercurial

Dev

Test

Prod

The entire application can be represented in an AWS CloudFormation template.

Use the version control system of your choice to store and track changes to this template

Build out multiple environments, such as for Development, Test, and Production using the template

This Example requires precreating IAM Roles

{ "AWSTemplateFormatVersion": "2010-09-09",

"Description": "Example CloudFormation to install Chef 12 Server using RHEL 6.5 ami in us-east-1. This template creates and starts a Chef 12 Server with the Web Management module (for up to 10 hosts), initializes knife in ec2-user account, and then uploads the aws cookbook to the running Chef 12 Server. Roles are used to create a private s3 bucket and upload a client validation key. A WaitCondition is used to pause the stack creation until the server is completely deployed. **WARNING** This template creates one or more Amazon EC2 instances. You will be billed for the AWS resources used if you create a stack from this template.",

"Parameters": { "KeyName": { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the Chef Server", "Type": "String", "MinLength": "1", "MaxLength": "255", "AllowedPattern" : "[\\x20-\\x7E]*", "ConstraintDescription" : "can contain only ASCII characters." }, "InstanceType" : { "Description" : "Chef 12 Server EC2 instance type", "Type" : "String", "Default" : "m3.large", "AllowedValues" : [ "t2.micro","t2.medium","m3.medium","m3.large","m3.xlarge","m3.2xlarge"], "ConstraintDescription" : "must be a valid EC2 instance type." },

CloudFormation Chef Server 12 Example 1/7

"ChefServerRole" : { "Description" : "Pre-create a Role - it needs at least S3 put/get", "Type" : "String" },

"SourceLocation" : { "Description" : "Source IP address range allowed SSH/Web to the Chef Server", "Type": "String", "MinLength": "9", "MaxLength": "18", "Default": "0.0.0.0/0", "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})", "ConstraintDescription": "must be a valid IP CIDR range of the form x.x.x.x/x." } },

"Mappings" : { "AWSRegion2AMI" : { "us-east-1" : { "id" : "ami-00a11e68" } } },


"Resources" : {

"ChefServer": { "Type": "AWS::EC2::Instance", "Metadata" : { "AWS::CloudFormation::Init" : { "config" : { "files" : { "/root/.aws/config" : { "content" : { "Fn::Join" : ["", [ "[default]\n", "region = us-east-1\n" ]]}, "mode" : "000600", "owner" : "root", "group" : "root" } } } } },


"Properties": { "SecurityGroups": [ { "Ref": "ChefServerSecurityGroup" } ], "IamInstanceProfile" : { "Ref" : "ChefServerRole" }, "ImageId": { "Fn::FindInMap": [ "AWSRegion2AMI", { "Ref": "AWS::Region" }, "id" ] }, "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [ "#!/bin/bash\n",

"export PATH=$PATH:/usr/local/bin:/opt/aws/bin\n",

"function error_exit\n", "{\n", " cfn-signal -e 1 -r \"$1\" '", { "Ref" : "ChefServerWaitHandle" }, "'\n", " exit 1\n", "}\n",

"curl https://bootstrap.pypa.io/ez_setup.py -o - | python\n", "easy_install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz\n", "cfn-init --region ", { "Ref" : "AWS::Region" }, " -s ", { "Ref" : "AWS::StackId" }, " -r ChefServer ", "|| error_exit 'Failed to run cfn-init'\n",

"# Bootstrap chef\n", "cd /home/ec2-user \n", "wget https://s3.amazonaws.com/awshat-chefcon2015/install-chef-aws.sh >> /tmp/install-chef-aws.log 2>&1 \n", "chmod +x /home/ec2-user/install-chef-aws.sh \n", "/home/ec2-user/install-chef-aws.sh >> /tmp/install-chef-aws.log 2>&1 \n",

"# Setup awscli on redhat\n", "curl https://bootstrap.pypa.io/get-pip.py -o - | python\n", "pip install awscli\n”,


"# use awscli to copy validation key to S3 bucket\n", "/usr/bin/aws s3 cp /home/ec2-user/.chef/chef-validator.pem s3://", {"Ref" : "ChefKeyBucket" } ,"/chef-validator.pem\n",

"# If all went well, signal success\n", "cfn-signal -e $? -r 'Chef Server configuration' '", { "Ref" : "ChefServerWaitHandle" }, "'\n" ]]}}, "KeyName": { "Ref": "KeyName" }, "InstanceType": { "Ref": "InstanceType" } } },

"ChefServerSecurityGroup" : { "Type" : "AWS::EC2::SecurityGroup", "Properties" : { "GroupDescription" : "Open up SSH/Web access to Chef Server from allowed Group and Source IP range", "SecurityGroupIngress" : [ { "IpProtocol": "tcp", "FromPort": "22", "ToPort": "22", "CidrIp": { "Ref" : "SourceLocation"} }, { "IpProtocol": "tcp", "FromPort": "443", "ToPort": "443", "SourceSecurityGroupName": { "Ref" :"ChefClientSecurityGroup" }}, { "IpProtocol": "tcp", "FromPort": "443", "ToPort": "443", "CidrIp": { "Ref" : "SourceLocation"} } ] } },


"ChefClientSecurityGroup" : { "Type" : "AWS::EC2::SecurityGroup", "Properties" : { "GroupDescription" : "Group with client access to Chef Server", "SecurityGroupIngress" : [ { "IpProtocol": "tcp", "FromPort": "22", "ToPort": "22", "CidrIp": { "Ref" : "SourceLocation"} }, { "IpProtocol": "tcp", "FromPort": "80", "ToPort": "80", "CidrIp": { "Ref" : "SourceLocation"} } ] } },

"ChefKeyBucket" : { "Type" : "AWS::S3::Bucket", "Properties" : { "AccessControl" : "Private" }, "DeletionPolicy" : "Delete" },

"ChefServerWaitHandle" : { "Type" : "AWS::CloudFormation::WaitConditionHandle" },


"ChefServerWaitCondition" : { "Type" : "AWS::CloudFormation::WaitCondition", "DependsOn" : "ChefServer", "Properties" : { "Handle" : { "Ref" : "ChefServerWaitHandle" }, "Timeout" : "7200" } } }, "Outputs" : { "ServerURL" : { "Description" : "URL of newly created Chef 12 server - login and change password", "Value" : { "Fn::Join" : ["", ["https://", {"Fn::GetAtt" : [ "ChefServer", "PublicDnsName" ]}, ":443/organizations/chef"]]} }, "ChefKeyBucket" : { "Description" : "Private S3 bucket with validation key for client bootstrap automation:", "Value" : {"Ref" : "ChefKeyBucket" } }, "ChefSecurityGroup" : { "Description" : "EC2 Security Group for access to Chef 12 Server", "Value" : { "Ref" :"ChefClientSecurityGroup" } } } }


Run CloudFormation template in AWS Console

Enter parameters for CloudFormation template

Using CloudFormation from the CLI

685b358e3054:ChefCon2015 smcdon$ pwd/Users/smcdon/Documents/ChefCon2015

685b358e3054:ChefCon2015 smcdon$ cat add.server

aws cloudformation create-stack --stack-name chef12-server5 --template-body file://./server.cfn --disable-rollback --parameters \ParameterKey=InstanceType,ParameterValue=m3.2xlarge \ParameterKey=KeyName,ParameterValue=awshat_key01 \ParameterKey=ChefServerRole,ParameterValue=devops \

Deploying Chef Server 12 via CloudFormation

685b358e3054:ChefCon2015 smcdon$ ./add.server

{ "StackId": "arn:aws:cloudformation:us-east-1:162012790422:stack/chef12-server5/f30bbc20-d48c-11e4-a271-50443313686e"}

685b358e3054:ChefCon2015 smcdon$

Infrastructure as Code under Version Control!

685b358e3054:ChefCon2015 smcdon$ git commit -m "ChefCon2015 presentation commit example"[master ccf204b] ChefCon2015 presentation commit example 1 file changed, 1 insertion(+), 1 deletion(-)

685b358e3054:ChefCon2015 smcdon$ git push origin masterCounting objects: 5, done.Delta compression using up to 4 threads.Compressing objects: 100% (3/3), done.Writing objects: 100% (3/3), 316 bytes | 0 bytes/s, done.Total 3 (delta 2), reused 0 (delta 0)To https://github.com/awshat/chefcon2015.git 190668d..ccf204b master -> master

685b358e3054:ChefCon2015 smcdon$

Watch CloudFormation build in AWS Console

CloudFormation Outputs when Completed

{

"AWSTemplateFormatVersion": "2010-09-09",

"Description": "Sample template to bring up a redhat linux ec2 instance and bootstrap a client node to be managed by an existing Chef Server. **WARNING** This template creates an EC2 instance. You will be billed for the AWS resources used if you create a stack from this template.",

"Parameters": { "KeyName": { "Type": "String", "Description" : "EC2 KeyPair to enable SSH access to the client instance" }, "InstanceType": { "Default": "m3.medium", "Description" : "Type of EC2 instance for the client node", "Type": "String", "AllowedValues" : [ "t2.micro", "t2.medium", "m3.small", "m3.medium", "m3.large", "m3.xlarge", "m3.2xlarge"], "ConstraintDescription" : "must contain only alphanumeric characters." },

CloudFormation for Chef Clients in AWS 1/3

"ServerURL" : { "Description" : "Chef 12 Server URL", "Type": "String" }, "ChefSecurityGroup" : { "Description" : "Security group for clients to get access to Chef Server", "Type": "String" }, "S3Role" : { "Description" : "IAM S3 Role with Get access for chef client bootstrapping automation", "Type" : "String" }, "ChefKeyBucket" : { "Description" : "S3 bucket with validation key", "Type": "String" }, "ChefClientEnv" : { "Description" : "Environment setting for deployed instances", "Type": "String", "Default" : "_default" }},


"Mappings" : { "AWSRegion2AMI" : { "us-east-1" : { "id" : "ami-00a11e68" } }},"Resources" : {"ChefClient": { "Type": "AWS::EC2::Instance", "Properties": { "SecurityGroups": [ { "Ref": "ChefSecurityGroup" } ], "IamInstanceProfile" : { "Ref" : "S3Role" }, "ImageId": { "Fn::FindInMap": [ "AWSRegion2AMI", { "Ref": "AWS::Region" }, "id" ] }, "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [ "#!/bin/bash\n",

"# Bootstrap chef client\n", "cd /home/ec2-user \n", "wget https://s3.amazonaws.com/awshat-chefcon2015/install-chef-client.sh >> /tmp/install-chef-amzn.log 2>&1 \n", "chmod +x /home/ec2-user/install-chef-client.sh \n", "/home/ec2-user/install-chef-client.sh ", {"Ref" : "ChefKeyBucket" } ," ", {"Ref" : "ServerURL" } ," ", {"Ref" : "ChefClientEnv" } ," >> /tmp/install-chef-amzn.log 2>&1 \n" ]]}}, "KeyName": { "Ref": "KeyName" }, "InstanceType": { "Ref": "InstanceType" } } } }}


Enter parameters for CloudFormation in GUI

Create a Version Controlled Client Script

685b358e3054:ChefCon2015 smcdon$ cp add.client4 add.client5685b358e3054:ChefCon2015 smcdon$ vi add.client5 685b358e3054:ChefCon2015 smcdon$ cat add.client5aws cloudformation create-stack --stack-name $1 --template-body file://./client.cfn --disable-rollback --parameters \ParameterKey=KeyName,ParameterValue=awshat_key01 \ParameterKey=InstanceType,ParameterValue=m3.large \

ParameterKey=ServerURL,ParameterValue=https://ec2-52-1-228-71.compute-1.amazonaws.com:443/organizations/chef \ParameterKey=ChefKeyBucket,ParameterValue=chef12-server5-chefkeybucket-zut8xoz6apsn \ParameterKey=ChefSecurityGroup,ParameterValue=chef12-server5-ChefClientSecurityGroup-NIR4XTVU1POF \ParameterKey=S3Role,ParameterValue=s3access

685b358e3054:ChefCon2015 smcdon$ git add -A685b358e3054:ChefCon2015 smcdon$ git commit -m "ChefCon2015 example client script for presentation"[master f655195] ChefCon2015 example client script for presentation 1 file changed, 8 insertions(+) create mode 100755 add.client5685b358e3054:ChefCon2015 smcdon$ git push origin masterCounting objects: 4, done.Delta compression using up to 4 threads.Compressing objects: 100% (3/3), done.Writing objects: 100% (3/3), 599 bytes | 0 bytes/s, done.Total 3 (delta 1), reused 0 (delta 0)To https://github.com/awshat/chefcon2015.git ccf204b..f655195 master -> master

Use CloudFormation to Bootstrap Clients

685b358e3054:ChefCon2015 smcdon$ ./add.client5 server5-client1{ "StackId": "arn:aws:cloudformation:us-east-1:162012790422:stack/server5-client1/73936ff0-d497-11e4-8f0c-50e2416294a8"}685b358e3054:ChefCon2015 smcdon$ ./add.client5 server5-client2{ "StackId": "arn:aws:cloudformation:us-east-1:162012790422:stack/server5-client2/75a5fc40-d497-11e4-bf18-50e24162947c"}685b358e3054:ChefCon2015 smcdon$ ./add.client5 server5-client3{ "StackId": "arn:aws:cloudformation:us-east-1:162012790422:stack/server5-client3/77368750-d497-11e4-adea-50018ffe9e62"}

Watch CloudFormation for Clients in Console

Watch Clients show up in Chef 12 Webmanager

CloudFormation = Easy Teardown

• Fully automated

• Delete resources

• Complete teardown

• Entire stacks are disposable

• For this Chef 12 Example – Deleting clients in

CloudFormation automatically deregisters nodes

from Chef Server managementAWS CloudFormation

Teardown Chef Server via CloudFormation

Watch Teardown in CloudFormation Console

Confirm deleted resources gone in console

Teardown for Chef Clients in AWS Console

Teardown both easily in AWS Console

Example Code: https://github.com/awshat/chefcon2015

Thank you & Have fun!

Scott McDonald - [email protected]

aws presents: infrastructure as code on aws - chefconf 2015

Technology

aws cloudformation template

code aws cloudformation

aws example

cloudformation chef

aws cookbook

aws region

example cloudformation

aws resources vpc