self service agile infrastructure for product teams - pop-up loft tel aviv
TRANSCRIPT
Automated Agile Infrastructure Do It Yourself
Oron AdamCTO & ArchitectEmind Cloud Experts
@oronoa
Started as a Developer
CTO of Emind Cloud Experts
Love to code AWS infrastructure and other things as well ...
About Myself
Key Challenges for Ops of Agile Product Teams● Dev, Staging & Production Parity● Let Dev experiment and fail fast● Innovate while reducing risk● Continuous uninterrupted delivery of service (and new features) ● Feedback loops, automation, metrics and goals
Dev/Prod Parity● Keep development staging and production as identical as possible● Backing services same across environments
Server ImmutabilityPhoenix servers
Fast Server Deployment
Starting from machine images
Fails sometimes, not a big deal
It’s testable and thus more stable
Automation CriticalNeeds to be 100% reproducible
Configure Server and SW automatically
Be able test success / failure of server startup
AWS CloudFormation● Declarative Language ● Integrates with git or any VC system ● Simple JSON Format● Supports Templates and Stacks● Supports All AWS resource types● Part of the CI/CD Pipeline
AWS Cloud revolution
>>> from troposphere import Ref, Template
>>> import troposphere.ec2 as ec2
>>> t = Template()
>>> instance = ec2.Instance("myinstance")
>>> instance.ImageId = "ami-951945d0"
>>> instance.InstanceType = "t1.micro"
>>> t.add_resource(instance)
<troposphere.ec2.Instance object at 0x101bf3390>
>>> print(t.to_json())
{
"Resources": {
"myinstance": {
"Properties": {
"ImageId": "ami-951945d0",
"InstanceType": "t1.micro"
},
"Type": "AWS::EC2::Instance"
}
}
}
Can be programmatically generated
Cloud Formation Dev Workflow
Code infrastructure parallel to App
Create and Test
Iterate
Bootstrapping Applications & Handling Update"Resources" : {
"Ec2Instance" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"KeyName" : { "Ref" : "KeyName" },
"SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ],
"ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]},
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["",[
"#!/bin/bash -ex","\n",
"yum -y install gcc-c++ make","\n",
"yum -y install mysql-devel sqlite-devel","\n",
"yum -y install ruby-rdoc rubygems ruby-mysql ruby-devel","\n",
"gem install --no-ri --no-rdoc rails","\n",
"gem install --no-ri --no-rdoc mysql","\n",
"gem install --no-ri --no-rdoc sqlite3","\n",
"rails new myapp","\n",
"cd myapp","\n",
"rails server -d","\n"]]}}
}
}
CloudFormation app deployment best practicesUse AWS::CloudFormation::Init
Use IAM roles to securely download software and data
Use Amazon CloudWatch logs for debugging
Use cfn-hup for updates
Use custom AMIs to minimize application boot times
Bootstrapping Applications & Handling Update
"Resources": {
"MyInstance": {
"Type": "AWS::EC2::Instance",
"Metadata" : {
"AWS::CloudFormation::Init" : {
"config" : {
"packages" : {
:
},
"groups" : {
:
},
"users" : {
:
},
"sources" : {
:
},
"files" : {
:
},
"commands" : {
:
},
"services" : {
:
}
}
}
},
"Properties": {
:
}
}
}
AWS::CloudFormation::Init
"files" : {
"/tmp/setup.mysql" : {
"content" : { "Fn::Join" : ["", [
"CREATE DATABASE ", { "Ref" : "DBName" }, ";\n",
"CREATE USER '", { "Ref" : "DBUsername" }, "'@'localhost' IDENTIFIED BY '",
{ "Ref" : "DBPassword" }, "';\n",
"GRANT ALL ON ", { "Ref" : "DBName" }, ".* TO '", { "Ref" : "DBUsername" },
"'@'localhost';\n",
"FLUSH PRIVILEGES;\n"
]]},
"mode" : "000644",
"owner" : "root",
"group" : "root"
}
},
Creating Files
"rpm" : {
"epel" : "http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm"
},
"yum" : {
"httpd" : [],
"php" : [],
"wordpress" : []
},
"rubygems" : {
"chef" : [ "0.10.2" ]
}
Installing Packages
"services" : {
"sysvinit" : {
"nginx" : {
"enabled" : "true",
"ensureRunning" : "true",
"files" : ["/etc/nginx/nginx.conf"],
"sources" : ["/var/www/html"]
},
"php-fastcgi" : {
"enabled" : "true",
"ensureRunning" : "true",
"packages" : { "yum" : ["php", "spawn-fcgi"] }
},
"sendmail" : {
"enabled" : "false",
"ensureRunning" : "false"
}
}
}
Setting up running services
"sources" : {
"/etc/myapp" : "https://s3.amazonaws.com/mybucket/myapp.tar.gz"
}
"sources" : {
"/etc/puppet" : https://github.com/user1/cfn-demo/tarball/master
}
Downloading the app
AWS Cloud Formation
CF Stack EC2 Instance
cfn-init
Get MetadataRun actions
cfn-init
AWS Cloud Formation CF Stack EC2 Instance
cfn-signal
send signal on setup done
cfn-signal
"AutoScalingGroup": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
"AvailabilityZones": { "Fn::GetAZs": "" },
"LaunchConfigurationName": { "Ref": "LaunchConfig" },
"DesiredCapacity": "3",
"MinSize": "1",
"MaxSize": "4"
},
"CreationPolicy": {
"ResourceSignal": {
"Count": "3",
"Timeout": "PT15M"
}
},
"UpdatePolicy" : {
"AutoScalingScheduledAction" : {
"IgnoreUnmodifiedGroupSizeProperties" : "true"
},
"AutoScalingRollingUpdate" : {
"MinInstancesInService" : "1",
"MaxBatchSize" : "2",
"PauseTime" : "PT1M",
"WaitOnResourceSignals" : "true"
}
}
},
"LaunchConfig": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"ImageId": "ami-16d18a7e",
"InstanceType": "t2.micro",
"UserData": {
"Fn::Base64": {
"Fn::Join" : [ "", [
"#!/bin/bash -xe\n",
"yum update aws-cfn-bootstrap\n",
"/opt/aws/bin/cfn-signal -e 0 --stack ", { "Ref": "AWS::StackName" },
" --resource AutoScalingGroup ",
" --region ", { "Ref" : "AWS::Region" }, "\n"
] ]
}
}
}
}
Wait for completion withCreation Policy
Cloud Formation Functions"Properties" : {
"MyMyLBDNSName" : {
"Fn::GetAtt" : [ "MyLoadBalancer", "DNSName" ]
}
}
Fn::Base64
Conditional Functions
Fn::And, Fn::Equals, Fn::If, Fn::Not, Fn::Or
Fn::FindInMap
Fn::GetAtt
Fn::GetAZs
Fn::Join
Fn::Select
Ref
External Wait Conditions"myWaitHandle" : {
"Type" : "AWS::CloudFormation::WaitConditionHandle",
"Properties" : {
}
}
"myWaitCondition" : {
"Type" : "AWS::CloudFormation::WaitCondition",
"DependsOn" : "Ec2Instance",
"Properties" : {
"Handle" : { "Ref" : "myWaitHandle" },
"Timeout" : "3600"
}
}
External Wait Conditions"UserData" : {
"Fn::Base64" : {
"Fn::Join" : [ "", ["SignalURL=", { "Ref" : "myWaitHandle" } ] ]
}
}
curl -X PUT -H 'Content-Type:' --data-binary '{"Status" : "SUCCESS","Reason" : "Configuration Complete","
UniqueId" : "ID1234","Data" : "Application has completed configuration."}' "https://cloudformation-
waitcondition-test.s3.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-1%3A034017226601%3Astack%2Fstack-
gosar-20110427004224-test-stack-with-WaitCondition--VEYW%2Fe498ce60-70a1-11e0-81a7-5081d0136786%
2FmyWaitConditionHandle?
Expires=1303976584&AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&Signature=ik1twT6hpS4cgNAw7wyOoRejVoo%3D"
SignalURL
AWS Cloud Formation CF StackB
cfn-signal
WaitCondition Count : 2
cfn-signal
cfn-signal
A
"Parameters" : { "EnvType" : { "Description" : "Environment type.", "Default" : "test", "Type" : "String", "AllowedValues" : ["prod", "dev", "test"], "ConstraintDescription" : "must specify prod, dev, or test." } },
Parameters for Dev / Prod / Test
"Conditions" : {
"CreateProdResources" : {"Fn::Equals" : [{"Ref" : "EnvType"}, "prod"]},
"CreateDevResources" : {"Fn::Equals" : [{"Ref" : "EnvType"}, "dev"]}
},
"Resources" : {
"EC2Instance" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]},
"InstanceType" : { "Fn::If" : [
"CreateProdResources",
"c1.xlarge",
{"Fn::If" : [
"CreateDevResources",
"m1.large",
"m1.small"
]}
]}
}
},
Conditionals
AWS Cloud FormationTemplate
Prod Stack
Dev, Stage, Prod
Dev Stack
Prod
Dev
Customer Stories
Management VPC peered to all other VPCs
Per Customer VPC
Need to be able to quickly setup new accounts
Per Environment Application Stack
for each dev, staging, production envs
Services Oriented Architecture (SOA) in the insurance industry.
Cloud Formation
management VPC
Access VPC
VPC for dev, staging, prod
Lambda - CIDR and subnet calculations
Peering of VPCs requires inter stack communication via outputs
"NetworkSubnetFunction": { "Type": "AWS::Lambda::Function", "DependsOn": [ "LambdaBasicExecutionRole", "LambdaBasicExecutionPolicy" ], "Properties": { "Code": { "S3Bucket": "ENTER BUCKER NAME HERE", "S3Key": "network/get_subnets_from_cidr.zip" }, "Role": { "Fn::GetAtt": ["LambdaBasicExecutionRole", "Arn"] }, "Timeout": 60, "Handler": "get_subnets_from_cidr.get_subnets_from_cidr", "Runtime": "python2.7", "MemorySize": 128 } },
def get_subnets_from_cidr(event, context): """ Calculates subnets for a given CIDR and returns the result as a Custom Resource Response Object to CloudFormation.
Arguments (encapsulated in the 'event' argument by CloudFormation):
cidr - the CIDR from which to extract the subnets in slash notation ('172.16.0.0/16'). number_of_subnets - the number of subnets to extract. Must be at least 1. subnet_size - the number of hosts in each extracted subnet. Must be a power of 2. start_from - return the desired number of subnets starting at this subnet number. Optional.
Example: event = { ... "ResourceProperties": { "cidr": "172.16.0.0/16", "number_of_subnets": "4", "subnet_size": "256", "start_from": "3" } }
result = { "Id0": "172.16.2.0/24", "Id1": "172.16.3.0/24", "Id2": "172.16.4.0/24", "Id3": "172.16.5.0/24" }
"GetVpcSubnetsCustomEvent": { "Type": "Custom::GetVpcSubnetsCustomEvent", "Properties": { "ServiceToken": { "Fn::GetAtt" : ["NetworkSubnetFunction", "Arn"] }, "cidr": { "Ref": "DevelopmentVPCCIDR" }, "number_of_subnets": 15, "subnet_size": 256 } },
"WebDevelopmentSubnetA": {"Type": "AWS::EC2::Subnet","DeletionPolicy": "Delete","Properties": {
"VpcId": {"Ref": "DevelopmentVPC"
},"MapPublicIpOnLaunch": "true","AvailabilityZone": {
"Fn::GetAtt": ["StackOutputInfo", "AZ1"] },
"CidrBlock": { "Fn::GetAtt" : ["GetVpcSubnetsCustomEvent", "Id0"] },"Tags": [{
"Key": "Name","Value": {
"Fn::Join" : [ "", [ "development-web-", { "Fn::GetAtt": ["StackOutputInfo", "AZ1"] } ] ] }
}]}
},
What packer fixes?Images are produced from templates
Images are maintainable
Images are cross cloud provider
Automated, Repeatable and Fast
Can also produce Vagrant Boxes for Dev
packer use casesContinuous Delivery
Dev / Prod Parity
Fast scaling out with non vanilla images
Demo images for multiple cloud providers
packer providersAWS
GCE
Digital Ocean
VMWare
VirtualBox / Vagrant
Docker
packer & CloudFormation togetherPre bake images with packer form cmd line
Can be easily initiated from CI server
parameterize AMI ids and give to CF stack run
packer & CloudFormation togetherAMI is built at pack time, not scale timeCan still use CloudInit to build, just happens at pack-time, not scale timeNo reliance on any external services to bring another server onlineUpdated AMI is a command-line statement awayCan pack local VMs using identical code
Customer Stories
Using TerraForm for agile scrum environments
Env is created for scum duration
Env destroyed when done
TerraFormExecution Plans - see what’s going to happen
Relationships & Dependencies - reference other resources
State - get resources to desired state
Multiple Providers - on many clouds
TerraForm Basicsprovider "aws" {
access_key = "ACCESS_KEY_HERE"
secret_key = "SECRET_KEY_HERE"
region = "us-east-1"
}
resource "aws_instance" "example" {
ami = "ami-408c7f28"
instance_type = "t1.micro"
}
TerraForm Basicsresource "aws_elb" "frontend" {
name = "frontend-load-balancer"
listener {
instance_port = 8000
instance_protocol = "http"
lb_port = 80
lb_protocol = "http"
}
instances = ["${aws_instance.app.*.id}"]
}
resource "aws_instance" "app" {
count = 5
ami = "ami-408c7f28"
instance_type = "t1.micro"
}
TerraForm Plan$ terraform plan
...
+ aws_instance.example
ami: "" => "ami-408c7f28"
availability_zone: "" => "<computed>"
instance_type: "" => "t1.micro"
key_name: "" => "<computed>"
private_dns: "" => "<computed>"
private_ip: "" => "<computed>"
public_dns: "" => "<computed>"
public_ip: "" => "<computed>"
security_groups: "" => "<computed>"
subnet_id: "" => "<computed>"
Terraform Apply$ terraform apply
aws_instance.example: Creating...
ami: "" => "ami-408c7f28"
instance_type: "" => "t1.micro"
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
...
Terraform Show (State)$ terraform show
aws_instance.example:
id = i-e60900cd
ami = ami-408c7f28
availability_zone = us-east-1c
instance_type = t1.micro
key_name =
private_dns = domU-12-31-39-12-38-AB.compute-1.internal
private_ip = 10.200.59.89
public_dns = ec2-54-81-21-192.compute-1.amazonaws.com
public_ip = 54.81.21.192
security_groups.# = 1
security_groups.0 = default
subnet_id = ….
Key TakeawaysIt’s all code - Infra just as app
Automate all the things
Leverage the power of Lambda and CloudFormation
Talk to us, we can help...
Thank you