vbacd - introduction to opscode chef - 2/29

Automating Cloud Deployments with Chef

matt@opscode.comwww.opscode.com

Twitter/IRC/GitHub: mattray

•U has a cloud

•Now what?

http://www.flickr.com/photos/ian_munroe/4758240536/

Congratulations!!!Congratulations!!!

But then what?But then what?

http://www.flickr.com/photos/doctorow/2698336843

You need to configure them

APIs are awesome!APIs are awesome!

•You can provision compute resources in seconds

•You can provision storage resources in seconds

•That’s cool.

http://www.flickr.com/photos/jdhancock/3634246981/

Chef can help with thatChef can help with that

•knife ec2 server create

•knife rackspace server create

•knife terremark server create

•knife voxel server create

•knife cloudstack server create

•knife kvm server create

•knife vsphere server create

•knife eucalyptus server create

•knife openstack server create

http://www.flickr.com/photos/kyz/3122499444/

See NodeSee Node

Application Server

See NodesSee Nodes

Application Server

Application Database

See Nodes GrowSee Nodes Grow

Application Server

Application Databases

Application Servers

Application Databases

See Nodes GrowSee Nodes Grow

Application Servers

Application Databases

Load Balancer

See Nodes GrowSee Nodes Grow

See Nodes GrowSee Nodes Grow

Application Servers

Application Databases

Load Balancers

See Nodes GrowSee Nodes Grow

Application Servers

Application Database Cache

Load Balancers

Application Databases

Tied together with Config

Tied together with Config

Application Servers

Application Database Cache

Load Balancers

Application Databases

Infrastructure is a Snowflake

Infrastructure is a Snowflake

Application Servers

Application Database Cache

Load Balancers

Floating IP?

Application Databases

Evolving ComplexityEvolving Complexity

Load Balancers

Application Servers

NoSQL

Database Slaves

Application Cache

Database Cache

Database

Complexity Grows Quickly

Complexity Grows Quickly

DC1

DC3

DC2

http://www.flickr.com/photos/16339684@N00/2681435235/

And it Continues to EvolveAnd it Continues to Evolve

Golden Images are not the answerGolden Images are not the answer

•Gold is heavy

•Hard to transport

•Hard to mold

•Easy to lose configuration detail

http://www.flickr.com/photos/garysoup/2977173063/

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

Typical Boring InfrastructureTypical Boring Infrastructure

•Move SSH off port 22

•Lets put it on 2022

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

New Compliance MandateNew Compliance Mandate

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite 1 2

3

4

5

6

•edit /etc/ssh/sshd_config

6 Golden Image Updates6 Golden Image Updates

•Delete, launch

•Repeat

•Typically manually

8

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite 1 2

3 4 5 6 7

9

10 11

12

12 Instance Replacements12 Instance Replacements

5

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite 1 2

4 5 6 7

8 9

10 11

12

3

•Don't break anything!

In a Maintenance WindowIn a Maintenance Window

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

• Invalid Configs

•Bob just got fired :(

With Different IP Addresses?

With Different IP Addresses?

http://www.flickr.com/photos/francoforeshock/5716969942/

Configuration DesperationConfiguration Desperation

Configuration Management and Automated Systems Integration

is the Answer

http://www.flickr.com/photos/philliecasablanca/3354734116/

Chef Solves This ProblemChef Solves This Problem

•But you already guessed that, didn’t you?

Collections of ResourcesCollections of Resources

•Networking

•Files

•Directories

•Symlinks

•Mounts

•Routes

•Users

•Groups

•Tasks

•Packages

•Software

•Services

•Configurations

•Other Stuff

http://www.flickr.com/photos/stevekeys/3123167585/

http://www.flickr.com/photos/glowjangles/4081048126/

Acting in ConcertActing in Concert

http://www.flickr.com/photos/28309157@N08/3743455858/

To Provide a ServiceTo Provide a Service

Chef is Infrastructure as Code

Chef is Infrastructure as Code

http://www.flickr.com/photos/louisb/4555295187/

•Programmatically provision and configure

•Treat like any other code base

•Reconstruct business from code repository, data backup, and bare metal resources.

Declarative Interface to Resources

Declarative Interface to Resources

•Define policy

•Say what, not how

•Pull not Push

http://www.flickr.com/photos/bixentro/2591838509/

That looks like thisThat looks like this

extra_packages = case node['platform'] when "ubuntu","debian" %w{ ruby1.8 ruby1.8-dev rdoc1.8 ri1.8 libopenssl-ruby } endextra_packages.each do |pkg| package pkg do action :install endend

Or thisOr this

search(:users, '*:*') do |u| user u['id'] do uid u['uid'] shell u['shell'] home "/home/#{u['id']}" end directory "#{home_dir}/.ssh" do owner u['id'] group u['gid'] mode "0700" end template "#{home_dir}/.ssh/authorized_keys" do source "authorized_keys.erb" owner u['id'] group u['id'] mode "0600" variables :ssh_keys => u['ssh_keys'] endend

Recipes and CookbooksRecipes and Cookbooks

•Recipes are collections of Resources

•Cookbooks contain recipes, templates, files, custom resources, etc

•Code re-use and modularity

•Hundreds already on Community.opscode.com

http://www.flickr.com/photos/shutterhacks/4474421855/

•Chef-Client generates configurations directly on nodes from their run list

•Reduce management complexity through abstraction

•Store the configuration of your programs in version control

http://www.flickr.com/photos/ssoosay/5126146763/

NodesNodes

Upload your infrastructureUpload your infrastructureknife cookbook upload chef-client

knife cookbook upload java

knife cookbook upload jpackage

knife cookbook upload ntp

knife cookbook upload sudo

knife cookbook upload tomcat

knife cookbook upload users

knife cookbook upload sample

knife role from file base.rb

knife role from file tc.rb

knife role from file sample.rb

knife data bag create users

knife data bag from file users mray.json

Build it somewhereBuild it somewhere

#EC2

knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-a7a97dce -f m1.small -d omnibus -r 'role[base],role[tc],role[sample]’

#Rackspace

knife rackspace server create --image 110 --flavor 2 -i ~/.ssh/mray.pem -d omnibus -r 'role[base],role[tc],role[sample]’

#CloudStack

knife cs server create -S "small instance" -T "CentOS 5.5(64-bit) no GUI (KVM)" -i ~/.ssh/mray.pem -d omnibus -r 'role[base],role[tc],role[sample]’

#Ubuntu Linux VM

knife bootstrap test.lab -i ~/.ssh/mray.pem -x ubuntu --sudo -d omnibus -r 'role[base],role[tc],role[sample]'

Tomcat stack deployedTomcat stack deployedec2-107-21-179-169.compute-1.amazonaws.com [Thu, 23 Feb 2012 03:16:27 +0000] INFO: Chef Run complete in 125.548799554 seconds

ec2-107-21-179-169.compute-1.amazonaws.com [Thu, 23 Feb 2012 03:16:27 +0000] INFO: Running report handlers

ec2-107-21-179-169.compute-1.amazonaws.com [Thu, 23 Feb 2012 03:16:27 +0000] INFO: Report handlers complete

Instance ID: i-ee18148b

Flavor: m1.small

Image: ami-0c6ebd65

Region: us-east-1

Availability Zone: us-east-1b

Security Groups: default

SSH Key: mray

Root Device Type: instance-store

Public DNS Name: ec2-107-21-179-169.compute-1.amazonaws.com

Public IP Address: 107.21.179.169

Private DNS Name: ip-10-120-255-91.ec2.internal

Private IP Address: 10.120.255.91

Environment: _default

Run List: role[base], role[tc], role[sample]

Tomcat stack deployedTomcat stack deployed

Update the sample cookbookUpdate the sample cookbook

$ knife cookbook upload sample

Uploading sample [0.0.2]

upload complete

Update the nodesUpdate the nodes$ knife ssh "role:base" "sudo chef-client" -i ~/.ssh/mray.pem -x ubuntu -a cloud.public_ipv4107.21.179.169 [Thu, 23 Feb 2012 03:30:22 +0000] INFO: *** Chef 0.10.8 ***107.21.179.169 [Thu, 23 Feb 2012 03:30:30 +0000] INFO: Run List is [role[base], role[tc], role[sample]]107.21.179.169 [Thu, 23 Feb 2012 03:30:30 +0000] INFO: Run List expands to [chef-client, ntp, sudo, users::sysadmins, java, tomcat, sample]107.21.179.169 [Thu, 23 Feb 2012 03:30:30 +0000] INFO: Starting Chef Run for i-ee18148b107.21.179.169 [Thu, 23 Feb 2012 03:30:30 +0000] INFO: Running start handlers107.21.179.169 [Thu, 23 Feb 2012 03:30:30 +0000] INFO: Start handlers complete.107.21.179.169 [Thu, 23 Feb 2012 03:30:36 +0000] INFO: Loading cookbooks [chef-client, java, jpackage, ntp, sample, sudo, tomcat, users]107.21.179.169 [Thu, 23 Feb 2012 03:30:36 +0000] INFO: Storing updated cookbooks/sample/recipes/default.rb in the cache.<SNIP>107.21.179.169 [Thu, 23 Feb 2012 03:30:37 +0000] INFO: template[/var/lib/tomcat6/webapps/sample/index.html] updated content107.21.179.169 [Thu, 23 Feb 2012 03:30:37 +0000] INFO: template[/var/lib/tomcat6/webapps/sample/index.html] sending restart action to service[tomcat] (immediate)107.21.179.169 [Thu, 23 Feb 2012 03:30:37 +0000] INFO: Processing service[tomcat] action restart (tomcat::default line 42)107.21.179.169 [Thu, 23 Feb 2012 03:30:45 +0000] INFO: service[tomcat] restarted107.21.179.169 [Thu, 23 Feb 2012 03:30:45 +0000] INFO: Chef Run complete in 15.170097638 seconds107.21.179.169 [Thu, 23 Feb 2012 03:30:45 +0000] INFO: Running report handlers107.21.179.169 [Thu, 23 Feb 2012 03:30:45 +0000] INFO: Report handlers complete

Tomcat stack updatedTomcat stack updated

http://www.flickr.com/photos/kathycsus/2686772625

• IP addresses

• Hostnames

• FQDNs

• Search for nodes with Roles

• Find configuration data

SearchSearch

pool_members = search("node","role:webserver”)

template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb” owner "root" group "root” mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]”end

Pass results into Templates

# Set up application listeners here.listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |memb| -%> server <%= memb[:hostname] %> <%= memb[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%><% if node["haproxy"]["enable_admin"] -%>listen admin 0.0.0.0:22002 mode http stats uri /<% end -%>

Pass results into Templates

Jboss App

Memcache

Postgres Slaves

Postgres Master

So when thisSo when this

NagiosGraphite

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

Becomes this

Becomes this

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

Updates can be automatic

Updates can be automatic

NagiosGraphite

Count the resourcesCount the resources

Jboss App

Memcache

Postgres Slaves

• Load balancer config

• Nagios host ping

• Nagios host ssh

• Nagios host HTTP

• Nagios host app health

• Graphite CPU

• Graphite Memory

• Graphite Disk

• Graphite SNMP

• Memcache firewall

• Postgres firewall

• Postgres authZ config

12+ resource changes for 1 node addition

http://www.flickr.com/photos/evelynishere/2798236471/

CLONING CANNOT COPE WITH THISCLONING CANNOT COPE WITH THIS

• Chef can.

Build anythingBuild anything

•Simple internal applications

•Complex external applications

•Workstations

•Hadoop clusters

• IaaS infrastructure

•PaaS infrastructure

•SaaS applications

•Storage systems

•You name it

http://www.flickr.com/photos/hyku/245010680/

And manage it simplyAnd manage it simply

http://www.flickr.com/photos/helico/404640681/

•Automatically reconfigure everything

•Linux, Windows, Unixes, BSDs

•Load balancers

•Metrics collection systems

•Monitoring systems

•Cloud migrations become trivial

The Chef Community

The Chef Community

•Apache License, Version 2.0

•550+ Individual contributors

•100+ Corporate contributors

•Dell, Rackspace,VMware, RightScale, Heroku, and many more

•400+ Community cookbooks

•http://community.opscode.com

Questions?Questions?

http://www.flickr.com/photos/mrchippy/443960682/

Questions?Questions?

Thanks!Thanks!

matt@opscode.comwww.opscode.com

Twitter/IRC/GitHub: mattray