chef in the cloud [dbccg]

Copyright © 2010 Opscode, Inc - All Rights Reserved

Speaker:

‣ [email protected]‣ @jtimberman‣ www.opscode.com

Joshua Timberman Technical Evangelist

1

Chef in the Cloud

Monday, September 27, 2010

Copyright © 2010 Opscode, Inc - All Rights Reserved 2http://www.flickr.com/photos/anotherphotograph/2100904507/sizes/o/

System administrator

Many environments

Opscode: Training, Services


http://www.flickr.com/photos/anotherphotograph/2100904507/sizes/o/

http://www.flickr.com/photos/anotherphotograph/2100904507/sizes/o/


http://www.flickr.com/photos/timyates/2854357446/sizes/l/

3

Developers?Systems Administrators?“Business” people?




Copyright © 2010 Opscode, Inc - All Rights Reserved 4

Cloud Infrastructure

ProvisioningConfiguration ManagementSystem Integration



Nodes

lb01

web01web02

db-master01db-slave01db-slave02

Provisioning



Roles

loadbalancerwebserverdbmasterdbslave

Configuration Management



Load Balancer

Web Server Web Server

DB Master

DiskDisk

DB Slave DB Slave

Disk

Recipes

haproxyapache2myssql

Systems Integration



Infrastructure as Code

8

http://www.flickr.com/photos/wonderlane/2306082998/





Infrastructure as Code is...

9

A technical domain revolving around building and managing infrastructure programmatically

http://www.flickr.com/photos/kwerfeldein/2634561264/sizes/o/





Enable the reconstruction of the business from nothing

but a source code repository, an application

data backup, and bare metal resources.

10Monday, September 27, 2010

Copyright © 2010 Opscode, Inc - All Rights Reserved 11http://www.brooklynstreetart.com/theBlog/wp-content/uploads/2008/12/swedish_chef_bork-sleeper-cell.jpg


http://www.brooklynstreetart.com/theBlog/wp-content/uploads/2008/12/swedish_chef_bork-sleeper-cell.jpg

http://www.brooklynstreetart.com/theBlog/wp-content/uploads/2008/12/swedish_chef_bork-sleeper-cell.jpg

At a High Level...

‣ A library for configuration management

‣ A configuration management system

‣ A systems integration platform

‣ An API for your entire Infrastructure

http://www.flickr.com/photos/asten/2159525309/sizes/l/





Chef Client runs on your systems



Clients talk to a Chef Server



RESTful API w/ JSON responses



The Opscode Platform is a hosted Chef Server

16

http://www.opscode.com


Copyright © 2010 Opscode, Inc - All Rights Reserved http://www.flickr.com/photos/peterrosbjerg/3913766224/ 17

We call each system you configure a Node



Nodes have Attributes

18

{ "kernel": { "machine": "x86_64", "name": "Darwin", "os": "Darwin", "version": "Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386", "release": "10.4.0" }, "platform_version": "10.6.4", "platform": "mac_os_x", "platform_build": "10F569", "domain": "local", "os": "darwin", "current_user": "jtimberman", "ohai_time": 1278602661.60043, "os_version": "10.4.0", "uptime": "18 days 17 hours 49 minutes 18 seconds", "ipaddress": "10.13.37.116", "hostname": "cider", "fqdn": "cider.local", "uptime_seconds": 1619358 }

Kernel info!

Platform info!

Hostname and IP!



Nodes have a Run List

19

What Roles or Recipes to applyin Order


Copyright © 2010 Opscode, Inc - All Rights Reserved 20http://www.flickr.com/photos/laenulfean/374398044/

Nodes have Roles



Roles have a Run List

21

What Roles or Recipes to applyin Order



Chef manages Resources on Nodes

22

cookbook_file

template

service

package

deploy

git

http_request

link

ruby_block

logbash

execute

remote_file

userMonday, September 27, 2010

http://www.flickr.com/photos/xiaming/382205902/sizes/l/

Resources...

‣ Have a type

‣ Have a name

‣ Have parameters

‣ Take action to put the resource in the declared state

package "apache2" do version "2.2.11-2ubuntu2.6" action :installend

template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :createend

Declare a description of the state a part of the node should be in





Resources take action through Providers


Providers...

Multiple providers per resource type.

Know how to actually perform the actions specified by a resource.

Apt, Yum, Rubygems, Portage, Macports, FreeBSD Ports, etc.

http://www.flickr.com/photos/affableslinky/562950216/




Resources

http://www.flickr.com/photos/acurbelo/2628837104/sizes/o/

Platform

Provider




Copyright © 2010 Opscode, Inc - All Rights Reservedhttp://www.flickr.com/photos/roadsidepictures/2478953342/sizes/o/ 27

Recipes are lists of Resources


http://www.flickr.com/photos/roadsidepictures/2478953342/sizes/o/


Recipes...Apply resources in the order they are specified


package "apache2" do version "2.2.11-2ubuntu2.6" action :installend

template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :createend

1

2

‣ Evaluates resources in the order they appear

‣ Adds each resource to the Resource Collection

[ "package[apache2]", "template[/etc/apache2/apache2.conf]"]





Order Matters

29

http://www.infrastructures.org/papers/turing/turing.html



Cookbooks are packages for Recipes



Common Cookbook Components

recipes/default.rb

files/templates/attributes/

default.rb

metadata.rb



Cookbooks are shareable!

cookbooks.opscode.com



Data bags store arbitrary data



% knife data bag show users jtimberman{ "comment": "Joshua Timberman", "groups": "sysadmin", "ssh_keys": "ssh-rsa SUPERSEKRATS jtimberman@cider", "files": { ".zshrc": { "mode": "0644", "source": "dot-zshrc" }, ".vimrc": { "mode": "0644", "source": "dot-vimrc" } }, "id": "jtimberman", "uid": 7004, "shell": "/usr/bin/zsh", "openid": "http://jtimberman.myopenid.com/"}

A user data bag item...



Data bags make recipes awesome-r (that’s

totally a word)

sysadmins = search(:users, 'groups:sysadmin')

sysadminss.each do |u| user u['id'] do uid u['id'] shell u['shell'] comment u['comment'] supports :manage_home => true home "/home/#{u['id']}" end

directory "/home/#{u['id']}/.ssh" do owner u['id'] group u['id'] mode 0700 end

template "/home/#{u['id']}/.ssh/authorized_keys" do source "authorized_keys.erb" owner u['id'] group u['id'] mode 0600 variables :ssh_keys => u['ssh_keys'] endend



Nodes, Roles, Data Bags are Searchable

36

% knife search node “role:webserver”

search(:users, “group:sysadmins”)



Automating the Cloud with Chef

37http://www.flickr.com/photos/46183897@N00/3442880227/sizes/l/


http://www.flickr.com/photos/46183897@N00/3442880227/sizes/l/

http://www.flickr.com/photos/46183897@N00/3442880227/sizes/l/


lb1

web1 web2



Command-line API utility, Knife

39http://www.flickr.com/photos/myklroventine/3474391066/



Fog

EC2Rackspace/OpenstackTerremark/vcloudSlicehostOthers...?



Add your Cloud credentials to knife.rb

42

vi ~/chef-repo/.chef/knife.rb

# Cloud credentialsknife[:aws_access_key_id] = ENV['AWS_ACCESS_KEY_ID']knife[:aws_secret_access_key] = ENV['AWS_SECRET_ACCESS_KEY']



Download some cookbooks

% knife cookbook site vendor haproxyINFO: Downloading haproxy from the cookbooks site at version 0.7.0...INFO: Merging changes from haproxy version 0.7.0....INFO: Cookbook haproxy version 0.7.0 successfully vendored!

% knife cookbook site vendor apache2...INFO: Cookbook apache2 version 0.12.3 successfully vendored!



Upload Cookbooks!

knife cookbook upload -a

These run as root, kids.Let’s not blindly trust the upstream too much!



Build some roles% vi roles/lb.rb

name "lb"description "Load Balancer"run_list( "recipe[haproxy]")

% vi roles/webserver.rb

name "webserver"description "Systems that serve HTTP traffic"run_list( "recipe[apache2]")default_attributes( "apache2" => { "listen_ports" => [ "80" ] })



Upload Roles

% knife role from file lb.rbWARN: HTTP Request Returned 404 Not Found: Cannot load role lbWARN: Updated Role lb!

% knife role from file webserver.rbWARN: HTTP Request Returned 404 Not Found: Cannot load role webserver WARN: Updated Role webserver!



Launch a new Web Server and Load

Balancer

47

knife ec2 server create ‘role[webserver]’

knife ec2 server create ‘role[lb]’



knife ec2 server create

48

Create EC2 instance via APIRetrieve local configurationSSH to instance

‣ Write chef configuration and authentication‣ Install Ruby and Chef‣ Run Chef with specified run list



Chef runs on your new server

49

sudo chef-client

INFO: Starting Chef Run...INFO: Chef Run complete in 211.852033 seconds

Automatically.


listen application 0.0.0.0:80 balance roundrobin<% @webservers.each do |n| -%> server <%= n[:hostname] %> <%= n[:ipaddress] %>:80 weight 1 maxconn 50 check<% end -%>

webservers = search(:node, "role:webserver")

template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 variables(:webservers => webservers) notifies :restart, resources(:service => "haproxy")end

cookbooks/haproxy/recipes/default.rb

cookbooks/haproxy/templates/default/haproxy.cfg.erb

<html> <head> <title>Welcome to <%= node[:hostname] %></title> </head> <body> You have reached: <ul> <li>FQDN: <%= node[:fqdn] %></title> <li>Public FQDN: <%= node[:ec2][:public_hostname]%></id> <li>IP Address: <%= node[:ipaddress] %></id> <li>Public IP: <%= node[:ec2][:public_ipv4] %></id> <li>Platform: <%= node[:platform] %></id> <li>Plaform Version: <%= node[:platform_version] %></id> <li>Run List: <%= node.run_list %></id> </ul> </body></html>

cookbooks/apache2/templates/default/index.html.erb

<html> <head> <title>Welcome to <%= node[:hostname] %></title> </head> <body> You have reached: <ul> <li>FQDN: <%= node[:fqdn] %></title> <li>Public FQDN: <%= node[:ec2][:public_hostname]%></id> <li>IP Address: <%= node[:ipaddress] %></id> <li>Public IP: <%= node[:ec2][:public_ipv4] %></id> <li>Platform: <%= node[:platform] %></id> <li>Plaform Version: <%= node[:platform_version] %></id> <li>Run List: <%= node.run_list %></id> </ul> </body></html>


Lessons Learned



You own your Availability

53http://www.flickr.com/photos/jeffmcneill/4252968654/



http://www.flickr.com/photos/wwworks/3271208324/sizes/l/

55

‣ Amazon Machine Images

‣ Use one AMI with JEOS for each size

‣ Beware of Image Sprawl

‣ Rebundle for long-running installation

AMIs






56

‣ c1.medium is best bang for the $

‣ User data can inform configuration

‣ Use SSH key only access

‣ Allow SSH in default security group

Instances






57

‣ Internal and external network info is not unique

‣ Only trust the instance-id

Instances






58

‣ EBS

‣ Persistent storage

‣ Trivial to snapshot

‣ Snapshots can create new devices

Storage






59

‣ Static public routable addresses

‣ Easy and fast to re-assign

‣ Just an API call away

Elastic IP Addresses





Resources/Questions

60

www.opscode.com/chefIRC and Mailing lists‣ irc.freenode.net #chef‣ lists.opscode.com

Twitter:‣ @opscode, #opschef‣ @jtimberman

Questions?


chef in the cloud [dbccg]

Technology

run list

rights reservedhttp

rights reservedinfrastructure

rights reservednodes

applyin ordermonday

public ip

ip address

copyright