infrastructure-as-code and cloud deployments with opscode chef & co
DESCRIPTION
This talk was held at a Zühlke internal meetup of architects in 2012. It explains the motivation behind DevOps and Infrastructure-as-Code, then introduces Chef, looking at the essential Chef concepts and the Chef recipe DSL. Next comes a brief outlook to Vagrant for managing VMs, Librarian for dependency management, and various Chef testing tools. Finally, different options for cloud deployments with Chef are discussed: AWS CloudFormation, Mccloud and knife-ec2.TRANSCRIPT
© Zühlke 2012
Torben Knerr
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co
20. Juli 2012 Slide 1
© Zühlke 2012
Topics
Motivation
DevOps & Infrastructure-As-Code
Introduction to Chef
Terms & Concepts (with Examples)
Chef vs. Puppet
Cloud Deployment
Different Approaches for Cloud Provisioning
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr 20. Juli 2012 Slide 2
© Zühlke 2012
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co
Torben Knerr
Motivation
DevOps & Infrastructure-As-Code
20. Juli 2012 Slide 3
© Zühlke 2012
Motivation DevOps & Infrastructure-As-Code
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
DevOps
Drivers: Cloud & Virtualisation Mass Server Management Agile & Continous Delivery
Philosophy: Collaboration between Devs and Ops Devs understand the needs of Ops Ops learn from Devs (Infrastructure-As-Code) Ops are part of the (SCRUM) team
Disciplines: Operations, Monitoring Automation Programming (Ruby) Configuration Management
Goals: Traceability, Auditability Maintainability Reproducability Speed / Time-To-Market
Infrastructure-As-Code
20. Juli 2012 Slide 4
© Zühlke 2012
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co
Torben Knerr
Introduction to Chef
Knife, Cookbooks, Recipes, etc…
20. Juli 2012 Slide 5
© Zühlke 2012
Introduction to Chef What‘s in a Chef Repository?
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
chef-repo
20. Juli 2012 Slide 6
© Zühlke 2012
Introduction to Chef What‘s in a Chef Repository?
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
chef-repo
cookbooks
databags
environments
roles
20. Juli 2012 Slide 7
© Zühlke 2012
Introduction to Chef What‘s in a Chef Repository?
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
recipes
attributes
files
templates chef-repo
cookbooks
databags
environments
roles
20. Juli 2012 Slide 8
© Zühlke 2012
Introduction to Chef Chef Solo vs. Chef Server
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
node
couchdb cookbooks
roles databags
nodes … Chef Server
node node node node
chef-client
poll for node’s run_list and converge node
node
cookbooks roles
databags solo.rb
chef-solo
read run_list from solo.rb and converge node
Chef Solo Runs chef-solo locally No server required Easier to get started with Good for few nodes, onetime
bootstraps
chef-repo
upload to node
20. Juli 2012 Slide 9
© Zühlke 2012
Introduction to Chef Chef Solo vs. Chef Server
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
node
couchdb cookbooks
roles databags
nodes … Chef Server
node node node node
chef-client
poll for node’s run_list and converge node
node
cookbooks roles
databags solo.rb
chef-solo
read run_list from solo.rb and converge node
Chef Server chef-client polls regularly More complex workflow Additional features:
Search, WebUI Good for ongoing
operations of many nodes chef-repo
upload to chef server
20. Juli 2012 Slide 10
© Zühlke 2012
Chef Repo
Cookbooks
Attributes
Recipes
Resources
Databags
Roles
Environments
Nodes
Introduction to Chef Terms & Concepts
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
The Chef Repository is the place where cookbooks, roles, config files and other artifacts for managing systems with Chef will live, typically stored in a version control system.
20. Juli 2012 Slide 11
© Zühlke 2012
Chef Repo
Cookbooks
Attributes
Recipes
Resources
Databags
Roles
Environments
Nodes
Introduction to Chef Terms & Concepts
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Cookbooks are the fundamental units of distribution in Chef. They encapsulate all the resources you need to automate your infrastructure and are easily sharable with other Chef users.
20. Juli 2012 Slide 12
© Zühlke 2012
Chef Repo
Cookbooks
Attributes
Recipes
Resources
Databags
Roles
Environments
Nodes
Introduction to Chef Terms & Concepts
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Attributes are node data such as the IP address, hostname, etc... New attributes can be defined on cookbook, environment, role and node level.
cookbooks/apache2/attributes/default.rb
20. Juli 2012 Slide 13
© Zühlke 2012
Chef Repo
Cookbooks
Attributes
Recipes
Resources
Databags
Roles
Environments
Nodes
Introduction to Chef Terms & Concepts
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Recipes encapsulate collections of resources which are executed in order to configure the system.
cookbooks/apache2/recipes/default.rb
20. Juli 2012 Slide 14
© Zühlke 2012
Chef Repo
Cookbooks
Attributes
Recipes
Resources
Databags
Roles
Environments
Nodes
Introduction to Chef Terms & Concepts
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Recipes encapsulate collections of resources which are executed in order to configure the system.
Resources
cookbooks/apache2/recipes/default.rb
20. Juli 2012 Slide 15
© Zühlke 2012
Chef Repo
Cookbooks
Attributes
Recipes
Resources
Databags
Roles
Environments
Nodes
Introduction to Chef Terms & Concepts
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Resources are the cross platform abstraction of the thing you're configuring on the node. Examples:
20. Juli 2012 Slide 16
© Zühlke 2012
Chef Repo
Cookbooks
Attributes
Recipes
Resources
Databags
Roles
Environments
Nodes
Introduction to Chef Terms & Concepts
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Resources are the cross platform abstraction of the thing you're configuring on the node. Examples:
20. Juli 2012 Slide 17
© Zühlke 2012
Chef Repo
Cookbooks
Attributes
Recipes
Resources
Databags
Roles
Environments
Nodes
Introduction to Chef Terms & Concepts
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Resources are the cross platform abstraction of the thing you're configuring on the node. Examples:
20. Juli 2012 Slide 18
© Zühlke 2012
Chef Repo
Cookbooks
Attributes
Recipes
Resources
Databags
Roles
Environments
Nodes
Introduction to Chef Terms & Concepts
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Resources are the cross platform abstraction of the thing you're configuring on the node. Examples:
20. Juli 2012 Slide 19
© Zühlke 2012
Chef Repo
Cookbooks
Attributes
Recipes
Resources
Databags
Roles
Environments
Nodes
Introduction to Chef Terms & Concepts
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Resources are the cross platform abstraction of the thing you're configuring on the node. Examples:
20. Juli 2012 Slide 20
© Zühlke 2012
Chef Repo
Cookbooks
Attributes
Recipes
Resources
Databags
Roles
Environments
Nodes
Introduction to Chef Terms & Concepts
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Resources are the cross platform abstraction of the thing you're configuring on the node. Examples: Many, many more in the Opscode Wiki:
http://wiki.opscode.com/display/chef/Resources
You can also write your own: http://wiki.opscode.com/display/chef/Lightweight+Resources+and+Providers+%28LWRP%29
20. Juli 2012 Slide 21
© Zühlke 2012
Chef Repo
Cookbooks
Attributes
Recipes
Resources
Databags
Roles
Environments
Nodes
Introduction to Chef Terms & Concepts
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Data bags provide an arbitrary stores of globally available JSON data…
chef-repo/data_bags/users/bofh.json
20. Juli 2012 Slide 22
© Zühlke 2012
Chef Repo
Cookbooks
Attributes
Recipes
Resources
Databags
Roles
Environments
Nodes
Introduction to Chef Terms & Concepts
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
…which are then used in Recipes:
cookbooks/users/recipes/default.rb
20. Juli 2012 Slide 23
© Zühlke 2012
Chef Repo
Cookbooks
Attributes
Recipes
Resources
Databags
Roles
Environments
Nodes
Introduction to Chef Terms & Concepts
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
A role provides a means of grouping similar features of similar nodes, providing a mechanism for easily composing sets of functionality.
chef-repo/roles/webserver.rb
20. Juli 2012 Slide 24
© Zühlke 2012
Chef Repo
Cookbooks
Attributes
Recipes
Resources
Databags
Roles
Environments
Nodes
Introduction to Chef Terms & Concepts
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Environments in Chef provide a mechanism for managing different environments such as production, staging, development, testing, etc
chef-repo/environments/dev.rb
20. Juli 2012 Slide 25
© Zühlke 2012
Chef Repo
Cookbooks
Attributes
Recipes
Resources
Databags
Roles
Environments
Nodes
Introduction to Chef Terms & Concepts
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Nodes are the thing that Recipes and Roles are applied to. The primary features of a node, from Chef's point of view, are its Attributes and its run list.
chef-repo/nodes/web.example.com.rb
20. Juli 2012 Slide 26
© Zühlke 2012
How to get started?
• If you are working on Windows use Bill’s Kitchen https://ikm.zuehlke.com/topics/Pages/Bill's%20Kitchen.aspx
• Start with Vagrant and Chef-Solo (you will love Vagrant…) http://vagrantup.com/v1/docs/provisioners/chef_solo.html
• Get familiar with cookbook development and testing https://github.com/tknerr/bills-kitchen/blob/master/COOKBOOK_DEVELOPMENT.md
• Set up a Chef Server using knife-server http://fnichol.github.com/knife-server/
• Use Vagrant with Chef-Server Provisioning http://vagrantup.com/v1/docs/provisioners/chef_server.html
• Get familiar with knife and use advanced features like search, bootstrap, etc.. http://wiki.opscode.com/display/chef/Knife
Introduction to Chef Getting Started with Chef Development
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr 20. Juli 2012 Slide 27
© Zühlke 2012
Introduction to Chef VirtualBox Automation with Vagrant
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Vagranfile
20. Juli 2012 Slide 28
© Zühlke 2012
Introduction to Chef Cookbook Dependency Management with Librarian
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Cheffile
20. Juli 2012 Slide 29
© Zühlke 2012
Introduction to Chef Test-Driven Infrastrucure!
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Lint Checking
• foodcritic
Unit-Level Testing
• chefspec
• fauxhai
Smoke Tests (from the inside)
• chef-minitest-handler
Acceptance Testing (from the outside)
• cucumber-nagios
20. Juli 2012 Slide 30
© Zühlke 2012
It doesn’t really matter (for us)
Same concepts, slightly different terminology
External vs. internal DSL (like Maven vs. Gradle)
Both have solo / standalone mode
There are more Puppet books than Chef books
Both have a very active community (mailing list, IRC, etc..)
Do you have in-house knowledge of Chef/Puppet?
Introduction to Chef Chef vs. Puppet
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr 20. Juli 2012 Slide 31
© Zühlke 2012
It doesn’t really matter (for us)
Same concepts, slightly different terminology
External vs. internal DSL (like Maven vs. Gradle)
Both have solo / standalone mode
There are more Puppet books than Chef books
Both have a very active community (mailing list, IRC, etc..)
Do you have in-house knowledge of Chef/Puppet?
Introduction to Chef Chef vs. Puppet
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Use the tool that “operations” understands!
20. Juli 2012 Slide 32
© Zühlke 2012
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co
Torben Knerr
Cloud Deployment
Different Approaches for Provisioning Cloud Instances
20. Juli 2012 Slide 33
© Zühlke 2012
Cloud Deployment The Basic Mechanism: EC2 User Data
You can pass arbitrary user data when launching an instance…
…and access it on the instance via the metadata service:
Notes:
User data is limited to 16K in raw form (before base64 encoding)
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr 20. Juli 2012 Slide 34
© Zühlke 2012
Cloud Deployment Option 1: Provisioning with Shell Scripts
Convention: if the user data script starts with a `#!` it will be run at first boot
Notes:
Might not be supported on all AMIs (for the official Ubuntu AMIs it is)
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
path/to/bootstrap.sh
20. Juli 2012 Slide 35
© Zühlke 2012
Cloud Deployment Option 1: Provisioning with Shell Scripts
Convention: if the user data script starts with a `#!` it will be run at first boot
Notes:
Might not be supported on all AMIs (for the official Ubuntu AMIs it is)
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
path/to/bootstrap.sh
easiest method good if bootstrapping is simple one-off bootstrap (first boot)
20. Juli 2012 Slide 36
© Zühlke 2012
VM instances are bootstrapped with Chef/Puppet via Cloud-init
The rest is just AWS CloudFormation Templates
Supports Chef Server, Chef Solo and Puppet Client Server
Cloud Deployment Option 2: AWS CloudFormation Templates
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr 20. Juli 2012 Slide 37
© Zühlke 2012
VM instances are bootstrapped with Chef/Puppet via Cloud-init
The rest is just AWS CloudFormation Templates
Supports Chef Server, Chef Solo and Puppet Client Server
Cloud Deployment Option 2: AWS CloudFormation Templates
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
good for more complex scenarios
official, AWS supported documentation available
not meant for humans to read 20. Juli 2012 Slide 38
© Zühlke 2012
Vagrant-like approach for managing VMs in the cloud
Supports Chef, Puppet and Shell provisioning
Currently focused on AWS but other providers possible
Supports stacks via AWS CloudFormation Templates
Cloud Deployment Option 3: Mccloud
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr 20. Juli 2012 Slide 39
© Zühlke 2012
Vagrant-like approach for managing VMs in the cloud
Supports Chef, Puppet and Shell provisioning
Currently focused on AWS but other providers possible
Supports stacks via AWS CloudFormation Templates
Cloud Deployment Option 3: Mccloud
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Yay! :-) 20. Juli 2012 Slide 40
© Zühlke 2012
knife-ec2
knife plugin to let you launch, bootstrap and provision EC2 VMs with Chef
Works only with Chef Server – no Chef Solo support
Puppet Cloud Provisioner
Puppet extension to launch, bootstrap and provision EC2 VMs with Puppet
Works only with Puppet Master – no Puppet Standalone mode (?)
Cloud Deployment Option 4: Native Chef / Puppet Tools
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr 20. Juli 2012 Slide 41
© Zühlke 2012
knife-ec2
knife plugin to let you launch, bootstrap and provision EC2 VMs with Chef
Works only with Chef Server – no Chef Solo support
Puppet Cloud Provisioner
Puppet extension to launch, bootstrap and provision EC2 VMs with Puppet
Works only with Puppet Master – no Puppet Standalone mode (?)
Cloud Deployment Option 4: Native Chef / Puppet Tools
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
You can’t have both: it’s Chef OR Puppet Fallback if the generic tools (Option 2 & 3)
don’t work out as expected No Solo / Standalone mode
20. Juli 2012 Slide 42
© Zühlke 2012
Vagrant – create and configure lightweight, reproducible, and portable development environments
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
Vagranfile
20. Juli 2012 Slide 43
© Zühlke 2012
EC2 User Data Scripts http://alestic.com/2009/06/ec2-user-data-scripts https://help.ubuntu.com/community/CloudInit http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html
AWS CloudFormation Templates with Chef/Puppet https://s3.amazonaws.com/cloudformation-examples/IntegratingAWSCloudFormationWithOpscodeChef.pdf https://s3.amazonaws.com/cloudformation-examples/IntegratingAWSCloudFormationWithPuppet.pdf
Mccloud https://github.com/jedi4ever/mccloud https://github.com/jedi4ever/mccloud-demo https://gist.github.com/3175267
knife-ec2 http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+with+Knife
Puppet Cloud Provisioner http://docs.puppetlabs.com/guides/cloud_pack_getting_started.html
Cloud Deployment Overview & Resources
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr 20. Juli 2012 Slide 44
© Zühlke 2012
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co
Torben Knerr
Discussion Time!
Puppet, Mccloud, Foo, Bar, Baz
DevOps, NoOps, Blablabla
Vagrant, Chef, Deployment Stuff
Cloudy Cloud!
20. Juli 2012 Slide 45
Thank You! And let’s continue the discussion here: