multi node deployments - viewdeck€¦ · multi node deployments chefconf 2015 . kennon kwok...

Multi Node Deployments

ChefConf 2015

Kennon Kwok Solutions Engineer - Chef Software, Inc.

“mini-me”

Zack Zondlo

Previously: Senior Infrastructure Engineer, CareCloud

Previously: Senior Infrastructure Engineer, Selling Source

Support Engineer

5

Kevin Dickerson

Kevin is a former sandwich artisan from Alaska. He's previously worked at places like Carnegie Mellon University, Salesforce, and Chef. Interests include machine learning algorithms, super big web applications, and software automation.

Solutions Engineer - Chef Software, Inc.

Sean Carolan

Previously: Unix Systems Administrator at Electronic Arts Worked with CFEngine, Puppet, and now Chef Now: Sips martinis with the sales team

Solutions Architect - Chef Software, Inc.

Introduce Yourselves Name

Current job role

Previous job roles/background

Experience with Chef and/or config management

Favorite Text Editor

8

Agenda

Introduction to Chef Provisioning Emerging usage patterns - cluster cookbook - provisioning node - state of cookbook patterns Chef Provisioning walkthrough - WordPress cluster

Choose your own adventure Provision a stack of your own We’ll provide resources and help!

Morning Afternoon

Other logistics

•  Morning Break: 10:30-11:00 •  Lunch: 12:30-1:45 •  Afternoon Break: 3:15-3:45 •  All food is on the 2nd level of Hyatt •  Coffee outside of classroom all day •  Restrooms are 2 right turns when exiting classroom

Course style

Training is a discussion •  Hands-on labs •  Lots of typing •  Ask questions when they come to you •  Ask for help when you need it •  We will troubleshoot issues on the spot

Chef Provisioning An Introduction

Chef Provisioning •  Allows programatic creation of cloud resources

in Chef Recipes •  Allows for multiple tiers to be created in one

shot •  Moves more towards “Infrastructure as Code” •  Test and repair also applies to infrastructure •  Previously known as “chef-metal”

Chef Provisioning •  Plugin model lets you write bootstrappers for

your own infrastructure (e.g. VirtualBox, EC2, LXC, bare metal, etc)

•  Today we are going to use the AWS driver for Chef Provisioning

https://github.com/chef/chef-provisioning-aws

Provisioning workflows •  Chef Server with “provisioning node” •  Chef Client local-mode

Amazon Web Servies •  This isn’t a course on AWS (that’s next door) but

we’ll be touching on some AWS concepts. •  We’ve taken some steps before class to set up

your provisioning workstations for use with AWS.

Provisioning workstation •  EC2 instance, CentOS 6 •  Contains pre-installed software: ChefDK 0.4.0,

chef-provisioining, chef-provisioning-aws, awscli •  AWS credentials •  user: chef •  password: chef.io

Chef Provisioning WordPress walkthrough

Success Problem -  Fully provision WordPress

cluster -  Soup-to-nuts -  3 tiers:

-  database (mysql) -  application (php) -  load balancer (ELB)

-  github.com/kennonkwok/wordpress-simple

-  Build a cookbook that describes the cluster

-  One chef-client run to provision the entire cluster

Wordpress cluster steps 1.  Build database machine 2.  Build application server(s) 3.  Build load balancer

About those playing cards… http://git.io/jR9L

REMOTE

SSH to provisioning host

[email protected]'s password:

Last login: Mon Mar 16 22:26:01 2015 from 107.3.143.174

[chef@ip-10-221-14-71 ~]$

$ ssh YOUR_HOST

REMOTE

Chef generate cookbook

Usage: chef generate repo NAME [options]

-C, --copyright COPYRIGHT Name of the copyright holder - defaults to 'The Authors'

-m, --email EMAIL Email address of the author - defaults to '[email protected]'

-a, --generator-arg KEY=VALUE Use to set arbitrary attribute KEY to VALUE in the code_generator cookbook

-I, --license LICENSE all_rights, apache2, mit, gplv2, gplv3 - defaults to all_rights

-p, --policy-only Create a repository for policy only, not cookbooks

-g GENERATOR_COOKBOOK_PATH, Use GENERATOR_COOKBOOK_PATH for the code_generator cookbook --generator-cookbook

$ chef generate cookbook --help

REMOTE

Chef generate cluster

…

* directory[/home/chef/wordpress-cluster/recipes] action create

- create new directory /home/chef/wordpress-cluster/recipes

* template[/home/chef/wordpress-cluster/recipes/default.rb] action create_if_missing

- create new file /home/chef/wordpress-cluster/recipes/default.rb

- update content in file /home/chef/wordpress-cluster/recipes/default.rb from none to 9f3d82

(diff output suppressed by config)

* execute[initialize-git] action run

- execute git init .

* cookbook_file[/home/chef/wordpress-cluster/.gitignore] action create

- create new file /home/chef/wordpress-cluster/.gitignore

- update content in file /home/chef/wordpress-cluster/.gitignore from none to dd37b2 (diff output suppressed by config)

$ chef generate cookbook wordpress-cluster

REMOTE

Navigate to new cookbook

(no output)

$ cd wordpress-cluster

Anatomy of a cluster cookbook wordpress-cluster

+---.chef

\---knife.rb

+---attributes

+---cookbooks

+---recipes

\---_aws_settings.rb

\---database.rb

\---app.rb

\---lb.rb

•  .chef directory with knife.rb •  attributes work like other

cookbooks •  cookbooks directory is

in .gitignore •  recipes only contain

provisioning code •  recipes can contain

application tiers or driver specific code

REMOTE

Create .chef directory

(no output)

$ mkdir .chef

Create File

current_dir = File.dirname(__FILE__)

chef_repo_path "#{current_dir}/.."

.chef/knife.rb

Best Practice De-couple provider specific code

Create file

require 'chef/provisioning/aws_driver' with_driver 'aws' with_machine_options( bootstrap_options: { instance_type: node['wordpress-cluster']['aws']['flavor'], key_name: node['wordpress-cluster']['aws']['key_name'], security_group_ids: node['wordpress-cluster']['aws']['security_group_ids'] }, ssh_username: node['wordpress-cluster']['aws']['ssh_username'], image_id: node['wordpress-cluster']['aws']['image_id'] ) aws_key_pair node['wordpress-cluster']['aws']['key_name']

recipes/_aws_settings.rb

REMOTE

Make attributes directory

(no output)

$ mkdir attributes

Create file

default['wordpress-cluster']['aws']['flavor'] = 't1.micro'

default['wordpress-cluster']['aws']['key_name'] = YOUR_KEY_NAME_HERE

default['wordpress-cluster']['aws']['security_group_ids'] = 'sg-c49e0ff4'

default['wordpress-cluster']['aws']['ssh_username'] = 'root'

default['wordpress-cluster']['aws']['image_id'] = 'ami-37361807'

attributes/default.rb

•  Centos6 image with chef account •  default security group is open to world on 22,80,443 and

open internally •  use a unique key_name (e.g. kennon-wp-cluster)

Create file

require 'chef/provisioning' include_recipe 'wordpress-cluster::_aws_settings'

machine 'wordpress-database' do

tag 'wordpress-database' recipe 'wordpress-simple::database’ action :converge

end

recipes/database.rb

•  The machine resource defines one (or more) machines and can be combined into clusters of machines.

•  This allows clusters to be maintained in a version control system and to be defined using multi-machine orchestration scenarios.

Best Practice Create a destroy recipe

Create File

include_recipe 'wordpress-cluster::_aws_settings' machine_batch do machines search(:node, '*:*').map { |n| n.name } action :destroy end

recipes/destroy_all.rb

•  Just a quick clean up recipe. •  :destroy action terminates instance, cleans up node and

client data

Modify file

name 'wordpress-cluster' maintainer 'The Authors' maintainer_email '[email protected]' license 'all_rights' description 'Installs/Configures wordpress-cluster' long_description 'Installs/Configures wordpress-cluster' version '0.1.0’ depends 'wordpress-simple'

metadata.rb

Berkshelf •  Cookbook dependency manager •  Included in ChefDK •  We’ll use it here to download a WordPress

cookbook and its dependencies

Modify file

source 'https://supermarket.chef.io'

metadata

cookbook 'wordpress-simple', git: 'https://github.com/kennonkwok/wordpress-simple.git'

Berksfile

Helpful one-liners $ rm -fr cookbooks; berks vendor cookbooks

$ chef-client -z -o wordpress-cluster::destroy_all

-  First line: re-vendor cookbooks -  Second line: destroys cluster!

REMOTE

Gather cookbook dependecies

…

Vendoring selinux (0.9.0) to cookbooks/selinux

Vendoring smf (2.2.6) to cookbooks/smf

Vendoring tar (0.6.0) to cookbooks/tar

Vendoring wordpress-cluster (0.1.0) to cookbooks/wordpress-cluster

Vendoring wordpress-simple (0.1.0) to cookbooks/wordpress-simple

Vendoring yum (3.5.3) to cookbooks/yum

Vendoring yum-epel (0.6.0) to cookbooks/yum-epel

Vendoring yum-mysql-community (0.1.14) to cookbooks/yum-mysql-community

[chef@ip-10-157-113-135 wordpress-cluster]$

$ berks vendor cookbooks

REMOTE

Converge!

... * mysql_database_user[wordpressuser] action grant

- Granting privs for 'wordpressuser'@'%'

Running handlers:

Running handlers complete

Chef Client finished, 34/44 resources updated in 305.018417375 seconds

- run 'chef-client -l auto' on wordpress-database

[2015-03-23T23:58:43+00:00] WARN: Skipping final node save because override_runlist was given

Running handlers:



$ chef-client -z -o wordpress-cluster::database

Wordpress cluster steps 1.  Build database machine ✓ 2.  Build application server(s) 3.  Build load balancer

Create app cookbook

include_recipe 'wordpress-cluster::_aws_settings' machine 'wordpress-app' do tag 'wordpress-app' recipe 'wordpress-simple::app' action :converge converge true attributes lazy {{ 'wordpress-simple' => { 'dbhost' => search(:node, 'tags:wordpress-database').first['ipaddress'] } }} end

recipes/app.rb

REMOTE

Converge!

…

* service[httpd] action restart

- restart service service[httpd]

Running handlers:



- run 'chef-client -l auto' on wordpress-app


Running handlers:



$ rm -fr cookbooks; berks vendor cookbooks

$ chef-client –z –o wordpress-cluster::app

Wordpress cluster steps 1.  Build database machine ✓ 2.  Build application server(s) ✓ 3.  Build load balancer

Create load balancer recipe (using ELB)

include_recipe 'wordpress-cluster::_aws_settings'

load_balancer 'your-elb-name' do

load_balancer_options :availability_zones => ['us-west-2a', 'us-west-2b', 'us-west-2c'],

:listeners => [{

:port => 80,

:protocol => :http,

:instance_port => 80,

:instance_protocol => :http

}]

machines ['wordpress-app']

end

recipes/lb.rb

Load balancer name must be unique in class/account

REMOTE

Converge!

…

{:port=>80, :protocol=>:http, :instance_port=>80, :instance_protocol=>:http}

- with security groups

- add machines wordpress-app

- create data bag loadbalancers at http://localhost:8889

- create data bag item wordpress-elb at http://localhost:8889

- add location = {"driver_url"=>"aws", "driver_version"=>"0.4.0", "allocated_at"=>"2015-03-24 02:55:31 UTC"}


Running handlers:




$ chef-client –z –o wordpress-cluster::lb

Wordpress cluster steps 1.  Build database machine ✓ 2.  Build application server(s) ✓ 3.  Build load balancer ✓

Tie it all together!

#

# Cookbook Name:: wordpress-cluster

# Recipe:: default

#

# Copyright (c) 2015 The Authors, All Rights Reserved.

include_recipe 'wordpress-cluster::database'

include_recipe 'wordpress-cluster::app'

include_recipe 'wordpress-cluster::lb'

recipes/default.rb

Add lb to destroy_all recipe

include_recipe 'wordpress-cluster::_aws_settings' machine_batch do machines search(:node, '*:*').map { |n| n.name } action :destroy end load_balancer 'your-elb-name' do action :destroy end

recipes/destroy_all.rb

Wordpress cluster steps 1.  Build database machine ✓ 2.  Build application server(s) ✓ 3.  Build load balancer ✓ 4.  Scale out application layer!!

Scale out app servers

TODO

recipes/app.rb

REMOTE

Converge!

…

* service[httpd] action restart

- restart service service[httpd]

Running handlers:



- run 'chef-client -l auto' on wordpress-app


Running handlers:




$ chef-client –z –o wordpress-cluster::app

REMOTE

Soup to nuts!

- create data bag item wordpress-elb at http://localhost:8889

- add location = {"driver_url"=>"aws", "driver_version"=>"0.4.0", "allocated_at"=>"2015-03-24 03:34:09 UTC"}


Running handlers:



$ chef-client –z –o wordpress-cluster::default

Discussion What questions can I answer for you? •  Chef Provisioning •  machine resource

Introduction to Chef Provisioning Emerging usage patterns - cluster cookbook - provisioning node - state of cookbook patterns Chef Provisioning walkthrough - WordPress cluster

Choose your own adventure Provision a stack of your own We’ll provide resources and help!

Morning Afternoon

Choose your own Adventure •  Scale out the app layer •  LXC or Docker provisioning •  Vagrant provisioning (on your laptop) •  AWS allthethings! (VPC, RDS…) •  Try a different application stack

multi node deployments - viewdeck€¦ · multi node deployments chefconf 2015 . kennon kwok...

Documents