Chef & ApacheCloudStack™

Cloud Systems automation and configurationmanagement using Chef with Apache CloudStack™ and/or

Citrix CloudPlatform™Created by / Jeff Moody @fifthecho

#> whoamiJeff Moody

Cloud Engineer at DatapipeManaged Service Provider based in Jersey City, NJManaged AWS Services (World's largest AWS reseller)Infrastructure-as-a-Service Platform built using Citrix CloudPlatform(Stratosphere)Managed Services on Stratosphere

Primary author/maintainer of knife-cloudstack-fog, kitchen-cloudstack,CloudStack-PowerShell

DisclaimerMy opinions are my own and not necessarily representative of myemployerYMMV (Your Milage May Vary)No code is perfect, there may be bugs (please sent pull requests/submitissues if you find them)

What is Apache CloudStack™?

Virtualization Orchestration PlatformPrivate, Public, VPC Style CloudsHypervisor Agnostic

XenKVMVMware ESXi (with VCenter)LXCHyper-VOracle VM (OVM)Bare MetalDocker support in development (announced last week)

What is Apache CloudStack™?

Project founded in 2008Started by Cloud.com (First OSS release May 2010)Bought by Citrix (July 2011)Donated to ASF (April 2012)Accepted as an ASF Top-Level Project (March 2013)

Written in Java/TomcatRobust Native API

Baked-in AWS EC2 Compatibility with SOAP and REST API

Then what is Citrix CloudPlatform™?

Commercial release of CloudStackDeeper 3rd-Party Vendor integrationsNon-Apache Licence Compatible Libraries/ToolsIncludes licensing for Citrix XenServer

CloudStack and ChefWhy are there two knife plugins for CloudStack?What makes the two plugins different?What benefits does a knife plugin have over native API + knife bootstrap?

What is knife-cloudstack?Developed initially by Edmunds.Less API coverage than knife-cloudstack-fogProvides "stacks" for building multiple servers at once.

David Nalley of Apache/Citrix looking at making this part of knife, not justknife-cloudstack

Uses its own CloudStack API library.

What, then, is knife-cloudstack-fog?Forked from an earlier, pre-Edmunds knife-cloudstack which workedsometimes.Developed initially by me. Several contributors, but still mostly my project.Provides coverage for almost every CloudStack API call needed to manageinfrastructure.Uses FOG for its CloudStack API calls.

What is FOG?

Cloud-independent/agnostic toolkit for RubyInitial CloudStack support added by Brian Dorry (also of Datapipe)

Uniform-ish access to all Cloud resources (independent of provider) usingcommon-ish languageUsed by Chef first-party knife plugins

What is knife?(Hopefully you know this already)CLI interface for Chef

Built to be extensible and support plug-insMany of these plug-ins are for VM provisioningPlugins include support for:

AWS, CloudStack, OpenStack, Google Compute Engine, VMware,XenServer, Docker...

Used to interact with Chef server and nodes

What does knife-cloudstack-fogprovide?

One-stop-shop for all information to manage CloudStack instances** CLOUDSTACK COMMANDS **knife cloudstack diskoffering listknife cloudstack keypair create -k NAME (options)knife cloudstack keypair delete NAMEknife cloudstack keypair listknife cloudstack network create -n NAME -o NETWORKOFFERINGID -z ZONE (options)knife cloudstack network delete IDknife cloudstack network listknife cloudstack networkoffering listknife cloudstack portforwardingrule listknife cloudstack publicip create (options)knife cloudstack publicip listknife cloudstack securitygroup list (options)knife cloudstack server create -s SERVICEID -t TEMPLATEID -z ZONEID (options)knife cloudstack server delete INSTANCE_IDknife cloudstack server destroy INSTANCE_IDknife cloudstack server list (options)knife cloudstack server start INSTANCE_IDknife cloudstack server stop INSTANCE_ID (options)knife cloudstack serviceoffering listknife cloudstack template list (options)knife cloudstack volume listknife cloudstack zone list

With all these options, what is themain thing we use this plugin for?

Provisioning servers!

knife cloudstack server create --helpknife cloudstack server create -s SERVICEID -t TEMPLATEID -z ZONEID (options) --server-url URL Chef Server URL --chef-zero-port PORT Port to start chef-zero on --key KEY API Client Key -A KEY, Your Cloudstack Access Key ID --cloudstack-access-key-id --cloudstack-api-endpoint ENDPOINT Your Cloudstack API endpoint -g, --groupids SECURITYGROUPIDS Comma separated list of CloudStack Security Group IDs. -G SECURITYGROUPNAMES, Comma separated list of CloudStack Security Group names. Each group name must be encapuslated in quotes if it contains whitespace. --groupnames -w, --networkids NETWORKIDS Comma separated list of CloudStack network IDs. -K SECRET, Your Cloudstack API Secret Access Key --cloudstack-secret-access-key -s, --serviceid SERVICEID The CloudStack service offering ID. -t, --templateid TEMPLATEID The CloudStack template ID for the server. -Z, --zoneid ZONE The CloudStack zone ID for the server. --[no-]color Use colored output, defaults to false on Windows, true otherwise -c, --config CONFIG The configuration file to use --defaults Accept default values for all questions --disable-editing Do not open EDITOR, just accept the data as is -D DISKOFFERINGID, Specifies either the Disk Offering ID for the ROOT disk for an ISO template, or a DATA disk. --diskoffering -d, --distro DISTRO Bootstrap a distro using a template; default is 'chef-full' -e, --editor EDITOR Set the editor to use for interactive commands -E, --environment ENVIRONMENT Set the Chef environment -F, --format FORMAT Which format to use for output -H, --hostname NAME The instance host name -i PRIVATE_KEY_FILE, The Private key file for authenticating SSH session. --keypair option is also needed. --identity-file -k, --keypair KEYPAIR The CloudStack Key Pair to use for SSH key authentication. -z, --local-mode Point knife commands at local repository instead of server

With all those options, how do westart?

1. Pick a Zone2. Pick a Template3. Pick a Service Offering4. Pick a Network/Security Group5. Add any additional options

Then, launch the VM!

knife cloudstack zone listNo configurable options

knife cloudstack zone list ID Name Network Type Security Groups? 3 New York Metro EPN Advanced No 5 New York Metro Advanced Yes 6 Hong Kong Advanced Yes 7 Silicon Valley Advanced Yes 8 Hong Kong EPN Advanced No 9 London Advanced Yes 11 Shanghai Advanced Yes 12 London EPN Advanced No 13 Silicon Valley EPN Advanced No 14 Shanghai EPN Advanced No 08e8fdf1-f64a-4ab6-bd21-7bffd578be9e Iceland EPN Advanced No e20be308-482b-4d10-8885-cea306e87de9 Silicon Valley 2 Advanced Yes cb765f1b-2638-465e-9c3d-9013e04116da Silicon Valley 2 EPN Advanced No 51345d53-bf3b-4280-b5dd-8541703eff9a Singapore EPN Advanced No 4baf6857-c7db-437a-96e6-f420761e128c Singapore Advanced Yes

knife cloudstack template list --helpknife cloudstack template list (options) -s, --server-url URL Chef Server URL --chef-zero-port PORT Port to start chef-zero on -k, --key KEY API Client Key -A KEY, Your Cloudstack Access Key ID --cloudstack-access-key-id --cloudstack-api-endpoint ENDPOINT Your Cloudstack API endpoint -K SECRET, Your Cloudstack API Secret Access Key --cloudstack-secret-access-key --[no-]color Use colored output, defaults to false on Windows, true otherwise -c, --config CONFIG The configuration file to use --defaults Accept default values for all questions -d, --disable-editing Do not open EDITOR, just accept the data as is -e, --editor EDITOR Set the editor to use for interactive commands -E, --environment ENVIRONMENT Set the Chef environment -L, --filter FILTER The template search filter. Default is 'featured.' Other options are 'self,' 'self-executable,' 'executable,' and 'community.' -F, --format FORMAT Which format to use for output -H, --hypervisor HYPERVISOR Limit responses to templates only running on a specific hypervisor. Default provides templates from all hypervisors. --local-mode Point knife commands at local repository instead of server -u, --user USER API Client Username --print-after Show the data after a destructive operation -T, --templateid TEMPLATEID Limit responses to a single template ID. Default provides all templates. -V, --verbose More verbose output. Use twice for max verbosity -v, --version Show chef version -y, --yes Say yes to all prompts for confirmation -Z, --zone ZONE Limit responses to templates only located in a specific zone. Default provides templates from all zones. -z, --zoneid ZONEID Limit responses to templates only running in a specific zone (specified by ID #). Default provides

knife cloudstack template list knife cloudstack template listID Hypervisor Size (in GB) Zone Name Zone ID Name 086ce1e1-3e92-4a9e-b416-2dd5445c1e7f VMware 20 Iceland EPN 08e8fdf1-f64a-4ab6-bd211573395c-7489-4a93-9ca3-11a51a285457 VMware 20 Iceland EPN 08e8fdf1-f64a-4ab6-bd21d16563da-64c6-4358-b6e6-0c313c43223c VMware 20 Iceland EPN 08e8fdf1-f64a-4ab6-bd21187dc53e-8f4b-405b-9067-7ee056c75fcc VMware 20 Iceland EPN 08e8fdf1-f64a-4ab6-bd2120e58c82-2b7b-4552-adb2-59d4f28021cd VMware 40 Iceland EPN 08e8fdf1-f64a-4ab6-bd2127b96e7e-7315-48f2-8ac0-4a4aa7cc7e14 VMware 20 Iceland EPN 08e8fdf1-f64a-4ab6-bd2127fa2e69-ad13-43c2-b000-56cce241f769 VMware 20 Iceland EPN 08e8fdf1-f64a-4ab6-bd2188c62685-b6f5-4acc-9ca4-9b937e5112b5 XenServer 20 Iceland EPN 08e8fdf1-f64a-4ab6-bd210f755a47-92bf-4bbf-beb0-bb3fc2ae6c91 VMware 20 Iceland EPN 08e8fdf1-f64a-4ab6-bd21a494ea35-e64f-40d4-a30c-b5c9dc1971bd VMware 20 Iceland EPN 08e8fdf1-f64a-4ab6-bd21...

knife cloudstack serviceoffering listNo configurable options

knife cloudstack serviceoffering listID Name Description Number of CPUs CPU Speed Memory (f26eb96f-98d5-4f34-81b0-d817c81227a2 kilo-1-40-hpc 1 CPU core, 4G memory, high performance root volume 2b9a7b24-c977-467e-b0c8-eb0c97512f22 nano-h-5-hpc 0.5 CPU core, 512MB memory, high performance root volume 9d865b43-296e-4ea6-83a2-122bc9baf6ac nano-h-10-hpc 0.5 CPU core, 1G memory, high performance root volume 39 mega-2-160 2 CPU core, 16G memory, standard root volume 29ef5f70-8346-472a-9f84-f00a1fd727a7 kilo-4-80-hpc 4 CPU core, 8G memory, high performance root volume 38 mega-4-160 4 CPU core, 16G memory, standard root volume 4414bc19-7ee8-443f-80cc-0b19ff274ad9 nano-1-10-hpc 1 CPU core, 1G memory, high performance root volume 37 mega-8-160 8 CPU core, 16G memory, standard root volume 10f888d6-f6aa-4ebb-8f40-9acc5c4f0fa4 kilo-1-20-hpc 1 CPU core, 2G memory, high performance root volume ...

knife cloudstack network listUsed for Advanced Isolated or Advanced VPC Zones

knife cloudstack network listID Name Zone ID VLAN State 1a036de9-dd01-4f9a-af80-f5deac2d6131 Demo1 3 allocated 4cbc9ef7-d276-4858-beb0-ce6643a247b8 TestingNetwork 3 allocated 91b06fd1-bf74-4d02-a28a-0d9aca3d5fd5 SiliconValleyEPNTest 13 allocated 623 Default Network 12 implemented 535 Default Network 3 implemented 3d4af98c-7787-417a-8e26-8f265129e7c3 defaultGuestNetwork 4baf6857-c7db-437a-96e6-f420761e128c1124 Shared SG enabled network 5 setup ...

knife cloudstack securitygroup listUsed for Basic or Advanced with Security Groups Zones

knife cloudstack securitygroup listID Name Description 129 default Default Security Group 295 Network Testing 320 Chef Server 324 TestSG 432 Puppet Server f2bf049f-8c6b-4b8f-87d1-4846da688dbf AppTesting ...

knife cloudstack server createknife cloudstack server create -Z 4baf6857-c7db-437a-96e6-f420761e128c -t b5870d22-e586-4da9-add5-100dc1c18a9f

Waiting for server....

Name: 0bdb2cd0-d991-4d3c-8c4e-809597dfaf05Primary IP: 64.106.186.210Username: rootPassword: jA2rszsieWaiting for SSH.

Other information is available fromknife-cloudstack-fog for:

disk offeringsCreating data-disks with instancesCreating VMs from ISO templates for the ROOT disk

keypairsAssumes that the cloud-set-guest-sshkey is in place.Supports creation of and deletion of keypairs (end user feature request)

networksSupports creation and deletion of networks (for Advanced Zones)

portforwardingrulesFor Advanced Zone networks, NAT rules can be created with knife

Server stop/startSSH GatewayRandom port forwarding (for Advanced Zones)

FutureMove code to knife-cloud?

One plugin for all CloudsCommon semanticsLOTS of work to do

Merge with knife-cloudstack?Unlikely as some core knife-cloudstack devs are anti-FOGknife-cloudstack-fog more feature rich, minus stacks and system VMs

TestsThere currently are no tests.I don't know how to write tests (as of March 2014)If someone wants to start writing tests, please do!

Bonus content!There is a Test Kitchen driver for CloudStack/CloudPlatform! (...and I wrote it!)

The setup for the driver is on the GitHub page (https://github.com/test-kitchen/kitchen-cloudstack)

There's also a Vagrant driver for CloudStack/CloudPlatform (...and I didn'twrite it!)

So, how do I get in on this?Datapipe Stratosphere

Global IaaS Platform running on Citrix CloudPlatform

Install Apache CloudStack™Available from YUM and apt repositories for EL-6 and Ubuntu 12.04

I've built a Docker build script for CloudStack!Still under heavy work.Definitely not recommended for production.

(Currently coordinating with CloudStack peoples on how we want this inthe Docker Index)

http://www.datapipe.com/cloud/stratosphere/

http://cloudstack.apache.org/

https://github.com/fifthecho/docker-cloudstack-management

Questions?Jeff Moody

Datapipe (http://www.datapipe.com)E-Mail: jmoody@datapipe.com / fifthecho@gmail.comGitHub: https://github.com/fifthechoTwitter: @fifthechoYouTube: https://www.youtube.com/user/fifthecho

Thank you!Also many thanks to the Chef

community, FOG developers, mycontributors, and people who submit

issues.

Speaker Office Hours: Today 3:15-3:55 in Marina