Chef and Apache CloudStack (ChefConf 2014)

Download Chef and Apache CloudStack (ChefConf 2014)

Post on 27-Jan-2015




1 download

Embed Size (px)




<ul><li> 1. Chef &amp; Apache CloudStack Cloud Systems automation and configuration management using Chef with Apache CloudStack and/or Citrix CloudPlatform Createdby /Jeff Moody @fifthecho </li></ul> <p> 2. #&gt; whoami Jeff Moody Cloud Engineer at Datapipe Managed Service Provider based in Jersey City, NJ Managed AWS Services (World's largest AWS reseller) Infrastructure-as-a-Service Platform built using Citrix CloudPlatform (Stratosphere) Managed Services on Stratosphere Primary author/maintainer of knife-cloudstack-fog, kitchen-cloudstack, CloudStack-PowerShell 3. Disclaimer My opinions are my own and not necessarily representative of my employer YMMV (Your Milage May Vary) No code is perfect, there may be bugs (please sent pull requests/submit issues if you find them) 4. What is Apache CloudStack? Virtualization Orchestration Platform Private, Public, VPC Style Clouds Hypervisor Agnostic Xen KVM VMware ESXi (with VCenter) LXC Hyper-V Oracle VM (OVM) Bare Metal Docker support in development (announced last week) 5. What is Apache CloudStack? Project founded in 2008 Started by (First OSS release May 2010) Bought by Citrix (July 2011) Donated to ASF (April 2012) Accepted as an ASF Top-Level Project (March 2013) Written in Java/Tomcat Robust Native API Baked-in AWS EC2 Compatibility with SOAP and REST API 6. Then what is Citrix CloudPlatform? Commercial release of CloudStack Deeper 3rd-Party Vendor integrations Non-Apache Licence Compatible Libraries/Tools Includes licensing for Citrix XenServer 7. CloudStack and Chef Why are there two knife plugins for CloudStack? What makes the two plugins different? What benefits does a knife plugin have over native API + knife bootstrap? 8. What is knife-cloudstack? Developed initially by Edmunds. Less API coverage than knife-cloudstack-fog Provides "stacks" for building multiple servers at once. David Nalley of Apache/Citrix looking at making this part of knife, not just knife-cloudstack Uses its own CloudStack API library. 9. What, then, is knife-cloudstack-fog? Forked from an earlier, pre-Edmunds knife-cloudstack which worked sometimes. Developed initially by me. Several contributors, but still mostly my project. Provides coverage for almost every CloudStack API call needed to manage infrastructure. Uses FOG for its CloudStack API calls. 10. What is FOG? Cloud-independent/agnostic toolkit for Ruby Initial CloudStack support added by Brian Dorry (also of Datapipe) Uniform-ish access to all Cloud resources (independent of provider) using common-ish language Used by Chef first-party knife plugins 11. What is knife? (Hopefully you know this already) CLI interface for Chef Built to be extensible and support plug-ins Many of these plug-ins are for VM provisioning Plugins include support for: AWS, CloudStack, OpenStack, Google Compute Engine, VMware, XenServer, Docker... Used to interact with Chef server and nodes 12. What does knife-cloudstack-fog provide? One-stop-shop for all information to manage CloudStack instances **CLOUDSTACKCOMMANDS** knifecloudstackdiskofferinglist knifecloudstackkeypaircreate-kNAME(options) knifecloudstackkeypairdeleteNAME knifecloudstackkeypairlist knifecloudstacknetworkcreate-nNAME-oNETWORKOFFERINGID-zZONE(options) knifecloudstacknetworkdeleteID knifecloudstacknetworklist knifecloudstacknetworkofferinglist knifecloudstackportforwardingrulelist knifecloudstackpublicipcreate(options) knifecloudstackpubliciplist knifecloudstacksecuritygrouplist(options) knifecloudstackservercreate-sSERVICEID-tTEMPLATEID-zZONEID(options) knifecloudstackserverdeleteINSTANCE_ID knifecloudstackserverdestroyINSTANCE_ID knifecloudstackserverlist(options) knifecloudstackserverstartINSTANCE_ID knifecloudstackserverstopINSTANCE_ID(options) knifecloudstackserviceofferinglist knifecloudstacktemplatelist(options) knifecloudstackvolumelist knifecloudstackzonelist 13. With all these options, what is the main thing we use this plugin for? Provisioning servers! 14. knife cloudstack server create --help knifecloudstackservercreate-sSERVICEID-tTEMPLATEID-zZONEID(options) --server-urlURL ChefServerURL --chef-zero-portPORT Porttostartchef-zeroon --keyKEY APIClientKey -AKEY, YourCloudstackAccessKeyID --cloudstack-access-key-id --cloudstack-api-endpointENDPOINT YourCloudstackAPIendpoint -g,--groupidsSECURITYGROUPIDS CommaseparatedlistofCloudStackSecurityGroupIDs. -GSECURITYGROUPNAMES, CommaseparatedlistofCloudStackSecurityGroupnames. Eachgroupnamemustbeencapuslatedinquotesifit containswhitespace. --groupnames -w,--networkidsNETWORKIDS CommaseparatedlistofCloudStacknetworkIDs. -KSECRET, YourCloudstackAPISecretAccessKey --cloudstack-secret-access-key -s,--serviceidSERVICEID TheCloudStackserviceofferingID. -t,--templateidTEMPLATEID TheCloudStacktemplateIDfortheserver. -Z,--zoneidZONE TheCloudStackzoneIDfortheserver. --[no-]color Usecoloredoutput,defaultstofalseonWindows,true otherwise -c,--configCONFIG Theconfigurationfiletouse --defaults Acceptdefaultvaluesforallquestions --disable-editing DonotopenEDITOR,justacceptthedataasis -DDISKOFFERINGID, SpecifieseithertheDiskOfferingIDforthe ROOTdiskforanISOtemplate,oraDATAdisk. --diskoffering -d,--distroDISTRO Bootstrapadistrousingatemplate;defaultis'chef-full' -e,--editorEDITOR Settheeditortouseforinteractivecommands -E,--environmentENVIRONMENT SettheChefenvironment -F,--formatFORMAT Whichformattouseforoutput -H,--hostnameNAME Theinstancehostname -iPRIVATE_KEY_FILE, ThePrivatekeyfileforauthenticatingSSHsession. --keypairoptionisalsoneeded. --identity-file -k,--keypairKEYPAIR TheCloudStackKeyPairtouseforSSHkeyauthentication. -z,--local-mode Pointknifecommandsatlocalrepositoryinsteadofserver 15. With all those options, how do we start? 1. Pick a Zone 2. Pick a Template 3. Pick a Service Offering 4. Pick a Network/Security Group 5. Add any additional options Then, launch the VM! 16. knife cloudstack zone list No configurable options knifecloudstackzonelist ID Name NetworkType 3 NewYorkMetroEPN Advanced 5 NewYorkMetro Advanced 6 HongKong Advanced 7 SiliconValley Advanced 8 HongKongEPN Advanced 9 London Advanced 11 Shanghai Advanced 12 LondonEPN Advanced 13 SiliconValleyEPN Advanced 14 ShanghaiEPN Advanced 08e8fdf1-f64a-4ab6-bd21-7bffd578be9e IcelandEPN Advanced e20be308-482b-4d10-8885-cea306e87de9 SiliconValley2 Advanced cb765f1b-2638-465e-9c3d-9013e04116da SiliconValley2EPN Advanced 51345d53-bf3b-4280-b5dd-8541703eff9a SingaporeEPN Advanced 4baf6857-c7db-437a-96e6-f420761e128c Singapore Advanced 17. knife cloudstack template list --help knifecloudstacktemplatelist(options) -s,--server-urlURL ChefServerURL --chef-zero-portPORT Porttostartchef-zeroon -k,--keyKEY APIClientKey -AKEY, YourCloudstackAccessKeyID --cloudstack-access-key-id --cloudstack-api-endpointENDPOINT YourCloudstackAPIendpoint -KSECRET, YourCloudstackAPISecretAccessKey --cloudstack-secret-access-key --[no-]color Usecoloredoutput,defaultstofalseon Windows,trueotherwise -c,--configCONFIG Theconfigurationfiletouse --defaults Acceptdefaultvaluesforallquestions -d,--disable-editing DonotopenEDITOR,justacceptthedataasis -e,--editorEDITOR Settheeditortouseforinteractivecommands -E,--environmentENVIRONMENT SettheChefenvironment -L,--filterFILTER Thetemplatesearchfilter.Defaultis'featured.' Otheroptionsare'self,''self-executable,' 'executable,'and'community.' -F,--formatFORMAT Whichformattouseforoutput -H,--hypervisorHYPERVISOR Limitresponsestotemplatesonlyrunningona specifichypervisor.Defaultprovidestemplates fromallhypervisors. --local-mode Pointknifecommandsatlocalrepositoryinstead ofserver -u,--userUSER APIClientUsername --print-after Showthedataafteradestructiveoperation -T,--templateidTEMPLATEID LimitresponsestoasingletemplateID. Defaultprovidesalltemplates. -V,--verbose Moreverboseoutput.Usetwiceformaxverbosity -v,--version Showchefversion -y,--yes Sayyestoallpromptsforconfirmation -Z,--zoneZONE Limitresponsestotemplatesonlylocatedina specificzone.Defaultprovidestemplatesfromallzones. -z,--zoneidZONEID Limitresponsestotemplatesonlyrunningina specificzone(specifiedbyID#).Defaultprovides 18. knife cloudstack template list knifecloudstacktemplatelist ID Hypervisor Size(inGB) ZoneName ZoneID 086ce1e1-3e92-4a9e-b416-2dd5445c1e7f VMware 20 IcelandEPN 08e8fdf1-f64a-4ab6-b 1573395c-7489-4a93-9ca3-11a51a285457 VMware 20 IcelandEPN 08e8fdf1-f64a-4ab6-b d16563da-64c6-4358-b6e6-0c313c43223c VMware 20 IcelandEPN 08e8fdf1-f64a-4ab6-b 187dc53e-8f4b-405b-9067-7ee056c75fcc VMware 20 IcelandEPN 08e8fdf1-f64a-4ab6-b 20e58c82-2b7b-4552-adb2-59d4f28021cd VMware 40 IcelandEPN 08e8fdf1-f64a-4ab6-b 27b96e7e-7315-48f2-8ac0-4a4aa7cc7e14 VMware 20 IcelandEPN 08e8fdf1-f64a-4ab6-b 27fa2e69-ad13-43c2-b000-56cce241f769 VMware 20 IcelandEPN 08e8fdf1-f64a-4ab6-b 88c62685-b6f5-4acc-9ca4-9b937e5112b5 XenServer 20 IcelandEPN 08e8fdf1-f64a-4ab6-b 0f755a47-92bf-4bbf-beb0-bb3fc2ae6c91 VMware 20 IcelandEPN 08e8fdf1-f64a-4ab6-b a494ea35-e64f-40d4-a30c-b5c9dc1971bd VMware 20 IcelandEPN 08e8fdf1-f64a-4ab6-b ... 19. knife cloudstack serviceoffering list No configurable options knifecloudstackserviceofferinglist ID Name Description f26eb96f-98d5-4f34-81b0-d817c81227a2 kilo-1-40-hpc 1CPUcore,4Gmemory,highperformancerootvolume 2b9a7b24-c977-467e-b0c8-eb0c97512f22 nano-h-5-hpc 0.5CPUcore,512MBmemory,highperformancerootvo 9d865b43-296e-4ea6-83a2-122bc9baf6ac nano-h-10-hpc 0.5CPUcore,1Gmemory,highperformancerootvolum 39 mega-2-160 2CPUcore,16Gmemory,standardrootvolume 29ef5f70-8346-472a-9f84-f00a1fd727a7 kilo-4-80-hpc 4CPUcore,8Gmemory,highperformancerootvolume 38 mega-4-160 4CPUcore,16Gmemory,standardrootvolume 4414bc19-7ee8-443f-80cc-0b19ff274ad9 nano-1-10-hpc 1CPUcore,1Gmemory,highperformancerootvolume 37 mega-8-160 8CPUcore,16Gmemory,standardrootvolume 10f888d6-f6aa-4ebb-8f40-9acc5c4f0fa4 kilo-1-20-hpc 1CPUcore,2Gmemory,highperformancerootvolume ... 20. knife cloudstack network list Used for Advanced Isolated or Advanced VPC Zones knifecloudstacknetworklist ID Name ZoneID 1a036de9-dd01-4f9a-af80-f5deac2d6131 Demo1 3 4cbc9ef7-d276-4858-beb0-ce6643a247b8 TestingNetwork 3 91b06fd1-bf74-4d02-a28a-0d9aca3d5fd5 SiliconValleyEPNTest 13 623 DefaultNetwork 12 535 DefaultNetwork 3 3d4af98c-7787-417a-8e26-8f265129e7c3 defaultGuestNetwork 4baf6857-c7db-437a-96e6-f42076 1124 SharedSGenablednetwork 5 ... 21. knife cloudstack securitygroup list Used for Basic or Advanced with Security Groups Zones knifecloudstacksecuritygrouplist ID Name Description 129 default DefaultSecurityGroup 295 NetworkTesting 320 ChefServer 324 TestSG 432 PuppetServer f2bf049f-8c6b-4b8f-87d1-4846da688dbf AppTesting ... 22. knife cloudstack server create knifecloudstackservercreate-Z4baf6857-c7db-437a-96e6-f420761e128c-tb5870d22-e586-4da9-add5-100dc1c1 Waitingforserver.... Name:0bdb2cd0-d991-4d3c-8c4e-809597dfaf05 PrimaryIP: Username:root Password:jA2rszsie WaitingforSSH. 23. Other information is available from knife-cloudstack-fog for: disk offerings Creating data-disks with instances Creating VMs from ISO templates for the ROOT disk keypairs Assumes that the cloud-set-guest-sshkey is in place. Supports creation of and deletion of keypairs (end user feature request) networks Supports creation and deletion of networks (for Advanced Zones) portforwardingrules For Advanced Zone networks, NAT rules can be created with knife Server stop/start SSH Gateway Random port forwarding (for Advanced Zones) 24. Future Move code to knife-cloud? One plugin for all Clouds Common semantics LOTS of work to do Merge with knife-cloudstack? Unlikely as some core knife-cloudstack devs are anti-FOG knife-cloudstack-fog more feature rich, minus stacks and system VMs Tests There currently are no tests. I don't know how to write tests (as of March 2014) If someone wants to start writing tests, please do! 25. Bonus content! There is a Test Kitchen driver for CloudStack/CloudPlatform! (...and I wrote it!) The setup for the driver is on the GitHub page ( There's also a Vagrant driver for CloudStack/CloudPlatform (...and I didn't write it!) 26. So, how do I get in on this? Datapipe Stratosphere Global IaaS Platform running on Citrix CloudPlatform Install Apache CloudStack Available from YUM and apt repositories for EL-6 and Ubuntu 12.04 I've built a Docker build script for CloudStack! Still under heavy work. Definitely not recommended for production. (Currently coordinating with CloudStack peoples on how we want this in the Docker Index) 27. Questions? Jeff Moody Datapipe ( E-Mail: / GitHub: Twitter: @fifthecho YouTube: 28. Thank you! Also many thanks to the Chef community, FOG developers, my contributors, and people who submit issues. Speaker Office Hours: Today 3:15- 3:55 in Marina </p>