devops for drupal: why we cook with chef
Post on 15-Jan-2015
Embed Size (px)
DESCRIPTIONDevOps for Drupal presentation given at DrupalCon 2013 in Portland. Promet Source shares secrets for automation and how to make your infrastructure hum.
- 1. DevOps for Drupal: WhyWe Cook with ChefWednesday May 22ndDevOps TrackBeginner
2. Overview: PurposeWhy are we all here? Interested in DevOps practices Interested in automation tools like Chef Interested in using them with Drupal+ 3. Overview: SharingHow are we going to do this? Team style presentation and discussion withviews from both sides - Development andOperations Share our experiences and examples... wewill be keeping it real We are NOT doing a tool comparison+ 4. Overview: LearnWhat do we hope you will learn? Why DevOps matters Why automation is a must Why you can use Chef to help yourselves,make life easier, and save time+ 5. For more info on Chef, check out fellowDevOps track session:The Joy of Cooking - Whip Up a DrupalEnvironment with Chef Thursday May 23rd at 2:15pm OR 203 Presented by Opscodes Nathen HarveyOverview: More on Chef + 6. Who We AreA software development shop... specializing in custom Drupal development,systems integration, mobile, DevOps, and24x7 support based in Chicago with 30+ team membersworldwide...and yes we are hiring! Come talk to us!+ 7. Who We ArePlaying the part of:Moderator andProduct Owner@ Promet Source:Director of Productsjay@promethost.com+Jay Uhlinger 8. Who We ArePlaying the part of:Development@ Promet Source:Solutions Architectwill@promethost.comdrupal.org: wamiltontwitter/hub: @winmillwill+Will Milton 9. Who We ArePlaying the part of:Operations@ Promet Source:Systems Administratorgreg@promethost.com+Greg Palmier 10. Audience Quiz TimeHow many are Developers? (a Will)+CC Image courtesy of Pedro Lozano on Flickr 11. Audience Quiz TimeHow many are Sysadmins? (a Greg)+CC Image courtesy of Sharyn Morrow on Flickr 12. Audience Quiz TimeHow many are both? (changing Prod servers)+CC Image courtesy of Arthur Caranta on Flickr 13. Audience Quiz TimeHow many are other? (the Product Manager)+ 14. Audience Quiz TimeHow many are using automated configurationmanagement tools and processes now inyour everyday work with Drupal?How many are using Chef?+ 15. DevOps: What It IsKey Culture Foster a collaborative working relationship betweenDevelopment and OperationsKey Concept Infrastructure as codeKey Results Higher deployment rates Better reliability, stability, and resilience of Production+ 16. DevOps: 3 Key Patterns1.Make environments available early in theDevelopment process2.Shorten and amplify feedback loops3.Create reusable deployment procedures+ 17. DevOps: What It IsntIt is not a collection of bash, Drush, insertscripts here, etc.+DevOps is not about the tools... they dont fix everythingmagicallyBut DevOps relies on some tools to implement itsprinciples 18. DevOps: Why Is It ImportantOne-off environments and complicated manualbuilds and deployments = FAILUREMust automate to meet business demands --support agile, lean, faster, better...Every environment is Production to someoneDrupal is not different -- all of these apply+ 19. A Classic ScenarioJay: Will, you said this was working.Will: It works in Dev.Jay: Greg, can you look into why it isnt workingin Prod?Greg: Ok. (later on) @#$%! Dev and Prod arenot the same!+ 20. Another Classic OneThe Blog Post: Installing Drupal on AWS1. Start wizard2. 50+ steps later3. A Drupal install+ 21. Why We Chose ChefThe orange color matches oursof course!We already had experience (Marius)doing DevOps work with Chef inBay Area with leading tech companiesWe build Drupal applications -- we found Chefto be a good fit for application configurationmanagement+ 22. The Operations part......out with the old and in with the new+ 23. Our Operations StoryHow we started with Chef at PrometInconsistency was killing our timeHow realistic was automation and configurationmanagement of all of our stuff+ 24. Wasting TimeYou are not in the sudoers file. This incident willbe reported.$ which git$Fatal error: Class PDO not found inblahblahblahAll of these messages diverted your work+ 25. Manual Config Run Amok Dev environments != Prod environments Shared Dev/Prod/?? environments werecontinually bloated Over time modifications amounted to moreinconsistency between environments Documentation was lackingo No records explaining changeso Constant paper trail "paving"o No great place for it anyways (what wiki?!?) More customizations -> High Fragility+ 26. Typing Faster != Solution Runbooks, Bash and SSHo Process to stand up new infrastructure was donemanually via a "runbook Any configuration time was repeatedextensively Environment discovery bled timeo Snooping around to analyze how things were initiallystood up and potentially modified(caching, etc.) Endless SSH-ing into servers+ 27. Chef Migration In with the Newo 2 dedicated servers VM hostso Similar model, consolidatedo Entirely Chef managedo Individual client Chef-spun Dev instances Stand Up, QA, and Cutovero Virtualize legacyo Considerations mainly PHP Monitoring customizations Special clientso Non-Drupalo Contractually unique+ 28. Chef MigrationAnd migrating wasnt a complete nightmare....+ 29. Righting the Ship Server configuration in Gito Mitigated "paper trail"o Team awareness of configs by notificationso PHP, Apache, MySQL System config changes pushed to Chef forServer propagationo No more endless SSH-ingo No more runbook config for every servero Undoes "helpful" customizations by others Direct between me and Config Mgmto Configuration entirely local with Vagranto Git driven accountability and awareness (--stat)o `knife` is to Chef as drush is to Drupal+ 30. Client Ownership Clients maintain ownership of their assetso Any platform with a knife plugino Legacy apps migratedo Non-Drupal or Promet "old" clients Give us their Keyso Let us be the Designated Drivero Developers drop in, get pitted+ 31. Client Ownership Excessive customization (not perfect)o Non-Drupal clients or other worko More so, use clients Hosted Chef Contract flexibilityo No Dysfunctional Marriage Honor prior client / hosting relationshipso Help clients where they are+ 32. Things That Are Solved User Configso .gitconfigo .tmux.confo .ssh/configo shell customizationso Not necessary but....doable Server Configso logwatcho mailo mysqlo apacheo ssho automysqlbackup+ 33. Sysadmin TransformationHave time to explore/make new toolsDo support in a consistent fashionAutomate all the Things!"Grow" your team and infrastructure not withnumbers but with talentWe do more work with less resources!+ 34. Sysadmin Team EvolutionWe even spun-off a Startup company forinfrastructure and applications automationand scaling using Chef in the Bay Area (non-Drupal)I dont care what you automate with... pick oneand do it!+ 35. The Dev part......its not what you think+ 36. What I Thought I Would Say What it was like... How it sucked for Devs How we changed it How it sucked in new ways How that lead to the new thing Repeat.... Where we are now Whats really fun about this for Devs+ 37. The Problem We went through too much shit Its not even fun to talk about Its probably boring Someone else already said what I was tryingto justify Youre probably the choir if youre here+ 38. Drupal: OH @crell"If youre not using Features, you are not doingDrupal professionally"Also: If you dont have a one step build, you arentusing Features correctly=> Sucking at doing Drupal professionally Configuration module is also probably fine Its going into core+ 39. So Now My Presentation Is What were doing now Why you shouldnt NOT do what were doing Actual examples of how it has helped Why Chef is really awesome A surprise+ 40. What Were Doing Now Everything you need in the repoo Any scripts or scripting exampleso README fileso (small(-ish)) csvs/xml/etco ...pretty much any text file that is not too large and isuseful ...Still need a way to automate sourcingarchives of files and sql... Vagrantfile + Berksfile+ 41. Berks What?+ 42. + 43. Kirkberks? + 44. What a Dev Has to Do onOur Team Install git somehow Install the vagrant package Install the vagrant-berkshelf plugin git clone/pull && cd-to-the-root vagrant up (or reload) edit the hosts file or use dnsmasq or similar+ 45. Some Concrete Examples ofThis Helping Us Do Better Auditso http://reload.github.io/phing-drupal-templateo Not perfect...o But now no one has to repeat what is automated Every single deployment ever sinceo Your Production push should not be the first timeyou attempt syncing to a Production environmento ...unless you crave excitement Last minute accommodationso Ever test RewriteBase?+ 46. Add the .htaccessdirective... +...change the docroot in code! 47. Potential Examples: YouCant Prove the Negative PECL/Pear packageso Shame on professionals who cant figure it out...o ...but shame on me for betting the project Anything to do with email ever since ever Anything to do with external serviceso ...memcachedo ...SOLRo ...even MySQL+ 48. Why You Shouldnt NOT DoAs We DoChallenges you can deal with, excusesthat are bullshitOn stupid things people say re: vagrant andchef We make great docs! My native environment is fine Virtualization consumes resources I dont have enough time+ 49. + 50. My Native Environment isFine1.No, it isnt Your job is to ship to your downstreamo ...not manicure and fetishize your tools You are getting strictly less help than youcould geto ...even if youre getting a lot of helpo ...even if youre paying for help You are keeping yourself from helping+ 51. My Native Environment isFine (contd)2.Your native environment is personal You may even obsess over it You may be completely wrong You will be happier without each other+ 52. Virtualization ConsumesResources + 53. I Dont Have Time......to do everything perfectly We didnt either, and were doing far betterthan we were We do not have a massive continuousdelivery, cloud orchestration and scalinginfrastructure ... But now what we have is in code+ 54. Chef is Really Awesome...Puppet is probably cool too Stewardship and leadership Constantly e