Download - There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly
![Page 1: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/1.jpg)
Charity Majors @mipsytipsy
![Page 2: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/2.jpg)
Charity Majors @mipsytipsy
![Page 3: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/3.jpg)
There and back again: a Chef tale
How we drank the Kool-Aid, sobered up, and learned to cook responsibly.
![Page 4: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/4.jpg)
Mobile apps platform
500k+ apps
AWS
MongoDB, Cassandra, Mysql, Redis
ruby & rails => golang
![Page 5: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/5.jpg)
Our mission:
• Support relentless growth
• Ship products fast
• Solve mobile apps naively at scale
![Page 6: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/6.jpg)
Active monthly Parse installations
![Page 7: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/7.jpg)
API requests per second
![Page 8: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/8.jpg)
• Support relentless growth
• Ship products fast
• Solve mobile apps naively at scale
Our mission:
![Page 9: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/9.jpg)
our mission
your mission
![Page 10: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/10.jpg)
![Page 11: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/11.jpg)
![Page 12: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/12.jpg)
Chef the Base System!!
• bootstrapping nodes with knife-ec2
• configuring system packages
• managing deb versions
• ec2 hostname tags from chef node names
• route53 DNS records from hostname tags
• cron jobs, batch jobs
![Page 13: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/13.jpg)
Chef the Services!!
• haproxy configs
• generate yaml files
• generate host lists
• manage config files for Parse services
• monitoring and graphing based off roles
![Page 14: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/14.jpg)
Chef the Databases!!
• creating/managing mongo replica sets
• provisioning & assembling RAID devices
• assigning cassandra initial tokens
• backups, snapshotting & restores
• community cookbooks for mysql, redis
![Page 15: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/15.jpg)
Chef the Deploys!!
• deploy Parse services?
….??????
![Page 16: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/16.jpg)
wait …
![Page 17: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/17.jpg)
![Page 18: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/18.jpg)
1) Things we did with chef badly
2) Things that chef was not the right tool for
![Page 19: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/19.jpg)
mistakes were made …
• Overloading roles with too much work
• Confusion between role vs instantiation of service
• Using definitions instead of providers
• Using lots of data bags
• One attribute per config entry instead of a hash of all entries
• Using knife search extensively
![Page 20: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/20.jpg)
mistakes were made …
• Forking + modifying community cookbooks
• Importing community cookbooks with too many custom dependencies
• Not using repo-per-cookbook / Berkshelf
• Not investing the time into vagrant, unit tests, staging environment, versioning
• Where is my source of truth?!
![Page 21: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/21.jpg)
but these are all solvable problems.
![Page 22: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/22.jpg)
but these are all solvable problems.
what isn’t?
![Page 23: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/23.jpg)
sometimes, chef just ain’t enough.
![Page 24: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/24.jpg)
• Provisioning from scratch
• Service registration & discovery
• Managing software & configs
• Databases
Problem areas
![Page 25: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/25.jpg)
bootstrapping from vanilla AMIs
launching instances with knife-ec2
Provisioning
![Page 26: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/26.jpg)
bootstrapping from vanilla AMIs
launching instances with knife-ec2
Provisioning
Solution: bake AMI with chef, use ASGs
![Page 27: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/27.jpg)
realtime search needs realtime data
Service discovery
![Page 28: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/28.jpg)
realtime search needs realtime data
Service discovery
Solution: zookeeper, consul, etcd, etc
![Page 29: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/29.jpg)
Service discovery
avoid snowflake hosts
use distributed locking for cron jobs
![Page 30: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/30.jpg)
Managing software & configs
• System software (debs, rpms)
• Developer-owned services
• Internal operations software
![Page 31: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/31.jpg)
Managing software & configsSystem software
![Page 32: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/32.jpg)
Managing software & configsDeveloper-owned services
• Do not tie code deploys to system changes
• Perform the minimal set of changes
• Configs *are* software. Version together.
![Page 33: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/33.jpg)
Managing software & configsInternal operations software
• Treat software engineering like software engineering
• Treat systems-y packages like systems packages
• Package and version “util” scripts
• Manage package versions with Chef
![Page 34: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/34.jpg)
Databases at scale
![Page 35: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/35.jpg)
DatabasesDBA operations
Not really what chef is best at.
Imperative commands
Automatic remediation
Coordinating actions across nodes
![Page 36: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/36.jpg)
DatabasesDBA operations
• Create, tear down replica sets or nodes
• Verify backups
• Rolling version upgrade
• Elect new primary / switch masters
• Enable/disable query killer
• Change schemas or indexes
• Compaction, rotation
• Version replica set state
• Etc
![Page 37: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/37.jpg)
DatabasesDBA operations
If you don’t have to do a ton of DBA ops, Chef can manage databases.
Don’t over-engineer in advance of your actual needs.
![Page 38: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/38.jpg)
DatabasesSeparation of configuration and state
Base system => chef
Detect and publish state changes => chef, zk
Generate monitoring configs => chef
Imperative commands => db tooling
![Page 39: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/39.jpg)
Databases at scale
![Page 40: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/40.jpg)
We chef for:
• Building base AMIs
• Generating monitoring configs
• Storing encrypted secrets
• Cron jobs (with zk lock)
• Inferring and publishing db state changes
![Page 41: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/41.jpg)
Things we still suck at
• Single source of truth (git / chef-server)
• Isolated staging environment
• Full continuous testing for cookbooks
![Page 42: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/42.jpg)
• Realtime data
• Internal software packaging & management
• Database administration at scale
Things we don’t chef
![Page 43: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/43.jpg)
![Page 44: There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly](https://reader033.vdocuments.net/reader033/viewer/2022042819/55ce3184bb61eb341c8b4793/html5/thumbnails/44.jpg)
Charity Majors
@mipsytipsy