empowering developers to deploy their own data stores
TRANSCRIPT
![Page 1: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/1.jpg)
Empowering developers to deploy their own data stores.
A story of Terraform, Puppet and rage
Tomas Doran@bobtfish
![Page 2: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/2.jpg)
• Iterate on the things you do often
• Hide complexity
• Empower others
2
Devops = Workflow
![Page 3: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/3.jpg)
• A thing of the past (mostly) • Need to be able to scale up and down in hours • If not minutes
• Need to allow people to experiment • Cloud is expensive, unless you use it!
3
Artisanal hand-crafted servers
![Page 4: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/4.jpg)
• ‘Infra’ layer • DNS / puppet / apt - basic services • A(WS)?nycast - failover / HA
• ‘App’ layer • Smartstack - Service discovery + routing • Paasta (Mesos + Marathon) - Scheduling + Orchestration • search24-reviews-uswest1aprod - ugh!
4
2 Layer architecture
![Page 5: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/5.jpg)
• Remembering the . on PTR records
• For some people! • Why make them do this?
5
The hardest thing
![Page 6: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/6.jpg)
• Datastore PAAS • Elasticsearch clusters are the ‘easy’ case
• No ‘master’ - all machines are equal • Automatic sharding/replication
• ASG + ELB • Zookeeper for discovery
6
Next logical step
![Page 7: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/7.jpg)
• curl http://10.29.0.3:8142 (A(WS)nycast puppetmaster)
{ “habitat”: “uswest1aprod”
}
• “habitat”, “region”, “superregion”, “ecosystem”
7
Environment server
![Page 8: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/8.jpg)
• curl http://10.29.0.3:8142 (A(WS)nycast puppetmaster)
{ “habitat”: “uswest1aprod”
}
• “habitat”, “region”, “superregion”, “ecosystem”
8
Environment server
![Page 9: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/9.jpg)
• Hostname: search1-reviews-uswest1aprod • Parse out cluster name
elasticsearch_cluster { ‘reviews’: }
puppet/modules/elasticsearch_cluster/data/cluster/reviews.yaml
• Can locate the ‘data’ directory somewhere else! • Reuse the same YAML for service discovery + provisioning • Commit hook validation
9
puppet data in modules
![Page 10: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/10.jpg)
• External Node Classifer • Puppetmaster calls a script, returns node definition • Create node definition from EC2 tags
puppet::role::elasticsearch_cluster => cluster_name=reviews
• Stop needing individual hostnames! • Pre-allocate names using GENERATE
10
puppet ENC
![Page 11: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/11.jpg)
• Bad abstraction for contextual information • Which db server is the master? Does it have ‘master’ in it’s FQDN? • If it does, what happens when you promote another machine?
• Need key => value for cattle not pets
• Customize your monitoring system to actually tell you what’s wrong! • ‘The master db has crashed’ vs ‘A db has crashed’ • ‘10-46-11-54 is dead’ vs ‘zookeeper::10-46-11-54 is dead`
11
Hostnames
![Page 12: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/12.jpg)
• Got most of the pieces • Machines auto-configure themselves after launch.
• Remaining step is actually launching machines
• Terraform is awesome… • IF you treat it as a low level abstraction • IF you keep things in composeable units • IF you add enough workflow to not run with scissors
12
Terraform
![Page 13: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/13.jpg)
13
![Page 14: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/14.jpg)
14
![Page 15: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/15.jpg)
15
![Page 16: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/16.jpg)
• Terraform the most generic abstraction possible • Map JSON (HCL) DSL => CRUD APIs • Cannot do implicit mapping
• But puppet / ansible / whatever can??? • ‘Name’ tag => namevar • Only works in some cases - not everything has tags!
• Implicit mapping is evil • Duplicates will screw up your day
16
Low level
![Page 17: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/17.jpg)
17
Implicit mapping example - puppet AWS
![Page 18: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/18.jpg)
18
Implicit mapping example - puppet AWS
![Page 19: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/19.jpg)
19
Implicit mapping example - puppet AWS
![Page 20: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/20.jpg)
20
Implicit mapping example - puppet AWS
![Page 21: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/21.jpg)
• BUG - prefetch method eats exceptions (fixed now)
21
Implicit mapping example - puppet AWS
![Page 22: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/22.jpg)
• BUG - prefetch method eats exceptions (fixed now)
22
Implicit mapping example - puppet AWS
![Page 23: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/23.jpg)
• Reusable abstraction (in theory)
• Don’t try to use like puppet! • Flat hierarchy (do not nest modules) • Use version tags • Use other git repos
• Or just generate resources as JSON
• KISS23
Terraform modules
![Page 24: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/24.jpg)
• Why even is state? • How to cope with state
• Atlas • Workflow (locking!) is your problem • Remote state
• Shard terraform for (team) concurrency • S3 store
• Many read, few write • Wrap it yourself (make, Jenkins, don’t install terraform in $PATH)
24
State
![Page 25: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/25.jpg)
• Provides the workflow
• ‘awsadmin’ machine + IAM Role as slave
• Makefile based workflow
• Jenkins job builder to template things
25
Jenkins
![Page 26: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/26.jpg)
• Refresh state (upload refreshed state) • Plan + save as artifact • Filter plan! • Approve plan • Apply plan, save state
26
Split up the steps
![Page 27: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/27.jpg)
• Commit some files to git. • Push to a branch • Jenkins runs • Gated approval/application process
• Abstract away the scary parts • Enforce workflow
27
Cluster provisioning workflow
![Page 28: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/28.jpg)
• Self service cluster provisioning • Developers define their own clusters • 1 click from OPs to approve
• Owning team gets accounted • AWS metadata added as needed. • All metadata validated.
• Clusters built around best practices • Can abstract further in future
28
Nirvana
![Page 29: Empowering developers to deploy their own data stores](https://reader034.vdocuments.net/reader034/viewer/2022052514/58701e0d1a28ab7f428b746b/html5/thumbnails/29.jpg)
P.S. We’re hiring!
@bobtfish
engineeringblog.yelp.com
github.com/Yelp
github.com/bobtfish