de-centralise and conquer: masterless puppet in a dynamic environment
DESCRIPTION
"De-centralise and Conquer: Masterless Puppet in a dynamic environment" by Sam Bashton of Bashton Ltd., at Puppet Camp London 2013. Learn about upcoming Puppet Camps at http://puppetlabs.com/community/puppet-camp/TRANSCRIPT
![Page 1: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/1.jpg)
De-centralise and Conquer
Masterless Puppet in a Dynamic Environment
Sam Bashton, Bashton Ltd
![Page 2: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/2.jpg)
Who am I?
● Linux guy since Slackware, floppy disks and
root + boot
● Using Puppet since 2007
● Run a company Manchester, North West
England
![Page 3: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/3.jpg)
● We provide outsourced ops for other
companies
● High traffic environments
● Most are now on Amazon Web Services
● #1 reason for moving to AWS? The ability to
scale on demand
Our Environments
![Page 4: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/4.jpg)
Server instances, single day
![Page 5: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/5.jpg)
How we use Puppet
● No Puppetmaster
● Puppet manifests and modules distributed to
all machines
![Page 6: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/6.jpg)
What's wrong with standard Puppet?
● Pets vs Cattle
● Standard Puppet configuration assumes that
servers are pets, not cattle
![Page 7: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/7.jpg)
What's wrong with standard Puppet?
● Standard Puppetmaster/Puppet Client
configuration makes assumptions about
environments○ Machine creation is a manual operation
■ Sign certs
○ No in-built mechanism to automatically clean up old
machines
![Page 8: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/8.jpg)
What's wrong with standard Puppet?
● Puppetmaster is a single point of failure
● When servers are pets, this isn't too much of
a problem○ Existing servers continue to work, but not any
updates
![Page 9: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/9.jpg)
What's wrong with standard Puppet?
● When servers are auto-scaling cattle, new
instances can appear at any time
● New instances require config to become
operational
● Configuration requires Puppet
![Page 10: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/10.jpg)
What's wrong with standard Puppet?
● Our environments span multiple data centres
('availability zones')
● Imagine a data centre fails
● New instances get auto-provisioned to
replace missing capacity
● But these instances need the Puppetmaster
● ..which was in the failed AZ
![Page 11: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/11.jpg)
What's wrong with standard Puppet?
● Resource contention
● Even when Puppetmaster isn't in the failed zone, multiple concurrent connections slow things down
![Page 12: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/12.jpg)
What's wrong with standard Puppet?
● None of these problems are insurmountable
● We could have configured a Puppetmaster a
cluster of Puppetmasters for our needs○ With autosign
○ and some sort of certificate distribution mechanism
○ uuid certificate names
○ And a mechanism for cleaning up old machines
![Page 13: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/13.jpg)
Meanwhile, on the other side of the room...
● Another team was evaluating Pulp
● Provides yum repository management
● To be used for managing security updates
and deploying application code
http://pulpproject.org/
![Page 14: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/14.jpg)
Pulp
● Allows cloning of repos, copying packages
between repos
● Allows us to push packages to clients○ Uses qpid message queue
● Has 'content distribution servers' for easy
replication + clustering
![Page 15: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/15.jpg)
How we deploy code
● Everything managed via the Jenkins
continuous integration server
● Jenkins uses Pulp to install code on remote
machines
![Page 16: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/16.jpg)
How we deploy code
● Jenkins fetches code from source control
(git)
● An RPM is built
● Tests are run
● The RPM is added to the relevant Pulp
repository
● RPM installed on the target machine(s)
![Page 17: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/17.jpg)
How we deploy code
● Jenkins also manages deployment lifecycle
● 'Promoted Builds' plugin used to install
previously built RPMs on staging
● Promoted Builds plugin then used to install
the same RPMs on live once testing is
complete
![Page 18: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/18.jpg)
Deploying configuration as code
● Idea: Why not just build an RPM of our
Puppet manifests + modules?
● Have puppet apply as part of the %
postinst
![Page 19: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/19.jpg)
Deploying configuration as code
● Allowed us to reuse our existing code
deployment infrastructure
● Manage configuration deployment from
Jenkins
![Page 20: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/20.jpg)
How we deploy configuration
● Puppet manifests and modules are checked
into git
● Jenkins builds configuration into an RPM
● Jenkins promoted builds plugin applies the
updates to environments via Pulp
![Page 21: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/21.jpg)
Our system architecture
● Quite AWS specific
● Concepts could be applied to other clouds○ Once they catch up in terms of toolsets..
![Page 22: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/22.jpg)
Separation of Roles
● CloudFormation - defines infrastructure
● Puppet manages configuration
● Pulp manages package versions○ Pulp in turn managed via Jenkins for custom repos
![Page 23: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/23.jpg)
Instance Provisioning
● Minimal images used
● cloud-init the only addition beyond standard
CentOS install
● cloud-init allows us to specify script to be run
at boot
![Page 24: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/24.jpg)
Puppet bootstrap
● cloud-init script adds local Puppet yum repo
and installs the Puppet configuration RPM
● Installing the RPM installs Puppet and
applies the configuration
![Page 25: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/25.jpg)
Machine metadata
● cloud-init also sets some variables in
/etc/environment
● $HOST_TYPE - the type of machine this is, eg
web, cache
![Page 26: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/26.jpg)
Machine metadata
● Also set facts to be used by facter, eg RDS
database hostname○ Values from CloudFormation
● $FACTER_DBHOSTset via cloud-init too, eg /root/.my.cnf
![Page 27: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/27.jpg)
Defining machine roles
● For each machine type there is a manifest /etc/puppet/manifests/$HOST_TYPE.pp
● This file looks something like this:node default {
import global
...
}
![Page 28: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/28.jpg)
Building the RPM
● Puppet manifests and modules are all
packed into an RPM
● Owner set to root, mode 600
● %postinst creates an at job set for now + 1
minute to run puppet apply
![Page 29: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/29.jpg)
Deploying configuration
![Page 30: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/30.jpg)
Free wins!
![Page 31: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/31.jpg)
Free wins
● Greater control over the timing of Puppet
runs
● Improved visibility - for ops and devs
● Configuration changes now have to be
deployed to testing/staging first
![Page 32: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/32.jpg)
More free wins
● Puppet configs now have a version
● Easy to find config version on the machine
itself
● Config changelogs accessible on every
machine○ (Git changelog added to RPM)
![Page 33: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/33.jpg)
Cheap wins
![Page 34: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/34.jpg)
Cheap wins
● Jenkins performs syntax checks with puppet parser validate
● Jenkins also runs puppet-lint on
manifests
![Page 35: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/35.jpg)
Cheap wins
● Config change required for new code?○ Make the Puppet RPM version a dependency
![Page 36: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/36.jpg)
The downsides
● Puppet manifests and modules on all
machines○ Potentially a security issue?
● No reporting*
![Page 37: De-centralise and Conquer: Masterless Puppet in a Dynamic Environment](https://reader034.vdocuments.net/reader034/viewer/2022052522/554bb7afb4c90594278b4a6f/html5/thumbnails/37.jpg)
Alternative implementations
● Don't want to use Pulp?
● Could do basically the same thing with yum
s3 plugin
https://github.com/jbraeuer/yum-s3-plugin