Getting to Push Button Deploys

Moovweb

January 19, 2012

Things Everyone Wants

Reality

… is it done yet??? … is it done yet??? … is it done yet???

Gotta Have It!

PROS

Increase velocity of change/responsiveness Get Operations out of the loop Buzzword compliance – aka Shiny!!!

PROS

Increase velocity of change/responsiveness

PROS – Ops Gets Out of Loop

Less work for Operations Developers feel more in control

Gotta Have It!

CONS

Increase velocity of change Shared responsibility Not always easy

Make It So...

Components

Components

Components

Components

Components - MoovStack

CI

Jenkins ( maiden name is Hudson ) Checks out a deploy branch from git Runs a few sanity checks Packages software Uploads to software repo Tags build in git

Jenkins

http://jenkins-ci.org/

Jenkins

Jenkins

Rake

Packaging

But it's sooooo hard

save yourself first.....

Using FPM

https://github.com/jordansissel/fpm

Building a Gem

Right Tool Packages

Something that does not change often Don't need to roll back quickly Don't need more than 1 version at a time You cache external repos or limit to just yours

Right Tool Gems

Fast roll back – gem cache Easy setup of gem server Need more than 1 version at a time Bundler for dependencies Bundle exec for peace of mind

Configuration Management

Puppet everything from bare metal/base image Seperate data from manifests with an ENC Max troubleshooting time is 20 minutes Amazon free ”chaos monkey” service

Puppet – Start with a sane base

ENC – Use Groups!

http://docs.puppetlabs.com/guides/installing_dashboard.html

ENC – Node Entries Inherit

ENC – Scripting Additions

ENC - Output

Manifests Contain ENC Variables

Templates Contain ENC Variables

Puppet - Tips

Execs hurt – try to avoid them Recursive directories – use vcsrepo instead Write your own types – just restart in between Run mongrel or passenger Use behind a load balancer for fun Use reporting, but take the DB off of your

master Write your own facts and save lots of time

Puppet - Scaling

Mcollective – Do It NOW!!

Orchestration framework Crazy Parallel Write more ruby code... Clients and Agents

Mcollective – Crazy Parallel

6380 nrpe checks across 370 hosts in 3.8 seconds

Use Facts and Classes to Filter

Mcollective – Custom Agent

Mcollective – Custom Client

Mcollective - Tips

Remeber you have a machine gun pointed at your foot

Wrap command in IO.popen and log everything Log to syslog Use ApacheMQ For EC2 tune your keep alives (registerinterval) Remember you have a machine gun pointed at

your foot

Mcollective - Scaling

Sinatra

Web app in 5 minutes Much more minimal than rails Written in Ruby so I can interface with

Mcollective easily

http://www.sinatrarb.com/

Sinatra – Addons

sinatra-respond_to - useful as API Pony – send email from Sinatra Twitter Bootstrap – make things pretty

Public Example

https://github.com/maguec/mco_dashboard_example

Public Example

Really Simple

Sinatra - Tips

Run a few behind Nginx Use helpers whenever possible Log every push Use it to integrate all your data

Actual Push Procedure

Dashboard Functionality Push software Audit software Real-time monitoring check Performance data integration Log data integration Troubleshooting tools for developers Utilities for non-technical staff Open trouble tickets Cost information (in progress) Userful links to other data sources

Oppy – an example

Oppy – an example

Oppy – an example

Oppy – an example

Oppy – an example

Oppy – an example

Thanks

Michael Catlin (organizer) Tarun Desikan (giving me free reign/enough rope) R.I. Pienaar ( Mcollective author and helpful guy) Andrew Farmer ( for helping to get this going )

Links

http://blog.moovweb.com/ http://www.devco.net/ http://www.puppetlabs.com/ http://blog.mague.com/ http://www.sinatrarb.com/ http://jenkins-ci.org/

IRC #puppet,#mcollective,#devops

Author

Chris Mague

christianmague@gmail.com

IRC: bosszaru usually on #gaijin

Twitter: @maguec