operations playbook: monitoring and automation - rightscale compute 2013

Automation + Monitoring Chris Deutsch, RightScale Operations

Cloud Management Platform

Cloud Management Platform

What I'll be talking about • Meet RightScale Operations • Monitoring

o How monitoring works on RightScale o How to build a custom monitor o How we monitor web servers and cassandra

• Automation o The RightScale API o The chimp command line tool o How we automate releases

• Tips from Ops

Cloud Management Platform

RightScale Operations • Deployed over 5 continents • Over 700 cloud servers administered • RightScale runs on RightScale

Cloud Management Platform

collectd: what is it? • open source metric collection tool • modular architecture • uses the ubiquitous rrdtool • more information: http://collectd.org/

Cloud Management Platform

collectd: built-in plugins • host monitoring

o cpu o disk space o disk I/O o memory o network

• application monitoring o process state o memory use o cpu usage

Cloud Management Platform

How does monitoring work?

Cloud Management Platform

How does monitoring work?

Cloud Management Platform

How does monitoring work?

Cloud Management Platform

How does monitoring work?

Cloud Management Platform

collectd: custom plugins •  Custom plugins written using the Exec plugin •  Can be written in any language •  Ruby, python and perl are common •  Simple

Cloud Management Platform

collectd: custom plugins What we're going to look at: • building an example monitor using the Exec plugin • http error code monitor • cassandra database server monitor

Cloud Management Platform

collectd: custom plugins: example /etc/init.d/collectd/example.conf: example.rb: https://collectd.org/wiki/index.php/Plugin:Exec

#!/usr/bin/ruby while true do time = Time.now.to_i puts "PUTVAL "host/cpu-0/cpu_overview" interval=20 #{time}:1" sleep 20 end

<Plugin exec> Exec "nobody" "example.rb" </Plugin>

Cloud Management Platform

collectd: custom plugins: http codes

Cloud Management Platform

collectd: custom plugins: http codes

https://gist.github.com/christopherdeutsch/db2380a47b62730ddf69

Cloud Management Platform

collectd: custom plugins: cassandra •  cassandra is a key-value data store (aka nosql) server •  data is stored on a ring •  a ring consists of nodes

Cloud Management Platform

collectd: custom plugins: cassandra

Cloud Management Platform

automation: the rightscale api

Cloud Management Platform

automation: the rightscale api • RightScale API is RESTful and easy to traverse • right_api_client - ruby client library • CloudFlows - the future

Cloud Management Platform

automation: the command line • needed a tool that would let us be lazy • the "chimp" executes commands on servers • let's jump into a demo

Cloud Management Platform

automation: chimp • select what to update using tags • update across multiple deployments • update one server at a time so service isn't disrupted • track success/failure

Cloud Management Platform

automation: scripting languages • having a command line tool lets us use scripting languages like bash

or ruby to automate common tasks • we ended up using Ruby rake files to tie it all together

Cloud Management Platform

automation: a RightScale release • chimp used to run commands on servers • supports "rolling" operations • uses tag service to scope operations • we use rake to organise tasks that make up a release • developed chimpd so we could run more commands in parallel

Cloud Management Platform

automation: chimp release • RightScale has released chimp as open source! • gem install right_chimp

Cloud Management Platform

tips • assume instances will die eventually • always reboot test ServerTemplates • use tags. everywhere. all the time. • use chimp to make ad-hoc queries • monitor not just host metrics but system metrics • design everything to be runnable in a server array

Thanks!

Chris Deutsch, RightScale Operations christopher@rightscale.com @ispeakdeutsch

Cloud Management Platform