sensu at nycdevops meetup

SENSE AND SENSU-BILITYPainless Metrics And Monitoring

In The Cloud with Sensu

Bethany Erskinenycdevops Meetup

http://github.com/skymob/sensu-tutorial

Thursday, November 14, 13

DO YOU LOVE YOUR

MONITORING SETUP?


#MONITORINGLOVE


MY STORY

+

(╯（╰,)


WHY SENSU

✓Ruby

✓Plugins can be written in any language

✓sensu-chef cookbook

✓community


WHY SENSU

✓re-use Nagios checks

✓metrics and checks all collected by one system

✓Graphite integration

✓easy to scale


WHY SENSU

✓“Can I do X with Sensu?” probably!


WHY SENSU


WHY SENSU?

✓Sensu source is well-written and easy to parse

✓https://github.com/sensu


https://github.com/sensu

https://github.com/sensu

WHY SENSU?

✓sensu-community-plugins

✓80 contributors

✓over 600 plugins

✓https://github.com/sensu/sensu-community-plugins


https://github.com/sensu/sensu-community-plugins




TODAY at PAPERLESS

Two Sensu environments (prod/testing)~ 250 - 275 instances of sensu-client

4-6 Sensu-server instances25k Metrics/Hour to Graphite

1 custom dashboard1 custom CLI


RESOURCES✓All of our Sensu infrastructure is

virtualized.

✓We typically give a sensu-server box 1.5GB RAM and 4 processors, scaling up RAM for any box running more than one Sensu service on it.

✓ 4GB RAM for a monolithic Sensu install (Rabbit, Redis, all Sensu components on one)


AS WE GREWGrowing pains and lessons learned...


NEEDS MORE SENSU

✓High load on Sensu server

✓Backed-up queues in RabbitMQ

✓TIP: set up check to monitor the RabbitMQ ready queue size, you'll want an email when the queue grows about 10K and stays there


HOW TO SCALE

✓Add more sensu-server instances

✓No special configuration needed

✓checks will be distributed in round-robin fashion to the sensu-servers


GRAPHITE PAINS

✓symptoms: backed up queues in RabbitMQ, spotty graphs

✓cluster couldn’t keep up with the large amount of metrics we were now serving it via AMQP


GRAPHITE PAINS

✓Solution: stop collecting metrics every 10 seconds (excessive!)

✓moved staging metrics to staging Graphite cluster

✓Moved prod Graphite cluster to SSD


THE MIGRATIONor, How To Quit Nagios in Ten Easy Steps


STEP 1: NUKE AND PAVE


STEP 2: PLANMETRICS AND MONITORING SURVEY


METRICS AND MONITORING SURVEY


STEP 3: DEFINE GLOBALS

✓CHECKS: must be actionable!

✓METRICS: go nuts

✓HANDLERS: EMAIL for everything initially, added Pagerduty later.


OUR GLOBALS

✓CHECKS: disk usage, swap usage, zombie processes, RO filesystems

✓METRICS: vmstat, disk usage, cpu, memory, interface and disk perf

✓HANDLERS: Email, Campfire, Pagerduty


STEP 4: DEFINE SPECIFICS

✓For each server role, define additional states to be checked and alerted on:

✓Process Checks

✓System Checks

✓Service Checks

✓Service Metrics


STEP 5: SET UP A PLACE TO TEST

✓Set up a permanent testing Sensu stack using your CM tool of choice

✓we used sensu-chef cookbook


STEP 6: SET A WORKFLOW

✓Develop and document a workflow for implementing, testing, deploying and signing off on checks

✓You’ll get the best coverage if anyone (developers or ops) can easily add checks and metrics to Sensu


EXAMPLE WORKFLOW

✓add new sensu_check definitions to the appropriate cookbook in Chef

✓deploy new check to staging env using Chef

✓Pull Request with sample graphs or alerts

✓Code Review from colleague

✓Deploy to Prod


SENSU IN CHEF


STEP 7: EXECUTE WORKFLOW

✓Starting with the low-hanging fruit (plugins that already existed in sensu-community-plugins repository), configure and deploy each check in the worksheet to the testing Sensu server

✓deploy sensu-client to a few select machines


STEP 8: WATCH THE WATCHER

✓Set up some bare-minimum 3rd party monitoring for the Sensu servers


MONITOR THE MONITOR

✓Other ideas: have Testing Sensu monitor Prod Sensu

✓Sensu can collect metrics about itself


STEP 9: ROLLOUT

✓Deploy your Production server infrastructure

✓Roll out the client and checks to the rest of the your prod environments.


STEP 10: TUNE

✓Laissez le bon alertes roulent!

✓Expect to need to tune thresholds and alert occurrences.


SENSU ARCHITECTURE


OMNIBUS INSTALLER

is awesome


LET’S PLAY WITH SENSU

If you haven’t been able to get your sandboxes up and running,

please pair with someone near you.


SANDBOX GOALS✓Get familiar with Sensu

configuration

✓Install a Handler

✓Deploy a check

✓Trigger an alert on that check

✓Give you something to take home and hack on


OOPS

If you mess anything up:

vagrant halt; vagrant up

Worst case:

vagrant destroy; vagrant up


TWO VIRTUALBOXES

Sensu-Server and Sensu-ClientVagrant/Chef

Centos 6.4Sensu Version 0.10.2


SENSU CONFIGURATION

✓Please open up a terminal and SSH into both your sensu-server and sensu-client VMs

✓sudo su -

✓cd /etc/sensu


SENSU CONFIGURATION✓/etc/sensu/config.json - config for

redis, rabbitmq, api and dashboard

✓/etc/sensu/conf.d/ - checks go here

✓/etc/sensu/conf.d/client.json - client configuration, subscriptions

✓/etc/sensu/{extensions|handlers|mutators|plugins}


TRIGGER AN ALERT!

On sensu-client:

service sensu-client stop


CHECK YOUR DASHBOARD

✓Open a web browser and go to http://10.254.254.10:8080

✓username: admin / password: secret


http://localhost:8080/

http://localhost:8080/

HANDLERS

✓A HANDLER takes action on an event using a pipe, TCP, UDP, AMQP, or a set of other handlers

✓Examples: send an email, send event to Pagerduty, send metrics to Graphite

✓Default is “debug”


HANDLER EXAMPLES

✓BASIC: send an email to ops@

✓ADVANCED: attempt to remediate the alert (i.e. run a custom script that spins up additional ec2 instances)


HANDLERS

✓Let’s configure an EMAIL handler to send a informative email for an event.

✓/etc/sensu/handlers/mailer.rb plugin is installed for you, we just need to configure and install it


CONFIGURE THE PLUGIN

{ "mailer": { "mail_from": "[email protected]", "mail_to": "[email protected]" }}

ON SENSU SERVER:vim /etc/sensu/conf.d/handlers/mailer.json


mailto:[email protected]




CONFIGURE THE HANDLER

cp /etc/sensu/conf.d/handlers/default.json /etc/sensu/conf.d/handlers/email.json

vim /etc/sensu/conf.d/handlers/email.json


EMAIL.JSON

"handlers": { "email": { "type": "pipe", "command": "/etc/sensu/handlers/mailer.rb" }}


CHECK GEM DEPENDENCIES

/opt/sensu/embedded/bin/gem list | grep mail


FIX PERMISSIONS

chown -R .sensu /etc/sensu/conf.d/


RESTART SERVICES

service sensu-server restart

tail -100 /var/log/sensu/sensu-server.log | grep mail


CHECKS✓Sensu-client runs CHECKS that are

defined and scheduled either locally (standalone) or on the sensu-server (subscription).

✓A CHECK sends a RESULT as an EVENT to a HANDLER - this applies to anything - service checks, metrics, etc


CHECK EXECUTION

✓Either scheduled by the server (subscription) or scheduled by the client (standalone)

✓Today we will configure a subscription-based check on the server that will run on our client


LETS CONFIGURE A CHECK

✓Use check-procs.rb to make sure at least one instance of cornbread is running


DETERMINE OUR CHECK COMMAND

On your SENSU CLIENT:

/opt/sensu/embedded/bin/ruby /etc/sensu/plugins/check-procs.rb -p cornbread -W1


INSTALL OUR CHECK

✓On your SENSU SERVER:

✓vim /etc/sensu/conf.d/checks/cornbread_process.json


CORNBREAD_PROCESS.JSON


RESTART SERVICES

service sensu-server restart

tail -100 /var/log/sensu/sensu-server.log | grep cornbread


CHECK YOUR DASHBOARD


CHECK YOUR EMAIL


SENSU API

✓REST API

✓HTTP/4567

✓on SENSU SERVER try:

curl -l http://localhost:4567/events \ | python -mjson.tool


http://localhost:4567/events

http://localhost:4567/events

SENSU SERVICES

✓Sensu API

✓Sensu Server

✓Sensu Client

✓Sensu Dashboard


EVERYTHING OK?

✓/etc/init.d/sensu-service {client|server|api|dashboard} {start|stop|status|restart}

✓ps -ef | grep sensu

✓tail -f /var/log/sensu/*.log

✓curl -l localhost:4567/info


COOL SENSU TRICKS


SEND DIRECTLY TO SENSUnetcat to: 127.0.0.0:3030


AGGREGATE ALERTS

✓Handy for preventing alert floods

✓Alert when X% of checks are are not OK


MY SENSU TIPS

✓install the RabbitMQ management web interface and bookmark it (see http://10.254.254.10:15672/#/ )

✓lock your plugins’ gem dependency versions


http://localhost:15672/#/

http://localhost:15672/#/

TIPS TIPS TIPS

✓have alternate ways to access your Dashboard information

✓we integrated our command-line developer tools with Sensu API

✓we also created our own Ops dashboard that queries Sensu, Graphite and our app for data


MORE TIPS

✓Put NGINX in front of sensu-dashboard


HA SENSU

✓Redundancy is easy (bring up more sensu-servers)

✓Making Redis and RabbitMQ HA more challenging

✓We’re still running one solitary Redis and RabbitMQ but are OK with this risk for now


WHERE TO GO FOR HELP

✓http://docs.sensuapp.org

✓IRC: #sensu - freenode

✓sensu-users mailing list


QUESTIONS


THANK YOU

[email protected]@skymob - twitter

robotwitharose - #sensu on IRC (freenode)




sensu at nycdevops meetup

Technology

sensu service

instances of sensu

sensu conguration

new sensu

sensu serversthursday

sensu architecturethursday

sensu source

sensu components