sensu monitoring

SensuMohanasundaram Ponnusamy

Content• What is Sensu?• Why Sensu?• Why Competitive?• Architecture• Components• Important Keywords• Monitoring Flow• Sensu Server, Sensu Client, Redis/Redis Sentinel, API, Dashboard• Client Info, Checks, Handlers, Slashes, Muators, etc• Reference

What is Sensu?• Client-Server Architecture Monitoring Solution• Monitoring coverage leveraging to any Servers,

Containers, Services, Applications, Network or any Remote resources• Written in Ruby language• Uses simple JSON for Configuration

Why Sensu?• Design for Cloud• Open source well-defined (github.com/sensu)• Message Oriented Architecture• Extensible Plugins, handlers, Check Scripts can written in any language• Sensu Community Support• Single Point of Check metrics collection• Designed to Chef Cookbook, Puppet or Ansible• Backend support to Graphite, Logstash, Elasticsearch, etc.• Powerful Monitoring through Service Checks and Event Processing• Scalable and Extensible with community plugins• Sensu Framework includes Sensu Server, Redis, Redis Sentinel, RabbitMQ, Sensu Client, Sensu

API, Uchiwa Dashboard and etc.

Why Competitive

• Sensu vs Data Dog• https://stackshare.io/stackups/datadog-vs-sensu

• Sensu vs Nagios• https://stackshare.io/stackups/nagios-vs-sensu-vs-kibana

Architecture

Components

• Sensu Client – To Gather all check results• RabbitMQ – Default Transport MQ• Redis – Default Persistent Data Store• Redis Sentinel – Managing HA of Redis• Sensu Server – To manage actions on check results• Sensu API – To query the state• Uchiwa - Dashboard

Monitoring Flow

• Sensu Client • executes checks• publishes checks results to abbitMQ

• Sensu Server receives check results from RabbitMQ• Processes check results if required• Stores into persistent store - Redis

Monitoring Flow

Monitoring Flow

Monitoring Flow

Monitoring Flow

Important Keywords• Checks

• Commands / Script that collect metrics, events, etc. called check results.• Results are disk usage, swap, process, cpu, memory, file systems, etc.

• Event Handlers• Publish event data to destinations. • Example, Email, TCP, UDP, etc.

• Set• Group of Handlers • To pass one event data to multiple handlers

• Severity Filters• To run handlers at certain security level

• Mutators• Mutate event data before passing it to handlers

• Plugins, Extensions, Stashes, etc.

Sensu Client

• Monitoring Agent• Runs on the machine which to be monitored • Executes all checks (command/scripts) and collects all

check results• Sends following to Sensu Server through RabbitMQ

• Client info• Keep alive messages• Check Results• Subscriptions

Sensu Server• Monitoring Router• Sends execution request to Client through RabbitMQ• Uses Redis to hold persistent data• Uses RabbitMQ for heavy in exchange of data with Sensu

Client• Provides• Event Processing; Metric Routing; Event Management

• Two Flavor as below• Sensu Core• Sensu Enterprise

Rabbit MQ

• The default Sensu Transport Message Bus for Sensu• It can be single instance or a clustered• Uses for Sensu server and client to

communicate and to exchange the services, check results, etc.

Redis

• A default persistent data store for Sensu• Key-value data store with advanced cache• Stores • registry, • check results, • check history • and event data

Redis Sentinel• Service for managing Redis servers• Clustering • High Availability of Redis nodes such as slave to

master if the current master is not working as expected• Recommended is TWO systems for Redis Master-

Slave replica and a minimum of THREE Redis Sentinels.

Sensu API

• REST API for accessing various data on Sensu server (Redis)• It usually runs on the same server as

Sensu Server or Redis instance• JSON document results

Sensu-Dashboard

• Built on top of Uchiwa • Real-time Web UI based on Node.js• Dedicated view for each events, checks,

datacenter, etc• Basic operations can be done

Client Info

• Client info is published when monitoring service is started• The monitoring content is based on • Name - Client name• Address - IP address of the client• Subscriptions - Roles list or Role

equivalent

{ "Client": { "Name": “myHost", "Address": "127.0.0.1", "Subscriptions": [ "Production", “Openshift", “Docker" ] } }

What is Checks• Essential commands/scripts• Executed by Sensu Client on the machine to be monitored• Subscription Checks - the execution timing if held by Sensu Server• Standalone Checks – the execution timing if held by Sensu Client• Check Results

• JSON Document • Sends as Event to Handler

• Output to STDOUT / ERR and have an exit status code as follows• 0 : OK• 1 : WARNING• 2 : CRITICAL• 3 : UNKNOWN or CUSTOM

Check Definition• myClient• Unique Check name

• Command• Check script to execute

• Subscribers• Which subscriptions to execute Check

• Interval• interval Check script is executed

(seconds)

{ "Checks": { “myClient": { "Command": “mycheck-rule.rb", "Subscribers": [ "Production" ], "Interval": 60, "Handlers": [ “TCP", "Irc" ] } } }

Event Handlers• Action executed by the Sensu server on events• Each checks have its own handler(s)• Types

• Pipe Handler• Event Data through STDIN and executes commands/scripts• Server executes a script and results event as JSON to Standard Input.

• TCP/UDP handlers• Event data to TCP/UDP socket• Server connects to the host:port and sending event

• Transport handlers publish event data to RabbitMQ• It can be routing through

• Example:• Sending an email alert, creating or resolving an incident (e.g. in PagerDuty,

ServiceNow, etc.), or storing metrics in a time-series database (e.g. Graphite

Event Handler - Example

• Execute a command

• send mail

{ "Handlers": { "File": { "Type": "pipe", "Command": "/etc/sensu/handlers/file.rb" } } }

{ "Handlers": { "Mail": { "Type": "pipe", "Command": "mail - s 'sensu event' email@myaddress.com" } } }

{ "Handlers": { "Tcp_socket": { "Type": "tcp", "Socket": { "Host": "127.0.0.1", "Port": 4242 } } } }

Stashes

• Associated with path• Key Value Store• Store anything• Make your handlers to check

Reference

• Sensu Document• https://sensuapp.org/features

• Sensu Plugins• https://github.com/sensu

Q & A