sensu monitoring
TRANSCRIPT
SensuMohanasundaram Ponnusamy
Content• What is Sensu?• Why Sensu?• Why Competitive?• Architecture• Components• Important Keywords• Monitoring Flow• Sensu Server, Sensu Client, Redis/Redis Sentinel, API, Dashboard• Client Info, Checks, Handlers, Slashes, Muators, etc• Reference
What is Sensu?• Client-Server Architecture Monitoring Solution• Monitoring coverage leveraging to any Servers,
Containers, Services, Applications, Network or any Remote resources• Written in Ruby language• Uses simple JSON for Configuration
Why Sensu?• Design for Cloud• Open source well-defined (github.com/sensu)• Message Oriented Architecture• Extensible Plugins, handlers, Check Scripts can written in any language• Sensu Community Support• Single Point of Check metrics collection• Designed to Chef Cookbook, Puppet or Ansible• Backend support to Graphite, Logstash, Elasticsearch, etc.• Powerful Monitoring through Service Checks and Event Processing• Scalable and Extensible with community plugins• Sensu Framework includes Sensu Server, Redis, Redis Sentinel, RabbitMQ, Sensu Client, Sensu
API, Uchiwa Dashboard and etc.
Why Competitive
• Sensu vs Data Dog• https://stackshare.io/stackups/datadog-vs-sensu
• Sensu vs Nagios• https://stackshare.io/stackups/nagios-vs-sensu-vs-kibana
Architecture
Components
• Sensu Client – To Gather all check results• RabbitMQ – Default Transport MQ• Redis – Default Persistent Data Store• Redis Sentinel – Managing HA of Redis• Sensu Server – To manage actions on check results• Sensu API – To query the state• Uchiwa - Dashboard
Monitoring Flow
• Sensu Client • executes checks• publishes checks results to abbitMQ
• Sensu Server receives check results from RabbitMQ• Processes check results if required• Stores into persistent store - Redis
Monitoring Flow
Monitoring Flow
Monitoring Flow
Monitoring Flow
Important Keywords• Checks
• Commands / Script that collect metrics, events, etc. called check results.• Results are disk usage, swap, process, cpu, memory, file systems, etc.
• Event Handlers• Publish event data to destinations. • Example, Email, TCP, UDP, etc.
• Set• Group of Handlers • To pass one event data to multiple handlers
• Severity Filters• To run handlers at certain security level
• Mutators• Mutate event data before passing it to handlers
• Plugins, Extensions, Stashes, etc.
Sensu Client
• Monitoring Agent• Runs on the machine which to be monitored • Executes all checks (command/scripts) and collects all
check results• Sends following to Sensu Server through RabbitMQ
• Client info• Keep alive messages• Check Results• Subscriptions
Sensu Server• Monitoring Router• Sends execution request to Client through RabbitMQ• Uses Redis to hold persistent data• Uses RabbitMQ for heavy in exchange of data with Sensu
Client• Provides• Event Processing; Metric Routing; Event Management
• Two Flavor as below• Sensu Core• Sensu Enterprise
Rabbit MQ
• The default Sensu Transport Message Bus for Sensu• It can be single instance or a clustered• Uses for Sensu server and client to
communicate and to exchange the services, check results, etc.
Redis
• A default persistent data store for Sensu• Key-value data store with advanced cache• Stores • registry, • check results, • check history • and event data
Redis Sentinel• Service for managing Redis servers• Clustering • High Availability of Redis nodes such as slave to
master if the current master is not working as expected• Recommended is TWO systems for Redis Master-
Slave replica and a minimum of THREE Redis Sentinels.
Sensu API
• REST API for accessing various data on Sensu server (Redis)• It usually runs on the same server as
Sensu Server or Redis instance• JSON document results
Sensu-Dashboard
• Built on top of Uchiwa • Real-time Web UI based on Node.js• Dedicated view for each events, checks,
datacenter, etc• Basic operations can be done
Client Info
• Client info is published when monitoring service is started• The monitoring content is based on • Name - Client name• Address - IP address of the client• Subscriptions - Roles list or Role
equivalent
{ "Client": { "Name": “myHost", "Address": "127.0.0.1", "Subscriptions": [ "Production", “Openshift", “Docker" ] } }
What is Checks• Essential commands/scripts• Executed by Sensu Client on the machine to be monitored• Subscription Checks - the execution timing if held by Sensu Server• Standalone Checks – the execution timing if held by Sensu Client• Check Results
• JSON Document • Sends as Event to Handler
• Output to STDOUT / ERR and have an exit status code as follows• 0 : OK• 1 : WARNING• 2 : CRITICAL• 3 : UNKNOWN or CUSTOM
Check Definition• myClient• Unique Check name
• Command• Check script to execute
• Subscribers• Which subscriptions to execute Check
• Interval• interval Check script is executed
(seconds)
{ "Checks": { “myClient": { "Command": “mycheck-rule.rb", "Subscribers": [ "Production" ], "Interval": 60, "Handlers": [ “TCP", "Irc" ] } } }
Event Handlers• Action executed by the Sensu server on events• Each checks have its own handler(s)• Types
• Pipe Handler• Event Data through STDIN and executes commands/scripts• Server executes a script and results event as JSON to Standard Input.
• TCP/UDP handlers• Event data to TCP/UDP socket• Server connects to the host:port and sending event
• Transport handlers publish event data to RabbitMQ• It can be routing through
• Example:• Sending an email alert, creating or resolving an incident (e.g. in PagerDuty,
ServiceNow, etc.), or storing metrics in a time-series database (e.g. Graphite
Event Handler - Example
• Execute a command
• send mail
{ "Handlers": { "File": { "Type": "pipe", "Command": "/etc/sensu/handlers/file.rb" } } }
{ "Handlers": { "Mail": { "Type": "pipe", "Command": "mail - s 'sensu event' [email protected]" } } }
{ "Handlers": { "Tcp_socket": { "Type": "tcp", "Socket": { "Host": "127.0.0.1", "Port": 4242 } } } }
Stashes
• Associated with path• Key Value Store• Store anything• Make your handlers to check
Reference
• Sensu Document• https://sensuapp.org/features
• Sensu Plugins• https://github.com/sensu
Q & A