Bill Anderson, Redis Mad Scientist

Beyond Simple Failover

Using Sentinel for more than just managing a master/slave pair.

Who Am I?

❖ Rackspace’s Resident Redis Mad Scientist!❖ Redis since 1.x !❖ github.com/therealbill/libredis and others!❖ Currently focused heavily on large scale deployments

Topics

❖ Overview / Review!❖ Alternate Uses of Sentinel!❖ Dynamic Sentinel!❖ Client Requirements!❖ Operational Perspectives!❖ Red Skull

Overview / Review

❖ Sentinel: A Redis option for managing which node in a Redis pod is master and which are slaves.!

❖ Can call scripts on events to trigger external events!❖ Publishes events to a PUBSUB channel for better system

integration.!❖ Uses a Quorum based election and promotion system

"So I have a Redis cluster..."

❖ Redis Pod!❖ One Master with at least 1 slave, no limit to slave count

or configuration!❖ Sentinel Constellation!

❖ Any number of Sentinel instances which manage a Pod!❖ Sentinels can overlap

Redis Client Communication

Alternate Uses of Sentinel

❖ Sentinel as a monitoring tool!❖ Sentinel as a Service discovery mechanism!❖ Coming: info aggregator

Sentinel for Monitoring

❖ Replication Lag!❖ Track and Profile SDOWN events!

❖ Network insight !❖ System insight!❖ Redis delays - such as those caused by BGSAVE or

SAVE

Sentinel for Monitoring

❖ Configuration Changes!❖ Epoch changes!❖ Slave changes!❖ Sentinel changes

Dynamic Sentinel❖ Sentinel API!

❖ Add, remove, reconfigure a pod!❖ Get current master!❖ Get current slaves!❖ Get other sentinels*!

❖ Useful for integration w/config management systems and removing Redis information from client code/configuration

Client Requirements

❖ API Usage: Any Redis Client or client/library which supports the sentinel command or allows a ”raw” command to be sent.!

❖ Client Usage: !❖ Option 1: Client which allows SENTINEL command,

implement connection management & control yourself!❖ Option 2: Client library which has “native” support for

using Sentinel.

Sentinel Event Handling

❖ Pubsub!❖ Each event has a dedicated channel!❖ masterchange!❖ +SDOWN !❖ -SDOWN!❖ Use to react or record events in your constellation and

pods.

Auditing Configuration

❖ Do you have enough sentinels to form a quorum?!❖ Do you have too many?!❖ Do all your masters have slaves?!❖ Are they promotable?!❖ Do the slaves have enough memory to handle the

master?

Operational Perspectives❖ Using Sentinel for discovery means more flexible

configuration for clients and client code!❖ Using Sentinel over config API improves integration with

system management and avoids file based management mechanisms (Chef, Puppet, etc.)!

❖ Visitor discovery of Redis through API provides snapshot information!

❖ Comparing discovered configuration to mandatory allows for automated audits

The Fine Print❖ When using failover quorum should be an odd number representing half+1

of your constellation.!❖ number_other_sentinels is not “live” - it is a high water mark. If you use it,

validate it.!❖ Same with sentinels in the “sentinel sentinels <master>” command. If you

use it, validate it.!❖ Sentinels must be able to talk to each other, the masters, and all of the

slaves.!❖ Sentinel is unauthenticated!❖ MONITOR/SENTINEL REMOVE must be done on all members of the

constellation

Sounds great. But ...

❖ Constellations, pods, nodes, sentinels, Oh my!!❖ Most changes have to be repeated across the

constellation!❖ I don't want to deal with all that fine print. !❖ I don't blame you. Neither do I. !❖ But I did!❖ And you don't have to

Red Skull Sentinel Management Tool

What does it do?

❖ Manages a cluster of sentinels!❖ Provides an API to handle constellation-wide actions in a

single call!❖ Audits key aspects of a constellation config !❖ Can correct some of the issues found in auditing!❖ Provides a window into your cluster via a web based

interface

Dashboard view

Constellation View

Nodes Listing

This Pod Has Problems

Pods With Error Listing

Deployment

❖ Runs on each sentinel you operate!❖ Bootstraps itself from the config file for that sentinel!❖ One binary plus an HTML directory!❖ Easy docker container

Future Features

❖ Consul/Serf based clustering and shared store!❖ Sentinel API!❖ TaskMaster !

❖ Provisioning!❖ Clone!❖ Migrations