05/03/2023

1

REDIS- THE CLIFFS NOTES VERSION

05/03/2023

2 WHAT IS REDIS?

REmote DIctionary Server

05/03/2023 3

•Written in: C•Main point: Blazing fast•License: BSD•Protocol: Telnet-like, binary safe•Disk-backed in-memory database,

4

 in-memory data structure store Redis is an advanced key-value store, where keys can contain

data structures such as strings, hashes, lists, sets, and sorted sets. Supporting a set of atomic operations on these data types.

Redis can persist data to the disk

Redis is not only a key-value store

Can be used as Database, a Caching layer or a Message broker.

Redis is fast

Official webpage: http://redis.ioSource Code: https://github.com/antirez/redisOnline Demo: http://try.redis.io

5

Redis is not a replacement for Relational Databases nor Document Stores.

It might be used complementary to a SQL relational store, and/or NoSQL document store.

Best Used: For rapidly changing data with a foreseeable database size (should fit mostly in memory).

What Redis is not...

05/03/2023 6

REDIS ARCHITECTURE Written in ANSI C by Salvatore Sanfilippo (@antirez). Works in most POSIX systems like Linux, BSD and OS X. Linux is the recommended No official support for Windows, but Microsoft develops and

maintains an open source Win-64 port of Redis* Redis is a single-threaded server, not designed to benefit

from multiple CPU cores. Several Redis instances can be launched to scale out on several cores.

All operations are atomic (no two commands can run at the same time).

It executes most commands in O(1) complexity and with minimal lines of code.

05/03/2023 7

8

CachingPub/subBlocking/Delayed Queueso Such as https://github.com/resque/resqueo http://techblog.netflix.com/2016/08/distributed-delay-queues-based-on.htmlShort lived items, with ttlo Fraud detectiono User Sessionso Request Filtering ServiceCounting Reviews most bought itemsReal time AnalysisStoring Unique items over time.

Use cases for Redis

05/03/2023 9

WHEN TO USE REDIS Speed is key More than just key-value pairs Redis is an in memory data store, so there are memory

limitations.o An empty instance uses ~ 1MB of memory.o 1 Million small Keys -> String Value pairs use ~ 100MB of

memory.o 1 Million Keys -> Hash value, representing an object with

5 fields, use ~ 200 MB of memory. Dataset is not critical

10

Redis Performance

https://redislabs.com/blog/the-proven-redis-performance#.WHKDFLYrJZ1

11

Redis Throughput...

https://web.archive.org/web/20160304051043/http://techblog.netflix.com/2016/01/dynomite-with-redis-on-aws-benchmarks_14.html

05/03/2023 12

Stat’s From Twitter(2014) Timeline Service for one datacenter using Hybrid List:

o ~40TB allocated heapo ~30MM qpso > 6,000 instances

Use of BTree in one datacenter:o ~65TB allocated heapo ~9MM qpso >4,000 instances

05/03/2023 13

REDIS DATA TYPESRedis Data Type Contains Read/write ability

StringBinary-safe strings (up to 512 MB), Integers or Floating point values, Bitmaps.

Operate on the whole string, parts, increment/decrement the integers and floats, get/set bits by position.

Hash Unordered hash table of keys to string values

Add, fetch, or remove individual ítems by key, fetch the whole hash.

List Doubly linked list of stringsPush or pop items from both ends, trim based on offsets, read individual or multiple items, find or remove items by value.

Set Unordered collection of unique strings

Add, fetch, or remove individual items, check membership, intersect, union, difference, fetch random items.

Sorted SetOrdered mapping of string members to floating-point scores, ordered by score

Add, fetch, or remove individual items, fetch items based on score ranges or member value.

Geospatial index

Sorted set implementation using geospatial information as the score

Add, fetch or remove individual items, search by coordinates and radius, calculate distance.

HyperLogLogProbabilistic data structure to count unique things using 12Kb of memory

Add individual or multiple items, get the cardinality.(http://antirez.com/news/75)

05/03/2023 14

SCALING REDIS Persistence

Redis provides two mechanisms to deal with persistence: Redis database snapshots (RDB) and append-only files (AOF).

Replication A Redis instance known as the master, ensures that one or more instances kwown as the slaves, become

exact copies of the master. Clients can connect to the master or to the slaves. Slaves are read only by default. Partitioning

Breaking up data and distributing it across different hosts in a cluster. Can be implemented in different layers:

o Client: Partitioning on client-side code.o Proxy: An extra layer that proxies all redis queries and performs partitioning (i.e.

Twemproxy,Dynomite).o Query Router: instances will make sure to forward the query to the right node. (i.e. Redis

Cluster).o Redis Cluster is a mix between query routing and client side partitioning.

Failover o Manualo Automatic with Redis Sentinel (for master-slave topology)o Automatic with Redis Cluster (for cluster topology)

15

Redis Persistence(Read more about it at https://redis.io/topics/persistence )

• RDBo  performs point-in-time snapshots of your dataset at specified intervals

• AOFo AOF persistence logs every write operation received by the serve

• If you wish, you can disable persistence at all, if you want your data to just exist as long as the server is running

Master not persisting, Slave Persisting to disk.

• It is possible to combine both AOF and RDB in the same instance.

16

RDB advantageso RDB is a very compact single-file point-in-time representation of your Redis

data.o  RDB is very good for disaster recovery, being a single compact file can be

transferred to far data centerso RDB maximizes Redis performances since the only work the Redis parent

process needs to do in order to persist is forking a child that will do all the rest. The parent instance will never perform disk I/O or alike

o RDB allows faster restarts with big datasets compared to AOF.

RDB disadvantages• RDB is NOT good if you need to minimize the chance of data loss in case Redis

stops workingo Fork() can be time consuming if the dataset is big, and may result in Redis to

stop serving clients for some millisecond or even for one second if the dataset is very big and the CPU performance not great

05/03/2023 17

o AOF advantages• Using AOF Redis is much more durable: you can have different fsync policies: no fsync at

all, fsync every second, fsync at every query.• The AOF log is an append only log, so there are no seeks, nor corruption problems if

there is a power outage• Redis is able to automatically rewrite the AOF in background when it gets too big• AOF contains a log of all the operations one after the other in an easy to understand and

parse formato AOF Disadvantages• AOF files are usually bigger than the equivalent RDB files for the same dataset• AOF can be slower than RDB depending on the exact fsync policy. In general with fsync

set to every second performances are still very high.

05/03/2023Hystrix 18

PARTITIONING Partitioning in Redis serves two main goals:

It allows for much larger databases, using the sum of the memory of many computers. Without partitioning you are limited to the amount of memory a single computer can support.

It allows scaling the computational power to multiple cores and multiple computers, and the network bandwidth to multiple computers and network adapters.

Client side partitioning means that the clients directly select the right node where to write or read a given key. Many Redis clients implement client side partitioning.e.g JedisSharding

Proxy assisted partitioning means that our clients send requests to a proxy that is able to speak the Redis protocol, instead of sending requests directly to the right Redis instance. The proxy will make sure to forward our request to the right Redis proxy Twemproxy implements proxy assisted partitioning.

Query routing means that you can send your query to a random instance, and the instance will make sure to forward your query to the right node

05/03/2023 19

05/03/2023 20

REDIS CLUSTER – KEY DISTRIBUTION MODEL The key space is split into 16384 slots, Effectively setting an upper limit for the cluster size of

16384 master nodes (however the suggested max size of nodes is in the order of ~ 1000 nodes).

HASH_SLOT = CRC16(key) mod 16384 When the cluster is stable, a single hash slot will be

served by a single node https://redis.io/topics/cluster-spec#keys-distribution-

model

05/03/2023 21

CLUSTER LIVE RECONFIGURATION Redis Cluster supports the ability to add and remove

nodes while the cluster is running. (https://redis.io/topics/cluster-spec)

Adding or removing a node is abstracted into the same operation: moving a hash slot from one node to another.o To add a new node to the cluster an empty node is added to

the cluster and some set of hash slots are moved from existing nodes to the new node.

o To remove a node from the cluster the hash slots assigned to that node are moved to other existing nodes.

o To rebalance the cluster a given set of hash slots are moved between nodes.

05/03/2023 22

Twemproxy (single)

Twemproxy* works as a proxy between the clients and many Redis instances. Is able to automatically distribute data among different standalone Redis instances. Supports consistent hashing with different strategies and hashing functions. Multi-key commands and transactions are not supported.

Twemproxy (load balanced)

*Twemproxy is a project from Twitter and is not part of redis: https://github.com/twitter/twemproxy**Illustrations from Redis Essentials book by Maxwell Dayvson Da Silva and Hugo Lopes Tavares

05/03/2023 23Dynomite

In the age of high scalability and big data, Dynomite’s design goal is to turn those single-server datastore solutions into peer-to-peer, linearly scalable, clustered systems while still preserving the native client/server protocols of the datastores, e.g., Redis protocol.

05/03/2023 24

FAILOVERo Manualo Automatic with Redis Sentinel (for master-slave topology)o Automatic with Redis Cluster (for cluster topology)

05/03/2023 25

05/03/2023 26

DEPLOYMENT ARCHITECTURE-REDIS CLUSTER(PREFERRED)

05/03/2023 27

OTHER DEPLOYMENT ARCHITECTURES—STANDALONE REDIS WITH FAILOVER

05/03/2023 28

05/03/2023 29

OTHER DEPLOYMENT ARCHITECTURES Dynomite Twemproxy

05/03/2023 30

WHAT CLIENT SHOULD I USE Spring Data Redis project supports Jedis and Lettuce, two

of the most popular java clients for Redis. Other Java Redis client of note is Redisson.

05/03/2023 31

REDIS PIPELINING A Request/Response server can be implemented so that

it is able to process new requests even if the client didn't already read the old responses. This way it is possible to send multiple commands to the server without waiting for the replies at all, and finally read the replies in a single step.

Lettuce client supports pipelining on redis cluster Jedis does not. Without pipelining 1.185238 seconds with pipelining

0.250783 seconds https://redis.io/topics/pipelining