redis / redis labs overview and then busting 4 myths … · redis / redis labs overview and then...

Redis / Redis Labs Overviewand then busting 4 myths about Redis

Yiftach Shoolman Co-Founder & CTO

Home of Redis

2

Thank You AWS

3

Birthdays

5 years old

7 years old

+ 80 years old

4

Who We Are

The open source home and commercial provider of Redis

Open source, in-memory , data structure storeused as a NoSQL database, a caching layer or a message broker

5

Redis Tops Database Popularity Rankings

……..#1 NoSQL in User Satisfaction and Market Presence

……..#1 NoSQL among Top 10 Data Stores

……..#1 database on Docker

#1 NoSQL database deployed in containers

………#1 in growth among top 3 NoSQL databases

………#1 database in skill demand

6

World’s fastest & most powerful database:1.5M ops/sec @ <1msec with a single cloud instance

HIGHPERFORMANCE

A new database concept based on Data Structures & Lua scripting

Data structures used by developers like “Lego” building blocks

GAME CHANGER

Redis – the Cloud Native Database

Largest open source community among the NoSQL databases. Compatible with most programming languages and environments.

DYNAMICCOMMUNITY

Lists

Sorted Sets

Hashes

Hyperlog-logs

Geospatial Indexes

Bitmaps

Sets

Strings

7

Redis Labs – Home of Redis

• Founded in 2011

• HQ in Mountain View CA, R&D centers in Tel-Aviv, North-IL, Sicily, Beijing

• The commercial company behind OSS Redis

• Provides enhanced Enterprise-Class Redis:

As-a-service – Redis Cloud

On-premises – Redis Labs Enterprise Cluster (RLEC)

• 150,000+ databases under management; 200 new databases added every day

8

Tremendous Customer TractionRedis Cloud

Available since mid-2013

5,800+ paying customers40,000+ free

Redis Labs Enterprise Cluster (RLEC)Available since early-2015

100+ paying customers

9

Redis Cloud Service Offered Over IaaS and PaaS

4 Clouds, 45 data centers across the world

10 partner channels

10

On-Prem Service

Available: Mid-2015

Annual subscription

We provide:

Supports:

Enterprise Software (Cloud/ On-Prem)

Available: Early-2015

Annual subscription

We provide:

Supports:

Private Cloud Service

Available: Mid-2015

PAYG

We provide:

Supports:

Product Offering

DB-as-a-Service DB Software

Public Cloud Service

Available: Mid-2013

Monthly subscription, PAYG

We provide:

Supports:

Redis Cloud Redis Cloud Private Redis Labs Enterprise Cluster Managed-RLEC

11

TelcoBusiness Services

TechnologyRetail/E-

Commerce

Customers From All Verticals

MediaFinance

Travel Education GamingSocialAdvertising

12

Redis Encompasses Many Use Cases

Geo SearchData Ingestion Social Functionality

Following, Followers, Relations Location-based ApplicationsHigh Throughput Buffering

Job & Queue Caching

Any Business Application Any Web or Mobile App

High Speed Transactions Time-Series

Business Applications

Analytics

Real-time Computations Time-Based Analysis

13

Caching



Job & Queue Caching




Analytics


50%

14

Operational Database



Job & Queue Caching




Analytics


20%

15

Analytics



Job & Queue Caching




Analytics


20%

16

Job & Queue



Job & Queue Caching




Analytics


10% ≈

17

Myths #1All in-memory databases are equally fast

1818

1,000,000

0 200,000 400,000 600,000 800,000 1,000,000 1,200,000

1xc3.8xlarge EC2 instance

(1) Deployed in a VPC(2) Enhanced Network Interface

(ENI) enabled(3) Placement group enabled(4) IRQ Distribution enabled(5) Interrupt coalescing enabled(6) 4xNICs configured(7) Receive Packet Steering (RPS)

enabled

How many ops/sec can a single EC2 instance do?

1919

1,228,432

1,000,000

0 200,000 400,000 600,000 800,000 1,000,000 1,200,000 1,400,000

1xc3.8xlarge EC2 instanceFully optimized(1) Deployed in a VPC(2) ENI enabled(3) Placement group enabled(4) IRQ Distribution enabled(5) Interrupt coalescing enabled(6) 4xNICs configured(7) RPS enabled

Not optimized(1) Deployed on Classic EC2(2) No tuning (3) Reached packets/sec limit


2020

1,750,000

1,228,432

1,000,000

0 200,000 400,000 600,000 800,000 1,000,000 1,200,000 1,400,000 1,600,000 1,800,000 2,000,000

1



Half optimized(1) Deployed in a VPC(2) ENI enabled(3) RPS enabled(4) Only 1xNIC

Fully optimized(1) Deployed in a VPC(2) ENI enabled(3) Placement group enabled(4) IRQ Distribution enabled(5) Interrupt coalescing enabled(6) 4xNICs configured(7) RPS enabled


2121

2,000,000

1,750,000

1,228,432

1,000,000

0 500,000 1,000,000 1,500,000 2,000,000 2,500,000



Half optimized(1) Deployed in a VPC(2) ENI enabled(3) RPS enabled(4) Only 1xNIC

Fully optimized(1) Deployed in a VPC(2) ENI enabled(3) Placement group enabled(4) IRQ Distribution enabled(5) Interrupt coalescing enabled(6) 4xNICs configured(7) RPS enabled

Fully optimized


22

How many servers to get 1M writes/sec on GCP?

23


24In-memory databases


25

Real-World Write Intensive App

394.42381.31

372.31

161.94

71.22

0

50

100

150

200

250

300

350

400

450

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

Application Requests/Sec Application Latency (msec)

Ap

plic

atio

n R

equ

ests

Per

Sec

on

d

Late

ncy

in M

illis

eco

nd

s

NoSQL Performance Benchmark

Benchmarks performed by Avalon Consulting Group

26

Hadoop Hbase+Internal cache vs. Hbase+Redis

27

Spark with Redis

x100

x40

28

Same Redis core, same HW, different performance

Benchmarks performed by Altoros

2929

Most are written in C/C++…. but programming language isn’t the only thing to consider

So why aren’t in-memory DBs equally fast?

30

(1) Complexity of processing commands

How many lines of code per command ? What is the computation

complexity (e.g. in Redis most commands are O(1))?

(2) Query efficiency

Is it limited to blob queries? Can you query a discrete value?

(3) Pipelining

Can you send multiple requests at once to get lower latency and less

context switches?

What affects in-memory DB performance?

31

(4) Protocol efficiency

How long it takes to parse a request or to serialize a response

(5) TCP overhead

Long-lived (connection pool) vs. short-lived connections


32

(6) Single-threaded or multi-threaded architecture

Lock-free vs. parallel computing

(7) Shared-nothing (the best) vs. shared-something vs. shared-everything

(8) Built-in acceleration components


33

Myths #2Redis is not scalable

34

Clustering support

𝘝 Redis Cloud/RLEC – mid-2013 2000+ clusters under management

𝘝 OSS Cluster - April 1, 2015Strong momentum

35

When should you switch to cluster

More cores

Trigger: 50K+ ops/sec

More RAM

Trigger: 50GB

36

Myths #3Redis is not reliable

37

HA Redis

𝘝 Redis Cloud/RLEC mid-2012 100K+ DBs

𝘝 Sentinel 01/2013Many HA deployments

38

Mapping Features to Outage Type

39

Failover Benchmark

Vendors are (not in the same order): Azure Cache, AWS ElastiCache, Heroku Redis, Compose.io (now IBM)

40

We Survived

30+1000+NODE FAILURES COMPLETE DATA-CENTER

OUTAGES

41

Myths #4Redis is Expensive

42

use case:

• 500+ GB

• 400K writes/sec

• 1500 reads/sec

• 37.5KB average object size

Which Cost More?

No extra work at

app level

1.5Gbps 120Gbps

Tons of work at app level

Others

6-node cluster

30+ node cluster

43

Highly Dense Multi-Tenant Architecture

200 Redis process on a single 4vcore cloud instance

44

Sometimes Redis Can Be Very Expensive

#1200GB

#2200GB

#50200GB

50 x r3.8xlarge instances

#51200GB

#51200GB

#100200GB

1st replica for HA

#101200GB

#102200GB

#150200GB

2nd replica for quorum

HA deployment of 10TB Redis dataset on EC2

Total cost (reserved instances) = $2,132,250/yr

45

Do we really need 2 replicas?

#1200GB

#2200GB

#50200GB

50 x r3.8xlarge instances

#51200GB

#51200GB

#100200GB

1 replica for HA

#10115GB

Efficient HA deployment of 10TB in-memory dataset on EC2

Total cost (reserved instances) = $1,421,500/yr Savings = $710,750/yr

Redis Labs Patent Pending Technology

1 quorum server

46

..

Can We Save More?

47

Redis on Flash Flash used as a RAM extender and NOT as persistent storage

48

How to Achieve Optimal Price/Performance

By dynamically setting RAM/Flash ratio

49

10TB Redis Deployment on EC2

Redis (on RAM) 2 replicas

Redis (on RAM) 1 replica

Redis on Flash1 replica

Instance type r3.8xlarge r3.8xlarge i2.8xlarge

# of instances 150 100 10

RAM 30TB 20TB 2TB

Flash - - 64TB

Persistent storage (EBS) 150TB 100TB 80TB

1yr costs (reserved instances)

$2,132,250 $1,421,500 $345,310

Yearly savings - $710,750 $1,786,940

Savings % - 33.33% 83.81%

50

Always On - Highly Available & Persistent

Simple, SeamlessScaling and Clustering

Enterprise-Class Management and

Support

Enterprise-Class Redis

Stable & Predictable Top Performance

Operational Cost Savings

Home of Redis

Thank You!

redis / redis labs overview and then busting 4 myths … · redis / redis labs overview and then...

Documents