cost savings at high performance with redis labs and aws

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Cost Savings at High Performance

with Redis Labs and AWS

Frank Ober, Data Center Solutions Architect, Intel

Yiftach Shoolman, CTO, Redis Labs

August 11, 2016

2

Open source in-memory database platform

for high performance Operational/Analytics/Hybrid use cases

3

Redis Tops Database Popularity Rankings

……..#1 NoSQL in User Satisfaction and Market Presence

……..#1 NoSQL among Top 10 Data Stores

……..#1 database on Docker

#1 NoSQL database deployed in containers

………#1 in growth among top 3 NoSQL databases

………#1 database in skill demand

………# 1 database in Top Paying Technologies

4

Redis Labs – Home of Redis

• Founded in 2011

• HQ in Mountain View CA, R&D center in Tel-Aviv IL.

• The commercial company behind OSS Redis

• Provides enhanced Enterprise-Class Redis:

As-a-service – Redis Cloud

On-premises – Redis Labs Enterprise Cluster (RLEC)

5

Much More than a Key/Value store

High PerformanceKey/Value Store

Data-Structure Engine

ListsSorted Sets

HashesHyperlog-

logs

Geospatial Indexes

Bitmaps

SetsStrings

Bit field

6

Data Structure Engine

ListsSorted Sets

Hashes Hyperlog-logs

Geospatial Indexes

Bitmaps

SetsStrings

Bit field

• Replication, Data-Persistence, Auto-failover, Scaling

• ACID transactions, Pub/Sub, Eviction, Expiration, Lua, 150+ commands

• 9 data-structures:

8

Analytics & BI

Data Sink

Spark Architecture

Processing

Data Source

RDD, Data Source,Data Set

RDD, Data Source,Data Set

Spark SQL &Data Frame

9

Spark is Considered Fast But…

Read to RDD Deserialization Processing Serialization Write to RDD

Analytics & BI

1 2 3 4 5 6

Data SinkData Source

10

When Spark Meets Redis

Data Source Serving Layer

Analytics & BI

1 2

Read filtered/sorted

data

Writefiltered/sorted

data

Processing

11

Spark SQL &Data Frame

When Spark Meets Redis

Data Source Serving Layer

Analytics & BI

1 2

Processing

Spark-Redis connector

Read filtered/sorted

data

Writefiltered/sorted

data

12

Accelerating Spark Time-Series with Redis

Redis is faster by up to 100 times compared to HDFS and over 45 times compared to Tachyon or Spark

13

Support Any Use Case with Redis Modules

High PerformanceKey/Value Store

Data-Structure Engine

ListsSorted Sets

HashesHyperlog-

logs

Geospatial Indexes

Bitmaps

SetsStrings

Bit field

In-memory database Platform

14

Redis Modules

• Any C/C++ (soon Go and Python) program can now

run on Redis

• Use existing or add new data-structures

• Enjoy simplicity, high performance, infinite

scalability and high availability

• Can be created by anyone

15

Available Modules

redisearch redabloom topk countminsketch

GraphSecondary Indexes

Coming soon:

Available OSS:

JSON

+ >20 more

Machine Learning

1616

3.152.40

21.00

8.70

24.57

10.61

0.00

5.00

10.00

15.00

20.00

25.00

30.00

Full text search Prefix search

Average Latency (msec)

RLEC Elasticsearch Solr

20,045

6,831

690

3,686

621

3,133

0

5,000

10,000

15,000

20,000

25,000

Full text search Prefix search

Ops/sec

RLEC Elasticsearch Solr

85% higher

32x higher

7.8x faster 4.1x faster

How Fast Can a Module Run ?

Redisearch - the world’s fastest text search engine

17

• A marketplace for OSS and closed source modules

• All modules are certified by Redis Labs for these platforms:

Module Hub

1M+ instances 350K+ instances 100+ F500 Companies

www.redismodules.com

http://www.redismodules.com/

18

Is Redis Expensive ?

19

Sometimes Redis Can Be Very Expensive

#1200GB

#2200GB

#50200GB

50 x r3.8xlarge instances

#51200GB

#51200GB

#100200GB

1st replica for HA

#101200GB

#102200GB

#150200GB

2nd replica for quorum

HA deployment of 10TB Redis dataset on AWS

Total cost (reserved instances) = $2,132,250/yr

20

Do we really need 2 replicas?

#1200GB

#2200GB

#50200GB

50 x r3.8xlarge instances

#51200GB

#51200GB

#100200GB

1 replica for HA

#10115GB

Efficient HA deployment of 10TB in-memory dataset on AWS

Total cost (reserved instances) = $1,421,500/yr Savings = $710,750/yr

Redis Labs Patent Pending Technology

1 quorum server

21

..

Can We Save More?

22

Price/Performance of Memory Technology

0.10

1.00

90.00

250.00

0.10

1.00

25.00

500.00

0.10

1.00

10.00

100.00

1,000.00

$0

$1

$2

$3

$4

$5

$6

$7

$8

$9

$10

DRAM NV-DIMM/PM NVMe SSD SATA SSD

mic

rose

c

$/G

B

1 GB cost vs. Read/Write Performance

1 GB cost Read latency Write latency

$9

$0.4$1

$2

23

Redis on Flash (RoF)

• Flash used as a RAM extender (not as a persistent storage)

‒ In RAM: dict & keys & hot values

‒ On Flash: cold values

‒ Values are either in RAM or on Flash

• Multi-shard; multi-threaded; asynchronous

• Pluggable storage engine (RocksDB and IBM’s ark)

• 100% compatible with the OSS Redis

• Available on Redis Cloud Private (AWS VPC)

• Created by the team who develops and maintains the OSS Redis

24

Flexible RAM/Flash Ratios

25

Behind the Scenes

26

Increase RAM ratio Scale-out Cluster

27

10TB Redis Deployment on AWS

Redis (on RAM) 2 replicas

Redis (on RAM) 1 replica

Redis on Flash1 replica

Instance type r3.8xlarge r3.8xlarge i2.8xlarge

# of instances 150 100 10

RAM 30TB 20TB 2TB

Flash - - 64TB

Persistent storage (EBS) 150TB 100TB 80TB

1yr costs (reserved instances)

$2,132,250 $1,421,500 $345,310

Yearly savings - $710,750 $1,786,940

Savings % - 33.33% 83.81%

28

..

How Fast RoF Can Run ?

29

Factor #1 – RAM:Flash Hit Ratio

95% / 85% / 50% 5% / 15% / 50%Higher performance Lower performance

30

Factor #2 – Storage Speed (how many IOPS)Single Instance Max Ops/sec @ < msec latency

RAM/Flash hit ratio – 50%/50%, Read/Write ratio – 50%/50%, Object size – 100B

21K

59K

126K

175K

0K

20K

40K

60K

80K

100K

120K

140K

160K

180K

200K

i2.xlargeR/W IOPS - 35K/35K

i2.2xlargeR/W IOPS - 75K/75K



31

..

Thank you!Q&A

32

Remember to complete your evaluations!

Thank you!