redis in memory data processing - imcsummit.org · – redis stream events - trigger an execution...

57
RedisGears – Redis in memory data processing JUNE 2019 | PIETER CAILLIAU

Upload: others

Post on 17-Mar-2020

35 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

RedisGearsndashRedisinmemorydataprocessingJUNE 2019 | PIETER CAILLIAU

bull ProducedinBelgium

bull (instanceof)SETomTom

bull Consultantneo4j

bull SolutionArchitectRedisLabs

bull ProductManagerRedisLabs

bull cailliaup

Aboutme

2

1 WhatisRedisandRedisEnterprise

2 StreamProcessingwithRedisGears

3 RedisGearsasaMultimodelEngine

Agenda

4

RedisisFasthellip

5

hellipExtremelyFast

DB-EnginesRanking

6

7

Andyoursquovebeenusingitalready

RedisisExtensivelyandDiverselyUsed

8

UsesRedisforTimelinefollowing

Scope10-20TB

UsesRedisforLocalsiteglobalcaching

UsesRedisforRepositoryrouter

Scope10+TB

UsesRedisforGeosearchuserprofiles

Scope10-20TB

UsesRedisforAllmessages

Scope40TB

RedisTopDifferentiators

Simplicity ExtensibilityPerformanceNoSQLBenchmark

1

RedisDataStructures

2 3

RedisModules

9

Lists

Hashes

Bitmaps

Strings

Bitfield

Streams

Hyperloglog

SortedSets

Sets

GeospatialIndexes

10

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

11

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

bull Createyourowndatatypesandcommands

bull ReuseRedisrsquosimplicityperformancescalabilityandhighavailability

bull CanbewritteninCC++GoPythonRustZig

bull Leverageexistingdatastructures

bull TurnRedisintoaMulti-Modeldatabase

ModulesExtendRedisInfinitely

12

httpsredislabscomcommunityredis-modules-hub

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

13

14

Introducing

RedisEnterprise

15

DBaaSbull Availablesincemid2013

bull 8500+enterprisecustomers

Softwarebull Availablesinceearly2015

bull 300+enterprisecustomers550K+databases

managedworldwide

bull 6oftopFortune10companies

bull 3oftop5communicationscompanies

Customersbull 3oftop4creditcardissuers

bull 3oftop5healthcarecompanies

bull CloudProvider

bull Highermarginbyndash Idleness

bull CloudLock-in

CloudProvidershavedifferentincentives

16

bull DBAASProvider

bull Highermarginbybetterresourceutilizationndash Multi-tenancyndash ReducingRAMndash CPUutilization

RedisEnterpriseAUniquePrimaryDatabase

HIGHESTPERFORMANCELINEARSCALING

HIGHAVAILABILITYWITHINSTANTFAILOVER

DURABILITYATMEMORYSPEEDS

ACTIVE-ACTIVEGEODISTRIBUTION(CRDT-BASED)

BUILT-INHIGHPERFORMANCE SEARCH

MULTI-MODEL

FLEXIBLEDEPLOYMENTOPTIONS(CLOUDON-PREMHYBRID)

INTELLIGENTTIEREDDATAACCESS(RAMampFLASHMEMORY)

FAST RELIABLE FLEXIBLE

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 2: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

bull ProducedinBelgium

bull (instanceof)SETomTom

bull Consultantneo4j

bull SolutionArchitectRedisLabs

bull ProductManagerRedisLabs

bull cailliaup

Aboutme

2

1 WhatisRedisandRedisEnterprise

2 StreamProcessingwithRedisGears

3 RedisGearsasaMultimodelEngine

Agenda

4

RedisisFasthellip

5

hellipExtremelyFast

DB-EnginesRanking

6

7

Andyoursquovebeenusingitalready

RedisisExtensivelyandDiverselyUsed

8

UsesRedisforTimelinefollowing

Scope10-20TB

UsesRedisforLocalsiteglobalcaching

UsesRedisforRepositoryrouter

Scope10+TB

UsesRedisforGeosearchuserprofiles

Scope10-20TB

UsesRedisforAllmessages

Scope40TB

RedisTopDifferentiators

Simplicity ExtensibilityPerformanceNoSQLBenchmark

1

RedisDataStructures

2 3

RedisModules

9

Lists

Hashes

Bitmaps

Strings

Bitfield

Streams

Hyperloglog

SortedSets

Sets

GeospatialIndexes

10

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

11

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

bull Createyourowndatatypesandcommands

bull ReuseRedisrsquosimplicityperformancescalabilityandhighavailability

bull CanbewritteninCC++GoPythonRustZig

bull Leverageexistingdatastructures

bull TurnRedisintoaMulti-Modeldatabase

ModulesExtendRedisInfinitely

12

httpsredislabscomcommunityredis-modules-hub

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

13

14

Introducing

RedisEnterprise

15

DBaaSbull Availablesincemid2013

bull 8500+enterprisecustomers

Softwarebull Availablesinceearly2015

bull 300+enterprisecustomers550K+databases

managedworldwide

bull 6oftopFortune10companies

bull 3oftop5communicationscompanies

Customersbull 3oftop4creditcardissuers

bull 3oftop5healthcarecompanies

bull CloudProvider

bull Highermarginbyndash Idleness

bull CloudLock-in

CloudProvidershavedifferentincentives

16

bull DBAASProvider

bull Highermarginbybetterresourceutilizationndash Multi-tenancyndash ReducingRAMndash CPUutilization

RedisEnterpriseAUniquePrimaryDatabase

HIGHESTPERFORMANCELINEARSCALING

HIGHAVAILABILITYWITHINSTANTFAILOVER

DURABILITYATMEMORYSPEEDS

ACTIVE-ACTIVEGEODISTRIBUTION(CRDT-BASED)

BUILT-INHIGHPERFORMANCE SEARCH

MULTI-MODEL

FLEXIBLEDEPLOYMENTOPTIONS(CLOUDON-PREMHYBRID)

INTELLIGENTTIEREDDATAACCESS(RAMampFLASHMEMORY)

FAST RELIABLE FLEXIBLE

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 3: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

1 WhatisRedisandRedisEnterprise

2 StreamProcessingwithRedisGears

3 RedisGearsasaMultimodelEngine

Agenda

4

RedisisFasthellip

5

hellipExtremelyFast

DB-EnginesRanking

6

7

Andyoursquovebeenusingitalready

RedisisExtensivelyandDiverselyUsed

8

UsesRedisforTimelinefollowing

Scope10-20TB

UsesRedisforLocalsiteglobalcaching

UsesRedisforRepositoryrouter

Scope10+TB

UsesRedisforGeosearchuserprofiles

Scope10-20TB

UsesRedisforAllmessages

Scope40TB

RedisTopDifferentiators

Simplicity ExtensibilityPerformanceNoSQLBenchmark

1

RedisDataStructures

2 3

RedisModules

9

Lists

Hashes

Bitmaps

Strings

Bitfield

Streams

Hyperloglog

SortedSets

Sets

GeospatialIndexes

10

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

11

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

bull Createyourowndatatypesandcommands

bull ReuseRedisrsquosimplicityperformancescalabilityandhighavailability

bull CanbewritteninCC++GoPythonRustZig

bull Leverageexistingdatastructures

bull TurnRedisintoaMulti-Modeldatabase

ModulesExtendRedisInfinitely

12

httpsredislabscomcommunityredis-modules-hub

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

13

14

Introducing

RedisEnterprise

15

DBaaSbull Availablesincemid2013

bull 8500+enterprisecustomers

Softwarebull Availablesinceearly2015

bull 300+enterprisecustomers550K+databases

managedworldwide

bull 6oftopFortune10companies

bull 3oftop5communicationscompanies

Customersbull 3oftop4creditcardissuers

bull 3oftop5healthcarecompanies

bull CloudProvider

bull Highermarginbyndash Idleness

bull CloudLock-in

CloudProvidershavedifferentincentives

16

bull DBAASProvider

bull Highermarginbybetterresourceutilizationndash Multi-tenancyndash ReducingRAMndash CPUutilization

RedisEnterpriseAUniquePrimaryDatabase

HIGHESTPERFORMANCELINEARSCALING

HIGHAVAILABILITYWITHINSTANTFAILOVER

DURABILITYATMEMORYSPEEDS

ACTIVE-ACTIVEGEODISTRIBUTION(CRDT-BASED)

BUILT-INHIGHPERFORMANCE SEARCH

MULTI-MODEL

FLEXIBLEDEPLOYMENTOPTIONS(CLOUDON-PREMHYBRID)

INTELLIGENTTIEREDDATAACCESS(RAMampFLASHMEMORY)

FAST RELIABLE FLEXIBLE

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 4: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

4

RedisisFasthellip

5

hellipExtremelyFast

DB-EnginesRanking

6

7

Andyoursquovebeenusingitalready

RedisisExtensivelyandDiverselyUsed

8

UsesRedisforTimelinefollowing

Scope10-20TB

UsesRedisforLocalsiteglobalcaching

UsesRedisforRepositoryrouter

Scope10+TB

UsesRedisforGeosearchuserprofiles

Scope10-20TB

UsesRedisforAllmessages

Scope40TB

RedisTopDifferentiators

Simplicity ExtensibilityPerformanceNoSQLBenchmark

1

RedisDataStructures

2 3

RedisModules

9

Lists

Hashes

Bitmaps

Strings

Bitfield

Streams

Hyperloglog

SortedSets

Sets

GeospatialIndexes

10

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

11

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

bull Createyourowndatatypesandcommands

bull ReuseRedisrsquosimplicityperformancescalabilityandhighavailability

bull CanbewritteninCC++GoPythonRustZig

bull Leverageexistingdatastructures

bull TurnRedisintoaMulti-Modeldatabase

ModulesExtendRedisInfinitely

12

httpsredislabscomcommunityredis-modules-hub

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

13

14

Introducing

RedisEnterprise

15

DBaaSbull Availablesincemid2013

bull 8500+enterprisecustomers

Softwarebull Availablesinceearly2015

bull 300+enterprisecustomers550K+databases

managedworldwide

bull 6oftopFortune10companies

bull 3oftop5communicationscompanies

Customersbull 3oftop4creditcardissuers

bull 3oftop5healthcarecompanies

bull CloudProvider

bull Highermarginbyndash Idleness

bull CloudLock-in

CloudProvidershavedifferentincentives

16

bull DBAASProvider

bull Highermarginbybetterresourceutilizationndash Multi-tenancyndash ReducingRAMndash CPUutilization

RedisEnterpriseAUniquePrimaryDatabase

HIGHESTPERFORMANCELINEARSCALING

HIGHAVAILABILITYWITHINSTANTFAILOVER

DURABILITYATMEMORYSPEEDS

ACTIVE-ACTIVEGEODISTRIBUTION(CRDT-BASED)

BUILT-INHIGHPERFORMANCE SEARCH

MULTI-MODEL

FLEXIBLEDEPLOYMENTOPTIONS(CLOUDON-PREMHYBRID)

INTELLIGENTTIEREDDATAACCESS(RAMampFLASHMEMORY)

FAST RELIABLE FLEXIBLE

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 5: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

5

hellipExtremelyFast

DB-EnginesRanking

6

7

Andyoursquovebeenusingitalready

RedisisExtensivelyandDiverselyUsed

8

UsesRedisforTimelinefollowing

Scope10-20TB

UsesRedisforLocalsiteglobalcaching

UsesRedisforRepositoryrouter

Scope10+TB

UsesRedisforGeosearchuserprofiles

Scope10-20TB

UsesRedisforAllmessages

Scope40TB

RedisTopDifferentiators

Simplicity ExtensibilityPerformanceNoSQLBenchmark

1

RedisDataStructures

2 3

RedisModules

9

Lists

Hashes

Bitmaps

Strings

Bitfield

Streams

Hyperloglog

SortedSets

Sets

GeospatialIndexes

10

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

11

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

bull Createyourowndatatypesandcommands

bull ReuseRedisrsquosimplicityperformancescalabilityandhighavailability

bull CanbewritteninCC++GoPythonRustZig

bull Leverageexistingdatastructures

bull TurnRedisintoaMulti-Modeldatabase

ModulesExtendRedisInfinitely

12

httpsredislabscomcommunityredis-modules-hub

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

13

14

Introducing

RedisEnterprise

15

DBaaSbull Availablesincemid2013

bull 8500+enterprisecustomers

Softwarebull Availablesinceearly2015

bull 300+enterprisecustomers550K+databases

managedworldwide

bull 6oftopFortune10companies

bull 3oftop5communicationscompanies

Customersbull 3oftop4creditcardissuers

bull 3oftop5healthcarecompanies

bull CloudProvider

bull Highermarginbyndash Idleness

bull CloudLock-in

CloudProvidershavedifferentincentives

16

bull DBAASProvider

bull Highermarginbybetterresourceutilizationndash Multi-tenancyndash ReducingRAMndash CPUutilization

RedisEnterpriseAUniquePrimaryDatabase

HIGHESTPERFORMANCELINEARSCALING

HIGHAVAILABILITYWITHINSTANTFAILOVER

DURABILITYATMEMORYSPEEDS

ACTIVE-ACTIVEGEODISTRIBUTION(CRDT-BASED)

BUILT-INHIGHPERFORMANCE SEARCH

MULTI-MODEL

FLEXIBLEDEPLOYMENTOPTIONS(CLOUDON-PREMHYBRID)

INTELLIGENTTIEREDDATAACCESS(RAMampFLASHMEMORY)

FAST RELIABLE FLEXIBLE

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 6: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

DB-EnginesRanking

6

7

Andyoursquovebeenusingitalready

RedisisExtensivelyandDiverselyUsed

8

UsesRedisforTimelinefollowing

Scope10-20TB

UsesRedisforLocalsiteglobalcaching

UsesRedisforRepositoryrouter

Scope10+TB

UsesRedisforGeosearchuserprofiles

Scope10-20TB

UsesRedisforAllmessages

Scope40TB

RedisTopDifferentiators

Simplicity ExtensibilityPerformanceNoSQLBenchmark

1

RedisDataStructures

2 3

RedisModules

9

Lists

Hashes

Bitmaps

Strings

Bitfield

Streams

Hyperloglog

SortedSets

Sets

GeospatialIndexes

10

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

11

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

bull Createyourowndatatypesandcommands

bull ReuseRedisrsquosimplicityperformancescalabilityandhighavailability

bull CanbewritteninCC++GoPythonRustZig

bull Leverageexistingdatastructures

bull TurnRedisintoaMulti-Modeldatabase

ModulesExtendRedisInfinitely

12

httpsredislabscomcommunityredis-modules-hub

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

13

14

Introducing

RedisEnterprise

15

DBaaSbull Availablesincemid2013

bull 8500+enterprisecustomers

Softwarebull Availablesinceearly2015

bull 300+enterprisecustomers550K+databases

managedworldwide

bull 6oftopFortune10companies

bull 3oftop5communicationscompanies

Customersbull 3oftop4creditcardissuers

bull 3oftop5healthcarecompanies

bull CloudProvider

bull Highermarginbyndash Idleness

bull CloudLock-in

CloudProvidershavedifferentincentives

16

bull DBAASProvider

bull Highermarginbybetterresourceutilizationndash Multi-tenancyndash ReducingRAMndash CPUutilization

RedisEnterpriseAUniquePrimaryDatabase

HIGHESTPERFORMANCELINEARSCALING

HIGHAVAILABILITYWITHINSTANTFAILOVER

DURABILITYATMEMORYSPEEDS

ACTIVE-ACTIVEGEODISTRIBUTION(CRDT-BASED)

BUILT-INHIGHPERFORMANCE SEARCH

MULTI-MODEL

FLEXIBLEDEPLOYMENTOPTIONS(CLOUDON-PREMHYBRID)

INTELLIGENTTIEREDDATAACCESS(RAMampFLASHMEMORY)

FAST RELIABLE FLEXIBLE

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 7: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

7

Andyoursquovebeenusingitalready

RedisisExtensivelyandDiverselyUsed

8

UsesRedisforTimelinefollowing

Scope10-20TB

UsesRedisforLocalsiteglobalcaching

UsesRedisforRepositoryrouter

Scope10+TB

UsesRedisforGeosearchuserprofiles

Scope10-20TB

UsesRedisforAllmessages

Scope40TB

RedisTopDifferentiators

Simplicity ExtensibilityPerformanceNoSQLBenchmark

1

RedisDataStructures

2 3

RedisModules

9

Lists

Hashes

Bitmaps

Strings

Bitfield

Streams

Hyperloglog

SortedSets

Sets

GeospatialIndexes

10

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

11

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

bull Createyourowndatatypesandcommands

bull ReuseRedisrsquosimplicityperformancescalabilityandhighavailability

bull CanbewritteninCC++GoPythonRustZig

bull Leverageexistingdatastructures

bull TurnRedisintoaMulti-Modeldatabase

ModulesExtendRedisInfinitely

12

httpsredislabscomcommunityredis-modules-hub

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

13

14

Introducing

RedisEnterprise

15

DBaaSbull Availablesincemid2013

bull 8500+enterprisecustomers

Softwarebull Availablesinceearly2015

bull 300+enterprisecustomers550K+databases

managedworldwide

bull 6oftopFortune10companies

bull 3oftop5communicationscompanies

Customersbull 3oftop4creditcardissuers

bull 3oftop5healthcarecompanies

bull CloudProvider

bull Highermarginbyndash Idleness

bull CloudLock-in

CloudProvidershavedifferentincentives

16

bull DBAASProvider

bull Highermarginbybetterresourceutilizationndash Multi-tenancyndash ReducingRAMndash CPUutilization

RedisEnterpriseAUniquePrimaryDatabase

HIGHESTPERFORMANCELINEARSCALING

HIGHAVAILABILITYWITHINSTANTFAILOVER

DURABILITYATMEMORYSPEEDS

ACTIVE-ACTIVEGEODISTRIBUTION(CRDT-BASED)

BUILT-INHIGHPERFORMANCE SEARCH

MULTI-MODEL

FLEXIBLEDEPLOYMENTOPTIONS(CLOUDON-PREMHYBRID)

INTELLIGENTTIEREDDATAACCESS(RAMampFLASHMEMORY)

FAST RELIABLE FLEXIBLE

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 8: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

RedisisExtensivelyandDiverselyUsed

8

UsesRedisforTimelinefollowing

Scope10-20TB

UsesRedisforLocalsiteglobalcaching

UsesRedisforRepositoryrouter

Scope10+TB

UsesRedisforGeosearchuserprofiles

Scope10-20TB

UsesRedisforAllmessages

Scope40TB

RedisTopDifferentiators

Simplicity ExtensibilityPerformanceNoSQLBenchmark

1

RedisDataStructures

2 3

RedisModules

9

Lists

Hashes

Bitmaps

Strings

Bitfield

Streams

Hyperloglog

SortedSets

Sets

GeospatialIndexes

10

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

11

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

bull Createyourowndatatypesandcommands

bull ReuseRedisrsquosimplicityperformancescalabilityandhighavailability

bull CanbewritteninCC++GoPythonRustZig

bull Leverageexistingdatastructures

bull TurnRedisintoaMulti-Modeldatabase

ModulesExtendRedisInfinitely

12

httpsredislabscomcommunityredis-modules-hub

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

13

14

Introducing

RedisEnterprise

15

DBaaSbull Availablesincemid2013

bull 8500+enterprisecustomers

Softwarebull Availablesinceearly2015

bull 300+enterprisecustomers550K+databases

managedworldwide

bull 6oftopFortune10companies

bull 3oftop5communicationscompanies

Customersbull 3oftop4creditcardissuers

bull 3oftop5healthcarecompanies

bull CloudProvider

bull Highermarginbyndash Idleness

bull CloudLock-in

CloudProvidershavedifferentincentives

16

bull DBAASProvider

bull Highermarginbybetterresourceutilizationndash Multi-tenancyndash ReducingRAMndash CPUutilization

RedisEnterpriseAUniquePrimaryDatabase

HIGHESTPERFORMANCELINEARSCALING

HIGHAVAILABILITYWITHINSTANTFAILOVER

DURABILITYATMEMORYSPEEDS

ACTIVE-ACTIVEGEODISTRIBUTION(CRDT-BASED)

BUILT-INHIGHPERFORMANCE SEARCH

MULTI-MODEL

FLEXIBLEDEPLOYMENTOPTIONS(CLOUDON-PREMHYBRID)

INTELLIGENTTIEREDDATAACCESS(RAMampFLASHMEMORY)

FAST RELIABLE FLEXIBLE

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 9: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

RedisTopDifferentiators

Simplicity ExtensibilityPerformanceNoSQLBenchmark

1

RedisDataStructures

2 3

RedisModules

9

Lists

Hashes

Bitmaps

Strings

Bitfield

Streams

Hyperloglog

SortedSets

Sets

GeospatialIndexes

10

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

11

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

bull Createyourowndatatypesandcommands

bull ReuseRedisrsquosimplicityperformancescalabilityandhighavailability

bull CanbewritteninCC++GoPythonRustZig

bull Leverageexistingdatastructures

bull TurnRedisintoaMulti-Modeldatabase

ModulesExtendRedisInfinitely

12

httpsredislabscomcommunityredis-modules-hub

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

13

14

Introducing

RedisEnterprise

15

DBaaSbull Availablesincemid2013

bull 8500+enterprisecustomers

Softwarebull Availablesinceearly2015

bull 300+enterprisecustomers550K+databases

managedworldwide

bull 6oftopFortune10companies

bull 3oftop5communicationscompanies

Customersbull 3oftop4creditcardissuers

bull 3oftop5healthcarecompanies

bull CloudProvider

bull Highermarginbyndash Idleness

bull CloudLock-in

CloudProvidershavedifferentincentives

16

bull DBAASProvider

bull Highermarginbybetterresourceutilizationndash Multi-tenancyndash ReducingRAMndash CPUutilization

RedisEnterpriseAUniquePrimaryDatabase

HIGHESTPERFORMANCELINEARSCALING

HIGHAVAILABILITYWITHINSTANTFAILOVER

DURABILITYATMEMORYSPEEDS

ACTIVE-ACTIVEGEODISTRIBUTION(CRDT-BASED)

BUILT-INHIGHPERFORMANCE SEARCH

MULTI-MODEL

FLEXIBLEDEPLOYMENTOPTIONS(CLOUDON-PREMHYBRID)

INTELLIGENTTIEREDDATAACCESS(RAMampFLASHMEMORY)

FAST RELIABLE FLEXIBLE

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 10: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

10

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

11

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

bull Createyourowndatatypesandcommands

bull ReuseRedisrsquosimplicityperformancescalabilityandhighavailability

bull CanbewritteninCC++GoPythonRustZig

bull Leverageexistingdatastructures

bull TurnRedisintoaMulti-Modeldatabase

ModulesExtendRedisInfinitely

12

httpsredislabscomcommunityredis-modules-hub

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

13

14

Introducing

RedisEnterprise

15

DBaaSbull Availablesincemid2013

bull 8500+enterprisecustomers

Softwarebull Availablesinceearly2015

bull 300+enterprisecustomers550K+databases

managedworldwide

bull 6oftopFortune10companies

bull 3oftop5communicationscompanies

Customersbull 3oftop4creditcardissuers

bull 3oftop5healthcarecompanies

bull CloudProvider

bull Highermarginbyndash Idleness

bull CloudLock-in

CloudProvidershavedifferentincentives

16

bull DBAASProvider

bull Highermarginbybetterresourceutilizationndash Multi-tenancyndash ReducingRAMndash CPUutilization

RedisEnterpriseAUniquePrimaryDatabase

HIGHESTPERFORMANCELINEARSCALING

HIGHAVAILABILITYWITHINSTANTFAILOVER

DURABILITYATMEMORYSPEEDS

ACTIVE-ACTIVEGEODISTRIBUTION(CRDT-BASED)

BUILT-INHIGHPERFORMANCE SEARCH

MULTI-MODEL

FLEXIBLEDEPLOYMENTOPTIONS(CLOUDON-PREMHYBRID)

INTELLIGENTTIEREDDATAACCESS(RAMampFLASHMEMORY)

FAST RELIABLE FLEXIBLE

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 11: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

11

WritteninC

Servedentirelyfrommemory

Single-threadedlockfree

MostcommandsareexecutedwithO(1)

complexity

Accesstodiscreteelements

withinobjects

Reducedbandwidthoverheadrequirements

Easytoparsenetworkingprotocol

Pipeliningforreduced

networkoverhead

Connectionpooling

OPTIMIZEDARCHITECTURE ADVANCEDPROCESSING EFFICIENTOPERATION

RedisSpeeddifferentiators

bull Createyourowndatatypesandcommands

bull ReuseRedisrsquosimplicityperformancescalabilityandhighavailability

bull CanbewritteninCC++GoPythonRustZig

bull Leverageexistingdatastructures

bull TurnRedisintoaMulti-Modeldatabase

ModulesExtendRedisInfinitely

12

httpsredislabscomcommunityredis-modules-hub

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

13

14

Introducing

RedisEnterprise

15

DBaaSbull Availablesincemid2013

bull 8500+enterprisecustomers

Softwarebull Availablesinceearly2015

bull 300+enterprisecustomers550K+databases

managedworldwide

bull 6oftopFortune10companies

bull 3oftop5communicationscompanies

Customersbull 3oftop4creditcardissuers

bull 3oftop5healthcarecompanies

bull CloudProvider

bull Highermarginbyndash Idleness

bull CloudLock-in

CloudProvidershavedifferentincentives

16

bull DBAASProvider

bull Highermarginbybetterresourceutilizationndash Multi-tenancyndash ReducingRAMndash CPUutilization

RedisEnterpriseAUniquePrimaryDatabase

HIGHESTPERFORMANCELINEARSCALING

HIGHAVAILABILITYWITHINSTANTFAILOVER

DURABILITYATMEMORYSPEEDS

ACTIVE-ACTIVEGEODISTRIBUTION(CRDT-BASED)

BUILT-INHIGHPERFORMANCE SEARCH

MULTI-MODEL

FLEXIBLEDEPLOYMENTOPTIONS(CLOUDON-PREMHYBRID)

INTELLIGENTTIEREDDATAACCESS(RAMampFLASHMEMORY)

FAST RELIABLE FLEXIBLE

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 12: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

bull Createyourowndatatypesandcommands

bull ReuseRedisrsquosimplicityperformancescalabilityandhighavailability

bull CanbewritteninCC++GoPythonRustZig

bull Leverageexistingdatastructures

bull TurnRedisintoaMulti-Modeldatabase

ModulesExtendRedisInfinitely

12

httpsredislabscomcommunityredis-modules-hub

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

13

14

Introducing

RedisEnterprise

15

DBaaSbull Availablesincemid2013

bull 8500+enterprisecustomers

Softwarebull Availablesinceearly2015

bull 300+enterprisecustomers550K+databases

managedworldwide

bull 6oftopFortune10companies

bull 3oftop5communicationscompanies

Customersbull 3oftop4creditcardissuers

bull 3oftop5healthcarecompanies

bull CloudProvider

bull Highermarginbyndash Idleness

bull CloudLock-in

CloudProvidershavedifferentincentives

16

bull DBAASProvider

bull Highermarginbybetterresourceutilizationndash Multi-tenancyndash ReducingRAMndash CPUutilization

RedisEnterpriseAUniquePrimaryDatabase

HIGHESTPERFORMANCELINEARSCALING

HIGHAVAILABILITYWITHINSTANTFAILOVER

DURABILITYATMEMORYSPEEDS

ACTIVE-ACTIVEGEODISTRIBUTION(CRDT-BASED)

BUILT-INHIGHPERFORMANCE SEARCH

MULTI-MODEL

FLEXIBLEDEPLOYMENTOPTIONS(CLOUDON-PREMHYBRID)

INTELLIGENTTIEREDDATAACCESS(RAMampFLASHMEMORY)

FAST RELIABLE FLEXIBLE

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 13: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

13

14

Introducing

RedisEnterprise

15

DBaaSbull Availablesincemid2013

bull 8500+enterprisecustomers

Softwarebull Availablesinceearly2015

bull 300+enterprisecustomers550K+databases

managedworldwide

bull 6oftopFortune10companies

bull 3oftop5communicationscompanies

Customersbull 3oftop4creditcardissuers

bull 3oftop5healthcarecompanies

bull CloudProvider

bull Highermarginbyndash Idleness

bull CloudLock-in

CloudProvidershavedifferentincentives

16

bull DBAASProvider

bull Highermarginbybetterresourceutilizationndash Multi-tenancyndash ReducingRAMndash CPUutilization

RedisEnterpriseAUniquePrimaryDatabase

HIGHESTPERFORMANCELINEARSCALING

HIGHAVAILABILITYWITHINSTANTFAILOVER

DURABILITYATMEMORYSPEEDS

ACTIVE-ACTIVEGEODISTRIBUTION(CRDT-BASED)

BUILT-INHIGHPERFORMANCE SEARCH

MULTI-MODEL

FLEXIBLEDEPLOYMENTOPTIONS(CLOUDON-PREMHYBRID)

INTELLIGENTTIEREDDATAACCESS(RAMampFLASHMEMORY)

FAST RELIABLE FLEXIBLE

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 14: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

14

Introducing

RedisEnterprise

15

DBaaSbull Availablesincemid2013

bull 8500+enterprisecustomers

Softwarebull Availablesinceearly2015

bull 300+enterprisecustomers550K+databases

managedworldwide

bull 6oftopFortune10companies

bull 3oftop5communicationscompanies

Customersbull 3oftop4creditcardissuers

bull 3oftop5healthcarecompanies

bull CloudProvider

bull Highermarginbyndash Idleness

bull CloudLock-in

CloudProvidershavedifferentincentives

16

bull DBAASProvider

bull Highermarginbybetterresourceutilizationndash Multi-tenancyndash ReducingRAMndash CPUutilization

RedisEnterpriseAUniquePrimaryDatabase

HIGHESTPERFORMANCELINEARSCALING

HIGHAVAILABILITYWITHINSTANTFAILOVER

DURABILITYATMEMORYSPEEDS

ACTIVE-ACTIVEGEODISTRIBUTION(CRDT-BASED)

BUILT-INHIGHPERFORMANCE SEARCH

MULTI-MODEL

FLEXIBLEDEPLOYMENTOPTIONS(CLOUDON-PREMHYBRID)

INTELLIGENTTIEREDDATAACCESS(RAMampFLASHMEMORY)

FAST RELIABLE FLEXIBLE

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 15: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

RedisEnterprise

15

DBaaSbull Availablesincemid2013

bull 8500+enterprisecustomers

Softwarebull Availablesinceearly2015

bull 300+enterprisecustomers550K+databases

managedworldwide

bull 6oftopFortune10companies

bull 3oftop5communicationscompanies

Customersbull 3oftop4creditcardissuers

bull 3oftop5healthcarecompanies

bull CloudProvider

bull Highermarginbyndash Idleness

bull CloudLock-in

CloudProvidershavedifferentincentives

16

bull DBAASProvider

bull Highermarginbybetterresourceutilizationndash Multi-tenancyndash ReducingRAMndash CPUutilization

RedisEnterpriseAUniquePrimaryDatabase

HIGHESTPERFORMANCELINEARSCALING

HIGHAVAILABILITYWITHINSTANTFAILOVER

DURABILITYATMEMORYSPEEDS

ACTIVE-ACTIVEGEODISTRIBUTION(CRDT-BASED)

BUILT-INHIGHPERFORMANCE SEARCH

MULTI-MODEL

FLEXIBLEDEPLOYMENTOPTIONS(CLOUDON-PREMHYBRID)

INTELLIGENTTIEREDDATAACCESS(RAMampFLASHMEMORY)

FAST RELIABLE FLEXIBLE

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 16: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

bull CloudProvider

bull Highermarginbyndash Idleness

bull CloudLock-in

CloudProvidershavedifferentincentives

16

bull DBAASProvider

bull Highermarginbybetterresourceutilizationndash Multi-tenancyndash ReducingRAMndash CPUutilization

RedisEnterpriseAUniquePrimaryDatabase

HIGHESTPERFORMANCELINEARSCALING

HIGHAVAILABILITYWITHINSTANTFAILOVER

DURABILITYATMEMORYSPEEDS

ACTIVE-ACTIVEGEODISTRIBUTION(CRDT-BASED)

BUILT-INHIGHPERFORMANCE SEARCH

MULTI-MODEL

FLEXIBLEDEPLOYMENTOPTIONS(CLOUDON-PREMHYBRID)

INTELLIGENTTIEREDDATAACCESS(RAMampFLASHMEMORY)

FAST RELIABLE FLEXIBLE

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 17: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

RedisEnterpriseAUniquePrimaryDatabase

HIGHESTPERFORMANCELINEARSCALING

HIGHAVAILABILITYWITHINSTANTFAILOVER

DURABILITYATMEMORYSPEEDS

ACTIVE-ACTIVEGEODISTRIBUTION(CRDT-BASED)

BUILT-INHIGHPERFORMANCE SEARCH

MULTI-MODEL

FLEXIBLEDEPLOYMENTOPTIONS(CLOUDON-PREMHYBRID)

INTELLIGENTTIEREDDATAACCESS(RAMampFLASHMEMORY)

FAST RELIABLE FLEXIBLE

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 18: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

18

Unevennumberofsymmetricnodes

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 19: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

19

Singlemasterdatabase

M

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 20: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

20

AnHAdatabase

SM

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 21: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

21

AClusteredDatabase

M1 M2 M3

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 22: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

Howdokeysgetassignedtopartitions

22

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 23: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

Howdokeysgetassignedtopartitions

23

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 24: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

RedisEnterpriseCluster

Node1 Node2 NodeN(oddnumber)

24

AHighlyAvailableClusteredDatabase

M1 M2 M3S3 S1 S2

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 25: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

RedisEnterpriseNode

25

ClusterManager

EnterpriseLayer

OpenSourceLayer

RESTAPIZerolatencyproxy

RedisShards

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 26: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

RedisEnterpriseSharedNothingSymmetricArchitecture

ClusterManagementPath

NodeWatchdogClusterWatchdog

Node1 Node2 NodeN(oddnumber)

RedisShardsampProxies

DataPath

26

Data-PathandControlManagementPathSeparation

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 27: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

27

asadatagrid

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 28: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

MicroservicesArchitectureandPolyglotPersistence

28

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 29: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

TheCostofPolyglotPersistence

AuthenticationAuthentication

Authentication

KeyValue

K V

KeyValue

K V

KeyValue

K V

Customers

KeyValue

K V

Graph

CustomersKeyValue

K V

Graph

Catalog

RDBMS Cache

Search

SessionStore

SessionStore

SessionStore

Document

Document

Document

API

API

API

API

API

API

API

API

API

FraudDetection

API

ColumnarSearch

Fulfilment API

RDBMS

Increasedapplicationcomplexity CostlycommunicationApplicationdoesheavyliftinginsharingdatakeepingdatasetsinsync

HighoperationalburdenHighercostofownershipDifferentdatabaseshavespecializedadministrativescalingavailability

requirements

Sub-optimalResourceUsage HighercostDedicatingpodsserversforeachtypeofdatabasereducesdeploymentefficiency

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 30: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

API API API API APIAPI API

RedisEnterpriseAMulti-modelDatabaseforMicroservices

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 31: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

Authentication Customers Catalog Search FraudDetection SessionStore

SearchGraphKeyValue

RDBMS

KeyValueCache

RDBMS

Cache

Fulfilment

KeyValue Document

Built-inMessageBroker

Built-inPub-SubStreamsforeventsynchacrossdatastores

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 32: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

Whatarewemissing

bull Howtoconsumemessagesinthisldquobuiltinmessageborkerrdquo

bull GivenashardeddatabasehowcanIrunanalyticalqueries

bull MultiModeldatabasendash Singlecopyincoredatatypesndash Intermodulecommunicationndash ComponentXdoingtranslationsbetweenmodules

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 33: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

33

RedisGears

Introducing

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 34: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

RedisGearsisaServerlessengineformulti-modelandclusteroperationsinRedissupportingbotheventdriven

aswellasbatchoperations

WhatisRedisGears

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 35: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

GearsCoordinator MapReducerGearsExecuter

GearsinfrastructureiswritteninC

C-API

Soon Soon

HighPerformanceArchitecture

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 36: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

RedisGearsallowstodefineapipeofoperations

bull Returningvaluefromoneoperationispassedtothenextoperationinthepipe

bull Lastoperationreturningtheresulttotheuserbull Firstoperationiscalledlsquoreaderrsquo-responsibleforprovidingdata

ndash Keysreader-readkeysfromRedisndash Streamreader-readstreamsfromRedisndash Pythonreader-allowtousertowritehisownreadersinpython

ScriptingwithRedisGears

36

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 37: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

bull Map

bull FlatMap

bull Filterbull Groupby+Reducebull Aggregatebull Sortbull Limit

bull ForEachbull Distinct

SupportedOperations

37

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 38: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

Reader(Flat)Mapper

UsingRedisGearsndash(Flat)Mapping

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 39: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

UsingRedisGears-Filtering

Reader Filterrecordwith1doc

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 40: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

UsingRedisGears-Aggregate

Reader CountAggregator

1

1

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 41: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

Demo

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 42: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

GearshasastreamingAPItoallowtotriggergearsexecutiononeventsndash RedisStreamevents-Triggeranexecutionwheneveranewdataentersasteamndash RedisKeysevents-Triggeranoperationwheneverakeyistouched

UseCase1ndashStreamProcessing

RedisTimeSeriesRedisStreams

Everysec

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 43: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

BecauseofRedisGearsrsquoflexibility(itsactuallyrunningpython)youcanachieveinternalmoduleintegrationwithitndash ReadfromhashesandindexinRediSearchRedisGraphndash ReadRedisJSONdataandpasstoRedisTimeSeriesndash hellip

UseCase2ndashaMultiModelEngine

RediSearch

RedisHashesEveryupdate

RedisGraph

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 44: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

createthebuilderbuilder=GearsBuilder()filtereventsonkeyall_keysbuilderfilter(lambdaxx[key]=all_keys)addthekeystoall_keyssetbuildermap(lambdaxexecute(saddall_keysx[key]))registertheexecutiononkeyspacenotificationbuilderregister()

Buildagearthatcreatesmaintainsasetofallkeyswithinredis

Recipe1ndasheventriggering

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 45: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

createthepipebuilderKeysOnlyReaderisaperformanceimprovementonlypipingthekeysbuilder=GearsBuilder(KeysOnlyReader)getfromeachhashthegenresfieldbuildermap(lambdaxexecute(hgetxgenres))filterthosewhodonothavegenresbuilderfilter(lambdaxxisnotNone)splitgenresbycommabuilderflatmap(lambdaxxsplit())countforeachgenrethenumberoftimesitappearsbuildercountby()starttheexecutionbuilderrun(movie)

Buildagearthatcountshowoftenagenreisusedwithinasetofmovies

Recipe2ndashmapreducing

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 46: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

createthebuilderwithaStreamReaderbuilder=GearsBuilder(StreamReader)extracteachfieldvaluepairfromthemessageandincreasethepipegranularitybuilderflatmap(lambdax[(a[0]a[1])forainxitems()])filteroutthestreamIditselfbuilderfilter(lambdaxx[0]=streamId)makesurethegearsdatalivesinthecorrectshardbuilderrepartition(lambdaxx[0])applyeachfieldvaluepairtoakeybuilderforeach(lambdaxexecute(setx[0]x[1]))registeronnewmessagesonthestreaminputStreambuilderregister(inputStream)

Buildagearthatconsumesastreamandupdateskeysaccordingly

Recipe3ndashstreamprocessing

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 47: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

ExampleTriggerExplained

47

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 48: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

ExampleTriggerExplained

48

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 49: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

ExampleTriggerExplained

49

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 50: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

ExampleTriggerExplained-Flatmap

50

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 51: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

ExampleTriggerExplained-Repartition

51

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 52: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

ExampleTriggerExplained-executeCommand

52

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 53: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

Demo

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 54: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

DemoSetup

54

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 55: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

bull httpsgithubcomRedisGearsAnimalRecognitionDemo

bull redisfoundmycat

Challenge

55

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 56: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

RediSearch(GA) redisearchio

RedisBloom(GA) redisbloomio

RedisTimeSeries redistimeseriesio

RedisJSON(GA) redisjsonio

RedisAI redisaiio

RedisGraph(GA) redisgraphio

RedisModules

56

RedisGearsredisgearsio

Thankyoupieterredislabscom

57

Page 57: Redis In memory data processing - imcsummit.org · – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever

Thankyoupieterredislabscom

57