Потоковая фильтрация событий

113
Streaming event matching Dmitry Schitinin

Upload: cee-secr

Post on 06-Jan-2017

82 views

Category:

Software


0 download

TRANSCRIPT

Streaming event matching

Dmitry Schitinin

About

Backend infrastructure team

› Services› Libraries› Frameworks

@ Yandex.Classifieds

› auto.ru

› auto.yandex.ru

› rabota.yandex.ru

› realty.yandex.ru

› travel.yandex.ru

https://stat.yandex.ru 2

About

Backend infrastructure team

› Services› Libraries› Frameworks

@ Yandex.Classifieds

› auto.ru

› auto.yandex.ru

› rabota.yandex.ru

› realty.yandex.ru

› travel.yandex.ru

https://stat.yandex.ru 2

About

Backend infrastructure team

› Services› Libraries› Frameworks

@ Yandex.Classifieds

› auto.ru

› auto.yandex.ru

› rabota.yandex.ru

› realty.yandex.ru

› travel.yandex.ruhttps://stat.yandex.ru 2

Introduction

Audience

› Software engineers/architects

› Event stream processing

› Distributed systems

4

Features

› Stream of offers (tens per second)

› Up to 500K subscriptions

› Complex multi field conditions

› Low latency (seconds)

8

Features

› Stream of offers (tens per second)

› Up to 500K subscriptions

› Complex multi field conditions

› Low latency (seconds)

8

Features

› Stream of offers (tens per second)

› Up to 500K subscriptions

› Complex multi field conditions

› Low latency (seconds)

8

Features

› Stream of offers (tens per second)

› Up to 500K subscriptions

› Complex multi field conditions

› Low latency (seconds)

8

Features

› Stream of offers (tens per second)

› Up to 500K subscriptions

› Complex multi field conditions

› Low latency (seconds)

8

Motivation

Customer subscriptions

› Short notification delays (minutes)

› Instant subscription modifications› Notifications transport

› email, SMS, push, pigeons, owls, etc.

10

Customer subscriptions

› Short notification delays (minutes)

› Instant subscription modifications› Notifications transport

› email, SMS, push, pigeons, owls, etc.

10

Customer subscriptions

› Short notification delays (minutes)

› Instant subscription modifications

› Notifications transport

› email, SMS, push, pigeons, owls, etc.

10

Customer subscriptions

› Short notification delays (minutes)

› Instant subscription modifications› Notifications transport

› email, SMS, push, pigeons, owls, etc.

10

Non-functional requirements

› Scalability by subscriptions› millions

› Scalability by events› tens-hundreds per second

› Fault tolerance› nodes, network, data centers

11

Non-functional requirements

› Scalability by subscriptions› millions

› Scalability by events› tens-hundreds per second

› Fault tolerance› nodes, network, data centers

11

Non-functional requirements

› Scalability by subscriptions› millions

› Scalability by events› tens-hundreds per second

› Fault tolerance› nodes, network, data centers

11

Non-functional requirements

› Scalability by subscriptions› millions

› Scalability by events› tens-hundreds per second

› Fault tolerance› nodes, network, data centers

11

Naive solution

Pros:› Simplicity

› Working

Cons:› High notificationlatency (hours, days)

› Search systemextra load

› Scalability?

19

Naive solution

Pros:› Simplicity

› Working

Cons:

› High notificationlatency (hours, days)

› Search systemextra load

› Scalability?

19

Naive solution

Pros:› Simplicity

› Working

Cons:› High notificationlatency (hours, days)

› Search systemextra load

› Scalability?

19

Naive solution

Pros:› Simplicity

› Working

Cons:› High notificationlatency (hours, days)

› Search systemextra load

› Scalability?

19

Naive solution

Pros:› Simplicity

› Working

Cons:› High notificationlatency (hours, days)

› Search systemextra load

› Scalability?19

Not so naive approach

Pros:› Low latency

› No extra work› Possibly scalable

Cons:› More complex, butnot complicated

22

Not so naive approach

Pros:

› Low latency

› No extra work› Possibly scalable

Cons:› More complex, butnot complicated

22

Not so naive approach

Pros:› Low latency

› No extra work› Possibly scalable

Cons:› More complex, butnot complicated

22

Not so naive approach

Pros:› Low latency

› No extra work

› Possibly scalable

Cons:› More complex, butnot complicated

22

Not so naive approach

Pros:› Low latency

› No extra work› Possibly scalable

Cons:› More complex, butnot complicated

22

Not so naive approach

Pros:› Low latency

› No extra work› Possibly scalable

Cons:

› More complex, butnot complicated

22

Not so naive approach

Pros:› Low latency

› No extra work› Possibly scalable

Cons:› More complex

, butnot complicated

22

Not so naive approach

Pros:› Low latency

› No extra work› Possibly scalable

Cons:› More complex, butnot complicated

22

Matching

Data model

› Event:

› Subscription:mark=FORD&model=FOCUS&model=MONDEO&year=2015

25

A better way

Commonalities:› mark=FORD&model=FOCUS

› mark=FORD&year=2010

› year=2010&price_from=500000

Data structures› BDD

› DAG

› Search tree

› Indices

27

A better way

Commonalities:› mark=FORD&model=FOCUS

› mark=FORD&year=2010

› year=2010&price_from=500000

Data structures› BDD

› DAG

› Search tree

› Indices

27

A better way

Commonalities:› mark=FORD&model=FOCUS

› mark=FORD&year=2010

› year=2010&price_from=500000

Data structures› BDD

› DAG

› Search tree

› Indices

27

Architecture

Architecture

Matcher

Resilience

› Subscriptions => Tokens› Token ≈responsibility

› Nodes distribute tokens

51

Resilience

› Subscriptions => Tokens

› Token ≈responsibility

› Nodes distribute tokens

51

Resilience

› Subscriptions => Tokens› Token ≈responsibility

› Nodes distribute tokens

51

Resilience

› Subscriptions => Tokens› Token ≈responsibility

› Nodes distribute tokens

51

Architecture

Notifier

Implementation

Scala

› Functional & concise› Persistent data structures for matching tree(1Kloc)

› No need for synchronization

› Property-based testing (matching algorithm)

1http://www.scalacheck.org

68

Scala

› Functional & concise

› Persistent data structures for matching tree(1Kloc)

› No need for synchronization

› Property-based testing (matching algorithm)

1http://www.scalacheck.org

68

Scala

› Functional & concise› Persistent data structures for matching tree(1Kloc)

› No need for synchronization

› Property-based testing (matching algorithm)

1http://www.scalacheck.org

68

Scala

› Functional & concise› Persistent data structures for matching tree(1Kloc)

› No need for synchronization

› Property-based testing (matching algorithm)

1http://www.scalacheck.org

68

Scala

› Functional & concise› Persistent data structures for matching tree(1Kloc)

› No need for synchronization

› Property-based testing (matching algorithm)

1http://www.scalacheck.org

68

Akka

› actors (events)

› akka-remoting (networking)

› protobuf (serialization)

› akka-fsm (aggregation)

69

Akka

› actors (events)

› akka-remoting (networking)

› protobuf (serialization)

› akka-fsm (aggregation)

69

Akka

› actors (events)

› akka-remoting (networking)

› protobuf (serialization)

› akka-fsm (aggregation)

69

Akka

› actors (events)

› akka-remoting (networking)

› protobuf (serialization)

› akka-fsm (aggregation)

69

Akka

› actors (events)

› akka-remoting (networking)

› protobuf (serialization)

› akka-fsm (aggregation)

69

Zookeeper

› Exclusive tokenacquirement

› Service discovery

http://zookeeper.apache.org 70

Conclusion

Achievements

› Low customer notification latency (seconds)

› No extra load› All on 3 nodes x 2 DC

› 200K subs — 1G heap

72

Achievements

› Low customer notification latency (seconds)

› No extra load› All on 3 nodes x 2 DC

› 200K subs — 1G heap

72

300K subscriptions in production

73

300K subscriptions stress testing (cluster)

74

150K subscriptions stress testing (node)

75

ContactsDmitry SchitininBackend infrastructure team @ Yandex.Classifieds

[email protected]

76