Потоковая фильтрация событий
Embed Size (px)
TRANSCRIPT
Streaming event matching
Dmitry Schitinin
About
Backend infrastructure team
› Services› Libraries› Frameworks
@ Yandex.Classifieds
› auto.ru
› auto.yandex.ru
› rabota.yandex.ru
› realty.yandex.ru
› travel.yandex.ru
https://stat.yandex.ru 2
About
Backend infrastructure team
› Services› Libraries› Frameworks
@ Yandex.Classifieds
› auto.ru
› auto.yandex.ru
› rabota.yandex.ru
› realty.yandex.ru
› travel.yandex.ru
https://stat.yandex.ru 2
About
Backend infrastructure team
› Services› Libraries› Frameworks
@ Yandex.Classifieds
› auto.ru
› auto.yandex.ru
› rabota.yandex.ru
› realty.yandex.ru
› travel.yandex.ruhttps://stat.yandex.ru 2
Introduction
Audience
› Software engineers/architects
› Event stream processing
› Distributed systems
4
Features
› Stream of offers (tens per second)
› Up to 500K subscriptions
› Complex multi field conditions
› Low latency (seconds)
8
Features
› Stream of offers (tens per second)
› Up to 500K subscriptions
› Complex multi field conditions
› Low latency (seconds)
8
Features
› Stream of offers (tens per second)
› Up to 500K subscriptions
› Complex multi field conditions
› Low latency (seconds)
8
Features
› Stream of offers (tens per second)
› Up to 500K subscriptions
› Complex multi field conditions
› Low latency (seconds)
8
Features
› Stream of offers (tens per second)
› Up to 500K subscriptions
› Complex multi field conditions
› Low latency (seconds)
8
Motivation
Customer subscriptions
› Short notification delays (minutes)
› Instant subscription modifications› Notifications transport
› email, SMS, push, pigeons, owls, etc.
10
Customer subscriptions
› Short notification delays (minutes)
› Instant subscription modifications› Notifications transport
› email, SMS, push, pigeons, owls, etc.
10
Customer subscriptions
› Short notification delays (minutes)
› Instant subscription modifications
› Notifications transport
› email, SMS, push, pigeons, owls, etc.
10
Customer subscriptions
› Short notification delays (minutes)
› Instant subscription modifications› Notifications transport
› email, SMS, push, pigeons, owls, etc.
10
Non-functional requirements
› Scalability by subscriptions› millions
› Scalability by events› tens-hundreds per second
› Fault tolerance› nodes, network, data centers
11
Non-functional requirements
› Scalability by subscriptions› millions
› Scalability by events› tens-hundreds per second
› Fault tolerance› nodes, network, data centers
11
Non-functional requirements
› Scalability by subscriptions› millions
› Scalability by events› tens-hundreds per second
› Fault tolerance› nodes, network, data centers
11
Non-functional requirements
› Scalability by subscriptions› millions
› Scalability by events› tens-hundreds per second
› Fault tolerance› nodes, network, data centers
11
Naive solution
Pros:› Simplicity
› Working
Cons:› High notificationlatency (hours, days)
› Search systemextra load
› Scalability?
19
Naive solution
Pros:› Simplicity
› Working
Cons:
› High notificationlatency (hours, days)
› Search systemextra load
› Scalability?
19
Naive solution
Pros:› Simplicity
› Working
Cons:› High notificationlatency (hours, days)
› Search systemextra load
› Scalability?
19
Naive solution
Pros:› Simplicity
› Working
Cons:› High notificationlatency (hours, days)
› Search systemextra load
› Scalability?
19
Naive solution
Pros:› Simplicity
› Working
Cons:› High notificationlatency (hours, days)
› Search systemextra load
› Scalability?19
Not so naive approach
Pros:› Low latency
› No extra work› Possibly scalable
Cons:› More complex, butnot complicated
22
Not so naive approach
Pros:
› Low latency
› No extra work› Possibly scalable
Cons:› More complex, butnot complicated
22
Not so naive approach
Pros:› Low latency
› No extra work› Possibly scalable
Cons:› More complex, butnot complicated
22
Not so naive approach
Pros:› Low latency
› No extra work
› Possibly scalable
Cons:› More complex, butnot complicated
22
Not so naive approach
Pros:› Low latency
› No extra work› Possibly scalable
Cons:› More complex, butnot complicated
22
Not so naive approach
Pros:› Low latency
› No extra work› Possibly scalable
Cons:
› More complex, butnot complicated
22
Not so naive approach
Pros:› Low latency
› No extra work› Possibly scalable
Cons:› More complex
, butnot complicated
22
Not so naive approach
Pros:› Low latency
› No extra work› Possibly scalable
Cons:› More complex, butnot complicated
22
Matching
Data model
› Event:
› Subscription:mark=FORD&model=FOCUS&model=MONDEO&year=2015
25
A better way
Commonalities:› mark=FORD&model=FOCUS
› mark=FORD&year=2010
› year=2010&price_from=500000
Data structures› BDD
› DAG
› Search tree
› Indices
27
A better way
Commonalities:› mark=FORD&model=FOCUS
› mark=FORD&year=2010
› year=2010&price_from=500000
Data structures› BDD
› DAG
› Search tree
› Indices
27
A better way
Commonalities:› mark=FORD&model=FOCUS
› mark=FORD&year=2010
› year=2010&price_from=500000
Data structures› BDD
› DAG
› Search tree
› Indices
27
Architecture
Architecture
Matcher
Resilience
› Subscriptions => Tokens› Token ≈responsibility
› Nodes distribute tokens
51
Resilience
› Subscriptions => Tokens
› Token ≈responsibility
› Nodes distribute tokens
51
Resilience
› Subscriptions => Tokens› Token ≈responsibility
› Nodes distribute tokens
51
Resilience
› Subscriptions => Tokens› Token ≈responsibility
› Nodes distribute tokens
51
Architecture
Notifier
Implementation
Scala
› Functional & concise› Persistent data structures for matching tree(1Kloc)
› No need for synchronization
› Property-based testing (matching algorithm)
1http://www.scalacheck.org
68
Scala
› Functional & concise
› Persistent data structures for matching tree(1Kloc)
› No need for synchronization
› Property-based testing (matching algorithm)
1http://www.scalacheck.org
68
Scala
› Functional & concise› Persistent data structures for matching tree(1Kloc)
› No need for synchronization
› Property-based testing (matching algorithm)
1http://www.scalacheck.org
68
Scala
› Functional & concise› Persistent data structures for matching tree(1Kloc)
› No need for synchronization
› Property-based testing (matching algorithm)
1http://www.scalacheck.org
68
Scala
› Functional & concise› Persistent data structures for matching tree(1Kloc)
› No need for synchronization
› Property-based testing (matching algorithm)
1http://www.scalacheck.org
68
Akka
› actors (events)
› akka-remoting (networking)
› protobuf (serialization)
› akka-fsm (aggregation)
69
Akka
› actors (events)
› akka-remoting (networking)
› protobuf (serialization)
› akka-fsm (aggregation)
69
Akka
› actors (events)
› akka-remoting (networking)
› protobuf (serialization)
› akka-fsm (aggregation)
69
Akka
› actors (events)
› akka-remoting (networking)
› protobuf (serialization)
› akka-fsm (aggregation)
69
Akka
› actors (events)
› akka-remoting (networking)
› protobuf (serialization)
› akka-fsm (aggregation)
69
Zookeeper
› Exclusive tokenacquirement
› Service discovery
http://zookeeper.apache.org 70
Conclusion
Achievements
› Low customer notification latency (seconds)
› No extra load› All on 3 nodes x 2 DC
› 200K subs — 1G heap
72
Achievements
› Low customer notification latency (seconds)
› No extra load› All on 3 nodes x 2 DC
› 200K subs — 1G heap
72
300K subscriptions in production
73
300K subscriptions stress testing (cluster)
74
150K subscriptions stress testing (node)
75
