reactive data system in practice
TRANSCRIPT
![Page 1: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/1.jpg)
Reactive Data System in Practice
Or how to solve Fast Data Problem and reacting to the World faster
By Triều Nguyễn at http://trieu.xyz http://www.rfxlab.com (Reactive Data System Lab)
λ(x)
![Page 2: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/2.jpg)
2008: Java Developer, develop Social Trading Network for a startup (Yopco)2011: joined FPT Online, software engineer, worked in FPT ID project and VnExpress Mobile (Backend)2012: backend engineer at Greengar (Scaling server and real-time services)12/2012 to 05/2015: back to FPT Online, lead engineer at eClick Ad Platform06/2015 to now: lead engineer at iTVad, a startup project at FPT for OTT and TV Advertising Solutions
about me
![Page 3: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/3.jpg)
1. Just some concepts 2. Ideas in practice3. What is RFX framework ?4. Why is RFX ?5. How to solve problems with RFX ?
a. Counting pageview of websiteb. Counting unique user of websitec. Querying data more reactive and adaptive
6. Final ideas
Contents of this talk
![Page 4: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/4.jpg)
1 - Just some concepts
![Page 5: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/5.jpg)
Fast Data vs Big Data
![Page 6: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/6.jpg)
What is Reactive Data System ?
is a software design approach that uses the principles of event driven architecture for the design and implementation of data intensive applications.
Event Driven ArchitectureEDA is a framework that promotes production, detection, consumption and reaction to the events. Event based applications will run on RDS framework.
![Page 7: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/7.jpg)
Why is Reactive Data SystemWe want to ask bigger questions faster
![Page 8: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/8.jpg)
2 - Ideas in practice
![Page 9: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/9.jpg)
Problem (User Story in Practice)
● User does login with FB Connect at Ecommerce website
● User does a click on an item, then system should do the following tasks:1. Find best items to recommend 2. Find a best item to send email marketing3. If there are more than 100 users, that click on a item,
run a marketing campaign on Facebook automatically.
![Page 10: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/10.jpg)
An example from E-Commerce
![Page 11: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/11.jpg)
3 - What is RFX framework ?
![Page 12: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/12.jpg)
● A framework for big fast data problems● A collection of Open Source Tools from Netty to Spark● The mission of RFX
1. Build data product quickly with design patterns2. React to critical events in real-time
What is RFX or Reactive Function X ?
![Page 13: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/13.jpg)
RFX is inspired by the Actor Model Theory
![Page 14: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/14.jpg)
Paper: http://vialab.science.uoit.ca/textvis2011/papers/textvis%202011-rohrdantz.pdf
![Page 15: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/15.jpg)
RFX is designed for startup business
I design RFX because I have to solve following problems both at FPT and at RFXLab.com● Marketing Automation● Product UX personalization● Real-time Advertising● System monitoring (frontend and backend)● Email marketing● Product Analytics● Customer Analytics
→ So I build a full-stack framework (a collection of tools for both backend and frontend)
![Page 16: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/16.jpg)
Philosophy of RFX
![Page 17: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/17.jpg)
Sub modules in RFX framework
![Page 18: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/18.jpg)
4 - Why is RFX framework ?
![Page 19: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/19.jpg)
Why Rfx ?
● Ideas since 2012 (from Actor model theory)● R&D and Deployed in Production since 2013
● Open Source: Apache License, Version 2.0 ● Full Stack: from Frontend to Backend● Agile for Data Analytics● Lightweight Lambda Architecture
● Really fast and near-real-time processing● Tested with 1.000.000 events / second● Simple development model for data engineer
![Page 20: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/20.jpg)
Roadmap to the future
![Page 21: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/21.jpg)
5 - How to solve problems with RFX ?
![Page 22: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/22.jpg)
ProblemsUser story in Digital Marketing: 1. Counting pageview of website2. Counting unique user of website3. Real-time marketing
![Page 23: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/23.jpg)
Find rush hour in your system
![Page 24: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/24.jpg)
Which device user reach our content
![Page 25: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/25.jpg)
Solution Architecture
Tracking(RFX-track)
Event Queue (Apache Kafka)
Event Processor(RFX-stream)
RedisMongoDB Synch Data Jobs(RFX-job)
Report(RFX-report) Event Actor
(RFX-actor)Ad Server
![Page 26: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/26.jpg)
![Page 27: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/27.jpg)
An example topology
![Page 28: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/28.jpg)
6 - Final ideas
![Page 29: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/29.jpg)
Summary
● Fast Data is cool● Fast data: The next step after big data● RFX is designed to solve common Fast Data
problems in Digital Marketing and Advertising
![Page 31: Reactive Data System in Practice](https://reader035.vdocuments.net/reader035/viewer/2022062302/5870597e1a28aba2118b6243/html5/thumbnails/31.jpg)
The end and thank youhttps://github.com/rfxlab
http://rfxlab.com
λ