"real time streaming of big data", shimon tolts, general manager, data solutions at...
TRANSCRIPT
Shimon Tolts General Manager, Data Solutions
ironSource Atom
Data Flow Management
About ironSource: ironSource in Numbers
700Employees
Sep. 2010
Established
50%R&D Employees
700Advertisers
80KPartnered Apps
100MDevices using ironSource
solutions shipping in 2016
TEL AVIV ISRAEL
SAN FRANCISCO UNITED STATES
NEW YORK UNITED STATES
LONDON UNITED KINGDOM
BANGALORE INDIA
HONG KONG CHINA
KIEV UKRAINE
BEIJING CHINA SHANGHAI CHINA
About ironSource: ironSource Hypergrowth
People Reached Each Month
4200Apps Installed Every Minutewith the ironSource Platform
Registered & Analyzed Data EventsEvery Month
160B
800M
50B
0
100B
150B
200B
Jun 201
5
Jul 201
5
Aug 201
5
Sep 201
5
Oct 201
5
Nov 201
5
Dec 201
5
Jan 201
6
Feb 201
6
Mar 201
6
Apr 201
6
May 201
6
We needed a way to manage this data:
Our Business Challenge
ProcessCollect Store
Collection
● Multi region layer - Latency based
routing
● Low latency from client to Atom servers
● High Availability - AWS regions does
fail!
● Storing raw data + headers upon
receiving
Data Enrichment● Enrich data before storing in your Data
Lake and/or Warehouse○ IP to Country○ Currency conversion ○ Decrypt data○ User Agent parsing - OS, Browser, Device...
● Any custom logic you would like! - fully extendible
Data Targets● Near real-time data insertion - 1
minute!● Stream data to Google Storage and/or
AWS S3● Smart insertion of data into AWS
Redshift○ Set the amount of parallel copys○ Configure priority on tables
● BigQuery - Streaming data using batch files import (saves 20% cost)
Micro-Services Architecture● Everything is a service● Decoupling● Distributed systems
Separate lifecycle● Communication using RESTful /
Queue / Streams
Docker● Linux Container● Save provisioning time● Infrastructure as code● Dev-Test-Production - identical
container● Ship easily
Cloud infrastructure● Pay as you go - (grow)● SaaS services ● Auto-scaling-groups● DynamoDB● RDS *SQL● Redshift data warehouse
Continuous Integration● From commit to production● Jenkins commit hook● Git branching model● AWS dynamic slaves● Unit tests● Docker builds● Updating live environment
Diagram
Partners
Everybody needs a data pipeline
The AWS platform allowed us to build on top of it with an infrastructure that is exactly tailored to our clients’ needs.
Maximum FlexibilityAny data, from any source,
in any format.
Infinite ScalabilityAdapt to your evolving
needs with a pay-as-you-go model.
Own Your DataWe manage the flow,
the data is yours.
David FitcherFrom London, United Kingdom
The User JourneyTouchpoint #1
Customizing anew device
David bought a new LG device.Installed 12 apps overall, 6 were games.
Gender: Male76% Probability
Age: 25-3595% Probability
User Profile: Casual Gamer81% Probability
Touchpoint #2
Using a mobile app
David is now playing a game which uses our SDK.What we already knowProbably a ‘Casual Gamer’What we offerRewarded video ads for casual gaming apps
User Profile: Casual Gamer95% Probability
Subcategory Interest: Simulation GamesLTV Projection: High
One month later…
What we know now
