abdw17-lightning talks track-anatomy of modern, data-driven applications
TRANSCRIPT
©2017 Ampool, Inc. 1Prepared for: XXX
Anatomy of Modern Data-Driven ApplicationsApril 2017Nitin Lamba & Suhas Gogate
©2017 Ampool, Inc. 2Prepared for: XXX
Today, apps are driving customer experiences!
! " |Immediate Fulfillment
Anywhere, Real-time
Powered by Analytics
Ongoing Value
∞
©2017 Ampool, Inc. 3Prepared for: XXX
…and generating lots of data!
$ % &
TransactionsPoints
User actionsWorkflow
LocationSocial
Financial Data
Behavioral Data
Contextual Data
©2017 Ampool, Inc. 4Prepared for: XXX
Apps
Do apps get value from analytics?
Streaming;Batch processing/machine learning;
Interactive querying
Flexibility in data formats;Fast ingestion needs
Experiences/ insightsat scale and speed
Persistence
timely
©2017 Ampool, Inc. 5Prepared for: XXX
The memory shift is already happening!
SSD3,000 MB/s
$2 – 6/ GiB
HDD100 MB/s
$0.03 - 0.2/ GiB
SCM10,000 MB/s
$3 – 5/ GiB
DRAM60,000 MB/s
$6 – 10/ GiB
©2017 Ampool, Inc. 6Prepared for: XXX
There’s a need for a unified, performant,memory-based data store…
ACTIVE
DATA STORE
Manage hot data in-memory
Process where data is stored
Primary store; not a cache!
Highest ACTIONABLE
Value
Highest WORKLOAD
Share
©2017 Ampool, Inc. 7Prepared for: XXX
Apps
… that changes the game, and captures value, NOW!
Use ADS for all data processing needs
1. Store ALL active data & update in real-time
2. Analyze through ‘best-of-breed’ compute engines & frameworks
3. Serve data concurrently to multiple data processing stages, tenants & applications
4. Tier inactive, immutable data to long-term persistence
Long-termPersistence
©2017 Ampool, Inc. 8Prepared for: XXX
Driving ‘value creating’ experiences here and now
Users
View ContentHave InterestsServed Ads
Publishers
Add ContentUpdate Interests
ACME CorpServe Ads/ ContentCapture EventsServe analytics
Ad Server Infra.
Ad Ad Ad …
Ad Ad Ad …
Advertisers
Create AdsUpdate InterestsAd Performance
Data/ Admin
Monitor data flow service & infrastructure
Events
©2017 Ampool, Inc. 9Prepared for: XXX
Serve all data access patterns from a single in-memory store
Active Data Store
( REST)
Stream ProcessingEnrichment, convertorsAggregates & computations
Machine LearningShort-term behavioral modelMost-recent user interests
Interactive QueriesTop ads, campaigns,
product(s), …Joins w/ other user data
Searches productShows intent to buyServed promotions
User on Device
* +
User session infoUser profile updates
Data Ingestion
Recent interestsReal-time
recommendations
Alert/ Show
©2017 Ampool, Inc. 10Prepared for: XXX
Let’s dive into the streaming details…
KafkaAd Server
Use
rs
Ads
Con
tent
Ad ServerLogs
U-C-A Updates
Ad Server Logs
Micro-batch ProcessingContent-Ad
Apex
Ad Server
Users
Ads
Content
Kin Ad Server
Kin Users
Kin Ads
Kin Content
Mout Ad LogMout Users Mout Ads Mout Content
User-Ad Top Events
StrCounter
TopCamps TopAds
SimplificationConvertor, Enrichment, Dimension Computation
Pluggable AnalyticsConvertor, Enrichment, …
Analytics of interestMout Camp Mout Ad