polyglot persistence - confoo 2016
TRANSCRIPT
PolyglotPersistence
Majid Fatemian
R E D V E N T U R E S
@majidfn
Relational Non-Relational Key-Value Big Data
Application
Relational
Application
Relational Relational Non-Relational
Relational
BIGDATA
Capture
Search Storage
Q u e r y
Privacy Visualize
Analysis
Non-Relational
1 2 3Non-Relational Cluster Schemaless
2 x
1 x
1 x
2011
2012
2013
2014
2011
2012
2013
2014
Application
ACID
Normalized{ConsistencyRelational
Player ID Map ID Difficulty Score1000 223 E 52000 673 H 4.51000 451 M 42000 980 H 3
Activity
Player ID Name Location1000 John US2000 Catherine CA3000 Paul CA4000 Kylee US
Profile
Replication
Fault ToleranceAvailability{Relational
OPs / DBAs Tools Devs{DevOps
Relational
2011
2012
2013
2014
ScalabilityRelational
DistributionRelational
Data CData B
Data A
Data Access Layer
SchemaRelational
{ player_id: 1234, * level: 40, * duration: 75, * activity: activity, game_id: 4001, platform: WiiU, difficulty_level: medium, repetition: 178, score: 450, endurance:3}
{ player_id: 1234, level: 12, duration: 120, activity: activity}
Total
Detailed
stats
stats
Players’ Profiles
Individual Activities
Aggregatedstats
Categorized
ACID Transaction Partitioning Replication
Tools OPs
Data Volume Fixed Schema Scale Up Manual Sharding
+ -
NoSQL...
mongoDB
1 2 3OPs Devs Community
Schema-lessNon-Relational
player_id: 1234, * level: 40, * duration: 75, * activity: activity, game_id: 4001, platform: WiiU, difficulty_level: 2, repetition: 178, score: 450, endurance:3
player_id: 1234, level: 75, duration: 120, activity: activity
ScalabilityNon-Relational
ScalabilityNon-Relational
+
ShardingNon-Relational
Data A
{key}
ShardingNon-Relational
Data A
A
Data B
B
Data C
C
{key}
Map/ReduceNon-Relational
QueryK, V
K,V1 K,V2 K,V3
Map/ReduceNon-Relational M
ap
K,V1 K,V2 K,V3
Map/ReduceNon-Relational
ResultReduce
Map
K,V1 K,V2 K,V3
DenormalizedNon-Relationalread / }write
Disk Space
Player ID Map ID Difficulty Score1000 223 E 52000 673 H 4.51000 451 M 42000 980 H 3
Activity
Player ID Name Location1000 John US2000 Catherine CA3000 Paul CA4000 Kylee US
Profile
{Name: John, game:1, activity: Mission01, Score: 1245}
{Name: John, game:3, activity: Mission04, score: 5431}
{Name: John, game:1, activity: Mission01, Score: 1245}
Data ModelNon-RelationalQuery
Aggregation}
Eventual- Consistency
Non-Relational
X2
W
X1 X1
X2
Eventual- Consistency
Non-Relational
X2
W
X1 X1
R
X1
X2
Eventual- Consistency
Non-Relational
X2
W
X2
X2
R
X2
X2
BalancerNon-Relational
BalancerNon-Relational+
BalancerNon-Relational
Non-RelationalResilience
A B C
Non-RelationalResilience
A C
LockingNon-RelationalDB Level (2.2)}Document Level (3.x)
BackupNon-RelationalPoint-in-time
Denormalized data Disk Space Expertise Complex QueryingEventual Consistency Resource Usage DB-Level Locking
-
Schema-Less Aggregated Data Large Scale Data
Sharding Map/Reduce
Memory Storage Journaling
+
Random ProfileSearch(1M)
Full History Retrieval(6M)
Insert(6M)
AggregationMap/Reduce
(6M)
0
0.003
0.005
0.008
0.01
MySQLMongoDB
PersistencePolyglot
Players’ Profile Leaderboards
Activity Details
Relational Data
Schema-less Sharding
Total Sum
Activity 1
Activity n
Activity m
……
Shard 1
Shard 2
Config
Data Access Layer
Application
TheChallenge
Inconsistency1
Data Access Layer
W
Pending
Commit2-Phase
Data Access Layer
Pending
W Commit2-Phase
Data Access Layer
Done
Commit2-Phase
2 Latencyvs. Consistency
Operations3 Exponential complexity
Availability4 Plan for failure
Data Access Layer
Data Access Layer
Data Access Layer
Data Integrity5Backward compatibility
Unit{TEST IntegrationFunctional
Data Migration6
Profile
History }{ profile: { GUID:…, platform: { name:’xbox’, xuid:… } }, activity:…, duration:…, datetime:… }
3rd Party
1 One-Shot Migration4 hours !!
Live !!2 Profile versioning
V+11
V+11
V
+1
2
FinalResults
Scalability
2011 2012 2013 2014
9x response time
3 years uptime
Go Hybrid!Thoughtfully
>
ThankYOU!@majidfn
References
Polyglot PersistenceNoSQL Distilled