re-architecting with mongodb
TRANSCRIPT
RE-ARCHITECTING WITH MONGODB
FIXING A RELATIONAL MESS WITH A NON RELATIONAL DATABASE
ABOUT US
MICHAEL LUJANVP OF ENGINEERING @ VIXLET
BOBBY LINCOLNCOO @ VIXLET
A LITTLE ABOUT VIXLET
• LOCATED IN DOWNTOWN LA• A PASSION BASED SOCIAL NETWORK • SOCIAL NETWORKS FOR MLB, ATP,
LFC, SLIPKNOT, ETC
CAPSULES
• CAPSULES ARE USED TO REPRESENT TOPICS
• USERS AND BRANDS CAN POST INTO THESE CAPSULES
THE PULSE• THE PULSE IS WHERE YOU FIND
CONTENT FROM CAPSULES YOU HAVE SUBSCRIBED TO• SHOWS CAPSULE WHERE MEDIA IS
FROM• USER WHO POSTED CONTENT• WHETHER YOU HAVE LIKED THIS
CONTENT
MEDIA RECORDS
• ORIGINALLY DESIGNED BY DEVELOPERS WITH RDBMS EXPERIENCE
• FOREIGN KEYS JOINED AT RUNTIME IN CODE
{”_id”: ”
53d682bba374b38a686e9eeb”,“capsuleId”:
“53d681f9a374b38a686e9ee0”,“creator”:
“55b815ea54517d544e3f5bee”,“type”: “text”,“caption” “test”,“statuses”: {
“created”: 1445621494381,“updated“: 1465410655512
}}
ORIGINAL WORKFLOW
Feed MS Media MS Capsule MSoAuth MS
Capsule MS
Relationship MS
User MS
User MS
Relationship MS
AVERAGE RESPONSE TIMES
Call Response Time AveragesPulse 4000msCapsule Feed 2000msProfile Feed 2500ms
INSERT FUNNY PICTURE
TIME TO RE-ARCHITECT
• MICRO MODULE ARCHITECTURE• SINGLE API LAYER BUT BROKEN UP INTO BITE SIZE MODULES• REMOVES HTTP OVERHEAD OF SOME MICRO-SERVICE ARCHITECTURE• ALLOWS FOR SINGLE LAYER TO BE BROKE UP INTO MICRO-SERVICES• ALLOWS FOR EACH PIECE TO BE OPTIMIZED (INCLUDING DATABASES)
MEDIA CREATION
Service Layer
Create Relationship
sCreate Media
Record Add to feeds
Mongo Riak Riak
SIMPLIFY THE ARCHITECTURE
• WORKER ARCHITECTURE• CRUD IS HANDLED BY SERVICE LAYER• ALL NON CRITICAL WORK IS HANDED OFF TO WORKERS• MAIN DISPATCHER HANDLES ALL REQUESTS • IMPROVES SPEED AND SIMPLICITY
DISPATCHINATOR
• EVENT TRACKING• ERROR HANDLING / RETRY LOGIC
NEW MEDIA CREATION
Service Layer
Media Module
Dispatcher Queue
Create Relationship Neo4j
Mongo
CassandraAdd to Feeds
NEW MEDIA RECORDS
• COMPLETE CAPSULE AND CREATOR OBJECTS
• NO LONGER NEED TO JOIN ON FOREIGN KEYS
{ ”_id”: ” 53d682bba374b38a686e9eeb”, “capsule”: {
“id”: “53d681f9a374b38a686e9ee0”,“name”: “ATP Challenger Tour”,“cover”: “image.png”,“description”: “description”
}, “creator”: {
“id”: “55b815ea54517d544e3f5bee”,“username”: “atpworldtour”,“avatar”: “image.png”
}, “type”: “text”, “caption” “test”, “statuses”: {
“created”: 1445621494381,“updated“: 1465410655512
}}
CAPSULE / USER UPDATES
Service Layer Module Dispatcher
Queue
Update Media Mongo
Mongo
BUT THERE IS A PROBLEM
• WHAT HAPPENS WHEN THERE ARE 1,000 UPDATES? OR 1,000,000 UPDATES?
FAN OUT ON WRITE WORKER
• KEEP COUNT OF MEDIA RECORDS PER CAPSULE• BREAK UP WORK INTO MANAGEABLE SIZES
NEW UPDATE WORKFLOW
Service Layer Module Dispatcher
Queue
Fan Out Worker
MongoMongo Update
Media
NEW LOOK UP WORKFLOW
Service Layer
Feed Module
Relationship Module
NEW RESPONSE TIMES
Call Response Time AveragesPulse 200msCapsule Feed 150msProfile Feed 150ms
ANY QUESTIONS???