implementing analytics in high-traffic social games
DESCRIPTION
Setting up a fail-proof analytics process that delivers information where and when they are needed comes with plenty of challenges in high traffic social gaming. Social Point CTO, Marc Canaleta, explains in this Slideshare how Social Point has found a solution that is economic and highly scalable, with high availability.TRANSCRIPT
Social Point Analytics in AWSMarc Canaleta (CTO)
@mcanaletaAWS Summit Barcelona 2013
Social PointSocial Games developer for Mobile & Facebook
Founded in 2008, offices in Barcelona(22@) and San Francisco
Top #20 mobile grossing games worldwide
Top #3 Facebook developer
Games
Social Games: interaction between friends; virality
Freemium model: Play for free with in-app purchases
Midcore
Leader in Breeding & Collecting strategy games
Dragon City
Top 20 Grossing in iOS App Store worldwide
Recently launched for Android, featured on Google Play
6M DAU on Facebook
Why AWS?
No hardware maintenance or planning: business benefits from increased speed
Flexible: Pay for use
Facilitates scalability:Auto Scaling
Facilitates high availability: multiple availability zones
Managed components: Load Balancers, Databases …
Analytics Driven. Vital for almost every team.
Engineers: realtime analytics, monitoring, detecting problems
Product: taking decisions, A/B testing, game balancing
Marketing: optimizing campaigns
Finance: monitoring the business
Social Point Analytics
ANALYTICS QUEUES
BACKEND SERVERS
Architecture
BACKEND SERVERS
FLASH CLIENT IOS CLIENT ANDROIDCLIENT
ANALYTICS QUEUES ANALYTICS QUEUES
LOGFILES STORAGE
ANALYTICS DATABASE
GAM
E SY
NCH
RON
OU
S PA
RTAN
ALYT
ICS
ASYN
CHRO
NO
US
PART
BACKEND SERVERS Symfony 2
Redis
AWS S3
AWS Redshift
REDIS
Events generation Backend writes events in Redis lists
Why Redis? Cost and Performance: 10K events/second/server
Problem: it’s a memory-based database; queues have to be constantly consumed
Scaled and HA: randomly distributed N servers
BACKEND
REDISREDIS
Python processes continuously consume queues and:
Calculate Real Time metrics
Store event logfiles to upload to S3
Enqueue S3 object URL to SQS
Events Processing
Consumer
Redis Queue
LPOP event
Event Log File
Amazon S3
write event
put object
LOAD DATA
GENERATION Of EVENTS
RedisReal Time
INCR counter
Amazon SQS
enqueue S3 object URL
Events ProcessingWhy these technologies?
Python is well suited to developing workers and dealing with data
Redis: structures like counters, sets, sorted sets for Real Time metrics
S3: virtually infinite space, scalable, high availability
SQS reliability and availability at a higher cost than Redis
Consumer
LPOP event
RedisReal Time
INCR counter
Event Log File
Amazon S3
write event
put object
Amazon SQS
enqueue S3 object URL
LOAD DATA
Redis Queue
GENERATION OF EVENTS
Data loading
Amazon S3Amazon SQS
Importer
TSV
RedShift
A process called importer:
Reads SQS URLs
Downloads S3 logfiles
It converts them to TSV
And imports multiple logfiles to Redshift at the same time
EVENT PROCESSING
Allows flexibility-> schema changes with no downtime
Highly scalable (but you cannot write while scaling)
Low implementation risk It is an offline system. Gameplay is unaffected by downtime. We have backups. In the worst case, we restore a backup and reload.
Minimum maintenance: only vacuums, space monitoring
Good SQL support, unlike other columnar data bases
Database: Redshift
Daily transformations and calculations implemented in SQL
Example:
UPDATE USER SET total_revenues = (SELECT SUM(amount) FROM transaction t WHERE t.user_id = user.user_id);
Why not hadoop? Much slower and complex; for now SQL operations meet all of our
needs. In Redshift these SQL operations are very fast.
Transformations, calculations
¿Would you like to work in the video gaming sector?
Talent attracts talent. You have the talent, we have the playground.
www.socialpoint.es/jobs
¡Thank you!