jesper richter-reichhelm - continuous evolution at wooga - code.talks 2015
TRANSCRIPT
DevOps (just 2 devs involved)
Daily iterations (sometimes faster)
Monster World
Fixes
Optimizations
Bigger DBs
More DBs
app
lb
Browser
SQL
sql
appapp
lb
app appappapp appapp
SQL
sql
SQL SQL
sql sql
SQL SQL
sql sql
SQL SQL
sql sql
SQL SQL
sql sql
SQL
sql
SQL
sql
SQL
sql
SQL
sql
SQL
sql
SQL
sql
lb
Stateless approach
1+ million users / day
10+ billion requests / month
~10 DB writes / request
> 100,000 DB writes / second
http://www.flickr.com/photos/biblicone/3425903181/sizes/l/in/photostream/
Added Redis (to MySQL)
Migrating users (on demand)
Monster World
Tests worked fine (using their code)
Live it worked badly (50K users corrupted)
Fixed it quickly (and repaired the users)
SQL SQL
sql sql
SQL SQL
sql sql
SQL SQL
sql
Rd Rd
rd rd
Rd Rd
rd rd
Rd Rd
rd rd
Rd Rd
rd rd
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
SQL
EBS EBS EBS EBS
lb lblblb
SQL SQL
sql sql
SQL SQL
sql sql
SQL SQL
sql
Rd Rd
rd rd
Rd Rd
rd rd
Rd Rd
rd rd
Rd Rd
rd rd
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
SQL
EBS EBS EBS EBS
lb lblblb
38 single points of failure
Boyd Cycle
Col. John Boyd, USA
Korea war: F-86 won 10:1 fight against MiG-15
Time competitive cycles of action
Effects of faster decisions are cumulative!
OODAIndependent teams (autonomous decisions)
Experiments (ab tests)
Agile approach (Scrum, Kanban etc.)
Quick iterations (4h, 1d, 7d, 14d)
OODA loop
Stateful approach
1+ million users / day
4 sessions / user / day
1 DB write / session
~50 DB writes / second
Server
sessionsessionsessionsession
Server
sessionsessionsessionsession
Server
sessionsessionsessionsession
S3
flickr.com/groups/bestfavoriterelayrace/
Rotating jobs
Being transparent
flickr.com/photos/marcomagrini/698692268
offline
DD
Erlang
statefulML
2011
Redis
Ruby
Riak
Javascript
ev. Ruby
1 doc
LAMP
statelessBB BI
MW
HH
MW 2
PI
PP
MWM
2012
Erlang
Redis
Ruby
stateful
Riak
Javascript
offline
ev. Ruby
1 doc
JRuby
shared
LAMP
multiplay
statelessBB BI
MW
HH
MW 2
ML
DD
PI
PP
MWM PV
GX
FT
KB
BI 2
2013
Erlang
Redis
Ruby
stateful
Riak
Javascriptrealtime
offline
ev. Ruby
1 doc
JRuby Scala
shared
LAMP
generic
multiplay
stateless
services
BB BI
MW
HH
MW 2
ML
DD
PI
PP
MWM PV
SBS
DYN
GX
FT
KB
BI 2
DIV
MA
2014
Erlang
Redis
Ruby
stateful
Riak
Javascriptrealtime
offline
ev. Ruby
1 doc
JRuby Scala
shared
LAMP
generic
multiplay
stateless
services
BB BI
MW
HH
MW 2
ML
DD
PI
PP
MWM PV
SBS
DYN
GX
FT
KB
BI 2
DIV
MA …
Erlang
Redis
Ruby
stateful
Riak
Javascriptrealtime
offline
ev. Ruby
1 doc
JRuby Scala
shared
Go
Elixir
hybrid
LAMP
generic
multiplay
stateless
services
BB BI
MW
HH
MW 2
ML
DD
PI
PP
MWM PV
SBS
DYN
GX
FT
KB
BI 2
DIV
MA …
…
…
…2015
Erlang
Redis
Ruby
stateful
Riak
Javascriptrealtime
offline
ev. Ruby
1 doc
JRuby Scala
shared
Go
Elixir
hybrid
LAMP
generic
multiplay
stateless
services2015
MySQLPHP
PHP MySQL
Erlang
RedisRuby stateful
Riak
Javascript
realtime
offline
ev. Ruby
1 doc
JRuby
Scala
shared
Go
Elixir
hybrid
generic
multiplay
stateless
services
Languages DBs State mngt. Misc.
PHP MySQL
Erlang
RedisRuby stateful
Riak
Javascript
realtime
offline
ev. Ruby
1 doc
JRuby
Scala
shared
Go
Elixir
hybrid
generic
multiplay
stateless
services
Languages DBs State mngt. Misc.
Wooga’stoolbox
DD
PI
PP
MWM PV
JS
GX
BI 2
DIV
HTML5
Obj. C
Apportable
ext. Android
BB BI
MW
HH
MW 2
ML
Switch to Mobile
DD
PI
PP
MWM PV
JS
GX
BI 2
DIV
HTML5
Obj. C
Apportable
ext. Android
SD / DYN
… …
…
…
…
AAUnity
BB BI
MW
HH
MW 2
ML
Switch to Mobile
Wooga benefits
The ability to quickly adapt to change Survive disruptions and benefit from them
Resiliency and robustness Long-term perspective vs. short-term goals
To attract strong people We give transparency and trust, we want impact
Reinventing the wheel?
Knowledge sharing is the key
Transparency is mandatory
Reusing
code / SDKs
services
best practices
Erlang
Redis
Ruby
stateful
Riak
Javascriptrealtime
offline
ev. Ruby
1 doc
JRuby Scala
shared
Go
Elixir
hybrid
LAMP
generic
multiplay
stateless
services
BB BI
MW
HH
MW 2
ML
DD
PI
PP
MWM PV
SBS
DYN
GX
FT
KB
BI 2
DIV
MA …
…
…
…SBS
Erlang
Redis
Ruby
stateful
Riak
Javascriptrealtime
offline
ev. Ruby
1 doc
JRuby Scala
shared
Go
Elixir
hybrid
LAMP
generic
multiplay
stateless
services
BB BI
MW
HH
MW 2
ML
DD
PI
PP
MWM PV
SBS
DYN
GX
FT
KB
BI 2
DIV
MA …
…
…
…SBS
Flexibility Innovation
Individual Impact
Efficiency Planning
Scale benefits
Wooga
Autonomy Centralisation
Autonomy Centralisation
Flexibility Innovation
Individual Impact
Efficiency Planning
Scale benefits
Wooga
What’s right for your company?