8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

1/22

Lessons learned building a Web 2.0Application using MySQL

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

2/22

Who are we?

Were missing a vowl (must be web2.0!)

7 full-time employeesSeveral ex-Andover.net (Slashdot)

guys including our CEO

Boston based but distributed devBeen in existence about 2 years

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

3/22

What do we do?

Were all about

Create and share reading listsMerge and filter feedsPublish widgetsFuture: Do more advanced stuff

with feeds

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

4/22

Why Grazr

James Corbett Eirepreneurhttp://eirepreneur.blogs.com

January 2006

Im actually coming to the conclusion thatthe whole subscriptions mindset is a problemand that in future well graze for the most

part instead of subscribing. As Zigbeesensors, RFID chips and GPS trackersproliferate well be drowning in an RSS-everywhere world if we dont change ourapproach.

http://eirepreneur.blogs.com/http://eirepreneur.blogs.com/

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

5/22

http://grazr.com

http://grazr.com/http://grazr.com/

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

6/22

The heart of oursystem:

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

7/22

Lots of Mistakes

Warning: Some of these lessonswill seem obvious

Hindsight is 20/20We *did* have reasons for many

of these at the time (good ones?)

Tried a lot of experimentalconfigurations

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

8/22

Lesson 1: Beware

arch. momentumEarly system decisions affectedlater architecture, even whenabandoned

Careful about exotic w/out goodreasondb read on our feed processing

web boxes

The traditional setup, splittingMySQL from Apache = system

much happier

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

9/22

Lesson 2: ScalingDont worry about scaling - 37signalsYou must build for scaling or die!

- FriendsterThe truth? Somewhere in the

middle

Understand growth and scalingpatterns but dont build up frontYour scaling plan: wrong in

some way

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

10/22

Overemphasis on

scalingTwo hosted data centers + onetest/backup data center

Geographically separatedOver provisioned

18 servers, architecture mirroredTraffic could have been servedfrom two or three machines

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

11/22

Skynet Jr.

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

12/22

Lesson 3: Limits of

TestingStartup reality: not enough time for

thorough testing

Replication testing and simulationSpeed good enough, even cross

country

Problem: real world systembehaved differently

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

13/22

esson :Replication is fast,

until it isntWe knew better: but empiricaltesting seemed OK

Asynchronous nature of repl.sometimes hard to code, carefulwith state

Some of our code treated it as

synchronous (Fail!)

Smarter code was slow (retries,polls)

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

14/22

esson :Memcached is

your friendExcellent tool in the scalingtoolbox

Classic cache: limiting touchingyour database is good!Added benefit: on top of repl.

synchronizer

Good temporary storage for async.proc.

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

15/22

Lesson 6: Bulk

insert / lazy writeObvious: If you dont need it now,do it later

Disconnected / async good inthese cases

If you can do it later, glom many

together (bulk)

Much better write perf.

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

16/22

esson : serexperience vs.

ScalingEmphasis on scaling hurt userexperience

Characterize lazy vs. user affectingtransactionsFast, data correct transactions =

single data store (no read/write iosplit) or a sync buffer(memcached)

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

17/22

Lesson 8:

InstrumentationVisibility into system goodMore data = betterNon live testing != realityScaling is iterative process,

requires feedback loopStill somewhere were seriously

lacking (but working on it!)

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

18/22

Lesson 9: Try new

thingsBest practices are good, but new

ideas sometimes better!

Memcached as write/bulk buffer,good results

UDF, clever replication uses,triggers, virtual servers,background async. daemons

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

19/22

esson :Everyone has thesame problems

If you can, re-use!

Obvious: MySQL, Apache (not re-writing dbs, webservers)Less obvious: building batch job

processor, others have done thisbetter! (Gearman)

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

20/22

Lesson 11: Accept

changeArchitecture constantly in fluxThink ahead, but dont overthink,

design for now + 1, not now + 100

Accept change, react, constantlyre-evaluate

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

21/22

Lesson 12: Listen

Even amongst team, easy to bedogmatic

Listen to everyones ideas evenwhen it challenges your expertise

Some of our best ideas: thecombination of several approaches

8/14/2019 Lessons Learned Building a Web 2.0 Application Using MySQL Presentation

22/22

Questions?Discussion?