Lessons Learnt.An In Production,

Scaleable Django StackPyCon ireland - Dublin, Oct 13th-14th 2012

john.moylan@rte.ie / john@8t8.eu / @redsnapper88

Brief Intro to RTE.ie

Website of Ireland's national broadcaster

Problem

Mostly W.O.R.M, but some stuff can't be cached :(Lot's of content(100's GB's of HTML)10's TB of Audio and Video

Lots of strands (News, Sport, Business, Ent., TV, Radio, Teletext, Archives, Young Peoples, Performing Groups...)

Small team.Try to keep things as simple and efficient as possible.

...massive growth

Traffic Growth

Oct 2010 V's Oct 2011Traffic nearly doubled.

Flashmob Traffic

36Gb/s of Traffic. 41K Concurrent streams.Playlists were not cacheable........

But what a great fight.

Static?

Legacy system published to static files using XML/XSL Publisher.

Multiple versions per type of view.

SSI's inside SSI's inside SSI's

Republishing a strand could take days.

Dynamic Requirement

Evaluated some systems:

Wicket, Rails, Shale....Django

Hmmm... nice

All in package, sensible defaults. Similar problem domain.

Try Before you buy

2008 Wrote prototype video upload sharing site... (Django 0.96) and then decided to make it live.

http://web.archive.org/web/20090318064215/http://www.rte.ie/tv/theden/ugc/red/video/

Try Before You Buy

2009 Original RTÉ Player - Written in Django.

2011Let's use Django to make the whole site dynamic!

..continued

2012Sport, About, RTÉ Player v.2, Radio, News?, Archives, playlists, lifestyle, SSO......and on and on

2013..More

Django Delivery

LinuxApache with Event MPM and mod_wsgiMysql and MHA (tip Innodb plugin)Solr for search and 'NoSQL'Varnish (used to use Squid)LVS-NAT and LVS-DR Load-balancers.MemcacheDeploying Glusterfs

Code Management

GitRedmineGitosis

Deployment

CobblerFabricBuildout

Monitoring

Monitor & Measure everything!ZenossRRDToolMRTGGraphiteStatsd - mozilla's Django-statsdSentryLot's of custom monitoring

In House Monitoring projects

Pyvarnishhttps://github.com/redsnapper8t8/pyvarnish

In House Monitoring Projects

Digital Dash - (open sourced soon)

Benchmarking

Tool agnostic but seem to use these alot.

Bonnie++Netperf

#HTTPHttperfSiege

Performance Tips

1. Memory, memory, memory2. Caching - in memory3. Does it scale horizontally?4. KISS

Lessons Learnt.

Read from memory whenever possibleCache, learned this one a long time ago but whenever we have issues it can usually be tracked down bad design and lack of caching. Don't hit your web servers unless you have to.

Memcache, Varnish

Long live Rest - GET's should not have side effects. Varnish

KISS - and no resume driven development.

Thank you

Questions?

Also, did I mention that we are hiring?