cto at numberly gentoo linux developer · designing a scalable and distributed application...
TRANSCRIPT
![Page 1: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/1.jpg)
Gentoo Linux developerCTO at Numberly
@ultrabug
© Eric Fischer / Flickr
![Page 2: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/2.jpg)
Designing a scalable and distributed application
EuroPython 2015
© Eric Fischer / Flickr
![Page 3: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/3.jpg)
USA ⇄ EUROPE
Geo distributed page hit counter web application
© Highways England / Flickr
![Page 4: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/4.jpg)
spans over multiple and distant datacenters
#1Multi Datacenter
© Catherine Holmes
![Page 5: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/5.jpg)
sum of the page hits from all the datacenters
#2A unique page hit count
© Catherine Holmes
![Page 6: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/6.jpg)
no manual application (re)configuration
#3Automatic resizing
© Catherine Holmes
![Page 7: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/7.jpg)
immediately available to all web services
#4Distributed configuration
© Catherine Holmes
![Page 8: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/8.jpg)
...that’s some kind of contract !
Hell...
© C. K. Koay / Flickr
![Page 9: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/9.jpg)
be lazy !
Rely on your stack
© Tripp / Flickr
![Page 10: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/10.jpg)
Zen of pythonif the technology is hard to explain,
it's a bad idea
© HumanOS
![Page 11: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/11.jpg)
UNIX tools philosophykeep it small and simple
© Jason Scott / Flickr
![Page 12: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/12.jpg)
Isolated componentsbreaking it down to pieces
© Blake Thornberry / Flickr
![Page 13: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/13.jpg)
distributed architecture style
Microservices
© Martin Grandjean / wikipedia
![Page 14: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/14.jpg)
micromanagement = high operational cost
Microservices
© Martin Grandjean / wikipedia
![Page 15: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/15.jpg)
Communication reliabilitynetwork latency and failure
© Steven Shorrock / Flickr
![Page 16: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/16.jpg)
Locality vs consistencydata access lag or outage
© R2 HOX / Flickr
![Page 17: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/17.jpg)
feature rich application stack
nginx + uWSGI
© Terabass / wikipedia
![Page 18: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/18.jpg)
⇨ gets HTTP requests ⇨ produces jobs
⇦ returns the total hit count
collector
![Page 19: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/19.jpg)
consumes jobs ⇨ increments a counter
processor
![Page 20: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/20.jpg)
![Page 21: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/21.jpg)
simple and reliable job queue server
beanstalkd
© Mooms
![Page 22: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/22.jpg)
![Page 23: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/23.jpg)
![Page 24: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/24.jpg)
![Page 25: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/25.jpg)
![Page 26: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/26.jpg)
automated scaling and fault tolerance
Service discovery
© Cambridge University / wikipedia
![Page 27: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/27.jpg)
multi datacenter + key / value storage
Consul
![Page 28: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/28.jpg)
![Page 29: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/29.jpg)
seamless service registration / health check
uWSGI consul plugin
![Page 30: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/30.jpg)
consul-register = url=http://localhost:8500,name=my_service
![Page 31: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/31.jpg)
to victory
Let’s build this up
© Brian Rincker / Flickr
![Page 32: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/32.jpg)
![Page 33: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/33.jpg)
![Page 34: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/34.jpg)
![Page 35: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/35.jpg)
![Page 36: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/36.jpg)
![Page 37: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/37.jpg)
![Page 38: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/38.jpg)
(hopefully)
Live demo !
© shoutmeloud / Pinterest
![Page 39: CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application EuroPython 2015 © Eric Fischer / Flickr](https://reader036.vdocuments.net/reader036/viewer/2022081615/5fd2346d095c194b5651a4e9/html5/thumbnails/39.jpg)
source code : github.com/ultrabug/ep2015@ultrabug
Thanks
© Eric Fischer / Flickr