boring dot com - the virtues of boring technology
TRANSCRIPT
![Page 1: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/1.jpg)
The virtues of boring technology
Kristian Köhntopp, Scalability Team Veteran
Boring
![Page 2: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/2.jpg)
What people think Booking looks like…Photo: ©2012 Kristian Köhntopp
![Page 3: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/3.jpg)
What Booking actually looks like…Photo: ©2015 Kristian Köhntopp
![Page 4: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/4.jpg)
Interesting times…
![Page 5: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/5.jpg)
Interesting times…
![Page 6: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/6.jpg)
Interesting times…Photo: ©2012 Kristian Köhntopp
![Page 7: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/7.jpg)
Boring Setup…nginx
uwsgi MySQL
Replication
Write
Read
Apache
mod_perl
![Page 8: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/8.jpg)
Not using sharding…Web Server GroupApache (mod_perl)
bp Database av Database
Web Server GroupApache (mod_perl)
bp Database
![Page 9: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/9.jpg)
Doing it over an over again…Web Server GroupApache (mod_perl)
Many Databases
Fragile!
![Page 10: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/10.jpg)
Many web servers…
LB
Redirect
Many bp servers
Load balanceddirect connect
Doing something about it…
![Page 11: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/11.jpg)
Y U SO BORING?CC BY-SA 3.0 Kartin st, https://en.wikipedia.org/wiki/Pallas%27s_cat#/media/File:Manul2.jpg
![Page 12: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/12.jpg)
Growth kills…
Problem
appears Solution
must be
readySolution
in use Solution
falling apart
3 years, Factor 10
© 2006-2015 booking.com
![Page 14: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/14.jpg)
Keep it simple, that’s complicated enough!"AK-47 type II Part DM-ST-89-01131". Public Domain - https://commons.wikimedia.org/wiki/File:AK-47_type_II_Part_DM-ST-89-01131.jpg
![Page 15: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/15.jpg)
What’s wrong with this model?
Wait…Wait…
Result
Set
httpsSQL
HTML
![Page 16: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/16.jpg)
All the things we didn’t do…● Not doing many network hops.
● Not doing distributed.
● Not doing asynchronous.
● Not doing microservices.
● Not doing schemaless.
● Not doing sharding.
● Not doing virtualization.
● Not doing hyperconvergence.
● Not doing multiple datacenters.
● Not doing multirepo.
● Not doing test driven development.
● Not doing change management.All Lies!
![Page 17: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/17.jpg)
Not doing many network hops.● Reliable, predictable performance.
● Low response time.
● Low response time variance.
● Works in a regular data-center network.
![Page 18: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/18.jpg)
Not doing distributed…● Strictly speaking a bug.
●We do have per-schema masters,
● we do multi-schema updates.
● This happes in places where it does not matter:
● Low concurrency,
● non-critical information.
● Also: “fsck”
avMaster
userMaster
![Page 19: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/19.jpg)
Not doing asynchronous…● Perl can’t (with MySQL) and mostly doesn’t (elsewhere).
● MySQL couldn’t.
● Non-critical Javascript on page for animations and spellchecking.
![Page 20: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/20.jpg)
Not doing microservices…● Codebase has structure and contracts.
● This does not express itself in the form of service boundaries.
● Saves a lot serialization/copying/instantiation.
● Less overhead.
● Less Jitter.
● Less locking issues.
![Page 21: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/21.jpg)
Not doing schemaless…● There is no such thing as schemaless.
● The sooner you admit that, the less painful it is going to be.
![Page 22: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/22.jpg)
Not doing sharding…● Except where we are doing it.
● Unfortunately, in an increasing number of places.
● Even where we aren’t doing it, we are seeing application-side hash-joins.
● At least they are visible.
![Page 23: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/23.jpg)
Not doing virtualization…● Again, predictable, reliable performance.
● Also, lots of problems with high density networking and computing.
● Semi-manual is painful.
● Fully automated is unreliable, and doesn’t scale.
=2005, 2 Racks 2015, 2U
50 Cores, 50 GBit/s,
512 GB SSD = moar IOPS!
![Page 24: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/24.jpg)
Not doing hyperconvergence…● Again, predictable, reliable performance.
● Does painful things to your network.
● Has high commit-latency.
●Good for archive storage.
● Bleeding edge technology. 18U Rack (resize vertically) 18U Rack (resize vertically) 18U Rack (resize vertically)
![Page 25: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/25.jpg)
Not doing multiple datacenters…● Well, there is CDN…
●Many, many, many images.
● The load for the active content actually isn’t that much.
![Page 26: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/26.jpg)
Not doing multirepo…● »The booking.com git-insanity.«
● Actually, the technical term is
● »Monorepo with trunk-based development and separation of rollout and
activation.«
● Man, am I thankful for that, if only after the fact.
![Page 27: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/27.jpg)
Not doing TDD…● “Making it safe to test in production.”
● “Separation of rollout and activation.”
● “A/B-testing” aka “Requirements validation.”
● “Conversion tracking” aka “fast metrics.”
● Fast, lag-free monitoring.
![Page 28: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/28.jpg)
Not doing change-management…● “The Booking-Rules.”
● “Roll-Call.”
● “Separation of rollout and activation.”
● Duplicate data keeping.
● Fast, lag-free monitoring.
![Page 29: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/29.jpg)
All the things we didn’t do…● Not doing many network hops.
● Not doing distributed.
● Not doing asynchronous.
● Not doing microservices.
● Not doing schemaless.
● Not doing sharding.
● Not doing virtualization.
● Not doing hyperconvergence.
● Not doing multiple datacenters.
● Not doing multirepo.
● Not doing test driven development.
● Not doing change management.
![Page 30: Boring dot com - the virtues of boring technology](https://reader033.vdocuments.net/reader033/viewer/2022061306/58a157d61a28abbe3c8b64fd/html5/thumbnails/30.jpg)
http://booking.com/jobsPhoto: ©2012 Kristian Köhntopp
Want a boring job?