winops conf 2016 - peter mounce - dos yourself in production every night to prove you can take it
TRANSCRIPT
![Page 1: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/1.jpg)
1
DoS yourself inproduction everynight to prove you
can take it@petemounce@justeat_tech@petemounce + @justeat_tech
![Page 2: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/2.jpg)
@justeat_tech
2
Any questions?Shout them out as we go. That's more fun.
@petemounce + @justeat_tech
![Page 3: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/3.jpg)
3
Who are JUST EAT?
![Page 4: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/4.jpg)
4
Performance?
![Page 5: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/5.jpg)
5
When do you suppose peaktime is?
The same time we DoS ourselves, of course!
![Page 6: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/6.jpg)
6
We have cyclic demand
![Page 7: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/7.jpg)
7
The problem withcontinuous delivery
Everyone wants to change everything, all the time.
![Page 8: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/8.jpg)
8
Traditional approachLet's make an environment like
production and run load through that.
![Page 9: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/9.jpg)
9
Individual tests take toolong
![Page 10: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/10.jpg)
10
But of course...
![Page 11: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/11.jpg)
11
So, test all the time
![Page 12: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/12.jpg)
12
So, test in production
#YOLO!
We deploy 10s of small changes a dayand we have alerts. I bet we won't break
production (without noticing)
#WhatCouldPossiblyGoWrong?
Let's just do it in production with faketraffic at the same time as customers!
![Page 13: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/13.jpg)
13
Reasons why this isn'tinsane
![Page 14: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/14.jpg)
14
How did we start doing this?Technology aspects and people aspects
![Page 15: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/15.jpg)
15
Have the idea to start( )We didn't invent this
![Page 16: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/16.jpg)
16
Choose scenarios we care about
![Page 17: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/17.jpg)
17
Choose a load agent
![Page 18: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/18.jpg)
18
Gain confidence outside of peak time(This part is also about reassuring stakeholders that you've
got it all under control...)
![Page 19: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/19.jpg)
19
Start adding data variety
![Page 20: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/20.jpg)
20
Make the computer do it every dayThis is the most vital part!
![Page 21: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/21.jpg)
21
Get more elaborate laterFake away external dependencies
x‐traffic‐flavour: fake
![Page 22: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/22.jpg)
22
And even more elaborate...Fake away more complicated things
![Page 23: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/23.jpg)
23
How have we kept doingthis?
... and what did we learn to do better
![Page 24: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/24.jpg)
24
Didn't allow tests to be red (for long)
![Page 25: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/25.jpg)
25
Needed to tune levels over time
![Page 26: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/26.jpg)
26
Got smarter about data management
![Page 27: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/27.jpg)
27
Embraced the fact that things break
![Page 28: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/28.jpg)
28
What battle scars did weget... lately?
All of these would have hurt badly if we hadn't had theability to turn the pain off ourselves
![Page 29: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/29.jpg)
29
Find unbounded result sets beforecustomers
![Page 30: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/30.jpg)
30
Monitoring needs to be solid!
![Page 31: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/31.jpg)
31
Realise AWS account limits are closerthan we thought...
Credit: StickerMule.com
![Page 32: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/32.jpg)
32
Realise haproxy should balance, notmagnify load...
![Page 33: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/33.jpg)
33
Realise we're not as smart as wethink...
Dear WinOps.org, this is why I had to leave early lastyear...
![Page 34: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/34.jpg)
34
But...Discovered problems during peacetime, not peak time
![Page 35: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/35.jpg)
35
What did we gain?
![Page 36: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/36.jpg)
36
Peace of mind, #1Continuous, early, warning about:
Getting slowerRunning out of capacity
![Page 37: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/37.jpg)
37
Peace of mind, #2Good, simple, clear operational response to most surprises:
Is fake load running? Stop it.
Scale up
Now, start to think
![Page 38: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/38.jpg)
38
Peace of mind, #3If we find a problem Thursday night:
1. Turn off fake load for the weekend2. Enjoy weekend3. Fix it next week with less pressure
![Page 39: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/39.jpg)
39
Performance & operability == 1stclass concern
![Page 40: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/40.jpg)
40
Alerts become automated tests inproduction
![Page 41: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/41.jpg)
41
git push production is one stepcloser
Continuous testing in production can be applied to morethan just performance & capacity
![Page 42: WinOps Conf 2016 - Peter Mounce - DoS yourself in production every night to prove you can take it](https://reader034.vdocuments.net/reader034/viewer/2022052705/588ae0921a28abab6c8b53d1/html5/thumbnails/42.jpg)
42
Online takeaway. Harderthan you might think
We've got many open spots for talented engineers (London,Bristol, Kiev), if you're interested.
tech.justeat.com/jobs
Get in touch peter.mounce@justeat.com