disappointing disasters & other y2k · 2020-06-11 · microservices and loose coupling...
TRANSCRIPT
![Page 1: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/1.jpg)
@wiredferret
Y2K & Other Disappointing Disasters
Risk Reduction and Harm Mitigation
![Page 2: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/2.jpg)
@wiredferret
![Page 3: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/3.jpg)
@wiredferret
An actual pager
![Page 4: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/4.jpg)
@wiredferret
Ready for the big night
![Page 5: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/5.jpg)
@wiredferret
This was a vital job
![Page 6: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/6.jpg)
@wiredferret
No one remembers the crisis averted.
![Page 7: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/7.jpg)
@wiredferret
Risk Reduction
![Page 8: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/8.jpg)
@wiredferret
Risk Reduction
● Stay-home orders
● Vaccination● Train gates
![Page 9: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/9.jpg)
@wiredferret
How to reduce risk
![Page 10: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/10.jpg)
@wiredferret
Secure your zone
![Page 11: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/11.jpg)
@wiredferret
![Page 12: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/12.jpg)
@wiredferret
Predict states
![Page 13: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/13.jpg)
Accept Risk
● You can’t prevent everything
● Decide what matters to save
● Make mindful tradeoffs basedon data
![Page 14: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/14.jpg)
@wiredferret
Harm Mitigation
![Page 15: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/15.jpg)
@wiredferret
When Things Go
Wrong
● Seatbelts● Building codes● RAID
![Page 16: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/16.jpg)
@wiredferret
![Page 17: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/17.jpg)
@wiredferret
How to mitigate harm
![Page 18: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/18.jpg)
Fail safe or fail secure?
● What are you protecting?
● What is your risk?
![Page 19: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/19.jpg)
@wiredferret
Failure is inevitable.
Disaster is not.
![Page 20: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/20.jpg)
@wiredferret
What’s a disaster?
![Page 21: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/21.jpg)
@wiredferret
![Page 22: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/22.jpg)
@wiredferret
![Page 23: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/23.jpg)
@wiredferret
Let’s design for that
![Page 24: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/24.jpg)
@wiredferret
Fizzle factors
● Microservices and loose coupling
● Fault-tolerant messaging● Data duplication● Canary launches● Kill switches and circuit
breakers● Automatic recovery● Testing for load, stress,
and outage
![Page 25: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/25.jpg)
Rubber bands, not rigid joints
Microservices and loose coupling
![Page 26: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/26.jpg)
Fault-tolerant messaging
The internet routes around censorship as damage
![Page 27: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/27.jpg)
Data duplication
It’s not a backup if you haven’t tested restoration
![Page 28: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/28.jpg)
Canary launches
Verify
Image by Capri23auto from Pixabay
![Page 29: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/29.jpg)
Kill switches and circuit breakers
Stick a fork in it, it’s done
![Page 30: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/30.jpg)
Automatic recovery
Take two reboots and call me in the morning
![Page 31: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/31.jpg)
Testing for load, stress and outage
Elastic scaling only gets you so far
![Page 32: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/32.jpg)
@wiredferret
tl;rt● Expect failure● Make systems less
rigid● Plan for disaster● Degrade gracefully
![Page 33: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/33.jpg)
https://tinyurl.com/failoverconf-heidi
![Page 34: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/34.jpg)
THE careful text-books measure
(Let all who build beware!)
The load, the shock, the pressure
Material can bear.
So, when the buckled girder
Lets down the grinding span,
'The blame of loss, or murder,
Is laid upon the man.
Not on the Stuff - the Man!
The Hymn of Breaking Strain
But in our daily dealing
With stone and steel, we find
The Gods have no such feeling
Of justice toward mankind.
To no set gauge they make us-
For no laid course prepare-
And presently o'ertake us
With loads we cannot bear:
Too merciless to bear.
The prudent text-books give it
In tables at the end
'The stress that shears a rivet
Or makes a tie-bar bend-
'What traffic wrecks macadam-
What concrete should endure-
but we, poor Sons of Adam
Have no such literature,
To warn us or make sure!
![Page 35: Disappointing Disasters & Other Y2K · 2020-06-11 · Microservices and loose coupling Fault-tolerant messaging Data ... Automatic recovery Testing for load, stress, and outage. Rubber](https://reader030.vdocuments.net/reader030/viewer/2022041108/5f0c5b437e708231d434ff3a/html5/thumbnails/35.jpg)
We hold all Earth to plunder -
All Time and Space as well-
Too wonder-stale to wonder
At each new miracle;
Till, in the mid-illusion
Of Godhead 'neath our hand,
Falls multiple confusion
On all we did or planned-
The mighty works we planned.
We only of Creation
(0h, luckier bridge and rail)
Abide the twin damnation-
To fail and know we fail.
Yet we - by which sole token
We know we once were Gods-
Take shame in being broken
However great the odds-
The burden of the Odds.
Oh, veiled and secret Power
Whose paths we seek in vain,
Be with us in our hour
Of overthrow and pain;
That we - by which sure token
We know Thy ways are true -
In spite of being broken,
Because of being broken
May rise and build anew
Stand up and build anew.
-- Rudyard Kipling