learning from incidents at autotrader
TRANSCRIPT
![Page 1: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/1.jpg)
Learning from Incidents at Auto Trader
@4ndyHumphrey
![Page 2: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/2.jpg)
Learning from Failure at Auto Trader
@4ndyHumphrey
![Page 3: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/3.jpg)
What is a Learning Organisation? What is the Reality? What are my Choices? Incident Reviews - things to Avoid Incident Reviews - things to Encourage What about holding people to Account? A bit on Our process
Learning from Incidents
![Page 4: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/4.jpg)
Our People
PRIVATE Car Sellers
Trade Car Dealers
30,000
15,000
Auto Trader Staff
Product & Tech Teams
850
275
Our Customers
![Page 5: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/5.jpg)
Our Technology Platform
1.2 billion page views per month
70 million peak page views per day
15 million unique visitors per month
Supported by 100 live applications
![Page 6: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/6.jpg)
Further Reading up front
Links:John Allspaw - The Infinite HowsSteve Shorrock - if it werent for the peopleEuroControl - Systems Thinking for SafetyLyndsay Holmwood - Blame-Language-SharingSydney Dekker - Just Culture
Black Box Thinking – Matthew Syed
People:Steven Shorrock
Erik Hollnagel
Sidney Dekker
Matthew Syed
John Allspaw
Lindsay Holmwood
Dave Zwieback
Nancy Leveson
Field Guide to Understanding Human Error – Sidney Dekker
Beyond Blame – Dave
Zwieback
Nancy Leveson - Engineering a Safer World
Further Reading up front
![Page 7: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/7.jpg)
What is a Learning Organisation?
![Page 8: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/8.jpg)
The Loom
![Page 9: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/9.jpg)
A Learning Organisation
![Page 10: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/10.jpg)
Moral ResponsibilityJob SatisfactionEconomic Imperative
Why should I want to learn?
![Page 11: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/11.jpg)
What’s the reality?
![Page 12: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/12.jpg)
Blame management
![Page 13: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/13.jpg)
Blame - Fundamental Attribution Error
![Page 14: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/14.jpg)
Blame - Justice
![Page 15: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/15.jpg)
Blame - Hindsight
![Page 16: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/16.jpg)
Blame – Bad Apple Theory
![Page 17: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/17.jpg)
Blame – Ignoring context
Jonathan Caramanus/Green Renaissance/wwf.org.uk
![Page 18: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/18.jpg)
Blame - It’s Easy
![Page 19: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/19.jpg)
What are my choices?
![Page 20: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/20.jpg)
Things will always go wrong
https://www.youtube.com/watch?v=EvegBo4TUdQ
![Page 21: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/21.jpg)
You can blame people…
![Page 22: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/22.jpg)
Or say it’s a one off…
![Page 23: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/23.jpg)
Or you can look at the context…
![Page 24: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/24.jpg)
…Learn and make changes
![Page 25: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/25.jpg)
“Blame is the enemy of safety…”
But it is a choice:
Nancy Leveson
W. Edwards Deming
“Whenever there is fear, you will get wrong figures.”
![Page 26: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/26.jpg)
Incident Reviews:Things to avoid
![Page 27: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/27.jpg)
Culture of fear
![Page 28: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/28.jpg)
Top down
![Page 29: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/29.jpg)
Asking Why?
![Page 30: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/30.jpg)
Environment
Capabilities
Behavior
Values and Beliefs
Identity
Contexts – WHERE?
Methods, Approaches – HOW?
Skills and Actions – WHAT?
Motivation and permission - WHY?
Sense of Self, Role– WHO?
Questioning styles:
Dilts Model
![Page 31: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/31.jpg)
Don’t go too Deep!
Environment
Capabilities
Behavior
Values and Beliefs
Identity
Contexts – WHERE?
Methods, Approaches – HOW?
Skills and Actions – WHAT?
What is important/true – WHY?
Sense of Self – WHO?
Dilts Model
![Page 32: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/32.jpg)
Single Root Cause
![Page 33: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/33.jpg)
Points scoring
![Page 34: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/34.jpg)
Incident Reviews: How to encourage learning
![Page 35: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/35.jpg)
Priming
![Page 36: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/36.jpg)
Keep an open mind
![Page 37: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/37.jpg)
Explore how events unfolded
![Page 38: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/38.jpg)
Incident Review Prompts(from The Field Guide To Understanding Human Error, by Sidney Dekker)
At each juncture in the sequence of events (if that is how you want to structure this part of the accident story), you want to get to know:
• Which cues were observed (what did he or she notice/see or did not notice what he or she had expected to notice?)• What knowledge was used to deal with the situation? Did participants have any experience with similar situations that was useful in dealing with this one?• What expectations did participants have about how things were going to develop, and what options did they think they have to influence the course
of events?• How did other influences (operational or organizational) help determine how they interpreted the situation and how they would act?
Here are some questions Gary Klein and his researchers typically ask to find out how the situation looked to people on the inside at each of the critical junctures:
Debriefings need not follow such a scripted set of questions, of course, as the relevance of questions depends on the event. Also, the questions can come across toparticipants as too conceptual to make any sense. You may need to reformulate them in the language of the domain.
Cues What were you seeing?What were you focusing on?What were you expecting to happen?
Interpretation If you had to describe the situation to your colleague at that point, what would you have told?Errors What mistakes (for example in interpretation) were likely at this point?Previousexperience/knowledge
Were you reminded of any previous experience?Did this situation fit a standard scenario? Were you trained to deal with this situation? Were there any rules that applied clearly here?Did any other sources of knowledge suggest what to do?
Goals What were you trying to achieve?Were there multiple goals at the same time?Was there time pressure or other limitations on what you could do?
Taking action How did you judge you could influence the course of events?Did you discuss or mentally imagine a number of options or did you know straight away what to do?
Outcome Did the outcome fit your expectation?Did you have to update your assessment of the situation?
Communications What communication medium(s) did you prefer to use? (phone, chat, email, video conf, etc.?) Did you make use of more than one communication channels at once?
Help Did you ask anyone for help?What signal brought you to ask for support or assistance? Were you able to contact the people you needed to contact?
![Page 39: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/39.jpg)
Timelines
14:00 Alert received from
Site confidence
15:15 Incident communication
sent
16:00 Incident closure comms
sent
1. Factual timeline entries can be filled in prior to the Review Meeting
![Page 40: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/40.jpg)
Timelines
14:00 Alert received from
Site confidence
15:15 Incident communication
sent
16:00 Incident closure comms
sent
1. Factual timeline entries can be filled in prior to the Review Meeting
13:10 Slow server performance
observed by BIll
14:20 Bill spoke to John about SC issues and
decided to recover DB
15:50 John finished DB recovery
2. As a group, overlay the basic timeline with key decisions and junctures
![Page 41: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/41.jpg)
One conversation
![Page 42: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/42.jpg)
Actions
![Page 43: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/43.jpg)
Impartial facilitator
![Page 44: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/44.jpg)
Investigate what went well
![Page 45: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/45.jpg)
Practice – make it habit
![Page 46: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/46.jpg)
What about holding people to account?
![Page 47: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/47.jpg)
Accountability
![Page 48: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/48.jpg)
Our process:
![Page 49: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/49.jpg)
Major IncidentsHigh Severity IncidentsFailed Releases (all)Failed Changes (Large)
Our Process
![Page 50: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/50.jpg)
Priming – Timeline - Actions
![Page 51: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/51.jpg)
We understand and truly believe that everyone did the best job they could, given what they knew at the
time, their skills and abilities, the resources available, and the situation at hand
We are here to learn and find solutions to improve our ways of working
Why we are here:
![Page 52: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/52.jpg)
Open MindedGo back in time
No single ‘Root Cause’How not Why
Things that help us learn
![Page 53: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/53.jpg)
Blaming peopleHuman ErrorArse CoveringPoints scoring‘Trying Harder’
Talking over people
Things that stop us learning:
![Page 54: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/54.jpg)
After the review:
• Incident details recorded• Actions (owners, dates) recorded• Owned by Service Management Team
![Page 55: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/55.jpg)
Further Reading up front
Links:John Allspaw - The Infinite HowsSteve Shorrock - if it werent for the peopleEuroControl - Systems Thinking for SafetyLyndsay Holmwood - Blame-Language-SharingSydney Dekker - Just Culture
Black Box Thinking – Matthew Syed
People:Steven Shorrock
Erik Hollnagel
Sidney Dekker
Matthew Syed
John Allspaw
Lindsay Holmwood
Dave Zwieback
Nancy Leveson
Field Guide to Understanding Human Error – Sidney Dekker
Beyond Blame – Dave
Zwieback
Nancy Leveson - Engineering a Safer World
Further Reading Again
![Page 56: Learning From Incidents at Autotrader](https://reader036.vdocuments.net/reader036/viewer/2022062904/58826ffc1a28ab470c8b6891/html5/thumbnails/56.jpg)
Questions?