stay cool … when it gets hot! incident / problem management thinking correctly under pressure
TRANSCRIPT
Stay cool … when it gets hot!
INCIDENT / PROBLEM MANAGEMENT
Thinking Correctly Under Pressure
The changing service operations landscape
Consistency and quality becomes a competitive advantage
Uptime of services, applications and systems is key
Service is being recognized as a value added function
Being able to Think Correctly Under Pressure (TCUP) is a necessity, a must – not a want in today‘s
environments.
Being able to Think Correctly Under Pressure (TCUP) is a necessity, a must – not a want in today‘s
environments.
Can‘t take the heat? You are not alone..
Trying the same things over and over again expecting different results? It may be time to try another approach.Trying the same things over and over again expecting
different results? It may be time to try another approach.
TCUP taken to the extreme
Consistency and quality becomes a competitive advantage….
……or might even save lives!
Thinking under pressure – Trainable?
Obvious jobs with a high degree of TCUP
There is a wide spread consensus
that emergency services need to be
technically and mentally capable to
deal with life threatening situations!
There is a wide spread consensus
that emergency services need to be
technically and mentally capable to
deal with life threatening situations!
When I said I route traffic I meant data!
How about the IT Services
world?
How about the IT Services
world?
It might not be saving lives, but ..
And many others….
Customer satisfaction / customer retention
Cost of Service
Mean-Time-To-Restore (MTTR)
First time Fix Rate (FTFR)
Good quality consistent communication, decision making and thinking are driven
by key forces. You have the power to drive them!
Driving Forces
Driving forces of TCUP mean …
Use brain HW and SW
effectively
Right Channels
Allow effective review and continuous improvement
Feedback
Capture and share essential information
Right Infrastructure
Enabling predictable
performance
Communicate progress to
stakeholders
Driving forces of TCUP mean …
Use brain HW and SW
effectively
Right Channels
Allow effective review and continuous improvement
Feedback
Capture and share essential information
Right Infrastructure
Enabling predictable
performance
Communicate progress to
stakeholders
Let's do a quick math problem:
A bat and a ball cost together 1,10 €.
The bat is 1,00 € more expensive than the ball.
How much is the ball?
How much is the ball?
• 20 cents ???
• 10 cents ???
• 5 cents ???
• Intuition vs. reasoning(1+X)+X=1.101+2X=1.102x=0.10x=0.05
Enabling predictable performance
Having the right skills to stay in control, create oversight and avoid jumping to
conclusions
Neuroscience – Brain HardwareNeuroscience – Brain Hardware
Skills – Brain SoftwareSkills – Brain Software
Character – Ability to cope with StressCharacter – Ability to cope with Stress
Brain Hardware and OS…
Emotional tagging
Is where emotional information attaches itself to the thoughts and experiences stored in our memories. This emotional information tells us whether to pay attention to something or not.
Emotional tagging
Is where emotional information attaches itself to the thoughts and experiences stored in our memories. This emotional information tells us whether to pay attention to something or not.
Pattern recognition
Integrates as many as 30 different parts of the brain and means that in a new situation we make assumptions based on prior experiences and judgement
Pattern recognition
Integrates as many as 30 different parts of the brain and means that in a new situation we make assumptions based on prior experiences and judgement
2 primary hardwired processes to decide on what to do !
2 primary hardwired processes to decide on what to do !
(source: Harvard Business Review Feb 2009 )
Brain Hardware and OS…
Reward
Engagement increases dopamine levels
Improves collaboration
Increases quality of rational thinking
Reward
Engagement increases dopamine levels
Improves collaboration
Increases quality of rational thinking
Threat
Threat response is easily triggered.
Takes resources from prefrontal cortex
Inhibits solving complex problems
Threat
Threat response is easily triggered.
Takes resources from prefrontal cortex
Inhibits solving complex problems
2 primary hardwired responses 2 primary hardwired responses
(source: Neuroleadership Journal 2008 issue 1)
FACTORS IN RESPONSE BEHAVIOUR
STATUS - People perform worse when status is threatened
CERTAINTY - a sense of certainty is increased through explicit processes
AUTONOMY – tends to be reduced in emegencies; being in a corner inhibits performance
RELATEDNESS – we work better when we feel connected to other people
FAIRNESS – time pressure will tend to produce unfair demands
RECIPE FOR SUCCESS
1. Increase certainty through process
2. Practice on the easy
3. Select the best people
4. Filter out Status and Fairness threats
5. Use autonomy wisely
?
Individuals knowing what to do, why they are doing it,and how it is to be done
Problem Analysis
Describe the Problem
State the Problem Include both an object and a defect in the sentence. (WHAT IS WRONG WITH WHAT?)
One Way traffic on <trunk equipment>
Specify the Problem
IS IS NOTWHATWhat object? <trunk equipment> <comparable equipment>What defect? One way voice Packets being lost
WHERE Hutchison Telecom In labWhere <exact location> <test rig>geographically? <equipment name> <equipment name>Where on the
object? <card name> <packets sent see logs>
WHENWhen first? 18th August 00:30 BEFORE
When since? Intermittently Constant/Periodic
When in After loading patch set 11 After previous resetlife cycle?
EXTENTHow many objects? 1 <>What is the size? Complete loss for ~5 minutes <>How many defects? 40 instances reported <>
What is the trend? Stable Increasing / Decreasing
INCREASING CERTAINTY THROUGH PROCESS
Predictable performance
“Implementing and improving a process is key for predictable performance!”
0,00
5,00
10,00
15,00
20,00
25,00
1 11 21 31 41 51 61 71 81 91 101 111 121 131
To
tal t
ime
on
tas
k / M
in]
Implementation days
Reduction of total time on task in Iincident type x over time
Data
XBar
UCL
LCL
Target
List Concerns
Separate/Clarify
Set Priority
State and Specify Problem
Distinctions and Changes
Testing/Verifying Possible Causes
Logical Process Flow
Time
KT – A defined, logical sequence of steps
Sequential Logic
List Issues
Assess Priority
Assign Actions
Describe Problem
Evaluate Possible Causes
Apply Solution
Time
Understand the quality whilst under way
Case Study- the Bugmarket incident
1. You have details of a major incident under your seat / or you will get them right now
2. You have 10 minutes to respond initially
3. Write down / memorize what you believe are the 4 most important facts
4. What question would you like to ask next?
5. When asked to do so share your findings with the person next to you.
The Problem Statement
Bug Market Web Login takes 5 minutes
1 OBJECT 1 DEVIATION
• focus!!!
• make sure the team talks and thinks about the same problem
• foundation for any problem-knowledge-mgmt-system
• BugMarket web client•Login takes around 5 m in
•BugMarket Web client connecting to the web server in Spain •BugMarket web client on user computer
• January 21st, NMD• 2-4 times per day 06/02/08 @ 13:31:15 - 13:34 06/02/08 @ 14:48:33 - 14:51:28 • After clicking the "Login" button
• All users on all machines
•Problem lasts for 4 to 5 min
•BugMarket native client login•Login fails /< 4 min / > 6 min
• BugMarket Web client connecting to the web servers in India & Canada
•BugMarket native client on user computer
• before that• more frequently (NMD) Every login
•Trying to load the login page
• Only some users, all users on some machines• Longer or shorter lengths of time
IS IS NOT
WHAT
WHERE
WHEN
EXTENT
Bug Market Web Login takes 5 minutes
Let´s test possible causes!
• BugMarket web client•Login takes around 5 m in
•BugMarket Web client connecting to the web server in Spain •BugMarket web client on user computer
• January 21st, NMD• 2-4 times per day 06/02/08 @ 13:31:15 - 13:34 06/02/08 @ 14:48:33 - 14:51:28 • After clicking the "Login" button
• All users on all machines
•Problem lasts for 4 to 5 min
•BugMarket native client login•Login fails /< 4 min / > 6 min
• BugMarket Web client connecting to the web servers in India & Canada
•BugMarket native client on user computer
• before that• more frequently (NMD) Every login
•Trying to load the login page
• Only some users, all users on some machines• Longer or shorter lengths of time
IS IS NOT
WHAT
WHERE
WHEN
EXTENT
Bug Market Web Login takes 5 minutes
Possible Cause : Network issues?
X Why native client not affected??
Possible Cause : Out of date registry entries for the Local System Account for database connections still point to other databases that are no longer accessible.
Only CQ Web was affected, as the stale registry keys were only present in the HKEY_USERS\.DEFAULT hive, which does not affect the native client on the same machine.
Driving forces of TCUP mean …
Use brain HW and SW
effectively
Right Channels
Allow effective review and continuous improvement
Feedback
Capture and share essential information
Right Infrastructure
Enabling predictable
performance
Communicate progress to
stakeholders
Sample form to document progress
A template for sharing information
an infrastructure necessary to capture and share essential information
STAKEHOLDERS’ LEGITIMATE DEMANDS
1. See you have understood the situation
2. Show you understand the business impact
3. Have a plan towards resolution
4. Meet key milestones in the investigation
5. Are able to isolate the affected area
6. Have a credible fix (high probability of working)
Driving forces of TCUP mean …
Use brain HW and SW
effectively
Right Channels
Allow effective review and continuous improvement
Feedback
Capture and share essential information
Right Infrastructure
Enabling predictable
performance
Communicate progress to
stakeholders
Feedback loop
having a feedback loop in place to allow effective review and continuous improvement
Driving forces of TCUP mean …
Use brain HW and SW
effectively
Right Channels
Allow effective review and continuous improvement
Feedback
Capture and share essential information
Right Infrastructure
Enabling predictable
performance
Communicate progress to
stakeholders
Communication – right channels
Reduce Status and Autonomy threats by
having separate channels to
communicate progress to stakeholders
Leave the “solvers” to do what they do best
MANAGING AUTONOMY
Autonomy is not a free for all!
Major incidents need a strong framework of expectations in which an individal can navigate. This includes:
• Quality standards for incident data
• Time points when key data gathering needs to be completed
• Sequential logic supports setting of standards
BUILDING RELATEDNESS
Attacks from the users don‘t help our support teams feel part of the group
Build feelings of relatedness through a strong Incident Facilitator who:
• Advocates the issues to the technology community
• Builds understanding of the business issues at play
Improvement area Driver Goal Comments
Customer satisfaction
Time to resolution (TTR) &First time fix rate (FTFR)
Fastest competent resolution with optimum effort
Customers see a longer duration than any single function in Service
Cost of service Quality of analysis
Spare part usage
No. of field dispatches
High quality clarification and analysis to reduce spare parts and field dispatch to necessary minimum
Not all available information is used to diagnose before taking action
Quality of service Proactive problem ticket (PPT)
Quality audit and coaching loops for ticket content on regular basis(per KT standard)
Change the focus from reactive fault management to proactive fault prevention over time
The way towards fault prediction and prevention is certainly not short, but probably the most important goal for the change process
AREAS FOR IMPROVEMENT
So what do these driving forces mean for you?
Right ChannelsFeedback
Right InfrastructureEnabling
Predictableperformance
Q&A
Contact information
Stefan Brahmer Senior Consultant
Service Excellence EMEA
0049 170 1611566sbrahmer@kepner-
tregoe.com
Stefan Brahmer Senior Consultant
Service Excellence EMEA
0049 170 1611566sbrahmer@kepner-
tregoe.com
Kurt Plambech Associate ConsultantService Excellence
Nordic
0045 2334 [email protected]
Kurt Plambech Associate ConsultantService Excellence
Nordic
0045 2334 [email protected]