root cause analysis master plan

+18.0 Root Cause AnalysisRoot Cause Analysis is the method of problem solving that identifies the root causes of failures or problems. A root cause is the source of a problem and its resulting symptom, that once removed, corrects or prevents an undesirable outcome from recurring.

Failure Is Trying to Tell Us Something

Performance–Based Project Management®, Copyright © Glen B. Alleman, 2002 ― 2016 842

+ The Notion of Root Cause Analysis

Symptom: The result or outcome of the problem.An observation§ We have late additions to the release that break the software§ We have “core” defects that should been caught long before

production release§ We make changes to software, stored procedures, or the

database only to discover it was a mistake.§ We make promises to the customer before assessing the

impact on our resources or the technical difficulty

Problem: § Test coverage insufficient the detect latent bugs in software§ We commit before understanding the consequences

Root Cause: § No software structure to determine test coverage or change

impacts on baseline.§ No detailed understanding of our capacity for work and

productivity of our technical staff

18. Root Cause

Performance–Based Project Management®, Copyright © Glen B. Alleman, 2002 ― 2016

843

+ Four Root Causes for Program Cost and Schedule Overruns

844


Unrealistic Performance Expectations, missing Measures of Effectiveness (MOE) and Measures of Performance (MOP).

Unrealistic Cost and Schedule estimates based on inadequate risk adjusted growth models.

Inadequate assessment of risk and unmitigated exposure to these risks without proper handling plans.

Unanticipated technical issues without alternative plans and solutions to maintain effectiveness.

Unanticipated Cost and Schedule Growth

The

Lens

of P

erfo

rman

ce

Ass

essm

ent

“Borrowed” with permission from Mr. Gary Bliss, Director, Performance Assessment and Root Cause Analysis (PARCA), Office of Assistant Secretary of Defense for Acquisition, Technology and Logistics.

18. Root Cause

+ Our Path to Better Root Cause Analysisn Principles of Root Cause Analysis

n Understanding the weaknesses in our current method

n Introduction the Apollo Method

n Steps to applying Apollo

n Transition from the current method to Apollo


845

18. Root Cause

ISO/IEC 17025:2005 (4.11.2) ‒ The procedure for corrective action shall start with an investigation to determine the root cause(s) of the

problem.

+ Evaluation of Significance

n Can the nonconformity recur or does it raise doubt about compliance of the laboratory’s operations with its own policies and procedures? (ISO/IEC 17025, Clause 4.9.2)

n What criteria is utilized to determine significance? (ISO/IEC 17025, Clause 4.9.1.b)


846

18. Root Cause

+ A Simple Perspective of Problem Solvingn Every Problem in our lives

has three basic elements connected through causality.

n Each Effect, has at least two causes:n An Action n A Condition

847

Effect

Condition Cause

Action Cause

This Cause and Effect relationship is the core principle of effective problem solving

18. Root Cause


+

18.1 Beyond the Conventional Wisdom of Problem Solving

Original Quote from George Bernard Shaw

In Apollo Root Cause Analysis: Effective Solutions to Everyday Problems, Every Time, Dean L. Gano

The common approach to problem solving is to categorize causes or identify causal factors and look for root causes within the categories.

Categorization schemes do not reveal the cause and effect relationships needed to find effective solutions.

It is the effective solution we are after.

18. Root Cause


Ignorance is a most wonderful thing.

It facilitates magic.

It allows the masses to be led.

It provides answers when there are none.

It allows happiness in the presence of danger.

All this, while the pursuit of knowledge can only destroy the illusion. It is any wonder mankind chooses ignorance?

+ The Persistent Problem …

Increasing use of larger and more complex systems potentially results in greater number of problems, evidenced by the symptoms of unstable software, latent defects, unexpected performance issues.Many of these problems and their symptoms, have a serious impact on business operations than ever before. In many case these problems are also more difficult to solve.— A Management System for the Information Business,

Edward Van Schaik

18. Root Cause

We can not solve problems by using the same kind of thinking we used when we created them. — Albert Einstein


849

+ … and the Persistent Unsuccessful Solution to recurring problems.

In every human endeavor, a critical component to success is the ability to solve problems. Unfortunately, we often set ourselves up to fail with our problem–solving strategies and our inherent prejudices. We typically rely on what we believe to be common sense, storytelling, and categorizing to resolve our problems. Conventional wisdom has us believe that problem solving is inherent to the subject at hand.— Dean L. Gano, The Apollo Method

18. Root Cause


850

+ Some sources of Guidance for Root Cause Analysis†

§2.4.5.1 Event Management – is the process that monitors all events that occur through the IT infrastructure to allow for normal operation and also to detect and escalate exception conditions.

§2.4.5.2 Incident Management – concentrates on restoring the service to users as quickly as possible, in order to minimize business impact.

§2.4.5.3 Problem Management – involves root–cause analysis to determine and resolve the cause of events and incidents, proactive activities to detect and prevent future problems/incidents and a Known Error sub process to allow quicker diagnosis and resolution if further incidents do occur.

† §2.4.5. Processes within Service Operation, ITIL V3 Service Operation

18. Root Cause


851

+ Root Cause Analysis

n A structured approach to investigating and analyzing of significant adverse events or system deficiencies and their required improvement.

n Root Cause Analysis provides information and tools to be incorporated into risk management, quality management, independent verification and validation and improvement procedures in order to:n PREVENT future occurrence of adverse events that cause or can cause

harm to individuals; and,

n CORRECT practices that have led to identified deficiencies

18. Root Cause


852

+

n The cause of a system defect or failure needs to be identified so the cause can be eliminated in the future.n The symptom and problem are

recurring

n There are repeated failures ascribed to human error, process errors, technical failures.

n To review every incident or system failuren Only symptoms, failures, and

incidents that impact business functions, Service Level Agreements, maintenance, operations, security, and integrity of the system in the future.

n When the reason for the failure is obviousn If it is obvious – fix it.

n Use RCA When … n Do Not Use RCA …

Do’s and Don’ts of Root Cause Analysis

18. Root Cause


853

+ Root Cause

n Direct causes often result from another set of causes, which could be called intermediate causes, and these may be the result of still other causes.

n When a chain of cause and effects is followed from a known end-state, back to an origin or starting point, root causes are found.

n The process used to find root causes is called root cause analysis ---systematic problem solving.

n A root cause is an initiating cause of a causal chain which leads to an outcome or effect of interest.

18. Root Cause


854

+Discovering Root Cause starts with …

18. Root Cause


855

+ The Five Whys

Five Whys involves holding meetings immediately – in our case the Friday immediately following the release.

The problems can be anything – development mistakes, infrastructure performance issues, process failures, or even internal missed schedules.

Any time something unexpected happens, we start the root cause analysiswith the 5 Ways in the domain from the chart above.

Process – how do the process not catch the problem?Tools – where in the path did the tool not catch the problem?Training – what training was missing that created the problem?Environment(s) – how did the environment fail to protect the baseline or the production system?Communications – what communications created the problem?Management – how did management fail in its actions?

The 5 Whys NEVER seek blame. But are exclusively focused on discovering the root cause that creates the problem, evidenced by the symptom

18. Root Cause


856

+ Five Steps in the Five Whys

n Invite all affected parties

n Select the meeting leader (someone training in RCA)

n Ask Why 5 times for each topic area

n Assign responsibilities for collected actual factual data

n Publish the results

Start this process with the planned release of 29 January.

18. Root Cause


857

+ Why Root Cause Analysis is Hard

n Many problems are poorly defined.

n A systematic approach is not used to classify problem and cause.

n Investigations are stopped prematurely – moving on the next problem.

n Decisions are based on guesses, hunches or assumptions.

n An inadequate level of detail is used to get to the real root cause.

n Interim containment fixes are sometimes allowed to become "permanent."

n The skills, knowledge and experience needed to uncover the root cause are not available.

n Lack of organizational will to address the bigger issues

n Fear of being blamed

n I really don’t have time for this, we have bigger problems to solve

18. Root Cause


858

+ Testing Answers from the Five Whysn What evidence is there that this cause exists?

n Is it concrete? n Is it measurable?

n What evidence is there that this cause could lead to the observed effect? n Are we merely asserting causation?

n What evidence is there that this cause actually contributed to the problem? n Even given that it exists and could lead to this problem, how do we know it wasn't

actually something else?

n Is anything else needed, along with this cause, for the stated effect to occur? n Is it self-sufficient? n Is something needed to help it along?

n Can anything else, besides this cause, lead to the stated effect? n Are there alternative explanations that fit better? n What other risks are there?

18. Root Cause


859

+18.2 Understanding

the Weakness of our Current Approach to Root Cause Analysis†

n A common Root Cause Analysis approach is to start with a narrative of what happened and how we think that undesirable state was achieved.

n This is a storytelling approach. Storytelling uses linear language and linear thinking

n Stories start in the past, while causal relationships start in the present

n Stories are linear while causal relationships follow the branches of an infinite set

n Stories use inference to communicate meaning, while problems are known by sensed causal relationships

The overriding theme of traditional Root Cause Analysis is the focus on the Root Cause. We can eliminate the problem if we eliminate the Root Cause.

This assumes the causal relationships are linear and that problems come from a single source.

18. Root Cause

† Apollo Root Cause Analysis: Effective Solution's to Everyday Problems, Dean L. GanoPerformance–Based Project Management®, Copyright © Glen B. Alleman, 2002 ― 2016 860

+

Stories seldom identify causes because they are busy setting the stage for who was where and when some action occurred.

A story is a sequence of events starting in the past, leading to the consequences disguised as a root cause

Root Cause Analysis is Not about Story Telling


861

+ Core Failure of Story Telling and the Filling Out of Formsn Stories rely on experience and judgment of the authors to connect

the causes of the problem. The mapping between Event, Cause, and Effect not provided in the story narrative.

n Story telling can be used to document the investigation and describe the corrective actions. But stories are poor in providing the analytical connections between cause and effect.

n Measures of the effectiveness of corrective actions can not be provided by narratives. Traceability between Effect, Action, and Condition can not be provided by the narrative.

n It is a false premise that analysis of a problem, its causes, effects, and solutions can be reduced to filling out a form and checking boxes.

18. Root Cause


862

+ Story Telling Is Not Good Root Cause Analysis Approach

n Story telling describes an event by relating people (who), places (where), and things (what) in a linear time frame (when).

n When using storytelling to analyze an event (system, outage for example), the causes identified in the report are difficult to follow and hinder our ability to understand the relationships between all the causes and provide a critique of the analysis.

n The investigators may well understand all the causal relationships, but because they are not presented causally it is difficult to know these relationships.

n Peer reviews will result in more questions because of the missing connections between Primary, Intermediate Effects, and their Causes.

18. Root Cause


863

+ Problem with the Story Telling Approach to Root Cause Analysis

n Stories start with the past – we saw this happen and something else happened after that, and then something else happened…n Causal relationships leading to the Root Cause start with the present

and work backwards to the causes – both Activities and Conditions of this cause.

n Stories are linear – they come from the minds of the story tellers, usually as a linear time line.n The linear understanding of an event in a time sequence from past to

present, ignores the cause–and–effect principle.

n Since we do not understand the branched causes, we use our own understanding of cause rather than the actual causal connections.

n Stories use inference to communicate causes.n Causal relationships require clear evidence of the existence of each

cause and its connection to the effects.

18. Root Cause


864

+ Root Cause Analysis is theEvent, the Cause, and the Resulting Effect

n We need a structured approach to investigating and analyzing significant adverse events or system deficiencies and their required improvement – not based on Story Telling.

n We need an approach that provides information and tools to be incorporated into risk management, quality management, independent verification and validation and improvement procedures in order to:n PREVENT future occurrence of adverse events that cause or can cause

undesired performance of our systems.

n CORRECT practices that have led to identified deficiencies.

n This approach separates story telling from the Primary Effect, and the cause–effect chain leading to the Primary Effect.

18. Root Cause


865

+ A Better Approach to Root Cause Analysis for Primary Effect, Cause, and Effect

n Direct causes often result from another set of causes – the intermediate causes – and these may be the result of still other causes.

n This chain of cause and effect needs to be revealed in a way that clearly points to the corrective actions.

n When a chain of cause and their effects is followed from a known end–state (time now), back to an origin or starting point, root causes are revealed and corrective actions can be applied.

n A root cause is an initiating cause of the causal chain which leads to an outcome or effect of interest.

18. Root Cause


866

+ Why Root Cause Analysis is Hard

n The problem is poorly defined.

n Systematic approach is not used to classify problem and cause.

n Investigations stopped prematurely – moving on to next problem.

n Decisions based on guesses, hunches or assumptions.

n Inadequate level of detail used to get to the Primary Effect.

n Interim containment fixes sometimes allowed to become "permanent.”

n Skills, knowledge, and experience needed to uncover the root cause not available.

n Lack of organizational will to address bigger issues.

n Fear of being blamed.

n I really don’t have time for this, we have bigger problems to solve.

18. Root Cause


867

+ The Problems with Categorical Thinkingn We need to put order to the things we perceive.

n This is a natural process, but creates laziness in our thinking processes.

n The notion of good and bad is categorical thinking at its base level.

n Categorical thinking creates the believe that once categorized, we can establish relationships, and act on the according to other perceived solutions.

n Filling out root cause forms or assigning elements to Fishbone charts reinforces the perception we can put the rot causes in categories (boxes) and assign solutions.

18. Root Cause


868

+ The Real Problem with Categorical Thinkingn When interacting with others, we assume there is a single reality

and therefore their categories are like ours.

n They are not.

n We assign value that establishes our basis of understanding and prejudices.

n If this is not recognized there is danger these prejudices set us up for failure when trying to produce an effective solution to the root cause.

18. Root Cause


869

+ Testing Answers from the Five Whys Question Streamn What evidence is there that this cause exists?

n Is it concrete? n Is it measurable?

n What evidence is there that this cause could lead to the observed effects? n Are we merely asserting causation without evidence?

n What evidence is there that this cause actually contributed to the Primary Effect? n Even given that it exists and could lead to this problem, how do we know it wasn't actually

something else?

n Is anything else needed, along with this cause, for the stated effect to occur? n Is it self–sufficient? n Is something needed to help it along?

n Can anything else, besides this cause, lead to the stated effect? n Are there alternative explanations that fit better? n What other risks are there?

18. Root Cause


870

To navigate the path to the actual Root Cause, we need to connect

the Action and Conditioncauses to the primary Effect and all the Intermediate Effects in

a single picture revealing the corrective actions that prevent

the Primary Effect in the future.

18. Root Cause


871

+

18.3 The Apollo Method

Nothing happens without a cause. Every time we ask WHY we must find at least two causes – the Action and the Condition in which that action causes the effect.

18. Root Cause


+

n Causes are never part of a Linear Chain found in standard Fishbone diagram or narrative approach.

n Look for causes to create the effect. Two causes are needed for each Effect.n Conditions – may exist prior to the Effect. Or conditions may be in motion or

active during the Effect. Conditions are the causes often ignored or beyond our knowledge.

n Actions – momentary causes that bring conditions together to cause an effect. Actions are causes most easily recognized.

n Connect all causes (Actions and Conditions) with a Caused By phrase to either an action or a condition.

n Support each Cause with evidence or an answered question.

For each Primary Effect we need ask why that Effect occurred.

Principles of the Cause and Effect Map

18. Root Cause


873

+ Five Steps to using the Five Whys

n Invite all affected parties to contribute to the map.

n Select the leader (someone trained in RCA)

n Ask Why 5 times for each topic area in the Cause and Effect map.

n Assign responsibilities for collected actual factual data.

n Publish the map.

Start this process using Apollo now.

18. Root Cause


874

+ The Apollo Principles

n Cause and Effect are the same thing n If we look closely at cause and effect, we see that a “cause” and an

“effect” are the same thing.

n A single thing may be both a cause and an effect.

n They differ only by how we perceive them in time.

n Each effect has at least two causes in the form of actions and conditions. n This is the most important and overlooked principle of causation.

n Unlike storytelling used to capture the Fishbone style charts, which focuses on linear action causes, reality demands that each effect have at least one action cause and one or more conditional causes.

18. Root Cause


875

+ Cause and Effect are the Same Thing

n The cause of one thing becomes the effect when you ask why.

n The cause of the “Injury” was a “Fall”, and when you ask why “Fall”, it changes to an effect and the cause is “Slipped.”

n This relationship continues as long as we continue to ask why.

Effects CauseInjury Caused by Fall

Fall Caused by Slipped

Slipped Caused by Wet Surface

Wet Surface Caused by Leaky Faucet

Leaky Faucet Caused By Seal Failure

Seal Failure Caused by Not Maintained

18. Root Cause


876

+ Each Effect Has At Least Two Causes in the form of Actions and Conditions

n Primary Effect – is any effect we want to prevent

n Action – momentary causes that bring condition together to cause an effect

n Conditions – the fundamental causal element of all that happens. It is made up of an effect and its immediate causes that represent a single causal relationship. n As a minimum, the causes in this

set consist of an action and one or more conditions.

n Causal sets, like causes, cannot exist alone.

n They are part of a continuum of causes with no beginning or end, which leads us to the next principle: n Causes and Effects are Part of

an Infinite Continuum of Causes.

18. Root Cause


877

+ Causes and Effects are Part of an Infinite Continuum of Causesn Causes are not linear.

n They branch out into at least two causes each time we ask why of an effect and if we ask why of each of those causes we find an ever expanding set of causes as shown in the example to the right.

18. Root Cause


878

+ An Effect Exists Only if Its Causes Exist in the Same Space and Time Frame

n Cause-and-effect relationships exist with or without the human understanding

n We perceive them relative to time and space.

n Every causal relationship is made up of conditional causes with a history of existence over time combining with an action cause in some defined time frame and existing in the same space to create an effect.

18. Root Cause


879

+

18.4 Four Phases of the Apollo Method

1. Define the Problem

2. Create the Cause and Effect Chart

3. Identify effective solutions

4. Implement the best solutionsThese four phases are the basis of discovering the corrective actions for the undesirable Effects we see in our development, testing, and deployment efforts.

18. Root Cause


+

1. Define the problem• What is the problem?• When did it happen?• Where did it happen?• What is the significance of

the problem?

2. Create the Cause and Effect chart• For the primary Effect, ask

Why did this happen• Look for causes in Actions

and Conditions• Connect all the causes with

Caused By for the next cause and its effect

• Support causes with evidence or an open Question

3. Identify effective solutions must• Prevent recurrence• Be within our control• Meet our goals and

objectives

4. Implement the best solutions• Measure the effectiveness of

these solutions in units defined in the Action and Condition causes

Four Phases of the Apollo Method18. Root Cause


881

+ No Fishbone Charts or Narratives Allowed in the Apollo Method

18. Root Cause


882

+ The Flaw of our Linear Thinking Processn Like a string of dominos, asking why in the conventional Five Whys

method assumes, A caused B, B caused C, and C caused D.

n At the end of this chain we believe the Root Cause of the undesirable outcome can be found.

n In the traditional Fishbone approach we are looking for the event that caused the Effect.

n Instead we need to find the Actions and Conditions that ALLOWED the event to happen.

n These Actions and Conditions are the actual Root Cause.

18. Root Cause


883

+ The Principles of the Apollo Methodn Cause and Effect are the same thing.

n If we look closely at cause and effect, we see that a “cause” and an “effect” are the same thing.n A single thing may be both a cause and an effect. n They differ only by how we perceive them in time.

n Each Effect has at least two causes in the form of Actions and Conditions. n This is the most important and overlooked principle of causation. n Unlike storytelling used to capture the Fishbone style charts, which

focuses on linear action causes, reality demands that each effect have at least one action cause and one or more conditional causes.

n Causes and Effects are part of a continuum of causes.

n An Effect exists only if its causes exist in the same space and time frame.

18. Root Cause


884

+ Cause and Effect are the Same Thing

n The cause of one thing becomes the effect when you connect caused by.

n The cause of the “Injury” was a “Fall”, and when you ask why “Fall”, it changes to an effect and the cause is “Slip.”

n This relationship continues as long as we continue to ask why.

Effect Caused by Action or ConditionInjury Caused by Fall

Fall Caused by Slip

Slip Caused by Wet Surface

Wet Surface Caused by Leaky Faucet

Leaky Faucet Caused By Seal Failure

Seal Failure Caused by Seal Not Maintained

18. Root Cause


885

+ An Effect Exists Only if Its Causes Exist in the same Space and Time Frame

n Cause–and–effect relationships exist with or without the human understanding.

n We perceive them relative to time and space.

n Every causal relationship is made up of Conditionalcauses with a history of existence over time, combining with an Actioncause in some defined time frame and existing in the same space to create an effect.

18. Root Cause


886

+ The Apollo Method structures our current information collection method to identify solutions to the Root Cause

Every time-series entry in our current narrative method is an ACTION cause.

By focusing on ACTION’s and not associated CONDITION causes, we

leave out important causes that can be acted on to provide an effective

SOLUTION.

18. Root Cause


887

+ Example of a Naïve RCA18. Root Cause


888

Why?

Action Cause?

Condition Cause?

Without following the Why’s to the terminal node with Action and Condition, the actual Root Cause is still buried in the narrative, waiting to reoccur

+ Actual System Outage Root Cause Analysis using Apollo Method

889


Each stopping condition points to the Root cause.Until these are found only symptoms are

discovered, not the Root cause

18. Root Cause

+ Missing Elements of Success

n Linear thought process n A caused B, B caused C, C caused D.

n No causal chain from Primary Effect to related Cause and Effect

n No Actions and Conditions to connect to the Effectn Under what conditions was the Effect observed?

n What actions triggered the Effect?

n No evidence for each Action and Condition

n Stopping too soon, before actual Root Cause found

Every Effect is caused by momentary Action coming together with existing Condition in the same time

and space.

18. Root Cause


890

+

18.5 Seven Steps to Discovery

1. Define the Problem.

2. Determine the Causal Relationships.

3. Provide a Graphical Representation of Cause and Effect that is not linear thinking.

4. Provide Evidence for each Cause and Effect.

5. Determine if each Cause is Sufficient and Necessary.

6. Identify Effective Solutions � THIS IS WHAT WE’RE AFTER.§ Finding the cause is needed.§ Preventing the effect is needed.§ But installing an effective solution is

the desired business outcome.

7. Implement And Track the Effective Solutions.

Effective problem solving and Strategies for business success that move away from blame finding and linear thinking of Fish Bone diagrams.

And move toward finding the interconnected factors where Cause and Effect are intertwined.

These Seven Steps expand on the Four phases, to further detail the process of arriving at the Root Cause

18. Root Cause


+ Seven Steps of the Apollo Method

1. Define the Problem.

2. Determine the Causal Relationships.

3. Provide a Graphical Representation of Cause and Effect that is not linear thinking.

4. Provide Evidence for each Cause and Effect.

5. Determine if each Cause is Sufficient and Necessary.

6. Identify Effective Solutions

§ Finding the cause is needed.§ Preventing the effect is needed.§ But installing an effective solution is the desired business outcome.

7. Implement And Track the Effective Solutions.

18. Root Cause


892

This is what we’re after in Root Cause Analysis

+ An Effective Solution …

n Prevents recurrence of the Primary Effect.

n Assures corrections and prevents actions within our control.

n Meets our goals and objectives, including a solution that …n Does not cause unacceptable problems.

n Prevents similar occurrences.

n Provides reasonable value for the cost.

18. Root Cause


893

+

A Recent Example of Cause and Effect Analysis

18. Root Cause


894

+

18.6 An Ugly Truth About Root Causes

The truth? You can’t handle the Truth!

18. Root Cause


+ In the end it usually comes down to the People. The People are the Root Cause

This does NOT and CAN NOT mean a Blame Game. If we ask properly people will see their role in the failure of the process.Human beings cause problems, not tools, processes, systems, or culture.

18. Root Cause


896

+ This is our core problem†

n Getting the staff engaged in Preventing problems not just Correcting problems meansn If I do this what will happen?

n Is this the right thing to do at this time?

n Is this really what the customer wanted me to do?

n Did I consider the impact of my actions?

n If I don’t have time to be careful, then what damage will result from my actions?

n Did I consult with someone who knows more than I do about what the solution should look like?

n Am I being as careful as I should be, when I make a change?

n Am I being pressured to do something I know isn’t the right thing to do?

If the people change, the culture will change† IEEE 13th Annual Workshop on Human Performance / Root Cause / Trending / Operating Experience / Self Assessment, August 26-31,

2007, Monterey Marriott, Monterey, CA

18. Root Cause


897

+

18.7 Effective Problem Solving Culture

n Critical Elements

n Infrastructure

n Management Buy-In and Support

n Crating a Program Champion

n Dedicated Incident Investigators

n Integration

n Implementation Strategies

n Deployment

Every organization is different, so before implementing this plan you should determine who the players are and what level of training they need.

http://www.realitycharting.com/training/problemsolvingculture/plan

18. Root Cause


+

n Exposed to the principles of causation to understand that “stuff” does not just happen.

n Should know that we can find effective solutions to event-based problems by using RealityCharting® and RC Simplified™.

n Must understand that different perspectives are a key to effective solutions and easily accommodated when you use the Reality Charting process.

n Must know their role in defining problems and finding effective solutions to prevent recurrence.

n Must know that management is behind this initiative.

The Work Force should

Critical Elements18. Root Cause


899

+

n Top-level management support.

n A Program Champion.

n Dedicated Incident Investigation Facilitators.

n Incorporation of the Reality Charting process into existing procedures and protocol.

n Involve every employee in this effective problem-solving initiative.

n Utilize RealityCharting®, the RC Learning Center™, and RC Simplified™ throughout the organization.

The Plan musty implement

Infrastructure18. Root Cause


900

+

n Show the Effective Problem-Solving processes to top-level managers so they know the principles of effective problem-solving.

n Show RealityCharting® Overview processes to all managers so they know what the software does and why it is so effective.

n Read the RealityCharting – Seven Steps to Effective Problem Solving and Personal Success and work the interactive exercises in this book.

n Provide a manager’s workshop on Managing Effective Problem-Solving. Discuss application to Defect Elimination, Continuous Improvement programs n Six Sigma, Lean, Proactive Maintenance, Chronic (Systemic) issues.

Top level support of the RCA process is done

Management Buy-In and Support18. Root Cause


901

+

n Qualificationsn An experienced incident investigator and facilitator.

n Trainingn Familiar with RCA concepts

n Familiar with Apollo method

n User of reality Charting®

n Rolen The Go To persons for all things RCA

n Effectively facilitate incident investigations

The Program Champion is the Cheer Leader for the RCA Improvement initiative

Create a Program Champion18. Root Cause


902

+ Dedicated Incident Investigator

n Identify stakeholders ‒ who have to solve problems as part of their daily work scope. n System or Process Engineers

n Supervisors or Team Leaders

n Some Managers

n Training ‒ online training takes ½ the time classroom training does

n Mentoring ‒ program champion and investigators mentor practitioners and support them to develop skills for conducting Root Cause Analyses

18. Root Cause


903

+ Integration

n Entry Points ‒ Identify problem-solving entry points by reviewing the work processes. n Problems can occur anywhere and evaluation includes what the actions of

any employee should be if they identify a problem.

n This may include Non-Conformance Reports, Corrective Action Reports, and Customer Feedback Forms.

n Minor Problems ‒ If problem is below the threshold criteria for performing a full-blown RCA, use Simplified process to document the problem.

18. Root Cause


904

+ Implementation Strategy

n Initial Findings ‒ If the initial problem analysis provides effective solutions, implement them according to existing procedures and approval protocol.n Send the final analysis to the Program Champion for review and approval and

put into the organizations tracking and trending system and/or print, or otherwise transmit the final copy to stakeholders who may need it.

n The Program Champion will review to determine if the problem exceeds the threshold criteria for a formal investigation and respond accordingly.

n This review provides the Program Champion oversight of how well the program is working and when and where to adjust it.

n Incomplete Analysis ‒ If the event-analysis does not find effective solutions, then send the initial analysis to Program Champion n The Program Champion will determine if the problem exceeds the threshold

criteria for a formal investigation and respond accordingly.n The Program Champion determines if analysis can be finished or if a team is

needed to work the issue.n If a team is required, Program Champion will gather team and work with them

to finalize the analysis.

18. Root Cause


905

+ Deployment18. Root Cause


906

+

18.7 Root Cause Analysis in Agile Software Development

n Do just enough Root Cause Analysis sessions

n Have a knowledgeable facilitator

n Communicate the corrective actions

n Do Agile Root Cause AnalysisRoot Cause Analysis is used in software development to build a shared understanding of a problem to determine the first or “root” causes.

Knowing these causes identifies effective improvement actions to prevent similar problems in the future.

Root Cause Analysis in Agile stops problems that have been inhibiting the team for too long.


18. Root Cause

907

+ Do Just Enough RCA

n For any problem to be investigated, the loss or potential for loss must be significant to the business or the projectn The definition of significant is held by those paying never by those

providing the solution

n The loss needs to monetized or defined in some unit of measure meaningful to the decision makers

n There must exist a significant probability that similar problems will occur in the future if no corrective action is taken


18. Root Cause

908

+ Knowledgeable Facilitator

n The team itself cannot produce a credible Root Cause Analysis

n Facilitation of the process is needed by a separate party

n This can be: n Quality Assurance

n Systems Architecture

n Program Management

n Release Management

n The facilitator must follow a process ‒ the Apollo Method is one shown heren Tools are needed to capture the information and avoid the narrative

biases also shown here


18. Root Cause

909

+ Communicate the Corrective Actionsn Finding the Root Cause is just the start.

n Taking corrective actions is next, but that starts with communicating to all stakeholders what those actions are.

n The communication must shown The problem

n The Root Cause

n The Corrective Actions

n The expected outcomes in quantifiable measures of effectiveness and performance


18. Root Cause

910


Root Causes Have Root CausesIt’s not a matter of asking 5 Whys

You must keep asking until you reach the end of the chain or action and condition

18. Root Cause


“Kto Nie Pamięta Historie Skazany

Jest Na Jej Ponowne Preżycie” ‒ George

Santayana

“Those Who Do Not Remember the Past are Condemned to Repeat it” ‒ George

Santayana

18. Root Cause

+ Root Cause Analysis

n Events have two contributors ‒ the condition and the action. Both must be found before root cause can be determined for the primary effect.

n There is an endless chain of cause and effect, stopping too early is a common failure mode of Root Cause Analysis

n Formal Root Cause Analysis processes and tools provide information not available with the narrative approach

n Without determining the Root Cause and suggested solution just address the symptoms. This approach does not remove the cause and allows the symptoms to recur in a loop of fix, break, fix.


913

18. Root Cause

root cause analysis master plan

Technology

root cause analysis2

cause of events

root causes

problem management

causes of problems

different problems

method of problem

persistent problem