quality metrics

Copyright T. M. Kubiak 1996-2010

Performance Improvement Solutions

Conquering the Challenge!™ 1

Presented to

ASQ St. Petersburg-Tampa-Section 1508

08 February 2010

by

T. M. Kubiak, President


Developing Meaningful Metrics:

Driving Action for Results!




If you don’t have confidence in

the diagnosis, you won’t have

confidence in the prescription.

— Steven Covey




Can any of you help me with

metrics? My boss wants a

whole new set by the end of

the week and I don’t even

know where to start! HELP!




The Driving Force

Definitions

Why Measure?

Barriers to Measurement

Pitfalls to Avoid

What Measures Measure

The “E’s” of Measurement

Different Views of Measures

Overview

Characteristics of Measures

Development Considerations

Principles of Measurement

Where Should You Measure?

Goals, Targets, & Benchmarks

Evolution of a Measure

Examples of Measures

Some Final Words...




Measure (Metric)

Refers to numerical information that quantify

input, output, and performance of processes,

products, and services

Indicator

Measurement that relates to performance, but

is not a direct or exclusive measure of such

performance

Measure that is a predicator of some more

significant performance

A Few Definitions, Of Course!

Measure, Metrics, and Indicators! Oh my!




If you cannot measure it, you cannot control it.

If you cannot control it, you cannot manage it.

If you cannot manage it, you cannot improve it.

What gets measured and rewarded is what gets

done. Corollary: If the measurements don’t

change, neither do the results.

Measurements allow us to track where we have

been, where we are, and where we are going

Shows how effectively we use our resources

Why Measure?

Measurement: The Roadmap to Progress




Focuses attention on factors contributing to achieving the organization’s mission

Assists in setting goals and monitoring trends and progress

Provides input for analyzing root cause and sources of errors

Identifies opportunities for on-going improvement

Provides a means of knowing whether we’re winning or losing

More Reasons for Measuring

Measurement: The Roadmap to Progress



Conquering the Challenge!™

Integrated System of Metrics

Driving Customer Satisfaction and Business Success!

Corporate

Objectives

Business

Requirements

Corporate

Strategy

Operational

Processes

Customer

Requirements

Business

Strategy

Support

Processes

Supplier

ProcessesAre these processes capable of

meeting business requirements?

Lean Six Sigma drives

improvement at the individual

process level.

The Driving Force




Thinking some jobs can’t be measured

Thinking there isn’t enough time to measure

The process of accomplishing a task is not

understood

Fear that the measurement will become a club

Barriers to Measurement




Using the measure as a carrot or a stick

Reason: the goal becomes to manipulate the number

Pitfalls to Avoid!




Measuring everything that is possibly helpful

Reason: dilutes the effort

Pitfalls to Avoid!




Measuring parameters where the organization excels

Reason: reduces the effort to a publicity campaign

Pitfalls to Avoid!




Requiring precision in the data beyond the

requirements of the decision

Reason: exasperates all involved

Pitfalls to Avoid!




Quality (acceptable, good, in-spec, etc.)

Quantity (Number of...)

Time (Cycle time, rates)

Cost

What Measures Measure




Extent (volume-based or deployment)

How much?

Efficiency (productivity-based, usually

expressed as a rate)

How fast?

Effectiveness (customer-based or

objective-based)

How well?

The “E’s” of Measurement!

If It Were Only That Easy!




W IPM W IPM W IPM W EOPM

W IPM W IPM W IPM W EOPM

Organizational level

Operational measures (invisible to the customer)

Predictive measures (in line of sight of the customer)

Process Level

In-process measures (evaluate the method that

creates results)

End-of-process measures (measure the results of a

method)

Different Views of Measures




Actionable (So What?)

Easily measurable

Economical to collect

Focused on processes

Objective

Relevant

Specific

Timely

Traceable

Understandable

Characteristics of Measures

If It Ain’t Actionable...




How often will the measure be computed and

posted?

What about data stratification/aggregation?

Where will the data come from?

Is the data source reliable?

Can/should the data be normalized?

How will the measure be presented?

Who will see them?

Development Considerations

Critical Questions Require Thoughtful Answers




Answers: Who, How, What, Where, When

Defines: formulas and terms

Defines: interpretation (e.g., up is good)

Example:

Operational Definition

Establishes a common language and understanding

What is a “month?”




Cash for Clunkers Example Issue: 78 cars bumped from the list

Threshold: 18 mpg

Reason: More precise data (4 decimals) caused the revisions (Note: 0.0001 miles is ½ drop of fuel based on 18 mpg)

Measurement Methods:

Older: Dynamometer, fuel consumption measured, measured to 4 decimals, numbers rounded

Newer: Tailpipe emissions, measured to 4 decimals

Fix: Old data updated to be “effectively equivalent”

Quote: “Repeatability and accuracy is something we spend a lot of time on.” EPA scientist

21

Hey, Joe! Gotta light?

Source: WSJ 09-08-05




Cancer Deaths Example

Normalized

Stratified by age

•Gaps in age

stratification

•Aggregated across

gender

•Aggregated across

cancer type

Does this chart support

the conclusion?

Source: WSJ 09-02-09




Pareto Chart Example

23

0

2

4

6

8

10

12

14

16

18

20

Def 1 Def 2 Def 3 Def 4 Other

You can’t fix stupid!

0

2

4

6

8

10

12

14

16

18

20

Def 1

Def 3

Def 5

Def 7

Def 9

Def 11




Air Travel Example

What is your understanding of an “on-time arrival?”

How does plus or minus 15 minutes of the scheduled arrival time sound?

Is this a good metric or a bad metric?

How many customers are aware of it?

Related metrics (by carrier):

Percentage of flights arriving on-time

Percentage of canceled flights

Mishandled bag reports per 1,000 passengers

Complaint reports per 1,000,000 passengers

Bumped passengers per 10,000

24

Do key players understand the metrics?

Source: WSJ 1--01-07




ExtentEfficiency

Effectiveness

Measurements must be specific

Measure the outputs of highest value to

the customerMeasures can be applied to all performance

dimensions - external as well as internal

Understand the game before you decide

how you’ll keep score

Measure the process as well as the results

There is no single perfect measure

Principles of Measurement




Early in the process to promote prevention

At the point where a cause-and-effect

relationship can be established

Functional boundaries

Points of convergence

Points of divergence

At critical processes

Where Should You Measure?




Goals, Targets, & Benchmarks




If you can’t win the race, make

sure the guy ahead of you

breaks the record.

— Steve Prefontaine (1951-

1975), American runner




Extent

Efficiency

Effectiveness

Number of computer reports

delivered

Number of acceptable computer

reports delivered on time

Number of computer reports

delivered on time

Percent of acceptable computer

reports delivered on time

Evolution of a Measure




The value of this measure is unclear. It is not

customer-focused nor does it measure the

ability of an organization to deal with incoming

trouble calls since no processing of trouble

calls is evident in the metric. Essentially, this

metric measures nothing more than the

volume of incoming trouble calls. The fact that

it is expressed in the form of a rate is

misleading and may fool one into thinking it is

a measure of efficiency. Type of Measure: Extent

Measure: Trouble calls received/week





This metric focuses on measuring nothing

more than the volume of jobs scheduled. Other

than that, it provides little insight into the

underlying process of “scheduling jobs.” The

metric is not actionable.

Type of Measure: Extent

Measure: Job Scheduled





Measures the planning/forecasting process.

Type of Measure: Effectiveness

Measure: Actual versus planned sales





This metric is highly misleading and often

used for organization to organization

comparisons. It falsely assumes that all

employees generate sales and does not

consider the product being sold or the

associated supporting infrastructure required

to produce and sell the product.

Type of Measure: Efficiency

Measure: Sales/employee





This metric measures the deployment of

training hours to the workforce.

Type of Measure: Extent

Measure: Number of training hours/employee





This metric measures the impact of training the

sales workforce so long as a clear cause-and-

effect relationship can be established between

training and revenue.


Measure: Revenue/sales training hour





This metric measure the design process from

the point of view of the drawings. However,

note that this metric does not normalize or

adjust for the complexity of a given drawing.


Measure: Engineering change orders/drawing





Measure, Control, Manage, Improve

Extent, Efficiency, Effectiveness

Performance measurement is

irrelevant without a frame of

reference

Measure twice! But cut once!

Some Final Words...




People without information are

unable to change; those with

information are compelled to

change.

quality metrics

Education

performance of processes

ongoing improvement

exclusive measure

measure metric

measures measure evolution

resources measurement

services indicator measurement

business success