Eight Common Statistical Traps

Presented by Bob MitchellPast Chair, ASQ Statistics Division

Fellow, ASQ

Hiawatha SectionJanuary 8, 2004

ASQ Statistics Division

Vision:Statistical Thinking Everywhere

Process Variation Data Improvement

Statistical Thinking

Philosophy ActionAnalysis

Statistical Methods

Motto:We help keep you employable

Data Sanity

Statistical Thinking applied to everyday data

Statistical Thinking

• All work is a process• Variation exists in all processes• Knowledge and management of

variation are keys to success.

Process

ProcessOutputs

Customers

Feedback

Inputs

Suppliers

S I P O C

Process Metrics

Statistics

Statistics is not merely the science of analyzing data, but the art and science of collecting and analyzing data.

Use of Data as a Process

Given any improvement situation one must be able to:

1. Choose and define the problem in a process/ systems context

2. Design and manage a series of simple, efficient data collections

3. Use comprehensive methods presentable and understood across all layers of the organization (graphical analyses)

4. Numerically assess the current state, assess the effects of interventions, and hold the gains of any improvements made.

Use of Data as a ProcessPeople, Methods, Materials, Machines, Environment and Measurements inputs can be a source of variation for any one of the measurement, collection, analysis or interpretation processes!

Statistical TrapsCommon errors in data use, display and collection.

Organizations tend to have wide gaps in knowledge regarding the proper use, display and collection of data. These result in a natural tendency to either react to anecdotal data, or “tamper”.

Trap 1Treating all observed variation in a time series data sequence as Special Cause

• Most common form of “tampering” –treating common cause variation as special cause.

• Given two numbers, one will be bigger!

Two-Point Comparison

What Action is Appropriate?What Action is Appropriate?So

met

hing

Impo

rtan

t

This PeriodLast Period

Common vs. Special

Common CauseCommon Cause Special CauseSpecial Cause

It Depends!It Depends!

Example – Trap 1

Change from Change fromRegion Q4 (000) Q3 Q4 last year

Northeast 1148 17.6% 20.6%Southwest 1337 11.7% 11.8%Northwest 806 17.2% (8.2%)North Central 702 (5.5%) 4.7%Mid-Atlantic 781 (3.2%) (2.6%)South Central 359 (19.7%) (22.3%)

Northeast

Observation

Ind

ivid

ua

l V

alu

e

2018161412108642

1400

1200

1000

800

600

_X=1000.0

UC L=1367.5

LC L=632.4

Observation

Mo

vin

g R

an

ge

2018161412108642

600

450

300

150

0

__MR=138.2

UC L=451.6

LC L=0

1

1

I-MR Chart of Northeast

Southwest

Observation

Ind

ivid

ua

l V

alu

e

2018161412108642

1600

1400

1200

1000

_X=1195.9

UC L=1534.5

LC L=857.3

Observation

Mo

vin

g R

an

ge

2018161412108642

400

300

200

100

0

__MR=127.3

UC L=416.0

LC L=0

I-MR Chart of Southwest

Northwest

Obser vation

Ind

ivid

ua

lV

alu

e

2018161412108642

1400

1200

1000

800

600

_X=1025.9

UC L=1298.0

LC L=753.8

Obser vation

Mo

vin

gR

an

ge

2018161412108642

300

200

100

0

__MR=102.3

UC L=334.3

LC L=0

1

1

I-MR Chart of Northwest

North Central

Observation

Ind

ivid

ua

l V

alu

e

2018161412108642

700

600

500

400

_X=519.6

UC L=669.7

LC L=369.4

Observation

Mo

vin

g R

an

ge

2018161412108642

200

150

100

50

0

__MR=56.5

UC L=184.5

LC L=0

1

1

11

I-MR Chart of N. Central

Mid-Atlantic

Observation

Ind

ivid

ua

l V

alu

e

2018161412108642

800

700

600

500

_X=656.7

UC L=750.2

LC L=563.2

Observation

Mo

vin

g R

an

ge

2018161412108642

100

75

50

25

0

__MR=35.2

UC L=114.9

LC L=0

1

1

1

1

111

1

I-MR Chart of Mid-Atlantic

South Central

Observation

Ind

ivid

ua

l V

alu

e

2018161412108642

500

450

400

350

300

_X=420.3

UC L=534.2

LC L=306.4

Observation

Mo

vin

g R

an

ge

2018161412108642

160

120

80

40

0

__MR=42.8

UC L=140.0

LC L=0

I-MR Chart of S. Central

Trap 2

• Fitting inappropriate trend lines to a time-series data sequence

• Another form of tampering –attributing a specific type of special cause (linear trend) to a set of data that contains only common cause.

Trends

Example – Trap 2

It generally takes a run of length seven to declare a sequence a true trend. If the total number of observations is 20 or less, SIX continuously increasing or decreasing points can be used to declare a trend.

Trap 3

• Unnecessary obsession with and incorrect application of the Normal distribution

• A case of “reverse” tampering – treating special cause as common cause.

• Can cause misleading estimates and inappropriate prediction of process outputs.

Normal Distribution

The “Normal” Distribution is a distribution of data that has certain consistent properties.

These properties are very useful in our understanding of the characteristics of the underlying process from which the data were obtained.

Many natural phenomena and man-made processes are distributed normally, or can be represented as approximately normal.

ND PropertiesProperty 1Property 1:: A normal distribution can be described completely by knowing only the:

− Mean

− Standard deviation

Distribution OneDistribution One

Distribution TwoDistribution Two

Distribution ThreeDistribution Three

ND PropertiesProperty 2Property 2:: The area under sections of the curve can be used to estimate the cumulative probability of a certain “event” occurring

-4 -3 -2 -1 0 1 2 3 4

95%

99.73%

68%

Empirical RuleThe previous rules of probability apply even when a set of data is not perfectly normally distributed

Number of Standard

Deviations

Theoretical

Normal

Empirical – Almost any distribution

+/- 1σ 68%

60-75%

+/- 2σ 95%

90-98%

+/- 3σ 99.7%

99-100%

Example – Trap 33 potential vendors for new process

Customer information for paper tearTarget 26.5 g/cmSpecifications 23-30 g/cm

QC summary data (based on 30 jumbos from each vendor):

Vendor Mean StdevA 26.5 0.98B 26.6 0.67C 26.6 0.82

Cost of A < Cost of B < Cost of C

Graphical Analysis

24.4 25.4 26.4 27.4 28.4

Dotplot for Vendor A-Vendor C

Vendor A

Vendor B

Vendor C

Vendor CVendor BVendor A

28.5

27.5

26.5

25.5

24.5

Vendor

Tear

ANOVAOne-way ANOVA: Tear versus VendorSource DF SS MS F PVendor 2 0.160 0.080 0.12 0.890Error 87 59.931 0.689

Total 89 60.091

Individual 95% CIs For Mean Based on

Pooled StDev

Level N Mean StDev ---------+---------+---------+---------+

Vendor A 30 26.523 0.976 (--------------*--------------)

Vendor B 30 26.573 0.668 (--------------*--------------)

Vendor C 30 26.627 0.817 (--------------*--------------)

---------+---------+---------+---------+

26.40 26.60 26.80 27.00

p-value > .05; therefore, “no statistically significant difference”

Plot the Bloody Dots !

10 20 3024

25

26

27

28

29

Index

Vend

or A

10 20 3024

25

26

27

28

29

Index

Vend

or B

10 20 3024

25

26

27

28

29

Index

Vend

or C

Trap 4

• Incorrect calculation of standard deviation and sigma limits

• The traditional calculation of standard deviation typically yields a grossly inflated variation limit.

• Some people have arbitrarily changed decision limits to two standard deviations from the average.

Example – Trap 4

Observation

Indi

vidu

al V

alue

30272421181512963

5

4

3

2

1

_X=3.039

UCL=5.043

LCL=1.035

I Chart of Length of Stay

Decision limits based on overall standard deviation = 0.668Too wide for the process.

Example – Trap 4

Observation

Indi

vidu

al V

alue

30272421181512963

4.5

4.0

3.5

3.0

2.5

2.0

_X=3.039

UCL=4.049

LCL=2.028

11

1

1111

I Chart of Length of Stay

Decision limits based on process sigma.

Trap 5

• Misreading special cause signals on a control chart

• Just because an observation is outside the calculated three standard deviation decision limits does not necessarily mean that the special cause occurred at that point.

Example - Trap 5

Observation

Indi

vidu

al V

alue

30272421181512963

4.5

4.0

3.5

3.0

2.5

2.0

_X=3.039

UCL=4.049

LCL=2.028

11

1

1111

I Chart of Length of Stay

Special cause signals are the result of mean shifts

Example – Trap 5

Observation

Indi

vidu

al V

alue

30272421181512963

5

4

3

2

1

_X=4

UCL=4.754

LCL=3.246

1 2 3

I Chart of Length of Stay by Level

Correct Interpretation

Trap 6

• Choosing arbitrary cutoffs for “above” average and “below” average values

• There is actually a “dead band” of common cause variation on either side of an average that is determined from the data themselves.

Dead BandCoin Toss

With a “fair” coin, we expect about 50% headsWith 20 flips, would you be suspicious about the coin’s fairness…

– If someone flipped 11 heads?– If someone flipped 19 heads?

When would you become suspicious? – Draw lines below

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Confidence IntervalsFlipping 20 coins and looking for heads produces the following distribution:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Empirical Rule - Revisited

−3σ −2σ −1σ 0 −3σ −2σ −1σ 0 1σ 2σ 3σ 1σ 2σ 3σ

µ + 1σ (60-75% of Obs)

µ + 2σ (90-98% of Obs)

µ + 3σ (99-100% of Obs)

Example – Trap 6MD Incidents PTCA Incid / PTCA

1 1 36 2.78 %

2 4 53 7.55 %

3 4 79 5.06 %

4 2 58 3.45 %

5 5 110 4.55 %

Total 16 336 4.76 %

Are physicians 2 & 3 really “above average”?

By definition, approx 50% of the data will be below and 50% above the mean

Example – Trap 6

Trap 7

• Improving processes through arbitrary numerical goals and standards

• Any process output has a natural, inherent capability within a common cause range.

• Goals are merely wishes.

Trap 8

• Using statistical techniques on “rolling” or “moving” averages

• Another form of tampering –attributing special cause to a set of data that may only contain common cause variation, plus some structure.

• The rolling average technique creates the appearance of special cause

Rolling Averages

Structural VariationExamples:

– Sales steadily increasing (true trend)– Seasonal pattern in revenues– Business cycle patterns for orders

SALES

-1000

0

1000

2000

3000

4000

5000

6000

Mar

-95

Apr-9

5

May

-95

Jun-

95

Jul-9

5

Aug-

95

Sep-

95

Oct

-95

Nov

-95

Dec

-95

Jan-

96

Feb-

96

Mar

-96

Apr-9

6

May

-96

Jun-

96

Jul-9

6

Aug-

96

Sep-

96

Oct

-96

Nov

-96

Dec

-96

Jan-

97

Feb-

97

Mar

-97

MONTH, YR

Indi

vidu

al V

alue

Special Cause Flag

Seasonal or periodic patternTrend over time

For more information…

www.asqstatdiv.org

Thank You !