components of measurement error - gage rr - the last stand - bw print version 5-20-08

1

Components of Measurement Components of Measurement Components of Measurement Components of Measurement

System AnalysisSystem AnalysisSystem AnalysisSystem Analysis

Gage RGage Repeatabilityepeatability eproducibilityeproducibility&& RR

and more………………………………

2

Components of Measurement System AnalysisComponents of Measurement System AnalysisComponents of Measurement System AnalysisComponents of Measurement System AnalysisComponents of Measurement System AnalysisComponents of Measurement System AnalysisComponents of Measurement System AnalysisComponents of Measurement System Analysis

Each component of a measurement system contributes to variation,which affects the decisions being made

• Resolution / Discrimination

• Accuracy (bias effects)

• Linearity

• Stability (consistency)

• Repeatability (test-retest)

• Reproducibility

3

Definitions:Definitions:Definitions:Definitions:Definitions:Definitions:Definitions:Definitions:

• Resolution/Discrimination

– Capability to detect the smallest significant change.

Guideline: “10 Bucket Rule”

– Increments in the measurement system should be one-tenth

the product specification or process variation. For

example, if a length is being measured to the nearest

millimeter, the measurement system resolution must at

least be to the nearest 0.1 mm.

Measurement units that are too large mask the

variation present in the components being measured.

4

What does this mean?

Measure the same parts twice.

Reading one is plotted along the

X-Axis and Reading two is

plotted along the Y-Axis.

The thinner the sausage, the

more precise the

measurements.

The greater the aspect ratio

(Length/Repeatability), the more

accurate the measurement as

long as the readings are on the

45°line.

This type of plot is the basis for

a type of gage R & R called a

Youdin plot.

Accuracy and PrecisionAccuracy and Precision

"Accurate and Precise"

L

repeatability

Accuracy: Closeness to the true value,

or to an accepted reference value

Precise: The variation seen when the

same part is measured repeatedly with

the same device.

5

Accuracy and PrecisionAccuracy and Precision

"Precise but Inaccurate"

Error

Bias

"Accurate but Not Precise"

"Inaccurate and Not Precise"

"Accurate and Precise"

L

repeatability

6

Linearity

Full Range of Gage (Reference Value)

No Bias

Observ

ed

Avera

ge V

alu

eBias

(Reference Value(Reference Value

Equals observedEquals observed

Average value)Average value)

Note: Bias + Increasing Variance

The change in bias over

the normal operating range The difference between the

observed average of measurements

and the reference value

7

StabilityStabilityStabilityStabilityStabilityStabilityStabilityStability

• Measurements remain constant and predictable over time

– For both mean and standard deviation

Time 2

Time 1

Master Value(Reference Standard)

This is not stable in time because:

Although the Time 1 data and the Time 2 data are both clustered, the

locations are not the same.

– Evaluated using control charts

– No drifts, sudden shifts, cycles, etc.

The change in bias over time

8

RepeatabilityRepeatabilityRepeatabilityRepeatabilityRepeatabilityRepeatabilityRepeatabilityRepeatability

• Variation that occurs when repeated measurements are made of the same

item under identical conditions

– Same:

• Repeatability is affected by:

• Operator

• Set-up

• Units

• Environmental conditions

– Repairing, replacing, adjusting equipment

– Not following or incomplete SOP (Standard Operating Procedures)

• Operators not following the Operator Instructions

• Inadequate Operator Instructions.

9

ReproducibilityReproducibilityReproducibilityReproducibilityReproducibilityReproducibilityReproducibilityReproducibility

The variation that results when different conditions are used to make the

measurements

– Different:

• Set-ups

• Environmental conditions

Reproducibility is affected by:

• Operators

• Test units

• Locations

• Companies

• Operator to operator differences

• Operator to part interaction (on the gage),

10

Good Precision &

Good AccuracyPoor Precision

A = S + R

σ2total = σ2

product/process + σ2repeatability + σ2

reproducibility

Master Value

A B

ll of it ome of it est of it

11

σ2Total = σ2

R&R + σ2Process output

σ2Total = σ2

Repeat + σ2Reproducibility + σ2

Process output

σ2Total = σ2

Repeat + σ2Oper + σ2

Oper • Process output + σ2Process output

The Big Picture: Linking Them All Together

12

Keys To Successful Keys To Successful Keys To Successful Keys To Successful Keys To Successful Keys To Successful Keys To Successful Keys To Successful

Measurement System AnalysisMeasurement System AnalysisMeasurement System AnalysisMeasurement System AnalysisMeasurement System AnalysisMeasurement System AnalysisMeasurement System AnalysisMeasurement System Analysis

• Establish on-going assessment criteria and schedules

• Define and validate measurement process

• Identify known elements of the measurement process

(operators, gauges, SOP, setup, etc.)

• Clarify purpose and strategy for evaluation

• Set acceptance criteria

• Implement preventive/corrective action procedures

13

Variable DataVariable DataVariable DataVariable Data

Measurement System Analysis

14

Variable DataVariable DataVariable DataVariable DataVariable DataVariable DataVariable DataVariable Data• Make sure that the samples that are utilized cover the full

range of variation. The supplier should know how much variation is expected.

– This does not necessarily mean that we use the entire tolerance band.

• Ensure that the data is analyzed using the ANOVA method (as opposed to the X bar R method).

• Ensure that the number reported is % of study variation, or (% of tolerance, whichever is appropriate). If you have questions about when % of tolerance is appropriate, see your MBB.

• Make sure that the samples are not collected in a row. The samples must be collected over time.

– (see your MBB or be able to justify doing anything different)

15

When do we use % of Tolerance?When do we use % of Tolerance?When do we use % of Tolerance?When do we use % of Tolerance?When do we use % of Tolerance?When do we use % of Tolerance?When do we use % of Tolerance?When do we use % of Tolerance?

• When there is no problem to solve, and

• When the parts pulled represent the entire range of expected variation,

and

• When the distribution is not close to the specification limits.

16

Percent of Tolerance versus Percent of StudyPercent of Tolerance versus Percent of StudyPercent of Tolerance versus Percent of StudyPercent of Tolerance versus Percent of StudyPercent of Tolerance versus Percent of StudyPercent of Tolerance versus Percent of StudyPercent of Tolerance versus Percent of StudyPercent of Tolerance versus Percent of Study

As the process variation increases, the percent of study

variation decreases for any measurement error.

What we really want is low process variation, but low

process variation will drive the percent of study value

higher.

As long as the entire expectedexpected range of variation is covered,

the percent of tolerance value can be used if the process

is in control, stable, and capable.

AS LONG AS THERE IS NO PROLBEM TO BE SOLVED.AS LONG AS THERE IS NO PROLBEM TO BE SOLVED.

So, if the distribution is away from the specification limits

(high capability) – percent of tolerance can be used…

17

So, how should samples be selected?So, how should samples be selected?So, how should samples be selected?So, how should samples be selected?So, how should samples be selected?So, how should samples be selected?So, how should samples be selected?So, how should samples be selected?

Samples must be selected in such a way to cover the

entire range of variationentire range of variation.

This means DO NOTDO NOT select consecutive parts.

18

Why Not?Why Not?Why Not?Why Not?

Consecutive parts are a lot alike.

They likely do not cover the entire range of variation produced by the process.

Because only a small portion of the tolerance will

utilized, and because there will only be small differences between the parts, the percent study

variation will be large. This is not a good thing.

19

So, how should parts be selected?So, how should parts be selected?So, how should parts be selected?So, how should parts be selected?So, how should parts be selected?So, how should parts be selected?So, how should parts be selected?So, how should parts be selected?

Select parts from:

- Different shifts

- Different lots

- Different days

Make sure that the entire range entire range

of process variation is coveredof process variation is covered.

20

Variable DataVariable DataVariable DataVariable DataVariable DataVariable DataVariable DataVariable Data

• With at least 2 operators, use a minimum of 10 samples (and more is better), collected over time, measured at least 2 times by each operator (and again, more is better).

• Use as many samples as possible, measured at least twice but three times is recommended (and preferred).

• It is important that the entire range of normal entire range of normal process variation is covered by the parts collectedprocess variation is covered by the parts collected.

21

So, How do we run the MiniSo, How do we run the MiniSo, How do we run the MiniSo, How do we run the Mini----Tab Tab Tab Tab

analysis using variable data?analysis using variable data?analysis using variable data?analysis using variable data?

22

Setting up the AnalysisSetting up the AnalysisSetting up the AnalysisSetting up the AnalysisSetting up the AnalysisSetting up the AnalysisSetting up the AnalysisSetting up the Analysis

• Select the samples. Label the samples.

• Have each operator measure the parts, in random order.

• Repeat the step above until the parts have been measured therequired number of times (at least twice – three times is better)by each operator.

23

Set up 3 columns in Mini-Tab. Label the first column ‘Part’, the second column ‘Operator’, and the third column ‘Data’.

24

For every data point, fill in the row of data in the Mini-Tab spread sheet. Every row must have the part identifier, operator number, and the measured value.

25

Click on Stat > Quality Tools > Gage Study > Gage R & R (crossed).

Note: This is for non-destructive tests only. If the test is destructive, see your Master Black Belt.

26

And the following screen will appear………

27

Click in the space next toClick in the space next toClick in the space next toClick in the space next to ‘Part numbersPart numbers’, then click on the then click on the then click on the then click on the

wordwordwordword ‘PartPart’ in the box to the left. The wordin the box to the left. The wordin the box to the left. The wordin the box to the left. The word ‘PartPart’ will appear

to the right of the box labeled ‘Part numbersPart numbers’.

Click in this space

28

Click in the space next toClick in the space next toClick in the space next toClick in the space next to ‘operatorsoperators’, the click on the wordthe click on the wordthe click on the wordthe click on the word

‘operatoroperator’ in the box to the leftin the box to the leftin the box to the leftin the box to the left. The wordThe wordThe wordThe word ‘operatoroperator’ will appear will appear will appear will appear

to the right of the box labeledto the right of the box labeledto the right of the box labeledto the right of the box labeled ‘operatorsoperators’.

Click in this space

29

Click in the space next toClick in the space next toClick in the space next toClick in the space next to ‘MeasurementMeasurement datadata’, click on the click on the click on the click on the

wordwordwordword ‘DataData’ in the box to the left. The wordin the box to the left. The wordin the box to the left. The wordin the box to the left. The word ‘DataData’ will will will will

appear to the right of the box labeledappear to the right of the box labeledappear to the right of the box labeledappear to the right of the box labeled ‘MeasurementMeasurement datadata’.

Click in this space

30

For the analysis method, click the ANOVA radio button

31

Click on the Click on the Click on the Click on the ‘‘‘‘OKOKOKOK’’’’ button and the analysis will run.button and the analysis will run.button and the analysis will run.button and the analysis will run.

32

Analyze the Results Analyze the Results

And……………..This is how.

33

Gage name:

Date of study:

Reported by:

Tolerance:

Misc:

0

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1 1 2 3

Xbar Chart by Operator

Sam

ple

Mean

Mean=0.8075UCL=0.8796

LCL=0.7354

0

0.00

0.05

0.10

0.15 1 2 3

R Chart by Operator

Sam

ple

Range

R=0.03833

UCL=0.1252

LCL=0

1 2 3 4 5 6 7 8 9 10

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

Part

OperatorOperator*Part Interaction

Ave

rage

1

2

3

1 2 3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

Operator

By Operator

1 2 3 4 5 6 7 8 9 10

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

Part

By Part

%Contribution

%Study Var

%Tolerance

Gage R&R Repeat Reprod Part-to-Part

0

100

200

Components of Variation

Perc

ent

Gage R&R (ANOVA) for Response

MSA Health Side

MSA Troubleshooting Side

Graphical Output Graphical Output Graphical Output Graphical Output ---- 6 Graphs In All6 Graphs In All6 Graphs In All6 Graphs In All

If only 1 operator,

you won’t get

these graphs

In a nested

study, you won’t

get this graph

These barsshould belarge

These bars

should be small

34

R Chart by Operator

%Contribution

%Study Var

Part-to-PartReprodRepeatGage R&R

100

50

0

Components of VariationP

erc

ent

%Contribution

%Study Var

Gage R&R Repeat Reprod Part-to-Part

0

50

100


Perc

ent

Bar Charts for Components of VariationBar Charts for Components of Variation

Much better

Needs helpBad Gage

It Answers the Question: “Where is the variation coming from?”

The bar chart identifies the components of variation

Bar Chart: Distinguishes

R&R output

from Process Output.

35

0

300400500600700800900

100011001200 1 2 3


Sam

ple

Mean

Mean=725.7UCL=797.1

LCL=654.3

0

0

50

100

150 1 2 3

R Chart by OperatorS

am

ple

Range

R=37.97

UCL=124.0

LCL=0

Closer Look At The Xbar & R ChartsCloser Look At The Xbar & R Charts

R chart: in control; Xbar: at least 50% outside limits;

R Chart:

Exposes gage

Repeatability,

Resolution &

Stability issues

R Chart:

Helps identify

unusual

measurements

Resolution/

repeatability

THIS SHOULD BE IN CONTROL

THIS SHOULD BE 50% OUT OF CONTROL

BUT IT ISN’T!

BUT IT ISN’T!

36

0

300400500600700800900

100011001200 1 2 3


Sam

ple

Mean

Mean=725.7UCL=797.1

LCL=654.3

0

0

50

100

150 1 2 3

R Chart by Operator

Sam

ple

Range

R=37.97

UCL=124.0

LCL=0

Closer Look At The Xbar & R ChartsCloser Look At The Xbar & R Charts

Xbar: at least 50% outside limits; R chart: in control

Xbar Chart:

Test of

sensitivity,

bias, &

population

variety

37

More R Chart IndicatorsMore R Chart IndicatorsMore R Chart IndicatorsMore R Chart Indicators

Both may indicate poor gage resolution

0

0.005

0.004

0.003

0.002

0.001

0.000

321

R Chart

Sam

ple

Range

R=4.33E-04

UCL=0.001416

LCL=0

0

0.15

0.10

0.05

0.00

321

R Chart by Operator

Sam

ple

Range

R=0.03833

UCL=0.1252

LCL=0

Rbar too small?

Plateaus

38

%Study

%Tolerance

%Contribution

Tabular Output MetricsTabular Output Metrics

Number of Distinct Categories

Don’t forget to check the number of distinct categories (which must be greater than five for a variable gage).

This is not the number to be reportedThis is not the number to be reported

Report one of these Report one of these

numbers (whichever numbers (whichever

is appropriate)is appropriate)

39

Tabular Output MetricsTabular Output Metrics

Number of Distinct Categories

If ten different parts are measured, and Minitab reports the number of

distinct categories as four, this means that your measurement system cannot detect the difference between some of the parts. Increasing the precision of the gage will increase the number of distinct categories.

The number of distinct

categories is the

number of groups within the process

data that your measurement system

can distinguish.

40

Why should the ANOVA method be used?Why should the ANOVA method be used?

Consider the following………………………..

41

Perc

ent


100

50

0

% Contribution

% Study Var

Sam

ple R

ange 0.02

0.01

0.00

_R=0.0015UCL=0.00386

LCL=0

1 2 3

Sam

ple M

ean

14.30

14.28

14.26

__X=14.28701UCL=14.28855LCL=14.28548

1 2 3

part trial

10987654321

14.30

14.28

14.26

operator

321

14.30

14.28

14.26

part trial

Avera

ge

10 9 8 7 6 5 4 3 2 1

14.30

14.28

14.26

1

2

3

operator

Gage name:

Date of study :

Reported by :

Tolerance:

M isc:


R Chart by operator

Xbar Chart by operator

response by part trial

response by operator

operator * part trial Interaction

Gage R&R (Xbar/R) for response

Gage R&R Study - XBar/R Method

Study Var %Study Var

Source StdDev (SD) (6 * SD) (%SV)

Total Gage R&R 0.0011729 0.0070371 7.71

Repeatability 0.0008860 0.0053160 5.83

Reproducibility 0.0007685 0.0046110 5.05Part-To-Part 0.0151642 0.0909853 99.70

Total Variation 0.0152095 0.0912571 100.00

Number of Distinct Categories = 18

Think about this analysis:

These sections should never be blank


42


Source StdDev (SD) (6 * SD) (%SV)

Total Gage R&R 0.0030774 0.0184644 20.95

Repeatability 0.0022336 0.0134015 15.20Reproducibility 0.0021170 0.0127017 14.41

operator 0.0003395 0.0020367 2.31

operator*part trial 0.0020896 0.0125374 14.22

Part-To-Part 0.0143640 0.0861838 97.78

Total Variation 0.0146899 0.0881395 100.00


Perc

ent


100

50

0

% Contribution

% Study Var

Sam

ple R

ange 0.02

0.01

0.00

_R=0.0015UCL=0.00386

LCL=0

1 2 3

Sam

ple M

ean

14.30

14.28

14.26

__X=14.28701UCL=14.28855LCL=14.28548

1 2 3

part trial

10987654321

14.30

14.28

14.26

operator

321

14.30

14.28

14.26

part trial

Avera

ge

10 9 8 7 6 5 4 3 2 1

14.30

14.28

14.26

1

2

3

operator

Gage name:

Date of study :

Reported by :

Tolerance:

M isc:


R Chart by operator

Xbar Chart by operator

response by part trial

response by operator

operator * part trial Interaction

Gage R&R (ANOVA) for response

Now consider this study:These sections should never be blank


43

Summary of Data

620.95ANOVA

187.71X Bar R

Number of

Distinct

Categories

% Study

Variation

44

Which Gage is Better?Which Gage is Better?Which Gage is Better?Which Gage is Better?Which Gage is Better?Which Gage is Better?Which Gage is Better?Which Gage is Better?

The same data was used for both analyses.

• The difference is the analysis method.

• The first graph used the Xbar R method.

• The second graph used the ANOVA method.

45

So, why the difference in outcomes?So, why the difference in outcomes?So, why the difference in outcomes?So, why the difference in outcomes?So, why the difference in outcomes?So, why the difference in outcomes?So, why the difference in outcomes?So, why the difference in outcomes?

• The Xbar R method masked the operator to part

interaction.

• In fact, a look at the session window for the

Xbar R method will show that operator to

part interaction was not an output, but…

• It is an output when using the ANOVA

method

46

Gage R&R

%Contribution

Source VarComp (of VarComp)

Total Gage R&R 0.0000010 3.61

Repeatability 0.0000004 1.50

Reproducibility 0.0000006 2.11Operator 0.0000006 2.11

Part-To-Part 0.0000279 96.39

Total Variation 0.0000289 100.00


Source StdDev (SD) (6 * SD) (%SV)Total Gage R&R 0.0010219 0.0061313 19.00

Repeatability 0.0006587 0.0039523 12.25

Reproducibility 0.0007812 0.0046875 14.53

Operator 0.0007812 0.0046875 14.53Part-To-Part 0.0052805 0.0316828 98.18

Total Variation 0.0053784 0.0322706 100.00


Another ANOVA Another ANOVA

example!example!

Add the Variance (% Contribution) for Total Gage R&R to the Part-To-Part and it does add up to 100%.

In this case, 3.61 + 96.39 = 100%

Why doesn’t this add up to 100%

Perc

ent


100

50

0

% Contribution

% Study Var

Sam

ple R

ange

0.0030

0.0015

0.0000

_R=0.001

UCL=0.002574

LCL=0

Darlene Jason Judy

Sam

ple M

ean

0.784

0.776

0.768

__X=0.77481UCL=0.77583LCL=0.77379

Darlene Jason Judy

Sample

10987654321

0.78

0.77

0.76

Operator

JudyJasonDarlene

0.78

0.77

0.76

Sample

Avera

ge

10 9 8 7 6 5 4 3 2 1

0.784

0.776

0.768

Darlene

Jason

Judy

Operator

Gage name:

Date of study :

Reported by :

Tolerance:

Misc:


R Chart by Operator


Readings by Sample

Readings by Operator

Operator * Sample Interaction

Gage R&R (ANOVA) for Readings

Remember to fill in these sections


47

So, what is acceptable?So, what is acceptable?So, what is acceptable?So, what is acceptable?So, what is acceptable?So, what is acceptable?So, what is acceptable?So, what is acceptable?

Ideally, the ANOVA % of study variation for

Total Gage R & RTotal Gage R & R should be less than 10%.

But, for Chassis components…..

Under no circumstances should anything over 20% of Under no circumstances should anything over 20% of Under no circumstances should anything over 20% of Under no circumstances should anything over 20% of

study variation for Total Gage R & R be acceptedstudy variation for Total Gage R & R be acceptedstudy variation for Total Gage R & R be acceptedstudy variation for Total Gage R & R be accepted.

Unless………

Using the ANOVA % of tolerance is appropriate

(see your MBB).

48

So, when is using % of tolerance

appropriate?

• When the entire range of process variation is covered by the parts selected.

• The process is extremely stable.

• Only a very small portion of the tolerance is used.

• The portion of the tolerance that is used, is away from

the specification limits.

Again…see your MBB if you are not sure

And there is no problem to solve.

49

How do we enter the gage informationHow do we enter the gage information

Perc

ent


100

50

0

% Contribution

% Study Var

Sam

ple R

ange

0.0030

0.0015

0.0000

_R=0.001

UCL=0.002574

LCL=0

Darlene Jason Judy

Sam

ple M

ean

0.784

0.776

0.768

__X=0.77481UCL=0.77583LCL=0.77379

Darlene Jason Judy

Sample

10987654321

0.78

0.77

0.76

Operator

JudyJasonDarlene

0.78

0.77

0.76

Sample

Avera

ge

10 9 8 7 6 5 4 3 2 1

0.784

0.776

0.768

Darlene

Jason

Judy

Operator

Gage name:

Date of study :

Reported by :

Tolerance:

Misc:


R Chart by Operator


Readings by Sample

Readings by Operator

Operator * Sample Interaction

Gage R&R (ANOVA) for Readings



50

To input the gage informationTo input the gage informationTo input the gage informationTo input the gage information……………………....Click on Stat > Quality Tools > Gage Study > Gage R & R (crossed)

51

This dialogue box will appear…At this point, click on At this point, click on ““Gage InfoGage Info””

52

This screen will appear on your monitorThis screen will appear on your monitorThis screen will appear on your monitorThis screen will appear on your monitorThis screen will appear on your monitorThis screen will appear on your monitorThis screen will appear on your monitorThis screen will appear on your monitor……………………………………………………………………………………................

53

Now input the required informationNow input the required informationNow input the required informationNow input the required informationNow input the required informationNow input the required informationNow input the required informationNow input the required information…………………………………………........

Then click Then click ““OKOK”………………”………………

54

Click Click Click Click Click Click Click Click ““““““““OKOKOKOKOKOKOKOK”””””””” againagainagainagainagainagainagainagain………………………………………………………………........

55

And this is what youAnd this is what you’’ll getll get……

Perc

ent


80

40

0

% Contribution

% Study Var

Sam

ple R

ange 0.0010

0.0005

0.0000

_R=0.000243

UCL=0.000626

LCL=0

1 2 3

Sam

ple M

ean 0.4185

0.4180

0.4175

__X=0.417921

UCL=0.418170

LCL=0.417672

1 2 3

Part

10987654321

0.419

0.418

0.417

Operator

321

0.419

0.418

0.417

Part

Avera

ge

10 9 8 7 6 5 4 3 2 1

0.4185

0.4180

0.4175

1

2

3

Operator

Gage name: Name this gage

Date of study : 7/9/07

Reported by : Your name

Tolerance: 3

Misc: Whatev er is important


R Chart by Operator


Data by Part

Data by Operator

Operator * Part Interaction

Gage R&R (ANOVA) for Data

56

Measurement Error and oneMeasurement Error and oneMeasurement Error and oneMeasurement Error and oneMeasurement Error and oneMeasurement Error and oneMeasurement Error and oneMeasurement Error and one--------sided sided sided sided sided sided sided sided

Tolerances!Tolerances!Tolerances!Tolerances!Tolerances!Tolerances!Tolerances!Tolerances!

57

To analyze a oneTo analyze a oneTo analyze a oneTo analyze a one----sided tolerancesided tolerancesided tolerancesided tolerance……………………Click on Click on Click on Click on

Stat>Quality Tools>Gage Study> Gage R&R Study (Crossed)Stat>Quality Tools>Gage Study> Gage R&R Study (Crossed)Stat>Quality Tools>Gage Study> Gage R&R Study (Crossed)Stat>Quality Tools>Gage Study> Gage R&R Study (Crossed)

58

And the following screen will appear………

59

Click in the space next toClick in the space next toClick in the space next toClick in the space next to ‘Part numbersPart numbers’, then click on the then click on the then click on the then click on the

wordwordwordword ‘PartPart’ in the box to the left. The wordin the box to the left. The wordin the box to the left. The wordin the box to the left. The word ‘PartPart’ will appear will appear will appear will appear

to the right of the box labeledto the right of the box labeledto the right of the box labeledto the right of the box labeled ‘Part numbersPart numbers’.

Click in this space

60

Click in the space next toClick in the space next toClick in the space next toClick in the space next to ‘operatorsoperators’, the click on the wordthe click on the wordthe click on the wordthe click on the word

‘operatoroperator’ in the box to the leftin the box to the leftin the box to the leftin the box to the left. The wordThe wordThe wordThe word ‘operatoroperator’ will appear will appear will appear will appear

to the right of the box labeledto the right of the box labeledto the right of the box labeledto the right of the box labeled ‘operatorsoperators’.

Click in this space

61

Click in the space next toClick in the space next toClick in the space next toClick in the space next to ‘MeasurementMeasurement datadata’, click on the click on the click on the click on the

wordwordwordword ‘DataData’ in the box to the left. The wordin the box to the left. The wordin the box to the left. The wordin the box to the left. The word ‘DataData’ will will will will

appear to the right of the box labeledappear to the right of the box labeledappear to the right of the box labeledappear to the right of the box labeled ‘MeasurementMeasurement datadata’.

Click in this space

62

For the analysis method, click the ANOVA radio buttonFor the analysis method, click the ANOVA radio buttonFor the analysis method, click the ANOVA radio buttonFor the analysis method, click the ANOVA radio button

63

Click on Click on Click on Click on ““““OptionsOptionsOptionsOptions””””

64

Enter either the upper or lower Enter either the upper or lower Enter either the upper or lower Enter either the upper or lower

specification. Then click specification. Then click specification. Then click specification. Then click ““““OKOKOKOK””””....

Enter your one-side

specification. Either

upper or lower.

Then click “OK”

This is the default for Mini-Tab Version

14. The default is 5.15 for version 13.

65

ThenThenThenThen………….Click .Click .Click .Click ““““OKOKOKOK””””

66

And… This is the resulting graphPerc

ent


100

50

0

% Contribution

% Study Var

% Tolerance

Sam

ple R

ange

0.030

0.015

0.000

_R=0.011

UCL=0.02832

LCL=0

Chandra Darby Sam

Sam

ple M

ean

2.0

1.5

1.0

__X=1.541UCL=1.552LCL=1.529

Chandra Darby Sam

Part

10987654321

2.0

1.5

1.0

Operator

SamDarbyChandra

2.0

1.5

1.0

Part

Avera

ge

10 9 8 7 6 5 4 3 2 1

2.0

1.5

1.0

Chandra

Darby

Sam

Operator

Gage name:

Date of study :

Reported by :

Tolerance:

Misc:


R Chart by Operator


Data by Part

Data by Operator

Operator * Part Interaction

Gage R&R (ANOVA) for Data



67

Output from the session windowOutput from the session windowOutput from the session windowOutput from the session windowOutput from the session windowOutput from the session windowOutput from the session windowOutput from the session window

Two-Way ANOVA Table Without Interaction

Source DF SS MS F P

Part 9 7.76256 0.862507 25875.2 0.000

Operator 2 0.00000 0.000000 0.0 1.000

Repeatability 78 0.00260 0.000033

Total 89 7.76516

Gage R&R

%Contribution

Source VarComp (of VarComp)

Total Gage R&R 0.0000333 0.03

Repeatability 0.0000333 0.03

Reproducibility 0.0000000 0.00

Operator 0.0000000 0.00

Part-To-Part 0.0958304 99.97

Total Variation 0.0958637 100.00

Lower process tolerance limit = 0.8

Part to part variation

is significant

because

P-value < 0.05

Operator to operator

variation is

insignificant because

P-value > 0.05

68

More session window output….

Study Var %Study Var %Tolerance

Source StdDev (SD) (6 * SD) (%SV) (SV/Toler)

Total Gage R&R 0.005774 0.03464 1.86 2.34

Repeatability 0.005774 0.03464 1.86 2.34

Reproducibility 0.000000 0.00000 0.00 0.00

Operator 0.000000 0.00000 0.00 0.00

Part-To-Part 0.309565 1.85739 99.98 125.39

Total Variation 0.309619 1.85771 100.00 125.41


69

Attribute DataMeasurement System Analysis

70

Attribute Gage R & RAttribute Gage R & RAttribute Gage R & RAttribute Gage R & RAttribute Gage R & RAttribute Gage R & RAttribute Gage R & RAttribute Gage R & R

• Use at least 50 samples (more is better). The sampling criteria will be discussed in the pages that follow.

• The samples must include parts from both sides of the

specification limits. To determine the part dimensions,

measure the parts using a known, calibrated measuring

system such as a CMM, height gage, etc.

• Use at least 3 operators, and measure the parts at least 3 times.

• Construct a truth table, and construct an attribute performance

curve as described in the AIAG Measurement System Analysis

(MSA) manual. This will be the subject of another module.

71

So how are the samples for a

Attribute Measurement

System Analysis selected?

72

Attribute MSA MethodAttribute MSA MethodAttribute MSA MethodAttribute MSA MethodAttribute MSA MethodAttribute MSA MethodAttribute MSA MethodAttribute MSA MethodStep 1: Select a minimum of 50 parts (take parts from the process, and if necessary, and if necessary,

create parts beyond the gage boundary limits to meet the requirecreate parts beyond the gage boundary limits to meet the requirements belowments below).

– 25% of the parts in your study should be near the lower gage boundary

limit (on both sides of the boundary).

– 25% of the parts in your study should be near the upper gage boundary

limit (on both sides of the boundary).

– 30% of the parts should represent the normal process variation.

– 10% of the parts should be outside the upper gage boundary limit and

beyond the 25% of the parts near the boundary as described above.

– 10% of the parts should be outside the lower gage boundary limit and

beyond the 25% of the parts near the boundary as described above.

All of the above parts must be independently measured with All of the above parts must be independently measured with All of the above parts must be independently measured with All of the above parts must be independently measured with a variable gage (such as a CMM or other known calibrated a variable gage (such as a CMM or other known calibrated a variable gage (such as a CMM or other known calibrated a variable gage (such as a CMM or other known calibrated standard) so that the physical measures of each part is standard) so that the physical measures of each part is standard) so that the physical measures of each part is standard) so that the physical measures of each part is known. known. known. known.

73

Attribute Gage R & R

LSL USL

25% at

Lower

Boundary

25% at

Upper

Boundary

10% outside

Lower

Boundary

10% outside

Upper

Boundary

30% representing range of

normal process variation

74

Attribute MSA MethodAttribute MSA MethodAttribute MSA MethodAttribute MSA MethodAttribute MSA MethodAttribute MSA MethodAttribute MSA MethodAttribute MSA Method

Step 4: Enter the data into MINITAB to report the

effectiveness of the attribute measurement system

Step 5: Document the results. Implement appropriate actions to fix the

inspection process if necessary

Step 6: Construct an attribute performance curve and document the

results.

Step 2: Identify the appraisers.

Step 3: Have each appraiser assess these parts and determine

whether or not they pass or fail the gage. This must be done

independently and in a random order. Each appraiser must repeat

this step a minimum of three times

75

Label for Column 1: Label for Column 1: Label for Column 1: Label for Column 1: TrialTrialTrialTrial

76

Label for Column 2: Label for Column 2: Label for Column 2: Label for Column 2: TruthTruthTruthTruth

77

Label for C3: Label for C3: Label for C3: Label for C3: Oper 1 FirstOper 1 FirstOper 1 FirstOper 1 First. This is the 1. This is the 1. This is the 1. This is the 1stststst OperatorOperatorOperatorOperator’’’’s s s s

measured value for the first measurement of the part.measured value for the first measurement of the part.measured value for the first measurement of the part.measured value for the first measurement of the part.

78

Label for C4: Label for C4: Label for C4: Label for C4: Oper 1 SecondOper 1 SecondOper 1 SecondOper 1 Second. This is the 1. This is the 1. This is the 1. This is the 1stststst OperatorOperatorOperatorOperator’’’’s s s s

measured value for the second measurement of the part.measured value for the second measurement of the part.measured value for the second measurement of the part.measured value for the second measurement of the part.

79

Label for C5: Label for C5: Label for C5: Label for C5: Oper 1 ThirdOper 1 ThirdOper 1 ThirdOper 1 Third. This is the 1. This is the 1. This is the 1. This is the 1stststst OperatorOperatorOperatorOperator’’’’s s s s

measured value for the third measurement of the part.measured value for the third measurement of the part.measured value for the third measurement of the part.measured value for the third measurement of the part.

80

Label for C6: Label for C6: Label for C6: Label for C6: Oper 2 FirstOper 2 FirstOper 2 FirstOper 2 First. This is the 2. This is the 2. This is the 2. This is the 2ndndndnd OperatorOperatorOperatorOperator’’’’s s s s


81

Label for C7: Label for C7: Label for C7: Label for C7: Oper 2 SecondOper 2 SecondOper 2 SecondOper 2 Second. This is the 2. This is the 2. This is the 2. This is the 2ndndndnd OperatorOperatorOperatorOperator’’’’s s s s


82

Label for C8: Label for C8: Label for C8: Label for C8: Oper 2 ThirdOper 2 ThirdOper 2 ThirdOper 2 Third. This is the 2. This is the 2. This is the 2. This is the 2ndndndnd OperatorOperatorOperatorOperator’’’’s s s s


83

Label for C9: Label for C9: Label for C9: Label for C9: Oper 3 FirstOper 3 FirstOper 3 FirstOper 3 First. This is 3. This is 3. This is 3. This is 3rdrdrdrd OperatorOperatorOperatorOperator’’’’s s s s


84

Label for C10: Label for C10: Label for C10: Label for C10: Oper 3 SecondOper 3 SecondOper 3 SecondOper 3 Second. This is the 3. This is the 3. This is the 3. This is the 3rdrdrdrd OperatorOperatorOperatorOperator’’’’s s s s


85

Label for C11: Label for C11: Label for C11: Label for C11: Oper 3 ThirdOper 3 ThirdOper 3 ThirdOper 3 Third. This is the 3. This is the 3. This is the 3. This is the 3rdrdrdrd OperatorOperatorOperatorOperator’’’’s s s s


86

Input the data into Mini-Tab

1. Place the sample numbers in Trial (Column C1)

2. Place the pass/fail value from the variable independent

measurement system in Truth (Column C2)

3. Place the values for Operator 1 first value in C3. Make

sure that the value corresponds to the correct sample

number in column C1.

4. Place the values for Operator 1 second value in C4.

5. Place the values for Operator 1 third value in C5.

6. Repeat step 3, 4, and 5 for Operator 2 and Operator 3.

87

Click on Stat > Quality Tools > Attribute Analysis

To run the analysis:

The “T” denotes a non-

numeric field.

88

This will appear on your screenThis will appear on your screenThis will appear on your screenThis will appear on your screenThis will appear on your screenThis will appear on your screenThis will appear on your screenThis will appear on your screen

89

Click the radio button in front of Click the radio button in front of

““Multiple columnsMultiple columns””

90

Now do the followingNow do the followingNow do the followingNow do the followingNow do the followingNow do the followingNow do the followingNow do the following……………………

Click in here firstThis list will appear

Next, highlight the required columns

Finally, click “Select”

91

And now the required columns are in theAnd now the required columns are in theAnd now the required columns are in theAnd now the required columns are in theAnd now the required columns are in theAnd now the required columns are in theAnd now the required columns are in theAnd now the required columns are in the

““““““““Multiple ColumnMultiple ColumnMultiple ColumnMultiple ColumnMultiple ColumnMultiple ColumnMultiple ColumnMultiple Column”””””””” boxboxboxboxboxboxboxbox

92

Now enter the Now enter the Now enter the Now enter the Now enter the Now enter the Now enter the Now enter the ““““““““Number of appraisersNumber of appraisersNumber of appraisersNumber of appraisersNumber of appraisersNumber of appraisersNumber of appraisersNumber of appraisers”””””””” and the and the and the and the and the and the and the and the

““““““““Number of trialsNumber of trialsNumber of trialsNumber of trialsNumber of trialsNumber of trialsNumber of trialsNumber of trials””””””””

Enter the number

of appraisers

Then enter the number of trials

93

Enter the known standard Enter the known standard Enter the known standard Enter the known standard Enter the known standard Enter the known standard Enter the known standard Enter the known standard

informationinformationinformationinformationinformationinformationinformationinformation………………………………………………………………

Then double-click hereor single click and thenpick “Select” button

First Click here

94

This dialogue box will be on your screen. Click on ‘Options’

95

The dialogue box below will be on your screen.

Click “OK”

This is the default value.

Make sure that the appropriate value is used.

96

Now, click on information and this box will appearNow, click on information and this box will appearNow, click on information and this box will appearNow, click on information and this box will appearNow, click on information and this box will appearNow, click on information and this box will appearNow, click on information and this box will appearNow, click on information and this box will appear……………………................

97

Fill in the required informationFill in the required informationFill in the required informationFill in the required information…………........

98

Click on Click on Click on Click on ““““OKOKOKOK”…”…”…”…........

99

Click Click Click Click ““““OKOKOKOK”””” againagainagainagain

100

This graph is the resultThis graph is the resultThis graph is the resultThis graph is the result

Appraiser

Perc

ent

321

100

90

80

70

60

95.0% C I

Percent

Appraiser

Perc

ent

321

100

90

80

70

60

95.0% C I

Percent

Date of study: 7/7/07

Reported by: What is your name

Name of product: Product

Misc: Whatever is important

Assessment Agreement

Within Appraisers Appraiser vs Standard

The graph on the left shows the within appraiser agreement

– How well does each appraiser agree with himself/herself.

The graph on the right shows the appraiser agreement to

the standard.

– How well does each appraiser agree with the Truth (C2)

101

Now go back to the session windowNow go back to the session windowNow go back to the session windowNow go back to the session window

Click in this area

and scroll up or

down to the desired

location.

102

This is how well each appraiser agreed with him/herselfThis is how well each appraiser agreed with him/herselfThis is how well each appraiser agreed with him/herselfThis is how well each appraiser agreed with him/herself

Attribute Agreement Analysis• Attribute Agreement Analysis for Oper 1 First, Oper 1 Secon, Oper 1

Third, ...

• Within Appraisers

• Assessment Agreement

Appraiser # Inspected # Matched Percent 95 % CI

1 29 29 100.00 (90.19, 100.00)

2 29 29 100.00 (90.19, 100.00)

3 29 24 82.76 (64.23, 94.15)

# Matched: How well each appraiser agrees with him/herself

across trials.

Notice that Appraiser 3 has some issues, as he/she does Notice that Appraiser 3 has some issues, as he/she does Notice that Appraiser 3 has some issues, as he/she does Notice that Appraiser 3 has some issues, as he/she does

not match as well as Appraisers 1 & 2!not match as well as Appraisers 1 & 2!not match as well as Appraisers 1 & 2!not match as well as Appraisers 1 & 2!

103

Kappa statistics for within appraiser agreementKappa statistics for within appraiser agreementKappa statistics for within appraiser agreementKappa statistics for within appraiser agreement

• Fleiss' Kappa Statistics

Appraiser Response Kappa SE Kappa Z P(vs > 0)

1 Fail 1.00000 0.107211 9.32738 0.0000

Pass 1.00000 0.107211 9.32738 0.0000

2 Fail 1.00000 0.107211 9.32738 0.0000

Pass 1.00000 0.107211 9.32738 0.0000

3 Fail 0.66331 0.107211 6.18697 0.0000

Pass 0.66331 0.107211 6.18697 0.0000

• Cohen's Kappa Statistics

– There are more than two trials within each appraiser. Cannot compute Cohen’s Kappa.

104

Kappa compares the proportion of agreement between Appraisers after

removing agreement by chance

The proportion that the judges are in agreement is Pobserved

The proportion expected to occur by chance is:

Pchance = (P Insp1 Good) (P Insp2 Good) + (P Insp1 Bad)(PInsp2 Bad)

Definition Of Kappa

chance

chanceobserved

P1

PPKappa

−

−=

105

This is the appraiser agreement to the standardThis is the appraiser agreement to the standardThis is the appraiser agreement to the standardThis is the appraiser agreement to the standard

• Each Appraiser vs. Standard


Appraiser # Inspected # Matched Percent 95 % CI

1 29 27 93.10 (77.23, 99.15)

2 29 28 96.55 (82.24, 99.91)

3 29 23 79.31 (60.28, 92.01)

# Matched: Appraiser's assessment across trials agrees with the known standard.

• Assessment Disagreement

# Pass / # Fail /

Appraiser Fail Percent Pass Percent # Mixed Percent

1 1 14.29 1 4.55 0 0.00

2 1 14.29 0 0.00 0 0.00

3 1 14.29 0 0.00 5 17.24

# Pass / Fail: Assessments across trials = Pass / standard = Fail.

# Fail / Pass: Assessments across trials = Fail / standard = Pass.

# Mixed: Assessments across trials are not identical.

106

FleissFleissFleissFleiss’’’’ Kappa Statistic for appraiser to standard agreementKappa Statistic for appraiser to standard agreementKappa Statistic for appraiser to standard agreementKappa Statistic for appraiser to standard agreement

Fleiss' Kappa Statistics


1 Fail 0.811688 0.107211 7.57092 0.0000

Pass 0.811688 0.107211 7.57092 0.0000

2 Fail 0.900855 0.107211 8.40261 0.0000

Pass 0.900855 0.107211 8.40261 0.0000

3 Fail 0.745591 0.107211 6.95441 0.0000

Pass 0.745591 0.107211 6.95441 0.0000

107

CohenCohenCohenCohen’’’’s Kappa Statistic for appraiser to standard agreement.s Kappa Statistic for appraiser to standard agreement.s Kappa Statistic for appraiser to standard agreement.s Kappa Statistic for appraiser to standard agreement.

Cohen's Kappa Statistics


1 Fail 0.811688 0.107211 7.57092 0.0000

Pass 0.811688 0.107211 7.57092 0.0000

2 Fail 0.901024 0.106685 8.44566 0.0000

Pass 0.901024 0.106685 8.44566 0.0000

3 Fail 0.745704 0.106861 6.97829 0.0000

Pass 0.745704 0.106861 6.97829 0.0000

108

Between Appraiser Agreement!Between Appraiser Agreement!Between Appraiser Agreement!Between Appraiser Agreement!

Between Appraisers


# Inspected # Matched Percent 95 % CI

29 23 79.31 (60.28, 92.01)

# Matched: How well all appraisers assessments agree with each other.

This is analogous to Reproducibility.

109

FleissFleissFleissFleiss’’’’ Kappa Statistics for between appraiser agreement!Kappa Statistics for between appraiser agreement!Kappa Statistics for between appraiser agreement!Kappa Statistics for between appraiser agreement!


Response Kappa SE Kappa Z P(vs > 0)

Fail 0.839286 0.0309492 27.1182 0.0000

Pass 0.839286 0.0309492 27.1182 0.0000

You must have multiple trials per appraiser to compute Fleiss’ Kappa.


You must have two appraisers and single trial per appraiser to compute Cohen’s Kappa.

110

This is how well all appraisers agree with the standard.This is how well all appraisers agree with the standard.This is how well all appraisers agree with the standard.This is how well all appraisers agree with the standard.

All Appraisers vs. Standard


# Inspected # Matched Percent 95 % CI

29 22 75.86 (56.46, 89.70)

# Matched: How well all appraisers assessments agree with theknown standard.

In a Measurement System Analysis, this is analogous to

an accuracy check.

111

Below are both Kappa Statistics for all appraiser agreement withBelow are both Kappa Statistics for all appraiser agreement withBelow are both Kappa Statistics for all appraiser agreement withBelow are both Kappa Statistics for all appraiser agreement with the standard.the standard.the standard.the standard.



Fail 0.819378 0.0618984 13.2375 0.0000

Pass 0.819378 0.0618984 13.2375 0.0000



Fail 0.819472 0.0617298 13.2752 0.0000

Pass 0.819472 0.0617298 13.2752 0.0000

112

Key Learning PointsKey Learning Points

For nominal data, the Kappa Coefficient provides a measure of relative agreement between appraisers.

For ordinal data, Kendall’s Coefficient of Concordance provides a measure of relative agreement between

Appraisers.

Now, what is the difference between Now, what is the difference between

these types of data?these types of data?

113

What are Data Types?What are Data Types?What are Data Types?What are Data Types?What are Data Types?What are Data Types?What are Data Types?What are Data Types?

Mathematical versus Measurement Model

• Math

– Continuous

– Discrete

• Measures of Scale

– Nominal

– Ordinal

– Interval

– Ratio

114

Mathematical Data ModelsMathematical Data ModelsMathematical Data ModelsMathematical Data Models

• Continuous

– No boundaries between adjoining values

– Most non-counting intervals and ratios

• Example: Time

• Discrete

– Clear boundaries

– Includes nominal, counts and rank orders

• Example: Calendar

115

Scales of MeasureScales of MeasureScales of MeasureScales of MeasureScales of MeasureScales of MeasureScales of MeasureScales of MeasureNominal

– Assigns items to groups or categories. Information is qualitative, not quantitative

• Marital status, Religious preference, Race, Sex

Ordinal

– Higher numbers represent higher values, but intervals between

numbers are not necessarily equal. The zero point is chosen arbitrarily.

• Race finish, opinion poll response (difference between rating of 2 and 3

may not be the same as the difference between ratings of 4 and 5)

Interval

– Equal intervals have equal differences

• Calendar year, Fahrenheit temperature

Ratio

– Are like Interval scales except they have a true zero point

– Has a real zero value

• Absolute income, Kelvin temperature

116

Data Structure •Data can be nominal or ordinal

•Nominal: categorical variables with 2 or

more possible levels and no natural ordering.

Example:

• rubbery, squishy, brittle

• Go/No-go

• Pass/Fail

• Yes/No

•Ordinal: categorical variables with 3 or more

possible levels with a natural ordering.

Example:

• Strongly disagree, disagree, neutral, agree, strongly agree

• Numeric scale such as 1- 5

Purpose of Attribute Data TypesPurpose of Attribute Data Types

117

Purpose Of Attribute MSAPurpose Of Attribute MSAPurpose Of Attribute MSAPurpose Of Attribute MSAPurpose Of Attribute MSAPurpose Of Attribute MSAPurpose Of Attribute MSAPurpose Of Attribute MSA

• Accuracy checks

– Assess standards against customers’ requirements

– Identify conformance to a “known master”

• Precision checks

– To determine if inspectors (Appraisers) agree across all shifts,

machines, lines, etc… Reproducibility

– To quantify the repeat inspection decisions by each appraiser –

Repeatability

• To identify agreement to a master:

– How often do appraisers accept defective product?

– How often do appraisers reject acceptable product?

118

Kappa compares the proportion of agreement between Appraisers after

removing agreement by chance

The proportion that the judges are in agreement is Pobserved

The proportion expected to occur by chance is:

Pchance = (P Insp1 Good) (P Insp2 Good) + (P Insp1 Bad)(PInsp2 Bad)

DefinitionDefinition Of KappaOf Kappa

chance

chanceobserved

P1

PPKappa

−

−=

119

The kappa coefficient must exceedexceedexceedexceed .70

Interpreting the Kappa CoefficientInterpreting the Kappa Coefficient

Kappa Value InterpretationKappa Value InterpretationKappa Value InterpretationKappa Value Interpretation

Below 0.00 Poor

0.00 to 0.20 Slight

0.21 to 0.40 Fair

0.41 to 0.60 Moderate

0.61 to 0.80 Substantial

0.81 to 1.00 Almost perfect

before

You can proceed with the analysis.

Does the Kappa Coefficient apply to

• within appraiser agreement?within appraiser agreement?within appraiser agreement?within appraiser agreement?

• appraiser to standard agreement?appraiser to standard agreement?appraiser to standard agreement?appraiser to standard agreement?

• between appraiser agreement?between appraiser agreement?between appraiser agreement?between appraiser agreement?

• all appraisers to standard?all appraisers to standard?all appraisers to standard?all appraisers to standard?

• all of the above!!!!all of the above!!!!all of the above!!!!all of the above!!!!

120

This is the output from the MiniThis is the output from the Mini--Tab session window.Tab session window.

This is how well each operator agreed with themselves (repeatabiThis is how well each operator agreed with themselves (repeatability).lity).

Note: Appraiser 3 has an issue

121

This slide is Appraiser agreement to the standard.This slide is Appraiser agreement to the standard.

Note Appraiser 3 has an issue

122

Note : There is general agreement between Fleiss’ Kappa and Cohen’s Kappa. Fleiss and Cohen are just two different statisticians.

123

This is a picture of Operator to Operator reproducibility

124

This is a picture of reproducibility to the standard

125

Appraiser

Perc

ent

321

100

90

80

70

60

95.0% C I

Percent

Appraiser

Perc

ent

321

100

90

80

70

60

95.0% C I

Percent

Date of study:

Reported by:

Name of product:

Misc:


Within Appraisers Appraiser vs Standard

This is graphic output of the session window.

126

Kappa CoefficientKappa Coefficient

Proportion of times that the appraisers agree

Maximum proportion of time that the appraisers could agree

• The ratio of the proportion of times that the appraisers

agree (corrected for chance agreement) to the maximum

proportion of times that the appraisers could agree

(corrected for chance agreement).

• When you have a known standard, Kappa is the average

of Kappa across trials.

• If Kappa = 1, then there is perfect agreement.

If Kappa = 0, then the agreement is the same as would

be expected by random chance.

127

Kappa vs. KendallKappa vs. Kendall’’s Coefficient of Concordances Coefficient of Concordance

• Kappa statistics represent absolute agreement among

ratings; they treat all misclassifications equally.

• Kendall's coefficients measure the associations among

ratings; they do not treat all misclassifications equally. The

consequences of misclassifying a perfect (rating = 5) object

as bad (rating = 1) are more serious than misclassifying it as

very good (rating = 4).

Kendall is only used for Ordinal data. If you have Ordinal data, see your friendly neighborhood MBB.

Ordinal

– Higher numbers represent higher values, but intervals between

numbers are not necessarily equal. The zero point is chosen arbitrarily.

• Race finish, opinion poll response (difference between rating of 2 and 3

may not be the same as the difference between ratings of 4 and 5)

128

REMEMBER……………

The kappa coefficient must exceed 0.70

Greater than 0.80 is preferred.

129

Additional Information for inquiring minds

130

How To Calculate Kappa How To Calculate Kappa How To Calculate Kappa How To Calculate Kappa How To Calculate Kappa How To Calculate Kappa How To Calculate Kappa How To Calculate Kappa –––––––– By HandBy HandBy HandBy HandBy HandBy HandBy HandBy Hand

Build contingency table

Sum columns and rows

Calculate Pobs by adding diagonal

Calculate Pchance

Calculate Kappa

Pchance = (PR1bad)(PR2bad)+(PR1good)(PR2good)

= (.2)(.5)+(.8)(.5)

= .5

Pobs - Pchance

Kappa =

1 - Pchance

=(.7-.5)/(1-.5)

= 0.4

.2

.8

.5 .5

Bad

Good

Ap

pra

iser 1

Good Bad

Appraiser 2

Pobs = .5 + .2 = .7

Add

3/10 = .3

2/10 = .20/10 = 0

5/10 = .5

Add

Add

Add

Add

131

% Study Variation% Study Variation% Study Variation% Study Variation% Study Variation% Study Variation% Study Variation% Study Variation

• Looks at standard deviations instead of variance

100* ationStudy Vari %TOTAL

R&R

σ

σ=

• Includes both repeatability and reproducibility

• Measurement System Standard Deviation (R&R) as a

percentage of Total Observed Process Standard Deviation

132

% Contribution% Contribution% Contribution% Contribution% Contribution% Contribution% Contribution% Contribution

100* onContributi %

TOTAL

2

R&R

2

σ

σ=

Includes both repeatability and reproducibility

Measurement System Variation (R&R) as a percentage

of Total Observed Process Variation

133

% Tolerance% Tolerance% Tolerance% Tolerance% Tolerance% Tolerance% Tolerance% Tolerance

Measurement error as a percent of tolerance

• Includes both repeatability and reproducibility

100*Tolerance

*155 Tolerance %

P/T Tolerance to Precision

R&Rσ.=

components of measurement error - gage rr - the last stand - bw print version 5-20-08

Documents

measurement system resolution

measurement units

variation present

process variation

type of gage r r

variancethe change

type of plot

smallest significant