1

Optimizing Subjective Results

Julia C. O'NeillKarl R. Wursthorn

Rohm and Haas Company

ASA/ASQ Fall Technical Conference

October 18, 2002

2

Outline

• Problem: Topaz Appearance

• Solution Techniques:– Factorial Design– Paired Comparisons– Multidimensional Scaling

• References and Software

3

The Topaz Finish: Objective

• To develop a formula which will consistently provide the customer the appearance they want; and investigate the effects of formulation and application variables on Topaz appearance.

• To become 100% supplier of powder coatings to this customer.

4

Problem: Topaz Appearance

• Does our coating “look right”?– What matters is what the customer sees.

• The Topaz coating appearance depends on many variables.– Formulation– Application conditions

• Measurable attributes do not give the whole picture for multi-color finishes.– Gloss and color are “averages”.

5

The Solution

• Factorial experiments allow the estimation and comparison of many effects.

• Paired comparisons simplify the task of judging appearance.

• Multidimensional scaling reveals the underlying dimensions of appearance.

6

Experiment Design

• Variables:– Grinding equipment (3 types)– Particle size (3 distributions)– Additive (A, B, or none)– Spray gun tip (conical or slot)– Powder charge (60 or 90 kV)– Gun feed (cup or fluid bed)

• Initial screening design had 16 runs• 110 panels sprayed• Optimal selection of 20 panels

7

20 Panels Evaluated

Charge Additive Grinder PSD Feed Tip 100 none production fine cup slotted 100 none lab lab cup slotted 60 none lab lab fluid slotted 60 B production fine cup conical 60 none lab lab cup conical 60 A production coarse cup conical

100 A lab lab fluid slotted 60 A production fine cup slotted 60 A lab lab cup slotted

100 B production coarse fluid slotted 100 none production coarse cup slotted 100 none production coarse cup conical 100 B production fine fluid conical 60 none production fine cup slotted

100 none production fine cup conical 100 A production fine fluid slotted 100 A lab lab cup conical 100 B lab lab cup slotted 100 A production fine cup conical 100 A production coarse fluid conical

8

Appearance Data Collection

• Discover, rather than impose, dimensions of appearance

• Do not specify attributes

• Trained, experienced “eyes” are critical

9

Paired Comparisons

• When objects to be compared can be judged only subjectively.

• When differences between objects are small.• When comparison of two objects at one time is

simpler than comparing many objects.• When the comparison between 2 objects should

not be influenced by other objects.• When we want to uncover “hidden structure” of

objects.

10

Paired Comparisons vs. Ranking vs. Rating

• Ranking is preferred when:– Several objects can easily be compared

simultaneously.– Differences between objects are fairly apparent.

• Rating is preferred when:– Several grades can be distinguished with

consensus among judges.– Ratings can be treated like measurements.

11

Instructions to Evaluators

• Cluster based on similarity: Looking at all 20 panels, form groups of any panels which are indistinguishable– Panels in same group have dissimilarity 0

• Select one panel to represent each group• Rate each pair of distinct panels 1 to 4:

– 1 means hardly any difference– 4 means very different

• Rate based on “appearance”, whatever that means to the judge.

12

Dissimilarity Data Collection

• Number of pairs is:

• For 20 objects, 190 pairs.• Similar to a Balanced Incomplete Block design

with block size 2• Modification of all possible pairs of panels

– Sort first into indistinguishable groups

• Can be optimized for order of presentation• Can be fractionated

2

I

13

Dissimilarity Ratings – averages from 3 evaluators

Panel a2 c5 d1 e2 f5 h1 h3 m3 p2 p4 s2 s3 s4 u1

a2 0.00 2.00 2.00 3.67 3.00 0.67 2.33 2.33 3.00 2.67 0.67 2.67 2.67 1.00

c5 2.00 0.00 3.33 4.00 4.00 2.33 4.00 0.33 4.00 0.67 1.67 0.33 2.00 2.00

d1 2.00 3.33 0.00 2.67 2.33 1.33 2.33 3.33 3.00 3.33 2.00 3.33 3.67 1.33

e2 3.67 4.00 2.67 0.00 2.00 4.00 1.67 4.00 2.33 4.00 3.67 4.00 3.00 3.67

f5 3.00 4.00 2.33 2.00 0.00 3.33 1.00 4.00 1.67 4.00 3.33 4.00 2.67 3.33

h1 0.67 2.33 1.33 4.00 3.33 0.00 3.33 2.00 3.33 2.00 1.00 2.33 2.67 0.33

h3 2.67 4.00 2.33 1.67 1.00 3.33 0.00 4.00 0.67 4.00 3.00 4.00 2.67 3.33

m3 2.33 0.33 3.33 4.00 4.00 2.00 4.00 0.00 4.00 0.33 2.00 0.33 1.67 1.67

p2 3.00 4.00 3.00 2.33 1.67 3.33 0.67 4.00 0.00 4.00 3.33 4.00 3.33 3.33

p4 2.33 0.67 3.33 4.00 4.00 2.00 4.00 0.33 4.00 0.00 2.33 0.67 1.67 1.67

s2 0.67 1.67 2.00 3.67 3.33 1.00 3.00 2.00 3.33 2.33 0.00 2.33 3.00 1.33

s3 2.67 0.33 3.33 4.00 4.00 2.33 4.00 0.33 4.00 0.67 2.33 0.00 2.00 2.00

s4 2.67 2.00 3.67 3.00 2.67 2.67 2.67 1.67 3.33 1.67 3.00 2.00 0.00 2.33

u1 1.00 2.00 1.33 3.67 3.33 0.33 3.33 1.67 3.33 1.67 1.33 2.00 2.33 0.00

u2 2.00 3.67 2.33 2.00 2.33 3.33 1.67 3.33 2.00 3.33 2.67 3.67 2.67 3.67

w1 3.67 4.00 2.67 0.00 2.00 4.00 1.67 4.00 2.33 4.00 3.67 4.00 3.00 3.67

w3 3.00 3.33 3.67 3.67 2.33 3.33 2.67 3.00 3.33 3.33 3.67 3.33 1.67 3.33

x1 2.67 4.00 2.67 2.00 1.33 3.33 1.67 4.00 2.00 4.00 3.33 4.00 3.00 3.33

y1 3.67 4.00 3.67 2.33 2.67 4.00 2.33 4.00 1.67 4.00 4.00 4.00 3.67 4.00

y4 3.00 4.00 2.33 2.00 0.00 3.33 1.00 4.00 1.67 4.00 3.33 4.00 2.67 3.33

14

Mapping Example of Multidimensional Scaling

• Start with distances between objects (cities).

• Given only the distances, produce the map.City Seattle Phoenix Minn. Madison Chicago Phil. New.York Boston Orlando WashingtonDCSeattle 0 1110 1390 1620 1710 2370 2410 2490 2450 2300Phoenix 1110 0 1270 1390 1440 2070 2150 2290 1760 1950Minneapolis 1390 1270 0 227 333 977 1020 1120 1210 905Madison 1620 1390 227 0 108 762 817 928 1010 682Chicago 1710 1440 333 108 0 676 737 863 904 587Philadelphia 2370 2070 977 762 676 0 93 278 808 135New.York 2410 2150 1020 817 737 93 0 186 894 227Boston 2490 2290 1120 928 863 278 186 0 1080 411Orlando 2450 1760 1210 1010 904 808 894 1080 0 696WashingtonDC 2300 1950 905 682 587 135 227 411 696 0

15

U. S. Map Reconstruction

-1600 -1100 -600 -100 400 900

Dim1

-500

-100

300

700

Dim2

Seattle

Phoenix

Minneapolis

MadisonChicago Philadelphia

New.York

Boston

Orlando

WashingtonDC

16

U. S. Map Reconstruction

-1600 -1100 -600 -100 400 900

Dim1

-800

-400

0

400

Dim2neg

Seattle

Phoenix

Minneapolis

MadisonChicago Philadelphia

New.York

Boston

Orlando

WashingtonDC

17

Distances among Objects

44434241

34333231

24232221

14131211

No meaning when i=jSame distance from i to j as from j to i.

44434241

34333231

24232221

14131211

dddd

dddd

dddd

dddd

D

18

Dissimilarities among Objects

44434241

34333231

24232221

14131211

44434241

34333231

24232221

14131211

No meaning when i=jNo effective difference for ij versus ji

19

Motivating Concept of MDS

• Dissimilarities behave like distances.ij should correspond to dij.

• Configuration should place similar objects near each other.

20

Advantages of MDS

• Uncover hidden structure of data

• Reduce to a few dimensions– Compare objects at opposite ends– Examine physical configuration of objects

21

Common applications:

• Psychologists– Perception of speech and musical tones– Perception of colors and faces

• Anthropologists– Comparing different cultural groups

• Marketing researchers– Consumer reactions to products

22

Multidimensional Scaling for Topaz

• Dissimilarity is analogous to distance.• Given only the dissimilarities, produce the

configuration.• If configuration is good, unlike objects will

be far apart.• More complicated than distances:

– Noise in the data– Number of dimensions is unknown

23

Types of MDS: d = f()

• Metric– d = b– d = a + b

• Nonmetric– ordinal relationship, increasing or decreasing

• Choice of function has little effect on configuration

24

Computational Approach

• Define an objective function (badness-of-fit).– Stress is sum of squared discrepancies divided

by a scaling factor.

• Specify f.

• Find the best configuration X to minimize the objective function.

25

Determining Dimensionality

• Useful– People can focus on only a few attributes

– Difficult to interpret or display more than 3

• Rule of thumb– R (I-1)/4

• Statistical– range of

reasonabledimensions

1 2 3 4 5 6 7

Dimensions

0

5

10

15

Stress

26

Three-Way MDS

• Can use average dissimilarities in two-way MDS.

• ALSCAL/PROXSCAL in SAS, SPSS, make full use of information from multiple judges

• For Topaz data, judges were very consistent, not much difference with 3-way MDS.

27

The Configuration

-0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8

Dim1

-0.5

-0.3

-0.1

0.1

0.3

0.5

Dim2

d1

p4

m3

e2u1

x1

h3

u2

f5

s3

s4

p2

a2

h1

c5

y1w3

s2

w1

y4

28

Shepard Diagram

0 1 2 3 4

Dissimilarity

0.0

0.4

0.8

1.2

Distance

29

Interpretation

• Examine the Configuration!• Dimensional Approach

– Regress other variables on coordinates• Attends most to large distances

• Neighborhood Approach– Shared characteristics in the same region

• Focus is on small distances

• Eclectic Approach– Use any means at your disposal to understand as much

as possible about the configuration

30

Examining the Configuration

-0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8

Dim1

-0.5

-0.3

-0.1

0.1

0.3

0.5

Dim2

d1

p4

m3

e2u1

x1

h3

u2

f5

s3

s4

p2

a2

h1

c5

y1w3

s2

w1

y4

Glossy,Fine texture

Lighter,More metallic

Darker,Less metallic

Finer

Coarser

31

Relating Effects to Dimensions

-0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8

Dim1

-0.5

-0.3

-0.1

0.1

0.3

0.5

Dim2

conical

conical

conical

slotconical

slot

slot

slot

slot

conical

slot

slot

conical

conical

conical

slotslot

conical

slot

slot

Lighter,More metallic

Darker,Less metallic

32

Relating Effects to Dimensions

-0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8

Dim1

-0.5

-0.3

-0.1

0.1

0.3

0.5

Dim2

fine

coarse

coarse

finefine

fine

lab

lab

lab

coarse

coarse

lab

fine

fine

lab

finecoarse

lab

fine

lab

Finer

Coarser

33

Hierarchical Clustering – 2D

-0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8

Dim1

-0.5

-0.3

-0.1

0.1

0.3

0.5

Dim2

d1

p4

m3

e2u1

x1

h3

u2

f5

s3

s4

p2

a2

h1

c5

y1w3

s2

w1

y4

34

Hierarchical Clustering – 3D

Components:

Dim1

Dim2

Dim3

a2

d1 h1

s2

u1

e2w1

y1

x

y

z

Spinning Plot

35

Gun Tip Effect

-0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8

Dim1

-0.5

-0.3

-0.1

0.1

0.3

0.5

Dim2

conical

conical

conical

slotconical

slot

slot

slot

slot

conical

slot

slot

conical

conical

conical

slotslot

conical

slot

slot

36

Charge Effect

-0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8

Dim1

-0.5

-0.3

-0.1

0.1

0.3

0.5

Dim2

60

100

60

100100

100

100

100

60

100

100

60

100

100

100

60100

60

60

100

37

Additive Effect

-0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8

Dim1

-0.5

-0.3

-0.1

0.1

0.3

0.5

Dim2

C

none

A

Anone

none

C

none

A

A

none

none

A

C

A

AC

none

none

A

38

Equipment Effect

-0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8

Dim1

-0.5

-0.3

-0.1

0.1

0.3

0.5

Dim2

production

production

production

productionproduction

production

lab

lab

lab

production

production

lab

production

production

lab

productionproduction

lab

production

lab

39

Particle Size Effect

-0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8

Dim1

-0.5

-0.3

-0.1

0.1

0.3

0.5

Dim2

fine

coarse

coarse

finefine

fine

lab

lab

lab

coarse

coarse

lab

fine

fine

lab

finecoarse

lab

fine

lab

40

Gun Type Effect

-0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8

Dim1

-0.5

-0.3

-0.1

0.1

0.3

0.5

Dim2

cup

cup

cup

fluidcup

cup

cup

cup

cup

fluid

cup

fluid

cup

fluid

cup

cupfluid

cup

cup

fluid

41

Topaz Results

• Created a formulation the customer accepted.

• Discovered that gun tip is critical to the appearance of this formula.

• Identified variables to consider when trouble-shooting complaints.

42

Summary

• Factorial experiments allow the estimation and comparison of many effects.

• Paired comparisons simplify the task of judging appearance.

• Multidimensional scaling reveals underlying dimensions of appearance.

43

References

• DAVID, HERBERT A. (1988) “The Method of Paired Comparisons.” London: Charles Griffin & Company Limited.

• KRUSKAL, JOSEPH B. and WISH, MYRON (1978) “Multidimensional Scaling.” Sage University Paper series on Quantitative Applications in the Social Sciences, 07-011. Newbury Park and London: Sage Pubns.

• VENABLES, W. N. and RIPLEY, B. D. (1994) “Modern Applied Statistics with S-Plus.” New York: Springer.

44

Software

• SPSS

• SAS

• S-Plus– MASS library from Venables & Ripley

• MASS@stats.ox.ac.uk