24 September 2004 David Drain 1

Experiment Design

• Suppose you want to do some experiments to improve a manufacturing process you think may be influenced by a dozen different factors– What do you really want from the experiment –

what does “optimal” mean?– How can you most economically get the

information you want?

24 September 2004 David Drain 2

An Experiment

• Three “factors” (potential influences)

• Seven “runs” – the process is run at this combination of settings and data is collected

X1 X2 Z1-1.00 -1.00 0.22-1.00 -0.70 0.00-1.00 -0.45 -0.220.00 0.00 0.001.00 0.45 0.221.00 0.70 0.001.00 1.00 -0.22

24 September 2004 David Drain 3

Noise Variables and Process Targets

• A robust process target gives good results regardless of noise variables

• Control-noise interaction terms measure robustness

2 20 1 1 2 2 11 1 22 2 12 1 2

1 1 2 2

11 1 1 12 2 1 21 2 1 22 2 2

y x x x x x x

z z

x z x z x z x z

24 September 2004 David Drain 4

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

-2 -1 0 1 2 3

Constraints in Industrial Design Problems

• Number of runs

• Mandatory center points

• Blocks

• Constraints on control variables

24 September 2004 David Drain 5

Contour Plot of PEV

• Plot of prediction error variance for modified CCD with 2 control and 2 noise variables:

C7M 40 50 60 75 95 135 250

X2 .

-2

-1

0

1

2

X1-2 -1 0 1 2

1 0.75

0.75 1

0.80 0.50 0.66

0.225 0.50 0.66

ZV

24 September 2004 David Drain 6

Sensitivity of a Classic DesignFCC with 23 runs

F 7 9 2 6 2 7 2 9 3 1 3 2 3 4 3 6

X 2 .

-1 .0

-0 .5

0 .0

0 .5

1 .0

X 1

-1 .0 -0 .5 0 .0 0 .5 1 .0

No noise Noise parameters of magnitude 0.18

F 1 5 2 6 2 7 2 9 3 1 3 2 3 4 3 6

X 2 .

-1 .0

-0 .5

0 .0

0 .5

1 .0

X 1

-1 .0 -0 .5 0 .0 0 .5 1 .0

24 September 2004 David Drain 7

Design Discovery Methods

• Try the classic designs (CCD,…)• Exchange methods

– Choose candidates, then an algorithm (DETMAX, for example) selects a “best” subset

– Candidate set must include optimal runs to get optimal design

– Selecting too many candidates makes the algorithm computationally intractable

• Heuristic methods

24 September 2004 David Drain 8

Hybrid Heuristics

• A hybrid heuristic is some combination of basic heuristics

• Starting a GA with a random population is a simple hybrid: scattersearch followed by GA

• Talbi (2002) gives a complete taxonomy of hybrid heuristics

24 September 2004 David Drain 9

The No Free Lunch Theorem

• Scattersearch is just as good as an efficient hillclimbing algorithm, if their performance is averaged over all possible problems.

• No heuristic can be applied successfully to all problems: the heuristic must be chosen to match the problem being solved.

“Roughly speaking, the average performance of any pair of algorithms across all possible problems is identical.” Wolpert and Macready (1997)

24 September 2004 David Drain 10

Genetic Algorithm

• Based on evolution: genetic material from two parents is combined to produce offspring, and some mutation is allowed

Parent A Parent B Child

Crossover

24 September 2004 David Drain 11

Simulated Annealing

• Based on physical annealing processes• Early in the process, while at a high

“temperature”, relatively large changes from local optimum are allowed. – Disadvantageous changes are allowed with a

probability inversely related to temperature

• Requires multiple starting points because of tendency to become trapped at local optima

24 September 2004 David Drain 12

GASA: Genetic Algorithm – Simulated Annealing Hybrid

• Integrated hybrid of GA and SA– Exploration with temperature-controlled GA– Exploitation with SA within each GA step– Discrete changes with GA (entire runs)– Continuous changes with SA – Temperature parameters tunable for

computational efficiency and effectiveness

24 September 2004 David Drain 13

GASA Algorithm

Read starting parameters

Iterate: Adjust GA parameters

Crossover

Mutate

SA sub-algorithm

Report results

24 September 2004 David Drain 14

GASA Initialization

• Initialize population from candidate list• Read test vectors, or generate random test vectors• Define objective function

– Desirability functions are supported

• Read starting parameters• Define rules for parameter modification by

iteration

24 September 2004 David Drain 15

GASA Parameters

• Number of iterations

• Population size

• Number of elites

• Cross-breeding requirements

Roulette rate Mutation rate Number of test vectors

used for objective function evaluation

Distance and probability rules for SA

Fixed Iteration Dependent

24 September 2004 David Drain 16

Using GASA Effectively

• Parameter choices have a significant effect on algorithm effectiveness and computational efficiency

• Some parameter settings can be determined through the use of designed experiments (these must estimate interactions)

• Ideal parameter values seem to depend on the stage of optimization:– Early in the process, high mutation rates make sense– Late in the process, more intense SA may be needed– Dynamic parameter adjustment is a necessity

24 September 2004 David Drain 17

Using GASA Effectively

• Use the best starting points available• Ensure sufficient genetic diversity

– Unintentional cloning occurs otherwise– Large populations and forced crossbreeding enhance

genetic diversity

• Choose discriminating objective functions– No population member should achieve a perfect score,

and most members should not receive a failing score– Objective function parameters might need to be refined

as optimum is approached

24 September 2004 David Drain 18

GASA Results

• GASA was validated on a problem recently solved by Borkowski (2003)– D-optimal design for a quadratic model in a square

region using six runs

• GASA can improve the best known designs• Greatest benefits are in slope estimate variance

reduction and rotatability, rather than prediction error variance reduction

• Smaller designs exhibit the most improvement

24 September 2004 David Drain 19

Rotatability Improvement

C7M 40 50 60 75 95 135 250

X2 .

-2

-1

0

1

2

X1-2 -1 0 1 2

Win23M 30 45 60 90150 200 300

X2 .

-2

-1

0

1

2

X1

-2 -1 0 1 2

24 September 2004 David Drain 20

Future Research

• Parameter setting optimization• Dynamic and adaptive parameter determination• “Neighborhood” definition (for SA) based on

design properties rather than Euclidean distance• Find alternatives to simulated annealing for

exploitation mechanism– Simple hill-climbing did not work well– Gravitating to areas of high variance did not work well

24 September 2004 David Drain 21

References• Borkowski, J.J. (2003). "Using a Genetic Algorithm to Generate Small Exact Response Surface

Designs". Journal of Probability and Statistical Science 1, pp. 65-88.• Borror, C. M.; Montgomery, D. C. and Myers, R. H. (2002). “Evaluation of Statistical Designs

for Experiments Involving Noise Variables”. Journal of Quality Technology, 34, pp. 54-70.• Box, G. E. P. and Wilson, L. B. (1951). “On the Experimental Attainment of Optimum

Conditions”. Journal of the Royal Statistical Society B, 13, pp. 1-45. • Forouraghi, B. (2000). "A Genetic Algorithm for Multiobjective Robust Design". Applied

Intelligence 12, pp. 151-161.• Haines, L. M. (1987). "The Application of the Annealing Algorithm to the Construction of Exact

Optimal Designs for Linear-Regression Models". Technometrics 29, pp. 439-447. • Hamada, M.; Martz, H.F.; Reese,C.S. and Wilson, A.G. (2001). "Finding Near-Optimal Bayesian

Experimental Designs via Genetic Algorithms". The American Statistician 55, pp.175-181.• Heredia-Langner, A.; Carlyle, W.M.; Montgomery, D.C.; Borror, C.M. and Runger, G.C. (2003).

“Genetic Algorithms for the Construction of D-Optimal Designs”. Journal of Quality Technology, 35, pp. 28-46.

• Kragelund, L. V. (1997). "Solving a Timetabling Problem using Hybrid Genetic Algorithms". Software- Practice and Experience, 27, pp. 1121-1134.

24 September 2004 David Drain 22

References (cont.)• Li, W. D.; Ong, S. K. and Nee, Y. C. (2002). "Hybrid genetic algorithm and simulated

annealing approach for the optimization of process plans for prismatic parts". International Journal of Production Research, 40, pp. 1899-1922.

• Myers, R. H.; Khuri, A. I. and Vining, G. (1992). "Response Surface Alternatives to the Taguchi Robust Parameter Design Approach". American Statistician, 46, pp. 131-139.

• Parkinson, D.B. (2000). "Robust Design Employing a Genetic Algorithm". Quality and Reliability Engineering International, 16, pp. 201-208.

• Taguchi, G. (1986). Introduction to Quality Engineering: Designing Quality into Products and Processes. Kraus International Publications, White Plains, NY.

• Talbi, E.-G. (2002). "A Taxonomy of Hybrid Metaheuristics". Journal of Heuristics 8, pp. 541-564

• Welch, W.J. (1982). "Branch-and-Bound Search for Experimental Designs Based on D Optimality and Other Criteria". Technometrics 24, pp. 41-48.

• Wolpert, D.H. and Macready, W.G. (1997). "No Free Lunch Theorems for Optimization". IEEE Transactions on Evolutionary Computation, 1, pp. 67-82.