A Genetic Algorithm Approach A Genetic Algorithm Approach to Multiple Response to Multiple Response
OptimizationOptimization
Francisco Ortiz Jr.
James R. Simpson
Joseph J. Pignatiello, Jr.
Alejandro Heredia-Langner
Department of Industrial EngineeringFlorida A&M and Florida State University
• Many industrial problems involve many response solutions – “It is not unusual to find RSM applications with 20 or more
responses…” (Montgomery, JQT, 1999)
• Current methods don’t always work
• Work by Heredia-Langner, and suggestions from Carlyle, Montgomery and Runger (JQT, 2000)
Motivation
Department of Industrial EngineeringFlorida A&M and Florida State University
CompositesProduction
INPUTS (Factors) OUTPUTS (Responses)
PROCESS:
Resin Transfer Molding
Resin Flow Rate
Mold Complexity
Gate Location
Type of Resin
Fiber Weave
Fiber Weight
Resin Flow Properties
Tensile Strength
Fiber Permeability
Dynamic Mechanical Analysis
Compression Strength
Flexural Strength
Acoustic Properties
Department of Industrial EngineeringFlorida A&M and Florida State University
Charge Control Agents
Charge Voltage
Colorants
Release Agents
Transfer Voltage
Surface Additives
Laser Toner Development
INPUTS (Factors) OUTPUTS (Responses)
PROCESS:
Toner Transfer
Background
Developer Roll Mass/Area
Spitting
Powder Flow
Isopel Optical Density
Toner-to-Cleaner
Film Onset
24 responses
Department of Industrial EngineeringFlorida A&M and Florida State University
• As the number of response and decision variables increases– The combined response function typically can be highly
nonlinear, multi-modal and heavily constrained – Conventional optimization methods can get trapped at a
local optimum and even fail to find feasible solutions
Challenges for Current Multiple Response Optimization Methods
Department of Industrial EngineeringFlorida A&M and Florida State University
• This paper develops and evaluates a multiple response solution technique using a genetic algorithm in conjunction with an unconstrained desirability function
Purpose
Department of Industrial EngineeringFlorida A&M and Florida State University
• A desirability function for the genetic algorithm
• “Robust-ising” the genetic algorithm for multiple response optimization
• Performance evaluation and comparison
Topics to Cover
Department of Industrial EngineeringFlorida A&M and Florida State University
• Most methods convert multiple responses with different units of measurements into a single commensurable objective
• Distance or Loss Functions– Khuri and Conlon, 1981– Pignatiello, 1993– Vining, 1998
• Desirability Methods– Derringer and Suich, 1980– Del Castillo, Montgomery, and McCarville, 1996
• Desirability approach converts individual response ŷi into individual desirability di(ŷi)
• Used in combination with optimization algorithm to locate a single solution
Combining Responses
Department of Industrial EngineeringFlorida A&M and Florida State University
• For example, target as the goal response• Constraint developed based on response’s utility• 0 < di(ŷi) < 1
• Overall desirability
• Optimization using Nelder-Mead simplex or GRG
Desirability Approach
ˆˆ,
ˆˆ ˆ( ) ,
0 , otherwise
i
i
s
i ii i i
i i
t
i ii i i i i
i i
y LL y T
T L
y Hd y T y H
T H
si ti
1/
1 1 2 2ˆ ˆ ˆm
m mD d y d y d y x
Department of Industrial EngineeringFlorida A&M and Florida State University
• A genetic algorithm (GA) is a search technique that is based on the principles of natural selection
– “survival of the fittest”• Uses the objective function magnitude directly in the search• The GA generates and maintains a population pool of
solutions throughout • It then evaluates the quality of each individual chromosome
using a fitness function – In this case the desirability function = fitness function
Genetic Algorithm
Department of Industrial EngineeringFlorida A&M and Florida State University
• A designed experiment produces an empirical system model
• A coding scheme widely used is to transform the natural variables into coded variables that fall between -1 and 1
• Here we have multiple fitted responses
),...,(ˆ 1 kxxfy
x1 x2 x3 x4 x5 x6 x7 x8
-0.451 0.346 0.518 0.701 -0.573 -0.634 0.937 -0.448
1 2ˆ ˆ ˆ ˆ[ , ,..., ]my y y y
Using the GA in Response Surface Studies
Chromosome
Gene
Department of Industrial EngineeringFlorida A&M and Florida State University
• Multiplicative desirability functions– Overall desirability is
– If any di (x) = 0, D(x) = 0
– The GA is unable to compare infeasible solutions
– Perhaps we could extend the D(x) function
Limitations of Current Desirability for the GA
1/
1 1 2 2ˆ ˆ ˆm
m mD d y d y d y x
Department of Industrial EngineeringFlorida A&M and Florida State University
• GA can be designed to handle constraint violations by using a penalty method
• where
Formulating a Desirability Function
*( ) ( ) ( )DSD D P x x x
21/
1 1 2 2ˆ ˆ ˆ( ) ( )... ( )m
m mP p y p y p y c x
Department of Industrial EngineeringFlorida A&M and Florida State University
• Every fitted response has an individual desirability di(ŷ) and a penalty pi(ŷ)
• where c is a small constant (e.g. c= 0.0001)
Formulating a Desirability Function
ˆˆ,
ˆˆ ˆ( ) ,
0 , otherwise
i
i
t
i ii i i
i i
s
i ii i i i i
i i
y HT y H
T H
y Ld y L y T
T L
ˆˆ ,
ˆ ˆ( ) ,
ˆˆ ,
i ii i
i i
i i i i i
i ii i
i i
y Lc y L
T L
p y c L y H
y Hc H y
T H
Department of Industrial EngineeringFlorida A&M and Florida State University
• Hence, the overall unconstrained desirability function is
• The penalty function enables the GA to find feasible solutions
• After feasible solutions are found, P(x) = 0, and no effect on the D*(x)
Formulating a Desirability Function
21/ 1/*
1 1( ) ( ) ( ) ( ) ( )m m
m mD d y d y p y p y c x
Department of Industrial EngineeringFlorida A&M and Florida State University
• Overall GA desirability function for a case with one response and linear weights (s1 = t1= 1) for the d(y1)
Unbounded Desirability Function
D*
Department of Industrial EngineeringFlorida A&M and Florida State University
• Proposed desirability function D*(x) vs. DDS (x)
Desirability Function Investigation
Responses Value Low Target High d i d i p i
ŷ1 60 50 60 70 1 1 0.000001
ŷ2 75 50 75 100 1 1 0.000001
ŷ3 50 47 50 53 1 1 0.000001ŷ4 30 40 45 50 0 0 2.000001
0
Responses Value Low Target High d i d i p i (penalty)
ŷ1 60 50 60 70 1 1 0.000001
ŷ2 45 50 75 100 0 0 0.200001
ŷ3 60 47 50 53 0 0 2.333334ŷ4 30 40 45 50 0 0 2.000001
0
Case # 1 Chromosome #1
Overall Desirability -1.34E-09
Case # 1 Chromosome #2
Overall Desirability -9.66E-04
)(xDSD )(* xD
)(xDSD )(* xD
)(xDSD
)(xDSD
)(* xD
)(* xD
Department of Industrial EngineeringFlorida A&M and Florida State University
• Proposed desirability function D*(x) vs. DDS (x)
Desirability vs. Generations
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
0 5 10 15 20 25 30 35 40 45 50
Generations
De
sira
bil
ity
GA DF
D-S DF
GA Performance Investigation
Department of Industrial EngineeringFlorida A&M and Florida State University
• The GA can be sensitive to the many parameter choices that must be made for its application
• The GA parameters considered for this study are– Population size– Parent-to-Offspring ratio– Selection type– Mutation type– Mutation rate– Crossover rate
Tuning the GA Parameters
x1 x2 x3 x4 x5 x6 x7 x8
-0.257 -0.914 0.028 0.383 0.429 -0.111 -0.207 -0.812
x1 x2 x3 x4 x5 x6 x7 x8
-0.451 0.346 0.518 0.701 -0.573 -0.634 0.937 -0.448
Department of Industrial EngineeringFlorida A&M and Florida State University
• Done by incorporating four problem environment parameters incorporated into a designed experiment framework
• Considered noise variables for the purposes of robust design
Factor Problem Parameters Low High A Number of Decision Variables 4 8B Number of Responses 4 16C Percent of 2nd order Models 25 75D Constraints (%of target) 5 15
Creating Multiple Response Problem Scenarios
Department of Industrial EngineeringFlorida A&M and Florida State University
• Certain rules were used to ensure that the problems generated mimic what typically is found in a real world situation
• Specifications – The decision variables that appear in each response are
chosen randomly– For second order models, ½ of the terms are linear (main
effects), ¼ of the terms are interactions, ¼ of the terms are pure quadratic
– The exact values of the regression coefficients are selected from a uniform probability distribution U(5, 20)
– Model hierarchy is always maintained
Creating Multiple Response Problem Scenarios
Department of Industrial EngineeringFlorida A&M and Florida State University
• For example, a case with 4 responses and 4 decision variables, 75% of the responses are second order models
Creating Multiple Response Problem Scenarios
21 1 2 1 2 1
2 2 3 2 3
23 1 2 1 2 3
24 3 4 3 4 4
ˆ 50 16 6 12 7
ˆ 50 19 8 13
ˆ 50 19 18 8 11
ˆ 50 7 19 10 14
y x x x x x
y x x x x
y x x x x x
y x x x x x
Department of Industrial EngineeringFlorida A&M and Florida State University
Factor GA Parameters Low High TypeA Parent Pop. Size 20 50 ControlB Parent/Offspring Ratio 1:1 1:07 ControlC Selection Type Ranking Tournament ControlD Crossover Rate 0.5 0.85 ControlE Mutation Type Uniform Gaussian ControlF Mutation Rate 0.1 0.4 Control
Problem Parameters Low High G Number of Decision Variables 4 8 NoiseH Number of Responses 4 16 NoiseJ Percent of 2nd order Responses 25 75 NoiseK Constraints (% of target) 5 15 Noise
Robust Designed Experiment
• Robust parameter design using a combined array 210-4 resolution IV (80 runs includes 16 pseudo-centers)
Department of Industrial EngineeringFlorida A&M and Florida State University
Robust Designed Experiment Results
• GA performance metrics were the number of evaluations until– Feasible– Within 10% of optimal, or D*(x) > 0.90
• Response model of the form
• Significant effects included– Noise factor main effects– Control x control interactions– Control x noise interactions
6 6 6 4 6 4
01 1 1, 1 1 1
ˆ , i i ij i j i i ij i ji i j j i i i j
y b b x b x x c z d x z
x z
Department of Industrial EngineeringFlorida A&M and Florida State University
Robust Designed Experiment Results
• Response: Achieving D*(x) > 0.90
Term Term Type FindingAC Control x Control For parent population = 20, selection type does not matter, whereas if the
parent population = 50, choose tournament selectionAE Control x Control Better performance is achieved with parent population = 20 and Gaussian
mutationAF Control x Control Performance is enhanced using the parent population = 20 and a mutation
rate = 0.4BK Control x Noise An offspring ratio of 1:7 is best with tight constraints (5%), but as the
constraints widen, the offspring ratio is unimportantEG Control x Noise Gaussian mutation works better with only 4 decision variables and both
mutation methods perform equally with 8 decision variablesEK Control x Noise Gaussian performs better with tight constraints (5%), but as the constraints
widen, the mutation type is insignificantAEK Control x Noise For mutation rates of 0.1, use a parent population = 20 and Gaussian
mutation. For all other mutation rates, factor AE is not significantCFK Control x Noise For tournament selection, use mutation rate = 0.4 regardless of constraint
setting. For ranking selection, the FK interaction is not significant
Department of Industrial EngineeringFlorida A&M and Florida State University
Robust GA Parameter Settings
• All but one determined by response surface models
Factor Parameters Setting Rationale
A Parent Population 20 Designed experiment result
B Parent/Offspring Ratio 1:7 Designed experiment result
C Selection Type Tournament Designed experiment result
D Crossover Rate 0.85 Diversify offspring pool
E Mutation Type Gaussian Designed experiment result
F Mutation Rate 0.4 Designed experiment result
Department of Industrial EngineeringFlorida A&M and Florida State University
Performance Evaluation
• Evaluate and compare proposed GA method• Considered the multiple response problem scenarios
– Dropped percentage of second order models– 23 factorial plus a center point
Factor A Factor B Factor CCase # No. of Decision Variables No.of Responses Constraint Range
1 4 4 52 8 4 53 4 16 54 8 16 55 4 4 156 8 4 157 4 16 158 8 16 159 6 10 10
Department of Industrial EngineeringFlorida A&M and Florida State University
• Results of investigation using 30 starting point locations
GRG Performance
Case # No. of Variables No.of Responses ConstraintsPct of GRG Solutions
Optimal1 4 4 5 0.302 8 4 5 0.073 4 16 5 0.074 8 16 5 0.005 4 4 15 0.836 8 4 15 0.277 4 16 15 0.578 8 16 15 0.009 6 10 10 0.07
Department of Industrial EngineeringFlorida A&M and Florida State University
• Performance of proposed GA using four replicates
Case # No. of Variables No.of Responses Constraints Mean St. Dev. High Low1 4 4 5 0.989 0.004 0.994 0.9852 8 4 5 0.969 0.011 0.979 0.9583 4 16 5 0.997 0.002 0.998 0.9954 8 16 5 0.987 0.005 0.991 0.9825 4 4 15 0.998 0.001 0.998 0.9976 8 4 15 0.994 0.003 0.997 0.9927 4 16 15 0.987 0.004 0.991 0.9838 8 16 15 0.958 0.027 0.984 0.9319 6 10 10 0.979 0.016 0.995 0.962
(95.0% )Confidence Intervals
GA Performance Investigation
Department of Industrial EngineeringFlorida A&M and Florida State University
• Current study shows consistent, effective performance
• Can be effectively combined with direct search methods (e.g. GA with GRG) to improve run-time performance
• Flexible to handle a host of objective functions– Distance or loss functions can all be applied as long as
covariance information is available
• Able to perform reasonable mapping of response function over design space
Why Use the GA for Multiple Response Optimization?
Department of Industrial EngineeringFlorida A&M and Florida State University
• 2.4 Genetic Algorithm
– Coding
00001010000110000000011000101010001110111011
is partitioned into 2 halves
0000101000011000000001 and 1000101010001110111011
these strings are then converted from base 2 to base 10 to yield:
x1 = 165377 and x2 = 2270139
The following is an example of real-value coding
0.125 -1.000 0.525
Department of Industrial EngineeringFlorida A&M and Florida State University
• Recombination – Exchange information/genes between parent chromosome to make new offspring
Parent 12500 85 1780 103 1200 95 1900 98 2500 168 500 40
Parent 21850 53 2200 99 785 67 1000 102 750 75 900 69
Offspring2500 85 1780 103 1200 95 1900 98 750 75 900 69
Crossover Point
• GA cannot rely on recombination alone.
Department of Industrial EngineeringFlorida A&M and Florida State University
• Mutation – Uniform mutation– Multiple uniform mutation – Guassian mutation
• All entities of the chromosome are mutated such that the resulting chromosome lies somewhere within the neighborhood of it parent.
X1
X2