an empirical study of the prediction performance of space-filling designs rachel t. johnson douglas...
TRANSCRIPT
An Empirical Study of the Prediction Performance of Space-filling Designs
Rachel T. Johnson
Douglas C. Montgomery
Bradley Jones
Computer Models
• Widely used in engineering design
• Use continues to grow
• May have lots of variables, many responses
• Can have long run times
• Complex output results
• Need efficient methods for designing the experiment and analyzing the results
2Johnson QPRC 2009
Computer Simulation Types
Discrete Event
Simulation(DES)
Computational Fluid
Dynamics(CFD)
3Johnson QPRC 2009
Computer Simulation Models
Stochastic Simulation Models
Deterministic Simulation Models
Comparing Designs
• Space-filling designs compared– Sphere packing (SP)– Latin Hypercube (LH)– Uniform (U) – Gaussian Process Integrated Mean Square Error
(GP IMSE)
• Assumed surrogate model– Gaussian Process model
• Comparisons based on prediction
Johnson QPRC 2009 4
Designs
Johnson QPRC 2009 5
-1
-0.5
0
0.5
1
X2
-1 -0.5 0 0.5 1
X1
Sphere Packing Sphere Packing:• Johnson et al. (1990)• Maximizes the minimum distance between pairs of design points
-1
-0.5
0
0.5
1
X2
-1 -0.5 0 0.5 1
X1
Maximum Entropy
-1
-0.5
0
0.5
1
X2
-1 -0.5 0 0.5 1
X1
Latin HypercubeLatin Hypercube:• McKay et al. (1979)• A random n x s matrix, in which columns are a random permutation of {1, . . ., n}
-1
-0.5
0
0.5
1
X2
-1 -0.5 0 0.5 1
X1
Uniform
Uniform:• Feng (1980)• A set of n points uniformly scattered within the design space
-0.75
-0.5
-0.25
0
0.25
0.5
0.75
X2
-0.75 -0.5 -0.25 0 0.25 0.5
X1
GASP IMSE
GP IMSE:• Sacks et al. (1989) • Minimizes the integrated mean square error of the Gaussian Process model
-1
-0.5
0
0.5
1
X2
-1 -0.5 0 0.5 1
X1
I - OptimalI - Optimal:• Box and Draper (1963)• Minimizes the average prediction variance (of a linear regression model) over a design region
Maximum Entropy:• Shewry and Wynn (1987) • Maximizes the information contained in the distribution of a data set
Model Fitting Techniques
• Gaussian Process Model
– Assumes a normal distribution
– Interpolator based on correlation between points – Prediction variance calculated as
6Johnson QPRC 2009
Evaluation of Designs for the GP Model
• Recall the prediction variance:
• Prediction variance dependent on: – Design – Value of unknown θ– Sample size– Dimension of x
• Design of Experiments can be used to evaluate the designs
7Johnson QPRC 2009
Example FDS Plots
Sphere Packing
Uniform
GP IMSE
Latin Hypercube
8Johnson QPRC 2009
Maximum Entropy
Research Questions
• Does it matter in practice what design you choose? – Is there a dominating experimental design that performs better in terms
of model fitting and prediction?
• What is the role of sample size in experimental designs used to fit the GASP model? – At what point does prediction error variance and other measures of
prediction performance become reasonably small with respect to N, the sample size chosen?
Johnson QPRC 2009 9
Empirical Comparison
• Used test functions to act as surrogates to simulation code
• Evaluated designs based on RMSE and AAPE
• Interested in effect of: – Design type – Sample size – Dimension of design – *Gaussian Correlation Function
Johnson QPRC 2009 10
Comparison Procedure
• Step 1: Choose a test function
• Step 2: Choose a sample size and space-filling design
• Step 3: Create the design with the number of factors equal to the number in the test function chosen in step 1) and the specifications set in step 2)
• Step 4: Using the test function in 1) find the values that correspond to each row in the design
• Step 5: Fit the GASP model
• Step 6: Generate a set of 40,000 uniformly random selected points in the design space and compare the predicted value (generated by the fitted GASP model) to the actual value (generated by the test function) at these points. Compute the Root mean square error (RMSE).
Johnson QPRC 2009 11
Test Function #4
Test Function #3
Test Function #2
Test Function #1
12
Test Functions
Johnson QPRC 2009
Test Function #1 Results
Johnson QPRC 2009 13
RM
SE
-1
0
1
2
3
4
5
6
7
8
9
10
11
GP
IMS
E
LHD
SP U
GP
IMS
E
LHD
SP U
GP
IMS
E
LHD
SP U
SS_10 SS_20 SS_40
Design within Sample Size
Test Function #2 Results
Johnson QPRC 2009 14
RM
SE
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
GP
IM
SE
LH
D
SP U
GP
IM
SE
LH
D
SP U
GP
IM
SE
LH
D
SP U
SS_10 SS_20 SS_40
Design within Sample Size
Test Function #3 Results
Johnson QPRC 2009 15
RM
SE
0
5
10
15
20
GP
IMS
E
LHD
SP U
GP
IMS
E
LHD
SP U
GP
IMS
E
LHD
SP U
SS_10 SS_20 SS_40
Design within Sample Size
Test Function #2 Results
Johnson QPRC 2009 16
RM
SE
0
10
20
30
40
50
GP
IMS
E
LHD
SP U
GP
IMS
E
LHD
SP U
GP
IMS
E
LHD
SP U
SS_10 SS_20 SS_40
Design within Sample Size
ANOVA Analysis
Johnson QPRC 2009 17
-4
-2
0
2
4
Log(
RM
SE
)
-0.3
1293
±0.1
7581
5
TF 1
TF 2
TF 3
TF 4
TF 1
TF
GP
IMS
E
LHD S
P U
GP IMSE
Design
10 15 20 25 30 35 40
23.333
Sample Size
Jackknife Plots
Johnson QPRC 2009 18
-5
0
5
10
Tes
t F
un
ctio
n 2
-5 0 5 10
Test Function 2
Jackknife Predicted
-10
-5
0
5
10
Test
Fun
ctio
n 2
-10 -5 0 5 10
Test Function 2
Jackknife Predicted
Test Function #2 – LHD with a sample size of 20
Test Function #2 – LHD with a sample size of 50
Conclusions
• No one design type is better than the others• Increasing sample size decreases RMSE• There is a strong interaction between sample size and
test function type – the more complex the function the more runs required
• Jackknife plot is an excellent indicator of a “good fit”
Johnson QPRC 2009 19
References
• Fang, K.T. (1980). “The Uniform Design: Application of Number-Theoretic Methods in Experimental Design,” Acta Math. Appl. Sinica. 3, pp. 363 – 372.
• Johnson, M.E., Moore, L.M. and Ylvisaker, D. (1990). “Minimax and maxmin distance design,” Journal for Statistical Planning and Inference 26, pp. 131 – 148.
• McKay, N. D., Conover, W. J., Beckman, R. J. (1979). “A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code,” Technometrics 21, pp. 239 – 245.
• Sacks, J., Welch, W. J., Mitchell, T. J., and Wynn, H. P. (1989). “Design and Analysis of Computer Experiments,” Statistical Science 4(4), pp. 409 – 423.
• Shewry, M.C. and Wynn, H.P. (1987). “Maximum entropy sampling,” Journal of Applied Statistics 14, pp. 898 – 914.
20Johnson QPRC 2009
Correlation Function
Johnson QPRC 2009 21
Gaussian Correlation Function Cubic Correlation Function
Sample Size
Design
SS 50
SS 40
SS 20
SS 10
RMSE
GCF
RMSE
CCF
RMSE
GCF
RMSE
CCF
RMSE
GCF
RMSE
CCF
RMSE
GCF
RMSE
CCF
4
3
2
1
0
RM
SE
Boxplot of RMSE