AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 1
AIAA 2002-5531OBSERVATIONS ON CFD SIMULATION
UNCERTAINTIES
Serhat Hosder, Bernard Grossman, William H. Mason, and Layne T. Watson
Virginia Polytechnic Institute and State University Blacksburg, VA
Raphael T. HaftkaUniversity of Florida
Gainesville, FL
9th AIAA/ISSMO Symposium on Multidisciplinary Analysis and Optimization
4-6 September 2002Atlanta, GA
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 2
Introduction
• Computational fluid dynamics (CFD) as an aero/hydrodynamic analysis and design tool
• CFD being used increasingly in multidisciplinary design and optimization (MDO) problems
• CFD results have an associated uncertainty, originating from different sources
• Sources and magnitudes of the uncertainty important to assess the accuracy of the results
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 3
• Design uncertainties due to computational simulation error
• Optimization is an iterative procedure subject to convergence error.
• Estimating convergence error may require expensive accurate optimization runs
• Many simulation runs are performed in engineering design. (e.g., design of experiments)
• Statistical analysis of error
Motivation
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 4
• Find error characteristics of a structural optimization of a high speed civil transport
• Estimate error level of the optimization procedure
• Identify probabilistic distribution model of the optimization error
• Estimate mean and standard deviation of errors without expensive accurate runs
• Improve response surface approximation against erroneous simulation runs via robust regression
Objectives
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 5
+ x
z Leading edge radius (fixed)
Outboard LE sweep (fixed)
x y
v1
v3
Location of maximum thickness (fixed)
v2
v4
Wing semi span (fixed)
v5: Fuel Weight • 250 passenger aircraft,
5500 nm range, cruise at Mach 2.4
• Take off gross weight (WTOGW) is minimized
• Up to 29 configuration design variables including wing, nacelle and fuselage geometry, fuel weight, and flight altitude
• For this study, a simplified 5DV version is used
High Speed Civil Transport (HSCT)
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 6
• For each HSCT configuration, wing structural weight (Ws) is minimized by structural optimization (GENESIS) with 40 design variables
• Structural optimization is performed a priori to build a response surface approximation of Ws
Structural optimization of HSCT
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 7
design
Op
timu
mw
ing
stru
ctu
ralw
eig
ht
(Ws)
5 10 15 2040000
60000
80000
100000
120000
140000
Case 1Case 2
Case 1 Case 2
Average error in Ws
5.51% 5.34%
For Case 2, the initial design point was perturbed from that of Case 1, by factors between 0.1 ~ 1.9
In average, Case 2 has the same level of error as Case 1
The errors were calculated with respect to higher fidelity runs with tightened convergence criteria
Effects of initial design point on optimization error
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 8
• To get optimization error we need to know true optimum, which is rarely known for practical engineering optimization
• Optimization error = OBJ* - OBJ*true
• To estimate OBJ*true
• Find convergence setting to achieve very accurate optimization
• This approach can be expensive
Estimating error of incomplete optimization
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 9
x
f(x)
0 1 2 3 4 50.0
1.0
2.0
3.0
= 0.5, = 1 = 1.0, = 1 = 2.0, = 1 = 4.0, = 1
• Widely used in reliability models (e.g., lifetime of devices)• Characterized by a shape parameter and a scale parameter
otherwise
xifx
x
0
0exp1
2
)1
(1
)2
(2
)1
(
22 Variance
Mean
PDF function according to
The Weibull distribution model
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 10
• Maximum likelihood estimation (MLE) to estimate distribution parameters of the assumed distribution.
• Find to maximize the likelihood function.
2 goodness of fit test
• Comparison of histograms between data and fit
• p-value indicates quality of the fit
nx
xfl
i
n
ii
size of sample for
1
);()(
Estimation of distribution parameters
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 11
Errors of two optimization runs of different initial design point
s = W1 – Wt
t = W2 – Wt , (Wt is unknown true optimum.)
The difference of s and t is equal to W1 – W2
x = s – t = (W1 – Wt) – (W2 – Wt ) = W1 – W2
We can fit distribution to x instead of s or t.
• s and t are independent • Joint distribution of g(s)
and h(t)
g(s; 1)
s, t
h(t; 2)
x=s-t
dsxshsgxf )()(),;( 21
• MLE fit for optimization difference x using f(x)
• No need to estimate Wt
Difference fit to estimate statistics of optimization errors
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 12
Error in Ws (%)
Fre
qu
en
cie
s
0 10 20 30 40 50 60 700
20
40
60
80
DataExpected by error fitExpected by difference fit
• Difference fit is applied to the differences between Cases 1 and 2, assuming the Weibull model for both cases
• Even when only inaccurate optimizations are available, difference fit can give reasonable estimates of uncertainty of the optimization
Error in Ws (%)
Fre
qu
en
cie
s
0 10 20 30 40 50 60 70 800
20
40
60
80
100
DataExpected by error fitExpected by difference fit
Comparison between error fit and difference fit
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 13
Cases Case 1 Case 2 Average of Abs(W1-W2) 5941
From data 4458 4321 Error fit
(discrepancy) 4207
(-5.63%) 3952
(-8.54%) Estimate of mean, lb. Difference fit
(discrepancy) 3804
(-14.7%) 3481
(-19.4%)
From data 8383 9799 Error fit
(discrepancy) 7157
(-14.6%) 7505
(-23.4%) Estimate of STD, lb. Difference fit
(discrepancy) 9393
(12.0%) 9868
(0.704%)
p-value of 2 test 0.5494
• Another set of optimization with a difference initial design point was enough, which is straightforward and no more expensive
• The difference fit gave error statistics as accurate as the error fits involving expensive higher fidelity runs
Estimated distribution parameters
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 14
• Robust regression techniques• Alternative to the least squares fit that may be
greatly affected by a few very bad data (outliers)• Iteratively reweighted least squares (IRLS)
• Inaccurate response surface approximation is another error source in design process Improve response surface approximation by
repairing outliers
Outliers and robust regression
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 15
• Weighted least squares: Initially all points have same weight of 1.0.
• At each iteration, the weight is reduced for points that are far from the response surface.
• This gradually moves the response surface away from outliers.
• IRLS approximation leaves out detected outliers
x
y
IRL
Sw
eig
htin
g
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
1.2
1.4
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5DataLSIRLSweighting
Outliers
1 dimensional example
Iteratively Reweighted Least Squares
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 16
• Regular weighting functions• Symmetrical• Huber’s function gives non-
zero weighting to big outliers
• Biweight rejects big outliers completely
• Nonsymmetric weighting function• make use of one-sidedness
of error by penalizing points of positive residual more severelyResidual, r
We
igh
ting
,w(r
)
-4 -2 0 2 40
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6HuberBiweightNIRLS
Weighting functions of IRLS
Nonsymmetric weighting function
Aggressive search by reducing tuning constant B
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 17
RMSE(lb.) R2 Before repair
6755 (8.7%)
0.9297
IRLS repair
3257 (4.7%)
0.9769
NIRLS repair
3042 (4.1%)
0.9824
All 117 pts repaired
2578 (3.5%)
0.9879
Design
Ws
(lb
)
5 10 15 200
20000
40000
60000
80000
100000
120000
140000
GENESIS: RepairedRS: Without repairRS: IRLS repairRS: NIRLS repair
Comparison of RS before and after repair
Improvements of RS approximation by outlier repair
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 18
Drag polar results from 1st AIAA Drag Prediction Workshop (Hemsch, 2001)
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 19
Objective of the Paper
• Finding the magnitude of CFD simulation uncertainties that a well informed user may encounter and analyzing their sources
• We study 2-D, turbulent, transonic flow in a converging-diverging channel
• complex fluid dynamics problem• affordable for making multiple runs• known as “Sajben Transonic Diffuser”
in CFD validation studies
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 20
x/ht
y/h
t
-4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.00.0
0.5
1.0
1.5
2.0
2.5
y/h
t
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Transonic Diffuser Problem
y/h
t
0.3
0.4
0.5
0.6
0.7
0.8
0.9
x/ht
y/h
t
-4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.00.0
0.5
1.0
1.5
2.0
2.5
0.3
0.4
0.5
0.6
0.7
0.8
0.9
P/P0i
P/P0i
Strong shock case (Pe/P0i=0.72)
Weak shock case (Pe/P0i=0.82)
Separation bubble
experiment CFD
Contour variable: velocity magnitude
streamlines
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 21
Uncertainty Sources (following Oberkampf and Blottner)
• Physical Modeling Uncertainty• PDEs describing the flow
• Euler, Thin-Layer N-S, Full N-S, etc.• boundary conditions and initial conditions • geometry representation • auxiliary physical models
• turbulence models, thermodynamic models, etc.• Discretization Error• Iterative Convergence Error• Programming Errors
We show that uncertainties from different sources interact
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 22
Computational Modeling
• General Aerodynamic Simulation Program (GASP)• A commercial, Reynolds-averaged, 3-D, finite volume
Navier-Stokes (N-S) code• Has different solution and modeling options. An
informed CFD user still “uncertain” about which one to choose
• For inviscid fluxes (commonly used options in CFD)• Upwind-biased 3rd order accurate Roe-Flux scheme • Flux-limiters: Min-Mod and Van Albada
• Turbulence models (typical for turbulent flows) • Spalart-Allmaras (Sp-Al)• k- (Wilcox, 1998 version) with Sarkar’s
compressibility correction
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 23
Grids Used in the Computations
Grid level Mesh Size (number of cells)
1 40 x 25
2 80 x 50
3 160 x 100
4 320 x 200
5 640 x 400
A single solution on grid 5 requires approximately 1170 hours of total node CPU time on a SGI Origin2000 with six processors (10000 cycles)
y/ht
Grid 2
Grid 2 is the typical grid level used in CFD applications
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 24
Nozzle efficiency
Nozzle efficiency (neff ), a global indicator of CFD
results:
H0i : Total enthalpy at the inlet
He : Enthalpy at the exit
Hes : Exit enthalpy at the state that would be reached by
isentropic expansion to the actual pressure at the exit
esi
eieff HH
HHn
0
0
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 25
Uncertainty in Nozzle Efficiency
+ ++ + + +
+ + ++ + + + +
+ + + + +
+
+
x x x x x x x x x x x xx
xx
xx
xx
x
x
++ + + +
+ + + ++ + + + +
+ + +
+ + ++
++
x x x x x x x x x x x x x x x xx
xx
xx
x
x
o
o
o
o
+
xo
+
xo +
+
x
x
Pe/P0i
ne
ff
0.70 0.72 0.74 0.76 0.78 0.80 0.82 0.840.700
0.725
0.750
0.775
0.800
0.825
0.850
0.875
0.900
grid 1grid 2
grid 4grid 3
grid 5
Sp-Al k-
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 26
Discretization Error by Richardson’s Extrapolation
)( 1 ppexactk hOhff
Turbulence model
Pe/P0i estimate of p (observed order of
accuracy)
estimate of (neff)exact
Grid level
Discretization error (%)
Sp-Al
0.72
(strong shock)
1.322 0.71950
1 14.298
2 6.790
3 2.716
4 1.086
Sp-Al
0.82
(weak shock)
1.578 0.81086
1 8.005
2 3.539
3 1.185
4 0.397
k- 0.82
(weak shock)
1.656 0.82889
1 4.432
2 1.452
3 0.461
4 0.146
order of the method
a measure of grid spacing grid level
error coefficient
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 27
x/ht
y/h
t
-1.0 -0.5 0.0 0.5 1.0 1.5 2.00.98
0.99
1.00
1.01
1.02
1.03
1.04
1.05
1.06
1.07
1.08
1.09
modified experimentaldata points
upper wall contour obtainedwith the analytical equation
upper wall contour of the modified-wallgeometry (cubic-spline fit to the modified data points)
upper wall contour of the modified-wallgeometry (cubic-spline fit to the data points)
x/ht
y/h
t
-4.00 -3.00 -2.00 -1.00 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00
1.00
1.10
1.20
1.30
1.40
1.50 upper wall contour of the modified-wallgeometry (cubic-spline fit to the data points)
upper wall contour obtainedwith the analytical equation
Error in Geometry Representation
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 28
Error in Geometry Representation
x/ht
P/P
0i
-4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.00.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0Pe/P0i=0.72Top Wall
experiment
Sp-Al, Min-Mod, grid 2mw , wall contourfrom modified experimental data
Sp-Al, Min-Mod, grid 2,wall contour from equattion
Sp-Al, Min-Mod, grid 2mw , wall contourfrom experimental data
• Upstream of the shock, discrepancy between the CFD results of original geometry and the experiment is due to the error in geometry representation.
• Downstream of the shock, wall pressure distributions are the same regardless of the geometry used.
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 29
Downstream Boundary Condition
x/hty/
ht
0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.00.5
1.0
1.5
2.0Extended geometry, Sp-Al, Van Albada, grid 3ext, Pe/P0i=0.7468
x/ht
y/h
t
0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.00.5
1.0
1.5
2.0Extended geometry, Sp-Al, Van Albada, grid 3ext, Pe/P0i=0.72
x/ht
y/h
t
0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.00.5
1.0
1.5
2.0Original geometry, Sp-Al, Van Albada, grid 3, Pe/P0i=0.72
x/ht
y/h
t
2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.51.0
1.1
1.2
1.3
1.4
1.5
Extended geometry, Sp-Al,Van Albada, grid 3ext, Pe/P0i=0.72
Original geometry, Sp-Al, Van Albada, grid 3, Pe/P0i=0.72
Extended geometry, Sp-Al, Van Albada,grid 3ext, Pe/P0i=0.7468
x/ht
y/h
t
2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.51.0
1.1
1.2
1.3
1.4
1.5
x/ht
y/h
t
2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.51.0
1.1
1.2
1.3
1.4
1.5
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 30
Downstream Boundary Condition
x/ht
P/P
0i
-4.0 -2.0 0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.00.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0Top Wall
experiment
Sp-Al, Van Albada,grid 3ext, Pe/P0i=0.72
Sp-Al, Van Albada, grid 3
Sp-Al, Van Albada,grid 3ext, Pe/P0i=0.7468
Extending the geometry or changing the exit pressure ratio affect:
• location and strength of the shock
• size of the separation bubble
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 31
the total variation in nozzle efficiency 10% 4%
the difference between grid level 2 and grid level 4
6%
(Sp-Al)
3.5%
(Sp-Al)
the relative uncertainty due to the selection of turbulence model
9%
(grid 4)
2%
(grid 2)
the uncertainty due to the error in geometry representation
0.5%
(grid 3, k-)
1.4%
(grid 3, k-)
the uncertainty due to the change in exit boundary location
0.8%
(grid 3, Sp-Al)
1.1%
(grid 2, Sp-Al)
Uncertainty Comparison in Nozzle EfficiencyStrong Shock
Weak ShockMaximum value of
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 32
Conclusions• Based on the results obtained from this study,
• For attached flows without shocks (or with weak shocks), informed users may obtain reasonably accurate results
• They may get large errors for the cases with strong shocks and substantial separation
• Grid convergence is not achieved with grid levels that have moderate mesh sizes (especially for separated flows)
• Difficult to isolate physical modeling uncertainties from numerical errors
AIAA 2002-5531
9th AIAA/ISSMO Symposium on MAO, 09/05/2002, Atlanta, GA 33
Conclusions• Uncertainties from different sources interact,
especially in the simulation of flows with separation
• The magnitudes of numerical errors are influenced by the physical models (turbulence models) used
• Discretization error and turbulence models are dominant sources of uncertainty in nozzle efficiency and they are larger for the strong shock case
• We should asses the contribution of CFD uncertainties to the MDO problems that include the simulation of complex flows