overview of cfd verification and validation

Overview of CFD Verification and Validation 1

Overview of CFD Verification and Validation

Last Updated: Thursday, 17-Jul-2008 08:53:41 EDT

Responsible NASA Official/Curator: John W. Slater

Introduction

This page presents an overview of the process of the verification and validation of computational

fluid dynamics (CFD) simulations. The overall objective is to demonstrate the accuracy of CFD

codes so that they may be used with confidence for aerodynamic simulation and that the results be

considered credible for decision making in design.

One should first understand the distinctions between a code, simulation, and model. The formal

definitions of these terms are defined on the page entitled Glossary of Verification and Validation

Terms. Essentially, one implements a model into a computer code and then uses the code to perform

a CFD simulation which yield values used in the engineering analysis. Verification and validation

examines the errors in the code and simulation results.

Credibility is obtained by demonstrating acceptable levels of uncertainty and error. A discussion of

the uncertainties and errors in CFD simulations is provided on the page entitled Uncertainty and

Error in CFD Simulations. The levels of uncertainties and errors are determined through verification

assessment and validation assessment.

Verification assessment determines if the programming and computational implementation of the

conceptual model is correct. It examines the mathematics in the models through comparison to

exact analytical results. Verification assessment examines for computer programming errors.

Validation assessment determines if the computational simulation agrees with physical reality. It

examines the science in the models through comparison to experimental results.

There is professional disagreement on exact procedures for verification and validation of CFD

simulations. CFD is maturing, but still an emerging technology. CFD is a complex technology

involving strongly coupled non-linear partial differential equations which attempt to

computationally model theoretical and experimental models in a discrete domain of complex

geometric shape. A detailed assessment of errors and uncertainties has to concern itself with the

three roots of CFD: theory, experiment, and computation. Further, the application of CFD is rapidly

expanding with the growth in computational resources. In this work, we primarily follow the

verification and validation guidelines established by the AIAA [AIAA-G-077-1998]. Note that this

is a guide - no standards yet exist for CFD simulation verification and validation. Other ideas from

other researchers in this discipline will also be included. Their papers are referenced in the

bibliography. Notable among them is the book on verification and validation published by Roache.


Verification and validation are on-going activities due to the complex nature of the CFD codes and

expanding range of possible applications. Some basic verfication should be done prior to release of

a code and basic validation studies should be performed on classes of flow features prior to use of

the code for similar flows. However, as the code continues to develop, verification and validation

should continue.

Use of CFD Results

The level of accuracy required from a CFD analysis depends on the desired use of the results. A

conceptual design effort may be content with general shock structure information, whereas a

detailed design may require accurate determination of the pressure recovery. Each quantity to be

determined generally has its own accuracy requirement. Levels of credibility may vary according to

the information required.

The application of CFD for design and analysis may be catagorized into three levels according to

increased levels of required accuracy: 1) provide qualitative information, 2) provide incremental

quantities, and 3) provide absolute quantities. This discussion follows the ideas of Benek et al.

Provide qualitative information. CFD provides details on the entire flow field not possible with

experimental methods. This is useful in understanding on a qualitative level the behavior of the flow

field. Accuracy requirements are low.

Provide incremental quantities. Corrections to experimental observations can be provided by

CFD at a higher accuracy than existing with the CFD method. This is due to cancellation of part of

the error when taking the differences. For example, if there is a design change from a baseline, for

which the quanity is known, Pbaseline, the quantity P for the design change can be expressed as:

P = Pbaseline + dP

where dP is the increment in P corresponding to the design change. If two CFD simulations are

performed, the first with the baseline geometry and the second with the modified geometry, then the

increment in P due to the modified geometry can be estimated as,

dP = ( P + E )2 - ( P + E )1 = dPactual + dE.

The E is the error associated with the quantity P obtained from the CFD simulation. As can be seen,

the error to the increment is dE, which cancels out some of the error.

Provide absolute quantities. This level involves determining absolute values of the quantity P and

requires the highest level of accuracy. The accuracy required is usually stated as part of the design

process. The accuracy observed from the CFD simulation varies according to the character of the

quantity, and so, it is not possible to state an accuracy or error band that applies to all quantities

obtained from the CFD simulation. The verification methods discussed below regarding a grid

convergence study will provide the error band for the calculations.

Flow Characteristics

In applying CFD for flows typical of aerospace systems, we must first understand the characteristics

of the flow. We must understand the reality upon which we will validate the CFD code and

processes.


The flow is characterized primarily by the Mach number. We are interested in analyzing flows

spanning the Mach number range from Mach 0 (static conditions) to 25 (access to space).

The flow is characterized by high Reynolds numbers which result in regions of laminar flow

transitioning to turbulent flow. Flows along the body and inlet surfaces create boundary layers.

Adverse pressure gradient may be present for internal flows. At transonic, supersonic, and

hypersonic speeds, shock waves are present. Under these conditions, the boundary layer may

separate.

At hypersonic Mach numbers, real gas effects may become important. This requires use of

chemistry models for calorically and thermally perfect gases, equilibrium air, and chemically

reaction of gas mixtures.

Often the geometry of the system is complex, which has to be physically modeled.

Unsteady flow may become important.

Physical Models

There are several physical models that are commonly used within CFD codes:

Spatial Dimension. The geometry of the inlet may be modelled in some cases using two-

dimensional or axisymmetric space rather than full three-dimensional.

Temporal Dimension. One may assume steady-state flow or attempt to capture the time variations.

Navier-Stokes Equations. The Navier-Stokes equations govern the continuum flow. Viscous and

heat conduction effects are modelled. If these are removed, then inviscid flow can be used.

Turbulence Models. Various algebraic, one-equation, and two-equation turbulence models exist

with various parameters and freestream boundary conditions. The option of wall functions exists.

Thermodynamic and Transport Properties. Constants and relations for thermodynamic and

transport properties are generally constants, algebraic equations, or curve fits.

Air Chemistry Models. Inlet flows typically involve calorically perfect air adequately described by

the perfect gas equation of state. At higher temperatures (greater than 700K), modeling of thermally

perfect air, equilibrium air, and chemically reacting air (temperatures greater than 2000 K) may be

needed.

Flow Boundary Conditions. These include subsonic and supersonic freestream inflow and

outflow. Also inflow and outflow of plenum chambers.

Bleed / Blowing. These can be treated as boundary conditions as a mass flow or porous boundary.

Another option is to grid the slots and holes of the actual geometry.

Flow Control Devices. Flow control is important and several new technologies have developed.

Vortex generators are the primary flow control devices used in inlets. These can be modeled or an

approximation of the geometry can be gridded.


CFD Analysis Process

The general process for performing a CFD analysis is outlined below so as to provide a reference

for understanding the various aspects of a CFD simulation. The process includes:

1. Forumulate the Flow Problem

2. Model the Geometry and Flow Domain

3. Establish the Boundary and Initial Conditions

4. Generate the Grid

5. Establish the Simulation Strategy

6. Establish the Input Parameters and Files

7. Perform the Simulation

8. Monitor the Simulation for Completion

9. Post-process the Simulation to get the Results

10. Make Comparisons of the Results

11. Repeat the Process to Examine Sensitivities

12. Document

In further detail, these steps include:

1. Formulate the Flow Problem

The first step of the analysis process is to formulate the flow problem by seeking answers to

the following questions:

o what is the objective of the analysis?

o what is the easiest way to obtain those objective?

o what geometry should be included?

o what are the freestream and/or operating conditions?

o what dimensionality of the spatial model is required? (1D, quasi-1D, 2D,

axisymmetric, 3D)

o what should the flow domain look like?

o what temporal modeling is appropriate? (steady or unsteady)

o what is the nature of the viscous flow? (inviscid, laminar, turbulent)

o how should the gas be modeled?

2. Model the Geometry and Flow Domain

The body about which flow is to be analyzed requires modeling. This generally involves

modeling the geometry with a CAD software package. Approximations of the geometry and

simplifications may be required to allow an analysis with reasonable effort. Concurrently,

decisions are made as to the extent of the finite flow domain in which the flow is to be

simulated. Portions of the boundary of the flow domain conicide with the surfaces of the

body geometry. Other surfaces are free boundaries over which flow enters or leaves. The

geometry and flow domain are modeled in such a manner as to provide input for the grid

generation. Thus, the modeling often takes into account the structure and topology of the

grid generation.

3. Establish the Boundary and Initial Conditions


Since a finite flow domain is specified, physical conditions are required on the boundaries of

the flow domain. The simulation generally starts from an initial solution and uses an

iterative method to reach a final flow field solution.

4. Generate the Grid

The flow domain is discretized into a grid. The grid generation involves defining the

structure and topology and then generating a grid on that topology. Currently all cases

involve multi-block, structured grids; however, the grid blocks may be abbuting, contiguous,

non-contiguous, and overlapping. The grid should exhibit some minimal grid quality as

defined by measures of orthogonality (especially at the boundaries), relative grid spacing

(15% to 20% stretching is considered a maximum value), grid skewness, etc... Further the

maximum spacings should be consistent with the desired resolution of important features.

The resolution of boundary layers requires the grid to be clustered in the direction normal to

the surface with the spacing of the first grid point off the wall to be well within the laminar

sublayer of the boundary layer. For turbulent flows, the first point off the wall should exhibit

a y+ value of less than 1.0.

5. Establish the Simulation Strategy

The strategy for performing the simulation involves determining such things as the use of

space-marching or time-marching, the choice of turbulence or chemistry model, and the

choice of algorithms.

6. Establish the Input Parameters and Files

A CFD codes generally requires that an input data file be created listing the values of the

input parameters consisted with the desired strategy. Further the a grid file containing the

grid and boundary condition information is generally required. The files for the grid and

initial flow solution need to be generated.

7. Perform the Simulation

The simulation is performed with various possible with options for interactive or batch

processing and distributed processing.

8. Monitor the Simulation for Completion

As the simulation proceeds, the solution is monitored to determine if a "converged" solution

has been obtained, which is iterative convergence. Further discussion can be found on the

page entitled Examining Iterative Convergence.

9. Post-Process the Simulation to get the Results

Post-Processing involves extracting the desired flow properties (thrust, lift, drag, etc...) from

the computed flowfield.

10. Make Comparisons of the Results


The computed flow properties are then compared to results from analytic, computational, or

experimental studies to establish the validity of the computed results.

11. Repeat the Process to Examine Sensitivities

The sensitivity of the computed results should be examined to understand the possible

differences in the accuracy of results and / or performance of the computation with respect

to such things as:

o dimensionality

o flow conditions

o initial conditions

o marching strategy

o algorithms

o grid topology and density

o turbulence model

o chemistry model

o flux model

o artificial viscosity

o boundary conditions

o computer system

Further information can be found on the pages entitled Verification Assessment and

Validation Assessment.

12. Document

Documenting the findings of an analysis involves describing each of these steps in the

process.

Uncertainty and Error in CFD Simulations

This page provides a classification of uncertainties and errors that cause CFD simulation results to

differ from their true or exact values. This discussion not only applies to the CFD code, but other

computer programs used in the analysis process such as CAD packages, grid generators, and flow

visualizers.

Defining Uncertainty and Error

Uncertainty and Error are commonly used interchangeably in everyday language. Here we follow

the definitions of the AIAA Guidlines:

Uncertainty is defined as:

"A potential deficiency in any phase or activity of the modeling process that is due to the lack of

knowledge." (AIAA G-077-1998)

Error is defined as:


A recoqnizable deficiency in any phase or activity of modeling and simulation that is not due to lack

of knowledge. (AIAA G-077-1998)

The key phrase differentiating the definitions of uncertainty and error is lack of knowledge.

The key word in the definition of uncertainty is potential, which indicates that deficiencies may or

may not exist. Lack of knowledge has primarily to do with lack of knowledge about the physical

processes that go into building the model. Sensitivity and uncertainty analyses can be used to better

determine uncertainty. Uncertainty applies to describing deficiencies in turbulence modeling. There

is a lot about turbulence modeling that is not understood. One approach for determining the level of

uncertainty and it effect on one's analysis is to run a number of simulations with a variety of

turbulence models and see how the modeling affects the results.

The definition for error implies that the deficiency is identifiable upon examination. Errors can also

be classified as acknowledged or unacknowledged:

Acknowledged errors (examples include round-off error and discretization error) have procedures

for identifying them and possibly removing them. Otherwise they can remain in the code with their

error estimated and listed.

Unacknowledged errors (examples include computer programming errors or usage errors) have no

set procedures for finding them and may continue within the code or simulation.

One can differentiate between local and global errors. Local errors refer to errors at a grid point or

cell, whereas global errors refer to errors over the entire flow domain. We are interested here in the

global error of the solution that accounts for the local error at each grid point but is more than just

the sum of the local errors. Local errors are transported, advected, and diffused throughout the grid.

The definition of error presented here is different than that an experimentalist may use, which is

"the difference between the measured value and the exact value". Experimentalist usually define

uncertainty as "the estimate of error". These definitions are inadequate for computational

simulations because the exact value is typically not known. Further these definitions link error with

uncertainty. The defintions provided in the above paragraphs are more definite because they

differentiate error and uncertainty according to what is known.

Classification of Errors

Here we provide a classification or taxonomy of error.

Acknowledged Error

1. Physical approximation error

o Physical modeling error

o Geometry modeling error

2. Computer round-off error

3. Iterative convergence error

4. Discretization error

o Spatial discretization error

o Temporal discretization error

Unacknowledged Error

1. Computer programming error


2. Usage error

Each of these types of errors are discussed below.

Physical Approximation Error

Physical modeling errors are those due to uncertainty in the formulation of the model and deliberate

simplifications of the model. These errors deal with the continuum model only. Converting the

model to discrete form for the code is discussed as part of discretization errors. Errors in the

modeling of the fluids or solids problem are concerned with the choice of the governing equations

which are solved and models for the fluid or solid properties. Further, the issue of providing a well-

posed problem can contribute to modeling errors. Often modeling is required for turbulence

quantities, transistion, and boundary conditions (bleed, time-varying flow, surface roughness).

Mehta lists sources of uncertainty in physical models as 1) the phenomenon is not thoroughly

understood; 2) parameters used in the model are known but with some degree of uncertainty; 3)

appropriate models are simplified, thus introducing uncertainty; and 4) an experimental

confirmation of the models is not possible or is incomplete. Even when a physical process is known

to a high level of accuracy, a simplified model may be used within the CFD code for the

convenience of a more efficient computation. Physical modeling errors are examined by performing

validation studies that focus on certain models (i.e. inviscid flow, turbulent boundary layers, real-

gas flows, etc...).

Computer Round-Off Error

Computer round-off errors develop with the representation of floating point numbers on the

computer and the accuracy at which numbers are stored. With advanced computer resources,

numbers are typically stored with 16, 32, or 64 bits. Round-off errors are not considered significant

when compared with other errors. If computer round-off errors are suspected of being significant,

one test is to run the code at a higher precision or on a computer known to store floating point

numbers at a higher precision. One can attempt to iterate a coarse grid solution to a residual of

machine zero; however, this may not be possible for more complex algorithms.

Iterative Convergence Error

The iterative convergence error exists because the iterative methods used in the simulation must

have a stopping point eventually. The error scales to the variation in the solution at the completion

of the simulations.

Discretization Errors

Discretization errors are those errors that occur from the representation of the governing flow

equations and other physical models as algebraic expressions in a discrete domain of space (finite-

difference, finite-volume, finite-element) and time. The discrete spatial domain is known as the grid

or mesh. The temporal discreteness is manifested through the time step taken. Discretization error is

also known as numerical error. A consistent numerical method will approach the continuum

representation of the equations and zero discretization error as the number of grid points increases

and the size of the grid spacing tends to zero. As the mesh is refined, the solution should become

less sensitive to the grid spacing and approach the continuum solution. This is grid convergence.

Such thinking also applies to the time step. The grid convergence study is a useful procedure for

determining the level of discretization error existing in a CFD solution. "Ordered" discretization

errors are those dependent on the grid size and vanish as the grid size approaches zero. These are


the errors that are addressed by a grid convergence study. Further details can be found on the pages

entitled Examining Spatial (Grid) Convergence and Examining Temporal Convergence.

The discretization error is of most concern to a CFD code user during an application. Discretization

errors are of major concern because they are dependent on the quality of the grid; however, it is

often difficult to precisely indicate the relationship between a quality grid and an accurate solution

prior to beginning the simulation. The level of discretization error is dependent on grid quality. The

grid should be generated with consideration of such things as resolution, density, aspect ratio,

stretching, orthogonality, grid singularities, and zonal boundary interfaces.

The level of discretization error is dependent on the features of the flow as resolved by the grid.

Errors may develop due to representation of discontinuities (shocks, slip surfaces, interfaces, ...) on

a grid. Interpolation errors come about at zonal interfaces where the solution of one zone is

approximated on the boundary of the other zone.

The truncation error is the difference between the partial differential equation (PDE) and the finite

equation. The truncation error is a function of the grid quality and flow gradients. Dispersive error

terms causes oscillations in the solution. One fix to this is adding artificial dissipation to decrease

the size of the dispersive errors. Dissipation error terms cause a smoothing of gradients. However, a

level of dissipation comparable to the actual physical viscosity may contaminate the solution.

Boundary layers may thicken. The truncation error terms are those of the expansion which are not

used in the discretized equation. If the order of the leading term of the truncation error is of second-

order, it is known as a numerical viscosity (dimensions of length2 / time), which is dimensions of

kinematic viscosity. A positive viscous term will indicate that errors will be damped whereas a

negative viscous term will indicate that errors will grow (unstable).

Included in the discretization error are errors due to not properly converging the solution with

respect to the iterations to the steady-state solution or within a time step. This is reffered to as

iterative convergence.

Computer Programming Errors

Programming errors are "bugs" and mistakes made in programming or writing the code. They are

the responsibility of the programmers. These type of errors are discovered by systematically

performing verification studies of subprograms of the code and the entire code, reviewing the lines

of code, and performing validation studies of the code. The programming errors should be removed

from the code prior to release.

Usage Errors

Usage errors are due to the application of the code in a less-than-accurate or improper manner.

Usage errors may actually show up as modeling and discretization errors. The user sets the models,

grid, algorithm, and inputs used in a simulation, which then establishes the accuracy of the

simulation. There may be blatant errors, such as attempting to compute a known turbulent flow with

an assumption of inviscid flow. A converged solution may be obtained; however, the conclusions

drawn from the simulation may be incorrect. The errors may not be as evident, such as proper

choice of turbulence model parameters for separated flows with shocks. The potential for usage

errors increases with an increased level of options available in a CFD code. Usage errors are

minimized through proper training and the accumulation of experience.


The user may intentionally introduce modeling and discretization error as an attempt to expedite the

simulation at the expense of accuracy. This may be proper in the conceptual stage of a design study

where more general information is needed at less accuracy. Even in the later stages, there may not

be proper computational resources to simulate at the proper grid density. One has to understand the

level of accuracy accompanying the results.

Usage errors should be controlable through proper training and analysis.

Usage errors can exist in the CAD, grid generation, and post-processing software, in addition to the

CFD code.

Verification Assessment

This page discusses Verification Assessment, which focuses on the methods for Verification of CFD

codes and simulation results.

Verification

Verification is defined as:

The process of determining that a model implementation accurately represents the developer's

conceptual description of the model and the solution to the model. (AIAA G-077-1998)

Verification assessment examines 1) if the computational models are the correct implementation of

the conceptual models, and 2) if the resulting code can be properly used for an analysis. The

strategy is to identify and quantify the errors in the model implementation and the solution. The two

aspects of verification are the verification of a code and the verification of a calculation. The

objective of verifying a code is error evaluation, that is, finding and removing errors in the code.

The objective of verifying a calculation is error estimation, that is determining the accuracy of a

calculation. Each are discussed below.

Verification has also been described as solving the equations right. It is intended to concern itself

more with mathematics rather than engineering. It is intended to look for errors in the programming

and implementation of the models.

Roache considers two aspects of verification: 1) verification of a code and 2) verification of a

calculation. These two aspects are described below:

Verification of a code involves error evaluation, which is, looking for bugs, incorrect

implementations of conceptual models, errors in inputs, and other errors in the code and usage. This

is typically done by the developers prior to release of the code. First, consistency checks are

performed which examine basic relationships expected in the solutions (i.e. mass conservation).

Then the code is used to simulate a suite of ``highly accurate'' verification cases. These cases

should be analytic or numeric solution to ordinary and partial differential equations. Verification

should not performed with experimental data. A grid refinement study should be conducted to bring

out potential errors. All the options of the code should be examined. This becomes more

complicated as the number of options available within a CFD code increase. Identifying and

quantifying each type of error is important because errors can interact and cancel each other -

leading to erroneous conclusions in the validation process. One potentially useful method of


verification is comparing the results of two codes. However, verification is not a democratic activity

and one should watch for comparing with an inaccurate code. The comparison is strengthened when

the two codes use differing numerical methods. The following paragraphs discuss specific checks

that can be performed as part of a code verification process.

Verification of a calculation involves error estimation, which is determining the accuracy of a

single calculation and putting an error band on the final value. The approach involves peformimg a

grid convergence study and determine the observed order of convergence, error bands, and grid

convergence indices (GCI).

Verification Assessment Process

The process for Verification Assessment of a CFD code and / or simulation can be summarized as:

1. Examine the Computer Programming of the Code.

One of the most basic tasks of verification assessment is the review of the computer programming

or coding to check for and identify computer programming errors or "bugs". This is done by

visually checking the coding and by computationally running subprograms using a test code. This is

aided by complete and clear documentation, both internal and external. This step is to directly

detect computer programming errors.

2. Examine Iterative Convergence.

Verification assessment requires that a simulation demonstrates iterative convergence. Further

details can be page entitled Examining Iterative Convergence.

3. Examine Consistency.

One should check for consistency in the CFD solution. For example, the flow in a duct should

maintain mass conservation through the duct. Further total pressure recovery in an inlet should stay

constant or decrease through the duct.

4. Examine Spatial (Grid) Convergence.

The CFD simulation results should demonstrate spatial convergence. Further details and methods

can be found on the page entitled Examining Spatial (Grid) Convergence.

5. Examine Temporal Convergence.

The CFD simulation results should demonstrate temporal convergence. Further details and methods

can be found on the page entitled Examining Temporal Convergence.

6. Compare CFD Results to Highly Accurate Solutions

The veracity of a code can be examined by comparing the CFD simulation results to highly accurate

solution to the models used within the CFD code. This can include analytical solutions, benchmark

numerical solutions to ordinary differential equations (ODEs), and benchmark numerical solutions

to partial differential equations (PDEs).


Validation Assessment

This page discusses Validation Assessment, which focuses on the methods for the Validation of a

CFD codes for simulation of a certain type of flows.

Validation

Validation is defined as:

The process of determining the degree to which a model is an accurate representation of the real

world from the perspective of the intended uses of the model. (AIAA G-077-1998)

Validation has also been described as "solving the right equations". It is not possible to validate the

entire CFD code. One can only validate the code for a specific range of applications for which there

is experimental data. Thus one validates a model or simulation. Applying the code to flows beyond

the region of validity is termed prediction.

Validation examines if the conceptual models, computational models as implemented into the CFD

code, and computational simulation agree with real world observations. The strategy is to indentify

and quantify error and uncertainty through comparison of simulation results with experimental data.

The experiment data sets themselves will contain bias errors and random errors which must be

properly quantified and documented as part of the data set. The accuracy required in the validation

activities is dependent on the application, and so, the validation should be flexible to allow various

levels of accuracy.

The approach to Validation Assessment is to perform a systematic comparison of CFD simulation

results to experimental data from a set increasingly complex cases.

Each CFD simulation requires verification of the calculation as specified in the discussion of

Verification Assessment.

Validation Assessment Process

The process for Validation Assessment of a CFD simulation can be summarized as:

1. Examine Iterative Convergence.

Validation assessment requires that a simulation demonstrates iterative convergence. Further details

can be page entitled Examining Iterative Convergence.

2. Examine Consistency.

One should check for consistency in the CFD solution. For example, the flow in a duct should

maintain mass conservation through the duct. Further total pressure recovery in an inlet should stay

constant or decrease through the duct.

3. Examine Spatial (Grid) Convergence.

The CFD simulation results should demonstrate spatial convergence. Further details and methods

can be found on the page entitled Examining Spatial (Grid) Convergence.


4. Examine Temporal Convergence.

The CFD simulation results should demonstrate temporal convergence. Further details and methods

can be found on the page entitled Examining Temporal Convergence.

5. Compare CFD Results to Experimental Data.

Experimental data is the observation of the "real world" in some controlled manner. By comparing

the CFD results to experimental data, one hopes that there is a good agreement, which inreases

confidence that the physical models and the code represents the "real world" for this class of

simulations. However, the experimental data contains some level of error. This is usually related to

the complexity of the experiment. Validation assessment calls for a "building block" approach of

experiments which sets a hierarchy of experiment complexity.

6. Examine Model Uncertainties.

The physical models in the CFD code contain uncertainties due to a lack of complete understanding

or knowledge of the physical processes. One of the models with the most uncertainty is the

turbulence models. The uncertainty can be examined by running a number of simulations with the

various turbulence models and examine the affect on the results.

Building-Block Approach for Experiments

A building-block approach is followed in performing the validation assessment for a complex

system such as an aircraft inlet. The approach consists of phases involving successively more

complex flow physics, geometry, and interactions. These phases include:

Unit Problems involve simple geometry, one element of the complex flow physics, and one

relevant flow feature. An example is the measurement of a turbulent boundary layer over a flat

plate. The experiment data set contains detailed data collected with high accuracy. The boundary

conditions and initial conditions are accurately measured.

Benchmark Cases involve fairly simple hardware representing a key feature of the system. The

flow field contains only two separate flow features of the flow physics which are likely coupled. An

example is a shock / boundary layer interaction. The experiment data set is extensive in scope and

uncertainties are low; however, some measurements, such as, initial and boundary conditions, may

not have been collected.

Subsystem Cases involve geometry of a component of the complete system which may have been

simplified. The flow physics of the complete system may be well represented; but the level of

coupling between flow phenomena is typically reduced. An example is a test of a subsonic diffuser

for a supersonic inlet. The exact inflow conditions may not be matched. The quality and quantity of

the experiment data set may not be as extensive as the benchmark cases.

Complete System Cases involve actual hardware and the complete flow physics. All of the

relevant flow features are present. An example is a test of a mixed-compression inlet in the 10x10

wind tunnel at NASA Glenn. Less detailed data is collected since the emphasis is on system

evaluation. Uncertainties on initial and boundary conditions may be large.

Requirements for Experimental Data

The experimental data likely has uncertainties and error associated with it. In comparing the CFD

simulation results to experimental data, one should discuss the experimental errors. Plots comparing


CFD results and experimental data should include a visual display of the error bars on the

experimental data.

Examining Iterative Convergence

Generally, CFD methods involve some iterative scheme to arrive at the simulation results. Here it is

assumed that the iteration is with respect to time or a pseudo-temporal quantity and some type of

time step is taken at each iteration. A steady-state flow simulation involves starting with a uniform

or fabricated flow field and iterating in time until the steady-state flow field is obtained. This is

termed iterative convergence, but requires some criteria for determining convergence. Criteria

include:

Residuals. The residuals of the equations are the change in the equations over an iteration. These

are usually scaled or normalized. One usually looks for the residuals to reach a certain level and

then level-off as an indication of iterative convergence. For a time-marching, steady-state strategy,

this involves examining whether the residual has been reduced a certain number (usually 3-4) of

orders of magnitude.

Results. The CFD simulation has the objective of determining some quantity such as lift, drag,

recovery, etc... One can track the values of such engineering quantities with respect to iteration and

define iterative convergence when these quantities converge. The convergence criteria is usually

defined by acceptable error in these values. It is often the case that certain quantities may reach

convergence at a different rate than other quantities. One can check that a monitored flow value

(such as thrust, drag, or boundary layer profile) has remained unchanged with respect to the number

of iterations.

Time-Accurate Simulations. For a time-marching, time-accurate strategy, this involves examining

whether the final time has been reached with proper convergence at each time step.

Space-Marching Simulation For a space-marching strategy, this involves examining whether the

end of the marching segment has been reached with proper convergence at each marching step.

Examining Solution Consistency

One can evaluate convergence by checking for consistency in the flow field. The conservation

relations require a balance of fluxes through a control surface. For a closed duct, the flow through

the duct should be conserved. Low-speed flow over a closed body should have zero drag. Other

such consistency relations can be defined for specific flow fields. These provide verification of the

code since the consistency relations are usually a statement of some analytic result.


Examining Spatial (Grid) Convergence

Introduction

The examination of the spatial convergence of a simulation is a straight-forward method for

determining the ordered discretization error in a CFD simulation. The method involves performing

the simulation on two or more successively finer grids. The term grid convergence study is

equivalent to the commonly used term grid refinement study.

As the grid is refined (grid cells become smaller and the number of cells in the flow domain

increase) and the time step is refined (reduced) the spatial and temporal discretization errors,

respectively, should asymptotically approaches zero, excluding computer round-off error.

Methods for examining the spatial and temporal convergence of CFD simulations are presented in

the book by Roache. They are based on use of Richardson's extrapolation. A summary of the

method is presented here.

A general discussion of errors in CFD computations is available for background.

We will mostly likely want to determine the error band for the engineering quantities obtained from

the finest grid solution. However, if the CFD simulations are part of a design study that may require

tens or hundreds of simulations, we may want to use one of the coarser grids. Thus we may also

want to be able to determine the error on the coarser grid.

Grid Considerations for a Grid Convergence Study

The easiest approach for generating the series of grids is to generate a grid with what one would

consider fine grid spacing, perhaps reaching the upper limit of one's tolerance for generating a grid

or waiting for the computation on that grid to converge. Then coarser grids can be obtained by

removing every other grid line in each coordinate direction. This can be continued to create

additional levels of coarser grids. In generating the fine grid, one can build in the n levels of coarser

grids by making sure that the number of grid points in each coordinate direction satisfies the

relation

N = 2n m + 1

where m is an integer. For example, if two levels of coarser grids are desired (i.e. fine, medium, and

coarse grids) then the number of grid points in each coordinate direction must equal 4 m + 1. The m

may be different for each coordinate direction.

The WIND code has a grid sequencing control that will solve the solution on the coarser grid

without having to change the grid input file, boundary condition settings, or the input data file.

Further, the converged solution on the coarser grid then can be used directly as the initial solution

on the finer grid. This option was originally used to speed up convergence of solutions; however,

can be used effectively for a grid convergence study.

It is not necessary to halve the number of grid points in each coordinate direction to obtain the

coarser grid. Non-integer grid refinement or coarsening can be used. This may be desired since

halfing a grid may put the solution out of the asymptotic range. Non-integer grid refinement or

coarsening will require the generation of a new grid. It is important to maintain the same grid


generation parameters as the original grid. One approach is to select several grid spacings as

reference grid spacings. One should be the grid spacing normal to the walls. Others may be spacings

at flow boundaries, at junctures in the geometry, or at zonal interfaces. Upon picking the ratio as

which the grid is to be refined or coarsened, this same ratio is applied to these spacings. The number

of grid points are then adjusted according to grid quality parameters, such as grid spacing ratio

limits. The surface and volume grids are then generated using the same methods as the original grid.

The grid refinement ratio should be a minimum of r >= 1.1 to allow the discretization error to be

differentiated from other error sources (iterative convergence errors, computer round-off, etc...).

Order of Grid Convergence

The order of grid convergence involves the behavior of the solution error defined as the difference

between the discrete solution and the exact solution,

where C is a constant, h is some measure of grid spacing, and p is the order of convergence. A

"second-order" solution would have p = 2.

A CFD code uses a numerical algorithm that will provide a theoretical order of convergence;

however, the boundary conditions, numerical models, and grid will reduce this order so that the

observed order of convergence will likely be lower.

Neglecting higher-order terms and taking the logarithm of both sides of the above equation results

in:

The order of convergence p can be obtained from the slope of the curve of log(E) versus log(h). If

such data points are available, the slope can be read from the graph or the slope can be computed

from a least-squares fit of the data. The least-squares will likely be inaccurate if there are only a few

data points.

A more direct evaluation of p can be obtained from three solutions using a constant grid refinement

ratio r,

The order of accuracy is determined by the order of the leading term of the truncation error and is

represented with respect to the scale of the discretization, h. The local order of accuracy is the

order for the stencil representing the discretization of the equation at one location in the grid. The

global order of accuracy considers the propagation and accumulation of errors outside the stencil.

This propagation causes the global order of accuracy to be, in general, one degree less than the local

order of accuracy. The order of accuracy of the boundary conditions can be one order of accuracy

lower than the interior order of accuracy without degrading the overall global accuracy.

Asymptotic Range of Convergence

Assessing the accuracy of code and caluculations requires that the grid is sufficiently refined such

that the solution is in the asymptotic range of convergence. The asymptotic range of convergence is


obtained when the grid spacing is such that the various grid spacings h and errors E result in the

constancy of C,

C = E / hp

Another check of the asymptotic range will be discussed in the section on the grid convergence

index.

Richardson Extrapolation

Richardson extrapolation is a method for obtaining a higher-order estimate of the continuum value

(value at zero grid spacing) from a series of lower-order discrete values.

A simulation will yield a quantity f that can be expressed in a general form by the series expansion:

f = fh=0 + g1 h + g2 h2 + g3 h

3 + ...

where h is the grid spacing and the functions g1, g2, and g3 are independent of the grid spacing. The

quantity f is considered "second-order" if g1 = 0.0. The fh=0 is the continuum value at zero grid

spacing.

If one assumes a second-order solution and has computed f on two grids of spacing h1 and h2 with

h1 being the finer (smaller) spacing, then one can write two equations for the above expansion,

neglect third-order and higher terms, and solve for fh=0 to estimate the continuum value,

where the grid refinement ratio is:

r = h2 / h1

The Richardson extrapolation can be generalized for a p-th order methods and r-value of grid ratio

(which does not have to be an integer) as:

Traditionally, Richardson extrapolation has been used with grid refinement ratios of r = 2. Thus, the

above equation simplifies to:

In theory, the above equations for the Richardson extrapolation will provide a fourth-order estimate

of fh=0, if f1 and f2 were computed using exactly second-order methods. Otherwise, it will be a third-

order estimate. In general, we will consider fh=0 to be p+1 order accurate. Richardson extrapolation

can be applied for the solution at each grid point, or to solution functionals, such as pressure

recovery or drag. This assumes that the solution is globally second-order in addition to locally

second-order and that the solution functionals were computed using consistent second-order

methods. Other cautions with using Richardson extrapolation (non-conservative, amplification of

round-off error, etc...) are discussed in the book by Roache.


For our purposes we will assume f is a solution functional (i.e. pressure recovery). The fh=0 is then

as an estimate of f in the limit as the grid spacing goes to zero. One use of fh=0 is to report the value

as the an improved estimate of f1 from the CFD study; however, one has to understand the caveats

mentioned above that go along with that value.

The other use of fh=0 is to obtain an estimate of the discretization error that bands f obtained from

the CFD. This use will now be examined.

The difference between f1 and fh=0 is one error estimator; however, this requires consideration of the

caveats attached to fh=0.

We will focus on using f1 and f2 to obtain an error estimate. Examining the generalized Richardson

extrapolation equation above, the second term on the right-hand side can be considered to be an an

error estimator of f1. The equation can be expressed as:

A1 = E1 + O( hp+1

, E12)

where A1 is the actual fractional error defined as:

A1 = ( f1 - fh=0 ) / fh=0

and E1 is the estimated fractional error for f1 defined as:

where the relative error is defined as:

This quantity should not be used as an error estimator since it does not take into account r or p. This

may lead to an underestimation or overestimation of the error. One could make this quantity

artificially small by simply using a grid refinement ratio r close to 1.0.

The estimated fractional error E1 is an ordered error estimator and a good approximation of the

discretization error on the fine grid if f1 and f2 were obtained with good accuracy (i.e. E1<1). The

value of E1 may be meaningless if f1 and fh=0 are zero or very small relative to f2-f1. If such is the

case, then another normalizing value should be used in place of f1.

If a large number of CFD computations are to be performed (i.e for a DOE study), one may wish to

use the coarser grid with h2. We will then want to estimate the error on the coarser grid. The

Richardson extrapolation can be expressed as:

The estimated fractional error for f2 is defined as:


Richardson extrapolation is based on a Taylor series representation as indicated in Eqn.

\ref{eq:series}. If there are shocks and other discontinuities present, the Richardson extrapolation is

invalid in the region of the discontinuity. It is still felt that it applies to solution functionals

computed from the entire flow field.

Grid Convergence Index (GCI)

Roache suggests a grid convergence index GCI to provide a consistent manner in reporting the

results of grid convergence studies and perhaps provide an error band on the grid convergence of

the solution. The GCI can be computed using two levels of grid; however, three levels are

recommended in order to accurately estimate the order of convergence and to check that the

solutions are within the asymptotic range of convergence.

A consistent numerical analysis will provide a result which approaches the actual result as the grid

resolution approaches zero. Thus, the discretized equations will approach the solution of the actual

equations. One significant issue in numerical computations is what level of grid resolution is

appropriate. This is a function of the flow conditions, type of analysis, geometry, and other

variables. One is often left to start with a grid resolution and then conduct a series of grid

refinements to assess the effect of grid resolution. This is known as a grid refinement study.

One must recognize the distinction between a numerical result which approaches an asymptotic

numerical value and one which approaches the true solution. It is hoped that as the grid is refined

and resolution improves that the computed solution will not change much and approach an

asymptotic value (i.e. the true numerical solution). There still may be error between this asymptotic

value and the true physical solution to the equations.

Roache has provided a methodology for the uniform reporting of grid refinement studies. "The

basic idea is to approximately relate the results from any grid refinement test to the expected results

from a grid doubling using a second-order method. The GCI is based upon a grid refinement error

estimator derived from the theory of generalized Richardson Extrapolation. It is recommended for

use whether or not Richardson Extrapolation is actually used to improve the accuracy, and in some

cases even if the conditions for the theory do not strictly hold." The object is to provide a measure

of uncertainty of the grid convergence.

The GCI is a measure of the percentage the computed value is away from the value of the

asymptotic numerical value. It indicates an error band on how far the solution is from the

asymptotic value. It indicates how much the solution would change with a further refinement of the

grid. A small value of GCI indicates that the computation is within the asymptotic range.

The GCI on the fine grid is defined as:

where Fs is a factor of safety. The refinement may be spatial or in time. The factor of safety is

recommended to be Fs=3.0 for comparisons of two grids and Fs=1.25 for comparisons over three or

more grids. The higher factor of safety is recommended for reporting purposes and is quite

conservative of the actual errors.

When a design or analysis activity will involve many CFD simulations (i.e. DOE study), one may

want to use the coarser grid h2. It is then necessary to quantify the error for the coarser grid. The

GCI for the coarser grid is defined as:


It is important that each grid level yield solutions that are in the asymptotic range of convergence

for the computed solution. This can be checked by observing two GCI values as computed over

three grids,

GCI23 = rp GCI12

Required Grid Resolution

If a desired accuracy level is known and results from the grid resolution study are available, one can

then estimate the grid resolution required to obtain the desired level of accuracy,

Independent Coordinate Refinement and Mixed Order Methods

The grid refinement ratio assumes that the refinement ratio r applies equally in all coordinate

directions (i,j,k) for steady-state solutions and also time t for time-dependent solutions. If this is not

the case, then the grid convergence indices can be computed for each direction independently and

then added to give the overall grid convergence index,

Effective Grid Refinement Ratio

If one generates a finer or coarser grid and is unsure of the value of grid refinement ratio to use, one

can compute an effective grid refinement ratio as:

where N is the total number of grid points used for the grid and D is the dimension of the flow

domain. This effective grid refinement ratio can also be used for unstructured grids.

Example Grid Convergence Study

The following example demonstrates the application of the above procedures in conducting a grid

convergence study. The objective of the CFD analysis was to determine the pressure recovery for an

inlet. The flow field is computed on three grids, each with twice the number of grid points in the i

and j coordinate directions as the previous grid. The number of grid points in the k direction

remains the same. Since the flow is axisymmetric in the k direction, we consider the finer grid to be

double the next coarser grid. The table below indicates the grid information and the resulting

pressure recovery computed from the solutions. Each solution was properly converged with respect

to iterations. The column indicated by "spacing" is the spacing normalized by the spacing of the

finest grid.


Grid Normalized Grid Spacing Pressure Recovery, Pr

1 1 0.97050

2 2 0.96854

3 4 0.96178

The figure below shows the plot of pressure recoveries with varying grid spacings. As the grid

spacing reduces, the pressure recoveries approach an asymptotic zero-grid spacing value.

We determine the order of convergence observed from these results,

p = ln[ ( 0.96178 - 0.96854 ) / ( 0.96854 - 0.97050 ) ] / ln(2) = 1.786170

The theoretical order of convergence is p=2.0. The difference is most likely due to grid stretching,

grid quality, non-linearities in the solution, presence of shocks, turbulence modeling, and perhaps

other factors.

We now can apply Richardson extrapolation using the two finest grids to obtain an estimate of the

value of the pressure recovery at zero grid spacing,

Prh=0 = 0.97050 + ( 0.97050 - 0.96854 ) / ( 21.786170

- 1 )

= 0.97050 + 0.00080 = 0.97130

This value is also plotted on the figure below.

The grid convergence index for the fine grid solution can now be computed. A factor of safety of

FS=1.25 is used since three grids were used to estimage p. The GCI for grids 1 and 2 is:

GCI12 = 1.25 | ( 0.97050 - 0.96854 ) / 0.97050 | / ( 21.786170

- 1 ) 100% = 0.103083%

The GCI for grids 2 and 3 is:

GCI23 = 1.25 | ( 0.96854 - 0.96178 ) / 0.96854 | / ( 21.786170

- 1 ) 100% = 0.356249%

We can now check that the solutions were in the asymptotic range of convergence,

0.356249 / ( 21.786170

0.103083 ) = 1.002019

which is approximately one and indicates that the solutions are well within the asymptotic range of

convergence.

Based on this study we could say that the pressure recovery for the supersonic ramp is estimated to

be Pr = 0.97130 with an error band of 0.103% or 0.001.


VERIFY: A Fortran program to Perform Calculations Associated with a Grid

Convergence Study

The Fortran 90 program verify.f90 was written to carry out the calculations associated with a grid

convergence study involving 3 or more grids. The program is compiled on a unix system through

the commands:

f90 verify.f90 -o verify

It reads in an ASCII file (prD.do) through the standard input unit (5) that contains a list of pairs of

grid size and value of the observed quantity f.

verify < prD.do > prD.out

It assumes the values from the finest grid are listed first. The output is then written to the standard

output unit (6) prD.out. The output from the of {\tt verify} for the results of Appendix A are:

--- VERIFY: Performs verification calculations ---

Number of data sets read = 3

Grid Size Quantity

1.000000 0.970500

2.000000 0.968540

4.000000 0.961780

Order of convergence using first three finest grid

and assuming constant grid refinement (Eqn. 5.10.6.1)

Order of Convergence, p = 1.78618479

Richardson Extrapolation: Use above order of convergence


and first and second finest grids (Eqn. 5.4.1)

Estimate to zero grid value, f_exact = 0.971300304

Grid Convergence Index on fine grids. Uses p from above.

Factor of Safety = 1.25

Grid Refinement

Step Ratio, r GCI(%)

1 2 2.000000 0.103080

2 3 2.000000 0.356244

Checking for asymptotic range using Eqn. 5.10.5.2.

A ratio of 1.0 indicates asymptotic range.

Grid Range Ratio

12 23 0.997980

--- End of VERIFY ---

Examples of Grid Converence Studies in the Archive

A grid convergence study is performed in the Supersonic Wedge case.

Examples of Grid Converence Studies in Literature

Other examples of grid convergence studies that use the procedures outlined above can be found in

the book by Roache and the paper by Steffen et al..

NPARC Alliance Policy with Respect to Grid Converence Studies

For the WIND verification and validation effort, it is suggested that the above procedures be used

when conducting and reporting results from a grid convergence study.

Examining Temporal Convergence

Time-accurate simulations involve taking discrete time steps. One must examine the sensitivity of

the simulation results to the magnitude of the time step. The effects and possible errors are usually

related to the time filtering of various time scales existing in the unsteady flow field.

Glossary of Verification and Validation Terms

The following is a glossary for verification and validation. The objective is to clarify meanings and

indicate how some of the terms have specific definitions that may be different than common usage

of those terms. For example, verification and validation share the same meaning in most

dictionaries and even in common usage among technical personnel; however, here we are

concerned with attaching to them specific technical definitions that are associated with different

aspects of CFD. Most of the definitions come directly from AIAA G-077-1998, "Guide for the

Verification and Validation of Computational Fluid Dynamics Simulations". Other definitions and

discussions come from other literature cited in the bibliography. The terms are listed alphabetically.


Calibration

Calibration in the context of CFD simulations is defined as

The process of adjusting numerical or physical modeling parameters in the computational model

for the purpose of improving agreement with experimental data. (AIAA G-077-1998)

Calibration has more to do with improving agreement of computational results with experimental

results rather than assessing error and uncertainty. Calibration arises from uncertainty in modeling

complex physical processes. It adjusts unmeasured or poorly characterized experimental

parameters. One use of calibration may be the adjustment of emperical constants found within a

turbulence model. While calibration improves agreement for a class of problems, it may make the

code less general.

Certification.

Certification of a code encompases verification and validation, but also includes such things as

documentation, quality assurance, and version control. Certification is concerned with managerial

and programmatic concerns on the use of the CFD code for design activities. Certification is

intended to put a "seal" of credibility on a production code for use in a design project. Certification

can mean that a certain standard has been satisfied (i.e. ISO, NASA). NASA Glenn has a center

level procedure LeR-P2.10.2 entitled "Software Product Assurance" which offers some high level

guidance on verification and validation of software.

Code.

A code is a set of computer instructions and data inputs and definitions. This is the CFD code

(WIND, NPARC, OVERFLOW, USM3D, ...) which may include ancillary codes and

documentation as part of the software package. A code typically has three stages related to the level

of validation completed: research, pilot, and production. The production code has been fully

validated for the intended design applications, including a system-level validation (full inlet, inlet-

aircraft integration).

Credibility.

An improvement of credibility is considered to be the same as confidence building or providing

quality to the customer.

Error.

Error is defined as

A recoqnizable deficiency in any phase or activity of modeling and simulation that is not due to lack

of knowledge. (AIAA G-077-1998)

Further discussion of error can be found at the page entitled Uncertainty and Error in CFD

Simulations.

Grid Convergence.

Grid convergence indicates that as the grid spacing is reduced, the computed simulation results

approach the continum result. Here "grid spacing" can refer to both spatial spacing, as well as, time


step for the case of unsteady, time-accurate simulations. Further discussion can be found on the

page entitled Examining Spatial (Grid) Convergence.

Iterative Convergence.

Iterative convergence indicates that as the discrete equations are iterated, the computed simulation

results approach a fixed value. Further discussion can be found on the page entitled Examining

Iterative Convergence.

Model.

A model is defined as:

A representation of a physical system or process intended to enhance our ability to understand,

predict, or control its behavior. (AIAA G-077-1998)

A conceptual model for CFD consists of the observations, mathematical modeling data, and

mathematical (partial differential) equations that describe the physical system. It also includes

initial and boundary conditions.

The computational model is the computer program or code that implements the conceptual model.

This may be finite-difference, finite-volume, finite-element, or other type of discretization. It

includes the algorithms and iterative strategies. Parameters for the computational model include the

number of grid points, algorithm inputs, and similar parameters.

Modeling.

Modeling is defined as

The process of construction or modification of a model. (AIAA G-077-1998)

Prediction

. Prediction is defined as

Use of a CFD model to foretell the state of a physical system under conditions for which the CFD

model has not been validated. (AIAA G-077-1998)

Prediction is going beyond the validation database and performing simulations of untested systems.

This of course, is a tremendous power of CFD and of use in design studies. However, we would like

to be able to estimate the accuracy of the predictions. Unfortunately, the verification and validation

processes can not formally provide these estimates. The best one can do is point to the verification

results and the most similar validation case.

Robustness

. Robustness defines the ability of the numerical method to provide a solution despite variabilities in

the initial solution and control parameters. This incorporates issues of fault tolerance. Generally,

robustness is achieved at the expense of accuracy.

Simulation.

A simulation is defined as

The exercise or use of a model. (That is, a model is used in a simulation). (AIAA G-077-1998)


For a CFD analysis the application or run of the CFD code is a simulation.

Uncertainty

. Uncertainty is defined as

"A potential deficiency in any phase or activity of the modeling process that is due to the lack of

knowledge." (AIAA G-077-1998)

Further discussion of uncertainty can be found at the page entitled Uncertainty and Error in CFD

Simulations.

Validation

. Validation is defined as

The process of determining the degree to which a model is an accurate representation of the real

world from the perspective of the intended uses of the model. (AIAA G-077-1998)

Further discussion of validation can be found at the page entitled Validation Assessment.

Verification.

Verification is defined as:

The process of determining that a model implementation accurately represents the developer's

conceptual description of the model and the solution to the model. (AIAA G-077-1998)

Further discussion of verification can be found at the page entitled Verification Assessment.


Bibliography of CFD Verification and Validation

The following is a list of references related to CFD Verification and Validation of CFD codes. The

references are grouped according to whether they are:

• Policy Statements

• Books

• Compilations and Proceedings

• Individual Papers

If anyone has any additional references they would like to have listed, please contact the

Webmaster.

Policy Statements

Roache, P.J., K. Ghia, and F. White, "Editorial Policy Statement on the Control of Numerical

Accuracy," ASME Journal of Fluids Engineering, Vol. 108,No. 1., March 1986, p. 2.

AIAA, "Editorial Policy Statement on Numerical Accuracy and Experimental Uncertainty," AIAA

Journal, Vol. 32, January 1994, p. 3.

AIAA, "Guide for the Verification and Validation of Computational Fluid Dynamics Simulations,"

AIAA G-077-1998, 1998.

ASME Editorial Board, "Journal of Heat Transfer Editorial Policy Statement on Numerical

Accuracy," ASME Journal of Heat Transfer, Vol. 116, November 1994. pp. 797-798.

Books

Anderson, D.A., Tannehill, J.C., and Pletcher, R.H. , Computational Fluid Mechanics and Heat

Transfer, McGraw-Hill Book Company, New York, 1984.

Briggs, W.L., A Multigrid Tutorial, SIAM, Philidelphia, PA, 1987.

ERCOFTAC, Best Practices Guidelines for Industrial Computational Fluid Dynamics, Version 1.0,

January 2000.

Hirsch, C. Numerical Computation of Internal and External Flows, Volume I: Fundamentals of

Numerical Discretization. New York: John Wiley & Sons, 1988.

Hirsch, C. Numerical Computation of Internal and External Flows, Volume II: Computational

Methods for Inviscid and Viscous Flows. New York: John Wiley & Sons, 1990.

Roache, P.J., Verification and Validation in Computational Science and Engineering, Hermosa

Publishers, Albuquerque, New Mexico, 1998.


Roache, P.J., Fundamentals of Computational Fluid Dynamics, Hermosa Publishers, Albuquerque,

New Mexico, 1998.

Shyy, W., Computational Modeling for Fluid Flow and Interfacial Transport, New York: Elsevier,

1994.

Compilations and Proceedings

AGARD, Validation of Computational Fluid Dynamics, Lisbon, Portugal, May 2-5, 1988, NATO

Advisory Group for Aeronautical Research and Development, AGARD CP 437, December 1988.

AIAA, AIAA Journal of Spacecraft and Rockets, Vol. 27, No. 2, March-April 1990, pp. 97-215.

This issue contains 5 papers on CFD Code Validation / Verification / Certification with the

emphasis on hypersonic flight. Another section entitled CFD Code Applications has 8 papers on

applying CFD codes for hypersonic flight and includes some discussion on validation. Some of

these 13 papers are listed individually below.

AIAA, AIAA Journal, Vol. 36, No. 5, May 1998, pp. 665-764. This issue contains 12 papers in a

special section entitled Credible Computational Fluid Dynamics Simulations. Some of these 12

papers are listed individually below.

AGARD, Experimental Data Base for Computer Program Assessment, Report of the Fluid

Dynamics Panel Working Group 04, AGARD-AR-138, May 1979. This is the report that contains

the papers and data for the RAE2822 airfoil and ONERA M6 wing.

Individual Papers

Aeschliman, D.P., W.L. Oberkampf, and F.G. Blottner, "A Proposed Methodology for

Computational Fluid Dynamics Code Verification, Calibration, and Validation," Paper presented at

the 16th International Congress on Instrumentation in Aerospace Simulation Facilities (ICIASF),

July 18-21, 1995, Wright-Patterson AFB, OH 45433.

Aeschliman, D.P. and W.L. Oberkampf, "Experimental Methodology for Computational Fluid

Dynamics Code Validation," AIAA Journal, Vol. 36, No. 5, pp. 733-741.

Barber, T.J., "Role of Code Validation and Certification in the Design Environment," AIAA

Journal, Vol. 36, No. 5, pp. 752-758.

Bardina, J.E., P.G. Huang, and T.J. Coakley, "Turbulence Modeling Validation, Testing, and

Development," NASA TM 110446, April 1997.

Benek, J.A., E.M. Kraft, and R.F. Lauer, "Validation Issues for Engine - Airframe Integration,"

AIAA Journal, Vol. 36, No. 5, pp. 759-764.

Blottner, F.G., "Accurate Navier-Stokes Results for the Hypersonic Flow over a Spherical Nosetip,"

AIAA Journal of Spacecraft and Rockets, Vol. 27, No. 2, pp. 113-122.


Bobbitt, P.J., "The Pros and Cons of Code Validation," AIAA Paper 88-2535 (NASA TM 100657),

July 1988.

Coleman, H.W. and F. Stern, "Uncertainties and CFD Validation," ASME Journal of Fluids

Engineering, Vol. 119, December 1997, pp. 795-803.

Cosner, R.R., "Issues in Aerospace Application of CFD Analysis," AIAA Paper 94-0464, January

1994.

Cosner, R.R., "CFD Validation Requirements for Technology Transition," AIAA Paper 95-2227,

June 1995.

Dolling, D.S, "High-Speed Turbulent Separated Flows: Consistency of Mathematical Models and

Flow Physics,"

AIAA Journal, Vol. 36, No. 5, pp. 725-732.

Dudek, J.C., N.J. Georgiadis, and D.A. Yoder, "Calculation of Turbulent Subsonic Diffuser Flows

Using the NPARC Navier-Stokes Code," AIAA Paper 96-0497, January 1996.

Dudek, J.C., "NPARC Validation - Subsonic Turbulent Diffusing Pipe Flow," The NPARC

Alliance, April 1996.

Dudek, J.C., "Testing Guidelines for NPARC Alliance Software Development," The NPARC

Alliance, April 1997.

Dudek, J.C., D.O. Davis, and J.W. Slater, "Validation and Verificaiton of the WIND Code for

Supersonic Diffuser Flows," AIAA Paper 2001-0224, January 2001.

Gnoffo, P.A., "CFD Validation Studies for Hypersonic Flow Prediction," AIAA Paper 2001-1025,

January 2001.

Habashi, W.G., J. Dompierre, Y. Bourgault, M. Fortin, and M.-G. Vallet, "Certifiable

Computational Fluid Dynamics Through Mesh Optimization," AIAA Journal, Vol. 36, No. 5, pp.

703-711.

Jameson, A. and L. Martinelli, "Mesh Refinement and Modeling Errors in Flow Simulations,"

AIAA Journal, Vol. 36, No. 5, May 1998, pp. 676-686.

Lewis, C.H., "Comments on the Need for CFD Code Validation," AIAA Journal of Spacecraft and

Rockets, Vol. 27, No. 2, pp. 97.

Marvin, J.G., "Perspective on Computational Fluid Dynamics Validation," AIAA Journal, Vol. 33,

No. 10, October 1995, pp. 1778-1787.

Mehta, U.B., "Computational Requirements for Hypersonic Flight Performance Estimates," AIAA

Journal of Spacecraft and Rockets, Vol. 27, No. 2, pp. 103-112.

Mehta, U.B., "Some Aspects of Uncertainty in Computational Fluid Dynamics Results,"

Transactions of the ASME, Vol. 113, December 1991, pp. 538-543.


Mehta, U.B., "Guide to Credible Computer Simulations of Fluid Flows," AIAA Journal of

Propulsion and Power, Vol. 12, No. 5, September-October 1996, pp. 940-948. (Also AIAA Paper

95-2225).

Mehta, U.B., "Credible Computational Fluid Dynamics Simulations," AIAA Journal, Vol. 36, No.

5, May 1998, pp. 665-667.

NPARC Alliance, "NPARC Alliance Policies and Plans", August, 1999.

Oberkampf, W.L., "A Proposed Framework for Computational Fluid Dynamics Code Calibration /

Validation," AIAA Paper 94-2540, June 1994.

Oberkampf, W.L. and F.G. Blottner, "Issues in Computational Fluid Dynamics Code Verification

and Validation," AIAA Journal, Vol. 36, No. 5, May 1998, pp. 687-695.

Oberkampf, W.L. and T.G. Trucano, "Validation Methodology in Computational Fluid Dynamics"

AIAA Paper 2000-2549, June 2000.

Paynter, G.C. and E. Tjonneland, "Accuracy Issues in the Prediction of Supersonic Inlet Flows,"

ASME 92-GT-400.

Reed, H.L., T.S. Haynes, and W.S. Saric, "Computational Fluid Dynamics Validation Issues in

Transition Modeling," AIAA Journal, Vol. 36, No. 5, pp. 742-751.

Rizzi, A. and J. Vos, "Toward Establishing Credibility in Computational Fluid Dynamic

Simulations," AIAA Journal, Vol. 36, No. 5, May 1998, pp. 668-675.

Roache, P.J., "Need for Control of Numerical Accuracy," AIAA Journal of Spacecraft and Rockets,

Vol. 27, No. 2, pp. 98-102.

Roache, P.J., "Perspective: A Method for Uniform Reporting of Grid Refinement Studies", ASME

Journal of Fluids Engineering, Vol. 116, September 1994.

Roache, P.J., "Quantification of Uncertainty in Computational Fluid Dynamics," Annual Review of

Fluid Mechanics, Vol. 29, 1997, pp. 123-160.

Roache, P.J., "Verification of Codes and Calculations," AIAA Journal, Vol. 36, No. 5, May 1998,

pp. 696-702.

Slater, J.W., J.C. Dudek, and K.E. Tatum, "The NPARC Verification and Validation Archive",

ASME Paper 2000-FED-11233, June 2000.

Slater, J.W., "Verification Assessment of Flow Boundary Conditions for CFD Analysis of

Supersonic Inlet Flows", AIAA Paper 2001-3882, July 2001.

Slater, J.W., J.M. Abbott, and R.H. Cavicchi, "Validation of WIND for a Series of Inlet Flows",

AIAA Paper 2002-0669, January 2002.

Steffen, C.J. Jr., Reddy, D.R., and K.B.M.Q. Zaman, "Analysis of Flowfield from a Rectangular

Nozzle with Delta Tabs", AIAA 95-2146, June, 1995.


Tatum, K.E. and J.W. Slater, "The Validation Archive of the NPARC Alliance", AIAA Paper 99-

0747, January, 1999.

Towne, C.E. and R.R. Jones, "Results and Current Status of the NPARC Alliance Validation

Effort", AIAA Paper 96-0387, January, 1996.

van Wie, D.M. and T. Rice, "Quantification of Data Uncertainties and Validation of CFD Results

in the Development of Hypersonic Airbreathing Engines," AIAA Paper 96-2028, June 1996.

Yee, H.C. and P.K. Sweby, "Aspects of Numerical Uncertainties in Time Marching to Steady-State

Numerical Solutions," AIAA Journal, Vol. 36, No. 5, pp. 712-724.

Links to Web Sites for CFD Verification and Validation

This is a list of web sites that have information related to CFD Verification and Validation:

NPARC Alliance Verification and Validation Archive

CFD Resources at NASA Glenn

CFD Resources at NASA Langley

MADIC/NASA Code Certification Project

CFD Online

ERCOFTAC Databases

UIUC Airfoil Data Site

NASA NAS Data Sets

FLOWnet Test-Cases Database

overview of cfd verification and validation

Documents