verification and validation in computational geomechanicssokocalo.engr.ucdavis.edu/~jeremic/ ·...

Before We Start Why Do You Trust a Simulation? Verification and Validation Summary

Verification and Validation in ComputationalGeomechanics

Boris JeremicMahdi Taiebat (UBC), Nima Tafazzoli, Panagiota

Tasiopoulou

Department of Civil and Environmental EngineeringUniversity of California, Davis

GheoMatMasseria Salamina

Italy, June 2009

Jeremic Computational Geomechanics Group

Verification and Validation in Computational Geomechanics


Outline

Before We Start

Why Do You Trust a Simulation?Verification, ValidationPrediction

Verification and ValidationThe T ExperimentsSelect V and V Examples

Summary




Before We Start

Important SourcesI W. L. OBERKAMPF, T. G. TRUCANO, AND C. HIRSCH.

Verification, validation and predictive capability incomputational engineering and physics. In Proceedings ofthe Foundations for Verification and Validation on the 21stCentury Workshop, pages 1–74, Laurel, Maryland,October 22-23 2002. Johns Hopkins University / AppliedPhysics Laboratory.

I P. J. ROACHE. Verification and Validation in ComputationalScience and Engineering. Hermosa publishers, 1998.ISBN 0-913478-08-3.

I Material from Verification and Validation in ComputationalMechanics web site http://www.usacm.org/vnvcsm/at the USACM.




Before We Start

Motivation

I How much can (should) we trust model implementations(verification)?

I How much can (should) we trust numerical simulations(validation)?

I How good are our numerical predictions?

I Can simulation tools (SimTool) be used for assessingpublic safety?




Before We Start

Verification

I Verification: the process of determining that a modelimplementation accurately represents the developer’sconceptual description and specification. Mathematicsissue. Verification provides evidence that the model issolved correctly.




Before We Start

Validation

I Validation: The process of determining the degree to whicha model is accurate representation of the real world fromthe perspective of the intended uses of the model. Physicsissue. Validation provides evidence that the correct modelis solved.




Before We Start

Prediction

I Prediction: use of computational model to foretell the stateof a physical system under consideration under conditionsfor which the computational model has not been validated




Before We Start

Role of Verification and Validation

MathematicalModel

ComputerImplementationDiscrete Mathematics

Continuum Mathematics

Programming

Analysis

Code Verification

SimulationComputer

ValidationModel

Reality Model Discovery

and Building




Verification, Validation

Outline

Before We Start



Summary





Importance of V & V

I V & V procedures are the primary means of assessingaccuracy in modeling and computational simulations

I V & V procedures are the tools with which we buildconfidence and credibility in modeling and computationalsimulations





Maturity of Computational SimulationsNational Research Council committee (1986) identified fivestages of maturity in computational fluid dynamics

I Stage 1: Developing enabling technologies (scientificpapers published)

I Stage 2: Demonstration of and Confidence in technologiesand tools (capabilities and limitations of technologyunderstood)

I Stage 3: Compilation of technologies and tools(capabilities and limitations of technology understood)

I Stage 4: Spreading of the effective use (changes theengineering process, value exceeds expectations)

I Stage 5: Mature capabilities (fully dependable, costeffective design applications)





Alternative V & V DefinitionsIEEE V & V definitions (1984):

I Verification: The process of determining whether theproducts of a given phase of the software developmentcycle fulfill the requirements established during theprevious phase

I Validation: The process of evaluating software at the endof the software development process to ensure compliancewith software requirements.

I Other organization have similar definitions:I Software quality assurance communityI American Nuclear Society (safety analysis of commercial

nuclear reactors)I International Organization for Standardization (ISO)





Certification and AccreditationI Certification: A written guarantee that a system or

component complies with its specified requirements and isacceptable for operational use (IEEE (1990)).

I Written guarantee can be issued by anyone (codedeveloper, code user, independent code evaluator)

I Code certification is more formal than validationdocumentation

I Accreditation: The official certification that a model orsimulation is acceptable for use for a specific purpose(DOD/DMSO (1994))

I Only officially designated entities can accreditI Normally appointed by the customers/users of the code or

legal authorityI Appropriate for major liability or public safety applications





Independence of Computational ConfidenceAssessment

1. V&V conducted by the computational tool developer, NoIndependence

2. V&V conducted by a user from same organization

3. V&V conducted by a computational tool evaluatorcontracted by developer’s organization

4. V&V conducted by a computational tool evaluatorcontracted by the customer

5. V&V conducted by a computational tool evaluatorcontracted by the a legal authority High Independence





Fundamentals of Verification and Validation

Real World

Benchmark PDE solutionBenchmark ODE solutionAnalytical solution

Complete System

Subsystem Cases

Benchmark Cases

Unit ProblemsHighly accurate solution

Experimental Data

Conceptual Model

Computational Model

Computational Solution

ValidationVerification





VerificationThe process of determining that a model implementationaccurately represents the developer’s conceptual descriptionand specification.

Benchmark PDE solutionBenchmark ODE solutionAnalytical solution

Highly accurate solution

Conceptual Model

Computational Model


Verification





Verification

Mathematics issue. Verification provides evidence that themodel is solved correctly.

I Identify and remove errors in computer codingI Numerical algorithm verificationI Software quality assurance practice

I Quantification of the numerical errors in computed solution





Verification

I Improving software quality through development process

I Application Programming Interface (API) defines softwareservices developed by Developers that are used by Users

I Public API is forever (asset, liability)I Realistically, API evolvesI API design is tough (perfection unachievable, try anyway)I Literate programming for API

I Two sided software platform connect Developers ANDusers through positive feedback (through API)





ValidationThe process of determining the degree to which a model isaccurate representation of the real world from the perspectiveof the intended uses of the model.

Real World

Complete System

Subsystem Cases

Benchmark Cases

Unit Problems

Experimental Data

Conceptual Model

Computational Model


Validation





Validation

Quantification of uncertainties and errors in the computationalmodel and the experimental measurements

I Goals on validationI Tactical goal: Identification and minimization of

uncertainties and errors in the computational modelI Strategic goal: Increase confidence in the quantitative

predictive capability of the computational modelI Strategy is to reduce as much as possible the following:

I Computational model uncertainties and errorsI Random (precision) errors and bias (systematic) errors in

the experimentsI Incomplete physical characterization of the experiment





Validation Procedure UncertaintyI Aleatory uncertainty→ inherent variation associated with

the physical system (variation in external excitation,material properties...). AKA irreducible uncertainty,variability and stochastic uncertainty.

I Epistemic uncertainty→ potential deficiency in any phaseof the modeling process that is due to lack of knowledge(poor understanding of mechanics...). AKA reducibleuncertainty, model form uncertainty and subjectiveuncertainty.

Hei

sen

ber

gp

rin

cip

le

un

cert

ain

tyA

leat

ory

un

cert

ain

tyE

pis

tem

ic

Det

erm

inis

tic





Use of Physical Experiments for Validation

I Traditional ExperimentsI Improve the fundamental understanding of physics involvedI Improve the mathematical models for physical phenomenaI Assess component performance

I Validation ExperimentsI Model validation experimentsI Designed and executed to quantitatively estimate

mathematical model’s ability to simulate well definedphysical behavior

I The simulation tool (SimTool) (conceptual model,computational model, computational solution) is thecustomer





Validation Experiments

I A validation experiment should be jointly designed andexecuted by experimentalist and computationalist

I Need for close working relationship from inception todocumentation

I Elimination of typical competition between eachI Complete honesty concerning strengths and weaknesses

of both experimental and computational simulations

I A validation experiment should be designed to capture therelevant physics

I Measure all important modeling data in the experimentI Characteristics and imperfections of the experimental

facility should be included in the model





Validation Experiments (contd)I A validation experiment should use any possible synergism

between experiment and computational approachesI Offset strength and weaknesses of computations and

experimentsI Use high confidence simulations for simple physics to

calibrate of improve the characterization of theexperimental facility

I Conduct experiments with a hierarchy of physics complexityto determine where the computational simulation breaks(remember, SimTool is the customer!)

I Maintain independence between computational andexperimental results

I Blind comparison, the computational simulations should bepredictions

I Neither side is allowed to use fudge factors, parameters





Validation Experiments (contd)I Validate experiments on unit level problems, hierarchy of

experimental measurements should be made whichpresent an increasing range of computational difficulty

I Use of qualitative data (e.g. visualization) and quantitativedata

I Computational data should be processed to match theexperimental measurement techniques

I Experimental uncertainty analysis should be developedand employed

I Distinguish and quantify random and correlated bias errorsI Use symmetry arguments and statistical methods to identify

correlated bias errorsI Make uncertainty estimates on input quantities needed by

the SimTool




Prediction

Outline

Before We Start



Summary




Prediction

Prediction

I Prediction: use of computational model to foretell the stateof a physical system under consideration under conditionsfor which the computational model has not been validated

I Validation does not directly make a claim about theaccuracy of a prediction

I Computational models are easily misused (unintentionallyor intentionally)

I How closely related are the conditions of the prediction andspecific cases in validation database

I How well is physics of the problem understood




Prediction

Relation Between Verification, Validation andPrediction

Quantification of confidence in a prediction:

I How do I quantify verification and its inference value in aprediction?

I How do I quantify validation and its inference value in apredictions?

I How far are individual validation experiments from physicalsystem of interest?




Prediction

Application Domain Complete or Partial Overlap

System Parameter

Sys

tem

com

plex

ity

DomainValidation

DomainApplication Domain

Application

System Parameter

Sys

tem

com

plex

ity

DomainValidation

I Rarely applicable to infrastructure objectsI Environmental influences (generalized loads, conditions,

wear and tare) are hard to predictI Human factors (I-35 bridge failure, forgot to multiply loads

with safety factor...)




Prediction

Application Domain – No Overlap

System ParameterS

yste

m c

ompl

exity

ApplicationDomain

DomainValidation

Inference

I Inference⇒ Based on physics or statistics (or both)I Validation domain is actually an aggregation of tests and

thus might not be convex (bifurcation of behavior)I Validation domain for infrastructure objects (bridges, dams,

buildings, ports...) is exclusively non–overlapping with theapplication domain.




Prediction

Importance of Models and Numerical Simulations

I Verified and Validated models can be used for assessingbehavior of

I components orI complete systems,

I with the understanding that the environmental influencescannot all be taken into the account prior to operation

I but with a good model, their influence on system behaviorcan be assessed as need be (before or after the event)




Prediction

Prediction under Uncertainty

I Ever present uncertainty needs to be estimated forpredictions

I Identify all relevant sources of uncertainty (see next lectureon uncertain soils)

I Create mathematical representation of individual sources

I Propagate representation of sources through modeling andsimulation process




The T Experiments

Outline

Before We Start



Summary




The T Experiments

Errors in Scientific SoftwareI Les Hatton, Oakwood Computing

I "Extensive tests showed that many software codes widelyused in science and engineering are not as accurate as wewould like to think."

I "Better software engineering practices wold help solve thisproblem,"

I "Realizing that the problem exists is an important firststep."

I Large experiment over 4 years measuring faults (T1) andfailures (T2) of scientific and engineering codes




The T Experiments

The T1 ExperimentsI Measured defects without running the code, measuring

formal consistency of 3,305,628 lines f77 and 1,928,011lines of C

I 100 codes, 40 application areas: graphics, nuc. mech.chem. aero. civil engineering, comms, DBs, med. systems

I Safety–critical and non–safety–critical applicationsI Applications with and without internationally standardized

systems of quality controlI Mature codes (1 - 20 years old), in regular useI Errors (select), arguments sequence in function calls,

misunderstanding of finite precision arithmetic, codecomplexity




The T Experiments

T1: C Sources




The T Experiments

T1: f77 Sources




The T Experiments

The T2 Experiments

I Specific application area: seismic data processing (inverseanalysis)

I Echo sounding of underground and reconstructing"images" of subsurface geological structure

I Nine mature packages, using same algorithms were usedon a same data set!

I 14 primary calibration points for results checkI Results "fascinating and disturbing"




The T Experiments

T2: Disagreement at Calibration Points




The T Experiments

T2: Stage 14, Interpretation of Data




Select V and V Examples

Outline

Before We Start



Summary





Constitutive Integration Error MapsNormalized error: δr =

√(σij − σ∗

ij )(σij − σ∗ij )/

√σ0

pqσ0pq

von Mises, linear isotropic hardening:

0 50 100 150 2000

20

40

60

80

100

p (kPa)

q (k

Pa)

5e−010

5e−0101e−009

1e−009

1.5e−009(a) At θ = 0

0 0.2 0.4 0.6 0.8 10

20

40

60

80

100

θ (rad)

q (k

Pa)

1e−010

2e−0103e−0104e−010

5e−0106e−010

(b) At p = 100 kPa





Constitutive Integration Error Mapsvon Mises, non–linear kinematic hardening(Armstrong-Frederick):

0 50 100 150 2000

20

40

60

80

100

p (kPa)

q (k

Pa)

0.002

0.0040.006

0.0080.01 0.010.0120.0140.016

(a) At θ = 0

0 0.2 0.4 0.6 0.8 10

20

40

60

80

100

θ (rad)

q (k

Pa)

0.002

0.0040.0060.008

0.01 0.010.0120.0140.016

(b) At p = 100 kPa





Constitutive Integration Error MapsDafalias-Manzari, rotational kinematic hardening, boundingsurface:

0 50 100 150 2000

20

40

60

80

100

p (kPa)

q (k

Pa)

0.05

0.05

0.1

0.1

0.1

0.1

0.15

0.15

0.15

0.15

0.15

0.2

0.2

0.2

0.2

0.2

0.2

0.25

0.25

0.25

0.3

0.3

0.35

0.4

(a) At θ = 0

0 0.2 0.4 0.6 0.8 10

20

40

60

80

100

θ (rad)

q (k

Pa)

0.05 0.05

0.10.1

0.150.15

0.20.2

0.25

(b) At p = 100 kPa





Coupled Formulation and Implementation

(after Gajo 1995)Jeremic Computational Geomechanics Group




Coupled Formulation and Implementation





Material Model Validation (Dafalias-Manzari)




Summary

I Importance of proper verification and validation fornumerical predictions used in design of infrastructureobjects

I Public (yours and mine) safety issue!

I Would you trust (design using) numerical simulations ifSimTool did not follow (extensive) Verification andValidation procedures?



verification and validation in computational geomechanicssokocalo.engr.ucdavis.edu/~jeremic/ ·...

Documents