we 0201 – pkit2date : november 2004slide : 1 / 74 weag – thales – jp 11.20 revvarevva
TRANSCRIPT
WE 0201 – PKIT2 Date : November 2004 Slide : 1 / 74
WEAG – THALES – JP 11.20WEAG – THALES – JP 11.20WEAG – THALES – JP 11.20WEAG – THALES – JP 11.20
REVVAREVVAREVVAREVVA
WE 0201 – PKIT2 Date : November 2004 Slide : 2 / 74
Presentation Outline
Project facts ; the consortium
Background and rationale
The REVVA concepts
The REVVA generic process
The REVVA products
Techniques and support
REVVA technical achievements
REVVA future
WE 0201 – PKIT2 Date : November 2004 Slide : 3 / 74
Project Facts
Purpose:The THALES JP 11.20 is a Western European Union / WEAG research program funded by 5 nations: France (lead nation), Denmark, Italy, the Netherlands, Sweden.
Objectives:To develop a basis for a common methodological framework for the Verification, Validation and Accreditation (VV&A) of data, models and simulations. To be a starting point for a draft standard defining VV&A taxonomy, process and methodology.
Project timeline:March 2003 – September 2004
Common Validation, Verification and Accreditation Framework for Simulation (REVVA)
WE 0201 – PKIT2 Date : November 2004 Slide : 4 / 74
The consortium
The THALES JP 11.20 is conducted by a multi-national consortium led by ONERA (France).
UNI-C (DK)
TNO (NL)
FOI – FMV(SW)
DATAMAT (IT)
ONERA – ALEKTO(FR)
WE 0201 – PKIT2 Date : November 2004 Slide : 5 / 74
REVVA background and rationale
WE 0201 – PKIT2 Date : November 2004 Slide : 6 / 74
REVVA Background (1)
JP 11.20REVVA
ITOP
AFDRG
DMSO
Capitalization
ExchangeSISO
NATO
RTP 11.13
Exchange
CapitalizationInformation
WE 0201 – PKIT2 Date : November 2004 Slide : 7 / 74
REVVA Background (2)
• VV&A state-of-the-art:– DMSO, SISO, NATO, ITOP.
• M&S scientific contributions:– Systems Theory (B. Zeigler)
• Contributions on the concept of validation coming from the epistemology of simulation
WE 0201 – PKIT2 Date : November 2004 Slide : 8 / 74
VV&A rationale (1)
Doing (only) what makes sense for customers
and users in their search for trust.
• Trust:– identification of trust components,
– support (based on project knowledge, consolidation by shared domain knowledge)
WE 0201 – PKIT2 Date : November 2004 Slide : 9 / 74
VV&A rationale (2)
Trust
M&S supplier skillsand reputation
Audit trail of experiments and uses
Formal analysis ofproducts, processes andorganisation
VV&A
Informal
Formal
WE 0201 – PKIT2 Date : November 2004 Slide : 10 / 74
VV&A rationale (3)
Processes
Organisation ProductsNeedsPurpose
FormalAnalysis
Acceptance
Validation
Verification
WE 0201 – PKIT2 Date : November 2004 Slide : 11 / 74
VV&A rationale (4)
Activities driven and prioritised by:
• Requirements (acceptance requirements)
• Identification / perception of risks
• Organisational needs, risks and benefits
• Costs
WE 0201 – PKIT2 Date : November 2004 Slide : 12 / 74
REVVA Concepts
WE 0201 – PKIT2 Date : November 2004 Slide : 13 / 74
REVVA Terminology (1)
Focus on Acceptance and Validation
• Decomposition of VV&A in Verification, Validation
and Acceptance; (temporal) independence of each
concept.– Verification: Model Verification.– Validation: Model Validation within an Experimental Frame (EF)
– Acceptance: Before use / After use
WE 0201 – PKIT2 Date : November 2004 Slide : 14 / 74
REVVA Terminology (2)
• Correctness:– The property of a simulation model to comply with
formal rules and bodies of reference information for its representation and for the transformation of its representation into another one.
• Verification:– The process which is used to construct, under a set of
time, cost, skills, and organisational constraints a justified belief about model correctness.
WE 0201 – PKIT2 Date : November 2004 Slide : 15 / 74
REVVA Terminology (3)
• Validity:– The property of a simulation model to have, within
a specific experimental frame, a behaviour which is indistinguishable under a set of validation criteria from the behaviour of the System of Interest.
• Validation:– The process which is used to construct, under a set
of time, cost, skills, and organisational constraints a justified belief about model validity.
WE 0201 – PKIT2 Date : November 2004 Slide : 16 / 74
REVVA Terminology (4)
• Acceptance:
– The informed decision of a M&S customer to use
the results of a simulation for a specific purpose.
– The informed decision of a M&S customer to use
a simulation, within an experimental frame, for a
specific purpose
WE 0201 – PKIT2 Date : November 2004 Slide : 17 / 74
REVVA Assumptions
• Acceptance (and Validation) should:
– be purpose oriented,
– be development neutral,
– be technology neutral,
– recognise the impact of organisations,
– recognise the impact of timeframe of M&S and VV&A,
– rely on data and knowledge management.
WE 0201 – PKIT2 Date : November 2004 Slide : 18 / 74
A comprehensive view
The"Real World"
ObjectivesNeeds
DecisionsActions
ProblemFormalisation
Problem solution
M&SQuestions
M&S Answers
SystemRequirements
M&S Enabling Products
The"Problem World"
The"M&S World"
The"M&S Product
World"
WE 0201 – PKIT2 Date : November 2004 Slide : 19 / 74
Methodological hints
• Independence
• Incremental nature of activities and results
• Focus on the intended purpose
• Objective assessment (Acceptance Criteria)
• Careful construction of a justified belief
• Tailoring
WE 0201 – PKIT2 Date : November 2004 Slide : 20 / 74
Tailoring
The EF used for Acceptance (and for Validation) is derived from the Acceptance Requirements All the activities are planed, designed and controlled by them. The Acceptance Requirements are not (necessarily) the M&S Requirements ; they are specific of each use and purpose
EF
M
Question
Answer
WE 0201 – PKIT2 Date : November 2004 Slide : 21 / 74
Impact of time
t| | |
Before M&S use1
BeforeM&S use2
AfterM&S use1
t1 t2 t3
Validation t1 Validation t2 Validation t3
• Verification is (mostly/hopefully) a one-shot process
• Validation is a permanent/recurrent issue
• Acceptance and Validation are purpose oriented and tailored
WE 0201 – PKIT2 Date : November 2004 Slide : 22 / 74
The REVVA Generic Process
WE 0201 – PKIT2 Date : November 2004 Slide : 23 / 74
Acquire Information
Develop ToVV
Conduct V&V
Assess Evidence
Assess Evidence
Integration
Evaluate V&V Report
DevelopToA
ToA
ToVV
V&V Items
Assessed items of evidence
V&V report
Acceptance Recommendation
Model information and system knowledge
Phase
Product
Intended purpose
Product flow
Back step
The REVVA Generic Process
ToA: Target of Acceptance
ToVV: Target of Verification and Validation
WE 0201 – PKIT2 Date : November 2004 Slide : 24 / 74
Phase Overlaps
Develop ToA
Acquire Information
Develop ToVV
Conduct V&V
Assess Items of Evidence
Assess Evidence Integration
Evaluate V&V Report
VV&A Endeavour DurationMain effort
WE 0201 – PKIT2 Date : November 2004 Slide : 25 / 74
Develop ToA
• Performed by the Acceptance Agent in close cooperation with the user
• Develop Acceptability Criteria (AC), including Validity Criteria (VC)
• Document rationale for development
WE 0201 – PKIT2 Date : November 2004 Slide : 26 / 74
Intended Purpose
Objectives and Sub-Objectives(”Questions”)
(Fictive) Experimental Frames
Precision Requirements for the Behavior Characteristics
Develop ToA (2)
WE 0201 – PKIT2 Date : November 2004 Slide : 27 / 74
Intended Purpose
AC1
AC2
Acceptability Criterion m
Sub-Objective 1 Sub-Objective 2 Sub-Objective n
AC3
ToASO1.1 SO1.1 SO1.1
WE 0201 – PKIT2 Date : November 2004 Slide : 28 / 74
Acquire Information
• Led by the V&V Agent, performed by the V&V Team
• Get overview over the data, information, and knowledge (likely to become) available– about the Simulation Executable Model– about the System of Interest
• Create data, information, and knowledge repository
• Identify additional sources
WE 0201 – PKIT2 Date : November 2004 Slide : 29 / 74
Develop ToVV
• Performed by the V&V Agent, supported by the V&V Team
• Develop an approach to demonstrate that the Validation Criteria are passed or failed, with the data, information, and knowledge available
• Identify data, information, and knowledge desirable for more efficient V&V
WE 0201 – PKIT2 Date : November 2004 Slide : 30 / 74
Intended Purpose
AC1
AC2
Acceptability Criterion m
Sub-Objective 1 Sub-Objective 2 Sub-Objective n
AC3
ToA
ToVV
SO1.1 SO1.1 SO1.1
TaskTask
Task
Sub-Criterion m.1
SC m.1.1 SC m.1.2 SC m.1.3
WE 0201 – PKIT2 Date : November 2004 Slide : 31 / 74
Conduct V&V
• Led by the V&V Agent, performed by the V&V Team
• Choose techniques and tools
• Run the technique, e.g.,– Use 2-test to measure deviation– Use SME to assess structural decomposition
• Record technique, referrenced information, and result as V&V item
• Modify ToVV, if creation of V&V item fails
WE 0201 – PKIT2 Date : November 2004 Slide : 32 / 74
Assess Evidence
• Performed by the V&V Agent
• Review documented V&V items
• Assess the possibility of an erroneous result
• Accept V&V item as assessed item of evidence, or reject it
• Direct feedback loop
WE 0201 – PKIT2 Date : November 2004 Slide : 33 / 74
Assess Evidence Integration
• Performed by the Acceptance Agent
• Assemble items of evidence according to most
recent ToVV
• Derive which VC are passed or failed– Intuitively, subjectivly– Based on methods for dealing with uncertainty
• Assess possibility of misperception
WE 0201 – PKIT2 Date : November 2004 Slide : 34 / 74
Intended Purpose
AC1
AC2
Acceptability Criterion m
Sub-Objective 1 Sub-Objective 2 Sub-Objective n
AC3
ToA
ToVV
Item ofEvidence IoE IoE
SO1.1 SO1.1 SO1.1
TaskTask
Task
Sub-Criterion m.1
SC m.1.1 SC m.1.2 SC m.1.3
WE 0201 – PKIT2 Date : November 2004 Slide : 35 / 74
Evaluate V&V Report
• Performed by the Acceptance Agent
• Review expressiveness and completeness of AC with newly gained knowledge
• Weigh impact of misperception according to possibility of misperception
• Make acceptance recommendation
WE 0201 – PKIT2 Date : November 2004 Slide : 36 / 74
The REVVA Products
WE 0201 – PKIT2 Date : November 2004 Slide : 37 / 74
Acquire Information
Develop ToVV
Conduct V&V
Assess Evidence
Assess Evidence
Integration
Evaluate V&V Report
DevelopToA
ToA
ToVV
V&V Items
Assessed items of evidence
V&V report
Acceptance Recommendation
Model information and system knowledge
Phase
Product
Intended purpose
Product flow
Back step
The REVVA Generic Process
WE 0201 – PKIT2 Date : November 2004 Slide : 38 / 74
Target of Acceptance (ToA)
• Precise specification of the Acceptability Criteria (coming from validity related functional requirements)
• Stakeholder oriented• “What exactly needs to be assessed?”• Important elements in the ToA nodes:
requirement, MoE, constraint, level of impact, decomposition argument
Acquire Information
Develop ToVV
DevelopToA
ToA
Intended purpose
WE 0201 – PKIT2 Date : November 2004 Slide : 39 / 74
ToA (2)
• Building– Getting the requirements– Deriving MoE and constraints– Determining the level of impact– Validation of the ToA – Reuse
• Using– Part of contract– Basis for assessment report
• Maintaining– Due to changes in science, stakeholder changing mind,
corrections in ToA found during V&V work, V&V findings may indicate more limited validity range
WE 0201 – PKIT2 Date : November 2004 Slide : 40 / 74
Model information and system knowledge
• Information used to determine the V&V strategy for demonstrating the model’s correctness and validity
• Identifies all sources of information and knowledge
Acquire Information
Develop ToVV
Conduct V&V
Assess Evidence
Integration
Model information and system knowledge
• Important elements: – design documents, code,
modeller’s perception of the real world, behaviour data (of model and sub-models), conceptual model.
– Real world data (referent): data on (related or similar) systems, expert opinions
WE 0201 – PKIT2 Date : November 2004 Slide : 41 / 74
Target of Verification and Validation (ToVV)
• V&V items: evidence required to substantiate the ToA Acceptability Criteria
• When necessary: further decomposition of criteria (extension of ToA)
Develop ToVV
Conduct V&V
Assess Evidence
IntegrationToVV
• Important additional elements in the ToVV: method to use for collecting evidence, required rigor, cost estimate, aggregation instructions
• Tasks
WE 0201 – PKIT2 Date : November 2004 Slide : 42 / 74
V&V Items
• A V&V item consists of:– relevant information, – the evaluation objective, – reference information, – an evaluation technique, – evaluation result.
• Must be reproducible
Conduct V&V
Assess Evidence
V&V Items
WE 0201 – PKIT2 Date : November 2004 Slide : 43 / 74
Assessed items of evidence
• Assessed items of evidence:– V&V agent assessment of the
V&V result, – the acceptance or rejection of
each V&V item, – probative force (depending on
method or technique and reference information)
Assess Evidence
Assess Evidence
Integration
Assessed items of evidence
WE 0201 – PKIT2 Date : November 2004 Slide : 44 / 74
V&V report
• The assessed items of evidence assembled and integrated based on the ToVV
• Links the assessed items of evidence to the Acceptability Criteria
• Indicates if– evidence is considered sufficient to pass an
acceptability criterion
– V&V items for a criterion are missing or too weak,
– disproving evidence was found
Assess Evidence
Integration
Evaluate V&V Report
V&V report
WE 0201 – PKIT2 Date : November 2004 Slide : 45 / 74
Acceptance Recommendation
• The final recommendation on acceptance• Based on the ToA• Residual uncertainty after V&V
– from probative forces,
– use level of impact to determine recommendation
• The recommendation states (with consequences for M&S product’s utility):– if required evidence is missing or too weak,
– if a weakness in the ToA was detected
Evaluate V&V Report
Acceptance Recommendation
WE 0201 – PKIT2 Date : November 2004 Slide : 46 / 74
A VV&A project management view
The "3 pillars" concept
Process
Organisation ProductsPurpose
Objectives
WE 0201 – PKIT2 Date : November 2004 Slide : 47 / 74
The M&S and VV&A roles
• M&S stakeholders
• M&S suppliers
• VV&A agents
• Experts
WE 0201 – PKIT2 Date : November 2004 Slide : 48 / 74
Levels of independence
• DV&V:– dependent V&V; V&V performed by suppliers and
accepted as it is
• IA:– independent assessment; V&V performed by supplier
and assessed by the V&V agent
• IV&V:– independent V&V; V&V performed under the control
of the V&V agent
WE 0201 – PKIT2 Date : November 2004 Slide : 49 / 74
Feasibility assumptions
• High-level organisational decision
• Standards of V&V documentation
• Incremental accumulation of knowledge:– project vs corporate knowledge management
• Process for knowledge revision
WE 0201 – PKIT2 Date : November 2004 Slide : 50 / 74
The REVVA Techniquesand support
WE 0201 – PKIT2 Date : November 2004 Slide : 51 / 74
REVVA point of view
• Identification of the most important techniques and support
• Techniques to support decision making, …
• Next direction: develop techniques to support the whole process
WE 0201 – PKIT2 Date : November 2004 Slide : 52 / 74
Most important techniques and support
• Most importance of Requirements Engineering– Develop ToA & ToVV
• Design of Experiments (DOE) – Conduct V&V: calibration, sensitivity analysis & validation
• Computation of probative force and convincing force– Assess evidence – Assess argument
• Uncertainty analysis– Assess evidence– Evaluate V&V Report
• Decision tools– Evaluate V&V Report
WE 0201 – PKIT2 Date : November 2004 Slide : 53 / 74
Claim-Argument-Evidence
Types of argument
Evidence
Evidence
Evidence
Claim
Argument• Deterministic – relying upon axioms, logic and proof
• Probabilistic – relying upon probabilities, statistical analysis
• Qualitative – relying upon adherence to standards, design codes
WE 0201 – PKIT2 Date : November 2004 Slide : 54 / 74
Decision Tools
• The model is fit for purpose
• The model is not fit for purpose
Fact
Perception
Valid Not valid
Accepted OK Type II error
Rejected Type I error OK
WE 0201 – PKIT2 Date : November 2004 Slide : 55 / 74
Uncertainty Analysis
• Integrated approach for uncertainty estimation based on– Behaviour Characteristics– Propositional Logic– Hierarchical Breakdown Structures– Possibility Theory– Basic Model of Decision Theory
• Increases quantitative share in decision making• Leaves ”subjectivity gaps” with
– Probative Forces – Impressions of confirmation and denial
WE 0201 – PKIT2 Date : November 2004 Slide : 56 / 74
p1
p1’: p1.1 p1.2 p1.3
+(--)
p1.1 p1.2 p1.3
Composition and Inference
+ : inference relation (p1’ p1 : p1’ is a sufficient condition, but not a necessary one)
: composition relation (p1’ is a conjunction
of p1.1 through p1.3 )
(--) : quantification of denial impression (from (-) to (----))
(if p1’ is not verified, how strongly would it mean that p1 is not verified?)
WE 0201 – PKIT2 Date : November 2004 Slide : 57 / 74
Completeness and consistency checking
• The dependencies and consistency between the sponsor needs and requirements, any structured problem description, the model (conceptual, formal or executable), the ToA and ToVV and the final simulation results are checked
• Contents (evidences, specifications, etc.) without requirements and requirements without contents need to be resolved
WE 0201 – PKIT2 Date : November 2004 Slide : 58 / 74
REVVA technical achievements
WE 0201 – PKIT2 Date : November 2004 Slide : 59 / 74
REVVA Project Breakdown
The project includes six technical Work Packages:
WP 1 : Problem Analysis
WP 2 : Definition of AccreditationTarget
WP 3 : VV&A Techniques
WP 4 : Optimisation of Testing
WP 5 : Outline for a VV&A Methodological Framework
WP 6 : Evaluation of Results
WE 0201 – PKIT2 Date : November 2004 Slide : 60 / 74
Technical achievements (1)
• Taxonomy: Initial contribution dealing with V&V state of the art
• VV&A problem statement
• Review of acceptance criteria and levels of confidence
• Process models
WE 0201 – PKIT2 Date : November 2004 Slide : 61 / 74
Technical achievements (2)
• Emphasis on Acceptance Agent role: Target of Acceptance (ToA) and assessment of ToA satisfaction
• Integration of qualitative and quantitative confidence in a logical framework of argumentation
• Guidance for ToA and ToVV definition
WE 0201 – PKIT2 Date : November 2004 Slide : 62 / 74
Technical achievements (3)
• Review of existing techniques and tools resources
• V&V techniques state-of-the-art
• User guidance
WE 0201 – PKIT2 Date : November 2004 Slide : 63 / 74
Technical achievements (4)
• Capitalisation of data and knowledge
• Experience optimisation
• Relation with field trial data
• Identification of data/knowledge management issues
WE 0201 – PKIT2 Date : November 2004 Slide : 64 / 74
• Deep understanding of the terminology
• Definition of the VV&A scope
• Identification of VV&A organisations depending on level of impact
• Definition of VV&A products and processes
Technical achievements (5)
WE 0201 – PKIT2 Date : November 2004 Slide : 65 / 74
REVVA Outcomes
• Generic VV&A process, described by:
– Methodology Reference manual (METHGU)
– User’s Manual (PROSPEC)
• Illustration by case studies
• Draft contributions to guidance
• REVVA Website (www.JP1120-REVVA.com)
WE 0201 – PKIT2 Date : November 2004 Slide : 66 / 74
Lessons learnt at methodological level
• Compliant with the mainstream idea that every euro spent in requirements generates high Returns on Investment (ROI)
• The cost driver is the criticality of the use, not the quality of the process
• Generic, versatile methodology: impact on the easiness of tailoring
• Ability to comply with formal supports for guidance or automation of the process
WE 0201 – PKIT2 Date : November 2004 Slide : 67 / 74
Lessons learnt from the case studies
The REVVA application on Case Studies has shown capital points:
• To answer the right question:– It is important to formalise at early steps the problem to be
solved by M&S use AND the question asked to VV&A.
• To answer the question right:– It is enabled by the derivation of the ToVV from the ToA.
• To construct a justified belief:– The confidence is substantiated by gathering and assessing the
items of evidence.
WE 0201 – PKIT2 Date : November 2004 Slide : 68 / 74
Practical experience on case studies
• REVVA method helps to balance between purpose, model and data
• Difficult to gather the right information
• ToA seems to be particularly useful for large
projects
WE 0201 – PKIT2 Date : November 2004 Slide : 69 / 74
REVVA communications
• 04S-SIW-153 : A Common Validation, Verification and
Accreditation Framework for Simulations.
• 04S-SIW-005 : T&E Based on M&S VV&A:
The Importance of Experience Capitalization.
• 04S-SIW-152 : Assessment Criteria and V&V Levels in
JP11.20.
Three papers have been presented in a special session during the Spring '04 SIW
WE 0201 – PKIT2 Date : November 2004 Slide : 70 / 74
REVVA Future
WE 0201 – PKIT2 Date : November 2004 Slide : 71 / 74
Follow-up
REVVA is a first methodological step;the work will continue with
a EUROPA project (REVVA2) which is to start during Spring 05.
WE 0201 – PKIT2 Date : November 2004 Slide : 72 / 74
REVVA2: Conceptual challenges
• Organisation and roles clarification
• Incremental knowledge management (i.e. repository …)
• Formalisation of VV&A products
• Formalisation of the methodology
• Tailoring
• Costs model and project management
WE 0201 – PKIT2 Date : November 2004 Slide : 73 / 74
REVVA2: Technical challenges
• VV&A requirements formalisation
• V&V evidences : collection, organisation
• V&V argumentation: formalisation and support
• V&V and decision-making under uncertainty
• Formal foundations for V&V tools
• V&V tools selection and guidance (tailoring)
• V&V actual processes support
WE 0201 – PKIT2 Date : November 2004 Slide : 74 / 74
REVVA2: Current status
• Objective: to produce draft documents for standardisation.
• The project timeline will be 36 months, expected start in March 2005.
• EUROPA MoU• FR, DK, SW and NL intend to join REVVA2.• The UK may consider to join after CConVV&A.• The WBS are refined taking into account the roadmap
defined in Munich.