research issues in verification and validation d. e. stevenson computer science clemson university...

Research Issues in Verification and Validation

D. E. Stevenson Computer Science Clemson University

steve@cs.clemson.edu

Foundations ‘02

Foundations for V&V in the 21st Century was held at Johns Hopkins University Applied Physics Laboratory on 22-23 October 2002.

The Workshop

• 198 Participants.• 19 sessions, covering most VV&A topics.• Plenary review session each day. • Final plenary sessions on findings and

research issues.• Proceedings available at DMSO site www.dmso.mil>VV&A>Foundations02.

Take Home Message

• VV&A is about risk management across the entire spectrum of research, development, and management.

• VV&A cannot succeed unless we properly incorporate risk management throughout the cycle.

• Success hinges on development of staff competencies in VV&A.

Grand Challenge

“It remains impossible to quantify, either technically or managerially, how much resources must be allocated to VV&A tasks.”

From the Executive Summary

Challenges

• Management Challenges: How do we implement what we know how to do?

• Research Challenges: What areas must we understand better in order to find viable technical solutions?

General Findings

• The primary motivation is risk reduction.• Effective communication remains a

problem. • Advances in M&S framework/theory is

essential for increasing automated VV&A techniques.

• Limitations posed by lack of detailed characterization of associated uncertainties must be addressed.

General Findings II

• There is a great need to develop formal methods for both managerial and technical areas.

• Education and training are crucial to developing staff competencies. [BOK sessions tomorrow]

Research Challenges

Overview

• Lingering Issues

• Managerial Challenges

• Research Challenges

Lingering Issues

• How should VV&A change with M&S size, type, application, and complexity?

• How to develop better cost estimation processes?

• How to make better use of visualization, especially to enhance SME reviews?

• How to better connect statistical processes appropriately to SME validation reviews?

Lingering Issues II

• How to better disseminate insights from VV&A experiences to communities?

• How to provide more and better automation support (tools) for VV&A?

• How to adopt or adapt tools from the software industry?

Management Challenges

• Qualitative assessment. • Appropriate and effective use of formal

assessment processes.• M&S/VV&A costs/resources.• How to ensure that “best practices” are

employed where they exist and where pertinent?

Research Challenges

• Inference.• Coping with adaptive systems.• Aggregation.• Human involvement and

representation of human behavior.

Conclusions

• Risk related to enormous complexity of models and simulations.

• Management of complexity is crucial. • Oberkampf and Trucano suggested

development of effective methods of using phenomenon identification and ranking tables (PIRT) for planning and assessment system.

Conclusions II

• We need research into the cost effectiveness

• We need to establish “best-practices.”

Y’all come toFoundations ‘04

Questions?All Comments Welcome!

Details

Inference

• Data availability to support assessment of simulation “predictions” is a fundamental problem.

• Comparison can be described statistically in terms of accuracy, error, resolution, etc.

• Action. Develop scientifically rigorous methods for making inferences about relationships between simulation results and elsewhere in the application domain.

Adaptive Programming

• Adaptive programs include artificial intelligence (AI), expert systems, genetic algorithms, fuzzy logic, machine learning, etc.

• Presents fundamental challenges to the prediction and assessment of performance.

• Action. Develop scientifically rigorous methods to ensure adaptive programming performance meet VV&A demands.

Aggregation

• As simulations become more complex, especially multi-resolution, better methods for determining the potential impact on simulation results from such variation in levels of detail are required to minimize potential misuse of simulation results.

• Action. Develop supporting theory and assessment procedures.

Human Involvement/Representation

• Representing human behavior is a major challenge.

• There are many significant research issues concerning interactions among simulation characteristics, the people involved, and appropriate simulation uses.

• Action. Develop representations of cognitive processes.

HWIL

• Hardware in the loop (HWIL) continues to present significant VV&A challenges. There is a need to document conceptual models of components of HWIL and distributed simulation systems, particularly in regard to model detail and semantic consistency. Some problems continue to exist from our inability to manage the communications latency in distributed systems and the need to manipulate and store dense environmental data for real time effectiveness. HWIL shares a problem with much of general computer science: research is needed to deal with non-determinism in parallel applications.

Complexity

• To measure goodness, we must develop and standardize validation metrics. – Develop methods for the construction and use

of a validation hierarchy

– specification and use of quantitative assessment criteria for validation metrics

• We must understand propagation of validation metric information in the validation hierarchy.

Model-Based Development

• This is an evolving area in computer science as well as M&S. Some specific challenges were issued:– How to use multiple frameworks. – How to generate executables and test-harnesses from

declarative models. – How to use goal-oriented thinking in modeling. – How to understand non-deterministic systems better.– Improve methods such as static analysis, runtime

verification, and model checking.

Formal Methods

• Research into effective methods for generating complete coverage test cases from formal specifications.

• Development of standardized test problems • Method of Manufactured Solutions Foundations.• Research into formal verification, and

“lightweight formal methods” to formally do VV&A early and often.

Mathematical Issues

• Formal methods are one avenue, but continued development of statistical methods for software quality assurance (SQA), M&S to establish the principles of predictable compositional modeling. Statistical methods are also central to the validation process.

Subject Matter Expert Areas

• SME-related Knowledge Engineering• Research into methods of guaranteeing

consistency in SME assessments• Capture in formal mechanisms of SME

knowledge• What truly qualifies someone to fulfill the SME

role?

REFERENCES

Dale K. Pace, D. E. Stevenson, and Simone Youngblood. Executive Summary in Foundations ’02: Foundations for VV&A in the 21st Century. San Diego, CA: Society for Computer Simulation. 2002.

Participants

• 198 attendees– 40% from the U.S. Defense community.– 15% from other U.S. government

organizations.– 25% from academia.– 10% from other industry organizations– 10% from outside the U.S.

Global Ideas

• Cost and resource requirements for M&S VV&A are not as well understood.

• More information about cost and resource requirements needs to be collected and made available to facilitate development of more reliable estimation processes.

• Many areas of M&S VV&A need to employ more formal (repeatable and rigorous) methods to facilitate better judgments about appropriateness of simulation capabilities for intended uses.

Management Challenges

Qualitative assessment involves human judgment in assessment: “peer review,” “subject matter expert (SME)” evaluation, face validation, etc. The managerial challenge is to guarantee that people have appropriate credentials and/or that formal processes are in place.

Qualitative Assessment

Formal Assessment

Formal assessment can be difficult to employ fully. The management challenge is to develop appropriate “light-weight” variants of the processes which can be more easily employed in M&S VV&A to enhance the quality of formal assessments.

Costs/Resources

Correct estimation of resources is a primary challenge in M&S applications. We lack adequate information for reliable estimation of M&S VV&A costs/needed resources. The management challenge is to collect and organize appropriate cost and resource information from whatever sources to develop for M&S/VV&A cost/resource estimation can be developed.