Formalized Model Development & Test Generation: Key Role of Abstraction

Bernard P. Zeigler

Arizona Center for Integrative Modeling and Simulation (ACIMS)www.acims.arizona.edu

and Joint Interoperability Test Command (JITC)

Fort Huachuca, AZ

Foundations 04, ASU, Tempe, Oct.13-15

Outline

• Systems-based framework for M&S– Bernard P. Zeigler and Hessam S. Sarjoughian, “Implications of

M&S Foundations for the V&V of Large Scale Complex Simulation Models” Proc. Foundations 02, Laurel, MD, November, 2002, available from www.scs.org

• Application to Systems Development– Interoperability Standards Conformance Testing

• Role of Abstraction– Nature of abstraction, scalability– Abstractions employed in JITC application

Simulation-based V&V as an afterthought

development of the system

simulation tests of the system

Government

Contractor

Automated test suite design

Informal BehaviorRequirementsat one or morelevels of SystemSpecification

Simulation based testing

Reference Model

FormalizedBehaviorSpecification

Software-IntensiveSystemImplement-ations

Formalized Model System Development Process

• Informal Behavior Requirements at one or more levels of System Specification: Systems theory offers well-characterized levels at which requirements for system behavior can be stated informally.

• Formalized Behavior Specification: The behavior requirements are translated into a formalized model using framework for M&S methodology..

• Reference Model: serves as a reference model for any implementation of the standard. This model can be analyzed and simulated with a simulation protocol to study logical and performance attributes Using model continuity, it can be executed in real-time protocol and provides a proof-of-concept prototype for an operational system.

• Automated test suite design: Branching in the lower path from the formalized specification, we develop a test suite consisting of test models within a test frame that can interact with a System Under Test (SUT) to test its behavior relative to the proposed standard.

• Simulation based testing: The test suite is implemented in a distributed simulation infrastructure and executed against the SUT. The test suite provides explicit pass/fail/unresolved results with leads as to components/rules that might be sources of failure.

• Optimization and Fielded Execution: The reference model provides a basis for correct implementation of the standard in a wide variety of technologies. The test suite provides a basis for testing such implementations as SUTs in a suitably extended test infrastructure.

• Iterative nature of development: The process is iterative allowing return to modify the master formalized model and its informal precursor requirements. Model formalization minimizes the artifacts that have to be modified as the process proceeds.

Formalized Model System Development Process

How is simulation software different from other software?

• It represents the behavior of dynamic systems whose state changes intrinsically depend on time

• Properly controlling the flow of time is critical• Simulation software may combine:

– continuous (time-driven) and discrete (event-driven) processes

– actual operating hardware and software representations

– wall clock and {faster/slower} than real time advance

independent specification of the dynamic system behaves over time

methods needed to compare the behavior of the system implementation and the independent specification

A Formal Framework for Modeling and Simulation should provide

DEVS

Experimental

Frames

• DEVS = Discrete Event System Specification

• Provides formal M&S framework: specification,simulation

• Derived from Mathematical dynamical system theory

• Supports hierarchical, modular composition

• Object oriented implementation

• Supports discrete and continuous paradigms Exploits efficient parallel and distributed simulation

techniques

Theory of Modeling and Simulation, 2nd Edition, Academic Press, 2000, Bernard P. Zeigler, Herbert Praehofer, Tag Gon Kim

DEVS Background

DEVS Hierarchical Modular Model Framework

Atomic: lowest level model, contains structural dynamics -- model level modularity

Atomic

AtomicAtomic

Atomic

+ coupling

Atomic

Atomic

Atomic

Coupled: composed of one or more atomic and/or coupled models

hierarchical construction

Automated test suite design: DEVS Federates

Standards DocumentMil-Std 6016C (JSLIRS)

HLA InfrastructureSimulation based testing

Reference Master DEVS Model

Captured As DEVSTemporalRules

DEVSModelContinuity:SystemImplement-ations

Formalized DEVS-based System Development Process

Interoperabilty Standards Test Generaton

• Express the Mil-Std 6016C (JSLIRS) as a collection of state variables and transition rules

• Rules are written as conditions, actions, and exceptions– If D is true now then do action E at time t, unless F occurs

in the interim

• State variables and rules characterize the dynamic behavior of the system

• Basis for semi-automated reasoning tools to generate test cases that target specific system behavior

Transactions consist of sets of Rules

Constraints(Exception)Rules

Stop

Modify C2Record for TN

1 2 3

RuleProcessing

Stop, Do Nothing,Alerts, Or jump to other

Transaction

TrackDisplay

Operatordecisions

Validity checking

TransmitMsg

Other ConsequentProcessing

Jumps (stimuli) to other

Transactions of specification

Stimuli

Transaction Level - example P.1.2 = Drop Track Transmit

Preparation Processing

Timeouts

PeriodicMsg

Abstractions: Needed For Scalability

Base Specification

Abstraction Analyze

Analyze useful information?

Preservation

mapping

•Different abstractions serve to address different questions•They may be sequenced combined, and crosschecked•Abstractions may be made more concrete

(If you could)

inverse

mapping

Answers

Rules Dependency Abstraction

Rule 2 depends on

(can be affected by the execution of )

Rule 1

if there is a non-empty intersection between the action variables of Rule 1 and the condition variables of Rule 2

Rule 1Rule 1

A

B

C

Rule 2Rule 2C D

Dependency Abstraction and Concretization

Rule Abstractions

ExecutableRules

DependencyGraphAnalysis

Rule EngineSimulation

Textualprocessing

Rules ExpressedIn textual form(XML)

Manual Concretization

candidatetest paths

Feasibility Verification

test plan

Example Dependency Graph

start rule

ordinary rule

output rule

rule

text

dependenciesrule

level

Rule set: Operator action to initialize the terminal or change data

Example: Path Analysis

shortest path

(no cycles allowed within)

rule

under

test

test sequence for rule under test = feasible path from a start to an output that traverses the rule under test

feasible = can be forced on the system under test by starting in a state (variablevalue assignments) that allows the designatied sequence of rule activiations to proceed

feasibility has to be checked by rule engine

Rule test sequence:

1. Initialize Terminal

2. Check input data validity

3. Transmit J2.2 PPLI message

Abstraction via Modulo Operation on Variable Names

G = Rules, VariablesDependency Graph

GModulo

Message types

Messagetypes

Recordtypes

invocations

GModulo

Record types

GModulo

Direct calls

Message Flow

DataStore

Dependency

Invocationdependency

+

Analysis

Analysis

Analysis

. 32. 32xxx J yyy J

Abstraction via Rule Set Partitioning

G = Rules, VariablesDependency Graph

G/group1

G/group2

G/group3

HierarchicalModular

DEVS

Employing Module Abstraction to Discover Test Paths

• Module abstraction provides a “bird’s eye” view of the overall interdependence of transactions.

• For example, we might seek a path that tests a desired sequence of transactions, and therefore, traverses a corresponding sequence of transaction modules

• Such abstraction can eliminate large numbers of possible paths from consideration since paths not present at a higher level cannot appear at a lower level.

• Moreover discovery of potential paths can be broken into two stages:– discover a path through modules– verify that internal connecting paths within the modules along this path exist

• Potential paths need to be tested for feasibility using rule engine

• For large rule sets, this multi-abstraction/multi-stage approach can significantly reduce the search necessary for path discovery.

Summary• A Formalized Model System Development Process provides the rigorous basis for

specification of behavior. Such specification drives integrated and concurrent development of effective processes to test system implementations.

• Test generation needs to exploit both micro and macro level analysis. Requirement might be for detailed local function testing or for overall sequence of behaviors

• Abstraction is the key to the scalability needed to develop and test large complex systems

• Multi-abstraction/multi-stage path discovery is a scalable methodology

• We will apply these methods and tools to generate actual test cases, and conduct reviews by SME’s to guide further development

• This formalized model, semi-automated test case development approach allows JITC to efficiently extract test cases from a standard, convert them to test procedures and apply these directly to a system for Standards Conformance and other testing efforts.