sas_08_model_val_tech_heimdahl mac-t ivv-08-152 model-validation in model-based development kurt...

SAS_08_Model_Val_Tech_Heimdahl MAC-T IVV-08-152

Model-Validation in Model-Based Development

Kurt Woodham

L-3 Communications

Ajitha Rajan, Mats Heimdahl

University of Minnesota

OSMA SAS ’08OSMA SAS ’08 September 8-12September 8-12

2 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl

Problem: Model Validation

• Model-Based Development (MBD) is here to stay Use of MBD is accelerating

Estimate 50% of NASA development projects using some form of MBD

Many advantages: model-checking, code generation, desktop testing, closed-loop simulation

Enhances early detection of requirement, design, or implementation defects

“Executable Specifications” enable evaluation of behavior that might otherwise be relegated to Inspections and Testing

• How do we know the models are “right”? Manually develop black-box tests

• When have we validated enough? Measure test coverage on an implementation/model


Problem : Current Practice

• Measure black-box test coverage over the model Indirect measure

Defects of omission in model not exposed.

Executable artifact is necessary Adequacy can only be determined late in the

development process

Incomplete ModelWeak

Black-Box Test set


Goals of Project

• Define metrics for objective, implementation-independent measure of adequacy of a black-box test suite

• Develop tools to measure validation adequacy based on the defined metrics

• Provide capability for autogeneration of black-box test suites


Testing – What does it mean?

Specification

Implementation

Does it implement?

Assertions

Model

Does it implement?

Assertion Based Testing (ABT) to Validate Model

Model

Source Code

Does it implement?

Model-Based Testing (MBT) to Verify Code

Our contribution is in providing novel ABT capabilities

In General


What are Assertions?

AssertionsProperties/ Formal

Assertions

Defined over

System

in1

ink

out1

outm

Can also be over components, interfaces,...


Contributions - ABT

Assertions Black-Box Tests

Measure Adequacy

Auto-generate

2

Model

Validate

Assess Model and Assertion Completeness

31

We provide the following contributions in the Assertion-Based testing domain (indicated by in the above figure):

1. Objective, implementation-independent measure of adequacy of a black-box test suite

2. Auto-generation of black-box validation tests directly from assertions

3. Objective assessment of completeness of model as well as assertions


The Idea

…then define structural coverage metrics to directly and objectively describe coverage of

assertions

Write assertions in a formal notation…G (FD_On -> Cues_On);

G((¬ Onside_FD_On Λ ¬ Is_AP_Engaged) → X(Is_AP_Engaged → Onside_FD_On))

Temporal Logic

1

Property_Satisfied

AND

OR

NOTAND

4

Right_FGS_Active

3

Left_FGS_Active

2

Right_Independent_Mode

1

Left_Independent_Mode

Synchronous Observers

microwave_library_temp/mode_logic

Printed 14-Jul- 2006 12:51:47

ON

OFFentry:mode=1;

Okentry: mode=3;

FAILEDentry: mode=2;

[steps_remaining>0]/steps_remaining--;

3

{steps_remaining=steps_to_cook;}

[start && steps_to_cook>0]

[steps_remaining<=0]

2 [door_closed]

1[start && ...door_closed]

1

[clear_off || ...!door_closed]

1

2

[clear_off]/steps_remaining=0;

2


A

LTL Temporal Operators

Operator Notation Meaning

Globally A G(A) Formula A is true in all states

Future A F(A) Formula A is true in some future state

A until B A U B Formula A is true in every state until B becomes true. B must eventually become true for the property to be true.

Next A X(A) Formula A is true in the next state

S0 S1 S2 S3 Si

A A A

B

A


Formalizing Assertions

“If the onside FD cues are off, the onside FD cues shall be displayed when the AP is engaged”

G((¬ Onside_FD_On ¬ Is_AP_Engaged) → X(Is_AP_Engaged → Onside_FD_On))

• Possible Coverage Metrics Assertion coverage: single test case that

demonstrates that assertion is satisfied Prone to “dumb” tests, e.g., execution in which AP is

never engaged. More rigorous metric is necessary


Task - 1

• Define a collection of assertion coverage criteria

• Formalize the assertion coverage obligations


Antecedent Coverage

• Many of the assertions in the FGS are of the form :

Globally if ‘A’ occurs then ‘B’ will occur

G (A → B) Two ways of satisfying (A → B)

– A is false– A is true and B is true

• Antecedent Coverage – test cases will exercise the antecedent.

S0 S1 Sn

Not A Not A A, B

What if: ACD → B


Modified Condition/Decision Coverage (MC/DC)

• To satisfy MC/DC Every point of entry and exit in the model should be invoked

at least once, Every basic condition in a decision in the model should take

on all possible outcomes at least once, and Each basic condition should be shown to independently

affect the decision’s outcome

A B A or B

F F F

T F T

F T T

Independent effect of AIndependent

effect of B

Basic Conditions


Unique First Cause (UFC) Coverage

“System shall eventually generate an Ack (A) or a Time Out (B)”

Req. LTL property - F(A B) .

Formal UFC obligation for A : ¬(A B) U (A ¬B)

for B : ¬(A B) U (B ¬A)

S0 S1 S2 S3 Si

A, ¬ B¬A, ¬B ¬A, B¬A, ¬B ¬A, ¬B

Path satisfies UFC obligation for A but not B.

S0 S1 Si

To show independence of B, ¬A, B¬A, ¬B ¬A, ¬B


UFC Coverage

• G(A)+ = {A U (a G(A)) | a є A+} G(A)- = {A U a | a є A-}

• F(A)+ = {¬A U a | a є A+} F(A)- = {¬A U (a G(¬A))| a є A-}

• (A U B)+ = {(A ¬B) U ((a ¬B) (A U B)) | a є A+} {(A ¬B) U b | b є B+}

(A U B)- = {(A ¬B) U (a ¬B) | a є A-} {(A ¬B) U (b ¬(A U B)) | b є B-}

• X(A)+ = {X(a) | a є A+}X(A)- = { X(a) | a є A-}

Michael Whalen, Ajitha Rajan, Mats Heimdahl and Steven Miller. Coverage Metrics for Requirements-Based Testing. In Proceedings of ISSTA 2006.


Task 2 – Validation Adequacy Measurement Tool

Formal Assertions (Eg. LTL

Properties)

Assertion Cov. Criteria

Assertion. Cov. Obligations/formulas

Validation Test Suite

Evaluate and Check formulas

Calculate ratio of formulas that

were true

Run Test Suite

Derive

Assertion Coverage Achieved by Test

Suite

We currently support the following coverage metrics:

• Assertion Coverage

• Assertion Antecedent Coverage

• Assertion UFC Coverage


Task - 3

• Automatically generate requirements-based tests from … Formal assertions Abstract model called Assertion Model created

using assertions and environmental constraints (specified as invariants)

… to provide the defined assertion coverage.


Assertion Model

Formal Assertions (Eg. LTL properties)

Trap Properties (for Cov. Oblig.)

Model Checker

Assertion Cov. Criteria

(eg. UFC)

Counter-examples(Assertion-based test

cases)

Inputs, Ouputs, Environmental

constraints

Generate Generate

Automatically Generating Requirements-Based Tests

Common with the Adequacy Measurement

ToolAssertions and environmental

constraints specified as invariants


What Are Model Checkers?

• Breakthrough technology of the 1990’s• Widely used in hardware verification

(Intel, Motorola, IBM, …)

• Several different types of model checkers Explicit, Symbolic, Bounded, Infinite Bounded, …

• Exhaustive search of the global state space Consider all combinations of inputs and states Equivalent to exhaustive testing of the model Produces a counter example if a property is not true

• Easy to use “Push button” formal methods Very little human effort unless you’re at the tool’s limits

• Limitations State space explosion


Preliminary Evaluation

Interested in determining:• Feasibility of generating assertion-based tests from a set of

assertions Generated assertion-based tests to provide UFC coverage over the

assertions

• Effectiveness of these test sets in validating the system model

Measured MC/DC achieved by the test sets over the system model

Used three realistic sized examples: • Flight Guidance System (FGS), • and two models related to the Display Window Manager

system (DWM1 and DWM2)

http://w

ww

.umsec.um

n.edu

21

Results

Ajitha Rajan, Michael Whalen, and Mats Heimdahl. Model Validation using Automatically Generated Requirements-Based Tests. In Proceedings of 10th IEEE High Assurance Systems Engineering Symposium, Nov 2007.


Results and Analysis

• UFC test suites achieved high MC/DC coverage over DWM models – well defined set of assertions

• Test-suite generated for UFC achieved very low MC/DC over the FGS model“When the FGS is in independent mode, it shall be active”.

G(m_Independent_Mode_Condition.result → X(Is_This_Side_Active = 1))

RSML–e MacroStructure of Independent_Mode_Condition is not captured in the property

Independent_Mode_Condition = ((Is_LAPPR_Active & Is_VAPPR_Active & IS_Offside_LAPPR_Active & Is_Offside_VAPPR_Active) |

( Is_VGA_Active & Is_Offside_VGA_Active))


Benefits of ABT

• Saves time and effort in generating validation test suites from assertions

• Effective method for generating model validation tests when the assertions are well defined

• Helps in identifying missing assertions and over constrained models


Bonus Task – Adequacy of Conformance Testing

Model

Conformance Tests

Code

Run

Measure Adequacy

Assertions

Measure Adequacy

1. Direct assessment of how well tests exercise the assertions

2. Will expose defects of omission

3. Assertion coverage could necessitate longer test cases than for model coverage

Useful ??


Assertion Coverage as an Adequacy Measure for Conformance Testing

Hypothesis 1(H1): Conformance tests providing assertion UFC coverage are more effective than conformance tests providing MC/DC over the model

Hypothesis 2(H2): Conformance tests providing assertion UFC coverage in addition to MC/DC over the model are more effective than conformance tests providing only MC/DC over the model


Experiment

• Used four industrial systems : Two models from the display window manager Two models representing the mode logic of a

flight guidance system

• Assessed effectiveness of test suites in terms of their fault finding ability

Ajitha Rajan, Michael Whalen, Matt Staats, and Mats Heimdahl. Requirements Coverage as an Adequacy Measure for Conformance Testing. To Appear in Proceedings of 10th International Conference on Formal Engineering Methods, Oct 2008.


Results – Hypothesis 1

MC/DC vs UFC

0

20

40

60

80

100

DWM1 DWM2 Latctl Vertmax

% F

au

lt F

ind

ing

Avg. MC/DC

Avg. UFC

Hypothesis 1 rejected at 5% statistical significance on all but the Latctl system


Analysis – Hypothesis 1

• Model coverage better than assertion coverage for measuring adequacy of conformance test suites

• Assertion UFC coverage is heavily dependent on the nature and completeness of the assertions

• Rigor and robustness of assertion coverage metric used is important UFC metric gets cheated when assertions are structured

to hide the complexity of conditions


Results – Hypothesis 2

MC/DC vs MC/DC + UFC

80

82

84

86

88

90

92

94

96

DWM1 DWM2 Latctl Vertmax

% F

au

lt F

ind

ing

Best MC/DC

Avg. Combined

Hypothesis 2 accepted at 5% statistical significance on all but the DWM2 system


Analysis – Hypothesis 2

UFC Achieved by MC/DC suites

Achievable UFC Rel. Diff

DWM1 28.3% 96.9% 70.8%

DWM2 59.7% 64% 6.7%

Latctl 94.7% 99.5% 4.8%

Vertmax 97.4% 99% 1.6%

Does UFC really help in revealing additional faults?


Summary – Bonus Task

• UFC > MC/DC FALSE 3 of the 4 case examples at 5% statistical significance

• UFC + MC/DC > MC/DC TRUE 3 of the 4 case examples at 5% statistical significance

• Combine rigorous metrics for assertion coverage and model coverage to measure adequacy of conformance test suites

• UFC metric is sensitive to structure of assertions Need assertion coverage metrics that are robust to

structure of assertions


Technology Readiness Level

• “Requirements-Based Test Generation Tool” TRL = 6

System/subsystem model or prototype demonstration in a relevant environment

• “Validation Adequacy Measurement Tool” TRL = 6

System/subsystem model or prototype demonstration in a relevant environment


Relevance to NASA

• MBD is here - estimate one-half of all NASA missions in development or on the books will use model-based subsystem development

Extensive use in avionics industry

• How do we know the models are right? Model validation problem

• We provide the capability to Objectively measure the “quality” of assertion-based black-box

validation tests Objectively assess the completeness of a model

Does the model address all assertions? Objectively assess the adequacy of a set of assertions

Are there enough assertions to adequately describe the model? Automatically generate truly assertion-based tests


Achievements to Date

• Formal assertion notation identified Most work with LTL Extended to work with Live Sequence Charts (LSC)

• Objective validation metrics defined Requirements, Antecedent, Unique First Cause, and Unique Cause

• Test case generation tool developed Developed tool generating tests from LTL

Capable of generating tests to all metrics defined Prototype tool working on LSC developed

• Developed test-adequacy measurement tool for the defined validation metrics

• Evaluation of metrics and tool • 12 papers and one PhD dissertation (Ajitha Rajan)


Next Steps

• Investigate alternative requirements notations to LTL

• Complete empirical evaluation of the effectiveness in model validation Flight Guidance Sensor (FGS) evaluation

complete Display Manager (DM) evaluation in work Coordinate evaluation on NASA IV&V project

• Coordinate technology transfer


Discussion

sas_08_model_val_tech_heimdahl mac-t ivv-08-152 model-validation in model-based development kurt...

Documents

assertions model

model model source code

heimdahl mact ivv

modelbased testing mbt

heimdahl testing

coverage of assertions

heimdahl problem

blackbox test suite