software specification 2ix20jschmalt/teaching/2ix20/2ix20_lecture11... · 2017-03-13 · software...

Software Specification 2IX20

Julien Schmaltz (with slides jointly with J. Tretmans, TNO&RUN)

Lecture 11:Introduction to Model-Based Testing

Context & Motivation

© Jan Tretmans and Julien Schmaltz

Testing

Testing:

checking or measuring some quality characteristics

of an executing object by performing experimentsin a controlled way

w.r.t. a specification

IUT

tester

specification


Formal Methods

FUse mathematics to model relevant parts of software

FPrecise, formal semantics: no room for ambiguity or misinterpretation

FAllow formal validation and reasoning about systems

FAmenable to tools: automation

F Examples: ♦ Z ♦ Temporal Logic ♦ First order logic ♦ SDL (Specification and Description

Language) ♦ LOTOS ♦ Promela ♦ Labelled Transition Systems ♦ Finite state machine ♦ Process algebra ♦ ......


Software Testing

Testing is:

F important

Fmuch practiced

F30% - 50% of project effort

Fexpensive

F time critical

Fnot constructive(but sadistic?)

But also:

F ad-hoc, manual, error-prone F hardly theory / research F no attention in curricula F not cool :

“if you’re a bad programmer you might be a tester”

Attitude is changing: F more awareness F more professionalImprovements possible

with formal methods ! ?


Types of Testing

unit

integration

system

efficiencymaintainability

functionality

white box black box

Level of detail

Accessibility

Characteristics

usabilityreliability

module

portability


A Model-Based Development Process

specification

realization

design

code

formalizablevalidation

formalverification

testing

model-based

informal world

world of models

real world

informalideas


formal world

concrete world

Verification is only as good as the validity of the model on

which it is based

Verification and Testing

Verification : Fformal manipulation Fprove properties Fperformed on model

Testing : F experimentation F show error F concrete system

Testing can only show the presence of errors, not their

absence


Model-Based Testing

FModel based testing has potential to combine

♦ practice - testing

♦ theory - formal methods

FModel Based Testing :

♦ testing with respect to a (formal) model / specification

state model, pre/post, CSP, Promela, UML, Spec#, . . . .

♦ promises better, faster, cheaper testing:

• algorithmic generation of tests and test oracles : tools

• formal and unambiguous basis for testing

• measuring the completeness of tests


Model Based Testingwith Formal Methods

Claims :

F Formal specifications are ♦ more precise, ♦ more complete, ♦ unambiguous

F Formal models can be formally verified

F Formal models allow automatic generation of tests


Towards Model-Based Testing: TrendsFIncrease in complexity, and quest for higher quality

software

FMore abstraction

♦ less detail ♦ model based development; OMG’s UML, MDA

FChecking quality

♦ practice: testing - ad hoc, too late, expensive, lot of time

♦ research: formal verification - proofs, model checking, . . . .

Software bugs / errors cost US economy yearly:

$ 59.500.000.000 (~ € 50 billion) (www.nist.gov) $ 22 billion could be eliminated…


x : [0..9] 10 ways that it can go wrong

10 combinations of inputs to check

y : [0..9]

x : [0..9]100 ways that it can go wrong


x : [0..9]y : [0..9]z : [0..9]

1000 ways that it can go wrong


Towards Model-Based Testing: TrendsIncrease testing effort grows exponentially with system size testing cannot keep pace with development of complexity and size of systems


Formal methods: F proving properties F research F sound theories F “clean”

Testing : F trial & error F practice F heuristics F “dirty hands”

Testing & Formal MethodsA Perfect Couple ?

“Testing is not necessary after formal verification”

“Testing can only detect the presence of errors, not their absence”

“Formal methods are toys for boys”

“Formal methods have extreme potential - but not for my project”


Testing with Formal Methods

FTesting with respect to a formal specification

FPrecise, formal definition of correctness :

good and unambiguous basis for testing

FFormal validation of tests

FAlgorithmic derivation of tests :

tools for automatic test generation

FAllows to define measures expressing coverage

and quality of testing


Testing & Formal Methods

Claims :

FCombining the “mathematically clean” world of formal

methods and the “dirty hands” world of testing

FTesting and formal methods are both necessary in

software development

FFormal methods improve the testing process

FFormal testing stimulates the use of formal methods


Automated Model-Based Testing

model

IUT

IUT confto model

TTCNTTCNtestcases

pass fail

test tool

test generation

tool

IUT passes tests

IUT confto model

⇔⇑ ⇓ soundexhaustive


Approaches to Model-Based Testing

Several modelling paradigms:

FFinite State Machine

FPre/post-conditions

FLabelled Transition Systems

FPrograms as Functions

FAbstract Data Type testing

Labelled Transition Systems


Model Based Testing

s ∈ LTS

i ∈ IOTS

i ioco s

pass fail

test tool

gen : LTS → ℘(TTS)

t || ii || gen(s) → pass

i ioco s

⇑ ⇓ soundexhaustive

pass fail

model

IUT

IUT confto model

test tool

test generation

tool

test execution

toolIUT passes tests

IUT confto model

⇑ ⇓ soundexhaustive

with Transition Systems


test execution

pass / fail

Formal Testing

test generation

test suite TS

specificationS

implementationi

correctness criterion

implementation relation

i passes Ts

i conforms-to s

⇔


Model-Based Testing for LTS

test execution

testgeneration

tests

model

IUT

confto

Involves:

• specification model

• implementation IUT + models of IUTs

• correctness

• test cases

• test generation

• test execution

• test result analysis pass / fail

=


Formal Testing with Transition Systems

exec : TESTS × IMPS → ℘(OBS)


Ts ⊆ TTS

s ∈ LTS

IUT ∈IMPS

ioco

iIUT ∈IOTS

passes : IOTS × TTS →

{pass,fail}

Proof soundness and exhaustiveness: ∀i∈IOTS .

( ∀t∈gen(s) . i passes t )

⇔ i ioco s

Test hypothesis :

∀IUT∈IMP . ∃iIUT ∈IOTS .

∀t∈TTS . IUT passes t ⇔ iIUT passes t

pass / fail

=

LTS with inputs and outputs


Labelled Transition System 〈 S, L, T, s0 〉

?coin

?button

!alarm ?button

!coffee

states

actions transitions T ⊆ S × (L∪{τ}) × S

initial state s0 ∈ S

Models: Labelled Transition Systems



a

ba

a b

a b

ab

a a

cb

a τ

c

a

a

a

a

b

b

b

b



S0 S1dub transition

dub coffeeS0 S3transition composition

kwart teaS0 executablesequence

non-executablesequence

kwart soupS0

LTS(L) all transitionsystems over L

L = { dub,kwart, coffee,tea,soup}

dub

coffee

kwart

tea

S1 S2

S3

S0s

S4

S5

τ



dub

coffee

dub

tea

S1 S2

S3 S4

S0 traces (s) = { σ ∈ L* | s σ }

s after σ = { s’ | s σ s’ }

s Sequences of observable actions:

Reachable states:

traces(s) = { ε, dub, dub coffee, dub tea }

s after dub = { S1, S2 }

s after dub tea = { S4 }


Input-Output Transition Systems

dub

coffee

kwart

tea

S1 S2

S3 S4

S0

LI = { ?dub, ?kwart }

LU = { !coffee, !tea }

dub, kwart coffee, tea

from user to machine from machine to user initiative with user initiative with machinemachine cannot refuse user cannot refuse

input output LI LU

LI ∩ LU = ∅ LI ∪ LU = L

!

??

!



LI = { ?dub, ?kwart }

LU = { !coffee, !tea }

Input-Output Transition Systems IOTS (LI ,,LU ) ⊆ LTS (LI ∪ LU )

IOTS is LTS with Input-Output and always enabled inputs:

for all states s,for all inputs ?a ∈ LI :

?dub ?kwart

?dub ?kwart

?dub ?kwart

?dub ?kwart

?dub

!coffee

?kwart

!tea

S ?a


?kwart

?dub ?kwart

?kwart

?dub ?kwart

?dub ?kwart

?kwart ?dub

?dub ?kwart

?kwart ?dub

?kwart

?kwart ?dub

?dub ?kwart

?dub ?kwart


!coffee

?dub

!tea !coffee

?dub

?dub

!coffee

?dub

!tea


Formal Testing with Transition Systems

exec : TESTS × IMPS → ℘(OBS)


Ts ⊆ TTS

s ∈ LTS

IUT ∈IMPS

ioco

iIUT ∈IOTS

passes : IOTS × TTS →

{pass,fail}

Proof soundness and exhaustiveness: ∀i∈IOTS .

( ∀t∈gen(s) . i passes t )

⇔ i ioco s

Test hypothesis :

∀IUT∈IMP . ∃iIUT ∈IOTS .

∀t∈TTS . IUT passes t ⇔ iIUT passes t

pass / fail

=


Labelled Transition System Testing

FSPECS → LTS ( LI ∪ LU ) ⊆ LTS

FMODS → IOTS (LI , LU ) ⊆ LTS

FTESTS → TTS ( LU , LI ) ⊆ LTS

FOBS → traces Fobs → t || i Fder → der : LTS → ℘ ( LTS )

FWhich imp ? ♦ (strong, weak, branching, ... ) bisimulation ♦ trace-, testing-, refusal - preorder / equivalence ♦ conf, conf*, aconf,

ioco

A conformance relation


i ioco s =def ∀σ ∈ Straces (s) : out (i after σ) ⊆ out (s after σ)

Implementation Relation iocoCorrectness expressed by implementation relation ioco:

Intuition:

i ioco-conforms to s, iff

• if i produces output x after trace σ, then s can produce x after σ • if i cannot produce any output after trace σ, then s cannot produce any output after σ ( quiescence δ )



Implementation Relation iocoCorrectness expressed by implementation relation ioco:

p after σ = { p’ | p σ p’ }

p δ p = ∀ !x ∈ LU ∪{τ} . p !x

out ( P ) = { !x ∈ LU | p !x , p∈P } ∪ { δ | p δ p, p∈P }

Straces ( s ) = { σ ∈ (L∪{δ})* | s σ }


δ

δ

Implementation Relation ioco

out ( i after ε ) =

out ( i after ?dub ) =

out ( i after ?dub.?dub ) =

out ( i after ?dub.!coffee) =

out ( i after ?kwart ) =

out ( i after !coffee ) =

out ( i after ?dub.!tea ) =

out ( i after δ ) =

!coffee

?dub

?dub

?kwart

?dub ?kwart

i ?kwart

{ δ }

{ !coffee }

{ !coffee }

{ δ }

{ δ }

∅ ∅

{ δ }



!coffee

?dub

?dub

?dub

i

!coffee

?dub

s

out (i after ε) = { δ }

out (i after ?dub) = { !coffee }

out (i after ?dub.!coffee)= { δ }

out (s after ε) = { δ }

out (s after ?dub) = { !coffee }

out (s after ?dub.!coffee) = { δ }

ioco

Implementation Relation iocoi ioco s =def ∀σ ∈ Straces (s) : out (i after σ) ⊆ out (s after σ)


!coffee

?dub

?dub

?dub

i

!coffee

?dub

s

!tea

out (i after ?dub) = { !coffee } out (s after ?dub) = { !coffee, !tea }

ioco



!coffee

?dub

s

?dub

?dub

!coffee

?dub

i

!tea

?dub

out (i after ?dub) = { !coffee, !tea } out (s after ?dub) = { !coffee}⊄

ioco



out (i after ?dub) = { !coffee, !tea } out (s after ?dub) = { !coffee, !tea}

ioco?dub

?dub

?dub

!coffee

?dubi

!tea

?dub

?dub


!coffee

?dub

s

!tea


ioco

?dub

?dub ?kwart

!coffee

?kwarti

!tea !coffee

?dub

s


out (i after ?kwart) = { !tea }

out (s after ?dub) = { !coffee } out (s after ?kwart) = ∅

But ?kwart ∉ Straces ( s )




?dub

?dub ?kwart

!coffee

?kwarti

!tea

ioco

s?dub

!coffee

?kwart

!tea


out (i after ?kwart) = { !tea }

out (s after ?dub) = { !coffee } out (s after ?kwart) = { !tea }


!coffee

?dub

i?kwart

?dub

?kwart

?dub

?kwart

out (s after ?kwart) = { !tea }out (i after ?kwart) = { δ }

ioco


s?dub

!coffee

?kwart

!tea


out (s after ?dub) = { !coffee }out (i after ?dub) = { δ, !coffee }

ioco

?dub

?dub

?dub

!coffee

?dubi

?dub

!coffee

?dub

s



out (s after ?dub) = { δ, !coffee }out (i after ?dub) = { δ, !coffee }

ioco

!coffee

?dub

τ

s


?dub

?dub

?dub

!coffee

?dubi

?dub


out (i after ?dub.?dub) = out (s after ?dub.?dub) = { !tea, !coffee }

i ioco s


i

?dub

?dub

?dub ?dub

!tea

?dub

?dub

!coffee

?dub

s

!coffee

?dub

?dub

?dub ?dub

!tea

?dub

?dub

?dub

?dub

!tea

s ioco i

out (i after ?dub.δ.?dub) = { !coffee } ≠ out (s after ?dub.δ.?dub) = { !tea, !coffee }


?dub

!choc

?kwart

!tea

!coffee

?dub?kwart

?dub ?kwart

?dub ?kwart

!choc

?dub

!tea

ioco ioco

Implementation Relation ioco

!coffee

?dub

!tea

s

iocoioco

δ?dub

!coffee

?dub

Test cases


Test Cases

♦ labels in L ∪ { θ } ♦ tree-structured ♦ finite, deterministic ♦ final states pass and fail ♦ from each state ≠ pass, fail :

• either one input !a • or all outputs ?x and θ

?coffee

!dub

!kwart

?tea

?coffee?tea

θ

!dub

θ

pass

failfail

failpass

Test case t ∈ TTS

TTS - Test Transition System :


Test Cases

?coffee

!dub

!kwart

?tea

?coffee?tea

θ

!dub

θ

pass

failfail

failpass

test case t

!dub

!kwart ; Start timer1

?tea fail

?timer1 fail

?coffee

!dub ; Start timer2

?tea pass

?timer2 pass

?coffee fail


Observations and Verdicts

OBS → ( L ∪ { θ } )*

obs → obs : TTS × IOTS → ℘ ( ( L ∪ { θ } )* )

obs (t,i) = { σ ∈ ( L ∪ { θ } )* | t ⎤⎥ i σ pass ⎤⎥ i' or

t ⎤⎥ i σ fail ⎤⎥ i' }

νt → νt : ℘ ( ( L ∪ { θ } )* ) → { fail, pass }

νt (O) = pass if ∀ σ ∈ O . t σ pass

= fail otherwise


Test Generation

Look for test derivation algorithm der : LTS → ℘(TTS)

such that der is sound and exhaustive:

∀ i ∈ IOTS . ( ∀ t ∈ der ( s ) . νt ( obs ( t, i ) ) = pass ) ⇔ i ioco s


obs (t,i) = { σ ∈ (L∪{θ})* | t ⎤⎥ i σ pass ⎤⎥ i' or

t ⎤⎥ i σ fail ⎤⎥ i' }

νt (O) = pass if ∀ σ ∈ O . t σ pass

= fail otherwise


Test Generation


out (s after σ) = { !x, !y }

s

!x

σ

!y

i

!x

σ

!z

out (i after σ) = { !x, !z }

out (test after σ) = LU

pass fail

test

?x

σ

?y

?z

pass


Test Generation


δ

out (s after σ) = { !x, !y, δ }

σ

s

!x !yδ

i

!x

σ

!z

out (i after σ)

= { !x, !z, δ }

out (test after σ)

= LU ∪ { θ }

θ

passpass fail

test

?x

σ

?y ?z

pass


Algorithm

To generate a test case t(S) from a transition system specification with S, with S set of states ( initially S = s0 after ε )

1 end test case pass

Apply the following steps recursively, non-deterministically:

2 supply input

!a

t ( S after ?a ≠ ∅ )

Test Generation Algorithm

allowed outputs or δ: !x ∈ out ( S )

forbidden outputs or δ: !y ∉ out ( S )

3 observe output

fail

t ( S after !x )

fail

allowed outputsforbidden outputs?y θ ?x


?coffee

failpass

θ ?tea?choc

failfail

Test Generation Example

test

?coffee

failfail

θ ?tea?choc

pass

!dub

s

!tea

?dub

!coffee


?coffee θ ?tea

passfail fail

?coffee

passfail

θ ?tea

δ

δ

δ


s?dub

!coffee

?dubtest

!dub


Test Execution Example

Two test runs :

t ⎤⎥ i dub tea pass ⎤⎥ i'

fail ⎤⎥ i''t ⎤⎥ i dub choc i fails t

!choc

?dub

!tea

i

?coffee

failpass

θ ?tea?choc

failfail

test

?coffee

failfail

θ ?tea?choc

pass

!dub

i' i''



Two test runs :

t ⎤⎥ i dub θ pass ⎤⎥ i'

pass ⎤⎥ i't ⎤⎥ i dub coffee θ

i?dub

!coffee

?dub

?dub

?dub

i passes t

?coffee θ ?tea

passfail fail

?coffee

passfail

θ ?tea

test!dub


test

?-2?2

pass pass

otherwise

failTo cope with non-deterministic behaviour, tests are not linear traces, but trees


Equation solver for y2=x

specification

? x (x >= 0)

! √x

? x (x < 0)

! -√x

!4pass

?-3otherwise?3

fail

!9


?-3

pass

otherwise?3

fail

?-2?2

pass pass

otherwise

fail

? x (x >= 0)

! √x

? x (x < 0)

! -√x

? y

implementation test⎤ ⎥

!9

!4



Validity of Test GenerationFor every test t generated with algorithm we have:

FSoundness :t will never fail with correct implementation i ioco s implies i passes t

FExhaustiveness :each incorrect implementation can be detectedwith a generated test t i ioco s implies ∃ t : i fails t


Validity of Test Generation

Theorem:

FSoundness : i ioco s implies ∀ t ∈ der ( s ) . νt ( obs ( t, i ) ) = pass

FExhaustiveness : ∀ t ∈ der ( s ) . νt ( obs ( t, i ) ) = pass implies i ioco s


?coffee θ ?tea

passfail fail

passfailδ

δ

δs?dub

!coffee

?dubtest

Soundness and Exhaustiveness

///fail

!dub

Still sound !Not sound anymore !

///pass

?coffee θ ?tea

software specification 2ix20jschmalt/teaching/2ix20/2ix20_lecture11... · 2017-03-13 · software...

Documents