1 jan tretmans university of nijmegen [email protected] © jan tretmans university of nijmegen...

1

Jan TretmansUniversity of Nijmegen

[email protected]

© Jan Tretmans University of Nijmegen

Model Based Testing

Property Checking for Real

2 © Jan Tretmans University of Nijmegen

Overview

Introduction Testing

Formal methods and Testing

Formal Testing Framework

Pre/post-condition program testing

Transition system testing ioco test theory

a test tool

an application: "Rekeningrijden"

Conclusions


Overview






a test tool


Conclusions


Testing

Testing:

to check the quality of an objectby performing experimentsin a controlled wayw.r.t. a specification

Software Testing :

testing the quality of a software product

mn

n+

mtester

IUT


Paradox of Software Testing

Testing is:

important

much practiced

30% - 50% of project effort

expensive

time critical

not constructive(but sadistic?)

But also:

ad-hoc, manual, error-prone

hardly theory / research

no attention in curricula

not cool :“if you’re a bad programmer you might be a tester”

Attitude is changing: more awareness more professional

Improvements possible

with formal methods !?


Types of Testing

unit

integration

system

performancerobustness

functional behaviour

white box

black box

Level of detail

Accessibility

Characteristics

usability

reliability

module

stress


Testing

IUT

specification

property

IUTcorrect w.r.t.specification

tester

pass fail

test cases

scenarios


Overview






a test tool


Conclusions


Formal methods: proving properties research sound theories “clean”

Testing : trial & error practice heuristics “dirty hands”

Testing & Formal MethodsA Perfect Couple ?

“Testing is not necessary after formal verification”

“Testing can only detect the presence of errors, not their absence”

“Formal methods are toys for boys”

“Formal methods are only for toy problems"


Formal Testing

Why testing with a formal specification:

improves the testing process

precise and unambiguous basis for testing

allows automatic generation of tests

Why no formal verification (theorem proving, model checking, …) :

code/structure of system not accessible (black-box)

system too complex

verification only on model of implementation

prove evidence to customer/user

no formalization possible


Development Process

informalrequirements

specification

realization

design

code

formalizable


Goal:

Testing functional behaviour

of black-box implementation

with respect to specification

in formal language

based on formal definition

of conformance

Specification BasedFunctional Testing

with Formal Methods

implementationunder test

formal testing

Model assumed to be

correct

specifications


Overview






a test tool


Conclusions


test

executionpass / fail

Formal Testing

test

generation

test suite TS

specification S

implementation i

correctness criterion

implementation

relation

i passes Ts

i ioco s

soundexhaustiv

e


Formal Testing : Conformance

s SPECS SpecificationIUT Implementation under Test

IUT is concrete, physical object

Model the physical world

But IUT is black box ! ?

Assume that model iIUT exists

specification S

implementation IUT

correctness criterion

IUT correct wrt s


Formal Testing : Test Hypothesis

Test hypothesis :

IUT IMPS . iIUT MODS .

t TESTS . obs ( t, IUT ) = obs ( t, iIUT )

IUT iIUT

test t test t


Formal Testing : Conformance

specification S

implementation iIUT

formalcorrectness

criterioniIUT imp s

s SPECS Specification

iIUT MODS model of IUT

Test hypothesis : each concrete IUT can be modelled

by some iIUT MODS

Correctness : iIUT imp s

iIUT is not known ;

testing to learn about iIUT


Completeness

exhaustive

sound

Completeness of test suite Ts on models:

i MODS .

t Ts . obs ( t, i ) = pass i imp s


Testing for Conformance

IUT passes Ts def t Ts . IUT passes t

IUT passes t def obs ( t, IUT ) = pass

Test hypothesis :

t TESTS . obs ( t, IUT ) = obs ( t, iIUT )Proof obligation :

i MODS .

( t Ts . obs ( t, i ) = pass ) i imp s

IUT passes Ts IUT correct wrt s ?

Definition : IUT correct wrt sIUT correct wrt s

iIUT imp s

t Ts . obs ( t, iIUT ) = pass

t Ts . obs ( t, IUT ) = pass

t Ts . IUT passes t

IUT passes Ts


Formal Testing

exec : TESTS IMPS (OBS)

der : SPECS (TESTS)

Ts TESTS

s SPECS

IUT IMPS

imp

iIUT MODS

obs : TESTS MODS (OBS)

Proof soundness and exhaustivess:iMODS .

tTs . obs(t,i) = pass i imp s

Test hypothesis :

IUTIMPS . iIUT MODS .

tTESTS . obs(t,IUT) = obs(t,iIUT)

pass / fail

Then:

IUT is correct iff it passes the test


Approaches to Formal TestingInstantiations of Formal

Framework

Programs as functions (pre/post-conditions)

Finite State Machine / methods with state

Labelled Transition Systems

Abstract Data Type testing


Overview






a test tool


Conclusions


Simple Input/Output Programs

Specification: s X Y

Test hypothesis: implementation i :: X Y

Tests: T X

Passing a test t T : ( t, i(t) ) s

Tools: QuickCheck, GST

Problem/challenge: select "good" subset T X:classical techniques: equivalence partitioning, boundary analysis,

……

IUTx: X y: Y


Input/Output Program: Example

Specification:

propSqrt(x,y) = x 0 |y y - x| 0.00001

Possible test set: T = { 0, 0.00232, 1, 1.87392, 2738894 }

Tools like QuickCheck, GST easily generate thousands

of test cases

But still: what is a "good" set ?

IUT

i(x) = x

x: real

pre: x 0

y: real

post: |y y - x| 0.00001


Overview






a test tool


Conclusions


Formal Testing with Transition Systems

exec : TESTS IMPS (OBS)

der : LTS (TTS)

Ts TTS

s LTS

IUT IMPS

ioco

iIUT IOTS

obs : TTS IOTS (traces)

Proof soundness and exhaustivess:

iIOTS .

( tder(s) . t(obs(t,i)) = pass )

i ioco s

Test hypothesis :

IUTIMPS . iIUT IOTS .

tTTS . exec(t,IUT) = obs(t,iIUT)

pass / fail


Labelled Transition Systems

Labelled Transition System S, L, T, s0

?coin

?button

!alarm ?button

!coffee

states

actions transitionsT S (L{}) S

initial states0 S


i ioco s =def Straces (s) : out (i after ) out (s after )

Implementation Relation ioco

Correctness expressed by implementation relation ioco:

p p = !x LU {} . p !x

out ( P ) = { !x LU | p !x , pP } { | p p, pP }

Straces ( s ) = { (L{})* | s }p after = { p’ | p p’ }


i ioco s =def Straces (s) : out (i after ) out (s after )


Correctness expressed by implementation relation ioco:

Intuition:

i ioco-conforms to s, iff

• if i produces output x after trace , then s can produce x after

• if i cannot produce any output after trace , then s cannot produce any output after ( quiescence )


?dub

!choc

?kwart

!tea

!coffee

?dub?kwart

?dub?kwart

?dub?kwart !choc

?dub

!tea

ioco ioco


!coffee

?dub

!tea

s

ioco


Algorithm

To generate a test case from transition system specification s0

compute T(S), with S a set of states, and initially S = s0 after ;

1 end test case

pass

For T(S), apply the following recursively, non-deterministically:

2 supply input

!a

T( S after ?a )

Test Generation Algorithm

allowed outputs or : !x out ( S

)

forbidden outputs or : !y out ( S )

3 observe output

fail

T ( S after !x )

fail

allowed outputsforbidden outputs ?y

?x


Completeness of Test Generation

For every test t generated with algorithm we have:

Soundness :t will never fail with correct implementation

i ioco s implies i passes t

Exhaustiveness :each incorrect implementation can be detectedwith a generated test t

i ioco s implies t : i fails t


Overview






a test tool


Conclusions


A Test Tool : TorX

On-the-fly test generation and test execution

Implementation relation: ioco

Specification languages: LOTOS, Promela, FSP, Automata

TorX IUTobserveoutput

offerinput

nextinput

specification check

output

passfailinconclusive

user:manualautomatic


TorX


Overview






a test tool


Conclusions


TorX Case Studies

Conference Protocol

EasyLink TV-VCR protocol

Cell Broadcast Centre component

‘’Rekeningrijden’’ Payment Box protocol

V5.1 Access Network protocol

Easy Mail Melder

FTP Client

“Oosterschelde” storm surge barrier-

control

academi

c

Philips

CMG

Interpay

Lucent

CMG

academi

c

CMG


Interpay ‘’Rekeningrijden’’Payment Box Protocol


“Rekeningrijden”

Characteristics :

Simple protocol

Parallellism :

many cars at the same time

Encryption

Real-time issues

System passed traditional testing

phase


PaymentBox

(PB)Road SideEquipment

OnboardUnit

UDP/IPWireless

‘’Rekeningrijden’’ Highway Tolling System


spec

PB

TorX PaymentBox

‘’Rekeningrijden’’: Test Architecture

PCO


spec

PB+

ObuSim+

TCP/IP+

UDP/IP

PaymentBox

TorX

‘’Rekeningrijden’’: Test Architecture

SUT

Test Context

ObuSim

TCP/IP UDP/IP IAPPCO


‘’Rekeningrijden’’: Issues

Parallellism :

very easy

Encryption :

Not all events can be synthesized :

Leads to reduced testing power

Real-time :

How to cope with real time constraints ?

Efficient computation for on-the-fly testing ?

Lack of theory: quiescence vs. time-out


‘’Rekeningrijden” : Results

Test results :

1 error during validation (design error)

1 error during testing (coding error)

Automated testing :

beneficial: high volume and reliability

many and long tests executed ( > 50,000 test

events )

very flexible: adaptation and many configurations


Overview






a test tool


Conclusions


Concluding ……

Testing can be formal, too (M.-C. Gaudel, TACAS'95)

Testing shall be formal, too

But testing can never be complete

it increases confidence in correctness

Testing can be the first step:

counter-example finding before expensive formal verification

Or the last:

to show that not only the model is correct

but also the real system


Thanks to ...

University of Twente:Ed Brinksma, Lex Heerink, Axel Belinfante,Jan Feenstra, René de Vries, Machiel van der Bijl, ……

Eindhoven University of Technology:Nicolae Goga, Loe Feijs, Sjouke Mauw

University of Nijmegen: Pieter Koopman

Philips Research Labs Eindhoven: Lex Heerink, Ron Koymans

KPN Research: Erik Kwast, Henri Dol

Lucent Technologies - R&D Centre Twente: Arjan de Heer

Project Pampa, IRISA, Rennes

(Logica)CMG Information Technology:Peter Christian, Robert Spee, Wouter Geurts, ……

Interpay B.V.: Cornel van Mastrigt, Rommert Jorritsma

Ordina Utopics: Pim Kars

………

1 jan tretmans university of nijmegen [email protected] © jan tretmans university of nijmegen...

Documents

formal verification

specification software

formal specification

testing process precise

formal language

boys formal methods

absence formal methods

formal verification