a coverage analysis for safety property lists
DESCRIPTION
A Coverage Analysis for Safety Property Lists. Koen Lindström Claessen Chalmers University Gothenburg, Sweden. FMCAD 2007, Austin, Texas. Property-based Verification. Properties. Design. ?. Dynamic Verification. Checkers/ Monitors. PSL/SVA properties. Properties. Design. stimuli. - PowerPoint PPT PresentationTRANSCRIPT
A Coverage Analysis for Safety Property ListsKoen Lindström ClaessenChalmers UniversityGothenburg, Sweden
FMCAD 2007, Austin, Texas
Property-based Verification
Design Properties?
Dynamic Verification
Design Properties
stimuli ? ? ?
Checkers/Monitors PSL/SVA
properties
Simulation Coverage
Design Properties
stimuli
Statement coverage
Gate coverage
State space coverage
Path coverage
Register coverage
95%
75%
98%
66%
83%
In Formal Verification:
100%
How About the Properties?
Design1. A => B
2. B & C => next A3. not (A and C)4. next A => B
?
Removing / adding one of these does not
affect coverage…
Property Coverage
We need coverage for propertiesHow much is the design constrained by
properties?Do the properties express what we want to
say?Are any properties missing?(lots of work done on this)
The more the merrier!
!
Mutation Coverage
Design PropertiesDesign
?
“Point not covered”
Mutation Coverage Expensive
Many (failing) calls to model checker Involves the whole design
SolutionsSymbolically; still expensiveApproximations
Look at proof Subset of mutations
[Chockler, Kupferman,Vardi]
This Talk
A property coverage analysis Independent of the design
Finds “forgotten cases” …… which indicate forgotten properties
Relatively cheapOnly involves properties Independent of design size
NaNNot a quantitative analysis
Design does not have to exist
There may exist multiple designs
Design may change after
analysis
Forgotten Case Analysis
Design1. prop12. prop23. prop34. prop4
“Forgotten Case”?
Properties specify a function : In Out?Too strictMeans complete specificationSynthesizable
BalanceStrict enough: to catch forgotten casesLoose enough: to be practically usable
Choice: Forgotten Case
in1 1 0 1 0 0 0 1 1 1 0 0 …
in2 1 0 0 0 1 1 0 0 1 1 0 …
in3 0 0 0 1 1 0 0 0 0 1 1 …
out1 1 1 0 0 0 1 1 0 1 1 0 …
out2 0 1 1 1 0 0 0 0 1 0 1 …
PropertiesOK
0
Properties still OK
In this trace:out1 is not constrained by properties at time 6:
Forgotten case!
Example: a FIFO
FIFO
in
put
get fst
num
err
put overrides
get
fst always outputs the first element
err signals for 1 clock cycle when something goes wrong – the FIFO does not break
A First Property List
•always (put=1 & num=n next err=1)•always (put=1 & num<n next num=num+1)•always (put=1 & num=0 next fst=in)•always (put=1 & 0<num<n next fst=fst)•always (get=1 & put=0 & num=0 next err=1)•always (get=1 & put=0 & 0<num next num=num-1)
Safety properties
No restriction on the logic…
n = max. FIFO size
Analyzing err
get 0 …
put 0 …
in 0 …
num 0 …
fst 0 …
err ? …
Not constrained at time 0
Analyzing err: A Fix
•err=0
Holds at initial point in time
Add:
Analyzing err
get 0 0 …
put 0 0 …
in 0 0 …
num 0 0 …
fst 0 0 …
err 0 ? …
Nothing goes wrong…
Analyzing err: A fix
•always (get=0 & put=0 next err=0)
Add:
Analyzing err
get 0 0 …
put 1 0 …
in 1 0 …
num 0 1 …
fst 0 1 …
err 0 ? …
Nothing goes wrong…
Analyzing err: A fix
•always (put=1 & num<n next num=num+1)•always (get=1 & put=0 & 0<num next num=num-1)
•always (put=1 & num<n next num=num+1 & next err=0)
•always (get=1 & put=0 & 0<num next num=num-1 & next err=0)
Change to:
Analysis does not complain about err
anymore…
Analyzing num
get 0 …
put 0 …
in 0 …
num ? …
fst 0 …
err 0 …
Not constrained at time 0
Analyzing num: A Fix
•num=0
Holds at initial point in time
Add:
Analyzing num
get 0 0 …
put 0 0 …
in 0 0 …
num 0 ? …
fst 0 0 …
err 0 0 …Not
constrained
Analyzing num: A fix
•always (get=0 & put=0 next num=num)
Add:
Analyzing num
get 1 0 …
put 0 0 …
in 0 0 …
num 0 ? …
fst 0 0 …
err 0 1 …
An error has occurred
Analyzing num: A fix
•always (get=0 & put=0 next num=num)
Change to:
•always ((get=0 & put=0) v next err=1 next num=num)
The analysis is now happy about
num…
Dependencies between outputs…
Analyzing fst
get 0 …
put 0 …
in 0 …
num 0 …
fst ? …
err 0 …
Not restricted
But we don’t want to specify it!
Analyzing fst: A fix
Add:
•always (num=0 free fst)
A new keyword:free
Logically vacuous: “fst=fst”
For the analysis:free x means that x
is constrained: “don’t complain!”
Analyzing fst
get 0 0 1 0 …
put 1 1 0 0 …
in 3 5 0 0 …
num 0 1 2 1 …
fst 0 3 3 ? …
err 0 0 0 0 …Not restricted:
FIFO behavior not specified in properties
Unconstrained Outputs
Three causes: It is supposed to be unspecifiedWe decide not to specify it (complicated)We have forgotten to specify it
Cases we want to catch
Cases we have to be explicit
about
Analyzing fst: A fix
•always (get=1 & put=0 & 0<num next num=num-1 & next err=0)
Change to:
•always (get=1 & put=0 & 0<num next num=num-1 & next err=0 & next free fst)
Analyzing fst
get 0 0 …
put 0 0 …
in 0 0 …
num 0 0 …
fst 0 ? …
err 0 0 … Real forgotten case
Analyzing fst: A fix
•always ((get=0 & put=0) v next err=1 next num=num)
Change to:
•always ((get=0 & put=0) v next err=1 next num=num & next fst = fst)
The analysis is now happy about
fst…
Luckily, we used free…
New & changed properties
•err=0 & num=0
•always (get=0 & put=0 next err=0)
•always (put=1 & num<n next num=num+1 & next err=0)
•always (get=1 & put=0 & 0<num next num=num-1 & next err=0 & next free fst)
•always ((get=0 & put=0) v next err=1 next num=num & next fst = fst)
initial values
implicit behavio
rconstan
t behavio
r
Implementation For output s, find a trace where
Props(s) is OKProps(s’) is OKExists exactly one t where s[t]≠s’[t]free(s)[t] should be false
(this trace must be infinite)
Implementation (I)
Find a trace satisfying:Props(s) & Props(s’) & ◊!(s≠s’)
With the use of free:Props(s,free_s) & Props(s’,free_s) & ◊!
(~free_s & s≠s’)
Using a standard LTL model
checker (NuSMV)
Implementation (II)
Property observer for safety property phi:□OK holds iff. phi holds
Analysis for property observers:□OK(s) & □OK(s’) ◊!(s≠s’)
Discussion (I)
Forgotten caseGiven an output signal s and a time t, and
given the values of all other signals at all points in time, and given all values of s not at time t, do the properties force the value of s at time t?
Alternatives…
Discussion (II)
Freeness“free s” does not indicate that s can take on
any valuerather, it is an artifical way of constraining s
for the sake of the analysis, without actually restricting it logically
Conclusion: This Analysis Identifies forgotten cases
Which inspire forgotten properties Forces to specify when outputs are free
Distinction between forgotten cases and underconstrainedness
Is design-independentpre-design / multiple implementationscost