On Bridging Simulation and Formal Verification

Eugene GoldbergCadence Research Labs (USA)

VMCAI-2008, San Francisco, USA

Summary

• Introduction – motivation– main idea

• Stable sets of points (SSPs) and proofs– checking if a set of points contains an SSP– simulation based proof system– extraction of sufficient test sets from proofs

• Generation of practical tests– test generation for design change/manufacturing

faults– theoretical and experimental justification of our

approach• Conclusion

Motivation

Simulation is still the main workhorse of verification. It works surprisingly well taking into account that only negligible part of the search space is sampled!!

Given a design, formal verification is to check if property holds at all points of the search space.

Simulation is to “informally” check if property holds by testing it for a (relatively small) set of points.

In this paper, we study this phenomenon by the example of the satisfiability problem. As an application, we consider testing combinational circuits.

Simulation in Terms of SAT

Simulation: Given a CNF formula F, find a set of points T s.t. if F(T) = 0, then F is (most likely) unsatisfiable.

Satisfiability problem (SAT): Given F, find an assignment p such that F(p)=1 or prove that F(all_space)=0.

In other words, find T such that

F(T)=0 implies F(all_space) = 0

Let F be a CNF formula (i.e. conjunction of disjunctions also called clauses).

Interpretation of Points as Tests

Let CNF formula FN(X,Y,z) specify N. Checking satisfiability of N reduces to checking the satisfiability of FN z.

Let N be a circuit. Let X,Y describe the input and internal variables of N. Let z describe the output variable of N.

….

NY

z

A point p is a complete assignment to the variables X Y {z}. The corresponding test t is the projection of p to X.

So p consists of a test t and a computation vector (y,z). Since p falsifies clauses of FN it is computation with a fault.X

Main Idea

We formulate simulation as a proof system.

In contrast to “regular” simulation, we use points not only for sampling search space but for derivation.

In this derivation we employ the machinery of Stable Sets of Points.

Instead of proving one “big” theorem (F implies an empty clause), we deduce a sequence of simpler lemmas (F implies a clause L).

Each derived clause is added to F.

Summary

• Introduction – motivation– main idea

• Stable sets of points (SSPs) and proofs– checking if a set of points contains an SSP– simulation based proof system– extraction of sufficient test sets from proofs

• Generation of practical tests– test generation for design change/manufacturing

faults– theoretical and experimental justification of our

approach• Conclusion

A Stable Set of Points (SSP)

Example of 1-neighborhood of point p w.r.t a clause C (written Nbhd(p,C)) such that C(p)=0.

C = x z, p=(x=0,y=0,z=0). Nbhd(p,C) = {p1,p2}, where p1=(x=1,y=0,z=0) and p2=(x=0,y=0,z=1).

A complete assignment a point.

Let F be a CNF formula and T be a set of points.

T is called an SSP if

p T, clause C of F such that Nbhd(p,C) T.

A CNF formula F is unsatisfiable iff there is a set of points that is an SSP.

Testing if a Set of Points Contains SSP

Let F be a CNF formula and T be a set of points.

To check if T is an SSP : For every point p of T, check if there is a clause C of F s.t. Nbhd(p,C) T.

This procedure is linear both in T and F.

Even if T is not an SSP, a subset T* of T may be an SSP. To check if T contains an SSP : For every point p of T, check if there is a clause C of F s.t. Nbhd(p,C) T.

If C does not exist, remove p from T and restart this procedure.

This procedure is quadratic in T and linear in F.

Simulation as a Proof System

We formulate simulation as a proof system with two derivation rules.

Rule 1: Generate a point pi and derive the value of F(pi).

Rule 2: Let T={p1,…,pk} be the current set of points. Derive a clause C such that F implies C and add it to F.

Derivation of C is valid only if T contains an SSP of the CNF formula F ~C (and so F C holds).

A Sufficient Test SetGiven a CNF formula F, a set of points T is called sufficient if there is a set of lemma clauses L1,…,Lk (Lk=) such that all k derivations

F L1,

F L1 L2 ,

….

F L1 … Lk-1 Lk

are valid (i.e. they can be proved by “simulation” using T)

In other words, T is a sufficient test set if the set of valid derivations is sufficient to prove that F is unsatisfiable by applying only the second rule (no new points are needed).

Extracting a Sufficient Test Set from a Resolution Proof

Let L={L1,..,Lk} be the set of resolvents of a resolution proof R that F is unsatisfiable. We assume that resolvents Li are numbered in their derivation order (so Lk = ).

Then there is always a sufficient test set whose size is 2k (proving that FL1 … Li-1 Li needs only two points of T).

Sufficient test sets can be very small even for very large formulas!!!!

“Proving” Resolvent by Building a Stable Set of Two Points

Let C be the resolvent of C and C. One can prove that C C C by building an SSP of only two points.

Example: Let C = ~x y z, C = x w. The resolvent C is y z w. To prove C C C it suffices to show the CNF formula C C ~C is unsatisfiable.

(Here ~C= ~y ~z ~w).

After making assignments y=0,z=0,w=0, the formula C C ~C turns into ~x x. Any two points p and p with y=0,z=0,w=0, that are different only in the value of variable x is an SSP for C C ~C .

Summary

• Introduction – motivation– main idea

• Stable sets of points (SSPs) and proofs– checking if a set of points contains an SSP– simulation based proof system– extraction of sufficient test sets from proofs

• Generation of practical tests– test generation for design change/manufacturing

faults– theoretical and experimental justification of our

approach• Conclusion

Applying Our Theory to (design change/manufacturing testing)

We cannot apply our theory directly to checking if F is unsatisfiable. (To prove that a set of points is sufficient we need a set of lemma clauses i.e. another proof)

However, there are numerous indirect ways to apply our theory. One application is to detect incorrect design changes/ manufacturing faults.

Let F be a CNF formula. Let R be a proof that F is unsatisfiable.

Find a set of points such that most likely detect satisfiable variations of F. (These variations describe tech. faults/design changes.) Such a test set can be extracted from a sufficient set of points specified by R.

Tight Sufficient Tests

Let T be a sufficient test set for F. Informally, T is tight if every point p T falsifies as few clauses as possible.

Given a resolution proof L1,..,Lk, a sufficient test set T is built as T1 … Tk. Here Ti is a two point SSP that proves that C C Li where C C are the parents of resolvent Li.

When building Ti one can arbitrarily assign variables that are not in C or C.

To build a tight sufficient test, free variables are assigned so as to minimize the number of falsified clauses.

Big Picture

Let F be an unsatisfiable formula. To detect satisfiable variations (“faults” in F) we generate a tight sufficient test set T. Such a set can be extracted, for example, from a resolution proof that F is unsatisfiable.

The sufficiency of T guarantees its “completeness”. The tightness of T increases the probability of detec-ting small satisfiable variations.

Suppose, for example, that a point p of T falsifies only one clause C of F. Then if F* is a variation of F consisting of disappearance of C (along with some other changes) it is likely that F*(p) = 1. Hence p detects that F* is satisfiable.

“Theoretical” Justification of Our Approach

Let N1 and N2 be identical copies of circuit N. Let Rnat be a “natural” resolution proof that their miter is unsatisfiable.

Let T={p1,..,pm} be a tight sufficient test set specified by Rnat. We show that the set inp(T)= {inp(p1),…,inp(pm)} detects all (testable) stuck-at faults of N. So, in a sense, our theory

“predicts” the high quality of the stuck-at fault model for circuit testing.

Experimental Justification

We experimentally compared quality of random test sets and test sets extracted from resolution proofs that two copies of a circuit were identical.

Resolution proofs were generated by a state-of-the-art SAT-solver. We used MCNC benchmark circuits.

Experiments show that tests extracted from resolution proofs significantly outperform random tests.

We applied these two kinds of test sets (of the same size) to detecting literal appearance faults. Such faults are more “subtle” than stuck-at faults and so are harder to detect.

Conclusion

We use SAT to show that there is a close relation between high quality test sets and formal proofs.

We show how high quality tests can be extracted from proofs of unsatisfiability represented as a sequence of lemma clauses.

Our approach answers (at least in principle) two important questions.

a) When does one stop simulation? (When the set of generated points “encrypts” a formal proof).

b) What is an “ideal” metric in functional verification? (A formal proof.)