verification of evolving software natasha sharygina joint work with sagar chaki and nishant sinha...

Verification of Evolving Software

Natasha Sharygina

Joint work with Sagar Chaki and Nishant Sinha

Carnegie Mellon University

Motivation

• Component-based Software– Software modules shipped by separate developers– Undergo several updates/bug-fixes during their lifecycle

• Component assembly verification– Necessary on update of any component– High verification costs of global properties

Contribution

Assembly A

Component CComponent C’

P ?

Contribution

• Automated substitutability check – Preserving all old behaviors– Allowing new behaviors

• Focus on components that have been modified

• Allow multiple upgrades simultaneously

Substitutability Check• Given old and new components Ci, C’i and assembly A

– For each i check if C’i is substitutable for Ci in A

• Two phases:– Containment check

• Ensures that behaviors of the old component are contained in its substitutable counterpart

– Compatibility check• Safety with respect to other components in assembly: all global

specifications are satisfied

• Approach:– Obtain a finite behavioral model of all components by abstraction: Labeled

Kripke Structures (LKS)– Use model checkers to solve both phases automatically

Abstraction into LKS

• Labeled Kripke Structures– <Q,,T,P,L>

• State-event traces – <p,,q,,…..>

• Composition semantics– Synchronize on shared actions

• Two types of abstraction:– Predicate abstraction (over-approximations: existential transitions)– Modified predicate abstraction (under-approximations: universal

transitions)

p

!q

q

!p

Abstraction into LKS

• Program Statements , States

• Control flow , Transitions

• Transitions depict observable behavior

• Automated

Predicate Abstraction into LKS

L1L1

lock = 0

if (x < y)

lock=1

x < y

void OSSemPend(…) {

L1: lock = 1; if (x < y) { L2: lock = 0; … } if (x >= y) { … L3: lock = 0; … } else { … }}

void OSSemPend(…) {

L1: lock = 1; if (x < y) { L2: lock = 0; … } if (x >= y) { … L3: lock = 0; … } else { … }}

L2L2

L3L3

if (x >= y)

lock = 0

x < y

Verification

L1L1

if (P)

L2L2

L3L3

unlock

unlock

lock

ModelModel

ERROR

unlock unlock

lock

lockif (!P)

SpecificationSpecification

Verification

L1L1

if (*)

L2L2

L3L3

unlock

unlock

lock

ModelModel

ERROR

unlock unlock

lock

lockif (*)

SpecificationSpecification

ComFoRT: Component Formal Verification Framework

VerificationYes

System OKAbstraction

Model

CounterexampleValid?

Components Assembly

Yes

No

Counterexample

CounterExampleGuidedAbstractionRefinement

AbstractionRefinement

ImprovedAbstractionGuidance

No

SpuriousCounterexample

Containment

Step 1. Construct LKSs M’ and M such that (C1) C µ M and (C2) M’ µ C’

Step 2. Verify if M µ M’• If yes then from (C1) and (C2) C µ C’ and we terminate• Else CE is produced

Step 3. Validate CE

1. Check if CE C. If yes proceed to next step. Otherwise refine M and repeat from Step 2.

2. Check if CE C’. If yes than CE C \ C’, Feedback CE and stop. Else refine M’ (by adding the valid CE) and repeat from Step 2.

Containment (contd.)

• In contrast to Refinement-based approaches

– Containment ensures that all behaviors of M are present in M’– However, containment allows new behaviors in M’ which could

be possibly erroneous

• Containment check is not sufficient since new components

– may not be safe with respect to other components– must satisfy the global behavioral specifications

L* learnerL* learner

Learning Regular languages: L*• Forms the basis of the compatibility check

• L* proposed by D. Angluin

• Polynomial in the number of states and length of counterexample

Minimally adequate Teacher

Minimally adequate Teacher

IsMember( trace )

IsCandidate( DFA D )

a

b

a

b

UnknownRegular Language

±Counterexample/ Yes

Model checker

Yes/No

Minimum DFA

Compatibility check

• Recursive Assume-guarantee to verify assembly properties

• Generate a (smaller) environment assumption A– A: most general environment so that P holds

– Constructed iteratively using L* and R1, R2

– Recursion for the R2 rule

R1: M1 || A ² PR2: M2 || … || Mn ² A

M1 || … || Mn ² P

Compatibility check

R1: M1 || Ai ² P

R2: M2 || … || Mn ² Ai

trueL* Assumption

GenerationAi

true true

CE CE Analysis True CE

False CE

-CE for Ai

+CE for Ai

Compatibility check (contd.)• Generate a most general assumption for each M’

– M1 = M’

– M2 = Mn M (all other component LKSs)

• Membership queries:– M’ || CE µ P

• Candidate queries:– M’ || A µ P

– M2 µ A

Compatibility check (contd.)• CE analysis: M’ || CE µ P

– Yes ) False CE– No ) True CE

• Compatibility check infers

– Either M’ is substitutable– Or counterexample CE

• CE may be spurious wrt. C’ due to the predicate abstraction

– CE is present in component LKS M’– Must refine M’– Repeat substitutability check

Compatibility check (contd.)• Dynamic compatibility check

– Restarts learning using information from a previously learned candidate

– Account for incremental changes between successive assumptions

The dynamic nature is critical for 1) checking evolving software; 2) during abstraction refinement where a single component is updated at a time.

Feedback to Developers

Feedback - collection of CEs C identified during containment check

Feedback Delivery: Given a trace Feedback, and Rep() – representation of in terms of program control locations, predicate valuations and actions, the levels of feedback are

1) Rep(Prefix()) – to denote the divergence point of

2) Rep(Suffix()) – to show what exact behavior (exact code) of C is missing in C’

3) Rep’(Pref()) – to denote location in Ci where changes are to be made

Experiments• Prototype implementation in ComFoRT framework

• ABB IPC Module– Deployed by a world leader in robotics engineering systems– 15000+ LOC

– modified WriteMessageToQueue component– checked for substitutability inside the assembly of four components

(read, write, queue, critical section)– Over 30 billion states after predicate abstraction

• Discovered synchronization bug– Process can incorrectly block while writing to a queue

Related Work• Learning Assumptions: Cobleigh. et. al.

– Do not consider state labeling of abstract models– Do not incorporate a CEGAR framework for AG

• Compatibility of Component Upgrades: Ernst et. al.– Do not consider temporal sequence of actions in generating

invariants

• Interface Automata: Henzinger et. al.– Do not have CEGAR, AG

ComFoRT framework:

www.cs.cmu.edu/~natalie/

Questions?