An EvaluationAn Evaluationof BLASTof BLAST

John GallagherJohn Gallagher

CS4117CS4117

OverviewOverview

BLAST incorporates new, fascinating and BLAST incorporates new, fascinating and complex technology. The engine and complex technology. The engine and external components are evolving external components are evolving monthly.monthly.BLAST can be used successfully on BLAST can be used successfully on carefully scoped (systems) programs.carefully scoped (systems) programs.BLAST exists to serve software model BLAST exists to serve software model checking research and make it tangible. It checking research and make it tangible. It is not a good developer tool.is not a good developer tool.

CriteriaCriteria

How easy is it to generate the input?How easy is it to generate the input?How easy is it to generate the output?How easy is it to generate the output?How accurate is the output?How accurate is the output?How precise is the output?How precise is the output?How broad are the applications of the tool?How broad are the applications of the tool?What are the alternatives?What are the alternatives?How much time or process does the tool add How much time or process does the tool add or remove from software development?or remove from software development?Opportunity CostOpportunity Cost

The Good – BLAST FrameworkThe Good – BLAST Framework

BLAST internals illustrate interesting BLAST internals illustrate interesting working examples of compiler technology:working examples of compiler technology:

int main() { int i, x, y, ctr; x = ctr; ctr = ctr + 1; y = ctr; if (x == i) { assert (y == i + 1); }}

:-)

The Good – BLAST FrameworkThe Good – BLAST Framework

Based on very recent research (earliest Based on very recent research (earliest work cited in corresponding paper was work cited in corresponding paper was from 2002).from 2002).Given good input, correct use of BLAST’s Given good input, correct use of BLAST’s myriad of runtime options, and good myriad of runtime options, and good specification, BLAST performs well.specification, BLAST performs well.Able to find real bugs that were fixed Able to find real bugs that were fixed based on counterexample information that based on counterexample information that no other tools were providing.no other tools were providing.

The Good – BLAST FrameworkThe Good – BLAST Framework

Running time figures Running time figures and number of real and number of real counterexample counterexample traces make BLAST traces make BLAST look good. They are look good. They are the result of a solid the result of a solid framework.framework.Favorable position of Favorable position of having little having little competition in terms competition in terms of results achievable.of results achievable.

Program

Lines pre-

processed

Time(mins)

Predicates Total Average

kbfiltr 12k 3 72 6.5

floppy 17k 25 240 7.7

diskprf 14k 13 140 10

cdaudio 18k 23 256 7.8

parport 61k 74 753 8.1

parclss 138k 77 382 7.2

Future – BLAST FrameworkFuture – BLAST Framework

Using interpolation in predicate discovery Using interpolation in predicate discovery was a step forward. This reduced the was a step forward. This reduced the number of abstract predicates to track.number of abstract predicates to track.Lazy Interpolation-Based model checking Lazy Interpolation-Based model checking integrates interpolants further into the engine. integrates interpolants further into the engine. It is being explored by BLAST Kenneth It is being explored by BLAST Kenneth McMillan and BLASTer Ranjit Jhala.McMillan and BLASTer Ranjit Jhala.Expression abstraction may yet be able to Expression abstraction may yet be able to represent more expressive C patterns. Basic represent more expressive C patterns. Basic user-mode program specifications cannot be user-mode program specifications cannot be verified.verified.

The Bad – BLAST UsabilityThe Bad – BLAST Usability

Not easy to install (Specific path Not easy to install (Specific path dependencies, purported Cygwin dependencies, purported Cygwin compatibility a fallacy)compatibility a fallacy)Not easy runNot easy run BLAST Query Language pre-processor yields BLAST Query Language pre-processor yields

bad output, contains no usage informationbad output, contains no usage information BLAST main executable has 232 lines of BLAST main executable has 232 lines of

usage information, some of which is usage information, some of which is completely broken, some of which has completely broken, some of which has significant impact on results.significant impact on results.

The Bad – BLAST UsabilityThe Bad – BLAST Usability

Output with debugging suppressed contains Output with debugging suppressed contains about .5% useful information. With about .5% useful information. With debugging added, that number is significantly debugging added, that number is significantly less. (Even a developer would be hard less. (Even a developer would be hard pressed to decipher over one-hundred pressed to decipher over one-hundred consecutive lines of “: is the zero DD”).consecutive lines of “: is the zero DD”).The tool’s specification weaver does not The tool’s specification weaver does not correctly remember source code lines, so correctly remember source code lines, so useful output requires extensive useful output requires extensive corresponding grep into the source.corresponding grep into the source.

BLAST – RecommendationsBLAST – Recommendations

Integrate BLAST into development Integrate BLAST into development process early.process early.

Take advantage of BLAST’s typically fast Take advantage of BLAST’s typically fast run times to perform verification as a post-run times to perform verification as a post-compile operation.compile operation.

Take note when BLAST output changes Take note when BLAST output changes dramatically, and find the corresponding dramatically, and find the corresponding change in source.change in source.

BLAST – RecommendationsBLAST – Recommendations

BQL (BLAST Query Language) is a good BQL (BLAST Query Language) is a good way to separate specification from way to separate specification from implementation and eliminate BLAST implementation and eliminate BLAST checks that could turn into useless runtime checks that could turn into useless runtime costscosts

Build tools around BQL specification Build tools around BQL specification weaver or improve it (ahh, the gift and the weaver or improve it (ahh, the gift and the curse of open source).curse of open source).

Questions?Questions?

ReferencesReferences

The information shown here is based by The information shown here is based by corresponding paper on the BLAST tool, corresponding paper on the BLAST tool, “Software Verification with BLAST”. A “Software Verification with BLAST”. A good list of references can be found there.good list of references can be found there.