CS527Topics in Software Engineering(Software Testing and Analysis)

Darko Marinov

September 15, 2011

Schedule• First few lectures to help you select projects

– Shared memory: CHESS, IMUnit, CAPP– Message passing: Setac– Comment analysis: iComment– Regression testing: survey paper– Today: Model-based testing– Next week two papers

• 3 votes: generation/evaluation of test assertions• 2 votes: mock-based testing, GUI testing• 1 vote: genetic algorithms, concolic testing, web testing

• Your project can be something we don’t cover

Paper Today• Model-based

quality assurance of protocol documentation: tools and methodology Wolfgang Grieskamp, Nicolas Kicillof, Keith Stobie and Victor Braberman (STVR 2010)

• Journal paper– Not too long– Otherwise, we prefer conference papers

• Follows: personal slides on models for data, slides on MBT, questions

• Thanks to Wolfgang Grieskamp for MBT slides

Developed Two Kinds of Test Abstractions

• Declarative: describe what inputs look like– Two kinds of languages used

• Declarative language for properties of desired inputs[SOFTMC’01, ASE’01, FME’02, OOPSLA’02, SAT’03, MIT’03, J-ASE’04, SAT’05, LDTA’06, ALLOY’06]

– Properties written in the Alloy modeling language– Uses Alloy Analyzer for generation of test inputs

• Imperative language for properties of desired inputs[ISSTA’02, TR’03, MIT’04, ICSE-DEMO’07, STEP’07, FSE’07]

– Properties written in implementation language (Java, C#…)

– Developed Korat for generation of test inputs

• Imperative: describe how to generate inputs– More recent work on ASTGen [FSE’07]

Korat at Microsoft Research• Korat implemented in the AsmLT test tool

in Foundations of Software Engineering group– Predicates in Abstract state machine

Language (AsmL), not in Java or C#– GUI for setting finitization and manipulating

tests– Korat can be used stand-alone or to

generate inputs for method sequences– Some extensions

• (Controlled) non-exhaustive generation• Generation of complete tests from partial tests• Library for faster generation for common

datatypes

AsmLT/Korat at Microsoft• Used by testers in several product groups• Enabled finding numerous errors

– XML tools• XPath compiler (10 code errors, test-suite augmentation)• Serialization (3 code errors, changing spec)

– Web-service protocols• WS-Policy (13 code errors, 6 problems in informal spec)• WS-Routing (1 code error, 20 problems in informal spec)

– Others• SSLStream• MSN Authentication• …

• Errors found in– Important real-world applications– Code already well tested using best testing

practices

Some Comments from Microsoft• Positive comments on AsmLT and Korat

– “So far our stateless AsmL models are pretty successful.”

– “AsmL parameter generation tool is quite convenient and powerful.”

• Negative comments on AsmLT not Korat– “Most of our testers prefer to write as much

C# as possible.”– “Very difficult to debug AsmL.”

• Result: new tool for C# (.NET) SpecExplorer

Korat is

Korat

Questions for Discussion (1)• Are there any open source tools equivalent to

Spec Explorer? [KN]• How would this methodology change if

applied to something but protocols? [JT]• In the case where the number of internal unit-

tests that need to be tested, is it worth it at all to do MBT? [AY]– What's the right time to apply model-based

testing? [XW]• What other selection strategies exist but

transition coverage? [AK]

Questions for Discussion (2)• … possible to scale up MBT to make it work

on large-sized systems? [AD]• Does model based testing scale down to

small projects? [DeG]• NLP … to aid in … converting technical

documentation into testing reqs? [CM]– Is it possible to reverse engineer a spec from a

program or test suite? [SB]• test models generated automatically? [YL]• How to totally/partially automate slicing? [FS]

– How does the slicing affect the success? [SO]

Questions for Discussion (3)• integrate with regression testing? [KB]• generate a test that checks to see if an action

disallowed by the model program is in fact disallowed by the System Under Test? [DwG]

• Can Spec be used to generate proprieties for Chess? [SB]

• How are bugs in the model treated? [AK]• Is the modeling language enough to model

various types of applications? [SO]