the search for the perfect program
TRANSCRIPT
THE SEARCH FOR THE PERFECT PROGRAM
NATALLIE BAIKEVICH
@LU_A_JALLA
A ‘PERFECT PROGRAM’ IS …
- Maintainable
- Efficient
- Correct
A ‘PERFECT PROGRAM’ IS …
- Maintainable
- Efficient
- CorrectDoes anything else really matter without correctness?
THE PLAN- Assertions
- Random Testing
- Symbolic Execution
- Types
THE PLAN- Assertions
- Random Testing
- Symbolic Execution
- Types
1.Do something
2.Check it
3.Go to 1.
THE FAST TRACK
Write code -> Release
WELL, NOT THAT FAST
Dear <…>
…
We see XYZException!!!.............!
where n = enough for you to feel miserable
n times
OK, LET’S TEST IT
1. Choose your favourite library
(good chances it’s .*Unit)
2. Write some tests
aand
3. XYZException has gone
THE PROBLEM, PART 1
But…
- they only check what you think a program should do, not what it actually should do
- ... on what you think is all possible inputs
RANDOMIZE IT
Throw the thousands of inputs into your program with
- QuickCheck (Haskell)
- FsCheck (F#)
- ScalaCheck (Scala)
- or something else
DEMO
RANDOMIZE IT
FSCHECK- Expressive and
easy to use
- Checks your spec
- … in a large number
- … of randomly generated cases
“500 checks can’t be wrong!” © don’t remember who said that
THE PROBLEM, PART 2- Expressive and
easy to use
- Checks your spec
- … in a large number
- … of randomly generated cases
“500 checks can’t be wrong!” © don’t remember who said that
- Writing good tests might be tricky
- Checks your spec
- … in a large number
- … of randomly generated cases
500 checks can easily be wrong!
But:
DEMO
RANDOMIZE IT – WHAT’S MISSING?
I WILL BUILD MY OWN TEST GENERATOR
let f x y =
if x < 10 then x
else if x = 42 then
failwith “42”
else 1 / y
x < 10 or x >= 10
x = 42 or x <> 42
y = 0 or y <> 0
DEMO
QUOTATIONS
THE PROBLEM, PART 3
But it’s too complicated to do by hands:
- too many cases,
- lots of overlapping paths,
- difficult to solve when the number of variables grows,
- not everything is a quotation…
IDEA: USE A CONSTRAINT SOLVER
- Check out Z3 homepage
- Try Z3 in your browser
- LINQ to Z3 (ch9 video and a post by Bart de Smet)
Specially for fsharpers:
- Z3Fs on github
THERE’S A TOOL FOR THAT – MEET PEX
- Dynamic symbolic execution
- Analyses .NET instructions
- Uses constraint solver to find the inputs
PEX REFERENCES
- Project homepage
- Pex for fun in your browser
- Code Digger, addin for VS
- Code Hunt website
- Documentation, videos
and more
DEMO
PEX
TEST GENERATIONStatic:
- Conditional statements
- Check the formulas satisfiability
Dynamic:
- Collect the information during the program execution
- Unknown environments
- Enhanced values generation
- Better Performance
DYNAMIC EXECUTION EXAMPLE
int obscure(int x, int y) {
if (x == hash(y)) return -1; // error
return 0;
}
“Compositional Dynamic Test Generation”, Patrice Godefroid (paper)
THE PROBLEM, PART 4
Limitations:
- Nondeterministic cases (e.g. native code)
- Concurrency
- Constraint solver limitations
MORE AND LESS TYPES WITH F*
F* - an ML-like verification-oriented language
- F* project homepage
- GitHub repo
- Try F* in your browser
DEMO
TRYING F*
THE PROBLEM, PART 5
- currently under development
- issues on mono
- in more complex cases, the errors become quite cryptic
- you still need to come up with a way to define the requirements, at the type level
CONCLUSION
If you don’t know where you’re going, any road’ll take you there.
Lewis Caroll
QUESTIONS?