automated test data generation maili markvardt. outline introduction test data generation problem...
TRANSCRIPT
Automated Test Data Generation
Maili Markvardt
Outline
Introduction Test data generation problem Black-box approach White-box approach
Introduction
Improtance of testing growing since mission-criticality of the software in our everyday life
Software errors are more costly than ever Testing can be automated
Test execution automation Test generation automation Test data generation automation
Problem: example
User inputs three sides of a triangle (a, b, c). Which type is the triangle?
Requirements: IF a<=0 || b<=0 || c<=0 -> input incorrect IF p*(p-a)*(p-b)*(p-c) < 0 -> sides not forming a
triangle IF a==b || a==c || b ==c -> isoceles Kui a==b & b==c -> eqilateral Other -> scalene
What strategy? -> what data?
Input validation automation
The Concept “side of a triangle” equivalence partitions and boundary values Normal: ]0; ∞[ Erroneous: ]- ∞; 0[, missing values Border values: {0}
For testing the Input validation functionality, pick a random value from each equivalence partition for each side: P(-1, 2, 3), P(1, -2, 3), P(1, 2, -3)
Same with boundary values P(0, 1, 2), P(1, 0, 2), P(1, 2, 0)
Input validation with “normal” values P(1, 2, 3)
What about other requirements?
If input values are dependent and that affects output, random values can not be used! – we may not be able to find needed values with random generation if p*(p-a)*(p-b)*(p-c) < 0 -> sides don’t form a
triangle We must use specification-based (Black-box)
or program-based (White-box) test data generation
Black-Box approach
Generating test data from formal specifications (ie. Z-notation)
Classification Tree Method (CTM) ...
Classification Tree Method4
Based on equivalence partitions method: input and output properties are divided into equivalence partitions
Equivalence partitions are combined into test cases
The goal: minimal but sufficient amount of test cases
4Dai, Z. R., Deussen, P. H. Automatic Test Data Generation for TTCN-3 Using Classification Tree Method (2005)
CTM
Equivalence partitions form a tree structure Input dependencies are not resolved
White-Box approach
White-box test data generation – based on program structure
Test data generation problem: For program P and path u, find input x S, so that P(x) traverses path u, where S is the set of all input values
Remember: white-box approach is based on program formalisation (graph, FSM, …)
Test data generator structure2
2Edvardsson, J. Contributions to Program- and Specification-based test data generation. (2002). www.ida.liu.se/~joned/papers/joned_lic.pdf
Possible strategies (adequacy criteria)
Statement coverage Branch coverage Condition coverage Multiple-condition coverage Path coverage …
Numerous methods for Constraint generator & Constraint solver Symbolic Execution Actual Execution Symbolic/Actual Execution hybrid Simulated Annealing Iterative Relaxation Technique Chaining Approach Genetic Algorithms MEA-Graph Planning ...
Symbolic Execution2
Popular static method for finding path constraints Path constraints are rewritten using input variables
Not suitable for programs using pointers and arrays Not suitable for programs using precompiled units
read(a,b)
c=a+b;
d=a-b;
e=c*d;
if (e>5) {...}
a*a – b*b > 5=>
Actual Execution2
Program is executed several times On every execution: check, whether or not
the desired path is executed If desired path is not executed, program is re-
executed with slightly modified input values Program is re-executed until desired path is
traversed or user-defined limit (time, execution count) is exceeded
Solves some problems of symbolic execution since values of variables are available
Actual Execution
For each path condition bi objective function is found:
Fi(x) {<|<=|=} 0
If Fi(x) {<|<=|=} 0, then current path is executed
F(x)= Σ Fi(x), if branch consists of several conditions
How to minimize objective function so that Fi (x)=0
In other words, what input values are needed to execute desired path?
Simulated Annealing
Simulated Annealing – generic probabilistic meta-algorithm for finding good approximation to the global optimum for a given function in a large search space
Analogy from metallurgy: Process of annealing is used for reducing defects in material Metal is heated: atoms start to move Metal is cooled down slowly: greater
probability that atoms find a “suitable” place
Simulated Annealing
Goal: minimize the objective function -> desired path is executed
Find a “random” solution for objective function Compare the solution with current solution of
objective function Decide, whether or not the “random” solution
is better than current solution
Simulated Annealing
If “random” solution gives a better value (closer to 0) for objective function, the “random” value is always chosen (probability is 1)
If “random” solution is not better than current solution – “sometimes” it is chosen – depends of the “temperature”
The value of “temperature” is decreased In the beginning high “temperature” -> almost every
solution is chosen When temperature is lowered, the probability of
choosing worse solution is lowered until it is 0
Simulated Annealing: properties
Choosing worse solutions in the beginning lowers the probability of getting stuck in a local optimum (drawback of Gradient Descent/Hill Climbing/Greedy algorithms)
It is possible to show, that probability of finding global optimum is almost 1
Little use in practice, since finding the global optimum with sufficient significance by annealing takes more time than full search of the whole search space
Simulated Annealing
Parameters for successful simulated annealing: Art rather than science How to find a “random” solution – how to
minimize the count of iterations finding the optimum?
How to determine, whether or not the “worse” solution is picked?
Annealing schedule – from what “temperature” to start and how the “temperature” is lowered?
Genetic algorithms
Imitates the process of natural selection Evaluation Choice Recombination and mutation
Start with random set of solutions - population Solutions are evaluated for their fitness –
ability to generate good offspring Chosen (good) solutions are recombined and
mutated to generate a new generation of solutions
GA for test data generation5
Algorithm is driven by control dependency graph of the program Graph nodes = program statements Graph edges = control dependencies between
program statements Goal: find data for executing certain node
(program sentence) Node X is post-dominated by node Y, if every
directed path from X to the end of the program includes node Y
5Pargas, R., Harrold, J.M., Peck, R.R. Test-Data generation Using genetic Algorithms (1999)
GA for test data generation
Node Y is control dependent of Y, only if Exists a directed path from Y to X and all
nodes on this path (except X,Y) are post-dominated by Y and
X is not post-dominated by Y Control dependency predicate path (CDPP)–
predicates that must be satisfied on acyclic path from initial node to some other node X
GA for TDG: algorithm
Solution is set of test data Start with random set Evaluate fitness of data
Execute the program with data, mark predicates on executed path
Compare the found set of predicates with CDPP to desired node
The more the found set of data allowed to execute CDPP, the better the data is
GA for TDG
Best solutions are chosen, recombined and mutated to generate a new generation of solutions
Non-typical application of GA – several possible solutions, depending on the test goal ie. find data for executing nodes A, C – more
than one test may be needed if A and C are exclusive!
GA: example
int i, j, k;
1: read i, j, k;
2: if (i<j) {
3: if (j<k) {
4: i=k;
} else {
5: k=i;
}
}
6: print i, j, k;5 is test goal,
CDGpath {ET, 2T, 3F}
GA: example
Random population
Fitness f{2, 2, 0, 0} Probability of choosing pi = fi/Σfj ({0.5,0.5,0,0}) One solution can be chosen more than once
TestCase Input(i,j,k) Path
TC1 1, 6, 9 1,2,3,4,6
TC2 0, 1, 4 1,2,3,4,6
TC3 5, 0, 1 1, 2, 6
TC4 2, 2, 3 1, 2, 6
GA: Näide
New population {(1, 6, 9), (0, 1, 4), (0, 1, 4), (0, 1, 4)}
Recombination (one-point crossover): n first values form one parent and others from the other parent N=2: {(1, 6, 4), (0, 1, 9)}
Mutation: Value in random position is replaced with a random number (0, 6, 4), (5, 1, 4)
Summary
Numerous methods Black-Box White-Box
Choice of methods depends on knowledge and preference of tester, Technology of SUT
Viited
1Edvardsson, J. A survey on Automatic Test Data Generation. (1999). [WWW] www.ida.liu.se/~joned/papers/class_atdg.pdf
2Edvardsson, J. Contributions to Program- and Specification-based test data generation. (2002). [WWW] www.ida.liu.se/~joned/papers/joned_lic.pdf
3Gupta, N., M, Mathur, A., Soffa, M.L. Automated Test Data Generation Using an Iterative Relaxation Method (1999)
Viited
4Dai, Z. R., Deussen, P. H. Automatic Test Data Generation for TTCN-3 Using Classification Tree Method (2005)
5Pargas, R., Harrold, J.M., Peck, R.R. Test-Data generation Using genetic Algorithms (1999)