mutation testing implements grammar-based testing · grammar-based integration mutation testing...

www.tugraz.at

Mutation Testing implementsGrammar-Based TestingManuel Jelinek,AK Softwaretechnologie

Graz, 20. Januar 2014

2

Agenda

Agenda

1. Grammar-Based Mutation Analysis

2. Grammar-Based Testing of Programs

3. Grammar-Based Integration Mutation Testing

4. Grammar-Based Testing Using Models

5. Grammar-Based Input Testing

6. Conclusion

Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014

3

Grammar-Based Mutation Analysis



4


Software engineers often use structures from automatatheory:

Programming languages defined in BNF (backus-naur-form)Program behavior described as finite state machinesAllowable inputs defined by grammars

BNF syntax example for bank transactions:bank ::= action*

action ::= dep | deb

dep ::= "deposit" account amount

deb ::= "debit" account amount

account ::= digit3

amount ::= "$" digit5 "." digit2

digit ::= "0" | "1" | "2" | "3" | "4" |

"5" | "6" | "7" | "8" | "9"


5


bank -> action*

-> action action*

-> dep action*

-> deposit account amount action*

-> deposit digit3 amount action*

-> deposit digit digit2 amount action*

-> deposit 7 digit2 amount action*

-> deposit 7 digit digit amount action*

-> deposit 73 digit amount action*

-> deposit 739 amount action*

-> deposit 739 $ digit5 . digit2 action*

-> deposit 739 $digit2 .digit2 action*

-> deposit 739 $digit digit.digit2 action*

-> deposit 739 $1 digit.digit2 action*

-> deposit 739 $12. digit2 action*

-> deposit 739 $12. digit digit action*

-> deposit 739 $12.3 digit action*

-> deposit 739 $12.35 action*


6


Unmutated derivation test criteria

Syntax-based coverage criteria:

Terminal Symbol Coverage (TSC):The set of test requirements, TR, contains eachterminal symbol t in the grammar G(14 in bank example)Production Coverage (PC):TR contains each production p in the grammar G(17 in bank example)Derivation Coverage (DC):TR contains every possible string that can be derived from thegrammar G(200.000.000 possible strings in bank example)


7


Mutated derivation test criteria

Grammars describe both valid and invalid stringsBoth types can be produced as mutantsA mutant is a variation of a valid string

Mutants may be valid or invalid

Mutation is based on ”mutation operators”and ”ground strings”Never mutate more then one operator at the same time


8


Mutated derivation test criteria

Syntax-based coverage criteria:

Mutation Coverage (MC):For each m ε M, TR contains exactly one requirement, to kill mThe amount of mutants killed is called the mutation scoreMutation Operator Coverage (MOC):For each mutation operator, TR contains exactly onerequirement, to create a mutated string m that is derived usingthe mutation operatorMutation Production Coverage (MPC):For each mutation operator, TR contains several requirements,to create one mutated string m that includes every productionthat can be mutated by that operator


9

Grammar-Based Testing of Programs



10


Program-based mutation

The original and most widely known application of syntax-basedtesting is to modify programsOperators modify a ground string (program under test) to createmutant programsMutant programs must compile correctly (valid strings)Mutants are not tests, but used to find testsOnce mutants are defined, tests must be found to cause mutantsto fail when executedThis is called ”killing mutants”


11

Grammar-Based Integration Mutation Testing

Grammar-Based Integration MutationTesting


12

Grammar-Based Integration Mutation Testing

also called ”interface mutation testing”Testing connections among separate program unitsIn Java, testing the way classes, packages and components areconnectedThis tests features that are unique to object-orientedprogramming languagesFour types of mutation operators

Change a calling method by modifying values that are sentto a called methodChange a calling method by modifying the callChange a called method by modifying values that enter andleave a method

Includes parameters as well as variables from higher scopes (classlevel, package, public, etc.)

Change a called method by modifying return statementsfrom the method


13

Grammar-Based Testing Using Models

Grammar-Based Testing UsingModels


14


Model-based grammars

Languages that describe software in abstract terms

Formal specification languages such as Z, SMV, OCL, etc.Informal specification languagesDesign notations such as statecharts, FSMs and other UMLdiagram notations


15


Specification-based mutation

A FSM (finite state machine) is essentially a graph G

Nodes are statesEdges are transitions

A formalization of an FSM is:

States are implicitly defined by declaring variables withlimited rangeThe state space is then the Cartesian product of the rangesof the variablesInitial states are defined by limiting the ranges of some orall of the variablesTransitions are defined by rules that characterize the sourceand target of each transition


16


Example SMV (symbolic model verifier)

MODULE main

#define false 0

#define true 1

VAR

x, y : boolean;

ASSIGN

init (x) := false;

init (y) := false;

next (x) := case next (y) := case

!x & y : true; x & !y : false;

!y : true; x & y : y;

x : false; !x & y : false;

true : x; true : true;

esac; esac;


17


Mutations and Test CasesMutating FSMs requires mutation operatorsMost FSM mutation operators are similar to program languageoperators

Constant Replacement operator:changes a constant to each other constantin the next(y) case: !x & y: false

is mutated to !x & y: true

To kill this mutant, we need a sequence of states (a path) thatthe original machine allows but the mutated machine does notThis is what model checkers do

Model checkers find counterexamples – paths in themachine that violate some propertyProperties are written in ”temporal logic”– logicalstatements that are true for some period of time!x & y: false has different result from !x & y: true


18


Counter-example SMV machinenext (y) := case

x & !y : false;

x & y : y;

!x & y : false;

true : true;

esac;

SPEC AG (!x & y) -> AX (y=true)

The model checkershould produce:

/*state 1*/ {x=0, y=0}

/*state 2*/ {x=1, y=1}

/*state 3*/ {x=0, y=1}

/*state 4*/ {x=1, y=0}

This represents a test case thatgoes from nodes FF to TT to FT toTF in the original FSMIf no sequence is produced, themutant is equivalent


19

Grammar-Based Input Testing



20


Input space grammar

The set of allowable inputs to a program, method or softwareCan be described in many waysMost input spaces can be described as grammarsGrammars are usually not provided, but creating them is avaluable service by the tester (errors will often be found simplyby creating the grammar)Software should reject or handle invalid data


21


String mutation input-based testing

Mutating input grammars

Mutants are testsCreate valid and invalid stringsNo ground strings ⇒ no killing


22


Input grammar mutation operators

Nonterminal replacementEvery nonterminal symbol in a production is replaced by othernonterminal symbols

Mutate ‘dep ::= "deposit" account amount’ to:dep ::= "deposit" amount amount

--> deposit $1500.00 $3789.88

dep ::= "deposit" account digit

--> deposit 4400 5


23



Terminal replacementEvery terminal symbol in a production is replaced by otherterminal symbols

Mutate ‘amount ::= "$" digit+ "." digit2’ to:amount ::= "." digit+ "." digit2

--> deposit 4400 .1500.00

amount ::= "$" digit+ "$" digit2

--> deposit 4400 $1500$00

amount ::= "$" digit+ "1" digit2

--> deposit 4400 $1500100


24



Terminal and nonterminal deletionEvery terminal and nonterminal symbol in a production is deleted

Mutate ‘dep ::= "deposit" account amount’ to:dep ::= account amount

--> 4400 $1500.00

dep ::= "deposit" amount

--> deposit $1500.00

dep ::= "deposit" account

--> deposit 4400


25



Terminal and nonterminal duplicationEvery terminal and nonterminal symbol in a production isduplicated

Mutate ‘dep ::= "deposit" account amount’ to:dep ::= "deposit" "deposit" account amount

--> deposit deposit 4400 $1500.00

dep ::= "deposit" account account amount

--> deposit 4400 4400 $1500.00

dep ::= "deposit" account amount amount

--> deposit 4400 $1500.00 $1500.00


26

Conclusion

Conclusion

Grammars describe both valid and invalid stringsBoth types can be produced as mutantsA mutant is a variation of a valid stringMutation is based on ”mutation operators”and ”ground strings”Ground string: A string in the grammarMutation Operator: A rule that specifies syntactic variations ofstrings generated from a grammarMutant: The result of one application of a mutation operator


mutation testing implements grammar-based testing · grammar-based integration mutation testing...

Documents