mutation testing and mujava
TRANSCRIPT
1
Mutation testing of object Mutation testing of object oriented programs of Java : oriented programs of Java : µJavaµJava
Krunal Parmar(15CS60R13)
Guided By :Prof. Rajib Mall
1
2
OutlineOutline• Introduction • Overview of Object Oriented Testing.• Fault hypothesis for object-oriented
software• Mutation testing.• Mutation testing tool for java (µJava).• µJava operators and example.• Demo of testing program using µJava.• Conclusion
2
• Software testing is necessary to produce highly reliable systems.
• More than 50% of development effort is being spent on testing.
• Quality and effective test case design is equally important.
IntroductionIntroduction
3
4
Object Oriented TestingObject Oriented Testing• Research confirms that testing methods
proposed for procedural approach are not adequate for Object Oriented approach. Ex. Statement coverage
• Object Oriented software testing poses additional problems due to Object Oriented fetchers. Ex. Inheritance ,encapsulation and polymorphism
• Testing efforts for OO software found to be increased compared to testing procedural software.
4
• Object Oriented software testing is performed at different levels.
• Object oriented testing can be classified into three levels:
1)Class level.2)Cluster level.3)System level.
Object Oriented Testing Object Oriented Testing (contd.)(contd.)
55
1 ) Class level:–Testing interactions between
attributes and methods of class must be addressed.
2 ) Cluster level:–Tests the interactions among a
group of cooperating classes.–A sequence of interactions is
typically required to implement a visible behaviour (i.e. a use case).3 ) System level:
–Tests an entire operational system.
Object Oriented Testing Object Oriented Testing (contd.)(contd.)
6
7
fault hypothesis for object-fault hypothesis for object-oriented softwareoriented softwarefault hypothesis :an assumption about where faults are
likely to be found.
Because number of possible tests is infinite
for practical purposes, Rational testing must be
based on fault hypothesis. 7
8
fault hypothesis (Cont.)fault hypothesis (Cont.)There are two general fault hypothesis
whichcorrespond to two basic testing strategies:
( 1 ) conformance-directed testing, which seeks to
establish conformance to requirements or
specifications. ( 2 ) fault-directed testing, which seeks to
reveal implementation faults. 8
9
fault-directedfault-directed testing testing• Fault-directed testing is motivated by
the observation that conformance can be
demonstrated for an implementation that contains faults.
• Searching for faults is a practical and prudent
alternative to conformance (Myers, 1979).
• Since the combinations of input, state, output and
paths are astronomically large, efficient probing of
an implementation requires a specific fault hypothesis
to direct the search for faults.
9
10
Testing is a good thingTesting is a good thingBut how do we know our tests are
good?
10
Mutation TestingMutation Testing
• Mutation testing is a fault-based testing technique that measures the effectiveness of test cases.
• Proposed by Richard J. Lipton in 1971 (winner of 2014 Knuth Prize)
• A better way to measure the quality of your tests
• Surge of interest in the 1980s
“Who watches the watchmen?”
1111
TERMINOLOGY: MutationTERMINOLOGY: Mutation A mutation is a (small) change in your
codebase, for example:
1212
TERMINOLOGY: MUTANTTERMINOLOGY: MUTANT
DebitCard >= anotherDebitCard
^(type = anotherDebitCard type)
and: [ number = anotherDebitCard number ]
DebitCard >= anotherDebitCard
^(type = anotherDebitCard type)
or: [ number = anotherDebitCard number ]
Operator: Change #and: by #or:
A mutant is a mutated version of your class or method.
13
Process of creating the MutantProcess of creating the Mutant
The Source Code
The Mutation “Operator”
Mutation Process
The “Mutant”
1414
Mutation Testing (contd.)Mutation Testing (contd.)• Mutation testing is based on the
assumption that a program will be well tested if a majority of simple faults are detected and removed.
• Simple faults are introduced into the program by creating a set of mutants.
• These mutants are created from the original program by applying mutation operators, which describe syntactic changes to the programming language.
1515
1616
• Measure the effectiveness of a test set in terms of its ability to detect faults
17/20
Objective of mutation testingObjective of mutation testing
1717
Steps : Mutation testingSteps : Mutation testing
1. Run test suite2. Change code (mutate)3. Run test suite again4. Observe outcomes
18
Test cases are used to execute these mutants with the goal of causing each mutant to produceincorrect output.
A test case that distinguishes the program from one or more mutants is considered to be effectiveat finding faults in the program.
Steps :18
OUTCOME #1: OUTCOME #1: KILLEDKILLED
19
• If a test set can distinguish a mutant from the original program i.e., it produces different execution result, the mutant is said to be killed.
• In short, mutant is killed if a test fails (detecting the mutated code)
OUTCOME #2: OUTCOME #2: LIVEDLIVED• A mutant didn’t trigger a failing test
, mutant remains live because it is equivalent to the original program i.e. it is functionally identical to the original program or the test data is inadequate to kill the mutant. 19
OUTCOME #3: OUTCOME #3: TIMED OUTTIMED OUT• The mutant caused the
program loop, get stuck
20
Other outcomesOther outcomes1) NON-VIABLEJVM could not load the mutant byte-code
2)MEMORY ERRORJVM ran out of memory during test
3)RUN ERRORAn error but none of the above.
20
If process is not error-free,
fix it
Test Mutan
ts
Process
Mutation
MutationMutatio
nTests
Test Process
Create Mutants
YesTest
CompleteNoAny Live Mutants?
Problem with Tests?
Any Mutations that are
caught by tests are killed
New Test Data
The Mutation ProcessThe Mutation Process
21
Equivalent MutantsEquivalent Mutants• There may be surviving mutants that
cannot be killed, these are called Equivalent Mutants
• Although syntactically different, these mutants are indistinguishable through testing.
• They therefore have to be checked ‘by hand’
while......i++if(i>=5)break;
while......i++if(i==5)
break;Original Code Mutant code 22
23
24
If a test data is inadequate, it can be improved by adding test cases to kill the live mutant.
A test set which can kill all non-equivalent mutants is said to be adequate.How to measure the adequacy of a test
set?Adequacy of test set :
(No. of killed mutants)
(No. of non-equivalent mutants)
Adequacy of test setAdequacy of test set
24
25
Advantages of Mutation Testing1) It can show the ambiguities in test cases. 2) It leads to more reliable product.
Disadvantages of Mutation Testing1) It is time consuming technique, hence requires automated tools.2) Each mutation will have the same size as that of the original program. So, a large number of mutant programs may need to be tested against the candidate test suite.25
26
TOOLINGTOOLINGThere exist many mutation tools for different languages and technologies.
µJava: http://cs.gmu.edu/~offutt/mujava/
Jester: http://jester.sourceforge.net/
Jumble: http://jumble.sourceforge.net/
javaLanche: http://www.st.cs.unisaarland.de/mutation/
26
27
µjavaµjava• µJava is a automated mutation system for
Java programs. It automatically generates mutants for mutation testing.
• µJava can test individual classes and packages of multiple classes.
• Tests are supplied by the users as sequences of method calls to the classes under test encapsulated in methods in separate classes.
27
28
Input
output28
µjava (Contd.)µjava (Contd.)
29
• µµJava implements both inter- and intra-class mutation operators.
• Class mutation operators are classified into four groups, based on the language features that are affected
Mutation OperatorsMutation Operators
1. Encapsulation2. Inheritance3. Polymorphism4. Java-Specific Features
29
30
Mutation Operator for Mutation Operator for Encapsulation Encapsulation AMC – Access modifier change:
•The AMC operator changes the access level for instance variables and methods to other access levels. •The purpose of the AMC operator is to guide testers to generate test cases that ensure that accessibility is correct.for example,
30
31
Mutation Operators for Mutation Operators for Inheritance Inheritance
• Inheritance is a powerful and useful abstraction mechanism, but incorrect use of inheritance can lead to a number of faults.
• We define five mutation operators to try to test the various aspects of using inheritance, coveringvariable shadowing, method overriding, the useof super and definition of constructors.
31
32
Operators for Inheritance(Contd.)Operators for Inheritance(Contd.)• IHD – Hiding variable deletion
• IHI – Hiding variable Insertion
32
33
Operators for Inheritance(Contd.)Operators for Inheritance(Contd.)• IOD – Overriding method deletion
• IOP – Overriding method calling Position change
33
34
Mutation Operators for Mutation Operators for polymorphism polymorphism • Polymorphism allows the behaviour of
an object reference to be different depending the actual type. Therefore, it is important to identify and exercise the program with all possible type bindings.
• The polymorphism mutation operators are designed to ensure this type of testing.
34
35
Mutation Operators for Java Mutation Operators for Java Specific features Specific features • Some object-oriented features are not common
to all object-oriented languages. This group of operators attempt to ensure correct use of such features supported in Java
35
36
How µJava does it ?How µJava does it ?• µµJava uses a reflection technique to
satisfy those requirements, specifically, to generate and run mutants.
• Reflection is the ability of a program to observe and possibly modify its high level structure.reflection is a natural way to implement mutationanalysis for several reasons.
1 : it provides an API to easily change the behaviour of a program during execution.
2 : it lets programmers extract OO-related Information about a class by providing an object that represents a logical structure of the class
36
37
Example of Mutant generation Example of Mutant generation ( IOD operator)( IOD operator) • Parent
class• Child
class
• Mutant generation code
37
38
Generating mutants with µJavaGenerating mutants with µJava
Mutant generator screen38
39
Generating mutants with µJava Generating mutants with µJava (contd.)(contd.)
Class mutant viewer screen39
40
Generating mutants with µJava Generating mutants with µJava (contd.)(contd.)
Test case runner screen40
41
ConclusionConclusion
41
• Mutation testing is the powerful technique for the assessment and enhancement of tests.
• The effectiveness of mutation testing depends heavily on the types of faults that the mutation operators are designed to represent. Therefore, the quality of the mutation operators is key to mutation testing.
• µJava allows the tester to enter and run tests,and evaluates the mutation coverage of the tests.
42
ReferencesReferences
42
• Testing Object Oriented Software : a Survey. ROBERT V. BINDER , RBSC Corporation, 3 First National Plaza, Suite 1400,Chicago,IL 6060 24205,U.S.A.
• Inter-Class Mutation Operators for Java Yu-Seung Ma, Yong-Rae Kwon and Jeff Offutt. Proceedings of the 13th International Symposium on Software Reliability Engineering, IEEE Computer Society Press, Annapolis MD, November 2002, pp. 352-363.
• Wikipedia : Mutation Testinghttps://en.wikipedia.org/wiki/Mutation_testing
• µJava Home Pagehttps://cs.gmu.edu/~offutt/mujava/
43
4444