mutation testing and mujava

1

Mutation testing of object Mutation testing of object oriented programs of Java : oriented programs of Java : µJavaµJava

Krunal Parmar(15CS60R13)

Guided By :Prof. Rajib Mall

1

2

OutlineOutline• Introduction • Overview of Object Oriented Testing.• Fault hypothesis for object-oriented

software• Mutation testing.• Mutation testing tool for java (µJava).• µJava operators and example.• Demo of testing program using µJava.• Conclusion

2

• Software testing is necessary to produce highly reliable systems.

• More than 50% of development effort is being spent on testing.

• Quality and effective test case design is equally important.

IntroductionIntroduction

3

4

Object Oriented TestingObject Oriented Testing• Research confirms that testing methods

proposed for procedural approach are not adequate for Object Oriented approach. Ex. Statement coverage

• Object Oriented software testing poses additional problems due to Object Oriented fetchers. Ex. Inheritance ,encapsulation and polymorphism

• Testing efforts for OO software found to be increased compared to testing procedural software.

4

• Object Oriented software testing is performed at different levels.

• Object oriented testing can be classified into three levels:

1)Class level.2)Cluster level.3)System level.

Object Oriented Testing Object Oriented Testing (contd.)(contd.)

55

1 ) Class level:–Testing interactions between

attributes and methods of class must be addressed.

2 ) Cluster level:–Tests the interactions among a

group of cooperating classes.–A sequence of interactions is

typically required to implement a visible behaviour (i.e. a use case).3 ) System level:

–Tests an entire operational system.

Object Oriented Testing Object Oriented Testing (contd.)(contd.)

6

7

fault hypothesis for object-fault hypothesis for object-oriented softwareoriented softwarefault hypothesis :an assumption about where faults are

likely to be found.

Because number of possible tests is infinite

for practical purposes, Rational testing must be

based on fault hypothesis. 7

8

fault hypothesis (Cont.)fault hypothesis (Cont.)There are two general fault hypothesis

whichcorrespond to two basic testing strategies:

( 1 ) conformance-directed testing, which seeks to

establish conformance to requirements or

specifications. ( 2 ) fault-directed testing, which seeks to

reveal implementation faults. 8

9

fault-directedfault-directed testing testing• Fault-directed testing is motivated by

the observation that conformance can be

demonstrated for an implementation that contains faults.

• Searching for faults is a practical and prudent

alternative to conformance (Myers, 1979).

• Since the combinations of input, state, output and

paths are astronomically large, efficient probing of

an implementation requires a specific fault hypothesis

to direct the search for faults.

9

10

Testing is a good thingTesting is a good thingBut how do we know our tests are

good?

10

Mutation TestingMutation Testing

• Mutation testing is a fault-based testing technique that measures the effectiveness of test cases.

• Proposed by Richard J. Lipton in 1971 (winner of 2014 Knuth Prize)

• A better way to measure the quality of your tests

• Surge of interest in the 1980s

“Who watches the watchmen?”

1111

TERMINOLOGY: MutationTERMINOLOGY: Mutation A mutation is a (small) change in your

codebase, for example:

1212

TERMINOLOGY: MUTANTTERMINOLOGY: MUTANT

DebitCard >= anotherDebitCard

^(type = anotherDebitCard type)

and: [ number = anotherDebitCard number ]

DebitCard >= anotherDebitCard

^(type = anotherDebitCard type)

or: [ number = anotherDebitCard number ]

Operator: Change #and: by #or:

A mutant is a mutated version of your class or method.

13

Process of creating the MutantProcess of creating the Mutant

The Source Code

The Mutation “Operator”

Mutation Process

The “Mutant”

1414

Mutation Testing (contd.)Mutation Testing (contd.)• Mutation testing is based on the

assumption that a program will be well tested if a majority of simple faults are detected and removed.

• Simple faults are introduced into the program by creating a set of mutants.

• These mutants are created from the original program by applying mutation operators, which describe syntactic changes to the programming language.

1515

1616

• Measure the effectiveness of a test set in terms of its ability to detect faults

17/20

Objective of mutation testingObjective of mutation testing

1717

Steps : Mutation testingSteps : Mutation testing

1. Run test suite2. Change code (mutate)3. Run test suite again4. Observe outcomes

18

Test cases are used to execute these mutants with the goal of causing each mutant to produceincorrect output.

A test case that distinguishes the program from one or more mutants is considered to be effectiveat finding faults in the program.

Steps :18

OUTCOME #1: OUTCOME #1: KILLEDKILLED

19

• If a test set can distinguish a mutant from the original program i.e., it produces different execution result, the mutant is said to be killed.

• In short, mutant is killed if a test fails (detecting the mutated code)

OUTCOME #2: OUTCOME #2: LIVEDLIVED• A mutant didn’t trigger a failing test

, mutant remains live because it is equivalent to the original program i.e. it is functionally identical to the original program or the test data is inadequate to kill the mutant. 19

OUTCOME #3: OUTCOME #3: TIMED OUTTIMED OUT• The mutant caused the

program loop, get stuck

20

Other outcomesOther outcomes1) NON-VIABLEJVM could not load the mutant byte-code

2)MEMORY ERRORJVM ran out of memory during test

3)RUN ERRORAn error but none of the above.

20

If process is not error-free,

fix it

Test Mutan

ts

Process

Mutation

MutationMutatio

nTests

Test Process

Create Mutants

YesTest

CompleteNoAny Live Mutants?

Problem with Tests?

Any Mutations that are

caught by tests are killed

New Test Data

The Mutation ProcessThe Mutation Process

21

Equivalent MutantsEquivalent Mutants• There may be surviving mutants that

cannot be killed, these are called Equivalent Mutants

• Although syntactically different, these mutants are indistinguishable through testing.

• They therefore have to be checked ‘by hand’

while......i++if(i>=5)break;

while......i++if(i==5)

break;Original Code Mutant code 22

23

24

If a test data is inadequate, it can be improved by adding test cases to kill the live mutant.

A test set which can kill all non-equivalent mutants is said to be adequate.How to measure the adequacy of a test

set?Adequacy of test set :

(No. of killed mutants)

(No. of non-equivalent mutants)

Adequacy of test setAdequacy of test set

24

25

Advantages of Mutation Testing1) It can show the ambiguities in test cases. 2) It leads to more reliable product.

Disadvantages of Mutation Testing1) It is time consuming technique, hence requires automated tools.2) Each mutation will have the same size as that of the original program. So, a large number of mutant programs may need to be tested against the candidate test suite.25

26

TOOLINGTOOLINGThere exist many mutation tools for different languages and technologies.

µJava: http://cs.gmu.edu/~offutt/mujava/

Jester: http://jester.sourceforge.net/

Jumble: http://jumble.sourceforge.net/

javaLanche: http://www.st.cs.unisaarland.de/mutation/

26

27

µjavaµjava• µJava is a automated mutation system for

Java programs. It automatically generates mutants for mutation testing.

• µJava can test individual classes and packages of multiple classes.

• Tests are supplied by the users as sequences of method calls to the classes under test encapsulated in methods in separate classes.

27

28

Input

output28

µjava (Contd.)µjava (Contd.)

29

• µµJava implements both inter- and intra-class mutation operators.

• Class mutation operators are classified into four groups, based on the language features that are affected

Mutation OperatorsMutation Operators

1. Encapsulation2. Inheritance3. Polymorphism4. Java-Specific Features

29

30

Mutation Operator for Mutation Operator for Encapsulation Encapsulation AMC – Access modifier change:

•The AMC operator changes the access level for instance variables and methods to other access levels. •The purpose of the AMC operator is to guide testers to generate test cases that ensure that accessibility is correct.for example,

30

31

Mutation Operators for Mutation Operators for Inheritance Inheritance

• Inheritance is a powerful and useful abstraction mechanism, but incorrect use of inheritance can lead to a number of faults.

• We define five mutation operators to try to test the various aspects of using inheritance, coveringvariable shadowing, method overriding, the useof super and definition of constructors.

31

32

Operators for Inheritance(Contd.)Operators for Inheritance(Contd.)• IHD – Hiding variable deletion

• IHI – Hiding variable Insertion

32

33

Operators for Inheritance(Contd.)Operators for Inheritance(Contd.)• IOD – Overriding method deletion

• IOP – Overriding method calling Position change

33

34

Mutation Operators for Mutation Operators for polymorphism polymorphism • Polymorphism allows the behaviour of

an object reference to be different depending the actual type. Therefore, it is important to identify and exercise the program with all possible type bindings.

• The polymorphism mutation operators are designed to ensure this type of testing.

34

35

Mutation Operators for Java Mutation Operators for Java Specific features Specific features • Some object-oriented features are not common

to all object-oriented languages. This group of operators attempt to ensure correct use of such features supported in Java

35

36

How µJava does it ?How µJava does it ?• µµJava uses a reflection technique to

satisfy those requirements, specifically, to generate and run mutants.

• Reflection is the ability of a program to observe and possibly modify its high level structure.reflection is a natural way to implement mutationanalysis for several reasons.

1 : it provides an API to easily change the behaviour of a program during execution.

2 : it lets programmers extract OO-related Information about a class by providing an object that represents a logical structure of the class

36

37

Example of Mutant generation Example of Mutant generation ( IOD operator)( IOD operator) • Parent

class• Child

class

• Mutant generation code

37

38

Generating mutants with µJavaGenerating mutants with µJava

Mutant generator screen38

39

Generating mutants with µJava Generating mutants with µJava (contd.)(contd.)

Class mutant viewer screen39

40

Generating mutants with µJava Generating mutants with µJava (contd.)(contd.)

Test case runner screen40

41

ConclusionConclusion

41

• Mutation testing is the powerful technique for the assessment and enhancement of tests.

• The effectiveness of mutation testing depends heavily on the types of faults that the mutation operators are designed to represent. Therefore, the quality of the mutation operators is key to mutation testing.

• µJava allows the tester to enter and run tests,and evaluates the mutation coverage of the tests.

42

ReferencesReferences

42

• Testing Object Oriented Software : a Survey. ROBERT V. BINDER , RBSC Corporation, 3 First National Plaza, Suite 1400,Chicago,IL 6060 24205,U.S.A.

• Inter-Class Mutation Operators for Java Yu-Seung Ma, Yong-Rae Kwon and Jeff Offutt. Proceedings of the 13th International Symposium on Software Reliability Engineering, IEEE Computer Society Press, Annapolis MD, November 2002, pp. 352-363.

• Wikipedia : Mutation Testinghttps://en.wikipedia.org/wiki/Mutation_testing

• µJava Home Pagehttps://cs.gmu.edu/~offutt/mujava/

43

4444