automatically repairing test cases for evolving method declarations

Automatically Repairing Test Cases

for Evolving Method DeclarationsMehdi MirzaaghaeiFabrizio Pastore

Mauro Pezzè

Universitàdella Svizzeraitaliana

http://swiss-landmarks.ch/panos/Lugano9.jpg

Software evolves

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance };}

Software evolves

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance };}

Software evolves

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance };}

Software evolves

public class .....

public class AccountUtil { private double dailyInterestRate = 0.00005; private int interestTerm=365; public double interest(BankAccount account return account.getBalance() *dailyInterestRate*interestTerm; } ...

public class AccountFactory {

public static BankAccount create( AccountContext ctx, boolean special ){ BankAccount account = new BankAcccount( ctx.amount ); if ( special ){ account.setInterestRate(0.0001); } ...

public class .....

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance };}

Software evolves

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance };}

public class BankAccount { private int balance;

public void deposit(Money money){ balance += money.centsValue; }

public int getBalance(){ return balance; };}

Original classes are modified

public class .....

public class .....

Test cases need maintenance

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance };}

public class BankAccount { private int balance;

public void deposit(Money money){ balance += money.centsValue; }

public int getBalance(){ return balance; };}

Compilation error:Method deposit(Money) in the type BankAccount is not applicable for the arguments (int)

changes in method declarations: 23% changes

that impact on compilation

changes in method declarations: 23% changes

that impact on compilation

> 80% in maintenance releases

changes in method declarations: 23% changes

that impact on compilation

> 80% in maintenance releases

Automatic Repair to Save Effort

Automatic Test Repair

[Memon et. al. 2008, Grechanik 2009 ]Repair GUI tests

Well suited only for GUI tests

Automatic Test Repair

[Memon et. al. 2008, Grechanik 2009 ]Repair GUI tests Repair Oracles

[B. Daniel et al 2010]

testInterest(){ ... assertEquals( 50, result );} FAILURE: expected 50, found: 40

assertEquals( 40, result );

Well suited only for GUI tests Focus on oracles only

Automatic Test Repair

[Memon et. al. 2008, Grechanik 2009 ]Repair GUI tests

Refactoring Techniques [ReBa, Eclipse]

Repair Oracles[B. Daniel et al 2010]

testInterest(){ ... assertEquals( 50, result );} FAILURE: expected 50, found: 40

assertEquals( 40, result );

//calculate one year interestresult = account.interest();

result = account.interest( 0 );

Well suited only for GUI tests Focus on oracles only

Prevent only some compilation errors

Automatic Test Repair

TestCareAssistant repairs test case compilation errors

• Parameter add

• Parameter type change • Return type change

• Parameter remove

caused by method declaration changes

TestCareAssistant repairs test case compilation errors

testDeposit(){BankAccount account = new BankAccount();int amount = 500;account.deposit(amount); assertEquals(500, account.getBalance());

• Parameter add

• Parameter type change • Return type change

• Parameter remove

caused by method declaration changes

TestCareAssistant repairs test case compilation errors

testDeposit(){BankAccount account = new BankAccount();int amount = 500;account.deposit(amount); assertEquals(500, account.getBalance());

Replace: int amount = 500;with: Money amount = new Money(500); account.deposit(amount);

• Parameter add

• Parameter type change • Return type change

• Parameter remove

caused by method declaration changes

TestCareAssistant repairs test case compilation errors

testDeposit(){BankAccount account = new BankAccount();int amount = 500;account.deposit(amount); assertEquals(500, account.getBalance());

testDeposit(){BankAccount account = new BankAccount();Money amount = new Money(500);account.deposit(amount); assertEquals(500, account.getBalance());

Replace: int amount = 500;with: Money amount = new Money(500); account.deposit(amount);

• Parameter add

• Parameter type change • Return type change

• Parameter remove

caused by method declaration changes

TestCareAssistant repairs test case compilation errors

testDeposit(){BankAccount account = new BankAccount();int amount = 500;account.deposit(amount); assertEquals(500, account.getBalance());

testDeposit(){BankAccount account = new BankAccount();Money amount = new Money(500);account.deposit(amount); assertEquals(500, account.getBalance());

Replace: int amount = 500;with: Money amount = new Money(500); account.deposit(amount);

• Parameter add

• Parameter type change • Return type change

• Parameter remove

caused by method declaration changes

TestCareAssistant repairs test case compilation errors

testDeposit(){BankAccount account = new BankAccount();int amount = 500;account.deposit(amount); assertEquals(500, account.getBalance());

testDeposit(){BankAccount account = new BankAccount();Money amount = new Money(500);account.deposit(amount); assertEquals(500, account.getBalance());

Replace: int amount = 500;with: Money amount = new Money(500); account.deposit(amount);

• Parameter add

• Parameter type change • Return type change

• Parameter remove

caused by method declaration changes

TestCareAssistant repairs test case compilation errors

testDeposit(){BankAccount account = new BankAccount();int amount = 500;account.deposit(amount); assertEquals(500, account.getBalance());

testDeposit(){BankAccount account = new BankAccount();Money amount = new Money(500);account.deposit(amount); assertEquals(500, account.getBalance());

Replace: int amount = 500;with: Money amount = new Money(500); account.deposit(amount);

• Parameter add

• Parameter type change • Return type change

• Parameter remove

caused by method declaration changes

Repair parameter type change

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance; }}

public class BankAccount { private int balance;

public void deposit(Money money){ balance += money.CentsValue; }

public int getBalance(){ return balance; }}

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

Version 0 Version 1

Which parameter to initialize?

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance; }}

public class BankAccount { private int balance;

public void deposit(Money money){ balance += money.CentsValue; }

public int getBalance(){ return balance; }}

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

Version 0 Version 1

Which parameter to initialize?

Use code diff to identify the parameter1

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance; }}

public class BankAccount { private int balance;

public void deposit(Money money){ balance += money.CentsValue; }

public int getBalance(){ return balance; }}

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

Version 0 Version 1

How to initialize a complex object?

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance; }}

public class BankAccount { private int balance;

public void deposit(Money money){ balance += money.centsValue; }

public int getBalance(){ return balance; }}

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

Version 0 Version 1

How to initialize a complex object?

Find first uses of parameter fields in Version12

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance; }}

public class BankAccount { private int balance;

public void deposit(Money money){ balance += money.centsValue; }

public int getBalance(){ return balance; }}

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

Version 0 Version 1

How to determine fields values?

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance; }}

public class BankAccount { private int balance;

public void deposit(Money money){ balance += money.centsValue; }

public int getBalance(){ return balance; }}

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

Version 0 Version 1

How to determine fields values?

Diff to identify corresponding variable in Version 03

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance; }}

public class BankAccount { private int balance;

public void deposit(Money money){ balance += money.centsValue; }

public int getBalance(){ return balance; }}

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

Version 0 Version 1

What was the original value?

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance; }}

public class BankAccount { private int balance;

public void deposit(Money money){ balance += money.CentsValue; }

public int getBalance(){ return balance; }}

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

Version 0 Version 1

What was the original value?

Analyze the def-use chain of the variable back to the definition in the test4

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance; }}

public class BankAccount { private int balance;

public void deposit(Money money){ balance += money.CentsValue; }

public int getBalance(){ return balance; }}

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

Version 0 Version 1

What was the original value?

Analyze the def-use chain of the variable back to the definition in the test4

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance; }}

public class BankAccount { private int balance;

public void deposit(Money money){ balance += money.CentsValue; }

public int getBalance(){ return balance; }}

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

Version 0 Version 1

How can we repair the test?

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance; }}

public class BankAccount { private int balance;

public void deposit(Money money){ balance += money.centsValue; }

public int getBalance(){ return balance; }}

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

Version 0 Version 1

How can we repair the test?

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance; }}

public class BankAccount { private int balance;

public void deposit(Money money){ balance += money.centsValue; }

public int getBalance(){ return balance; }}

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

Version 0 Version 1

How can we repair the test?

Instantiate the new object, use original values to initialize fields5

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance; }}

public class BankAccount { private int balance;

public void deposit(Money money){ balance += money.centsValue; }

public int getBalance(){ return balance; }}

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

Version 0 Version 1

How can we repair the test?

Instantiate the new object, use original values to initialize fields5

Replace: int amount = 500;with: Money amount = new Money(500); account.deposit(amount);

public class BankAccount { private int balance;

public void deposit(int cents){ balance += cents; }

public int getBalance(){ return balance; }}

public class BankAccount { private int balance;

public void deposit(Money money){ balance += money.centsValue; }

public int getBalance(){ return balance; }}

testDeposit(){ BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());}

Version 0 Version 1

To what extent TestCareAssistant can fix test cases automatically?

To what extent TestCareAssistant can fix test cases automatically?

• Repair 22 test cases of 6 open source systems

• Continuum, Geronimo, xml-security, PMD, POI, Shindig

• 24 compilation errors caused by different changes

• 9 parameter types changed, 8 parameters added, 3 parameters removed, 4 return types changed

To what extent TestCareAssistant can fix test cases automatically?

Results Total Repaired %Test Cases 22 16 72

Compilation Errors 24 18 75

Values Initialized 36 29 80

• Repair 22 test cases of 6 open source systems

• Continuum, Geronimo, xml-security, PMD, POI, Shindig

• 24 compilation errors caused by different changes

• 9 parameter types changed, 8 parameters added, 3 parameters removed, 4 return types changed

To what extent TestCareAssistant can fix test cases automatically?

Results Total Repaired %Test Cases 22 16 72

Compilation Errors 24 18 75

Values Initialized 36 29 80

• Repair 22 test cases of 6 open source systems

• Continuum, Geronimo, xml-security, PMD, POI, Shindig

• 24 compilation errors caused by different changes

• 9 parameter types changed, 8 parameters added, 3 parameters removed, 4 return types changed

To what extent TestCareAssistant can fix test cases automatically?

• Static data flow analysis not always effective

• Use of complex data structures, e.g. hash tables

• Changes in method logic

• Changes in interfaces

Results Total Repaired %Test Cases 22 16 72

Compilation Errors 24 18 75

Conclusionspublic class BankAccount { public void deposit(int cents){ balance += cents; }}

public class BankAccount { public void deposit(Money money){ balance += money.CentsValue; }}

testDeposit(){BankAccount account = new BankAccount();int amount = 500;account.deposit(amount); assertEquals(500, account.getBalance());

testDeposit(){BankAccount account = new BankAccount();int amount = 500;account.deposit(amount); assertEquals(500, account.getBalance());

Conclusionspublic class BankAccount { public void deposit(int cents){ balance += cents; }}

public class BankAccount { public void deposit(Money money){ balance += money.CentsValue; }}

testDeposit(){BankAccount account = new BankAccount();int amount = 500;account.deposit(amount); assertEquals(500, account.getBalance());

testDeposit(){BankAccount account = new BankAccount();Money amount = new Money(500);account.deposit(amount); assertEquals(500, account.getBalance());

Conclusionspublic class BankAccount { public void deposit(int cents){ balance += cents; }}

public class BankAccount { public void deposit(Money money){ balance += money.CentsValue; }}

Replace: int amount = 500;with: Money amount = new Money(500); account.deposit(amount);

testDeposit(){BankAccount account = new BankAccount();int amount = 500;account.deposit(amount); assertEquals(500, account.getBalance());

testDeposit(){BankAccount account = new BankAccount();Money amount = new Money(500);account.deposit(amount); assertEquals(500, account.getBalance());

TestCareAssistant

Future Work

CopticChronology EthiopicChronology

Chronology + withUTC()

TestCopticUTC(){ .... assertEquals("AM",

copticChronology.withUTC() );}

Future Work

CopticChronology EthiopicChronology

Chronology + withUTC()

TestCopticUTC(){ .... assertEquals("AM",

copticChronology.withUTC() );}

TestEthiopicUTC(){ .... assertEquals("EE",

ethiopicChronology.withUTC() );}

CopticChronology EthiopicChronology

Chronology + withUTC()

Future Work

CopticChronology EthiopicChronology

Chronology + withUTC()

TestCopticUTC(){ .... assertEquals("AM",

copticChronology.withUTC() );}

TestEthiopicUTC(){ .... assertEquals("EE",

ethiopicChronology.withUTC() );}

CopticChronology EthiopicChronology

Chronology + withUTC()

Future Work

CopticChronology EthiopicChronology

Chronology + withUTC()

TestCopticUTC(){ .... assertEquals("AM",

copticChronology.withUTC() );}

TestEthiopicUTC(){ .... assertEquals("EE",

ethiopicChronology.withUTC() );}

CopticChronology EthiopicChronology

Chronology + withUTC()

Automate Test Generation by Identifying and Implementing Test Adaptation Patterns

Questions?public class BankAccount { public void deposit(int cents){ balance += cents; }}

public class BankAccount { public void deposit(Money money){ balance += money.CentsValue; }}

Replace: int amount = 500;with: Money amount = new Money(500); account.deposit(amount);

testDeposit(){BankAccount account = new BankAccount();int amount = 500;account.deposit(amount); assertEquals(500, account.getBalance());

testDeposit(){BankAccount account = new BankAccount();Money amount = new Money(500);account.deposit(amount); assertEquals(500, account.getBalance());

TestCareAssistant