new strategies for finding f ailures and their d omains
DESCRIPTION
New strategies for finding f ailures and their d omains. Mian Asbaat Ahmad 24-01-2013. Objectives. Development of new automated strategies with the following objectives: To f ind the maximum number of faults and their domains To use a minimum number of test calls - PowerPoint PPT PresentationTRANSCRIPT
New Strategies for Finding failures and its domains
New strategies for finding failures and their domainsMian Asbaat Ahmad24-01-2013
Development of new automated strategies with the following objectives:
To find the maximum number of faults and their domainsTo use a minimum number of test callsTo take the least possible timeTo require minimal resourcesTo give comprehensive resultsTo create an extensive test suite
ObjectivesDirt Spot Sweeping Random (DSSR) Strategy
Automated Discovery of Failure Domain (ADFD) Strategy
ADFD with Daikon (ADFD-D) Strategy
Thesis contributionsWhy random testing?Because:Testing of all values is typically impossibleTime and resources are limitedEasy to implementFree from human biasQuick and effective in finding faults
What is Random Testing?Black-box testing techniqueDynamic testing processRandom SelectionTest executionOutput evaluationResult generation
Automated Random TestingAutomation of the process for:Analysis of system under testSelection of test valuesExecution of testsEvaluation of resultsDelivery of results Generation of test suite
Automated Random Testing Tools
York Extensible Testing InfrastructureRandom testing toolDeveloped in JavaSupports multiple languagesHigh speed (up to 106 calls per minute)Both CLI and GUI interfacesInteractive testingExperimentally verifiedRunnable in cloud8Yeti is a random testing tool, strategy used is random testingCoded in Javaby testing java.lang and iTextDecoupling ?Language agnostic ?High speed of processingYeti can be operated from both CLI and GUITesting can be changed at run time
8YETI Front-end
Why Improve Random Testing?To increase code coverageIf(x == 3333) then where x is 32 bit variableTo increase efficiencyTo decrease over-headTo generate friendly outputTo introduce automationEnhanced versions of Random TestingAdaptive Random Testing (ART)Quasi Random Testing (QRT)Mirror Adaptive Random Testing (MART)Restricted Random Testing (RRT)Feedback-Directed Random Testing (F-DRT)Random Plus Testing (R+)
Types of Fault DomainPoint Fault DomainFaults lies scattered across input domain
Block Fault DomainFaults lies in a block across input domain
Strip Fault Domain Faults lies in a strip across the input domain
Main idea of DSSRWe start by testing with YETI (using random+)
When a new failure is found we add the neighboring values to the list of interesting values
These have higher priority, effectively sweeping around the faults found
Dirt Spot Sweeping Random StrategyBased on three strategiesRandomRandom PlusSpot Sweeping
Block sweepingStrip sweepingExample to illustrate DSSR strategy/** * Calculate square of given number and verify results. * The code contain 3 faults. * @author (Mian and Manuel) */public class Math1{public void calc (int num1) {// Square num1 and store result.int result1 = num1 * num1;int result2 = result1 / num1; // 1 assert Math.sqrt(result1) == num1; // 2 assert result1 >= num1; // 3} }
Performance of DSSR compared to R and R+ St.60 classes from 32 projects were tested by R, R+ and DSSR strategiesIn 43 classes all three strategies found the same number of faultsIn remaining 17 classes DSSR strategy found the highest number of unique failures (1117) followed by R+ (1106) and R (1090) respectivelyOn overall basis DSSR strategy performed better than R and R+ strategies
Test Results of 17/60 classesDSSR strategy better up to 33% than R, up to 17% than R+Limitations of DSSR strategyLack of improvement in point fault domainExcess time involved to find first fault5% over-head than R and 2% than R+ in executionAutomated Discovery of Failure Domain (ADFD)A new automated strategy implemented in YETI Provides GUI interfaceFacilitates control on lower and upper boundFinds faults and their domainsPresents results in graphical formMain idea of ADFDWe start by testing with YETI (using random+)
When a new failure is found ADFD generates programs that will test each direction independently
We then compile and execute these programsThey generate failure domains for each direction
We show the results in the interface
Using ADFDLaunch the ADFD applicationSelect language, ADFD strategy and timeSelect class for TestingSpecify lower and upper boundsClick run button to start testingProgram is dynamically generatedClick count button to find number of files generatedClick button to compile the programClick button to execute the programData is generatedClick button to plot graphFront-end of ADFD
Illustration of ADFD strategy by examples of Point, Block and Strip Fault DomainsPoint Fault Domain one Argument/** * Point Fault Domain example for one argument * @author (Mian and Manuel) */public class PointDomainOneArgument{
public static void pointErrors (int x){ if (x == -66 ) abort(); if (x == -2 ) abort(); if (x == 51 ) abort(); if (x == 23 ) abort();}}
Point Fault Domain two Argument
/** * Point Fault Domain example for two arguments * @author (Mian and Manuel) */public class PointDomainOneArgument{
public static void pointErrors (int x, int y){ int z = x/y;}
}
Block Fault Domain one argument/** * Block Fault Domain example for one argument * @author (Mian and Manuel) */public class BlockDomainOneArgument{
public static void blockErrors (int x){if((x > -2) && (x < 2))abort();if((x > -30) && (x < -25))abort();if((x > 50) && (x < 55))abort();
}}
Block Fault Domain two argument/** * Block Fault Domain example for two arguments * @author (Mian and Manuel) */public class BlockDomainTwoArgument{
public static void pointErrors (int x, int y){
if(((x > 0)&&(x < 20)) || ((y > 0) && (y < 20))){abort();} }
}
Strip Fault Domain one argument/** * Strip Fault Domain example for one argument * @author (Mian and Manuel) */
public class StripDomainOneArgument{
public static void stripErrors (int x){if((x > -5) && (x < 35))abort(); }}
Strip Fault Domain two argument/** * Strip Fault Domain example for two arguments * @author (Mian and Manuel) */public class StripDomainTwoArgument{
public static void pointErrors (int x, int y){
if(((x > 0)&&(x < 50)) || ((y > 0) && (y < 500))){abort();} }
}
Development of ADFD with Daikon (ADFD-D) strategySalient FeaturesTo execute SUT by Daikon to generate invariantsTo add values from invariants to the list of interesting valuesTo Execute DSSR strategyDaikon Invariant DetectorAutomated toolDeveloped by MIT Program Analysis GroupDynamically reports likely program invariantsDetect C, C++, Eiffel, IOA, Java and Perl programs.freely available: http://groups.csail.mit.edu/pag/daikon/download/Daikon with ADFD strategyWe are utilizing Daikon invariants:As oracle for comparing test resultsTo find border (interesting) valuesTo remove the dependence of DSSR on R+ strategyTo increase test performance by restricting the upper and lower bound.
Dirt Spot Sweeping Random (DSSR) StrategyUp to 33% better compared to R strategyUp to 17% better compared to R+ strategy Automated Discovery of Failure Domain (ADFD) StrategyFinds and Plots fault domainExhaustive search of projections on each axis
ADFD with Daikon (ADFD-D) StrategyCombination of ADFD strategy with DaikonResearch still in Progress
ConclusionsThank You