Download - An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows
![Page 1: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/1.jpg)
IEEE Transactions on Software EngineeringWes Masri, Andy Podgurski, David Leon
Presented by Jason R. Beck and Enrique G. Ortiz
![Page 2: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/2.jpg)
Introduction and background definitions Paper Objectives Filtering Techniques Profile Types and Tools Empirical Study Description Subject Programs Results Conclusion Pros - Cons / Suggestions
![Page 3: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/3.jpg)
Information Flow◦ Important concept in software testing research◦ Describes complex interactions between different
program elements.
Software failures◦ Often caused by untested information flows.◦ Why?
Information flows can be complex Too many to make testing them all feasible.
![Page 4: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/4.jpg)
Test Case Filtering◦ Involves selecting a manageable number of test
cases to use.
Software Profiles◦ Software profiles are recorded interactions during
program operation.◦ Can describe control flow, data flow, input or
variable values, object states, event sequences, and timing.
◦ Profiles can analyzed for how likely they are to generate errors and those can be tested further.
![Page 5: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/5.jpg)
1. Reduce the number of test cases to be executed.
2. Reduce the number of test executions which need a manually interpretation of correct output.
◦ Anything that requires a human interpretation of results as part of the test involves much effort.
◦ Can be eliminated if test cases are automated and self validating.
![Page 6: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/6.jpg)
Presents the results of an empirical study using many test case filtering techniques.
Evaluates techniques for their ability to reveal defects in programs.
Information profiles created using author developed tool.
![Page 7: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/7.jpg)
Generally graph theory models showing information flow in the software.
Many proposed techniques Authors focus on …
1. Information flow between objects. Data driven
2. Dynamic program slicing. Program statement driven (think stack trace when
debugging).
Both have static and runtime versions.
![Page 8: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/8.jpg)
Two techniques compared Each driven by execution profiles which
indicate execution frequency of program elements.
1.Coverage Based-Techniques2.Distribution-Based Techniques
![Page 9: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/9.jpg)
“Select test cases to maximize the proportion of program elements of a given type”◦ Attempts to cover as many elements of the
program as possible with the fewest number of test cases.
◦ Instance of a set-cover problem.
Algorithm◦ Each iteration selects a test case which covers the
largest number of program elements not covered by the previously selected tests.
![Page 10: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/10.jpg)
Clustering technique◦ Test cases are clustered and a test case from each
cluster can be selected to represent the group.◦ Created by observing execution profiles as patterns with
n dimensions. ◦ Each dimension represents the execution count of a basic
block of code.
Also uses failure-pursuit sampling ◦ Audits test cases near failures using a k-nearest neighbor
approach. ◦ This allows cases similar to the errors to be checked.
![Page 11: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/11.jpg)
Profiles characterize test executions by keeping track of execution frequencies of program elements.
The study takes into account eight types of profiles.◦ Generated using Byte Code Engineering Library to
examine the byte code of Java programs.◦ It also uses an existing tool the authors created
for dynamic information flow analysis.
![Page 12: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/12.jpg)
Method Calls (MC) ◦ contains a count of how many times a method M
was called.
Method Call Pairs (MCP)◦ a count of how many times a method M1 called a
method M2.
Basic Blocks (BB)◦ A count of how many times a given basic block of
code was executed.
![Page 13: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/13.jpg)
Basic Block Edges (BBE)◦ A count of how many times a basic block B1
branches to basic block B2.
Def-use pairs (DUP)◦ A count of how many times a variable definition is
defined and then later used.
All of the above combined (ALL)◦ Combination of all the above models.
![Page 14: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/14.jpg)
More complex profile types
Information flow pairs (IFP)◦ Count of how many times a variable x flowed into
variable y.
Slice Pairs (SliceP)◦ For each statement pair s1 and s2, s1 occurs
before s2 in at least one slice.
![Page 15: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/15.jpg)
Basic Coverage Maximization Cluster Filtering (One-per cluster sampling) Failure-Pursuit Sampling Simple Random Sampling
Empirical StudyEmpirical Study
![Page 16: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/16.jpg)
Ties◦ “different tests that each covers the maximal
number of program elements not covered by previously selected tests”
Ran 1,000 times per program/profile type Randomly selected order of the tests Recorded
◦ Number of tests selected◦ How many failures and defects detected
Basic Coverage Basic Coverage MaximizationMaximization
![Page 17: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/17.jpg)
Proportional Binary Metric and Agglomerative Hierarchical Clustering
Number of clusters varied to correspond to a range of percentages of the size of the test suite
Procedure1. Clustered into c clusters based on their profiles2. One test randomly selected from each cluster3. Recorded number of failures and defects
Run 1,000 times Failure Pursuit: Check 5 nearest neighbors
Cluster Filtering and Failure Cluster Filtering and Failure Pursuit SamplingPursuit Sampling
![Page 18: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/18.jpg)
Randomly select test without replacement Record number of failure-inducing tests and
defects Ran 1,000 Times
Simple Random SamplingSimple Random Sampling
![Page 19: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/19.jpg)
Subject Programs and Test Subject Programs and Test SuitesSuites
![Page 20: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/20.jpg)
28,639 lines of code Jacks Test Suite
◦ 3,140 tests◦ 233 cause failures
javac javac Java CompilerJava Compiler
![Page 21: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/21.jpg)
52,528 lines of code XML Conformance Test Suite
◦ Used 1,667 tests of 2,000 Difficult to determine pass/fail of dropped tests
◦ 10 cause failures◦ Only checks syntax
Xerces Xerces XML parserXML parser
![Page 22: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/22.jpg)
Test compliance with Java Language Specification
1,000 files (tests) from Google Groups◦ Failed on 47 of test cases
TidyHTML Syntax CheckerTidyHTML Syntax Checker
![Page 23: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/23.jpg)
Defects that caused errors were traced Results:
◦ Average percentage of defects that they revealed over a number of replicated applications viewed as a function of the number of tests selected
◦ Compared with respect to how often they reveled specific defects
AnalysisAnalysis
![Page 24: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/24.jpg)
Basic Coverage Maximization Results
![Page 25: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/25.jpg)
Basic Coverage Maximization Results
![Page 26: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/26.jpg)
Several defects revealed in 1,000 replications
Some defects only revealed when SliceP and IFP maximized
“Maximization with one type of profile revealed defects that were not revealed with another type of profile that seems to be more detailed.”
ResultsResults
![Page 27: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/27.jpg)
Simpler profile types (i.e. MC, MCP, BB, BBE, and DUP) revealed more defects than IFP
“Information Flow Pairs are recorded only when a variable is actually defined (assigned a value), but some defects may be triggered without executing such an operation.”
AnomaliesAnomalies
![Page 28: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/28.jpg)
Distribution Based Filtering Distribution Based Filtering ResultsResults
![Page 29: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/29.jpg)
Distribution Based Filtering Distribution Based Filtering ResultsResults
![Page 30: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/30.jpg)
Distribution Based Filtering Distribution Based Filtering ResultsResults
![Page 31: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/31.jpg)
Distribution Based Filtering Distribution Based Filtering ResultsResults
![Page 32: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/32.jpg)
Distribution Based Filtering Distribution Based Filtering ResultsResults
![Page 33: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/33.jpg)
Distribution Based Filtering Distribution Based Filtering ResultsResults
![Page 34: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/34.jpg)
Programs too broad Did not debug programs enough Wrongly classified defects Assumes size of the final set of tests is an
accurate measure of cost
Threats to ValidityThreats to Validity
![Page 35: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/35.jpg)
Time and space increases with level of profile detail
Time for collecting profile information, longer than time needed for analysis
Observations Cost and Observations Cost and AnalysisAnalysis
![Page 36: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/36.jpg)
Coverage maximization, One-Per-Cluster Sampling, and Failure Pursuit Sampling more effective than Random Sampling when proportion of failure high
Coverage maximization based on complex profiles revealed most defects
ConclusionsConclusions
![Page 37: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/37.jpg)
One-per-cluster sampling and failure pursuit did not clearly perform better than coverage maximization
No clear performance difference between one-per-cluster and failure pursuit sampling
ConclusionsConclusions
![Page 38: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/38.jpg)
Empirically evaluate test case filtering techniques
Compare with respect to:◦ Effectiveness for revealing defects◦ Simple Random Sampling
Complex profiles such as IFP and SliceP justifiable for when large number of tests necessary
ConclusionsConclusions
![Page 39: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/39.jpg)
Pros◦ Describes a good way to analyze programs.
◦ Uses profiles to help minimize complexity for only those most meaningful code chunks.
Cons◦ Programs tested were just compilers and syntax
checkers. ◦ Graphs could have better captions explaining
what is occuring0
![Page 40: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows](https://reader035.vdocuments.net/reader035/viewer/2022062809/568159a1550346895dc6f193/html5/thumbnails/40.jpg)
Have only one Test Suite◦ Several different program types that can be
tested with same suite◦ Eliminates an additional variable
Select several types of programs
SuggestionsSuggestions