The Impact of Test Case Reduction and Prioritization on Software Testing Effectiveness

Saif ur Rehman Khan, Inayat ur Rehman, Saif Ur Rehman Malik Department of Computer Science

COMSATS Institute of Information Technology Islamabad, Pakistan

{saif_rehman, Inayat, saif_ur_rehman@comsats.edu.pk}

Abstract- Software testing is critical but most expensive

phase of Software Development Life Cycle (SDLC). Development organizations desire to thoroughly test the software. But this exhaustive testing is impractical due to resource constraints. A large number of test suites are generated using automated tools. But the real challenge is the selection of subset of test cases and/or high order test cases crucial to validate the System Under Test (SUT). Test case reduction and prioritization techniques help test manager to solve this problem at a little cost. In this paper, we investigate their impact on testing process effectiveness using previous empirical studies. The results indicate that these techniques improve the testing effectiveness significantly. At the end, a case study is presented that suggests different useful combinations of these techniques, which are helpful for different testing scenarios.

Index Terms-- Software testing, exhaustive testing, test case

reduction, test case prioritization.

I. INTRODUCTION

Software testing is critical but most expensive phase of SDLC [20]. Every development organization desires to thoroughly test the developed system with all infinite combinations of values for program variables [1]. According to Rothermel et. al. [5], a product of about 20,000 lines of code requires seven weeks to run all its test cases and costs several hundred thousands dollars to execute them. Hence, it is impractical due to resource constraints. On the other hand, optimal testing ensures the judicious utilization of the testing resources. Out of plethora of optimal testing strategies, both test case reduction and prioritization plays an important role in achieving the optimal testing objective.

According to IEEE standard, a test case is: “A set of test inputs, execution, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement” [3]. A test suite is a combination of test cases. Generally, a large number of test suites are generated according to some criteria. Similarly due to formalization of requirements and design documents, test manager can automate the test execution and evaluation process [7]. But the real challenge is the selection of subset of test cases and/ or high order test cases crucial to validate the SUT. Test case reduction and prioritization techniques help test manager to solve this problem at a little cost [4], [5]. Eventually, it helps

to minimize the test case execution, management and storage cost.

The aim of test case reduction techniques is to compute a small representative set of test cases by removing the redundant and obsolete test cases from test suites [21]. Here we consider code coverage-based redundant test cases. Note, a test case may be redundant for one testing criterion, whereas it can be important with respect to another criterion. For example, given two test cases: tc1{x=1} and tc2{x=3}; given a predicate {if (x > 1)}. For the statement coverage criterion, one of tc1 and tc2 is redundant; while for branch coverage criterion, neither of them is.

The goal of this work is to find a suitable strategy for effective test case management in the context of optimal testing and also, to judge their impact on overall testing process. For this purpose, we use previous empirical studies related to these test case optimization techniques. The main contributions of this work are: 1. a brief review of test case reduction and prioritization

techniques, 2. analysis of previous empirical studies conducted by

different researchers related to these test optimization techniques, and

3. a case study that suggest some useful combination of these techniques with feasible scenarios.

II. TEST CASE REDUCTION (TCR)

TECHNIQUES

TCR techniques help to find out effective subset of test cases during maintenance phase and are helpful in reducing the testing cost. Their main focus is on reducing test suites to obtain a subset that yields equivalent coverage with respect to some criteria [11]. The optimal test suite reduction problem may be stated as follows [8]:

Given: Test suite TS, a set of test cases requirements r1, r2, …, rn that must be satisfied to provide the desired test coverage of the program, and subsets of TS1, TS2, TS3,…, TSn, one associated with each of the ri’s such that any one of the test cases tj belonging to Ti can be used to test ri. Problem: Find a representative set of test cases from TSi that satisfies all of the ri.

2009 International Conference on Emerging Technologies

978-1-4244-5632-1/09/$26.00 ©2009 IEEE 416

A representative set of test cases that satisfies all of the ri

must contain at least one test case from each TSi; such a set is called a hitting set of the group of sets TS1, TS2, TS3, …, TSn. [6].

In literature, a large number of coverage-based test case reduction techniques have been proposed by different researchers using statement-coverage, branch-coverage, path-coverage, or def-use association criterion [8], [21], [2], [22], [23], [24], [25]. The test reduction technique was initially proposed by Harrold et. al. [8]. It effectively finds out the representative test cases from large pool of test suites based on their cardinality (i.e. number of program statements covered by a test case. Ronne [2] presents a reduction technique called as Multi-Hit Minimization algorithm, which overcome the limitations of Harrold et. al. [8] technique. Recently, Khan et. al. [21] proposed a new TCR technique, called as TestFilter, which uses statement coverage criteria to determine representative test cases.

By using TCR techniques, test manager, 1) spends less time executing test cases, 2) examines test results, and 3) manages the data associated with testing [2]. However, it may be possible that by employing these techniques, we may loose the fault revealing test cases during test reduction process. We investigate this crucial question in Section 4.

III. TEST CASE PRIORITIZATION (TCP)

TECHNIQUES

Cheng et. al. report that National Institute of Standards and Technology (NIST) found that software defects cost the U.S. economy 59.9 billion dollars annually and also 22 million dollars could be saved through earlier and more effective defect detection [9]. These techniques schedule test cases for execution in an order that attempts to maximize some objective function [5].

A number of objective functions are applicable. One such function involves rate of fault detection, (i.e. how quickly faults are detected within the testing process). An improved rate of fault detection can provide faster feedback about SUT and let test manager begin locating and correcting faults earlier than might otherwise be possible. The test suite prioritization problem can be stated as [12], [15]:

Given: Test suite T, a set of permutations of T, PT, a function f from PT to the real numbers. Problem: Find pt є PT such that (∀pt´) (pt є PT′)

[f (pt) ≥ f(pt′)]

Here, PT is the set of possible prioritizations of T, and f is an objective function that, applied to any such order, yields an award value for the order [12]. Note, the high award values are always preferred than the lower one. There are many possible goals for prioritization, for example, testers may wish to increase the rate of fault detection of test suites, increase the coverage of the SUT at a faster rate, increase their confidence in the reliability of the SUT at a faster rate, or increase the likelihood of revealing faults related to specific code changes earlier in the testing process [5]. In the

definition of the TCP problem, f represents a quanti-fication of such a goal [12].

There are three main test prioritization techniques [5]: 1) control techniques, 2) statement-level techniques, and 3) function-level techniques. Most proposed techniques are code based. For example, test cases can be prioritized in terms of the number of code statements, basic blocks, or functions they covered. Note that performance of TCP techniques varies significantly with program attributes, change attributes, test suite characteristics and their interaction [14].

A metric called APFD (Average Percentage of Faults Detected) is used over the life of the test suite to measure, how rapidly a prioritized test suite detects faults [15]. Its values range from 0 to 100; higher APFD values means faster fault detection rates [5]. It relies on two assumptions: 1) all faults have equal severity, and 2) all test cases have equal cost. However, in reality faults severity and costs can vary due to different testing environment factors. To overcome this limitation, Elbaum et. al. [12] presents a new metric for assessing the rate of fault detection of prioritized test cases, that incorporates varying test cases and fault costs. Rothermel and Elbaum [13] identified three most prominent factors, which can affect the prioritization cost-effectiveness. The factors are, 1) test suite granularity, 2) modification location, and 3) level of analysis. IV. PREVIOUS EMPIRICAL STUDIES RELATED TO TCR

AND TCP TECHNIQUES

Different researchers have performed some empirical studies to investigate the issues related to test case reduction and prioritization techniques. In this section, we will briefly discuss these empirical studies and then judge their overall impact on testing effectiveness. I.e. whether they really improve the effectiveness or test manager is just wasting development resources by employing these techniques. Note, the first three studies are conducted to review the impact of reduction techniques, while other four are related to prioritization techniques.

Test manager always desire to achieve high percentage of size reduction and low percentage of fault-detection, while employing any TCR technique. The generic TCR procedures which were adopted by the researchers in their experimental studies are as follows [8]:

1. Select a test suite. 2. Create a set of faults detected by the original test suite. 3. Reduce the test suite using the TCR technique. 4. Form a set of faults detected by the reduced test suite. 5. Calculate the percentage size reduction and percentage

fault detection. For each reduction experiment, researcher can estimate the

percentage size reduction [11], as:

100 × (1 – SizeReduced / SizeFull), and percentage fault detection reduction, as [11]:

100 × (1- FaultsDetectedReduced / FaultsDetectedFull).

A. Harrold et. al. (TCR)

417

Harrold et. al. [8] performed three separate experiments to determine the possible reduction of the test suite size using TCR technique at different stages of software life cycle (i.e. during program development, maintenance, and program enhancements). Different program of varying size were used for this study. Table 1 briefly describes the results of reduction techniques, applied during program development.

TABLE 1

REDUCTION DURING PROGRAM DEVELOPMENT [8]

Pr

oced

ure

Sour

ce

Line

s

Du-

pairs

O

rigin

al

Test

Cas

es

R

edun

dant

Te

st C

ases

% R

educ

tion

of

Tes

t C

ases

trityp 21 39 16 3 18.7 Atof 17 63 2 1 50.0

getop 19 33 5 3 60.0 calc 33 3 11 4 36.4 qsort 20 43 4 2 50.0 sqroot 19 13 5 2 40.0

The main results of the studies were:

i. The reduction ranges from 19% to 60%, when actually developing the program.

ii. The reduction ranges from 20% to 55% for implementation changes, and from 0% to 40% when making program enhancements.

B. Wong et. al. (TCR)

Wong et. al.[10] performed an empirical study to determine the effect of test suite reduction on fault detection effectiveness. Ten common C UNIX utility programs were used for the experiment. Overall 2549 test cases were generated for the experiment using different testing techniques. A tool called ATAC was used. Finally, researcher used null- hypothesis testing to determine whether the minimized test suites had same fault detection capabilities as compared to un- reduced test suites. The following observations were made, on average reduction in fault detection effectiveness due to reduction based on all-uses coverage:

i. There is no reduction for test suites with (50-55) % block coverage.

ii. The average effectiveness reduction for test suites with (60-65)%, (70-75)%, (80-85)% and (90-95)% block coverage is less than 0.39%, 0.23%, 5.05% and 6.55%, respectively.

Similarly, the following observations were made, on average test size reduction due to all-uses reduction:

i. For every test set with (80-85)% and (90-95)% block coverage, there is size reduction.

ii. The maximal size reduction for test sets with (50-55)%, (60-65)%, (70-75)%, (80-85)%, and (90-95)% block coverage is 6.55%, 17.94%, 31.17%, 50.02%, and 68.23%, respectively.

iii. Null-hypothesis testing showed that test suites

reduced by ATAC retained a size effectiveness advantage over their corresponding randomly reduced test suites.

C. Rothermel et. al.(TCR)

Rothermel et. al. [4] performed two experiments to determine costs and benefits when test suites have a wider range of sizes. Six C programs were used as subjects in first experiment, while only one program was used for second experiment. Each program contains with many faulty versions. Total 32,742 test cases were generated for these two experiments. Then for each subject program, TCR techniques were applied to each of the test suite of the program. The main results of the study were:

i. Reduction can produce substantial savings in test suite size on coverage-adequate, coverage-redundant test suites. The results also indicate that as test suite size increases, the savings (in terms of size) produced by the TCR also increases.

ii. TCR can compromise the fault detection effective-ness of edge-adequate, coverage redundant test suites. Moreover, the results also suggest that as test suite size increases, the reductions in the fault detection effectiveness of those test suites will also increases.

iii. Reduced test suites find more number of faults than their randomly reduced counterparts.

D. Rothermel et. al. (TCP)

Rothermel et. al. [15] conducted an experimental study to determine, whether test case prioritization really improve the rate of fault detection of test suites? The researchers used 7×9 (seven C programs and nine prioritization techniques) factorial design with 1000 APFD measures per cell. For each subject program the researchers created a large tests pool using black box and white box techniques. Ten people performed fault-seeding process with out knowledge of each other’s work. There was exactly one seeded fault in each faulty version of each subject program. For each subject program, the prioritization techniques were applied to each of the 1000 sample test suites, yielding 8000 prioritized test suites. The APFD values of these prioritized test suites were calculated. The main results of their study were:

i. TCP techniques improves the rate of fault detection of test suites,

ii. Additional FEP (Fault Exposing Potential) out-performed all prioritization techniques based on coverage. Furthermore, total FEP prioritization outperformed all coverage-based techniques rather than total branch coverage.

E. Rothermel et. al. (TCP) Rothermel et. al. [5] conducted an empirical study to

investigate the effectiveness of “version specific prior-itization”. Researcher used eight C programs and fourteen prioritization techniques for the study. The overall results of the study were:

418

O

bjec

ts

N

o. o

f V

ersi

ons

N

o. o

f C

lass

es

N

o. o

f Tes

t C

ases

(Tes

t- C

lass

Lev

el)

N

o. o

f Tes

t C

ases

(Tes

t-M

etho

d Le

vel)

ant 9 627 150 877 xm 4 143 14 83 jm 6 389 28 78

jto 4 50 11 128

i. Version-specific TCP techniques improve the rate of fault detection during regression testing.

ii. Coarse-grained techniques (function level) are less costly, and involve less intrusive instrumentation than fine-grained techniques (statement level). However, statement level techniques can produce high effectiveness gains.

iii. The optimal techniques were significantly better than all other techniques. Suggesting the researchers that there is still room for improvement in existing TCP techniques.

F. Elbaum et. al. (TCP)

Elbaum et. al. [14] conducted an empirical study, which provides the guideline to test manager about the selection of appropriate prioritization technique with respect to particular testing environment.

The researchers used eight C programs. For prioritization, function-coverage based techniques were used. The concept of prioritization instances was introduced for comparison of two TCP techniques. Five cost-benefit threshold values (0%, 1%, 5%, 10%, and 25%) were used for comparison purpose. Test manager selects prioritization technique, which has greater instance value with respect to cost-benefit threshold.

G. Hyunsook and Gregg (TCP)

Previous 3 TCP experiments were conducted on C programs. Effectiveness of TCP techniques for Object Oriented (OO) program was still needed to be checked.

Saff and Ernst [16] and Do et. al. [17] considered TCP techniques for Java programs. Recently, experimental study by Hyunsook and Gregg [18] measured the ability of TCP techniques to improve fault detection rate, using 4 Java programs with JUnit test cases. Table 2 depicts their experimental objects.

TABLE 2

EXPERIMENTAL OBJECTS AND TEST DATA [18]

V. ANALYSIS OF PREVIOUS EMPIRICAL STUDIES RELATED TO TCR AND TCP

The overall results of experiments related to reduction techniques are summarized in Table 3. Some qualitative terms are used in Table 3. E.g. terms used in column 4 represents the size reduction of test cases achieved by employing TCR techniques. Where: High means more than 50%, and Good means less than 50% but more than 40%. Here column 5 represents the effectiveness-reduction of test

suite due to reduction techniques. Here Low means less than 7% reduction in test suite effectiveness as a result of reduction techniques.

TABLE 3

EMPIRICAL STUDIES RELATED TO TCR

Ex

perim

ent N

o.

Su

bjec

t

Prog

ram

s

N

o. o

f Tes

t

Cas

es

Size

Red

uctio

n

Ef

fect

iven

ess

Red

uctio

n

A 17 213 High Low B 10 2549 High Low C 7 32742 Good Low

Table 4 illustrates the summarized results related to

prioritization techniques. The terms used in column 5 of Table4 represents the effect of rate of fault detection achieved by the test manager by using prioritization techniques. Here Good means: all TCP techniques increase the fault detection rate, while High means some techniques achieved nearly optimal-level results after prioritization.

TABLE 4

EMPIRICAL STUDIES RELATED TO TCP

Expe

rimen

t No.

Su

bjec

t

Prog

ram

s

N

o. o

f Tes

t

Cas

es

N

o. o

f TC

P Te

chni

ques

Ef

fect

on

faul

t D

etec

tion

D 7 21807 9 High E 8 35392 14 Good F 8 3918191 14 High G 4 1369 7 Good

VI. CASE STUDY

In order to judge the usefulness & impact of TCR and TCP techniques, we conduct an experimental study. We select Triangle program for this study. It accepts three positive integer values as inputs (that represent three sides of the triangle) from user and determines whether it is a valid triangle or not. Finally on the basis of this input it displays corresponding message as follows:

• If three sides are equal : Equilateral triangle • If any two sides are equal : Isosceles triangle • If all sides are different: Scalene triangle • Otherwise: Invalid triangle

Note, we take the subject data from [20]. Total 49 test cases are developed using black box and white box techniques for validation purposes. Next we mark each condition/statement of the program as a separate testing requirement.

For example, consider Table 5, where first condition is marked as requirement 1 (Req1) and also tc1 to tc49 means that all these test cases are effective to test this requirement. Total 13 requirements are marked for this program.

419

It is a natural phenomenon that a high statement-coverage test case has high probability of determining fault (if any) from those statements. So it is a wise approach to execute test cases in the order in which TCP techniques determines them. So, next we apply TCP technique to determine the execution order of all developed 49 test cases. For that purpose, we use statement-coverage prioritization technique.

TABLE 5

COVERAGE REQUIREMENTS FOR TRIANGLE PROGRAM

Pr

ogra

m

Stat

emen

ts

R

eq’s

(Req

i)

Te

st C

ases

(tc i)

in

Ass

ocia

ted

Set

if! (a > 0 && b > 0 && c > 0) Req1 tc1 – tc49 cout<<”not a triangle”<<endl; Req 2 tc26 – tc39

if! ((2*a) < (a+b+c) && (2*b) < Req 3 tc1 – tc49 cout<<”not a triangle”<<endl; Req 4 tc11 – tc49

if (a == b) Req 5 tc1 – tc49 if (b == c) Req 6 tc1 – tc49

cout<<”triangle is equilateral Req 7 tc10 cout<<”triangle is isosceles Req 8 tc7

if (a == c) Req 9 tc1 – tc49 cout<<”triangle is isosceles Req 10 tc8

if (b == c) Req 11 tc1 – tc49 cout<<”triangle is isosceles Req 12 tc9

cout<<”triangle is scalene ”<<endl; Req 13 tc1, tc2, tc3, tc4, tc5, tc6

Table 6 depicts our prioritization results. It shows that tc29 has high priority as it covers three statements of the program. While tc26 to tc28, tc49, and tc30 to tc39 has 2nd high priority, as they all cover two statements of the program, each. The remaining test cases have least priority. Finally, we apply following two combinations of TCR and TCP techniques. A. Combination 1: TCR + TCP

In this combination, first we identify representative set of test cases using TCR techniques and then we determine their execution order using TCP techniques [26].Table 6 shows the execution order of these representative test cases. So by using this combination testing team is able to determine the execution order of representative test cases.

TABLE 6

STATEMENT-COVERAGE BASED PRIORITIZATION

Priority Test Case

1 tc29 2 tc26, tc27, tc28, tc30-tc39, tc49 3 tc1-tc25, tc40-tc48

Note, in our case we have only six representative test cases, but there may be a scenario where this number exceeds to thousands of test cases. This puts lot of pressure on test manager to determine their right execution order. Test manager can pick these test cases using any adhoc or random technique, but it might be possible that least

effective test cases are selected first. When the turn of high coverage test cases come, testing team thinks themselves lacking of resources (i.e., testing time, etc).

Ultimately they have to stop this testing process, and finally they found very few faults from the SUT. This certainly degrades the overall testing process. By using this combination, testing team can avoid such degradation.

B. Combination 2: TCP + TCR

This combination iteratively optimizes the large number of test cases. In this combination, first we identify high priority test cases using TCP techniques and finally we determine the representative test cases using TCR techniques from the computed prioritized test cases [26]. Note, we use statement-converge criterion for prioritization purposes. For example, (Table 6) where tc29 has high priority. While next high priority test cases (i.e. tc26, tc27, tc28, tc49, and tc30 to tc39). As tc29 is a single test case, so we take tc29 and next high priority test cases for reduction purpose. Finally, we get t26 as their representative. Next, we take next higher priority test cases (i.e. tc1 to tc25 and tc40 to tc48) and by applying reduction technique we found tc1, tc7, tc8, tc9, and tc10 as their representative test cases.

During software testing, test managers may want to analyze their testing process earlier, by detecting critical faults earlier. By using this combination test manager first identify higher priority test cases say ‘n’ and then using reduction technique, they found their representative test cases. This process continues until we finished least priority level test cases. By doing this, test manager can parallely manage test optimization and execution (& validation) process.

VII. CONCLUSION AND FUTURE WORK

In this paper, we examined the impact of test case reduction and prioritization on software testing effectiveness. The analysis of previous empirical studies shows that these techniques significantly improve the testing effectiveness as compared to any other available test optimization technique. They help test manager in achieving nearly exhaustive testing level by selecting effective test cases using limited testing budget. We also discuss the practical applicability of reduction and prioritization techniques, and present some useful combinations appropriate for different testing environments. Size effectiveness advantage is retained in TCR techniques as compared to randomly reduced test suites [10].

Our experimental study results show that these techniques improve the testing effectiveness. We have also observed some situations, where the combination of these techniques also helpful in improvement of overall testing process. But before using any reduction or prioritization technique, test manager needs to analyze its applicability according to the assigned testing environment. Available testing time also plays a very critical role in selection of these techniques. For example, test manager can face following three situations during testing:

420

i. In case of safety critical systems, if there is any danger of losing fault-detection of test suites using reduction techniques [19]. The obvious choice for test manager is to use prioritization techniques.

ii. If test manager has limited testing resources and project is also behind the planned schedule. TCR technique seems to be the wise solution of this situation.

iii. If test manager has sufficient available time to run relatively more test cases and fault detection is also very critical. So in this situation test manager can employ prioritization techniques.

In literature, few of the TCR techniques have been applied on object oriented systems. Further research should be conducted to determine the behavior of TCR techniques on object oriented systems. In previous empirical studies related to TCP the researchers only considered one objective i.e. rate of fault detection. In order to achieve some generic results, other prioritization objectives also need to be investigated. Moreover, additional studies of varying artifacts need to be conducted for reduction techniques. Truly generalizable results can only be achieved through additional understanding and careful control of the various factors (e.g., subject program, number of faults, number of test cases, testing time, etc.) that affect the cost-effectiveness of reduction techniques. Through the results reported in this paper, we hope to provide test manager with efficient and cost-effective test optimization techniques.Ultimately, this improves the testing effectiveness to some extent.

REFERENCES

[1] W. R. Adrison, M. A. Branstad, and J. C. Cherniavsky, ”Validation, verification, and testing of computer software,” In ACM Computing Sur. vol. 14, No. 2, pp. 159–92, June 1982.

[2] J. V. Ronne, “Test suite minimization: an empirical investigation,” Retrieved from URL: http://www.ics.uci.edu/~jronne/pubs/jvronne-uhc-thesis.pdf

[3] Institute of Electrical & Electronics Engineers, Standard 610(1990), Reprinted in IEEE Standard Collection: Software Engineering, 1994 Edition.

[4] G. Rothermel, M. J. Harrold, and C. Hong, “Empirical studies of test suite reduction,” In Journal of Software Testing, Verification, and Reliability, vol. 12, No. 4, December 2002.

[5] G. Rothermel, R. H. Untch, and M. J. Harrold, ”Prioritizing test cases for r egression testing,” In IEEE Trans. on Software Eng. vol. 27, No. 10, pp. 929–948, October 2001.

[6] I. Pomeranz, and S. M. Reddy, ”On the compaction of test sets produced by genetic optimization,” In Proceedings of IEEE 5th Asian Test Symposium (ATS’97), Akita Japan, November 1997.

[7] J. C. Gurguillo, M. Llamas, and M. J. Fernandez, ”Heuristic-driven techniques for test case selection,” In Electronic Notes in Theoretical Computer Science, No. 2, 2002.

[8] M. J. Harrold, R. Gupta, and M. L. Soffa, ”A methodology for controlling the size of test suite,” In ACM Trans. on Software Eng. and Methodology (TOSEM), NY USA, pp. 270–285, 1993.

[9] C. Cheng, A. Dunitrescu, and P. Schroeder, ”Generating small combinatorial test suites to cover input-output relationships,” In Proceedings of the 3rd International Conference on Quality Software (ICQS’03), Dallas Texas. 2003.

[10] W. E. Wong, J. R. Horgan, and S. London, “Effect of test set minimization on fault detection effectiveness,” In Proceedings of IEEE 17th International Test Conference Software Eng. (ITCSE’95), Seattle Washington USA, pp.41–50, April 1995.

[11] S. McMaster, and A. M. Memon, “Call stack coverage for test suite reduction,” In Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM'05), Budapest, pp. 539–548, Sept. 2005.

[12] S. A. Malishvesky, and G. Rothermel, “Incorporating varying test costs and fault severities into test case prioritization,” In Proceedings of the 23rd International Conference on Software Eng. (ICSE’01), Toronto Ontario Canada, pp. 329–338, 2001.

[13] G. Rothermel, and S. Elbaum, “Putting y our b est tests forward,” Published by the IEEE Computer Soc. pp. 22–25, 2003.

[14] S. Elbaum, G. Rothermel, S. Kanduri, and A. G. Malishevsky, “Selecting a cost-effective test case prioritization technique,” Technical Report 03-01-01, Department of Computer Science and Engineering, University of Nebraska -- Lincoln, January 2003.

[15] G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold, “Test case prioritization: an empirical study,” In Proceedings of the IEEE International Conference on Software Maintenance (ICSM’99), Washington DC USA, pp. 179–189, 1999.

[16] D. Saff, and M. D. Ernst, “Reducing wasted development time via continues t esting,” In Proceedings of International Symposium of Software Reliability Engineering (ISSRE’03), pp. 281–292, 2003.

[17] H. Do, G. Rothermel, and A. Kinneer, “Empirical studies of test case prioritization in a JUnit testing environment,” In Proceedings of International Symposium of Software Reliability Engineering (ISSRE’04), pp. 113–124, 2004.

[18] H. Do, and G. Rothermel, “A controlled experiment assessing test case prioritization techniques via mutation faults,” In Proceedings of IEEE International Conference on Software Maintenance (ICSM’ 05), pp. 411–420, 2005.

[19] M. P. E. Heimdahl, and D. J. George, ”Test-suite reduction for model based test: effects on test quality and implications for testing,” In Proceedings o f the 19th International Conference on Automated Software Engineering (ASE’04), 2004.

[20] S. R. Khan, A. Nadeem, and A. Awais, “TestFilter: a statement coverage based test case reduction technique,” In Proceedings of 10th IEEE International Multitopic Conference (INMIC’06), pp. 275–280, Islamabad, Pakistan, December, 2006.

[21] M. Hennessy, and J. F. Power, ”An analysis of rule coverage as a criterion in generating minimal test suites for grammar based software,” In Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering (ASE’05), Long Beach, CA, USA, pp. 104–113, November 2005.

[22] A. Kandel, P. Saraph, and M. Last, “Test cases generation and reduction by automated input-output analysis,” In Proceedings of 2003 IEEE International Conference on Systems, Man and Cybernetics (ICSMC’3), Washington, D.C., October, 2003.

[23] B. Vaysburg, L. H. Tahat, and B. Korel, “Dependence analysis in reduction of requirement based test suites,” In Proceedings of the 2002 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’02), Roma Italy, pp. 107–111, 2002.

[24] J. A. Jones, and M. J. Harrold, ”Test-suite reduction and prioritization for modified condition/decision coverage,” In IEEE Trans. on Software Engineering (TSE’03), Vol. 29, No. 3, pp. 195–209, March 2003.

[25] D. Jeffrey, and N. Gupta, ”Test suite reduction with selective redundancy,” In Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM’05), Budapest, Hungary, pp 549–558, September 2005.

[26] D. Leon, and A. Podgurski, ”A comparison of coverage based and distribution based techniques for filtering and prioritizing test cases,” In IEEE Proceedings of 14th International Symposium on Software Reliability Engineering (ISSRE’03), Denver, Colorado, 2003.

421