pairwise test suite generator tool based on harmony …

PERPUSTAKAAN UMP

DII ID ID ID 11111 DD 1111 00001 03246

PAIRWISE TEST SUITE GENERATOR TOOL

BASED ON HARMONY SEARCH ALGORITHM

(HS-PTSGT)

LAI Yl XIANG

BACHELOR OF sCIwir

(SOFTWARE ENGINEERING)

UNIVERSITI MALAYSIA PAHANG

ABSTRACT

In the era of technology, software systems are around us in our daily life. Thus, a good

system is needed to ensure the well function of gadget and devices with software

embedded. However, exhaustive testing that required lot of time and financial resources

is normally impossible. This thesis is about the research on developing a pairwise test

suite generator tool based on Harmony Search Algorithm (HS-PTSGT) to generate

optimum test suite. Test suite which considers all interaction of at most two factors will

be generated from two-valued input parameters that are not more than 500. Users are

allowed to enter parameter and values through the graphical user interface, and test

cases will be generated through the execution of Harmony Search Algorithm involved

random initialization of harmony memory and harmony memory improvisation. From

the comparison with other existing pairwise testing tools in term of size, it is shown that

the strategy used in HS-PTSGT can effectively generate optimum test suite with

minimized test data and performed better on certain parameter size.

V

ABSTRAK

Pada era teknologi mi, system perisian berada di sekeliling kita. Jadi, sistem perisian

yang berkualiti amat diperlukan untuk memastikan peralatan yang terbenam perisian

sentiasa berfungsi dengan baik. Zalimnya, ia adalah mustahil untuk kita menguji sesuatu

sistem dengan sepenuhnya memandangkan masa dan wang yang banyak diperlukan.

Tesis mi akan membincang tentang uji kaji untuk menciptapairwise test suite generator

tool based on Harmony Search Algorithm (HS-PTSGT) bagi menghasilkan koleksi set

pengujian yang optimum. Koleksi set pengujian yang mempertimbangkan semua

interaksi antara dua faktor tidak melebihi 500 parameter akan dihasilkan. Pengguna

boleh masukkan parameter berserta dengan nilainya ke dalam HS-PTSGT, dan koleksi

set pengujian akan dihasilkan setelah operasi Harmony Search Algorithm yang

melibatkan harmony memory initialization dan harmony memory improvisation.

Daripada keputusan kajian, HS-PTSGT terbukti mampu menghasilkan koleksi set

pengujian yang optimum dengan berkesan dan berprestasi lebih baik daripada pairwise

testing tool yang lain di keadaan yang tertentu.

II

TABLE OF CONTENTS

Page

VERIFIED FORM j

DECLARATION

SUPERVISOR'S DECLARATION

ACKNOWLEDGEMENTS iv

ABSTRACT v

ABSTRAK vi

CONTENTS vii

LIST OF TABLES x

LIST OF FIGURES xii

LIST OF ABBREVIATIONS xv

CHAPTER 1 INTRODUCTION

1.1 INTRODUCTION 1

1.2 PROBLEM STATEMENT 3

1.3 GOAL AND OBJECTIVES 4

1.4 RESEARCH SCOPE 4

1.5 METHODOLOGY 4

vii

viii

1.6 THESIS ORGANIZATION 6

1.7 SUMMARY 7

CHAPETR 2 LITERATURE REVIEW

2.1 INTRODUCTION 8

2.2 OVERVIEW 8

2. 2. 1 Pairwise Testing 9

2. 2. 2 Harmony Search Algorithm 14

2.3 SURVEY OF EXISTING PAIRWISE STRATEGIES 15

2. 3. 1 'Genetic Algorithm (GA) 16

2. 3. 2 Ant Colony Algorithm (ACA) 18

2. 3. 3 In-Parameter-Oder (IPO) 19

2.4 SUMMARY 21

CHAPTER 3 METHODOLOGY

3.1 INTRODUCTION 22

3.2 METHODOLOGY 23

3. 2. 1 Requirement Planning 24

3.2.2 User Design 25

3. 2. 3 Construction 25

3.2.4 Cutover 25

3.3 HARDWARE AND SOFTWARE SPECIFICATION 26

CHAPTER 4 THE DESIGN OF HS-PTSGT

4.1 INTRODUCTION 22

4.2 DESIGN APPROACH 23

4. 2. 1 Non-Deterministic Output 29

4. 2. 2 Automated Parameter Mapping Supports 30

4.3 THE DEVELOPMENT OF HS-PTSGT 32

ix

4. 3. 1 Interaction List Generation Algorithm 34

4. 3. 2 HS-PTSGT Test Suite Generation Algorithm 39

4. 3. 3 Actual Parameter Mapping Algorithm 44

4.4 SUMMARY 45

CHAPTER 5 IMPLEMENTATION

5.1 INTRODUCTION 47

5.2 IMPLEMENTATION. OF PAIR COMBINATION 47

5.3 IMPLEMENTATION OF TEST SUITE GENERATION 50

5.4 IMPLEMENTATION OF ACTUAL PARAMETER MAPPING 52

5.5 SUMMARY 53

CHAPTER 6 RESULT AND DISCUSSION

6.1 INTRODUCTION 54

6.2 DEMONSTRATION OF IMPLEMENTATION 56

6.3 DEMONSTRATION OF MINIMIZED TEST SUITE SIZE 58

6.4 COMPARISON AGAINST OTHER PAIRWISE TESTING TOOLS 59

6.5 ANALYSIS OF HS-PTSGT THROUGH HMCR & PAR VALUES .70

6.6 SUMMARY 72

CHAPTER 7 CONCLUSION

7. 1 CONCLUSION 74

7.2 FUTURE WORK 75

FEFERENCES 77

APPENDICES 79

LIST OF TABLES

Table Number Page

2.1 Input Parameter and Value Illustration 9

2.2 Exhaustive Combination of 4 Inputs 10

2.3 2-Way Combination for AB 10

2.4 2-Way Combination for AC 11

2.5 2-Way Combination for AD 11

2.6 2-Way Combination for BC 12

2.7 2-Way Combination for BD 12

2.8 2-Way Combination for CD 13

3.1 Hardware Specification 26

3.2 Software Specification 27

6.1 iPhone Keyboard Input Specification 56

6.2 Pairwise Coverage for iPhone Keyboard Setting Input Specification 57

6.3 Input Specification for Ti until T8 58

6.4 Input Data Specification for 12 Tests 60

6.5 Result of the Generated Test Case in term of the Test Case Size 61

6.6 Results of Test Size Comparison 69

x

xi

6.7 Values Specification for HMCR and PAR Analysis 71

6.8 Result of the Generated Test Suite in term of Test Case Size 71

LIST OF FIGURES

Figure Number Page

1.1 Display Tab in Options Dialog for Micrasoft Word 3

1.2 Rapid Application Development (RAD) Model 5

2.1 Merging of AB, AC, AD, BC, BD and CD 13

2.2 Harmony Search Algorithm Flow Chart 14

2.3 Genetic Algorithm 17

2.4 Ant Colony Algorithm 18

2.5 Search Space 19

2.6 IPO_H - Algorithm for Horizontal Growth of a Test Suite by Adding

Values for New Parameters 20

2.7 IPO_V - Algorithm for Vertical Growth of a Test Suite by Adding

Values for New Parameters 20

3.1 Rapid Application Development (RAD) Model 24

4.1 Effect of HMCR and PAR in Generating Optimum Solution 29

4.2 Process of Mapping Parameter 31

4.3 Overview of HS-PTSGT Framework 32

4.4 The General Flow of HS-PTSGT 34

4.5 Pseudo-Code of Interaction List Generation Algorithm 35

xli

xlii

4.6 Number of Pairs Calculation 36

4.7 The Interaction Pairs of AB, AC and AD 37

4.8 Initial uncoveredlnteractionPairsSearch Array 38

4.9 Initial uncoveredlnteractionPairsCOunt Array 39

4.10 Pseudo-Code of HS-PTSGT Test Suites Generation Algorithm 39

4.11 The Evaluation of Covered Interaction Pairs for a Test Case 42

4.12 The Final Data Structure of uncoveredPairsSearch 43

4.13 Pseudo-Code of Actual Parameter Mapping Algorithm 44

4.13 Mapping Symbolic Values with Actual Parameter Values 45

5.1 Input Screen for User-Defined Inputs Before Parameter Values Input 48

5.2 Inputs for Hostel Survey 49

5.3 Example of Input Specification for HS-PTSGT 49

5.4 Parameter Values with Symbolic Notation 50

5.5 Harmony Memory Evaluation for Best Harmony Test Case 51

5.6 The Symbolic Final Test List 51

5.7 Final Test Suite Before and After Mapping Process 52

6.1 The Keyboard Setting for iPhone 55

6.2 Final Test Suite Generated by HS-PTSGT 56

6.3 Exhaustive Test Suite Size against HS-PTSGT Test Suite Size 59

6.4 Test Suite Result of 51 61

6.5 Test Suite Result of S2 62


xlv







6.13 Test Suite Result of S 1 67

6.14 Test Suite Result of S 1 68


6.16 The Pattern of Test Suite Size with Various HMCR and PAR Values 72

LIST OF ABBREVIATIONS

HM: Harmony Memory

HMS : Harmony Memory Size

HMCR : Harmony Memory Considering Rate

PAR : Pitch Adjustment Rate

GA: Genetic Algorithm

ACA : Ant Colony Algorithm

IPO : In-Parameter-Order

RAD : Rapid Application Development

CASE: Computer-Aided Software Engineering tools

xv

CHAPTER 1

INTRODUCTION

1.1 INTRODUCTION

In the era of technology, software systems are around us in our daily life.

Electronic gadgets and devices that we're using are embedded with software, any

bugs in these systems can cause the gadgets and devices to be malfunction. For

critical system, the failure bring by software system may cause the loss of life and

huge sum of money. Thus, a good system is needed to ensure the well function of

these gadget and devices.

Start from the time when the term 'bug' is used to describe flaw in a system by

the famous Thomas Alva Edison, defect in software system has become a threat in

software development (Meerts, J. and Graham, D., 2013). In software development

process, software testing plays an important role and required the most financial

resources. Software Testing known as the process of discovers errors in system

execution and validating the system against its specification (Myers, G. J., 1979).

I

During testing phase, test suite that with a collection of test cases are used to

explore defects by entering test data as input into the system. In order to have

effective testing on system, the generation of an optimum test suite is essential to

avoid exhaustive testing which are practically impossible. This research is to develop

a pairwise test suite generator tool based on Harmony Search Algorithm (HS-PTSGT).

Pairwise testing is a combinatorial technique used to reduce the number of test

case inputs to a system, consider all interactions of at most two factors (McCaffrey, J.

D., 2009). Pairwise testing normally begins by choosing individual inputs variables

using domain partitioning; the values are then permuted to ensure the consideration of

all pair interactions (Bach, J. and Schroeder, P. J, 2004). From the studies of Kuhn,

D.R. and Reilly, M.J. (2002), more than 70% of bugs caused by two or fewer

interaction condition able to be noticed.

Harmony search algorithm is a music-based metaheuristic optimization

algorithm, divined from the aim to search for prefect state of harmony in jazz

musician's improvisation process (Yang, X. S., 2009). Harmony Search algorithm

works as follow: stochastic generation of an initial population harmony vectors and

stored these vectors in a harmony memory. After that, new candidate harmony will be

created based on all solutions in the harmony memory using a memory consideration

rule, a pitch adjustment rule and a random re-initialization. Lastly, the new candidate

harmony is compared with the worst harmony vector in the harmony memory. The

new candidate harmony will replace the worst harmony and update the harmony

memory if it is better thanO the worst harmony. This process will repeat until certain

termination criterion is met. Harmony Search is a generalized optimization method for

continuous, discrete, and constrained optimization and has been applied to numerous

types of optimization problems. Further information will be discussed in details in

Chapter 2.

3

1.2 PROBLEM STATEMENT

Testing is required to find faults in a software product. For a big system,

exhaustive testing that required lot of time and financial resources is impossible.

Figure 1.1 shows the display tab in the options dialog for Microsoft Word. There are

17 possible options that can take two values which are check and uncheck. Hence, the

combinations that need to be tested are 217 or 131,072. If a test case need 5 minutes to

be test, to completely test only the display tab will need approximately 15 months,

which are practically inefficient.

Word Options

GenerOlChange how document content in displayed on the screen and when printed.

[DiOpint IPage display options Proofing

Save ) Show SShute space between pages In Pant Layout H ew() Shnwglghhlghtnrnnarttø Typography

3 Show document toottipa on honor Language

AdvancedAtwaynohow thow 00000thng n.wtmon the onreen

fl lab rharootert -. Custonnine Ribbon

Quick Access Toolbar D Paragraph IStrilt I Hidden Smut iTS

Add.int Optional hyphens ,Trust Center ObJectan5hors isle

l Qptfonal break, - Show Ill formatting mark,

Printing option,

) Print dyawingi Wetted In Word 0 Print Qacttground colors and Images

rl Print document gropertles Pant hidden toyt

In Update tields before printing Update linked data before printing

Figure 1.1: Display Tab in Options Dialog for Microsoft Word

4

1.3 GOAL AND OBJECTIVES

The goal for this research is to develop a pairwise test suite generator tool

based on Harmony Search Algorithm. Several objectives that have been set to achieve

the goal are as follow:

i) To apply Harmony Search Algorithm in generating pairwise test suite.

ii) To generate optimum test suite with minimized test data based on

interactions of two factors.

iii) To evaluate the performance of this pairwise test suite generator tool in

term of test size against other existing pairwise testing strategies.

1.4 RESEARCH SCOPE

In the process of development, NetBeans IDE 8.0 will be used. The scopes of

this research are as follow:

i) Pairwise test suite generating based on Harmony Search Algorithm.

ii) Pairwise test suite generating with 2-valued input parameters that are not

more than 500.

iii) Pairwise test suite generator tool developed in Java language.

1.5 METHODOLOGY

Software methodology is the activities used to control and ensure the

achievement of stated objectives and finally reach the goal of this research. The

methodology that going to use in this research are Rapid Application Development

model, Figure 1.2 below shows the phases in the model:

5

• Phase 1: Requirements Planning

o Literature review is performed in this phase to get the idea on the

design of HS-PTSGT. The other tools is collected to be used in

evaluate the performance of this pairwise test suite generator tool in

term of test size against other existing pairwise testing strategies.

. Phase 2: User Design

o The architecture of HS-PTSGT will be design to implement Harmony

Search Algorithm. The interface design of the generator tool will be

carrying out to ensure good usability.

• Phase 3: Construction

o After the design of HS-PTSGT, this tool will be developed. The

concept of pairwise testing and harmony search algorithm will be

implemented in this tool, and test will be carry out to ensure correct

implementation and well function of this tool.

• Phase 4: Cutover

o The result of generated test suite of HS-PTSGT will be compared

with other existing tools to show efficiency of this tool in term of test

size. Study will also carry out to observe the effect of HMCR and

PAR on test size. The whole research will then be documented.

Future improvement will be suggested to improve this research.

Figure 1.2: Rapid Application Development (RAD) Model

rel

1.6 THESIS ORGANIZATION

This thesis consists of seven (7) chapters, each chapter discuss about its own

issues. Chapter 1 is Introduction. This chapter discusses on introduction about the

research that will be done. The introduction, problem statement, goal, objectives, and

scopes will be known.

Chapter 2 is Literature Review. Existing research and literature review that

related to the project will be discussed in this chapter.

Chapter 3 is Methodology. In this chapter, overall approach and framework of

research will be discussed. The method, technique or approach to be used for the

project will be fixed.

Chapter 4 is Design of HS-PTSGT. This Chapter will illustrate the framework

and model through flow work.

Chapter 5 is Implementation. All the processes involved in the research

development will be recorded.

Chapter 6 is Result and Discussion. The results of pairwise test suite generator

tool will be discussed on this chapter to determine the development is success or not.

The test suite generated will be compared with other strategies in term of test size.

Chapter 7 is Conclusion. In this chapter, summarization of the research being

done will be discussed. Besides that, the future suggestion and enhancement of the

research topic also proposed in this chapter.

7

1.7 SUMMARY

This chapter has discussed the introduction of research on Pairwise Test Suite

Generator Tool Based on Harmony Search Algorithm. Included in this chapter are

problem statement, goal and objective, research scope, and thesis organization. The

next chapter is literature review that discusses about the survey of existing pairwise

testing strategy and Harmony Search Algorithm techniques.

CHAPTER 2

LITERATURE REVIEW

2.1 INTRODUCTION

In Chapter 1, the introduction of this research has been discussed, including

problem statement, goal and objective, scope and methodology. Building on the

material in Chapter 1, this chapter will focus on relevant literature review survey. In

particular, pairwise testing techniques, Harmony Search Algorithm, and existing

pairwise testing strategy are elaborated to justify the current work.

2.2 OVERVIEW

The main objective of this research is to generate optimum test suite with

minimized test data based on interactions of two factors. To achieve this goal,

pairwise testing and Harmony Search Algorithm will be implemented into this test

suite generator tool.

8

Oj

2. 2. 1 Pairwise Testing

Pairwise testing is a combinatorial technique used to reduce the number of test

case inputs to a system, consider all interactions of at most two factors (McCaffrey, J.

D., 2009). In pairwise testing test suite that consider all combinations of the selected

test data values for each pair of variables is generated. To understand the pairwise

testing, following example is illustrated.

Table 2.1: Input Parameter and Value Illustration

A I B I C I D

True False

--V - T False 1H False

Table 2.1 shows the input for a four plugs socket. In this system, there are 4

inputs available to user with 2 options respectively. Here, the exhaustive combination

of all possible test data can be calculated by [number of test data = (number of value) number of input parameters = 2 4 = 16]. The exhaustive combinations of this four plugs socket

are shown in Table 2.2.

2-way combination

for AB

True

10

Table 2.2: Exhaustive Combination of 4 Inputs

1L I A B ]C j_

\Y11IG3 True True True True

_____________ L False [ False - False - ]L alse ]

- True :1 True True tTrue

1 True]__ True L_False]

True True False True

True True False_ False]

True False True True False True .__JL False

True False False I True- _False False False

False True True True _EaLsq ..HL ....Iriu True False •.j

rue False

False r True

False True False 9 False I

False False True LTrue T

False False True False 7 False False ][True

r False 11 False --71 False IFalse

In order to reduce the test data, pairwise testing technique can be used. In this

technique, the 2-way possible combinations are AB, AC, AD, BC, BD, CD. For AB

combination the total test size is reduced to 4 as only parameter A and B are

considered while parameter C and D are treated as don't care value (see Table 2.3).

Thus, C and D can randomly take either value True or False.

Table 2.3: 2-Way Combination for AB

s le LaJ

BC j D j

True True P True I True

I

The second combination is AC. For this combination the total test data also

reduced to 4 since only parameter A and parameter C are considered. Parameter B and

D can take either value of True or False randomly as they are treated as don't care

values (see Table 2.4).

Table 2.4: 2-Way Combination for AC

=

1npjaijiables Base Value I II

1u,r: bilL:

-- True --- - False

combination True for AC I 1: False

The third combination is AD. For this combination the total test data also

reduced to 4 since only parameter A and parameter D are considered. Parameter B

and C can take either value of True or False randomly as they are treated as don't care

values (see Table 2.5).

Table 2.5: 2-Way Combination for AD

MMVdbD [___4 True

- True True

OJ11iftW1ftiTh False Rig A@-- False -

C

LTci True False 1I iaJse

D True

True False - 1 True False 1

The fourth combination is BC, the total test data also reduced to 4 since only

parameter B and parameter C are considered. Parameter A and D can take either value

of True or False randomly as they are treated as don't care values (see Table 2.6).

Table 2.6: 2-Way Combination for BC

In the fifth combination (BD), the total test data again reduced to 4 since only

parameter B and parameter D are considered. Parameter A and C can take either value

of True or False randomly as they are treated as don't care values (see Table 2.7).

Table 2.7: 2-Way Combination for BD

Similar to previous combinations, the total test data for CD combination is

reduced to 4. This is because only C and D are considered. A and B are then treated s

don't care value where they can randomly take either True or False values (see Table

2.8).

12

pairwise test suite generator tool based on harmony …

Documents