final report 1.3

School of Electronic Engineering and Computer Science

Final Report

Supervisor: Stefan Poslad

Project Title:

Automatic Test Data Generator For

Web Service Testing

Student Name: Muhammad Yahya

Date: 20/04/2015

Programme of study:

BSc Computer Science

Supervisor: Stefan Poslad

Supervisor: Dr Mustafa Bozkurt

Declaration of Authorship

I, Muhammad Yahya, declare that this thesis titled, ‘Automatic Test Data Generator For

Web Service Testing’ and the work presented in it are my own. I conform that:

This work was done wholly in candidature for an undergraduate degree at this

University.

Where any part of this thesis has been previously been submitted for a degree of

any other qualification at this University or any other institution, this has been

clearly attributed.

Where I have consulted the published work of others, this has always been clearly

attributed.

Where I have quoted from the work of others, the source has always been given.

With the exception of these such quotations, this thesis is entirely my own work.

I have acknowledged all main sources of help.

Signed: Muhammad Yahya

Date: 20/04/2015

Automatic Test Data Generator For Web Service Testing Muhammad Yahya

of 50

Abstract:

Web service testing is a process of finding bugs and errors in the web services. There are

different methods of testing web services. Web service testing is usually performed manually

which is very costly and time consuming. There are also number of web service testing tools

currently available in the market. Some of them are free open-source, such as SoupUI free

version, WebInject, Membrane, TestMaker and SOAPSonar and some of them can be bought

such as IBM® Rational® for SOA Quality (RTSQ) and SOAPUI NG Pro. To my

knowledge, most of these commercially or widely available web service testing tools does

not have automatic test values generator. SOAPSonar, RTSQ and few academic works does

provide automatic test values generator, but they are not very useful. I have created a web

service testing application which allow user to enter constrains and the test values are

generated based on those constrain. This testing tool is very flexible in terms of test values

generator. Everyone now a days is creating web services for their applications, this tool will

allow them to test their web services easily, thoroughly and quickly.


of 50

Table of Contents

Introduction ................................................................................................ 7

Objectives ..................................................................................................................... 7

Motivation ...................................................................................................................... 8

Challenges .................................................................................................................... 8

Summary ..................................................................................................................... 10

Background ............................................................................................. 11

Web services .............................................................................................................. 11

WSDL .......................................................................................................................... 13

SOAP request and response ...................................................................................... 15

Regular Expression .................................................................................................... 15

ASCII keys .................................................................................................................. 16

XML Data Types ......................................................................................................... 16

Different types of testing ............................................................................................. 16

Summary ..................................................................................................................... 17

Critical review of the existing applications ................................................ 18

Existing applications ................................................................................................... 18

Advantages and limitation of FAWST ......................................................................... 20

Summary ..................................................................................................................... 21

Requirement specification........................................................................ 22

Primary functional requirements ................................................................................. 22

Secondary functional requirements ............................................................................ 23

Summary ..................................................................................................................... 24

Application Overview, Design and Application Walk Through .................. 25

Application Overviews ................................................................................................ 25

Design ......................................................................................................................... 26

Application Walk Through ........................................................................................... 28

Summary ..................................................................................................................... 36


of 50

Testing..................................................................................................... 37

Unit testing .................................................................................................................. 37

Boundary testing ......................................................................................................... 39

User Interface Testing ................................................................................................ 39

Acceptance Testing .................................................................................................... 41

Platform Testing .......................................................................................................... 42

Summary ..................................................................................................................... 42

Conclusions and recommendations ......................................................... 43

What I learned and achieved ...................................................................................... 43

What would I add or do different ................................................................................. 43

Conclusion .................................................................................................................. 45

Bibliography 46

Appendix 48

How FAWST stores test constrains in the Array Lists ................................................ 48


of 50

Figures and Tables

Figure 1.1 who uses website and web service [6] ............................................................................ 11

Table 2.2.1 WSDL four major elements .......................................................................................... 13

Figure 1.2 Web Service Description Language [2] .......................................................................... 14

Figure 1.3 Main structure of a WSDL document [3] ....................................................................... 14

Figure 2.3.1 SOAP request ............................................................................................................... 15

Figure 2.3.2 SOAP response ............................................................................................................ 15

Table 2.6.1 commonly used XML data types .................................................................................. 16

Figure 5.1.1 Flow chart .................................................................................................................... 25

Figure 5.2.2.1 Model View Controller [10] ..................................................................................... 27

Figure 5.3.1 Initail window after FAWST is executed .................................................................... 28

Figure 5.3.2 Closing confirmation box ............................................................................................ 28

Figure 5.3.4 Window for String data type constrains ....................................................................... 30

Figure 5.3.5 Window for Double data type constrains ..................................................................... 31

Figure 5.3.6 Window for Enum data type constrains ....................................................................... 31

Figure 5.3.7 Window for Char data type constrains ......................................................................... 32

Figure 5.3.8 Window for Boolean data type constrains ................................................................... 32

Figure 5.3.9 Window for Date data type constrains ......................................................................... 33

Figure 5.3.10 Help Window for Date data type ............................................................................... 33

Figure 5.3.11 Custom Double window with error message ............................................................. 34

Figure 5.3.12 Custom Integer window ............................................................................................. 35

Figure 5.3.13 Custom String window .............................................................................................. 35

Table 6.1.2. Unit testing ................................................................................................................... 38

Figure 6.2.1 ...................................................................................................................................... 39

Figure 6.2.2 ...................................................................................................................................... 39

Table 6.3.1 Result from user interface testing .................................................................................. 41

Table 6.4.1 Acceptance testing summary ......................................................................................... 42


of 50

Introduction

Web service testing is a process of finding bugs and errors in the web services. There are

different methods of testing web services and most of them are usually performed manually

which is very costly and time consuming. Now a days there are different machines and

software available to check and test the items we use in our daily life such as, cars, tyres,

and drugs etc. This manual process of testing is performed by the software testers, therefore,

the quality of the web service rely on the software testers’ ability and experience. Software

testers have to spend a huge amount of resources and time to make sure that the web service

is tested properly and rigorously.

To address the issue of manual web service testing I decided to create a tool which will test

web services automatically. This software is called Flexible Automatic Web Service Testing

(FAWST) and it will generate unique test cases based on user constrains.

Objectives

The main objectives of this project are following;

Validate and parse WSDL

Generate test data for any XML data type

Generate test cases automatically with

o Any number of test cases

o Each test case should be unique

Automatically send SOAP requests

Analyse SOAP response

Generate detailed test report


of 50

Motivation

There were number of different things that motivated me to do this project such as;

Software Testing: Software Testing is my hobby, I test software for fun and I love

doing it. The reason why I love software testing is because it is a challenging task. I

have to look at software from a different angle and come up with all the different test

cases and scenarios. The joy and satisfaction of finding and fixing the bugs is also

very motivational for me.

Software quality improvement: Poorly tested software can have a lot of

consequences such as security issue, loss of money and time, reliability and even

death or injury. With the help of FAWST, developers can develop better web services

which will contain minimum bugs. As a result, users will get more robust web service

to use.

Knowledge enhancement: In this project most of the things were completely new

to me. Learning about new things and facing new challenges is very motivational for

me. It also gives me courage and self-esteem when I learn new things.

Future Directions: I will be looking for job as a software tester. This project has

enhanced my chance of becoming a good software tester and it will open many

opportunities for me in the field of software testing.

Challenges

Testing web services is more challenging than testing traditional software due to the

complexity of Web service technologies and the limitations that are imposed by the Service-

Oriented Computing environment [1]. Also most of things in this project were completely

new to me because of this, I faced number of challenges right from the beginning.

1. Starting point – Starting of any project is difficult especially if something is new.

In my case I had the same issue because automatic test case generating is completely

new, I was not really sure how I should do this and I had nothing to compare it to.

2. API – Finding a right API for parsing WSDL and for SOAP request and response

was one of the biggest challenge I faced in this project. I spent almost six to seven


of 50

weeks on finding the correct API which can parse WSDL send SOAP request and

receive SOAP response. I went through all the major APIs such SOAP-UI, Selenium

and Wsdl4j. SoapUI and Selenium APIs are very complicated. In my point of view,

they are also very poorly documented. These APIs have their own forums and it is

almost impossible to find any information or get any help. Finding the right API

made the whole project very difficult because I spent a lot of time on it.

3. Complexity – By complexity I mean I had to use number of different java internal

and external libraries to achieve objectives of this project. I have also used NetBeans

GUI builder to create user interface. Understanding and changing NetBeans GUI

builder code is a very hard and complex task. Other libraries include XML parsers,

random number generator and date generator etc.

4. User Input – Finding a useful and simple way of getting test constrains from the

user is not easy task. I have to cover all the possible inputs. It was a difficult task but

most of the test constrains are covered. For example, for integer data type, number

can be a random, between two numbers, specific number or custom (combination of

specific, random and between two numbers). I had to get test constrains for all data

types.

5. Storing test input constrains – As FAWST allows the users to generate any test

inputs, this means user can do following:

1. Twenty characters long string

2. 1st character is ‘a’

3. 2nd character is between ‘h’ and ‘z’

4. 3rd character is random

.

.

.

20. 20th character is ‘4’.

It is just one example illustrated above; a string could be any characters long. I had

to store all this information because the test case data is generated based on this

information. Finding a good solution was also very hard. At the end, I used Array-

Lists and my own type of regular expressions to solve it.


of 50

This project is like a star, looks small from far, but as you get closer and closer it gets bigger

and bigger. In the beginning of this project, it looked fairly simple and easy to do but as I

progressed through, it became very hard and very complex.

Summary

This chapter introduced FAWST and what are the main objects of the FAWST. I have also

talked about my motivations behind do this project and what challenges I had to faces while

working on FAWST.


of 50

Background

This chapter explains in detail all the background research I have done for this project such

as web services, WSDL, XML data types, SOAP request and response, regular expressions,

ASCII keys and different types of testing.

Web services

Web service is a service which is available on web for application consumption/use and a

website is available on web for human consumption/use. In simple words web service

provides a way of communicating between software applications over the network, web

services can run on a variety of platforms and frameworks [25].

For example, on Twitter website we can post and read tweets but if a computer wants interact

with Twitter to post and read tweets, it has to use Twitter web service. We can write some

code which will allow the computer to access Twitter data though web service, this data then

can be analysed by computer without any human interaction. Similarly if we make weather

app, all we have to do is write some code which will automatically get data from met office

web service and then we can display it on our app however we like. We can see from the

above examples, that web services are very useful because they provide any easy way of

communication between two applications.

Figure 1.1 who uses website and web service [6]


of 50

There are two types of the web services, both do the same thing but they have their own

advantages and disadvantages. It is totally up to the programmer which technique suits

him/her most.

2.1.1 REST

REST stand for Representational State Transfer, it relies on stateless communication

protocol, typically HTTP. In the REST web services, data and functionality are considered

resources and they are accessed by using Uniform Resource Identifiers (URIs), typically

links on the Web [5]. This means that each unique URI is a representation of some object.

REST web service allows you to get and modify content though HTTP GET, POST, PUT or

DELETE methods. REST web services are lightweight, maintainable, and scalable [9]. A

web services based on REST are called RESTful services.

REST approach will work best in the following situations:

1. Limited bandwidth and resources - because the return structure can be in

any format but it is usually JSON which is very lightweight.

2. Totally stateless operations - it is good for stateless operations such as

Create, Read and Delete.

3. Caching situations - data can be cached because of the totally stateless

operation of the REST approach [12].

2.1.2 SOAP

SOAP stands for Simple Object Access Protocol. It is a communication protocol which

allows communication between applications. SOAP is platform and language independent

protocol and it is based on XML. It is simple and extensible and allows you the get around

fire walls, and it is also recommended by W3C [7].

As SOAP is platform and language independent, it provides a platform for communicate

between applications running on various different operating systems, with different

programming languages and technologies. SOAP web services needs to have WSDL [7].

SOAP approach will work best in the following situations:

1. Asynchronous processing and invocation: SOAP approach provides high

level of reliability and security


of 50

2. Formal contracts: “If both sides (provider and consumer) have to agree on

the exchange format then SOAP 1.2 gives the rigid specifications for this type

of interaction”[12]

3. Stateful operations: SOAP approach is good for stateful operations. Stateful

means the computer or program keeps track of the state of interaction [13]

WSDL

The Web Services Description Language (WSDL) is an XML-based language which is used

to describe the functionality offered by a web service. A real life example of a WSDL can

be a takeaway leaflet where we see all the different items that a particular takeaway offers.

Also Chinese and Indian takeaway leaflets will have different items and different phone

numbers to call them. Similarly WSDL tells us what each web service is offering and how

can we contact this web service. WSDL document can be divided into four major elements

shown below in table 2.2.1.

Element Description

<types> Data type of the parameter. Different data types are explained below

under heading 2.6 XML data types.

<message> Each message can consist of one or more parts. The parts are like

the parameters of a method in a traditional programming language

[3].

<portType> PortType is set of operation offered by a web service.

<binding> “A protocol and data format specification for a particular port type”

[3].

Table 2.2.1 WSDL four major elements


of 50

In the below Figure 1.2 we can see how each part of the WSDL are linked together and also

it show what the sub parts of the four main elements.

Figure 1.2 Web Service Description Language [2]

The main structure of a WSDL document looks like Figure 1.3. WSDL can also have other

elements such as extensions elements and service elements that allow us to group together

the definitions of several web services in one single WSDL document [3].

Figure 1.3 Main structure of a WSDL document [3]


of 50

SOAP request and response

1. SOAP Request is sent to invoke a method on the web service. Request call has to

have the parameters of the method. The request has to be in the XML format which

includes all the information about the request. See Example below in Figure 2.3.1.

<SOAP-ENV:Envelope

xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"

xmlns:example="http://www.webserviceX.NET/">

<SOAP-ENV:Header/>

<SOAP-ENV:Body>

<example:ChangeAngleUnit>

<example:AngleValue>10</example:AngleValue>

<example:fromAngleUnit>radians</example:fromAngleUnit>

<example:toAngleUnit>degrees</example:toAngleUnit>

</example:ChangeAngleUnit>

</SOAP-ENV:Body>

</SOAP-ENV:Envelope>

Figure 2.3.1 SOAP request

2. SOAP Response is sent back from the web service method, which includes return

value if the request was correct else error is sent back. A response is also sent back

in the XML format, which makes it easy to parse and use. See the Example below in

Figure 2.3.2.

<?xml version="1.0" encoding="UTF-16"?>

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"

xmlns:xsd="http://www.w3.org/2001/XMLSchema"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<soap:Body>

<ChangeAngleUnitResponse xmlns="http://www.webserviceX.NET/">

<ChangeAngleUnitResult>572.95779513082323</ChangeAngleUnitResult>

</ChangeAngleUnitResponse>

</soap:Body>

</soap:Envelope>

Figure 2.3.2 SOAP response

Regular Expression

A regular expression is a string which describes a string pattern. Which is used to produce

new strings based on the pattern or validate existing strings such as “\b [A-Z0-9._%+-

]+@[A-Z0-9.-]+\.[A-Z]{2, 4}\b” this regular expression checks if someone has entered the

correct format e-mail address.


of 50

ASCII keys

ASCII stands for American Standard Code for Information Interchange. ASCII is an integer

representation of the characters for example alphabet “a” can be represented as number 97.

XML Data Types

WSDL uses XML define data types, in table 2.6.1 we can see all the main commonly used

XML data types but it is not a full list. There are many more XML data types which are

based on data types in table 2.6.1.

Data Type Description

String String data types are used for the values which contain sequence of

character. Such as, sSsd, hello-world and 354sdse%3 etc.

Integer Integer is sequence of number ranging from -2147483648 through

2147483647.

Double Double is sequence of number with decimal point. Double hold 64 bit

long numbers.

Float Float is similar to Double but it can only hold 32 bit long numbers.

Char Char is for single character. For example A, b, % and s etc.

Enum Enum data type contains a list of acceptable values.

Date and time Date and time data type are for the values that contains date and time.

Boolean Boolean data type is for true or false.

Table 2.6.1 commonly used XML data types

Different types of testing

1. Unit testing – Programmer test each individual software components and parts. This

is mostly done by the programmers themselves.


of 50

2. Functional testing – “This type of testing ignores the internal parts and focus on

the output is as per requirement or not. Black-box type testing geared to

functional requirements of an application” [27].

3. End-to-end testing – It “involves testing of a complete application environment in a

situation that mimics real-world use, such as interacting with a database or using

network communications”.[27]

4. Sanity testing – “Testing to determine if a new software version is performing well

enough to accept it for a major testing effort. If application is crashing for initial use

then system is not stable enough for further testing and build or application is assigned

to fix”[27].

5. Load testing – “It is a performance testing to check system behaviour under load.

Testing an application under heavy loads, such as testing of a web site under a range

of loads to determine at what point the system’s response time degrades or fails”. [27]

6. Stress testing - “System is stressed beyond its specifications to check how and when

it fails. Performed under heavy load like putting large number beyond storage

capacity, complex database queries, continuous input to system or database load”.[27]

Summary

In this chapter, I have explained in detail what web service is, what are the different types of

web services and what is SOAP request and response is. I have also explained what WSDL

is and what data types WSDL accepts and I also talked a little bit about regular expressions,

ASCII keys and different types of software testing.


of 50

Critical review of the existing

applications

This chapter presents a review the existing applications, both commercially and non-

commercially available. It also includes a review of FAWST and it shows in detail what

functionality FAWST does and does not provide.

Existing applications

There are number of web service testing tools currently available in the market. Some of

them are free open-source such as SoupUI free version, WebInject, Membrane 1, TestMaker

and SOAPSonar and some of them can be bought such as IBM® Rational® for SOA Quality

(RTSQ), prices starts from $2995 and SOAP UI NG Pro €449 per year. To my knowledge,

most of these commercially or widely available web service testing tools do not have

automatic test case generator. SOAPSonar, RTSQ and few academic works such as WSTD-

Gen does provide automatic test case generator [21, 22, 23, 24].

Basically most of these widely available web service tools provide graphical user interface,

which allows the users to import and check WSDL, create and edit test cases manually, run

the test cases and see the result. I have personally used two of the web service testing tools

and their details are provided below;

1. SOAP UI

SoapUI is free and an open-source cross-platform functional testing tool for web

service, and is known as “Swiss-Army Knife of Testing”. SoupUI has well designed

and structured GUI. SoapUI allows the user to create and run automated functional,

regression, compliance, and load tests. It supports both SOAP and REST web service

approach. SoapUI also allows the user to run Functional/Load Tests and MockServices

from a task scheduler with a set of command-line tools. SoapUI received a number of

awards: ATI Automation Honors, 2009 , Jolt award in 2014 for the best testing tool

.InfoWorld Best of Open Source Software Award, 2008 and SOAWorld Readers'

Choice Award, 2007 [13] [16] [15].

1 www.service-repository.com


of 50

2. Service-repository.com

Service-repository.com is browser based web service testing tools. It provides very

basic web service testing functionality such as WSDL validation, send SOAP request

and display SOAP response. Users have to manually enter values for each test case one

by one. User can only test SOAP web service. This tool is only good for basic checks

and testing, it will be of no use if user wants to test their web service rigorously.

Some of these web service tools can generate test cases automatically. SOAPSonar utilizes

XSDMutation, they have submitted a patent for automation technique which is pending.

SOAPSonar generate a set of test cases, which are both negative and positive. “The test

mutations may occur at the data type, data value, message structure, or protocol binding

level. An example mutation is a buffer overflow type boundary conditions for an unbounded

string type field” [22].

SOAPSonar generates random test cases and they are ineffective in detecting defects.

SOAPSonar test case are based on the type information and data structure defined in the

WSDL. WSDL definition usually have basic information about the actual data type of

parameters. For example, the parameters of multiple method can be all of type String. If

generated values are strings such as are “ss” and “ds” the test case will fail because the

multiple method was expecting numeric values and generated values were string. These

randomly generated test cases will be ineffective in detecting defects as most of the test case

will fail because generated values were of different data type. [22]

RTSQ can generates random test cases based on the information taken from WSDL such

methods parameters data types, e.g. “1234” for integer data type. This type of testing is only

usefully from performance testing, they will of no help in actual errors and bugs detection.

WSTD-Gen is testing tool which provides a better way of generating test cases, this tool was

created for an academic work. Unfortunately I couldn’t find WSTD-Gen application on the

internet; I have reviewed this application based on a WSTD-Gen project report [22].

According to the report, users are allowed set constrains for each data-type as show in Figure

3.1.1.


of 50

According to my understandings of this application users can only select from these

predefined contains shown in figure 3.1.1, which puts a limit on number of test cases this

application can produce.

All the applications I reviewed above have some type of a limit on values they can generate

and most of the times these values were randomly generated based WSDL. This type of

testing is not very useful for a web services tester because it randomly generated values

usually ineffective in detecting bugs and errors [21, 22, 23, 24].

Advantages and limitation of FAWST

In most the web service testing tools which I reviewed (sub heading 3.1), someone has to

generate all the test cases, this is the task which takes most of the user time and resources.

FAWST have all the main functionality which any standard web service testing application

does such as WSDL checking and parsing, GUI and automatic SOAP request and response

but it also have some unique features such as, automatic test data generator, result verifier

and detail test report which most of the other web services testing applications lack.

FAWST automatic test data generation is very flexible, users can put any constrains on any

character as they wish. For example if a tester wants FAWST to generate four characters

long string or a number, he/she can put constrains on each character, such as first character

can be specific, second can be random between two numbers or alphabets, third character

can be random and fourth can be random again. This flexibility allow the users to test their

web services with any possible values they can think of.

Figure 3.1.1 Constrain window of WSTD-Gen


of 50

Users can perform many different types of testing using FAWST such as Unit testing,

Functional testing, End-to-end testing and Sanity testing. These are some of the main

and common types of testing usually performed on software for quality insurance.

There are just two main limitations of my application. First one is that user can only test

SOAP web service and second limitation is not being able to parse all version of WSDL.

This is because the API I have used, the API is unable to parse some of the WSDL. I found

this problem at very late stage so I was unable change my API. I have not integrated the

REST web services in this version FAWST because of the limited time and main point of

this project was implement automatic test case generator which can generate any number of

different test cases.

There are some limitation in result verification, FAWST allow users to check multiple tags

and values but if there are two tags with the same name, users would only be able to check

the value of the first tag.

Summary

This chapter includes review of the well-known web service testing applications and shows

their limitations and advantages. I have reviewed FAWST and how it is unique from all other

web service testing applications available in the market and what are FAWST limitations.


of 50

Requirement specification

This chapter describes the requirements identified for FAWST. Starting with functional

requirements, which are split into primary and secondary requirements.

Primary functional requirements

Primary functional requirements are the main requirements of FAWST. These requirement

were necessary to achieve the objectives to FAWST.

Checking WSDL: SOAP web service require WSDL file for it to know what

functionality web services are offering. So checking if WSDL file or a link to WSDL

is correct is very important part of FAWST, because with the wrong WSDL file

nothing is going to work.

Parsing and storing WSDL: Parsing WSDL means reading the WSDL files and

extracting all the key information, such as method names, parameters names and

data-type, complex data-type values and URL for SOAP requests. This information

has to be stored so it can be later displayed and used for SOAP requests. This

information can be stored on the RAM because this information is only required for

a small time.

Collecting and storing test constrains: Collecting and storing test constrains enter

by a user as regular expressions is essential part of FAWST Based on the regular

expressions all the test case data will be generated. User test constrains are not stored

as normal regular expressions but it will be based on regular expressions concept.

For example 4-a-d means generate four character long string and the values can only

be between ‘a’ and ‘d’. Also these test constrains can be stored on RAM because

they will only be required for a small time period and accessing this data from RAM

is a lot faster than hard drive.

Automatic test input data generator: This is the unique part of this project.

Automatic test inputs will only be generated once FAWST have collected and stored

test constrains in regular expressions. Based on the regular expressions FAWST will

generate all the test inputs. Without the user regular expressions FAWST will not be

able to generate any test inputs.


of 50

Automatic SOAP Request: Generating SOAP request and response is another main

functionality of FAWST. This functionality will allow the users to run as many test

cases as they want, without having to write a SOAP request for each test case.

Automatic Result Verification: All the SOAP responses should be checked if they

passed or failed. The software should allow the user to type in expected values and

tag names and the values should be automatically checked against the SOAP

response.

User Interface: User interface is necessary to have because it will allow the users to

interact with the FAWST easily. Without the user interface it would be incredibly

hard to navigate through the FAWST, because it is complex software and it requires

a fair amount user input at the start. Basic user interface will include following things;

Home window: On this window user would be able to enter WSDL

link.

Show method window: This window will display all the methods

names, parameter names and data types.

Test data input windows: This is where user should easily be able to

enter constrains for test input generator. Test data input window

should be slightly different for each data type.

Report: A report should be generated to display, what parameter values were

generated, SOAP request, SOAP response and result (pass/failed) of each test case.

This report should be in table so it can be easily checked by the user.

Secondary functional requirements

Secondary functional requirements are all the optional and additional features of FAWST.

These were implemented after the primary functional requirements.

Checking for same test cases: Same test cases should be checked, because there are

no point testing web services with the same test cases again and again. It will only

waste time and consume unnecessary resources.


of 50

Help windows: Help windows should be added to help the users to use FAWST and

to warn if user if they enters wrong values or no values.

Verification of user input: All the user input data should be checked and verified.

For example if a text field in FAWST is expecting an integer, it should only take an

integer for all the other value it should display an error message to inform users.

Critical actions confirmation: Critical actions like closing the software or Back to

main menu should be caught by a confirmation window.

Summary

In this chapter I have described the identified requirements for FAWST. The requirements

were divided into primary and secondary requirements. Each of the requirements was

explained in detail for user understanding.


of 50

Application Overview, Design and

Application Walk Through

This chapter explains in detail with flow chart how FAWST works. I will then reflect on the

design of the FAWST, which includes development environment and design patterns. I will

then explain in detail with screenshots how FAWST looks and works.

Application Overviews

Figure 5.1.1 Flow chart


of 50

In this topic, I will explain in detail how FAWST works and how it generates test inputs.

FAWST rely on some user input and based on that input it generates test case data. The

method is shown in Figure 5.1.1. Also this is just a basic overview of the FAWST main

functions and Figure 5.1.1 only shows main functions and flow.

Once FAWST is executed, user inputs a WSDL, which then checked to make user it is a

WSDL. Once the WSDL is parsed all the information is extracted such as methods name,

method input parameters name and data types, URL and complex data types etc. User selects

one of the method and enter an integer value for number test cases he/she wants FAWST to

generate. For each parameter, user have to enter test input constrains. There is almost no

limit on the test input constrains user can enter. There are number of basic options that user

can choose from such as:

1. Random value

2. Specific value

3. Between two values

4. Custom (combination of random, specific and between two values).

All the above options are explained in detail in application walk-through in sub section 5.3.

Once the user enters constrains for all parameters, FAWST generates test inputs based on

these constrains. Each test case is checked against all the pervious test cases to make sure

that there are no duplications. Once duplication is checked, SOAP request is sent. When a

SOAP response is received, user then enters tag names and expected values of each SOAP

response which is then checked and stored. User can only enter valid tag names otherwise

he/she will keep getting error (“Tag name not found”). Once all the test cases are generated,

a report is produced, which display all the generated parameter values, SOAP request and

response and result of each test case.

Design

5.2.1 Development Environment

I have used Java programming language to write code for FAWST because I have over two

years of experience of Java and I was more comfortable coding in Java. I have used NetBeans

IDE to help me manage and code FAWST because I have used NetBeans before and I knew


of 50

about the shoutcuts and help that I could get from using NetBeans. Most of the GUI

(graphical user interface) is created by NetBeans GUI builder. NetBeans GUI builder is very

helpful and it saves a lot of time once you know how to use it.

5.2.2 High level design

FAWST code design patteren is based on Model View Controller (MVC); MVC is design

pattern in which developers simply separate their code into three categories which are model,

view and controller. This allows developers to keep the code tidy and as result it makes it

easy to find and fix bugs, maintain code and enhance code usability. I have also divided the

code in to further sub folders to make it even simpler and tidier for example view package

has two sub-packages, MainGUI and ReportGUI.

Figure 5.2.2.1 Model View Controller [10]


of 50

Application Walk Through

Initail Window: The initial window the user will see is Figure 5.3.1. User can enter WSDL

URL or path to the WSDL store on the local machine. Once user enters WSDL, he/she has

to press the validate button. Once validate button is pressed WSDL URL or path is checked.

If correct, WSDL file is also checked to make sure it is correct. Once the WSDL is validated

“check methods” buttons is enabled, which opens new a window shown in Figure 5.3.3.

Check method button preform same checks as validate button, just to make sure user has not

changed WSDL. Clear button clears the text from the input WSDL test field. Exit button

closes the application but a confirmation dialog is display shown in Figure 5.3.2 to confirm

user intentions. Exit button does the exact same job throughout the application.

Methods Window: Method window is shown in Figure 5.3.3, The information extracted

from WSDL file is displayed here, such as, all the method names and paramter name and

data type. User can collapse or open each method to see input parameters name and data

Figure 5.3.1 Initail window after FAWST is executed

Figure 5.3.2 Closing confirmation box


of 50

type. User can select any method and click on the “Test” button to test that paticular method.

Once “Test” button is pressed, application checks if user selected a method, a parameter or

anything at all; based on result an error is displayed or if correct method was selected, a input

message dialog is displayed, where user is asked to enter number of test cases he/she wants

FAWST to generate, input message dialog only take integers, all ther other values are

rejected with an error message. “Back” button takes the user back to the initial window.

Different windows for different Data types: Once the user selects one of the methods and

clicks on “Test” Button in the method window and enters of number test he/she wants

FAWST to produce. Depending upon the method user wants to test, FAWST will open a

new a window for each parameter, where user can enters constrains. Different screens and

option are displayed for different data types. I will explain these different windows and

options in detail with screen shots below:

String: String window is shown in Figure 5.3.4. For string user have three basic options and

custom option which is explained in detail below under Custom heading. Any of these option

can be selected by clicking on their respective combo boxes;

1. Random String: User has to specify the length of the string.

2. Specific String: User can enter any string they want, it will stay the same though out

for every test case.

Figure 5.3.3 Method window


of 50

3. Random between two String: Users have to select alphabet one and two, alphabets

are between “a” and “z”. Users also have to enter length of the string.

Double: Double window is shown in Figure 5.3.5. It is mostly same as string window, the

only different is that user have to select decimal point position as well for random number,

decimal point position has to be less than random number length.

Specific and random between two numbers text field accept only numbers, which could be

either decimal or not decimal numbers.

Figure 5.3.4 Window for String data type constrains


of 50

Integer: Integer window is almost same as double window, the only difference is that all the

text fields only accepts integers and there is not decimal point position.

Enum: Enums are specific list of values which a method in the web service will accept,

FAWST extract these values from WSDL and display it to the user as shown in Figure 5.3.6.

Figure 5.3.6 Window for Enum data type constrains

Figure 5.3.5 Window for Double data type constrains


of 50

There are three options for a user, he/she can check the “Select Random one” check box or

chose one or multiple values from the list. If user selects multiple values, one of the value

will be randomly select for each test case.

Char: Char is similar to String window, the only difference is that there are no custom option

of char because char is a single character and there is no length option. Also user can only

enter single character in the specific option.

Boolean: Boolean window is very simple with just one options. User can either select true

or false.

Figure 5.3.7 Window for Char data type constrains

Figure 5.3.8 Window for Boolean data type constrains


of 50

Date: Date window simple and self-explanatory as shown in Figure 5.3.9. User have

following four options:

1. Random Date

2. Specific Date

3. Random between two dates

4. Today Date

For each option user have to select the date format so the generated date is in format that

user wants. Also there is a help window shown in Figure 5.3.10 for date format, this helps

the user to understand, what are difference between different date formats.

Figure 5.3.9 Window for Date data type constrains

Figure 5.3.10 Help Window for Date data type


of 50

Custom user constrain screens: Custom screens are for Integers, String and Double.

Custom screens allow the user to put constrains on each individual character, this means

FAWST will generate specific values for test cases based on user constrains. Each of custom

window is slightly different than the other, but they all have similar design and options. The

similarities and difference are following;

Similarities

1. Length of characters

2. Option of specific value, random between two values and random value.

3. User can only select values from combo boxes.

4. Error messages if user forgets to enter a value, shown in Figure 5.3.11

Differences:

1. Custom Integer window: Custom Integer window shown in Figure 5.3.12

only allow users to select numbers between 0 and 9.

Figure 5.3.11 Custom Double window with error message


of 50

2. Custom String window: Custom String window have one more option for

random value. User can have random character with or without punctuation.

Specific value can be any character and for between two values user can only

select alphabets.

Figure 5.3.12 Custom Integer window

Figure 5.3.13 Custom String window


of 50

3. Custom Double window: Custom Double window is very similar to custom

Integer window. The only difference is that user can have decimal point, only

one point can be selected, if more than one is selected user will get an error.

Summary

In this chapter, I have explained in how FAWST works. I have also reflected on the high

level design and design pattern of the FAWST code. I have also covered in detail how

FAWST looks and what are different options on the different screens with screenshots.

Figure 5.3.14 Custom Double window


of 50

Testing

This chapter will cover different types of testing performed on FAWST, this includes Unit

testing, Boundary testing, GUI testing, Acceptance testing and Platform testing, this is to

make sure that FAWST is bugs and error free.

Unit testing

Unit testing is a process in which the smallest testable parts of an application, called

units, are individually and independently scrutinized for proper operation [26]. I have

tested each and every class of this software individually. I wrote a main method of each class

and tested all the methods with different values.

The above Figure 6.1.1 is one of the example. I have three methods intGen, doubleGen and

charToInt in the CommonMethodsTwo class. All of these methods were tested with a lot of

different test cases such as, intGen takes two integers, so some of the test for this method

are as following;

1. intGen(1,4) // passsed

2. intGen(41,9) // failed, first value has to be smaller than second value

3. intGen(a,4) // failed, only take integers.

4. intGen(-10000,0) // passed

Table 6.1.2 shows a list of some of the methods that were individually tested. This table only

includes those methods whose test values were small and easy to understand.

Figure 6.1.1


of 50

NO# Method partial

header

Test values Expected

Result

Actual

Result

1 isDouble(String

parm)

2.3232 True passed

2 format(String xml) <?xml version="1.0"

encoding="utf-

8"?><soap:Envelope

xmlns:soap="http://schemas.xmlso

ap.org/soap/"

xmlns:xsi="http://www.w3.org/20

01"

xmlns:xsd="http://www.w3.org/20

01/XMLSchema"><soap:Body></

soap:Body></soap:Envelope>

Better

layout

Passed

3 smallerChar(Object

pramOne, Object

pramTwo)

C,K C Passed

4 biggerChar(Object

pramOne, Object

pramTwo)

C,K K Passed

5 checkConfirm(int

pram)

0 // this method is for check box,

this method true is check box is

checked

True Passed

6 isInt(String pram) 323232442424224 False Passed

7 intGen(int

rangeFrom, int

rangeTo)

1,4 2 or 3 Passed

8 doubleGen(double

rangeMin, double

rangeMax)

1.2,1.23 1.21 or

1.22

Passed

9 charToInt(String

pram)

abc 97 Passed

10 checkWSDLAvaila

ble(String wsdl)

http://www.webservicex.net/count

ry.asmx?WSDL

True Passed

11 checkPath(String

pram)

/homes/my301/year3/Project/other

/genericbarcode.wsdl

True Passed

Table 6.1.2. Unit testing


of 50

Boundary testing

Boundary testing is where test cases values are extreme values e.g. “maximum, minimum,

just inside/outside boundaries, typical values, and error values” [17]. I have checked all the

methods on which I could perform boundary testing e.g. intGen, doubleGen, intToChar,

isInt and charToInt etc. In Figure 6.2.1 shows one of the example.

In the above example (Figure 6.2.1) isInt method checks, if a given value is an int or not. As

we can see the four test cases, first one is a maximun value of an int, second value is minum

value of int, third testcase value is bigger than the int maxium value and fourth is less then

minmun value of the int. so the result should be true,true,false and false and we can see in

the Figure 6.2.2 it is correct.

User Interface Testing

The UI testing is process to make sure that all the functionality of the UI is working and it

does what it supposed to do. A thorough UI testing was performed on FAWST to make sure

that everything works. Table 6.3.1 shows all the main tests that were performed on the

FAWST UI.

Figure 6.2.1

Figure 6.2.2


of 50

No. Test Name Description Expected Result Actual

Result

1 Run

Application

Application was opened

in NetBeans and through

jar file execution.

Application first

window should open.

Passed

2 Clear button Click on the “Clear”

button.

Text field should get

clear.

Passed

3 Exit button

confirmation

box

Exit the application once

clicked but after the

confirmation.

Confirmation popup

box should appear.

Passed

4 Check

Methods

button

By clicking on check

methods button, it should

check the input WSDL. It

should call checkWSDL

method and if correct

open show method frame

Open show method

frame or display error

if WSDL is wrong.

Passed

5 Test button By clicking on test button,

it should display an input

dialog if a method is

selected else it should

display a message dialog.

Input dialog or

message dialog.

Passed

6 Main Menu

button with

confirmation

box

By clicking on the main

menu. Confirmation box

should appear for user

confirmation.

Confirmation box

should open.

Passed

7 Next button

with

message

dialog box

By clicking on the next

button. User constrains

should be validated.

Message box should

appear on the screen.

Passed

8 Done button

with

confirmation

box

By clicking on the done

button, confirmation box

should appear for user

confirmation.

Confirmation box

should appear

Passed

9 Click me

Jtable

column

By click on the click me

in report table screen

New frame should

open with extra

information

Passed


of 50

Acceptance Testing

The best way to determine whether a software had met all the user specifications and

requirements is to ask the user to check and test the software. Acceptance testing for FAWST

is performed by Dr Bozkurt. He tested the following WSDL which covers all the main data

types.

1. http://www.webservicex.net/geoipservice.asmx?WSDL for String

2. http://www.webservicex.net/ConverPower.asmx?WSDL for Double and Enum

3. http://www.webservicex.net/periodictable.asmx?WSDL for String

4. http://www.webservicex.net/MortgageIndex.asmx?WSDL for Int

5. http://www.webservicex.net/sunsetriseservice.asmx?WSDL for Int and Float

WSDL

Number

Issues Passed/Failed Comments Score 1-10

1 is low

1 N/A Passed N/A 8

2 N/A Passed Good example 9

3 Input didn’t

work

Passed/Failed Issue with one of the

method

7

4 N/A Failed Come up with some

sensible input.

5

5 N/A N/A N/A N/A

Overall Performance 5 4 3 2 1

10 Collapsible

table

On method screen, user

should be able to open

and collapse each method

and complex to see which

values are accepted by the

method.

Methods should open

and collapse

Passed.

Table 6.3.1 Result from user interface testing

http://www.webservicex.net/geoipservice.asmx?WSDL

http://www.webservicex.net/ConverPower.asmx?WSDL

http://www.webservicex.net/periodictable.asmx?WSDL

http://www.webservicex.net/MortgageIndex.asmx?WSDL

http://www.webservicex.net/sunsetriseservice.asmx?WSDL


of 50

Design 5 4 3 2 1

Met all user requirements

and specifications

5 4 3 2 1

Any comments Overall performance is okay. Design is good. FAWST have met

and exceeded all the user requirements and specifications.

Dr Bozkurt was happy with the FAWST and he said it met and exceeded all the user

requirements and specifications.

Platform Testing

It is very import for FAWST to run on all major operating system otherwise limited user

would be able to use FAWST. I have tested FAWST on the following platforms:

Linux

o Fedora

Windows

o Windows 8

o Windows 7

Apple OS

o OSX Yosimite 10.10.2

FAWST worked on all the above operating systems and there were no issues.

Summary

In this chapter I have covered different types of testing that were performed on FAWST with

examples. The different types of testing performed on FAWST includes Unit testing,

Boundary testing, GUI testing, Acceptance testing and Platform testing to make sure that all

the bugs and errors are removed.

Table 6.4.1 Acceptance testing summary


of 50

Conclusions and recommendations

This Chapter describes what I have learned and achieved from this project. I would also

explain what I will do differently next time and what new things I would like to add to this

project.

What I learned and achieved

I am very pleased with the outcome of my project. For me, this project has opened a whole

new world of programming and testing which I had no clue about. This project was by far

the biggest project I have ever done and by doing so learned and achieved a lot.

Now I have a very good knowledge of web service and test data generation. Both of them

these topics were very new to me and I knew very little about then. This project has

broadened my programming skills, I have learnt and practice many different things such

working with external libraries, recursion, error handling and much more. I have used and

learnt more about the MVC design pattern, which will come very useful in my future

programming projects and as well as in my day to day life.

This project has also helped me learnt more about project management and time

management. These are very useful skills because project management skills reduces the

chance of a project failing, it increases the quality of the software, increase efficiency and

saves time. As we all know time is money, there should be no questions about how useful

time management skill is. The advantage of time management are, it reduces stress, free

more time, get more done in same time and we get more time where it needed most. These

two skills are not only good for projects but they are useful in all acpects of the life.

I am well aware of the limitation of FAWST but I am very proud of what I have achieved,

this project does what is supposed to do and it satisfies all the primary and secondary user

requirements. This by far the biggest project I have ever created and I have a lot from this

project.

What would I add or do different

The only thing that I would do differently is to make use of more object oriented

programming style. I haven’t used inheritance and subclasses, and after completing 80% of


of 50

the project I realized using inheritance and subclasses would have been very useful for this

project. FAWST code would have been more generalized, easy to maintain and expand.

Apart of this I am very happy with the rest of the project and the way I coded it. However, I

certainly would like to add more features if I had more time available, these features are not

part of functional or non-functional requirement but it will make this project much better.

Printing and storing report: Printing and storing the report would be a very nice

feature; this will allow the users to analyse and compare the results.

Enhance GUI: Current FAWST GUI is not very simple because it is a prototype.

By making the following changes to the GUI will enhance user experience of using

FAWST:

o Make the layout better

o Use colours effectively manage attention

o Improve navigation

XPath for result verification: XPath is a language that describes a way to locate

and process items in XML [20]. XPath will allow FAWST to verify results more

accurately and also user would be able to check complex data types.

Enhanced Settings: User preference settings should be added. For example, only

unique test case on and off, check multiple elements or single element, keep the

same value for tag or different value to each test case and stop is certain number of

test case fails.

Increase testing capability: Currently user can only perform unit testing,

functional testing, end-to-end testing and sanity testing using FAWST. But I can

easily include the load and stress testing options. This will increase the testing

capability of FAWST and my application will be used by wider audience:

Help: My application is very straight forward to use but adding more help screens

will make it even simpler for user to use.


of 50

Conclusion

In conclusion to this report, I would simply say that I had great fun doing this project.

FAWST is an amazing application which can really help with the testing of web services. It

is also very unique and flexible and I am well aware that there is quite a few things needs to

be included in FAWST but, in my point of view creating FAWST was a great achievement

for me and as my supervisor said that FAWST had met and exceeded all the user

requirements and specifications. FAWST is also very thoroughly tested to make sure that it

is bugs and errors free. I would say again it was great fun doing this project and I learnt a lot

from this experience.


of 50

Bibliography

[1] M. Bozkurt, M. Harman and Youssef, “Testing & Verification In Service Oriented

Architecture :A Survey,” in Software Testing, Verification and Reliability, London, Wiley

InterScience, 2009, p. 259–350.

[2] A. Payong, “Examples of WSDL,” yurtopic, 27 03 2013. [Online]. Available:

http://www.yurtopic.com/tech/programming/wsdl-language.html . [Accessed 17 02 2015].

[3] “WSDL Documents,” w3schools, [Online]. Available:

http://www.w3schools.com/webservices/ws_WSDL_documents.as. [Accessed 17 02 2015].

[4] “WSDL,” webopedia, [Online]. Available:

http://www.webopedia.com/TERM/W/WSDL.html. [Accessed 16 02 2015].

[5] “What Are RESTful Web Services,” Oracle, [Online]. Available:

http://docs.oracle.com/javaee/6/tutorial/doc/gijqy.html. [Accessed 17 02 2015].

[6] Dr. P. Oliva, “ECS639U - Web Programming,” Queen Mary, [Online]. Available:

http://qmplus.qmul.ac.uk/course/view.php?id=4915. [Accessed 17 02 2015].

[7] “SOAP Introduction,” w3schools, [Online]. Available:

http://www.w3schools.com/webservices/ws_SOAP_intro.asp. [Accessed 17 02 2015].

[8] “SOAP Example,” w3schools, [Online]. Available:

http://www.w3schools.com/webservices/ws_SOAP_example.asp. [Accessed 17 02 2015].

[9] M. Vaqqas, “WEB DEVELOPMENT,” Dr.dobbs, 23 09 2014. [Online]. Available:

http://www.drdobbs.com/web-development/restful-web-services-a-tutorial/240169069.

[Accessed 17 02 2015].

[10] “MVC image,” wordpress, [Online]. Available:

https://imdjkoch.files.wordpress.com/2012/09/oaf-mvc-architecture.jpg. [Accessed 17 02

2015].

[11] “Software Testing Services in the US: Market Research Report,” IBISWorld, 2014.

[12] M. Rozlog, “REST and SOAP,” InfoQ, 04 01 2010. [Online]. Available:

http://www.infoq.com/articles/rest-soap-when-to-use-each. [Accessed 02 03 2015].

[13] F. M. Borger and J. , “stateless,” whatis, 01 04 2005. [Online]. Available:

http://whatis.techtarget.com/definition/stateless. [Accessed 02 03 2015].


of 50

[14] “What is SoapUI,” SoapUI, [Online]. Available: http://www.soapui.org/about-soapui/what-

is-soapui-.html. [Accessed 02 03 2015].

[15] “Award Winning Software,” SMARTBEAR, [Online]. Available:

http://smartbear.com/about-us/awards/ . [Accessed 02 03 2015].

[16] S. Hussain, Z. Wang and I. K. Tour, Web Service Testing Tools: A Comparative Study,

Beijing,.

[17] “Boundary testing,” Wikipedia, [Online]. Available:

http://en.wikipedia.org/wiki/Boundary_testing. [Accessed 04 03 2015].

[18] F. Usmani, “Product Scope vs Project Scope,” PM Study Circle, [Online]. Available:

http://pmstudycircle.com/2012/01/product-scope-vs-project-scope/. [Accessed 04 03 2014].

[19] A Guide to the Project Management Body of Knowledge (PMBOK® Guide), Fifth Edition,

Pennsylvania: Project Management Institute, 2013.

[20] M. Rouse, “XPath,” Search SOA, [Online]. Available:

http://searchsoa.techtarget.com/definition/XPath. [Accessed 05 03 2015].

[21] J. Z. ,. L.-J. Z. a. N. M. M. Zhong Jie Li, Z. J. Li, J. Zhu, L.-J. Zhang and N. , Towards a

Practical and Effective Method for Web Services Test Case Generation, Vancouver: IEEE,

2009, pp. 106 - 114.

[22] X. J. Offutt and W. , Generating Test Cases for Web Services Using Data, Fairfax, 2009.

[23] “Home,” crosschecknet, [Online]. Available: http://www.crosschecknet.com/. [Accessed 09

03 2015].

[24] “Rational Service Tester for SOA Quality,” IBM, [Online]. Available: http://www-

03.ibm.com/software/products/en/servicetest. [Accessed 09 03 2015].

[25] “What Are Web Services,” Oracle, [Online]. Available:

http://docs.oracle.com/javaee/6/tutorial/doc/gijvh.html . [Accessed 14 04 2014].

[26] M. Rouse, “Unit Testing,” Search Software Quality, [Online]. Available:

http://searchsoftwarequality.techtarget.com/definition/unit-testing. [Accessed 14 04 2015].

M.

[27] “Types of software Testing,” Software Testing Help, [Online]. Available:

http://www.softwaretestinghelp.com/types-of-software-testing/. [Accessed 16 02 2015].


of 50

Appendix

How FAWST stores test constrains in the Array Lists

8.1.1 Custom String

------ Start for loop ------

User constrains Custom String

User constrains Specific

User constrains b

User constrains Random withOut

User constrains 1

User constrains Random with

User constrains 1

Length of Array List: 6

------ End for loop ------

8.1.2 String with random value


User constrains String

User constrains Random

User constrains 4


------ End for loop ------

8.1.3 String with between two values



User constrains BetweenTwoValue

User constrains 17-d-e


------ End for loop ------

8.1.4 String with specific value




User constrains abc


of 50


------ End for loop ------

8.1.5 Custom Int


User constrains Custom Int


User constrains 3


User constrains 1


User constrains 4


------ End for loop ------

8.1.6 Int with specific value


User constrains Int


User constrains 3232


------ End for loop ------

8.1.7 Int with random value


User constrains Int


User constrains 4


------ End for loop ------

8.1.8 Int with between two values value


User constrains Int

User constrains BetweenTwoValue

User constrains 2131-232314


------ End for loop ------


of 50

8.1.9 Boolean with specific value


User constrains Boolean


User constrains False


------ End for loop ------

8.1.10 Enum values


User constrains Enum

User constrains None

User constrains Code

User constrains Type

User constrains Both


------ End for loop ------

8.1.11 Enum values


User constrains Enum

User constrains None

User constrains Modulo10


------ End for loop ------