Goal-based Testing of Semantic Web Services

M. Shaban Jokhio

Department of Computer Science, The University of Auckland, New Zealand

mjok002@aucklanduni.ac.nz

Abstract—Web services, the reusable software components,have brought automation to Internet computing. However,since they are currently described syntactically using XMLstandards, i.e., SOAP, WSDL and UDDI, the automationof web services tasks, e.g., web service discovery, selection,composition and execution, is still a challenge. In order tomake the most of automation in Service Oriented Architecture(SOA), the concept of the semantic web services (SWS), whichare described semantically using an ontology language, havebeen introduced. The research related to testing and qualityassurance aspects of web services is not mature [1]. This isespecially true for semantic web services, since research to-datehas mainly focused the automation of WS tasks. Furthermore,some semantic web service frameworks promote the client-oriented SOA, by formally specifying user requirements, called“goal specification”, and automatically resolve it by appropriateweb service detection. Keeping this in mind, we propose a novelapproach for testing semantic web services based on user goalspecifications. We believe this type of testing would add realvalue to web service users, unlike tradition web service testing.

Keywords-Web Services, Semantic Web Services, Web Ser-vice Testing, Verification and Validation, Formal Methods

I. INTRODUCTION

Web services (WS) are basic building blocks of services

over the Internet [2]. The ultimate objective of Service

Oriented Architecture (SOA) is to provide these services to

the clients with high accuracy and automation. However,

because web services are presently described syntactically

using XML-based standards, i.e., SOAP, WSDL and UDDI,

this objective is not achieved completely. To achieve this,

web services described semantically using some ontology

language, called semantic web services (SWS), are intro-

duced. This enables the service capabilities and behavior to

be understood by machines and WS tasks to be automated. A

few frameworks, e.g., WSDL-S, OWL-S and WSMO, have

been proposed.

Software testing is one of the crucial phases in the soft-

ware development process, consuming an average of 40% to

70% of software development resources [3]. Despite much

progress in SOA, research related to the testing and quality

assurance aspects of web services, is still not mature [1],

especially for semantic web services, which are relatively

new. This is also because of underlying challenges in web

service testing, i.e., web service have no user interface,

they are dynamic in nature [4], [5] and mapping from high

level semantic description to low level syntactic description

is an overhead in the testing process. The research in the

SOA community has mainly focused on the automation

of WS tasks including web service discovery, selection,

composition, mediation and execution, resulting in a lack

of research on web service testing.

A. Related Work

The work related to the testing of semantic web services

can be divided into three major categories, 1) WS testing

based on standard WS specification, 2) WS testing based

on semantic WS specification and 3) WS testing based on

commercial tools. In the first approach the test cases are

generated by syntactically parsing WSDL specifications [6],

[7], [4] or injecting some faults in web services using

mutation based testing techniques [8], [9]. In the second

approach, the OWL-S specification of web services is mainly

used for generating test-cases, e.g., Dai et. al [10] and Wang

et. al [11]. Dai proposed a methodology for ontology-based

testing of SWS. The given OWL-S specification of the SWS

was divided into input-output information and operational

behaviour, and the latter was represented as a state-machine.

The test-cases were generated by parsing different paths of

the state-machine, and analyzing the inputs and outputs of

every atomic process. Whereas, Wang proposed the contract-

based testing of SWS by verifying whether the service ful-

fills its contract as specified in the SWS profile. The test data

were generated from the service profile that describes the

service capabilities. Finally, there are some commercial tools

for WS testing available, e.g., [12], [13], [14], which help

in test-case generation, execution, replay and monitoring of

SOAP requests and response messages.

B. Limitations of related work and research question

There are two main limitations of the above approaches.

Firstly, these approaches perform only the syntax-based

testing of web services, i.e., the testing of interface signature,

sequence and input-output messages of operations and parts

and types of operation messages from WSDL specification.

Secondly, all these approaches use the provider’s web

service specification for the purpose of testing, e.g., WSDL

or OWL-S. This gives only the provider’s view of testing.

However, in the web service marketplace, the service user

is an important role, who uses a published service and is

concerned with its trustworthiness. To elaborate on this, we

know that there are three roles in WS marketplace, as shown

2009 IEEE/ACM International Conference on Automated Software Engineering

1527-1366/09 $29.00 © 2009 IEEE

DOI 10.1109/ASE.2009.74

691

2009 IEEE/ACM International Conference on Automated Software Engineering

1527-1366/09 $29.00 © 2009 IEEE

DOI 10.1109/ASE.2009.74

705

2009 IEEE/ACM International Conference on Automated Software Engineering

1527-1366/09 $29.00 © 2009 IEEE

DOI 10.1109/ASE.2009.74

705

2009 IEEE/ACM International Conference on Automated Software Engineering

1527-1366/09 $29.00 © 2009 IEEE

DOI 10.1109/ASE.2009.74

703

2009 IEEE/ACM International Conference on Automated Software Engineering

1527-1366/09 $29.00 © 2009 IEEE

DOI 10.1109/ASE.2009.74

707

in Figure 1, i.e., WS user who requests and uses desired

web services, WS provider, who provides and publishes web

services in a global registry, such as UDDI, and WS broker,

who serves as a third party for registering and searching.

Figure 1. Typical roles in WS marketplace

Amongst these three roles, WS user is more concerned

with the quality assurance (QA) of the web services, since he

uses the web services for achieving his goal. Therefore, the

ultimate objective of SWS testing should be to ensure a web

service user of the correctness of a service’s functionality.

The SWS testing should be done from the perspective of

the user’s specification. Whereas, there is no research so

far that addresses testing of web services based on the user

requirements specifications, to the best of our knowledge.

The existing WS technologies pay a little attention to the

client-side SOA. Hence, the importance of user requirements

is also ignored in SWS testing, to-date.Furthermore, in SWS frameworks, such as WSMO, a

user’s requirements are specified by a generic goal speci-

fication, and while the provider’s services are specified by

webservice specification. The goal specification is separate

and independent from the webservice specification. Every

concrete user request is an initialization of the generic goal

specification [2]. Furthermore, in an open and heterogenous

environment such as WWW, there is a high probability that

these two may differ. Therefore, a web service tested with

its own specification does not always ensure the robustness

to web service users. Thus, the question we address is, howto ensure the correctness of web services on all concreteuser requests to the end users?

II. PROPOSED SOLUTION

To ensure correctness of web services for the end user, we

need a methodology for testing web services based on user

goal specifications rather than web service specifications. We

propose a novel approach of investigating user requirements-

based testing of SWS. We call this goal-based testing ofsemantic web services. Figure 2 shows a high-level view of

our approach.

Figure 2. Overview of goal-based testing of semantic web services

We use the Web Service Modelling Ontology (WSMO)

framework to realize the goal-based testing of SWS. WSMO

defines user objectives in the form of WSMO goals as the

top level elements. The reason WSMO was chosen is that

WSMO goal specification has a rich semantic description

for specifying user objectives that can be used to identify

test cases for web service testing. WSMO components are

obtained from repositories of a WSMO execution environ-

ment, which consist of four elements, i.e., the goals, web

services, mediators and the ontology base. The OntologyBase has two components, i.e., the domain Ontology, which

consists of static domain information used by the goal

specifications and an Ontology Reasoner, which provides

useful inferencing for the purpose of reasoning based on

the domain ontology. The Goal Analyser component parses

the goal specification and extracts the useful information

from which the test cases can be generated, such as, the

goal capability, precondition, postcondition and transition

rules. It passes this information to the Test case Generatorcomponent to produce concrete Test cases based on standard

software testing techniques. The test cases are evaluated

by the Test case Evaluation process, which assures the

completeness and effectiveness of the generated test cases,

using the proposed coverage criteria. Evaluation Results are

fed back to the Ontology Base to improve the test case

generation process with the help of the ontology reasoner.

Test case Execution is achieved by running the test cases on

corresponding services which are connected to that particular

goal by mediators. The Execution Results are also fed back

to the Ontology Base to improve the test case generation

process.

The novelty of our approach is that unlike the traditional

approaches that give a provider’s view for testing, our

approach emphasizes the client’s requirements for testing

SWS. Since the test cases are generated from the goal

specification, which defines a generic template for concrete

goal requests, we can ensure the correctness of the tested

web service(s) with respect to the user. A web service that

692706706704708

passes all tests generated from a goal specifying particular

requirements for a user, is supposed to be robust from the

user’s end and can not fail on any concrete user request,

since every user request, is an instance of goal specification.

The application scenario is shown in Figure 3.

Figure 3. Application of goal-based testing of SWS

The goals representing the generic user objectives,

e.g., BuyABook, SearchFlight, BookFlight, BookHotel,

RentACar, etc., is written and stored in a WSMO repository

by ontology-engineers. A web service tester can retrieve

a goal specification and generate a set of test cases using

any test generation method, e.g., standard software testing

techniques or based on formal methods. The set of tests can

be executed after retrieving the web service(s) connected to

that particular goal. The testing results can be used to record

the correctness of the tested web service and can be referred

to by the user before the actual web service usage.

III. BACKGROUND

WSMO is an emerging framework for SWS. It is based

on the design principles of scalable mediation and strict

decoupling between its elements. It defines four top level

elements, i.e., ontology to formally specify the terms to be

used by other elements, webservice to model the description

of services, goal to formally specify the user objectives

and mediator to link any of these four components and

resolve mismatches. The Web Services Modelling Language

(WSML) provides the ontology language and the Web

Services Execution Environment (WSMX). An overview

of the relationship between the components is shown in

Figure 4 [15].

WSMO goals provide a formal specification of the user’s

expectations of the execution of the web service. A goal has

two important parts, i.e., requested capability- specifying

what functionality the user wants and requested interface-

specifying how the user wants to interact with the service.

Figure 4. Overview of the WSMO framework

In addition to this, a goal has some descriptive annotations

known as “non-functional properties” and can import an

external ontology and use mediators to resolve any mismatch

at the data, function and behaviour levels. A requested ca-

pability is defined in terms of a precondition, postcondition,

assumption and effect as WSML predicates. Precondition

defines the constrained information space for goal fulfillment

where as the postcondition defines the constrained output

space after the goal is fulfilled. The assumption and effect

describe conditions on the world state that must hold before

and after fulfilment of the goal. The requested interfacedefines the requested view of interaction for goal fulfilment

defined in terms of a requested choreography (a desired

format of interaction with the service that fulfills particular

goals) and a requested orchestration (a desired service flow

composition). Both contain two sub-elements, i.e., the state

signature, defining the states of concepts, which are used

to interact with the service and the transition rules which

dictate how the states of these concepts are changed based

on different conditions.

IV. SCOPE AND OBJECTIVES

There are many types of web service testing. We limit the

scope of our research as shown by the shaded area in Figure

5.

Figure 5. Scope of the research

Furthermore, we also refine the scope of our research with

respect to the WSMO framework by following assumptions.

693707707705709

• Goal specification contains all parts as defined in con-

ceptual framework, i.e., both the requested capability

and the interface specification.

• Both the web service and goal specifications use the

same ontology for defining terms, i.e., no mediator is

involved.

• Goal specification is defined at some concrete level,

which is enough to derive test cases.

We have set the following objectives in our research.

• Propose the solution for generating test cases from goal

specification and execute those in a real environment on

the web service(s) connected to the goal specification.

• Analyze test execution results and investigate how these

results can be used by web service users to trust web

service robustness.

• Propose solution for generating test-cases based on

ontology inference using the ontology reasoner.

V. CURRENT STATUS

We have made preliminary investigations related to the

problem definition and identified challenges that arise. We

have investigated various methods for test-case generation

and evaluation from the goal specification.

A. Proof of Concepts- Test Case Generation and Evaluation

We have investigated test case generation from goal

specifications using the standard software testing techniques,

such as boundary conditions and equivalence class testing.

To evaluate the effectiveness of generated test cases, we

propose the test case coverage criteria at different levels,

such as at WSMO goal and web service specification levels.

We use two coverage criteria at goal specification level,

i.e., boundary condition coverage, which states that every

boundary condition must be tested effectively and transition

rule coverage, which states that every transition rule must

be covered at least once. A web service specification can

restrict or loosen the conditions in a web service goal

specification, e.g., BuyBookGoal vs BuyBookbyCard web

service (restricts the condition). This is shown in Figure 6.

Therefore, we also propose to evaluate the test-coverage on

the corresponding webservice specification level. For more

details see [16].

Figure 6. WSMO goal Vs. web service specification

Since, a goal is a formal specification of user require-

ments, we have investigated possible applications of formal

methods-based test generation techniques on goal specifi-

cations, as shown in Figure 7. There are two possible ap-

proaches. One is using model checking, i.e., converting goal

specifications into a formal language and applying model

checking to generate counterexamples, e.g., converting goal

specification into B notation and using ProB model checker.

The second is using theorem proving, i.e., converting goal

specification into a formal notation supporting theorem

proving then converting the input domain into sub-partitions

and using a theorem prover to prove completeness and

disjointness of the input domain and finally generating tests

from individual input domain.

Figure 7. Test-case generation from WSMO goal specification using formalmethods

B. Formal specification based test case generation

Since goal specification is a state-based formal specifica-

tion for user requirements, we have used a model checking

approach for generating test cases from goal specifications.

Our approach is shown in Figure 8.

The goal specification described in WSML, is translated

to B abstract state machine. We have proposed a set of

translation rules from WSMO to B, for this purpose. A

set of the trap properties are calculated from the translated

B specification according to the particular test coverage

criteria. A Trap property is an assertion that is derived

as the asserted falsehood of a certain logical expression,

called test property. The trap properties are inserted into

B specifications in a form of assertions in the ASSER-

TIONS clause. Once a user goal specification is converted

to B notation, we use a B model checker to generate test

cases. The model checker generates counterexamples which

represent concrete test cases. This process is repeated for

the set of assertions representing the trap properties, to

obtain a set of test cases. Finally, the set of test cases are

694708708706710

Figure 8. Overview of the proposed test case generation framework usingmodel checking

converted to WSML format using B to WSML simple data

conversion rules, which can be ultimately used for the testing

purpose. We have evaluated our approach using the AmazonE-commerce Service specification case study [17].

VI. CONCLUSION

In this paper, we have proposed our approach for SWS

testing based on user goal specifications. We have discussed

the related work in this area and identified limitations and the

research question we are addressing. We have also discussed

our proposed solution to that, including our scope and

objectives. Finally, we have discussed the proof of concepts

and progress.

Our work may contribute in two-directions. Firstly, this

is a novel approach for SWS testing, based on user require-

ments. This will enable a SWS user to trust the correctness

of discovered services, w.r.t, his requirement specifications.

Secondly, our research can be used in the WSMO research

community, e.g., the results of the goal-based testing, can be

used to introduce a new QoS parameter to rank web services

in WSMX instead of simple instancebased ranking.

ACKNOWLEDGMENT

The author especially acknowledges and thanks to pro-

fessor Gillian Dobbie and Dr. Jing Sun from department

of Computer Science, the University of Auckland, New

Zealand, for their valuable feedback, guidelines and super-

vision on this work.

REFERENCES

[1] W.-T. Tsai, X. Zhou, Y. Chen, and X. Baiu, “On Testing andEvaluating Service-Oriented Software,” Computer, vol. 41,no. 8, pp. 40–46, 2008.

[2] M. Solberg, “Scalable Semantic Web Service Discovery forGoal-driven Service-Oriented Architectures,” Ph.D. disserta-tion, University of Innsbruck, 2008.

[3] A. Gargantini and E. Riccobene, “ASM-Based Testing: Cov-erage Criteria and Automatic Test Sequence,” J.UCS: Journalof Universal Computer Science, pp. 1050–1067, 2001.

[4] X. Bai, W. Dong, W.-T. Tsai, and C. Yinong, “WSDL BasedAutomatic Test Case Generation for Web Services Testing,” inProceedings of the IEEE International Workshop on Service-Oriented System Engineering. Washington, DC, USA: IEEEComputer Society, 2005, pp. 215–220.

[5] L. Baresi and E. D. Nitto, Test and Analysis of Web Services.Springer, 2007.

[6] W. T. Tsai, P. Ray, S. Weiwei, and Z. Cao, “Coyote: an XML-based Framework for Web services Testing,” in ”Proceedingsof 7th IEEE International Symposium on High AssuranceSystems Engineering”, 2002, pp. 173–174.

[7] H.M. Sneed and Shihong Huang, “WSDLTest - A Toolfor Testing Web Services,” in Proceedings of Eighth IEEEInternational Symposium on Web Site Evolution, Sep 2006.

[8] R. Siblini and N. Mansour, “Testing Web Services,” in Inter-national Conference on Computer Systems and Applications.Los Alamitos, CA, USA: IEEE Computer Society, 2005, pp.135–vii.

[9] W. Xu, J. Offut, and J. Luo, “Testing Web Services by XMLPerturbation,” in Proceedings of the 16th IEEE InternationalSymposium on Software Reliability Engineering. Washing-ton, DC, USA: IEEE Computer Society, 2005, pp. 257–266.

[10] G. Dai, X. Bai, Y. Wang, and F. Dai, “Contract-Based Testingfor Web Services,” in Proceedings of the 31st Annual In-ternational Computer Software and Applications Conference.Washington, DC, USA: IEEE Computer Society, 2007, pp.517–526.

[11] Y. Wang, X. Bai, J. Li, and R. Huang, “Ontology-Based TestCase Generation for Testing Web Services,” in Proceedingsof Eighth International Symposium on Autonomous Decen-tralized Systems, 2007, pp. 43–50.

[12] “MindReef.” [Online]. Available: http://www.mindreef.com/

[13] “SOAPTest.” [Online]. Available: http://www.parasoft.com

[14] “SOAPUI.” [Online]. Available: http://www.soapui.org

[15] D. Roman, H. Lausen, and U. Keller, “WSMO,WSMX andWSML Working Drafts,” October 2006, wSMO WorkingGroup, available at http://www.wsmo.org/TR/.

[16] M. S. Jokhio, G. Dobbie, and J. Sun, “Towards SpecificationBased Testing for Semantic Web Services,” in Proceedingsof the 20th Australian Software Engineering Conference,ASWEC ’09, April 2009.

[17] J. Kopecky, D. Roman, and J. Scicluna, “WSMO Use Case:Amazon E-commerce Service,” 2005. [Online]. Available:http://www.wsmo.org/TR/d3/d3.4/v0.1/

695709709707711