252 ieee transactions on software engineering, …referring throughout to a simplified air traffic...

Test Synthesis from UML Modelsof Distributed Software

Simon Pickin, Claude Jard, Thierry Jeron, Jean-Marc Jezequel, Member, IEEE, and Yves Le Traon

Abstract—The object-oriented software development process is increasingly used for the construction of complex distributed

systems. In this context, behavior models have long been recognized as the basis for systematic approaches to requirements capture,

specification, design, simulation, code generation, testing, and verification. Two complementary approaches for modeling behavior

have proven useful in practice: interaction-based modeling (e.g., UML sequence diagrams) and state-based modeling (e.g., UML

statecharts). Building on formal V&V techniques, in this article we present a method and a tool for automated synthesis of test cases

from scenarios and a state-based design model of the application, remaining entirely within the UML framework. The underlying “on

the fly” test synthesis algorithms are based on the input/output labeled transition system formalism, which is particularly appropriate for

modeling applications involving asynchronous communication. The method is eminently compatible with classical OO development

processes since it can be used to synthesize test cases from the scenarios used in early development stages to model global

interactions between actors and components, instead of these test cases being derived manually. We illustrate the system test

synthesis process using an air traffic control software example.

Index Terms—Formal methods, testing tools, object-oriented design methods.

Ç

1 INTRODUCTION

THE UML (“Unified Modeling Language”) notation [30]has now become the standard notation of object design

methodologies. As a consequence, the need for the auto-matic synthesis of functional test cases from UML models isincreasingly being felt in industry. Furthermore, UML isused in an ever wider range of contexts, in particular, that ofdistributed system development. The testing of distributedapplications has to take into account their use of asynchro-nous communication and their inherent concurrency.

In general, in conformance testing of concurrent applica-tions, testing of all possible invocation orderings isunrealistic due to a combinatorial explosion in the numberof such orderings permitted by the specification. Thus, inapplications involving concurrency, user-defined test objec-tives constitute a way of limiting the number of test cases tobe produced by test synthesis from a specification. Testobjectives can be described in the form of high-level testscenarios, which are then easily understood as behavioral test

patterns by developers.

Other advantages of using test case synthesis according totest objectives for both centralized and distributed applica-tions are the following:

. Productivity gain. A test objective specifies theessential aspects of a test, independent of low-leveldesign and implementation choices. While defininga high-level test scenario is not difficult when themain classes are identified, refining and adapting itto the final software product is an arduous process.Automating the completion of the test objective withthe low-level design details—obtained from theUML model—holds out the promise of significantproductivity gains.

. Coherent development process. The main expectedbehaviors can easily be represented as test objec-tives. Such test objectives can be derived from use-case scenarios, contributing to the overall coherenceof the development process.

. Version/product independence. Test objectives canbe chosen to be independent of software versionsand variants. This is particularly important in aproduct-line context [1], since it enables generictest objectives to be defined for an entire productline [27].

In this paper, we present a method, first described in[31], along with the prototype tool that supports it, for theautomated synthesis of UML test cases (with built-in oraclein the form of test verdicts) from UML test objectives and aUML system model. We thereby demonstrate the feasibilityof applying formally based test synthesis methods and toolsto UML models, without the user having to leave the UMLdomain. Here, we describe the method and supporting toolin more detail than in [31] and deal more fully with theissue of the UML models to which it is applicable.

252 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 33, NO. 4, APRIL 2007

. S. Pickin is with the Departamento de Ingenierıa Telematica, UniversidadCarlos III de Madrid, Madrid, Spain. E-mail: [email protected].

. C. Jard is with ENS Cachan, Campus de Ker Lann, F-35170 Bruz, France.E-mail: [email protected].

. T. Jeron is with IRISA/INRIA, Campus Universitaire de Beaulieu, F-35042Rennes Cedex, France. E-mail: [email protected].

. J.-M. Jezequel is with IRISA, Campus Universitaire de Beaulieu, F-35042Rennes Cedex, France. E-mail: [email protected].

. Y. Le Traon is with ENST Bretagne, 2, rue de la Chataigneraie, CS 17607,35576 Cesson Sevigne Cedex, France.E-mail: [email protected].

Manuscript received 29 Apr. 2005; revised 29 Sept. 2006; accepted 10 Oct.2006; published online 1 Mar. 2007.Recommended for acceptance by S. Uchitel, I.H. Krueger, M. Broy, andJ. Whittle.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference IEEECS Log Number TSE-0114-0405.

0098-5589/07/$25.00 � 2007 IEEE Published by the IEEE Computer Society

In defining our method, we address the following issuesconcerning conformance testing in a UML framework:

. the definition of a complete process with a formalbasis to synthesize test cases from UML modelsaccording to test objectives,

. the definition of a formal operational semantics forUML models, and

. the definition of a scenario-based language withinthe UML framework to express test objectives andtest cases.

Though the user deals only with UML, the formal basismeans that we have a precise notion of what is being testedand what is the meaning of a verdict. This underlyingformal basis is the synchronous product of Input-OutputLabeled Transition Systems (IOLTS); see the Appendix. Thetool implementing the method results from the incorpora-tion of the TGV tool into the Umlaut UML environment.Umlaut [16] is a CASE tool that manipulates the UMLmetamodel, enabling automatic model transformation. TGV[23] is a test synthesis tool based on an on-the-fly and partialtraversal of the enumerated state graph of the specification.It was chosen here for its formal basis, for the desirableproperties which have been proven of its test synthesisalgorithms, for the ability to treat systems of significant sizewhich its on-the-fly approach confers, and, finally, for thefact that it has demonstrated its capabilities in test synthesisfrom SDL [20] and Lotos [18] specifications.

The synthesis method could be usefully applied to gray-box testing, to the testing of individual components as wellas to testing the integration of different components. As wediscuss below, however, it is currently most easily appliedto black-box system testing, on which we concentrate in thisarticle. Being based on black-box testing, it does not rely onthe implementation under test (IUT) being derived frominstrumented code, unlike many other approaches to testingderived from UML descriptions.

In Section 2, we give an overview of the method,dividing it into four main parts. The first two of theseparts—concerning mapping the UML representation of theapplication model and that of the test objectives to theformal notation—are presented in Sections 3 and 4,respectively. The next part, concerning the test synthesison the formal models, is presented in Section 5. Section 6presents the last part, the mapping from a formal test caseto UML. In Section 7, we discuss the future enhancementsof our method. In Section 8, we place our work in context,and we conclude in Section 9. We provide the definition ofthe terms that are used throughout the paper in theAppendix.

We illustrate the method described in this article byreferring throughout to a simplified air traffic controlsystem example.

2 OVERVIEW OF THE METHOD

Fig. 1 gives an overview of the method for synthesizing testcases from a UML model, according to test objectives. Theinputs to the method are a test objective—in a UMLsequence-diagram-based notation and representing a high-level behavior scenario—and a UML model of the system

to be tested, including a UML object or deployment

diagram describing the initial state. The output is an

automatically synthesized test case—again in a UML

sequence-diagram-based notation—exactly defining the

ordering of call sequences and associated test verdicts

and including any required object creation or destruction.

The input and output can be provided as XMI files,

facilitating the use of different tools for the initial modeling

and the visualization of the synthesized test cases. In

particular, we used an XMI interface to the Objecteering

tool (see http://www.objecteering.com/).The method is divided into four main parts:

1. Formal specification derivation, in which a labeledtransition system (LTS) semantics (see the Appen-dix) is given to the UML model via the generation ofa simulation API that enables the LTS to be builtincrementally on demand (termed “on the fly”); seeSection 3.

2. Formal objective derivation, in which an LTSsemantics is given to the test objective representedin the sequence-diagram-based language; currently,this is not implemented on the fly; see Section 4.

3. Test synthesis on the formal models, in which a testcase, in the form of an IOLTS, is synthesized fromthe specification LTS according to the test objectiveLTS; see Section 5.

4. UML test case derivation, in which a sequence-diagram based representation of the test case isderived; see Section 6.

The sequence-diagram notation used was obtained by

adding constructs from the MSC notation [21] to UML

sequence diagrams in order to adequately handle concur-

rency, choice, asynchronous communication, internal ac-

tions, etc.; see [33]. Though a similar approach has been

taken to define the UML 2.0 sequence-diagram notation

[30], the syntax is not the same due to the fact that the two

languages were contemporary.

2.1 Restriction to System Testing

In the case of system testing, i.e., testing of implementations

of the whole UML model in which the external actors

constitute the IUT environment, there is a relatively good fit

between the policy toward inter-IUT-environment commu-

nication at the UML level and that at the IOLTS level. A

UML model cannot contain communications between

external actors. Similarly, an IOLTS as used by the TGV

tool cannot contain actions internal to the IUT environment

(and, in consequence, tests synthesized by TGV cannot

contain actions internal to the tester).In the system testing case also, the information required

to define an IOLTS from an LTS can easily be obtained

automatically from the UML model. This information is the

third input to the test synthesis, in addition to the two LTSs,

representing the semantics of the UML model and that of

the test objective, as shown in Fig. 1.Hereinafter, we will generally refer to the SUT (System

Under Test) rather than to the IUT (Implementation Under

Test).

PICKIN ET AL.: TEST SYNTHESIS FROM UML MODELS OF DISTRIBUTED SOFTWARE 253

3 FROM UML MODEL TO LTS

In this section, we show how the semantics of a UML modelis derived. First, we discuss the executability of the modeland then present part of the UML model of the exampleused to illustrate the method. Next, we present thetransformation of an executable UML model into one moreamenable to the derivation of an LTS, followed first by thederivation of the LTS from the transformed model andsecond by the derivation of an IOLTS from this LTS. Finally,we discuss the testability of UML models via this method.

3.1 An Executable UML System Model

The UML model must contain a class diagram, an objectdiagram, and a state machine for each of the main classes(usually including all the active classes).

3.1.1 The Dynamic Behavior of the System Objects

Deriving an LTS from the system model requires a dynamicbehavior specification for all system objects. Therefore, forsystem classes where a state machine is not provided, apassive “daisy” state machine—a machine having only onestate and a set of loop transitions (“petals”), each having anempty action expression and being triggered by theinvocation of one of the operations of the correspondingclass—is assumed. In the case of active classes, the looptransitions include spontaneous transitions invoking opera-tions of associated classes.

3.1.2 The State Machine Action Language

The behavior of the system as a whole is given by thecombined execution of the state machines contained in thesystem and the state machines that model its environment,taking into account the state of the active-object event

queues. However, transitions of UML state machines areparameterized by actions, which can be placed on transi-tions or in states. Thus, to make a UML state machineexecutable, an operational semantics must be given for thelanguage used to describe the actions.

The action expression of a UML state machine transitionmay be quite complex since the granularity of transitions inUML state machines is often quite coarse. Until UML 1.5, nosyntax was prescribed for these expressions. The Umlautsimulator currently uses an ad hoc action language in statemachine transitions to describe the creation/destruction ofobjects and links, the assignment of values to attributes, theinvocation of methods, etc. This syntax must therefore beused in the UML model defined in “your favorite case tool,”where it is treated simply as text. See Fig. 3 for some verysimple examples of this syntax.

In order to use the action language, we need to knowhow an object is to refer to other objects of the model. In theUmlaut simulator, each object has an implicit attribute foreach association in which it is involved, and the identifier ofthis attribute is the role name appearing at the other end ofthe association in the class diagram. If no role name isdefined, the name of the corresponding object is used. In thecase of an association with multiplicity greater than one, allobjects of the same class linked via an instance of the sameassociation play the same role. In such cases, rather thanidentifying a simple attribute, the role name identifies anarray in the Umlaut simulator action language.

3.1.3 The Environment of the Model

To simulate UML models exhaustively, we must alsosimulate an environment able to send any acceptable input,that is, the simulated system must be closed. In Umlaut, thesystem is closed by treating the external actors as active


Fig. 1. The synthesis method.

objects and providing them with state machines (explicitly

or implicitly) that send input to the system and receive

output from it.As is standard practice in the interleaving semantics

case, we reduce the number of possible interleavings by

making the assumption that the environment is “reason-

able,” i.e., it only sends stimuli to the system when the

system is in a stable configuration and is thus able to

proceed with the input immediately. This corresponds to an

assumption that the system is much faster than its

environment.

3.1.4 The Initial State of the Model

Class diagrams describe the possible configurations of

objects in the system. State diagrams describe how config-

urations, or global states, can evolve. The third element

required in order to derive an LTS is a specification of the

initial configuration. This information may be provided via

an object diagram, though if we also wish to show the

localization of each object in a distributed application, a

deployment diagram can be used. Note that an arbitrary

configuration cannot necessarily be represented by such a

diagram, nor is this necessary for test synthesis; cf. the

notion of test preamble.

3.2 The UML Model of the ATC Example

The class diagram of the ATC case study is shown in Fig. 2.

The system consists of four classes: Flight and FlightPlan,

used to store flight data, and FlightPlanManager and

ControllerWorkingPosition, which control the system and

interact with the environment. The thick borders on the

class diagram indicate that the latter two classes are active.As an example of a state machine, in Fig. 3 we provide

that of the (passive) class FlightPlan.


Fig. 2. The class diagram of the ATC application.

Fig. 3. The state machine of the passive FlightPlan class.

The initial configuration is represented in Fig. 4. Thediagram shows that the deployed system knows of twoflights (each with an associated flight plan), initially locatedout of the area managed by the ATC. Only two flights areused in order to keep the example simple.

3.3 Transforming the UML Model

The model is first transformed into an equivalent one usinga much simpler subset of UML, consisting mainly of classesand operations. The state machines of the specification aretransformed using a variant of the State design pattern [10]:the states are reified, and substates are represented by aninheritance hierarchy.

An active-object state machine has an event queue and athread that dispatches events taken from the queue. Theevent queue and the thread are made explicit in thetransformed model (which contains classes dedicated tothese concepts).

Each state of an active-object or passive-object statemachine is seen as a specific subtype of the class to whichthe state machine is attached (its context). This subtype hasthe same interface (signature) as the context: This interfacecomprises one operation for each event that the statemachine can react to. Note that the transformation musttake into account UML policy on unspecified reception (ifan event does not trigger any transition, it is discarded).1

If a given state has an outgoing transition triggered by agiven event, then the subtype corresponding to this (sub)staterealizes the operation corresponding to that event with amethod whose body contains the effect of the transition(that is, the set of actions to be executed when thetransition is fired). Under this scheme, a transitionbetween states of an object’s state machine is transformed

into a change from one type to another (UML supports

dynamic and multiple classification).The initial configuration described in the object or

deployment diagram is also transformed so as to include

the objects representing communication queues and event

dispatchers.

3.4 Deriving an LTS from the Transformed Model

Umlaut generates a simulation API with which to construct

an LTS defining the semantics of the UML model. This API

can then be used by TGV (or other tools) to construct all or

part of this LTS as required, enabling on-the-fly treatment.

The simulator API provides functions for

. the calculation of the initial global state, i.e., the stateof the whole system of objects,

. the calculation of the fireable transitions in a givenglobal state,

. the calculation of the successor state, given a globalstate and a transition which is fireable in that state,and

. the comparison of global states.

In a given global state, the set of fireable transitions is

calculated by enumerating the control states of the active

objects in the system (recall that after the transformations

described in the previous section, a control state is

represented as a type). The global states are stored using

a deep copy of the whole object structure. The comparison

function between global states enables cycles and con-

fluences in the accessibility graph to be detected. This

comparison function relies on local state comparison

functions for each class of objects.Concerning the implementation language of the simu-

lator, UML is compiled to an object-oriented programming

language first, to facilitate the handling of dynamic creation

and destruction of objects and, second, to facilitate the

implementation of dynamic reclassification.


1. We consider that an invocation to an operation that is not defined inthe class diagram, or the sending of a signal that is not so declared, is anerror that can be detected statically. Thus, our UML state machines onlyneed to be semantically completed with respect to the set of operations/signals defined in the corresponding class diagram.

Fig. 4. The object diagram of the ATC application.

3.4.1 Transitions of the Generated LTS

In the current implementation of Umlaut, global statescorrespond to “stable” configurations of the UML system,that is, configurations in which each object of the system isin a well-defined state of its associated UML state machine.Since calls to active objects are assumed to be asynchronousand pass via FIFO event queues, each LTS transition, that is,each simulation step, corresponds to a run-to-completionstep of an active-object state-machine, i.e., an active objectmethod execution. In state machine terms, this is one of thefollowing:

. The execution of the action expression of atriggered active-object transition whose guard eval-uates to true (i.e., a method execution provoked byan invocation), together with that of the actionexpressions of any passive object transitions in-voked from within this action expression (and soon, recursively).

. The execution of the action expression of a sponta-neous active-object transition whose guard evaluatesto true, together with the execution of the actionexpressions of any passive object transitions invokedfrom within this action expression (and so on,recursively).

In both cases, the executed action expressions mayinvolve invocations to other active objects, each suchinvocation leading to a new transition in the next simulationstep. Using LTS transitions that correspond to active object“run-to-completion” steps, rather than to general state-machine microsteps, defines a rather course-grain seman-tics but avoids the structure of the derived LTS beingdependent on the action language which parameterizesUML state machines.

Apart from the current state of the state machine of eachof the objects (represented as a type after precompilation),the other information used to define a global state of thesystem is the following:

. the values of the attributes of each of the objects,

. the state of the communication queues of the activeobjects, and

. the structure of the links between objects.

In accordance with the policy on unspecified reception inUML state machines, the Umlaut simulator generates a loopLTS transition from the reception of an asynchronousinvocation by an active object whether or not a correspond-ing transition is specified in the object’s state machine.

3.4.2 Labels on Generated LTS Transitions

Test synthesis proceeds by matching the labels on thetransitions of the test objective LTS (which may includeregular expressions) with those on the transitions of thesystem LTS. The labeling of the LTS transitions is thereforeof some importance, since it determines not only the labelson the transitions of the test cases—the test synthesisoutput—but also the labels that must be used in the testobjectives—one of the test synthesis inputs.

The format of the labels generated by Umlaut is of one oftwo types depending on the nature of the active object statemachine transition initiating the run-to-completion step:

. Reception of an asynchronous invocation. The LTStransition generated for

1. the reception at the active object origin of anasynchronous invocation, with parameter va-lues val1, . . . , valn, of the operation opname,

2. the execution of the action expression of thecorresponding state machine transition, and

3. the consequent execution of the action expres-sion of any other state-machine transitionscontained in the same simulation step,

is labeled

origin?opnameðval1; . . . ; valnÞ:

Notice that, on reception of an asynchronousinvocation, at the application level there is noknowledge of the identity of the sender, so thatwith the current implementation, such informationdoes not figure in the LTS transition label.

. Spontaneous transition. The LTS transition gener-ated for

1. the execution of the action expression havingthe Umlaut action-language textual descriptionspontaneous transition action expression of thespontaneous state-machine transition of theactive object origin and

2. the consequent execution of the action expres-sion of any other state-machine transitionscontained in the same simulation step

is labeled

origin!spontaneous transition action expression:

In the current implementation, the text of the actionexpression is simply copied, without instantiating anyvariables in it. Whether or not the transition is guarded,the guard does not figure in the transition label (since anLTS transition should not be generated if the guardevaluates to false).

If the action expression comprises a single invocation ofthe operation opname belonging to the object playing therole receiver role, with parameters param1, . . . , paramn, theaction language used means that the transition label willhave the following simple syntax:

origin!receiver role:opnameðparam1; . . . ; paramnÞ:

If no receiver role is defined, the name of the receivingobject is used. Note that the characters “!” and “?” arereserved so that any occurrence in an identifier must beescaped.

3.5 Deriving an IOLTS from the LTS

In this section, we discuss the generation of the informationneeded in test synthesis to derive an IOLTS from an LTS.We refer to the unit of behavior described by an LTStransition label as a �-action, in order to distinguish theseactions from UML actions since, in general, the two types ofaction are not in 1-1 correspondence.


The derivation of an IOLTS from an LTS has two aspects(see the Appendix):

. The partition of the �-actions into internal actionsand visible actions, thus defining the SUT interface.This is the role of the characteristic function �hide ofthe IOLTS definition (see the Appendix).

. The partition of the visible �-actions into SUT inputsand SUT outputs. This is the role of the characteristicfunction �io of the IOLTS definition (see theAppendix).

The derivation requires relating the synchronous com-munication architecture required by TGV to the commu-nication architecture assumed by the Umlaut simulator,taking into account the atomicity of the generated LTS.

3.5.1 The Characteristic Function �hide

The system testing case can be dealt with by consideringboth the communication queues of the external actors andthose of the SUT active objects that communicate with themto be internal to the SUT. The transitions of the derivedIOLTS labeled by visible �-actions will then be thosegenerated from the run-to-completion steps of the actorstate machines.

3.5.2 The Characteristic Function �io

With the SUT interface as defined in the previousparagraph, an SUT input can be defined as the action ofan external actor placing an invocation event in the inputqueue of an SUT active object, while an SUT output can bedefined as the action of an external actor consuming aninvocation event placed in its input queue by an SUT object.In order to ensure that all the LTS transitions correspondingto communications between the SUT and its environmentare labeled by one of these two types of actions, certainrestrictions must be imposed on the actor state machines;see Section 3.6.

3.5.3 Deriving the Characteristic Functions

The information needed to derive an IOLTS from the LTS isprovided to the TGV tool in files whose location is indicatedvia the command line options -hide (for the function �hide)and -io (for the function �io). The former file defines thesyntactic rules to identify labels used for hidden (orequivalently visible) actions, while the latter file definesthe syntactic rules to distinguish the labels used for SUTinputs (or equivalently SUT outputs) from among thevisible �-actions. In accordance with the labeling of thevisible transitions discussed in the previous sections, thesefiles are automatically generated from the UML modelaccording to the following specification:

. the visible �-actions are defined to be those whoselabel begins with an actor name, and

. the SUT inputs are defined to be those visible actionswhose label contains the character “!”.

As a consequence of this definition, actor autoinvoca-tions involving only internal processing would beclassified as SUT inputs. However, these are ruled outby the above-mentioned restrictions on actor state-machine transitions.

3.6 Testable UML Models

As is the case for other approaches to deriving anoperational semantics from UML models such as [26],[13], or [8], restrictions must be placed on the UML modelthat can be used as input to our method in order to be ableto derive that semantics. In the case where the semantics isthen used for test synthesis, additional restrictions must beimposed on the model for the following three reasons:

. in order to obtain meaningful labels on the transi-tions of the generated LTS, particularly those labelsthat define the visible actions,

. in order to be able to divide the visible actions intoinputs and outputs in a sensible manner,

. in order to be able to match actions from the testobjective with those of the system model.

Making these restrictions explicit helps to clarify thefeasibility of the current method as well as throwing lighton its possible enhancements. In this section, therefore, wepresent the restrictions imposed on the input UML model,dividing them into four categories, each corresponding toone of the above reasons.

3.6.1 Deriving an LTS

In Umlaut, active objects—and in particular externalactors—are assumed to communicate asynchronously viaFIFO event queues. The operations of the correspondingclasses have no return value and their state-machines mayhave both spontaneous and triggered transitions. Theemission of an invocation to an active object correspondsto the placing of an event in the event queue of that object.The reception of an invocation by an active objectcorresponds to the removal of an event from its eventqueue and the execution of the corresponding method.

Passive objects, on the other hand, are assumed tocommunicate synchronously without any intermediateevent queues. The operations of the corresponding classesmay or may not have a return value and their state-machines cannot have any spontaneous transitions.

These suppositions are consistent with a style ofdistributed system modeling in which all nonlocal callsare implemented as asynchronous calls to active objects.

Since the aim of our work was to demonstrate feasibilityrather than develop an industrial-strength tool, for simpli-fication purposes, we stipulate that hierarchical statemachines, i.e., those having composite states, cannot beused. Nor can deferred events, history pseudostates oraction expressions in states (including entry and exitactions). One way of lifting the former restriction wouldbe to incorporate work on flattening statecharts, suchas [36].

Finally, the method uses enumerative techniques and, asa consequence, data structures having infinite or very largedomains cannot be used.2 In contrast to the situationregarding data structures, the on-the-fly techniques usedmean that the tool chain can deal with arbitrarily large stategraphs where the state explosion arises from the controlstructures.


2. The INTEGER type used in the example is, in fact, bounded (and,therefore, an enumerative type).

3.6.2 Ensuring Meaningful Labels

We first make the not unreasonable assumption that allprocessing is by method invocation and restrict the actionexpressions used on spontaneous transitions of active objectstate machines to operation invocations.

If we wish to model an active object spontaneouslyperforming internal processing (e.g., initializations, dataobject creations, etc.), we stipulate that two state-machinetransitions must be used. The first is a spontaneoustransition whose action expression contains only a singleauto-invocation—therefore, a transition of the form stipu-lated in the previous paragraph—and the second istriggered by the first and contains the required processingin its action expression (possibly together with otherinvocations). Notice that a single LTS transition will begenerated from such a pair of state-machine transitions.

3.6.3 Dividing Visible Actions into Inputs and Outputs

Of the following three restrictions on the input UMLmodels, the first two arise from the use of the course-grainsemantics, in which visible LTS transitions correspond to“run-to-completion” steps of active object state machines,while the third is due to a limitation of the TGV tool.

We first wish to ensure that the removal of an event fromthe input queue of an actor, together with any subsequentprocessing, is a well-defined SUT output. We thereforestipulate that the action expression used on an actor state-machine transition to be triggered by an invocation from theSUT cannot contain any invocations to the SUT. Thisrestriction also helps to ensure that each invocation of anSUT operation by an external actor appears in the label of avisible LTS transition.

We next wish to ensure that each placing of an event inthe input queue of an SUT active object, as a consequence ofan invocation occurring in the action expression of an actorstate-machine transition, generates an LTS transition de-fined as an SUT input. We therefore stipulate that the actionexpression used on a spontaneous transition of an externalactor state machine must contain a single operationinvocation.

Finally, recall that in TGV, actions internal to the SUTenvironment cannot be handled. Any spontaneous actorinternal processing of the kind discussed above musttherefore be carried out as part of an SUT input. Bearingin mind also the reason for the previous restriction, wetherefore stipulate that the action expression of an actorstate machine transition that is triggered by a spontaneousactor autoinvocation must contain a single invocation tothe SUT.

3.6.4 Matching the Actions

Due to the use of the course-grain semantics and the currentlabeling function, the return value of a synchronousinvocation can never appear in a transition label. A valuesent from the SUT to an external actor cannot therefore bematched with any label on a test objective. With the currentimplementation, then, if such a return value is important toa test objective, the synchronous call should instead bemodeled as two asynchronous calls, one in each direction.To avoid such situations arising, we currently impose the

perhaps overly severe restriction that external actors canonly communicate with SUT active objects (by assumption,only via asynchronous calls).

To facilitate the use of SUT inputs generated fromspontaneous actor autoinvocations in test objectives, werestrict the name that can be used for actor operations to bespontaneously autoinvoked by imposing the followingnaming convention: Any operation that is spontaneouslyautoinvoked by an actor must have the same name as theSUT operation invoked in its action expression.

4 FROM UML TEST OBJECTIVE TO LTS

In this section, we present the derivation of an LTS from aUML test objective. Before doing so, however, we look moreclosely at the notion of test objective that is crucial to themethod and illustrate the LTS derivation using a testobjective for the ATC example.

4.1 Test Objectives

Here, we look at the notion of test objective at the UML andLTS levels.

4.1.1 UML Sequence-Diagram Test Objectives

In the test synthesis from UML models presented in thisarticle, test objectives are defined as one or more UMLsequence diagrams of two types:

. Accept scenarios (usually only one). Sequencediagrams specifying an abstract view of the SUTbehavior that the test designer wants to test, to beused as positive criteria for selecting the behavior tobe tested.

. Reject scenarios. Sequence diagrams specifying anabstract view of the SUT behavior that the testdesigner wants to avoid in the test, to be used asnegative criteria to avoid selecting unwantedbehavior.

The generated test case is accepted by each of the acceptscenarios and by none of the reject scenarios. Thesesequence diagrams use elements of the UML model to betested and may also require other classes to be defined. Thesyntax is a restricted version of that used for the generatedtest cases; see Section 6.

An action labeling a send or receive event of a testobjective incorporates the following information: the nameof the sending and receiving component, the method nameand the method parameters. The abstraction with regard toSUT behavior is obtained in two ways:

. by allowing the specification to perform actions thatare not specified in the test objective when matchingthe two, and

. by using wildcards in the component names(appearing in lifeline headers), in method names,and in method parameters.

In the test synthesis from UML models using Umlaut/TGV, due to the atomicity of the semantics currentlyimplemented by the Umlaut UML simulator, we cannoteasily use test objectives containing SUT-internal actions.For this reason, though the underlying tool allows internal


actions to be used, the actions of a test objective arerestricted to (abstract views of) communication actionsinvolving the SUT and its environment.

4.1.2 LTS Test Objectives

In test synthesis using TGV, test objectives are defined asLTS containing two types of sink states:

. Accept states. If a transition arriving at such a statein the specification LTS is fired, the test objective isachieved.

. Reject states (error states). If a transition arriving atsuch a state in the specification LTS is fired, the testobjective cannot be achieved.

These sink states are thus used to select the behaviorto be tested from the behaviors described by thespecification LTS.3 Due to the fact that the states of anLTS contain no information, the two sink states are in factmodeled as states having a single appropriately labeledoutgoing transition.

The transitions leading to a reject state specify the partsof the specification LTS which do not need to be explored inorder to synthesize a test case; the use of such states canconsiderably improve the efficiency of test synthesis. Theymay be used to exclude actions that are known to activelyinterfere with the purpose of the test. Perhaps the mostcommon use, however, is to help to minimize thesynthesized test case by excluding actions that are knownto be superfluous for the purposes of the test. Thisreduction of “noise” is particularly useful for synthesizingtests of concurrent applications in an interleaving-modelcontext such as that used here.

The actions of the test objective are simply the labels onthe test objective LTS transitions. The abstraction withregard to IUT behavior is obtained in three ways:

. by allowing the specification to perform actionswhich are not specified in the test objective whenmatching the two; this is done by implicitlycompleting all states of the test objective with regardto the alphabet of actions of the specification withtransitions looping back to the same state,

. by allowing the use of “otherwise” transitions; this isdone by interpreting transitions labeled by anasterisk as explicitly specifying completion withregard to to the alphabet of actions of the SUTspecification, and

. by allowing individual test objective actions to beabstractions of individual SUT actions (using regularexpressions).

4.2 A Test Objective for the ATC Example

In the ATC example, suppose we want to generate a test tocheck that flight number 0 will be correctly assumed (i.e.,that the controller ctrl will be notified that the flight status isupdated with the value 4) when the flight enters the ATCzone (i.e., when the radar ptg updates the flight position totrue, meaning “in zone”). This is described by the acceptscenario of Fig. 5a. To help to produce a test case, we alsospecify that we are not interested in any of the eventsconcerning flight number 1. This is described by the rejectscenario of Fig. 5b.

From these scenarios we then derive first the LTSs ofFig. 6 and then the LTS representation of the test objectiveshown in Fig. 7.

The loop transitions for the rest of the alphabet of themodel (the “otherwise” transitions)—in states 0 and 1 of theLTS of Fig. 6a and the LTS of Fig. 7, and in state 0 of the LTSof Fig. 6b—are implicit.


3. By comparison, the sink states of an LTS defining a test casecorrespond to states in which a test verdict is derived.

Fig. 5. UML test objective for the ATC example.

Fig. 6. Accept and reject LTSs derived from test objective of Fig. 5.

Fig. 7. LTS representation of test objective of Fig. 5.

Since states contain no information in an LTS, a

transition to an accept or reject state is actually modeled

as a transition to a state whose only outgoing transition is

labeled accept or reject, respectively. Both of the LTSs of

Fig. 6 have classic automata-theory accept states (except

that they are represented using an accept transition); the

reject state of Fig. 7 arises on completing the intersection of

the LTS of Fig. 6a with the negation of the LTS of Fig. 6b as

explained in Section 4.4.

4.3 Deriving an LTS from a Single Scenario

Test synthesis involves matching the labels on the transi-

tions of the LTS generated from the test objective with those

of the LTS generated from the UML model. The atomicity

and transition labeling scheme of the LTS derived from a

sequence diagram representing an accept or reject scenario

must therefore be consistent with the atomicity and labeling

scheme of the LTS derived from the UML model of the

system, as described in the previous section. The same

applies to the definition of the SUT interface and the

division of actions into inputs and outputs.

4.3.1 Consistent Atomicity

Recall that a single action of the LTS generated from the

UML system model—a �-action—may subsume a set of

UML actions. To ensure the correspondence between the

atomicity of the two LTSs, a single LTS transition must also

be generated from the same set of UML actions occurring in

a sequence diagram representing an accept or reject

scenario of a test objective.A systematic use of focus bars facilitates grouping the

UML sequence diagram events labeled by actions that

belong to the same active-object “run-to-completion” step.

However, given the restrictions imposed on actor state

machines, if we only generate LTS transitions labeled by

visible actions, that is, we only generate LTS transitions

from events on actor lifelines, the situation is much less

complex. To simplify the compatibility between the two

LTSs, we therefore do not take advantage of the possibilities

offered by TGV of generating LTS actions that are SUT-

internal from test objectives.With this simplification, the �-actions labeling the

transitions of the LTS derived from a test objective sequence

diagram are either actor emissions or actor receptions.

4.3.2 Consistent Labeling

Ensuring that the same labeling scheme is used for

triggered transitions (recall origin?opnameðval1; . . . ; valnÞ)s relatively straightforward. However, ensuring that the same

labeling scheme is used for spontaneous transitions (recall

origin!spontaneous transition action expression) is more com-

plicated since, in the general case, the full action expression

does not appear in a sequence diagram representation.

However, due to the restrictions imposed on actor state

machines, deriving only transitions labeled by visible

actions from a test objective sequence diagram, i.e., deriving

transitions only from actions on actor lifelines, greatly

simplifies the problem.

4.3.3 Consistent Definition of Inputs/Outputs

A message reception on an actor lifeline represents the actorconsuming an invocation event placed in its input queue bythe message emitter. A message emission on an actorlifeline represents the actor placing an invocation event inthe input queue of the message receiver.

From a message reception on an actor, we generate anLTS transition labeled

actorname?actor opnameðval1; . . . ; valnÞ;

where actor opname is the actor operation invoked.From a message emission on an actor, we generate a

transition labeled

actorname!:�:SUT opnameðparam1; . . . ; paramnÞ;

where SUT opname is the SUT operation invoked. The useof the regular expression ensures that this label also coversthe case in which the actor performs some internalprocessing in the same transition.

4.3.4 No SUT Internal Structure in UML Test Objectives

We have restricted the semantics of test objective sequencediagrams (the generated LTS) to visible actions. However,this does not mean that the syntax of these diagramscontains no SUT-internal actions. In fact, some SUT-internalactions will almost inevitably appear in the diagrams,namely, reception of messages from external actors andemission of messages to external actors.

One could wish to use other SUT-internal actions simplyas a convenient way of specifying different types ofordering relations on actions occurring on different actorlifelines. However, we limit the possible syntactic SUT-internal actions to the above two types by imposing therestriction that test objective sequence diagrams mustcontain a single SUT lifeline.

The reason for this restriction is the fact that theinformation as to which SUT instance invoked an actormethod is not available on the LTS transition labelcorresponding to the reception of that invocation, since, atthe application level, this information is not available. If wewish to be able to match labels of the two LTSs, it cannot,therefore, appear in the test objective LTS either.

4.4 Deriving an LTS from a Set of Scenarios

From each scenario of the test objective, we generate an LTSwhich defines its semantics in a way that is consistent withthe LTS generated from the UML model, as explainedabove. Regarding the ordering relations between the eventsof a sequence diagram, we currently use the rather simplesemantics of [25], though it may be possible to loosen thissemantics in the future.

The LTS generated from a scenario involving wildcardson component names (appearing in lifeline headers) isobtained by generating an LTS from each possible value ofthe wildcard and taking their union. In the case where adiagram uses wildcards on several component names, thecorresponding LTS is obtained by generating an LTS foreach of the possible combination of values and taking theirunion. See the ATC example for an illustration of the use ofwildcards. Note that the example uses the following


derivation: for the common reject case of a scenarioinvolving a single message exchanged between two wild-card-labeled instances, instead of the union of two transi-tions labeled:

:�!:� n:operationðargsÞ and :�?operationðargsÞ

(SUT and tester roles interchanged) we derive a singletransition labeled:

:�operationðargsÞ:Having derived an LTS from each accept or reject

scenario, we first complete each of them with regard to �,the alphabet of the application model, with loops in eachstate. That is, each state contains an implicit loop transitionwhose label is “any action in the set �� {other outgoingactions of that state}.” We then derive the LTS correspond-ing to the test objective as follows:

Suppose we have:

accept scenarios fseqiþgi2I defining LTS fSiþgi2I;

reject scenarios fseqj�gj2J defining LTS fSj�gj2J;

where these LTS are completed with regards to � by loopsin each state.

The LTS of the test objective is then defined as

\i2I Siþ \ : \j2J Sj�;

where negation denotes complement. The accepting statesof this LTS are the so-called accept states and it is completedwith regard to � by transitions to the so-called reject states.

5 TEST SYNTHESIS OF IOLTS TEST CASE

In this section, we first state the inputs and outputs of thetest synthesis and then present the result of applying thesynthesis tool to the ATC example, using the ATC testobjective and ATC model presented in the previoussections. We then briefly summarize the formal basis ofthe synthesis tool. More details can be found in [23].

5.1 Inputs and Outputs of the Test Synthesis Tool

The test synthesis uses the TGV tool [23] but could also useother tools with similar properties. The inputs to this toolare

. A simulation API, which can be used to lazilyconstruct the LTS representing the operationalsemantics of the specification of the entire system(including actors).

. An LTS representing the test objective, whichpartially describes sequences of the specification.The LTS is completed with regard to �, the alphabetof the specification, by (implicit) loops in each stateand the labels on its transitions may contain regularexpressions.

. The specification of which actions are internal (viathe -hide option, conventionally provided in a filewith suffix .hide), thus defining the SUT interface(note: hiding may produce nondeterminism).

. The specification of which actions on the IUTinterface are inputs and which are outputs (via the-io option, conventionally provided in a file withsuffix .io), thus defining IOLTSs from the LTSs.

The output is an IOLTS representing a test case, that is, apredefined set of interactions between the tester—assumingthe role of the system environment—and the implementa-tion. It is decorated with the test verdicts

. pass, on termination of executions which fulfill thetest objective (i.e., those that lead to an accept state ofthe test objective LTS), and

. inconclusive, on termination of executions which donot fulfill the test objective but on which thebehavior is consistent with the specification (wherethis verdict is derived as soon as possible).

The fail verdict is implicit on reception of any input notexplicitly specified. Note that it is not related to the rejectstates.

5.2 ATC Test Case Generation

Feeding Umlaut/TGV with the test objective LTS of Fig. 7together with the LTS, the .hide and the .io files derivedfrom the ATC UML model, we obtain the IOLTS illustratedin Fig. 8.

Such a test case could be executed by a tester on animplementation of the ATC model. Recall that the transitions


Fig. 8. The IOLTS test case synthesized from the ATC model according

to the test objective IOLTS of Fig. 7.

leading to a fail verdict on unexpected input of the testerare left implicit. The concurrency in the system leads tothe diamonds between states 7 and 10 and betweenstates 15 and 18.

5.3 Formal Basis

The test case derivation is based on the following formalnotion of conformance (known as ioco; see [35]): animplementation conforms to a specification if it cannotproduce outputs which are unexpected with regard to thespecification, after executing a trace of visible actions whichis allowed by the specification. In the theory, the absence ofvisible activity (quiescence), resulting from deadlocks,livelocks, or waiting for input, must be observable and istherefore considered to be a particular type of output. Inpractice, such “outputs” are detected by timers. The theoryalso assumes that an implementation (conformant or not)can never refuse an input and that, in the general case, theset of outputs of an implementation is a superset of the setof outputs of the specification.

The test case is derived by exploring that part of the statespace of the specification which is selected by the testobjective (the reject part of the test objective plays a crucialrole in this selection). This involves calculating the synchro-nous product of the two IOLTSs, hiding the internal actions,calculating an equivalent deterministic automaton (aftertaking into account quiescence), and extracting a test case asthe mirror image of a particular type of controllable subgraphof this determinized product. The mirror operation inter-changes inputs and outputs in order to move from aspecification viewpoint to a tester viewpoint. A synthesizedIOLTS is said to be controllable if none of its states has acontrollability conflict; a state has a controllability conflictif it has more than one outgoing transition and one ofthese transitions is an output. See [23] for details.

All but the controllability part of the test synthesisalgorithm is performed on-the-fly, that is, lazily withrespect to the construction of the state graph. This enablesthe test synthesis algorithms to handle specifications ofarbitrary size. However, on-the-fly determinization alsomeans that the generated test cases are not necessarilyminimal.

The test synthesis algorithms ensure essential propertieson the synthesized test cases with respect to ioco. Inparticular, test cases are sound, that is, they reject onlynonconformant implementations. The converse property,exhaustiveness, is unreachable in practice due to loops andto the observable nondeterminism that may be exhibited bythe IUT and which the tester cannot control. Nevertheless,the algorithms guarantee that the test synthesis methoditself is exhaustive, in the sense that for any nonconformantimplementation, it is possible to synthesize a test case thatmay reject it (only “may reject” due to the fact that the testercannot fully control the IUT).

6 FROM IOLTS TO UML TEST CASE

It remains to derive a scenario, in the UML sequence-diagram-based language used, called TeLa [33], from anIOLTS representing a test case produced by the synthesisengine. TeLa is not aimed exclusively at the representationof IOLTS produced by the test synthesis described here; it isalso intended to be used to directly describe test cases.

The semantics of TeLa diagrams can be given in terms ofpartial orders of events. However, such a general semanticsis not needed to represent the IOLTS test cases produced bytest synthesis. For this reason, TeLa also allows the use of amore restrictive semantics in terms of partial orders ofmessages, similar to the synchronous interworkings of [25];see [32] for details.

The TeLa diagram to be derived from an IOLTS is adiagram that has this IOLTS as (restricted) semantics. Aswith MSC and HMSC, different diagrams can have thesame semantics so that the same IOLTS can be often berepresented in different ways. To illustrate the derivation,in this section, we present some of the possible TeLarepresentations of the ATC test case IOLTS presented inFig. 8.

Note that, as well as the method being invoked, thelabels on the arrows of TeLa diagrams can contain the nameof the object which actually executes the method (usefulwhen the lifeline header contains the name of a componentcontaining that object); when it appears, the object nameprecedes the method name separated from it by a dot.

6.1 UML Test Case Using a Choice Operator

A test case produced by synthesis can always be correctlyrepresented using only two lifelines, one for the SUT andone for the tester. Fig. 9 shows a TeLa scenario represent-ing the IOLTS test case of Fig. 8, synthesized from the LTSof the ATC UML model according to the test objective LTSof Fig. 7, which is in turn represented by the scenarios ofFig. 5. The patently-improvable graphical syntax used forthe choice operator was chosen to be easy to implement inUML tools available at the time this work was undertaken.

The representation in Fig. 9 uses the TeLa goto/labelconstruct. TeLa also offers the alternative possibility ofconnecting sequence diagrams together using an activitydiagram in the way that an HMSC connects MSCs. Suchtwo-tier representations, anticipated the interaction over-view diagrams of UML 2.0. For example, a two-tierrepresentation of the test case of Fig. 8 would compriseseven sequence diagrams like those of Fig. 9, but withoutthe sequence-diagram choice operator, and a single activitydiagram with seven main nodes linking them together.

6.2 UML Test Case Using a Concurrency Operator

The test synthesis presented in this paper is based onIOLTSs, in which concurrency is modeled as interleaving,that is to say, choice and concurrency are not distinguished.One could hope to reconstruct concurrency from diamondstructures in the IOLTS output of the synthesis, for example,that involving states 7, 8, 9, and 10 and that involvingstates 15, 16, 17, and 18 in Fig. 8. However, apart from thedifficulty in recognizing diamond structures involving anarbitrary number of transition actions, such structurescannot be interpreted as representing concurrency withoutmaking assumptions about the application model.

Fig. 10 shows a TeLa scenario representing the same testcase as that represented in the previous figure. This time,the diamonds formed by states 7, 8, 9, and 10 and bystates 15, 16, 17, and 18 of Fig. 8 have been interpreted asrepresenting concurrency (by making certain assumptionsabout the UML model). The coregions are used to indicatethat the tester can expect to receive the correspondingmessages in any order.


6.3 UML Test Case Using Tester Internal Structure

In TeLa, the tester structure, or test architecture, can be

shown by using different lifelines to represent different

tester components. In the case where these lifelines

represent tester objects, the TeLa syntax allowing the name

of an invoked tester object to appear in an arrow label is no

longer needed.For a synthesized test case, the external actors and their

interaction with the SUT can be considered to define a

default test architecture.4 We may wish to show this

architecture explicitly in the TeLa scenario derived from

the synthesis, that is, we may wish to use a distinct

lifeline to represent each external actor. The restrictions on

actor state machines ensure that we can do so since the

Umlaut-generated labels used in the test case contain

enough information to identify the external actor involvedin each tester event. Note that, since we are using themore restricted semantics of TeLa, representing each actorwith a separate lifeline does not introduce furtherconcurrency. See Fig. 10 for an example.

6.4 No SUT Internal Structure in UML Test Cases

Though we are concerned with black-box testing, this doesnot mean that test cases described in a sequence diagramnotation such as TeLa cannot show part of the SUT internalstructure in terms of objects. This is since, in the objectparadigm, the tester needs references to the objects itcommunicates with, either having this knowledge in theinitial state or acquiring it in the course of the test.

However, aside from the difficulties of possibly introdu-cing spurious concurrency, the use of SUT structure in


Fig. 9. TeLa representation—using the label/goto construct—of the test case represented as an IOLTS in Fig. 8.

sequence diagrams representing test cases suffers from thesame problem as that mentioned when discussing SUTstructure in sequence diagrams representing test objectives.That is, with the current simulator implementation, theinformation as to which SUT instance invoked an actormethod is not available on the LTS transition labelcorresponding to the reception of that invocation. We aretherefore restricted to diagrams having a single lifelinerepresenting the SUT.

7 FUTURE WORK

In this section, we discuss some of the ways in which thedifferent parts of the synthesis method can be enhanced.

7.1 Formal Specification Derivation

Regarding the formal specification derivation, the mostobvious possible enhancement concerns the atomicity of theLTS transitions. In a more sophisticated derivation, an LTStransition would be generated for each state-machinemicrostep, or at least for each interobject communicationaction. This would necessarily involve a notion of globalstate in which objects of the system can be between controlstates of their associated state machines. A finer atomicitywould enable us to remove many of the restrictionsimposed on the UML model.

The handling of dynamic object creation constitutes asignificant advance but has thrown up some interestingproblems with the current implementation. First, if the statecomparison function distinguishes global states that differonly in the object identifiers used, a model containing a

loop involving object creation can lead to infinite branchingin the generated LTS. The solution is for the comparison tobe based on a notion of isomorphic object graphs. Second,if the scheme used for assigning identifiers to dynamicallycreated objects does not ensure that the same dynamicallycreated object always has the same identifier in differentglobal states, then, since these identifiers may appear inLTS transition labels, this can lead to spurious nondeter-minism in the generated LTS and may make it impossibleto correctly distinguish LTS transition labels involvingtruly different dynamically created objects. A work-aroundexists in certain cases at the TGV level to deal with thespurious nondeterminism without changing the simulatorimplementation.

7.2 Formal Objective Derivation

Regarding the formal objective derivation, the mostimportant possible enhancement concerns test coverage, ofwhich the method currently lacks all notion. In a black-boxcontext, it is coverage of the model that is of interest.Required here is a notion of coverage of a state-basedspecification by a set of high-level interaction scenarios.

7.3 Test Synthesis

Regarding the test synthesis, the enumerated character ofthe IOLTS basis can be palliated to some extent by usingtest-data generation techniques. However, a more promis-ing, but much more complex, alternative is to extend thesynthesis method to symbolic treatment of data. A tooldealing with symbolic data, which builds on the TGV work,is currently under development [7] but has not been used inthe Umlaut environment.

Another enhancement of great interest, but also of somecomplexity, concerns going beyond interleaving models. Inthe IOLTS model, on which the test synthesis presented inthis article is based, concurrency is modeled as interleaving,that is, choice and concurrency are not distinguished.Reconstructing concurrency from diamond structures inthe IOLTS output of the synthesis is not possible in thegeneral case. An alternative approach to distributed testingis to keep explicit the initial concurrency of the UML model,replacing the IOLTS representation by a true-concurrencymodel and performing test synthesis on this model, as in[22]. The sequence-diagram languages would in this casealso need to be used with a semantics based on partialorders of events.

8 RELATED WORK

Some of the earliest work on using scenarios to describe testpurposes and test cases (in this case using MSCs) is that of[12]. More recent developments on this line are thegraphical syntax for TTCN-3 [9] and the UML Test Profile[29] based on UML 2 [30]. The UML Testing Profile has notyet seen much use and, consequently, many issuesconcerning its use remain to be clarified, some of themrequiring clarification of the UML 2 specification itself,particularly concerning the sequence diagram part (e.g., theUML 2 sequence-diagram operators neg and assert; see also[33] for some discussion of these issues).


Fig. 10. TeLa representation—showing tester structure and using

coregions—of the test case represented as an IOLTS in Fig. 8.

A number of authors, such as [3], [4], [5], [17], and [28],have worked on deriving test data from UML statediagrams, though most of these approaches deal only witha single state diagram. In a similar way to the workpresented here, [15] builds a global state machine out ofmultiple state charts but the approach is not on-the-fly, isnot guided by test objectives, and lacks formality. Moreformal approaches are those of [11] and [34], the latterincluding a notion of test objectives (in the form of LTLformulas), though, again, they are both restricted to a singlestate diagram.

Similarly to our work, the collaboration-diagram-basedapproach of [2] is also oriented to deriving tests from use-cases. However, to our knowledge, the approach taken in theAgedis project [14] is the only other UML-based formalapproach building a global state machine (in this case,described in the IF language) from multiple communicatingstate charts and deriving tests of distributed applicationsfrom UML models on-the-fly, according to a formallydescribed conformance relation and guided by testobjectives. The Agedis tools are also centered on TGVand the project has built on the developments we presenthere with work on taking data into account to a certainextent, on deriving test objectives from test strategies andon using more flexible notions of test objective at the TGVlevel.

However, one of the main points of interest in ourapproach is that though the UML model is state-based, thetest objectives and the test cases are interaction-based. Weconsider the use of sequence-diagram based test objectivesgives our approach a significant usability advantage overother approaches such as that of [34], which uses LTL, orthat of [14], which uses state diagrams. The possibility ofusing use-case scenarios as test objectives is in keeping withtheir use in driving the testing in widely-used softwaredevelopment processes. Similarly, we consider the use ofsequence-diagram based test cases to be another advantageof our approach, such use being in keeping with recentwork reported on in [29] and [9]. Our work could berelatively-easily adapted to produce test cases in one ofthese latter notations.

9 CONCLUSIONS

In this paper, we have presented a method and a tool for theautomated production of test cases (expressed as scenarios)from state-based design models of distributed softwareguided by test objectives (also expressed as scenarios).

From the tooling point of view, we used the Umlautframework to automatically compile a UML model (with afew restrictions on the semantics of atomicity) into animplicit input/output labeled transition system (IOLTS),and then used the TGV tool to drive the resulting IOLTS togenerate test cases based on the TGV on-the-fly model-checking technology. This approach makes it possible todeal with arbitrarily large systems, without any a priorilimitation on the dynamics of the application (e.g., creationof objects). This has been validated on real-world casestudies (e.g., see [6] for an experience report), with veryreasonable processing time (on the order of a second foreach test case), despite our rather naive approach in

handling global state manipulations. On this note, ourideas have been reimplemented in the context of the Agedisproject [14] with a smarter handling of global statemanipulations.

From the methodological point of view, the specificcontribution of this work consists in driving the testgeneration process with behavioral test patterns, i.e.,reusable (and incomplete) testing scenarios. The resultspresented here constitute the first complete proposal tosynthesize conformance test cases from high-level scenariosthat is fully integrated in the UML framework but which, atthe same time, has a fully formal basis obtained by givingan operational semantics to UML models in the form oflabeled transition systems.

APPENDIX

DEFINITION OF BASIC CONCEPTS

In this section, we give more precise definitions of the termsused in the rest of the paper.

Black-box testing. Testing in which interaction with theIUT is carried out exclusively through the interfaces definedin its specification and without reference to lower-leveldescriptions.

Conformance testing. Testing that an implementationconforms to a specification; conformance testing thereforesupposes a priori that the specification is correct. Inconformance testing, then, the specification serves toprovide the test oracle. Conformance testing is generallyblack-box testing.

Test verdict. One of the following possible outcomes fora test case: pass, fail, inconclusive, and error [19]. The firstthree verdicts concern the observed behavior of the IUT andthe last verdict concerns correct execution of the testsoftware. In the enumerated data case dealt with here, theinconclusive verdict is only assigned when the behavior ofthe implementation under test, while correct according toits specification, does not permit the test objective to beverified.

Tester. All the software in the test setup that is not partof the IUT. From the IUT point of view, during testexecution, the tester plays the role of the IUT environment.The purpose of this software is to stimulate the IUT, observeits responses and deliver a corresponding test verdict.

Test result. The test result is the behavior actuallyperformed in a test execution—that is, the set of actionsexecuted by the tester (including parameter values) and theordering, and possibly timing, relations between them—together with the verdict for that test execution.

Note that we do not simply define the test result as theoutput of the IUT. In the distributed case, such a definitionis not sufficient, even if we view the IUT input and IUToutput as sequences of actions. The inherent concurrency ofa distributed system means that the IUT may exhibitobservable nondeterminism, that is, in different executions,it will not necessarily produce the same sequence of outputsfor the same sequence of inputs.

Labeled transition system (LTS). A quadruple M ¼ðQM; qM

0 ;�M;!MÞ where


. QM is a finite nonempty set of states,

. qM0 is the initial state,

. �M is the alphabet of actions,

. !M� QM � �M �QM is a transition relation.

Input-output labeled transition system (IOLTS). Atriple (M, �M

hide, �Mio ), where

. M ¼ ðQM; qM0 ;�

M;!MÞ is a labeled transition system,. �M

hide : �M ! f0; 1g is a black-box view on �M, i.e., acharacteristic function that partitions the set ofactions of M into visible actions �M

v ¼ ð�MhideÞ

�1ð1Þand internal actions �M

� ¼ ð�MhideÞ

�1ð0Þ.. �M

io : �Mv ! f0; 1g is an input-output view on �M

v , i.e.,a characteristic function that partitions the set ofvisible actions of M into input actions �M

I ¼ ð�Mio Þ�1ð0Þ

and output actions �MO ¼ ð�M

io Þ�1ð1Þ.

Note that, in this definition, as used by the TGV tool, it isnot assumed that all internal actions are identical—as is thecase for the usual labeled transition systems with internalaction—in order to allow test objectives to contain IUT-internal actions, though this facility cannot be used in themethod presented here for the reasons explained earlier.

IUT interface. The IUT interface is the set of points ofinteraction of the IUT with its environment. In the test case,where the tester plays the role of this environment, these arethe points of interaction with the tester, for which reason,the IUT interface is also termed the test interface. The IUTinterface therefore defines the boundary of the black boxused in black-box testing.

In a component-model allowing multiple componentinterfaces, the IUT interface is a set of such interfaces. Ina component model allowing instances of componentinterfaces, or ports, the IUT interface is a set of suchcomponent ports.

In a service-oriented context, the IUT interface com-prises two parts: the part describing the services offered bythe IUT and the part describing the services required bythe IUT.

In the LTS context, the IUT interface is defined implicitlyby identifying the set of actions occurring at that interface,i.e., the visible actions, �M

v ¼ ð�MhideÞ

�1ð1Þ, of the IOLTSdefinition above.

(Conformance) test case. A test case (where, here, weunderstand a platform-independent test case, or abstracttest case in the terminology of [19]) is a specification of aninteraction between the tester and the IUT in which thetester stimulates the IUT via the test interface, observes itsresponses at this interface and assigns a verdict to the resultof this interaction. The verdict is assigned in function ofwhether the test result is consistent with the IUT specifica-tion as defined by some conformance relation. A test case isdesigned to exercise a particular execution or verifycompliance with a specific requirement.

Test objective. A generic behavior specification repre-senting an abstract view of some specific behavior of theIUT that we wish to test. Note that this definition is moreflexible than one in which a test objective is simply definedto be an abstract view of a test or of a set of tests.

Test synthesis according to a test objective. The processof using a test objective as a criterion for selecting the

behavior to be tested from among the behaviors permitted

by the specification, in order to derive one or several test

cases. The role of the test objective in test synthesis is

analogous to that of a property in model-checking.

ACKNOWLEDGMENTS

This work has been partially supported by the COTE RNTL

National project [24] and the CAFE European project

Eureka �! 2023 Programme, ITEA project IP 0004.

REFERENCES

[1] C. Atkinson et al., Component-Based Product Line Engineering withUML. Addison-Wesley, 2002.

[2] M. Badri, L. Badri, and M. Naha, “A Use Case Driven TestingProcess: Towards a Formal Approach Based on UML Collabora-tion Diagrams,” Proc. Third Int’l Workshop Formal Approaches toSoftware Testing (FATES ’03), A. Petrenko and A. Ulrich, eds., Oct.2003.

[3] K. Bogadov and M. Holcombe, “Testing from Statecharts Usingthe Wp Method,” Proc. Second Workshop Formal Approaches toTesting of Software (FATES ’02), R. Hierons and T. Jeron, eds., Aug.2002, http://www.brunel.ac.uk/~csstrmh/concur2002/Fates02_proceedings.pdf.

[4] L.C. Briand, Y. Labiche, and J. Cui, “Automated Support forDeriving Test Requirements from UML Statecharts,” J. Softwareand Systems Modeling, vol. 4, no. 4, Nov. 2005.

[5] S. Burton, J. Clark, and J. McDermid, “Automatic Generation ofTests from Statecharts Specifications,” Proc. First Workshop FormalApproaches to Testing of Software (FATES ’01), E. Brinksma andJ. Tretmans, eds., Aug. 2001, http://www.brics.dk/NS/01/4/BRICS-NS-01-4.pdf.

[6] L.D. Bousquet, H. Martin, and J.-M. Jezequel, “ConformanceTesting from UML Specifications, Experience Report,” Proc. UML2001 Workshop: Practical UML-Based Rigorous Development Methods,A. Evans, R. France, A. Moreira and B. Rumpe, eds., 2001.

[7] D. Clarke, T. Jeron, V. Rusu, and E. Zinovieva, “STG: A SymbolicTest Generation Tool,” Proc. Eighth Int’l Conf. Tools and Algorithmsfor the Construction and Analysis of Systems (TACAS ’02),J.-P. Katoen and P. Stevens, eds., Apr. 2002.

[8] W. Damm, B. Josko, A. Pnueli, and A. Votintseva, “A Discrete-Time UML Semantics for Concurrency and Communication inSafety-Critical Applications,” Science of Computer Programming,vol. 55, nos. 1-3, 2005.

[9] The Testing and Test Control Notation, Version 3. Part 3: TTCN-3Graphical Presentation Format (GFT), ETSI ES 201 873, parts 1 to 7V3.0.0 (2005-03), European Telecomm. Standards Inst., 2005.

[10] E. Gamma, R. Helm, R. Johnson, and J. Vlissides, Design Patterns:Elements of Reusable Object-Oriented Software. Addison-Wesley,1995.

[11] S. Gnesi, D. Latella, and M. Massink, “Formal Test-CaseGeneration for UML Statecharts,” Proc. Ninth IEEE Int’l Conf.Eng. of Complex Computer Systems, P. Bellini, S. Bohner, andB. Steffen, eds., Apr. 2004.

[12] J. Grabowski, D. Hogrefe, and R. Nahm, “Test Case Generationwith Test Purpose Specification by MSCs,” SDL ’93: Using Objects,Proc. Sixth SDL Forum, O. Faergemand and A. Sarma, eds., 1993.

[13] D. Harel and H. Kugler, “The RHAPSODY Semantics ofStatecharts (Or on the Executable Core of the UML),” Integrationof Software Specification Techniques for Application in Engineering,Springer-Verlag, 2004.

[14] A. Hartman and K. Nagin, “The AGEDIS Tools for Model BasedTesting,” ACM SIGSOFT Software Eng. Notes, vol. 29, no. 4, July2004.

[15] J. Hartmann, C. Imoberdorf, and M. Meisinger, “UML-BasedIntegration Testing,” Proc. ACM SIGSOFT Int’l Symp. SoftwareTesting and Analysis (ISSTA ’00), Aug. 2000.

[16] W. Ho et al., “UMLAUT: An Extendible UML TransformationFramework,” Proc. 14th IEEE Int’l Conf. Automated Software Eng.(ASE ’99), Oct. 1999.

[17] H.S. Hong, Y.G. Kim, S.D. Cha, D.H. Bae, and H. Ural, “A TestSequence Selection Method for Statecharts,” Software Testing,Verification and Reliability, vol. 10, no. 4, 2000.


[18] LOTOS—A Formal Description Technique Based on the TemporalOrdering of Observational Behaviour, ISO 8807, Int’l Organization forStandardization, 1989.

[19] Conformance Testing Methodology and Framework—Part 1: GeneralConcepts, ISO/IEC 9646-1, Int’l Organization for Standardization/Int’l ElectroTechnial Commission, 1994.

[20] Specification and Description Language (SDL), ITU-T Z.100, 2000.[21] ITU-T, Message Sequence Chart (MSC), ITU-T Z.120, Int’l Tele-

comm. Union, 2000.[22] C. Jard, “Synthesis of Distributed Testers from True-Concurrency

Models of Reactive Systems,” Int’l J. Information and SoftwareTechnology, vol. 45, 2003.

[23] C. Jard and T. Jeron, “TGV: Theory, Principles and Algorithms, aTool for the Automatic Synthesis of Conformance Test Cases forNon-Deterministic Reactive Systems,” Software Tools for TechnologyTransfer (STTT), vol. 7, no. 4, Aug. 2005.

[24] C. Jard and S. Pickin, “COTE—Component Testing Using theUnified Modeling Language,” ERCIM News, no. 48, Jan. 2001.

[25] S. Mauw, M. van Wijk, and T. Winter, “A Formal Semantics ofSynchronous Interworkings,” SDL ’93: Using Objects, Proc. SixthSDL Forum, O. Faergemand and A. Sarma, eds., 1993.

[26] S. Mellor and M. Balcer, Executable UML: A Foundation for ModelDriven Architecture. Addison-Wesley, 2002.

[27] C. Nebut, S. Pickin, Y. Le Traon, and J.M. Jezequel, “AutomatedRequirements-Based Generation of Test Cases for ProductFamilies,” Proc. 18th IEEE Int’l Conf. Automated Software Eng.(ASE ’03), Nov. 2003.

[28] A.J. Offutt, S. Liu, A. Abdurazik, and P. Ammann, “GeneratingTest Data from State-Based Specifications,” Software Testing,Verification and Reliability, vol. 13, no. 1, Mar. 2003.

[29] Unified Modeling Language: Testing Profile, Object ManagementGroup, Apr. 2004.

[30] Unified Modeling Language: Superstructure, Object ManagementGroup, Oct. 2004.

[31] S. Pickin, C. Jard, Y. Le Traon, T. Jeron, J.M. Jezequel, and A.Le Guennec, “System Test Synthesis from UML Models ofDistributed Software,” Proc. Conf. Formal Techniques for Networkedand Distributed Systems (FORTE ’02), D. Peled, and M. Vardi, eds.,Nov. 2002.

[32] S. Pickin, “Test des Composants Logiciels Pour les Telecommu-nications,” PhD thesis, Universite de Rennes, 2003, (title inFrench, main content in English).

[33] S. Pickin and J.M. Jezequel, “Using UML Sequence Diagrams asthe Basis for a Formal Test Description Language,” Proc. Conf.Integrated Formal Methods (IFM ’04), E. Boiten, J. Derrick, andG. Smith, eds., Apr. 2004,

[34] L. Shuhao, J. Wang, and Q. Zhi-Chang, “Property-Oriented TestGeneration from UML Statecharts,” Proc. 19th IEEE Conf. Auto-mated Software Eng. (ASE ’04), Sept. 2004,

[35] J. Tretmans, “Test Generation with Inputs, Outputs and RepetitiveQuiescence,” Software—Concepts and Tools, vol. 17, no. 3, pp. 103-120, 1996.

[36] A. Wasowski, “Flattening Statecharts without Explosions,” Proc.Languages, Compilers, and Tools for Embedded Systems (LCTES ’04),June 2004.

Simon Pickin received the BSc degree inmathematics from Sussex University, UnitedKingdom, in 1982, the Certificate of AdvancedStudy in Mathematics (Part III) from CambridgeUniversity, in 1986, the MSc degree in mathe-matics from King’s College, London, in 1987, theMSc degree in Computing from Imperial Col-lege, London, in 1988, and the PhD degree incomputing from the Universite de Rennes I,France, in 2003. He has worked as a researcher

in France (France Telecom R&D and IRISA) and Spain (DIT/UPM andIT/UCIIIM) and as a university profesor in Spain (EUI/UPM and IT/UCIIIM). His research interests lie in formal methods and formalverification, model-based testing, and distributed systems.

Claude Jard received the engineer in telecom-munications degree in 1981 and the PhD degreein computer science from Rennes University1984. He was the head of the protocol validationgroup at the French National Research Centre inTelecommunications (CNET) from 1981 to 1986.He was a research supervisor and then directorat the French National Centre for ScientificResearch (CNRS) since 1987. Currently, he isa professor of computer science at the Ecole

Normale Superieure de Cachan. His research work relates to the formalanalysis of asynchronous parallel systems and the use of formalmethods in the specification, verification, and test of distributed softwareon networks of processors, the central topic being the study of dynamicmethods of verification (verification carried out during the execution ofthe—abstracted, simulated, or real—program). He is the author orcoauthor of more than 100 publications in the fields of theoreticalcomputer science, protocol engineering, and distributed systems.

Thierry Jeron received the PhD degree incomputer science from the University of Rennes1 (IRISA laboratory) in 1991. He then spent oneyear as a research engineer in the Alcatelresearch laboratory in Marcoussis near Paris.He returned to IRISA in 1993 as an INRIAresearch scientist. Since 2001, he is thescientific leader of the VerTeCS team and theresearch director since 2006. His researchtopics concern the formal verification and valida-

tion of reactive systems, including model-checking, model-basedtesting, control synthesis, and diagnosis of discrete event systems. Heis author or coauthor of around 50 international publications in thesedomains and has been involved in a number of academic and industrialcollaborations.

Jean-Marc Jezequel received the PhD degreein computer science from the University ofRennes, France, in 1989. He joined the CNRS(Centre National de la Recherche Scientifique)in 1991. Since October 2000, he has been aprofessor at the University of Rennes, leading anINRIA research team called Triskell. His inter-ests include model driven software engineeringbased on object oriented technologies for tele-communications and embedded systems. He is

a member of the IEEE and the IEEE Computer Society.

Yves Le Traon received the engineering degreeand the PhD degree in computer science fromthe Institut National Polytechnique de Grenoble,France, in 1997. From 1998 to 2004, he was anassociate professor at the University of Rennes.From 2004 to 2006, he joined France TelecomR&D as an expert on MDA and validation. He isnow assistant professor at the ENST Bretagne(Ecole Nationale des Telecommunications deBretagne) and an associate member of the

IRISA research laboratory in the Triskell team. His research interestsinclude OO testing, design for testability, model-driven validation, andsecurity.

. For more information on this or any other computing topic,please visit our Digital Library at www.computer.org/publications/dlib.


252 ieee transactions on software engineering, …referring throughout to a simplified air traffic...

Documents