Automating Testing of Service-oriented Mobile Applications with DistributedKnowledge and Reasoning

James Edmondson, Aniruddha Gokhale, Sandeep NeemaDept of EECS, Vanderbilt University

Nashville, TN 37212, USA{james.r.edmondson,a.gokhale,sandeep.neema}@vanderbilt.edu

Abstract—Automated testing of distributed, service-orientedapplications, particularly mobile applications, is a hard prob-lem due to challenges testers often must deal with, such as (1)heterogeneous platforms, (2) difficulty in introducing additionalresources or backups of resources that fail during testing, and(3) lack of fine-grained control over test sequencing. Dependingon the testing infrastructure model, the testers may also berequired to fully define all the hosts involved in testing, beforced to loosely define test execution, or may not be ableto dynamically respond to application failures or changes inhosts available for testing. To address these challenges, thispaper describes an approach that combines portable operatingsystem libraries with knowledge and reasoning, which togetherleverage the best features of centralized and decentralizedtesting infrastructures to support both heterogeneous systemsand distributed control. A domain-specific modeling languageis provided that simplifies, visualizes, and aggregates testsettings to aid developers in constructing relevant, feature-rich tests. The models of the tests are subsequently mappedonto a portable testing framework, which uses a distributedknowledge and reasoning engine to process and disseminatetesting events, successes, and failures. We validate the solutionwith automated testing scenarios involving service-orientedsmartphone-based applications.

Keywords-test sequencing; distributed control in testing;portability; knowledge dissemination; domain specific modelinglanguage;

I. INTRODUCTION

Automated testing of distributed, service-oriented applica-tions generally falls into two execution models: centralizedand decentralized. In a centralized model, a testing controlservice sends testing commands to services installed oneach host taking part in testing, generally in a push model.In a decentralized model, there is no centralized controlservice, and testing occurs in a purely distributed manner.If synchronization is required, decentralized infrastructuresmay use network lock files, or they may require separatedaemons with intimate knowledge of test sequencing orcustom messaging protocols.

To understand the two models, let us consider an examplewhere a developer would like to test a service-orientedsmartphone-based mobile application which sends a messageto an application server. Ideally, the smartphone should notsend a message to the server until the server is ready, andsuppose this should happen after 15 seconds.

In the centralized model shown in Figure 1, a controlserver sends a command to launch the application server,waits for 15 seconds, and then sends a command to thesmartphone or host connected to the smartphone, which runsa script or launches a unit test already on the smartphoneand sends a message to the application server. If no suchscript or unit test exists, a tester might emulate a messagethat looks like it came from a smartphone.

Figure 1. Centralized software testing

For simple tests like this example, the centralized modelworks, particularly when services or applications need toonly do one task and then exit. Additionally, the centralizedmodel can allow for fine-grained launch sequencing basedon time and provides a single point of configuration whenchanging tests.

In a decentralized model, portable scripts (e.g., in Perl,Python, or Ruby) are written that perform a series of stepsappropriate for the host running each script without using acentralized controller. If coordination is needed, the scriptsmay use network lock files. In the context of our example,the first script might launch an application server, wait for15 seconds, and then create a lock file called “server.ready”on a network file system. A second host runs a differentscript that keeps checking the network file system for a filecalled “server.ready”. When the file is created, the secondscript may launch another script or service to automate asmartphone sending a message.

The decentralized model is portable, elegant and intuitiveand avoids any type of centralized messaging bottleneckor single point-of-failure. It also allows for moving scriptsbetween hosts and not binding the testing infrastructure toa single host or port configuration, which may change ashosts go in and out of service. For clouds, LANs, and grids,decentralized testing infrastructures are highly appropriate,intuitive, and may be the only real available solution whenhost/port information are unknown before an experiment isswapped in (e.g., in a cloud), without the cloud API exposingthe information directly.

One of the major issues with the decentralized model isthe usage of network file systems as lock files for testing co-ordination. Network file systems, however, were not createdwith lock files in mind. Our extensive experience conductingdecentralized testing over local area networks reveals thatnetwork file systems, such as the widely-used Network FileSystem (NFS), make no guarantees that local caches wouldever be updated with the lock files. Moreover, in practice,timing delays could be thrown off by several minutes, even ifthe lock file appears at all. A custom communication modelbuilt on top of TCP or UDP could potentially solve thisissue, but we do not observe this approach being used intraditional testing solutions.

While centralized models can avoid network file systems,they are tightly coupled to hosts, present a single point-of-failure (the testing controller), and the configuration ofthe controller can become cumbersome as tests becomemore complex, especially when services need to be launchedin parallel with different timing delays. Additionally, mostcentralized models are not portable to all operating systemsand instead cater to a specific architecture (e.g., Windows).

In addition to these sequencing and portability concerns,the testers need to be able to codify and debug their testconcerns with an intuitive interface. A cumbersome, unus-able infrastructure is unlikely to benefit developers. Fromall these concerns, we form the following requirements of adistributed, automated testing infrastructure.

1) It should support heterogeneous OS platforms, net-working platforms and topologies

2) It should allow for dynamic sequencing of tests basedon testing events, successes, and failures

3) It should be host-agnostic and allow for moving testsetups easily to other nodes (e.g., in a cloud)

4) It should allow for seamless recovery from nodefailure, and facilitate the creation and integration ofbackup testing entities

5) It should be easily configurableOur solution to address these requirements brings together

the advantages of centralized and decentralized approacheswithout suffering from their individual drawbacks. In partic-ular, it automates the testing of distributed service-orientedapplications, illustrating the approach on case studies drawnfrom smartphone-based mobile applications. Our solution

is realized as an open-source tool kit called KATS (KaRLAutomated Testing Suite). KATS is based on an underlyingframework called the Knowledge and Reasoning Language(KaRL) engine [5] available from madara.googlecode.com.

The rest of this paper is organized as follows: SectionII details prior work relating to automated testing infras-tructures; Section III presents an outline of our solution ap-proach to distributed, automated testing using a DSML anddistributed knowledge and reasoning; Section IV discussesexamples of case studies that mimic real-world, distributedblack box testing that KATS is being developed for; Sec-tion V outlines some performance related experiments; andfinally Section VI presents concluding remarks.

II. RELATED WORK

A vast majority of networked testing infrastructures [21],[3], [4], [10] utilize a centralized controller. This centralizedparadigm extends to most deployment [15], [14] and testinginstrumentation technologies [14], [23], [24], [8] as well.

Though these related tool suites do not support trulydistributed test scheduling and automation, they often doinclude useful instrumentation tools [23] that allow for realtime performance analysis – which we do not include inour tool suite at the moment. Other instrumentation systemsplace themselves in between applications and the underlyingmiddleware or networking layer [22]. Our solution relieson distributed blackbox testing without any requirement forhooks into the operating system or services to be tested.

Model-driven testing systems exist for centralized or non-distributed systems [7], [20], [6], [11], [1]. Other systems usea generic programming approach [12] for functional testing.Both these types of systems demand intimate knowledgeof internal behaviors of the application or service beingtested – often requiring code insertion into the applicationor modeling/reading of source code.

Recent work using domain-specific modeling in testingwith mobile phones appears in the literature [18], but thereare many key differences between the two approaches. Wedo not require a hardware modification to directly send keyevents to the phone and instead rely on the Android DebugBridge which is part of the Android SDK. Additionally,we offer automated and barriered test cases in contrast tointeractive sessions between users and the phone in [18].

We are unaware of prior work using distributed knowledgeand reasoning in automating networked testing, nor havewe seen prior art using an anonymous publish/subscribe(pub/sub) paradigm to coordinate and sequence test exe-cution among generic networked processes or programs.Scripted distributed testing using pub/sub exists [25] but itrequires a networked file system and customized scriptingto sequence tests and suffers from all of the NFS problemsmentioned in decentralized testing in Section I.

III. AUTOMATED TESTING FRAMEWORK FORSERVICE-ORIENTED DISTRIBUTED APPLICATIONS

This section describes our solution approach to realizingautomated testing of distributed, service-oriented applica-tions focusing on mobile services and applications. Thecontributions of this solution are as follows:

1) A portable process and process group specificationthat addresses Requirement 1 by utilizing the Adap-tive Communication Environment (ACE) [19], whichhas been ported to most platforms including Linux,Windows, Mac, Android, and iOS (see Section III-A).

2) A portable knowledge and reasoning engine calledKaRL [5] that processes testing events, successes andfailures (Requirements 2, 4) and distributes knowl-edge across the testing network via an OpenSpliceDDS transport [17]. DDS is an anonymous pub-lish/subscribe protocol that is network architectureportable (Requirement 1) and helps us remain host-agnostic (Requirement 3) (see Section III-C).

3) A middleware solution called the KaRL AutomatedTesting Suite (KATS) which integrates and config-ures KaRL and standardizes the reasoning operationsinto a process lifecycle that facilitates fine-grainedsequencing (Requirement 2), and conditions for start-ing or using backup services (Requirement 4) (seeSection III-B).

4) Our solution provides a domain-specific modelinglanguage (DSML) [16] for visualizing test sequencesand parameters in an intuitive way (Requirement 5).From this DSML, we generate test configurations forthe decentralized agents to follow (see Section III-D).

The resulting architecture is shown in Figure 2.

Figure 2. Overview of Solution Architecture

A. Processes and Process Groups

Processes in the KATS DSML are organized into pro-cess groups which dictate how individual processes areconfigured and launched. Each process has over a dozenconfiguration settings to affect changes in OS scheduling,file I/O, test sequencing, and logging, and we accomplishOS portability by using the Adaptive Communication En-vironment (ACE) middleware, which supports a wide rangeof operating systems–including mobile phone platforms likeiOS and Android. Among the more useful settings for

services supported in the current version of the KATS DSMLare the following:

• Real-time scheduling at highest OS priority, whichensures minimum jitter, fewer context switches, andhigher priority for tested applications.

• Executable name, working directory, and command-linearguments for fine-grained control of applications thatare launched and what configurations and parametersthey will be started with.

• Redirecting stdin, stderr and stdout to files for emulat-ing user input and flexible file logging. Any redirectionfrom or to a file is buffered by the OS, which minimizesjitter and reduces the time spent in OS calls - whichare expensive.

• Kill times and signals (POSIX) for errant applications.Even on non-Unix operating systems (e.g., Windows),we allow for termination of applications after a speci-fied time delay if the application does not return first.This feature is configurable by the tester.

• Test sequencing information (e.g., preconditions andpostconditions).

These settings allow for flexible management of eachapplication or service launch within a process group. Processgroups have additional settings indicating whether to launchthe processes in the group in parallel or in sequence. Eachprocess group also includes the settings listed above exceptfor executable name and command-line arguments. Thisexpressiveness for process and process group settings ispart of our solution to address Requirement 1 (test settingsportable across heterogenous OS platforms).

B. Configuring Test Sequencing

Processes in the KATS infrastructure can be sequenced intwo complementary ways: temporal delays and conditions.Temporal delays occur after KATS conditions have beenchecked. These conditions come in three supported types:barriers, preconditions, and postconditions.

The sequence of these temporal delays and conditionsis shown in Figure 3. Barriers are groupings of processesor process groups that must rendezvous to the same pointbefore moving onward with process launching. Barriersrequire setting a process id and the number of processesexpected to participate in the barrier event.

After the barrier, preconditions are checked against thelocal context within that represent conditions that must betrue before the process may be launched. The applicationis then launched and its return code is saved into theKaRL variable .kats.return, which allows for postconditionsto feature logics that check whether or not the applicationsucceeded or failed based on the return code (the exit valuereturned by the process). Postconditions may also makeglobal state modifications, which may satisfy other appli-cation or service preconditions. Temporal delays, barriers,preconditions, and postconditions can be specified on either

Figure 3. KATS Process Lifecycle

a process or process group level. These lifecycle elementsare the foundation of our solution to meet Requirement2 (dynamic sequencing of tests based on test progress orfailure).

C. Augmenting Decentralized Testing with Knowledge andReasoning

The process lifecycle outlined in Section III-B providesthe infrastructure for meeting Requirement 2, but we stillneed a communication and reasoning mechanism for dis-tributing knowledge and learning from testing events, suc-cesses, and failures. In this section, we look at the distributedknowledge and reasoning engine we developed called theKnowledge and Reasoning Language (KaRL) engine [5].

KaRL allows developers to manipulate knowledge viaexpressions in a programming logic which are evaluatedagainst a local context (see Figure 4). The KaRL reasoningengine mechanisms used for evaluating logics allow foraccessing or mutating knowledge variables and provide fine-grained debugging capabilities for global knowledge state inan atomic, thread-safe manner.

Figure 4. KaRL evaluates user logics against a local context, which isthen synchronized with other contexts

The underlying reasoning engine mechanisms createknowledge events that are disseminated based on whetherthey are considered local (i.e., the variable begins witha period) or global (i.e., anything else) and whether or

not there are interested software entities in the networkthat would like information on the published knowledgedomain. Once the knowledge events in KaRL are ready tobe disseminated, they can be sent to the appropriate entitiesusing an underlying transport mechanism.

The coupling of KaRL with an anonymous pub-lish/subscribe paradigm fits our knowledge and reasoningneeds for testing in the following ways. First, the communi-cation is host-agnostic–which means we do not have to doany configuration when changing hosts or enlarging a servicecloud (Requirement 3). Second, the KaRL reasoning serviceis robust to application or node failures and allows us toseamlessly integrate backup services to take over in testingoperations, if necessary (Requirement 4). Third, we wantedan engine that could change preconditions, postconditions,and barriers on the fly, at runtime, and most other knowledgeand reasoning engines do not allow this without resetting allknowledge periodically.

The KaRL engine and its underlying transports are seam-lessly incorporated into the KATS testing framework withvery little interaction or configuration required by the tester.The only time a tester manually works with KaRL is whenthey are creating preconditions and postconditions. Thoughbarriers use KaRL, the user only has to specify a barriername, an id for this process, and the number of processesparticipating in the barrier. KATS builds the KaRL logicfrom this information provided by the user in the visualDSML of their testing procedures.

We now briefly outline the KaRL solution to encod-ing testing information into conditions for the expressedpurpose of distributed reasoning. KaRL is a knowledgerepresentation and predicate calculus engine built for real-time, continuous systems. KaRL features first class supportfor multi-modal knowledge values, and each knowledgevariable may be one of 264 possible values. KATS conditions(i.e., barriers, preconditions, and postconditions) are KaRLexpressions that take the following form Predicate =>Result, which can be read as “if the Predicate is true thenevaluate Result.” By default, Predicate is true in the KaRLlanguage, and KaRL expressions may be chained togetherwith the Both operator ;. This operator ; simply means thatboth the left and right expression should be evaluated.

For an example of a KaRL expression that can be used ina KATS condition, consider the following possible postcon-dition: server.finished => (no.backup => all.tests.stop= 1) . This KaRL expression will check the variableserver.finished and if it is not zero (i.e., true), then wewill check if there is a backup server available. If no.backuphas been set to anything but zero, we then indicate that alltests need to stop by setting all.tests.stop to 1.

KaRL supports most multi-modal logic operations andconditionals including ==, !=, &&, and || and mutationson knowledge, including +, −, ∗, /, %, where the latterthree are multiplication, division, and remainder or modulus.

For a more comprehensive listing, please see the projectsite. See Section IV for specific, simple examples of howKaRL expressions offer powerful sequencing capabilities toa tester.

D. Domain-Specific Modeling

KATS provides a modeling front-end (DSML) to over-come mundane and erroneous tasks handled by the testers(Requirement 5). The KATS DSML comprises a UML-basedmeta-model developed using the open-source and freelyavailable Generic Modeling Environment (GME) [13] tocodify the process lifecycle and all reasoning operationsin a highly configurable way. This DSML has four keycomponents that users can create and configure: processes,process groups, hosts, and barriers. The DSML in turngenerates an XML file that is interpreted by KATS tools.If testers do not wish to use the visual DSML, they canencode simple XML files according to the KATS schema.We show example testing XML files in Section IV.1

IV. DEMONSTRATING KATS ON CASE STUDIES

In this section we present experimental scenarios thathighlight typical distributed testing configurations. Theseconfigurations feature service-oriented applications usingsmartphones connected to hosts because these scenariosrepresent the types of experiments we are currently workingon for our sponsored R&D projects. To concretize our casestudies, we assume that the smartphones used in each ofthese examples are Android phones that allow for instrumen-tation of application launches through the Android DebugBridge [2].

A. Sequenced Smartphone Scenario

In our first test case, five smartphones are connected toa server and testers are interested in determining if theserver will receive the five smartphone requests in the correctorder with no arbitrary temporal delays between them.Specifically, the testers want to start the gateway service,wait for 15 seconds, and then send a message from Phone1,then Phone2, Phone3, etc. We assume that all smartphonesare connected by USB to the same host and the server resideson its own host.

In KATS, a model is created that creates a barrier namedSimplePhoneSequence, which has six processes andIDs from 0..5 inclusive and arbitrarily assigned. The fivephone processes can be setup in one of two ways andboth require an initial step of launching the server withno preconditions or postconditions. Additionally, we cansequence the phones in two different ways: (1) no pre- orpost-conditions and (2) with pre- and post-conditions.

The first option requires that KATS process group launchall the phone instrumentation scripts in parallel (a single

1We do not provide details of the modeling environment due to spacelimitations, and since it is optional in KATS.

element added to the process group XML configurationfile). This configuration is by far the easiest to setup, andrequires only setting command line arguments for the phonescripts (for the user-defined application logic) and possiblyredirecting stderr and stdout to files for logging.

Figure 5. Generated XML from DSML for the Sequenced SmartphoneScenario

The second configuration possibility requires that Phones1..5 have preconditions and/or postconditions set accordingto settings shown in Table I. Phone2 depends on Phone1completing, Phone3 depends on Phone2, and so on. Withthis intuitive mapping of preconditions and postconditions,we have accomplished distributed testing execution with nooperating system sleep calls and no centralized controller.The XML generated from the visual KATS DSML is shownin Figure 5.

Table IPRECONDITIONS, POSTCONDITIONS, AND TEMPORAL DELAY IN THE

SEQUENCED SMARTPHONE SCENARIO

Name Pre Post Delay

Phone1 Phone1.done=1 15Phone2 Phone1.done Phone2.done=1Phone3 Phone2.done Phone3.done=1Phone4 Phone3.done Phone4.done=1Phone5 Phone4.done

B. Backup Service for Smartphones Scenario

The testers have acquired a backup server host, and theywant to create tests that have phones switch over to thebackup server if the main server starts dropping connections.To complicate the scenario, testers bring in three moremachines and attach the five phones to random machines.The three phones are blasting clients that send 100 messagesa second to the gateway service. If any of these scripts failto send their message, they exit with a return code of 1.Otherwise, they return a 0 for success.

The two other phones start only if the blasting clients failwith a return code of 1. No phone should try to contact aserver until 15 seconds after the gateway service is ready,and all phones should be killed after 3 minutes regardless ofwhether they succeed in contacting the main server or not.We assume the backup gateway service is launched alongwith or sometime after the main gateway service.

With KATS, the testers can produce this more complicatedsequencing scenario with minimal configuration of precondi-tions and postconditions shown in Table II.2 “.kats.return” isthe return value of the launched process (in this case one ofthe blasting clients). The postconditions here indicate that “ifmy return value is non-zero, set the global variable backupto 1.” The other two phones have preconditions that say “ifthe global variable backup is non-zero, launch my process.”In this way, we facilitate the testing scenario created by thetesters, and we can move these process groups wherever weneed to in the network or cloud.

Table IISETTINGS FOR PHONES IN THE BACKUP SCENARIO

Name Pre Post Kill

Phone1 .kats.return => backup=1 180Phone2 .kats.return => backup=1 180Phone3 .kats.return => backup=1 180Phone4 backup 180Phone5 backup 180

To implement this same solution with a centralized modelwould require a custom controller with push and pull capa-bility and may require multiple versions of testing servicescustomized to a particular operating system. In KATS, weget this cross-platform functionality included.

V. EVALUATING KATS

In this section we briefly describe performance infor-mation regarding our testing infrastructure KATS. Thoughvery few testing infrastructures are concerned with latencyand performance, decentralized service-oriented softwarecan often have a high overhead especially in group com-munications like barriers. Because our solution is novel in

2Generated XML is not shown due to space limitations.

how it disseminates data to a testing infrastructure, it isimportant to investigate whether KATS features come at ahigh performance cost.

We divide these performance metrics into two sections:condition latency (the time it takes for a precondition orpostcondition to reach other interested hosts on a local areanetwork) and barrier latency (the time it takes for a group ofdistributed applications to barrier together across a network).

A. Experimental Testbed Setup

Unless specified otherwise, all experiments were con-ducted on five IBM blades with dual core Intel Xeonprocessors at 2.8 GHZ each and 1 GB of RAM runningFedora Core 10 Linux. Each blade is connected acrossgigabit ethernet and networked together with a switch. Thecode was compiled with g++ with level 3 optimization, andeach test featured a real-time class to elevate OS schedulingpriority to minimize jitter during the test runs. This testbedis similar to the one we are using for sponsored research.

B. Measuring Condition Latency

Condition latency is the time it takes for a precondition orpostcondition of one application to reach another applicationacross the network. Recall that in testing, a precondition is acondition that must be true before an application can launch.Consequently, this latency between the setting of a variablein one part of the network and the evaluation of this data ininterested testing entities is an important metric to have.

Setup. Two applications (App 1 and App 2) werelaunched on two separate hosts. Each application was con-figured with conditions indicated in Table III, and thenthe applications were launched repeatedly. These conditionscreate a roundtrip latency time on each host. We take thisroundtrip latency time and divide by two to get our averagedlatency numbers.

Table IIICONDITIONS USED FOR LATENCY TESTS

App 1 App 2

P0 == P1 => ++P0 P0 != P1 => P1 = P0

Results. The results of our condition latency tests areshown in Table IV. These results represent average latencyfor a condition to be sent and processed from one host tothe next. Each column represents the number of round tripscompleted. The Ping row is the latency reported betweenthe nodes via the Linux ping utility. Dissem is the averagedissemination latency of KATS conditions.

Analysis. The KaRL interpreter and OpenSplice DDStransport provide very low latency for the delivery of KATSconditions. On our network, this allows for sub-millisecondaccuracy when a tested application’s dependencies are met.This will vary depending on network latency in the developer

Table IVCONDITION RESULTS

5k 25k 50k 100k 500k

Ping 114 us 114 us 114 us 114 us 114 usDissem 650 us 440 us 437 us 315 us 317 us

testbed. The lower the latency, the more fine-grained the testsequencing can be, and the more expressive testers can bewith their distributed testing scenarios.

C. Measuring Barrier Latency

Barrier latency is the time it takes for a group of KATSprocesses to execute a barrier operation before launchingthe user-defined applications. Barrier operations are far moreexpensive than setting a condition, but barriers are usefulwhen testers need all testing participants to be up and ready.From our experience, all distributed testing scenarios requirea barrier, unless the individual tests do not interact withother testing participants. Consequently, the barrier time isan important metric because it is the overhead that must beperformed before each distributed test can start.

Setup. Five hosts with two CPUs apiece launch applica-tions (simple sleep statements of five seconds) after reachingbarriers under different networked process stress loads. Highresolution timers are used to measure the time it takes tocomplete a barrier. The tests are repeated 10 times and anaverage time is reported.

Each host in these tests launched its own barrier set of 2to 5 applications. We only have two CPUs per host, so as thenumber of applications increases, no further speed increaseis possible (it is pure overhead from context switching past2 application launches). We will discuss why we did notconnect the hosts in a LAN-wide barrier in the Analysissection. Each barrier grouping was executed 10 times perhost and averaged. We complement the average latencynumbers with minimum and maximum observed barriertime.

Results. The results of our barrier latency tests are shownin Table V. These results represent latencies experiencedon a single host launching applications >= the number ofCPUs on the host. The minimum latency was the smallesttime spent in a barrier in the 50 tests. The maximum wasthe largest, and the average was also for the 50 tests.

Table VBARRIER LATENCY RESULTS

Apps per host Min Avg Max

2 0.8 ms 2.0 ms 2.7 ms3 1.7 ms 9.4 ms 21.5 ms4 2.4 ms 17.2 ms 17.2 ms5 3.0 ms 17.3 ms 35.2 ms

Analysis. We use intrahost communication here to high-light the speed and precision that the KATS system is able toachieve with distributed processes. Within a single host, asis the case here, the cause of deviation between the min andmax is caused by context switching. Since our experimentalmachine had two cores, the deviation is very low on the twoapps test. Running this same series on a real-time kernel canreduce the deviation, but in a networked test, the benefits willbe negligible.

We have found that barrier latencies for networked pro-cesses across many hosts are hugely dependent on whenthe distributed tests are started on each host, far more thananything dictated by the latency of the underlying KaRLsystem in disseminating knowledge shown in Section V-B.

As an example, consider the case where KATS tests willbe started distributedly via cron jobs on each host at 5:00p.m., according to the system clock which is synchronizedwith the network time protocol (NTP). Even under thebest of circumstances, this protocol guarantees accuracybetween the networked clocks in the hundreds of millisec-onds – orders of magnitude larger than how long barriers,preconditions, or postconditions execute in KATS or itsKaRL infrastructure outside of situations with heavy contextswitching between the KATS processes and the OpenSpliceDDS daemon.

This may seem like a problem, but this is exactly thereason barriers are recommended among test participants.Barriers ensure that regardless of time synchronization ororder of test launches in the network, the exact sequence oftesting events will be launched precisely as specified by thetesters.

VI. CONCLUSIONS

In this paper we have described a novel, portable ap-proach to decentralized testing automation that accomplishesdistributed test sequencing and control for service-orientedapplications. Our solution provides testers with distributedbarriers for processes involved in a test sequence, precon-ditions for process entry that can be based on knowledge,and postconditions that can export knowledge to facilitatetest sequencing over the network using an anonymous pub-lish/subscribe paradigm. Testers generate XML-based testscenarios that can be used by our solution to distributedlyautomate and sequence thousands of tests in the appropriateorder.

We have validated our work with case studies involvingsmartphones, client applications, and custom applicationservices. Additionally, we show how to use the built-inKATS features to set knowledge based on the success orfailure of individual test processes based on their returnvalues.

Future work for our system includes more tools for foren-sic accounting, performance metrics reporting for testedapplications, knowledge observers for visualizing testing

progress, and more intuitive UI design that makes conditionseasier to create, understand, and edit. All source code for thisproject, including examples and case studies can be foundat our project site at http://madara.googlecode.com and isprovided under a BSD license.

REFERENCES

[1] C. Boyapati, S. Khurshid, and D. Marinov. Korat: Automatedtesting based on java predicates. In IN PROC. INTER-NATIONAL SYMPOSIUM ON SOFTWARE TESTING ANDANALYSIS (ISSTA, pages 123–133. ACM Press, 2002.

[2] G. Chang, C. Tan, G. Li, and C. Zhu. Developing mobileapplications on the android platform. In X. Jiang, M. Ma,and C. Chen, editors, Mobile Multimedia Processing, volume5960 of Lecture Notes in Computer Science, pages 264–286.Springer Berlin / Heidelberg, 2010.

[3] J. C. Cunha, J. Loureno, T. R. Anto, J. A. Lourenco, andT. R. A. Ao. An experiment in tool integration: the ddbgparallel and distributed debugger. In EUROMICRO Journalof Systems Architecture, nd Special Issue on Tools and Envi-ronments for Parallel Processing, pages 708–717. Springer,1999.

[4] C. Dumitrescu, I. Raicu, M. Ripeanu, and I. Foster. Diperf:An automated distributed performance testing framework. Inin 5th International Workshop in Grid Computing, pages 289–296. IEEE Computer Society, 2004.

[5] J. Edmondson and A. Gokhale. Design of a scalable reasoningengine for distributed, real-time and embedded systems. InProceedings of the 5th conference on Knowledge Science,Engineering and Management, KSEM 2011, Lecture Notesin Artificial Intelligence (LNAI). Springer, 2011.

[6] F. Fraikin and T. Leonhardt. Seditec ” testing based onsequence diagrams. In Proceedings of the 17th IEEE in-ternational conference on Automated software engineering,ASE ’02, pages 261–, Washington, DC, USA, 2002. IEEEComputer Society.

[7] P. Godefroid, N. Klarlund, and K. Sen. Dart: Directedautomated random testing. In PLDI, 2005.

[8] W. Gu, G. Eisenhauer, E. Kraemer, K. Schwan, J. Stasko,J. Vetter, and N. Mallavarupu. Falcon: on-line monitoringand steering of large-scale parallel programs. In Frontiersof Massively Parallel Computation, 1995. Proceedings. Fron-tiers ’95., Fifth Symposium on the, pages 422 –429, feb 1995.

[9] A. Hartman, M. Katara, and S. Olvovsky. Choosing a testmodeling language: a survey. In Proceedings of the 2ndinternational Haifa verification conference on Hardware andsoftware, verification and testing, HVC’06, pages 204–218,Berlin, Heidelberg, 2007. Springer-Verlag.

[10] M. J. Katchabow. Making distributed applications man-ageable through instrumentation. Journal of Systems andSoftware, 1999.

[11] S. Khurshid and D. Marinov. Testera: A novel framework fortesting java programs. In In IEEE International Conference onAutomated Software Engineering (ASE, pages 22–31, 2003.

[12] P. Koopman, A. Alimarine, J. Tretmans, and R. Plasmeijer.Gast: Generic automated software testing. In The 14thInternational Workshop on the Implementation of FunctionalLanguages, IFL02, Selected Papers, volume 2670 of LNCS,pages 84–100. Springer, 2002.

[13] A. Ledeczi, A. Bakay, M. Maroti, P. Volgyesi, G. Nordstrom,J. Sprinkle, and G. Karsai. Composing Domain-SpecificDesign Environments. Computer, 34(11):44–51, 2001.

[14] T. Li and T. Bollinger. T.: Distributed and parallel data miningon the grid. In Proc. 7th Workshop Parallel Systems andAlgorithms, page 2003.

[15] A. S. McGough, A. Akram, L. Guo, M. Krznaric, L. Dickens,D. Colling, J. Martyniak, R. Powell, P. Kyberd, and C. Kot-sokalis. Gridcc: real-time workflow system. In Proceedingsof the 2nd workshop on Workflows in support of large-scalescience, WORKS ’07, pages 3–12, New York, NY, USA,2007. ACM.

[16] M. Mernik, J. Heering, and A. M. Sloane. When and Howto Develop Domain-specific Languages. ACM ComputingSurveys, 37(4):316–344, 2005.

[17] Object Management Group. Data Distribution Service forReal-time Systems Specification, 1.2 edition, Jan. 2007.

[18] Y. Ridene, N. Belloir, F. Barbier, and N. Couture. A DSMLfor Mobile Phone Applications Testing. In proceedings of10th Workshop on Domain-Specific Modeling in SPLASH10th Workshop on Domain-Specific Modeling in SPLASH,page nc, France, 10 2010.

[19] D. C. Schmidt. The ADAPTIVE Communication Environ-ment: An Object-Oriented Network Programming Toolkit forDeveloping Communication Software. In Proceedings of the12th Annual Sun Users Group Conference, pages 214–225,San Jose, CA, Dec. 1993. SUG.

[20] K. Sen and G. Agha. Automated systematic testing of opendistributed programs. In FASE, 2006.

[21] J. Tufarolo, J. Nielsen, S. Symington, R. Weatherly, A. Wil-son, and T. C. Hyon. Automated distributed system testing:designing an rti verification system. In Proceedings of the31st conference on Winter simulation: Simulation—a bridgeto the future - Volume 2, WSC ’99, pages 1094–1102, NewYork, NY, USA, 1999. ACM.

[22] J. S. Vetter and B. R. de Supinski. Dynamic software testingof mpi applications with umpire. In Proceedings of the2000 ACM/IEEE conference on Supercomputing (CDROM),Supercomputing ’00, Washington, DC, USA, 2000. IEEEComputer Society.

[23] A. Waheed and D. T. Rover. A structured approach to instru-mentation system development and evaluation. In Proceed-ings of the 1995 ACM/IEEE conference on Supercomputing(CDROM), Supercomputing ’95, New York, NY, USA, 1995.ACM.

[24] A. Waheed, D. T. Rover, and J. K. Hollingsworth. Modeling,evaluation, and testing of paradyn instrumentation system. InProceedings of the 1996 ACM/IEEE conference on Super-computing (CDROM), Supercomputing ’96, Washington, DC,USA, 1996. IEEE Computer Society.

[25] M. Xiong, J. Parsons, J. Edmondson, H. Nguyen, andD. Schmidt. Evaluating technologies for tactical informationmanagement in net-centric systems. In Proceedings of theDefense Transformation and Net-Centric Systems conference,2007.