Cloud Testing ToolsXiaoying Bai∗ †, Muyang Li∗, Bin Chen∗, Wei-Tek Tsai∗ ‡, Jerry Gao∗ §

∗Department of Computer Science and Technology, TNList, Tsinghua University, Beijing, China† SKLSDE, Beijing University of Aeronautics and Astronautics, Beijing, China

‡School of Computing, Informatics, and Decision Systems Engineering, Arizona State University, USA§Department of Computer Science and Technology, San Jose State University, USA

Email: baixy@tsinghua.edu.cn, li-my08@mails.tsinghua.edu.cn, chenb.thu@gmail.com, wtsai@asu.edu, jerry.gao@sjsu.edu

Abstract—Cloud platform provides an infrastructure for re-source sharing, software hosting and service delivering in apay-per-use approach. To test the cloud-based software systems,techniques and tools are necessary to address unique qualityconcerns of the cloud infrastructure such as massive scalabilityand dynamic configuration. The tools can also be built on thecloud platform to benefit from virtualized platform and services,massive resources, and parallelized execution. The paper makesa survey of representative approaches and typical tools for cloudtesting. It identifies the needs for cloud testing tools includingmulti-layer testing, SLA-based testing, large scale simulation,and on-demand test environment. To address the needs, itinvestigates the new architecture and techniques for designingtesting tools for the cloud and in the cloud. Tool implementationsare surveyed considering different approaches including migratedconventional tools, research tools, commercial tools and facilitieslike benchmark and testbed. Based on the analysis of state-of-the-art practices, the paper further investigates future trend oftesting tool research and development from both capability andusability perspectives.

Keywords-Cloud Computing, Cloud Testing, Testing Tool

I. INTRODUCTION

Cloud computing proposes a new architecture of multi-layered resource sharing, such as infrastructure-as-a-service (I-aaS), data-as-as-service (DaaS), platform-as-a-service (PaaS),and software-as-a-service (SaaS) [1]. Resources can be dynam-ically allocated based on usage demands following negotiatedService-Level-Agreement (SLA) in a pay-per-use businessmodel to achieve cost-effective performance and resourceutilization. Cloud-based infrastructure has a significant impacton software technologies. It shifts the focus of software de-velopment from product-oriented activities to service-orientedreuse, composition, and online renting. It has a potentialto enhance the scalability, portability, reusability, flexibility,and fault-tolerance capabilities of software systems, takingadvantages of the new cloud infrastructure [2], [3].

However, cloud-based hosting introduces risks to the de-pendability of systems. Software is remotely deployed in a vir-tualized runtime environment using shared hardware/softwareresources, and hosted in a third-party infrastructure. Thequality and performance of the software highly depend on itsruntime environment that are usually out of the users’ control.For example, Amazon provides a cloud infrastructure and web-hosting system AWS (Amazon Web Services)[4], includingservices like EC2 (Elastic Compute Cloud) and S3 (SimpleStorage Service). It promises to keep customers’ sites up and

running 99.95% of the year, i.e., allows for 4.4 hours ofdowntime per year only. Unfortunately, an unexpected crashhappened in April, 2011 due to operation mistakes duringnetwork reconfiguration [5]. More than 70 organizations areaffected including FourSquare, the New York Times, andReddit, who pay to use AWS to run their websites on EC2. Dueto the accident, the performance of these websites is greatlydecreased, and some sites were even down for dozens ofhours. Applications hosted on remote clouds may have lowercontrollability and observability, compared with conventionalin-house hosted applications.

In addition, the mechanisms of massive resource sharingand usage-based allocation introduce uncertainties. From user-s’ perspective, it needs to ensure that system will providefunctionalities and performance with reasonable cost. From theproviders’ perspective, it aims to enhance resource utilizationby utilizing unused capacities based on guaranteed systemperformance. A SLA between the two parties usually definesthe QoS properties, including not only quality propertiesduring normal operations, but also fault-tolerance and recoverycapabilities in reaction to failures. However, as the cloudinfrastructure also changes its status (e.g., number of virtualmachine instances, number of applications, and usage loads)continuously, making testing a difficult task because the un-derlying infrastructure is also changing and thus performancewill be different even the same application is run.

Cloud testing issues have been addressed earlier [6]. Thepaper addresses tool support for cloud testing, specificallysupport for multi-layer testing, SLA-based testing, simulation-based testing, embedding testing with execution, and on-demand testing. Various tools are available [7], [8], [9], [10],[11], [12] to test cloud-based systems at various layers includ-ing hardware interface, platform interface, storage system, andapplication systems. One can make the following observation:

1) Testing faces the challenges of cost and scalability due toincreases in software size and complexity. However test-ing tools can also take advantages of from the enormousresources from a cloud infrastructure, resources suchas virtualized platforms and services, large computationpower and memory, parallel operations, and automatedrecovery mechanisms that comes in a PaaS. For exam-ples, conventional testing tools, such as JUnit test frame-work and symbolic execution tools, can be migrated toa cloud platform. They can be refactored using cloud

Proceedings of The 6th IEEE International Symposium on Service Oriented System Engineering (SOSE 2011)

978-1-4673-0412-2/11/$26.00 ©2011 IEEE 1

parallel programming techniques like MapReduce. Byparallelizing job units, tools’ performance and scalabilitycan be greatly enhanced.

2) To address cloud unique quality problems, new test-ing methods, techniques and tools are necessary. Someframeworks are proposed to provide testing as contin-uous infrastructure services [13], [14]. Testing archi-tectures like Collaborative Verification and Validation(CV&V), Testing-as-aservice (TaaS), self-testing of au-tomatic services, are proposed recently. Techniques suchas simulation, services mocking, job parallelization, andenvironment virtualization can also be used to enhancethe testing capabilities.

3) To compare across heterogeneous cloud infrastructureand development stacks, new metrics are defined andbenchmarks [15], [16], [17] are developed to setup thebaseline for testing and evaluation.

Cloud testing is an emerging area. This paper suggests newtrends of cloud testing tools, including the needs for onlineand adaptive testing, cross-cloud testing, SaaS multi-tenancytesting, real-time results processing, SLA conformance testing,dependability testing such as security testing and reliabilitytesting.

This paper is organized as follows. Section II analyzes theissues of cloud testing tools. Section III introduces novel ar-chitectures for cloud testing. Section IV reviews enabling tech-niques and typical tool implementations. Section V presentscloud benchmarks. Section VI briefly shows commercial test-ing tools. Section VII compares the tools and discusses futureneeds of tool development. Finally, section VIII concludes thispaper.

II. CHALLENGES AND NEEDS

As shown in TABLE I, software hosted in a cloud environ-ment differs from that in an in-house environment. Essentially,the in-house environment is usually fully controlled by soft-ware vendors with dedicated hardware and software resources;while the cloud-based hosting provides infrastructure as leasedservices from a theoretically unlimited resource pool. Softwaredeployed to the cloud are supposed to benefit from its built-in mechanisms of distributed parallel computing and fault-tolerance, using new techniques such as Hadoop [18] and GFS(Google File System)[19]. However, risks are introduced usingthe new model of resource sharing. Resources are shared ina virtualized and multi-tenancy architecture (MTA) [20], [21],[3]. To reduce the cost of leased resources, and to maximizeresource utilization, usage-based dynamic resource allocationis a basic principle of cloud management. With guaranteedSLA, it automatically allocates more resources when the loadincreases (scale up), and releases unused resources when theload decreases (scale down). In addition, it can also changesoftware configuration and runtime environment in response toSLA changes or the needs of failure recovery. The process ismostly carried out online following pre-defined policies, wherethe users usually have low control over.

TABLE ICOMPARISON OF SOFTWARE IN CLOUD AND IN-HOUSE ENVIRONMENT

In-house Hosting Cloud Hosting

Architecture Centralized, limitedparallel and fault tol-erance

Built-in distributedparallel computing,high fault-tolerance

Configuration Pre-definedconfiguration,offline deployment,relative fixed set-up

Dynamicdeploymentand onlinereconfiguration,transparent tosoftware providers

Resource Allocation Stable dedicated re-sources, limited up-per bound

Unlimited resourcepools, dynamic allo-cation based on real-time usage

Resource Sharing Limited sharing multi-tenancyarchitecture, large-scale resourcesharing

RuntimeEnvironment

Stable dedicated en-vironment, in-housecontrol

Virtualizedcomputing services,unpredictableenvironment, lowcontrollability

Scalability Offline scaleup/down

Online massive scal-ability with unlimit-ed resource pool, dy-namic scale up/downin response to usage

Many new techniques are introduced to support the desiredfeatures of cloud computing, such as virtualization and MTA.These techniques also introduce new fault models and threat-ens to system quality [22], [23], [24], [25], [26]. For example,even an application is allocated with promised hardware likeCPU and disks, its performance cannot be guaranteed. Theperformance of application in the cloud is affected by manyfactors such as the number of virtual machines (VM) instances,VM management policies, the effectiveness of hypervisorscheduling, the strategy of application migration and recoveryin case of failures. Some of these factors change at runtimeand thus cannot be fully predicted and controlled.

In traditional software lifecycle, testing is usually a stage inthe process, and is often performed offline by test engineersbefore product delivery. As cloud-based software has uniquelifecycle models and quality issues, new testing capabilities arenecessary to meet the needs of cloud testing such as continuousonline testing and massive scalability testing.

In [6], different testing types in the cloud environment,including service functional testing, integration testing, APIand connectivity testing, performance & scalability testing,security testing, interoperability & compatibility testing, andregression testing are discussed. Particularly, cloud testing hasthe following features.

Multi-layer testing: Faults may exist in various cloudcomponents including hardware, network, virtualization man-agement and storage system. As a large-scale distributedsystem, a cloud may also have complex fault-tolerant andfailure recovery mechanisms such as writing into three copies

2

Fig. 1. Taobao Order statictics [27]

in GAE (Google App Engine). It is difficult to locate faults incase an application fails. For thorough analysis, testing needsto be performed on each component at all of these layers. Eachlayer requires different testing focuses and techniques.

SLA-based testing: For conventional software, testing isbased on source code or software specifications that describeexpected software behavior using natural language or formalmodels. For software deployed on the cloud, source code maybe unavailable. Instead, SLA is negotiated between softwareand infrastructure providers, including functionalities and QoSproperties. SLA thus provides the basis not only for cloud pro-visioning, but also for test design, execution, and evaluation.

Large-scale simulation: Testing needs to simulate variousinputs and scenarios. As an open platform, a public cloudallows for flexible access and operation. The number of pos-sible usage scenarios is huge. The load is high and unexpectedand large fluctuations can occur. For example, Taobao [27] isa large e-commerce system in China. It has more than 300million registered customers with about 300 billion RMB peryear. It sold about 47,000 products per minute on average in2010. Figure 1 shows an example of a common shop’s orderstatistics in a week [27]. At the peak of the week, it has over3500 orders a day; while at the low, it is only about 100. Inthis example, each order is on average contributed by 318 pagebrowses from 201 customers.

To test the functionality and performance of such complexsystems, large-scale simulation is needed. Actually, it needs tosimulate not only the usage of the system, but also the changesin the environment such as infrastructure configurations.

On-demand test environment: Testing needs to be trig-gered online whenever a change occurs in the cloud includingthe application, runtime environment, and infrastructure. Anenvironment is helpful for test assets sharing, automatic testgeneration/ selection/execution, results collection and analysis.It usually takes effort to build test environment and maintainit for regression testing. To test the cloud under various usagescenarios, it is necessary to design testing mechanisms so thatvarious test environments can be deployed or invoked whenneeded.

Embedded continuous testing for SaaS: Most SaaS soft-ware use MTA in a cloud environment [20], [28], [29], [3].It is possible to embed monitoring and testing services in

a cloud that continuously track the inputs and outputs ofservices whenever the services are activated for execution.Once sufficient number of inputs and outputs are collected,test oracle may be established statistically with voting [29],[30]. MTA can also be simulated [31].

Test cases generation from metadata: One of the keyaspect of SaaS and PaaS is that the system is driven by meta-data. For example, GAE and Force.com both use metadata tomaintain and track system and application status. Furthermore,failure of metadata nodes can have disastrous effect on thesystem as it did happen to GAE earlier. Another feature ofmetadata is that it can be used to generate test cases. Testcases can be generated by examining metadata, e.g. Incomelength of customer must be 64 bits or so, hence some simpletest cases will be randomized with 64 bits. One can generatea collection of customers of 64 bits, another collection with128 bits or any other bits. This is another new feature to beexplored in cloud testing.

Policy enforcement as testing: Policy is a common service-oriented technique. Policies can be enforced at runtime duringcloud execution. Policies can also be automatically generatedfrom metadata [29]. Policy enforcement can be an integratedpart for testing and evaluation of cloud applications.

Scalability metrics: While traditional scalability metricsare available for parallel and distributed computing, cloudcomputing needs new metrics as it needs to consider boththe performance gain versus the resource needed [28], [32],[33]. Specifically,

1) T processing time and this reflects the traditional speed-up.

2) R ∗ Tr resource consumption and this reflects the re-source usage in the system.

3) PRR performance resource ratio, which will be definedin the following and reflects the relationship betweenperformance and the resources used.

4) Metric variance, such as the variance of speed-up, vari-ance of resource consumption and variance of PRR.The variance of PRR will be defined later in this paper.

The performance resource ratio is defined as:

Tw = Tq + Te (1)

CR =∑

Ri ∗ Ti (2)

where Tw represents the waiting time, Tq the queueing timeand Te the execution time, Ri is the allocation of resource i,which can be I/O bandwidth, CPU and memory usage and Ti

is the time resource i is used. PRR is defined as:

PRR =1

Tw∗ 1

CR(3)

Given PRR, the scalability of the SUT is measured by thePC (Performance Change) when workload changes.

PC =PRR(t)W (t)

PRR(t′)W (t′)(4)

3

Fig. 2. Trustworthy Service Broker Architecture [37]

with the ideal PC equals to unity.

III. ARCHITECTURES

Cloud testing systems are usually designed based on cloudplatform and service-oriented concepts. Some new architec-tures are proposed in recent years to provide testing function-alities as online services.

A. Collaborative Verification and Validation

Like a service-oriented system, applications in a cloud isoften composed by services developed by different parties.For the application to work, each party needs to deliver theircomponents right, and the integration must be successful. Thusthe verification and validation of the application need to becontributed by multiple parties in a collaborative manner, thusCV&V (collaborative verification and validation). One suchCV&V framework is the WebStrar [34], [35], [36] where testcases can be published and ranked for their potency. Mostpotent test cases will be used first to test new software toreduce the testing effort.

The conventional service broker can be extended withcheck-in and check-out testing process by adding just-in-timeservice testing, evaluation, and ranking capacities, as shownin Fig. 2 [37]. The check-in test process ensures that onlyqualified services are accepted for publishing; while the check-out test on the candidate services satisfying a request to assesswhether the services have changed after publishing and theimpacts of the changes.

This concept can be further extended to a contract-basedarchitecture [38], as shown in Fig. 3. The test broker enablesscalable and flexible collaborations among test participants.A test provider supplies test knowledge such as test cas-es, executable test scripts, test results, defects, test ranks,services ranks, and test/service evaluation models based ontesting statistics. A tester carries the published test cases andsimulates testing on the target services. Both test providersand testers can be any party including service providers,

Fig. 3. Contract-Based Collaborative Verification and Validation [38]

service users, or third party independent test participants.Contracts are identified from two perspectives: TSC (TestingService Contracts) and TCC (Test Collaboration Contracts).TSC is the communication between testing components andthe SUT (service under test), including test requirements andtest invocation protocols. TCC defines the way that testingcomponents collaboratively design test cases, execute test plan,and evaluate test results. Figure 3 depicts the collaborationactivities among different parties.

B. Testing as a Service (TaaS)

To reduce the cost of test design, execution, and main-tenance, a concept TaaS was proposed to establish a uni-fied service-based framework for promoting reuse of all testartifacts including test data, tools and process. It providesstatic/dynamic on-demand testing services in/on/over cloudsfor the third-parties at any time and all time (365/7/24) [6],[39]. According to [40], TaaS concept was initially introducedby Tieto in Denmark in 2009. TaaS has received wide attentiondue to its scalable testing environments, cost reduction, utility-based service models, and on-demand testing services.

As shown in Fig. 4, Yu, et al., [12] defined a 5-layer TaaSframework based on cloud infrastructure services, including:

• Test service tenant and contributor layer: This layer pro-vides functionality that supporting testing service tenantand contributor to interact with TaaS.

• Test task management layer: This layer is a middle-ware layer, supporting service registry and repository,scheduling and dispatching test tasks, and some otherfunctionality.

• Testing resource management layer: This layer acts as theCloud infrastructure, taking the responsibility of resourcemanagement and monitoring, test task provisioning.

• Test layer: This layer is the kernel part of the platform,consisting of service composition, service pooling andtest-reduce sub layer.

• Testing database layer: This layer is used to store the testtask of tenants, targets-under-test, service images, and bugtracking results.

Candea [7] identified three categories of testing services:TaaSD for developers, TaaSH for end users and TaaSC ascertification service. They argue that with a pricing model,

4

Fig. 4. 5-layer architecture of Testing-as-a-service [12]

TaaS can be operated as a public service and as a business,targeting at the ”long-tail” small business companies.

The concepts have been incorporated into commercial prod-ucts like Tieto [40] and Sogeti[41]. Sogeti provide a cloud-based provisioning platform so that partners’ testing tools,such as IBM Rational and HP Mercury, can be automaticallydeployed and executed following the pay-per-use approach.

C. Test Support as-a-Service (TSaaS)

To enhance testability of autonomic services, TSaaS wasproposed so that each service will expose both productionand test environment to external users. Test functions (suchas specification, execution, configuration and reporting) areexposed as API services. King, et al.,[42], [43] applied au-tonomic computing concepts to testing of adaptive systems,called autonomic self-testing (ATC). The technique was thenmigrated to the cloud platform [9], called TSaaS, so thatservices that are hosted on remote cloud platform can exposetheir test support APIs for partner providers. A self-test harnessis developed to manage testing workflow and activities. Itmonitors changes or updates on hosted services, utilizes nec-essary infrastructure services, and invokes TSaaS supportingservices to validate the changes. Test operations exposed assupporting services include test setup, input, assertions, andteardown operations. These services are provided for cloudpartners during the development, testing and maintenance oftailor-made cloud applications and services. They can alsobe used for design, build, and deployment of automated testsacross administrative domains.

Fig. 5. CloudSim Architecture [44]

IV. TECHNIQUES AND TYPICAL TOOL IMPLEMENTATIONS

The section identifies some techniques used in cloud testingsystems and their typical tool implementations.• Simulation. To reduce the complexity and separate qual-

ity concerns, a cloud simulator is necessary for cloudsystem testing so that it can focus on quality problems atparticular cloud component, and analyze system behaviorunder various scenarios.

• Service mocking. The systems hosted on the cloud mayuse services from various providers. Conventional stuband driver techniques need to be adapted to mock externalfunctionalities at the service interface level.

• Test job parallelization. Parallel programming becomesa natural practice for applications built upon cloud. Testscan benefit from the practice by dividing test activitiesinto independent jobs and parallelizing them to reducetesting time and cost.

• Environment virtualization. Throughout software life-cyle, testing and regression testing usually need to main-tain various testing environment for different versions,dependent software and platforms. It is always resourceconsuming to maintain the environment, and effort con-suming to set up the environment for each test execution.Virtual machines can help to ease and speedup theprocess, and to reduce test cost.

A. Simulation

1) CloudSim: CloudSim [44] is built by CLOUD (CloudComputing and Distributed Systems) Laboratory at the U-niversity of Melbourne in Australia. It aims to provide atoolkit for modeling and simulating the behavior of variouscloud components including data centers, virtual machines andresource provisioning services. It can be used for analyzingand evaluating cloud strategies in a controlled simulated envi-ronment. Particularly, CloudSim facilitates initial performancetesting with less time and effort to set up test environment.

5

Fig. 6. D-Cloud Architecture [45]

As shown in Fig. 5, CloudSim is built with two layers: theSimulation layer and the User Code layer. It is designed tosupport modeling and simulating typical cloud features such asnetwork behavior, VM allocation, Cloud federations, dynamicworkloads and power consumptions. It claims to have beenused for research in companies like HP and universities.

2) D-Cloud: D-Cloud [45] is developed by University ofTsukuba in Japan. D-Cloud is a dedicated simulated testenvironment built upon Eucalyptus, an open-source cloud in-frastructure providing similar functionalities as Amazon EC2.It uses QEMU, an open-source virtual machine software,to build virtual machine for simulating faults in hardwareincluding disk, network and memory.

Fig. 6 shows D-Cloud architecture. D-Cloud can simulatevarious typical faults and inject them into host/guest OS.It supports the definition of fault types, fault injection timeand fault duration. A XML-based language is designed fordescribing test environment configuration and test scenarios.A tester first submit test plan in the specific language toset up infrastructure parameters, workload and fault injectionscenarios. D-Cloud then initiates virtual machines to simulatethe execution process following the test plan. In this way,D-Cloud enables flexible and observable platform for testingdistributed system on the cloud.

3) PreFail: PreFail [46], [47] is developed by ParLab(Parallel Computing Laboratory) at University of Californiaat Berkley. For the cloud built out of tens of thousandsof unreliable computers, it usually experiences frequent anddiverse failures. Failure testing is thus important to validatethe correctness and efficiency of cloud’s recovery protocols.However, it is expensive to enumerate all of the possiblefailure scenarios. PreFail thus introduces a framework forsystematically and efficiently explore failures.

PreFail introduces a failure abstraction called failure ID(FID), composed of an I/O ID (abstract information of an I/Ocall) and the injected failure, to identify every failure. UnlikeD-Cloud that provides simulated ”actual” faults, PreFail inserts

a ”failure surface” into the target system between the targetsystem (e.g., HDFS [18]) and the OS library (e.g., Java SDK).In this way, testers can flexible program failure testing policies.PreFail provides optimization techniques to reduce redundantfailure scenarios before fault injections and simulation, andparallelization method to divide failure sequences into inde-pendent parts that can be exercised in parallel on differentmachines.

PreFail has been adopted by Cloudera Inc. for Hadoopsoftware testing.

4) Distributed Load Simulation: Cloud storage system ischaracterized by highly parallelism and non-determinism. Aspart of the research and implementation of the Cloudy2distributed database system, ETH Zurich proposed a newdistributed testing architecture for simulating parallel jobs[10].

This framework contains two types of nodes - Master andSlave. Master is unique identified, which is responsible fordistribution, synchronization and management of all slavenodes. Master is started with a given test. It waits for enoughslaves to connect to it, then sends every slave correspondingtasks. During the execution, the master controls the executionsequence of slaves to guarantee that all tasks in a step startat the same time. The slaves run testing tasks and store testresults locally, including nodes’ states collected by daemonthread at each node. At the end of the test, master recollectsevery slave’s results, analyzes and then generates statistics andgraphs for analyzing test execution.

A workload description is organized at three layers: task,step and test. A task is an atomic job in a test, which givesa clear set of instruction to execute, containing configurationsabout loads, actions, and limits. Each step is a collection oftasks, which are performed in parallel with a group of slaves.A test is a well-defined sequence of steps. In this way, thisframework can simulate a variety of workload scenarios.

B. Service Mocking

Software deployed on the cloud platform may use third-party services. Such dependencies make it hard for test setup and fault localization. Stubs and drivers are widely usedtechniques for incremental integration testing. Stubs providesimulated implementations while drivers provide simulatedinvocations for those functionalities that are unavailable orindifference for current integration testing. Similar to stubs,service mocking is used to break a service’s external depen-dencies so that it can be tested independently. Service mockingreplace remote service with a simulated one which behaves asif the real one is called. The simulated implementation canprovide the same interface functions, accept messages, andreturn results.

Service mocking has been used in many service testingsystems, such as soapUI and InfoQ. iTKO@LISA analyzesthe constraints in enterprise systems of restricting availabilityand accessibility for development and delivery. It providesa constraints-free cloud environment with fully virtualizedservices.

6

C. Test Job Parallelization

Compared with traditional software, the size and complex-ity of cloud testing increase tremendously in following twoaspects:

1) Huge number of testing tasks. Cloud is becoming acommon practice for software hosting. Massive servicesare migrated to cloud platforms in recent years. Forexample, up to July 2011, Amazon has attracted over400,000 customers and Google App Engines over 10million users. To have online quality control over theseservices, the cloud platform has to devote a lot of timeand resources for testing. According to a Google report[48], it needs to support testing of over 500,000 buildsof 20,000 projects per day. It maintains over 120Ktest suites in the code base. Over 7.5M test suites areexercised per day.

2) Large scale testing. Cloud services, including IaaS, PaaSand SaaS, are expected to provide higher quality and de-pendability than traditional software, such as scalabilityand fault-tolerance capabilities. In accordance, testingneeds to be exercised in a large scale to verify andvalidate these properties.

Parallel techniques are thus used to speed up testing[49],[50], [51]. Lastovetsky [52] once exercised a case studyof parallel execution of 588 Orbix test suites on a clusterof multiprocessor workstations. The results showed that itaccelerates test executions by 3.8 to 3.9 times using singleSolaris 4-processor workstations, and 6.8 to 7.7 using a clusterof two 4-processor workstations. Cloud by nature provideparallelization support with easy-to-use programming models.Some testing tools are developed on, or migrated to, the cloud,taking the advantage of cloud virtual machines to parallel testexecutions.

1) Cloud9: Cloud9 [8], an academic research project fromEPFL in Switzerland, migrates symbolic execution to thecloud platform. Symbolic execution is an important testingtechnique introduced in 1970s. It reasons all the possibleexecutions by exploring program path-by-path. In spite ofresearch over 30 years on path exploration, it still faces thechallenging problem of scalability. With increasing programsize and complexity, its memory and CPU consumptions goexponentially. It is difficult to be applied to industry softwarein general where software often contains millions of linesof code. Basically, Cloud9 parallelizes symbolic executionon large shared-nothing clusters of computers. To do this, itdivides the path exploration work into independent jobs to beallocated to different worker nodes. The conventional searchstrategy is distributed to the workers, with reduced couplingand coordination among workers. The global load balancer isused to balance the workload of each worker, in order to gethigh efficiency of testing service. Fig. 7 shows the architectureof Cloud9. Each worker consists of a runtime, a searcher anda constraint solver. It independently explores a sub-tree ofprogram’s execution tree.An initial prototype is implementedwith the single-node Klee symbolic execution engine running

Fig. 7. Cloud9 Architecture [8]

on the infrastructure service provided by Amazon EC2. Itreduced testing time of 32 real UNIX utilities on a average bya factor of 47, with a maximum of 250-fold speedup.

2) HadoopUnit: HadoopUnit [53] migrates JUnit testframework to Hadoop platform. JUnit test cases are createdas independent Hadoop MapReduce jobs. The map() functionreceives test jobs as < testname, testcommand > pair. Ateach node, the command is executed as a process. The reducergets < testname, testresult > from each map and combinesall the results. Experiments shows that a 150-node clustercan produce 30x improvement compared with sequential testexecutions on a local computer.

3) YETI: YETI (York Extensible Testing Infrastructure)[11] also provides a cloud version random testing tools. Ituses MapReduce to parallelizes the processing of test inputsand results. A preliminary evaluation was carried using A-mazon EC2. It showed clearly performance improvements byemploying more computers and distributing the jobs.

D. Environment Virtualization

Cloud can also be used for simulating domain-specificinstruments for large-scale application testing. For example,the Network Management System (NMS) needs to managenetworks comprised of a huge number of elements. To testNMS applications, it either tests with real elements, or usessimulators to simulate the required environments, both ofwhich require significant time, cost and resources. To addressthe challenge, cloud infrastructure services can facilitate tosimulate different virtual elements and test scenarios[13].Agents that emulate the managed elements are deployed inthe cloud using public infrastructure services. The virtualmachines (VM) created in the cloud simulates heterogeneousdistributed environment with different OS and runtime config-urations. Metrics such as CPU utilization and memory utiliza-tion are collected through VM hypervisors for performanceanalysis.

Similarly, S. Baride, et al., proposed a cloud-based approachfor mobile application testing where infrastructure services areused to simulate diversified mobile devices, hardware config-urations, heterogeneous application platforms, and complexdependencies [14]. Based on the cloud infrastructure, a testenvironment is built including the emulator which providesunified interfaces to different simulated/actual mobile devices,emulator of mobile environments and service platforms, test-

7

ing tools to perform various types of testing such as securityand performance testing.

OCT (Open Cloud Testbed) is representative large-scaletestbed [54]. Up to 2009, it has 120 nodes in four data centersgeographically distributed located at Baltimore, Chicago andSan Diego, which are connected with high performance 10G-b/s network. It aims to investigate the performance of dataintensive computing systems with extremely large data setsand very high volume data streams. It has tested heterogeneouscloud platforms such as Eucalyptus, CloudStore, Haddop,Sector/Sphere, and Thrift.

V. CLOUD BENCHMARKS

This section introduces typical benchmarks and testbeddeveloped to support cloud testing.

A. YCSB

To achieve the objectives such as scale-out, elasticity andhigh availability, cloud systems typically sacrifice sophisticat-ed mechanisms in traditional ACID databases such as complexjoins and aggregates, and strong transaction model with two-phase commit. Special design tradeoffs are made to optimizesystem performance, such as read versus write performance,latency versus durability, synchronous versus asynchronousreplication and data partitioning.

YCSB (Yahoo! Cloud Serving Benchmark)[15] by Yahoo!as a benchmark framework. It is designed to be extensibleand portable to heterogeneous clouds to give a reasonablecomparison among cloud storage systems (e.g., Cassandra andHBase). The framework is intended to deal with various qual-ity concerns including performance, scalability, availabilityand replication. Two tiers, performance and scalability, havebeen published as open source tools. Performance tier teststhe latency of request with increasing throughput until thedatabase system is saturated. Scaling tier tests how systemperformance changes with increasing number of machines.Two scaling metrics are used: scale-up and speedup. Scale-up shows the system’s static scalability, in which, the systemshould remain a constant performance, given a larger amountof data with more servers. Elastic speedup shows the system’sdynamic scalability, in which, the system should achieve aperformance improvement, as new servers added when thesystem is still running.

YCSB tool consists of two parts: a workload generatorand a package of standard workload. The workload packageprovides a collection of programs representing typical cloudoperations like read/write, data size, request distributions, andso on. The operations supported consist of insert, update, readand scan. It uses different distribution model to select theoperations and records to perform. Four built-in distributionsare used including Uniform, Zipfian, Latest, and Multinomial.Typical workload patterns include update heavy (50% Read,50% Update), read heavy (95% Read, 5% Update), read only(100% Read), read latest (95% Read, 5% Insert), and shortranges (95% Scan and 5% Insert).

YCSB supports workload generation using user-definedworkload description, which can be a mix of workloads inthe core package, or complete user-defined workloads withnew combination of operation and distribution.

B. Enhanced TPC-W

C. Binning, et al., [17] identified following novel featuresof cloud storage systems compared with transactional databasesystems:• Elasticity to changing conditions. Conventional systems

are mostly used in a managed environment using afixed configuration. Cloud enables dynamic resourcesallocation and deallocation so that the systems can beadapted to load fluctuations on the fly.

• Tradeoffs between consistency and availability. It is im-possible for cloud storage systems to provide availabil-ity and strong consistency together in the presence ofnetwork failures. Most cloud providers thus offer onlyweaker forms of consistency for the sake of availability.

• Pricing problem. Cloud aims to provide economy ofscale. However, the promise of unlimited scalabilityis difficult to achieve. Given various pricing plan andgranularity by different providers, such as infrastructureservices and platform services, it leads to different overallcost.

Existing benchmarks like TPC-benchmarks are mostly builtfor transactional database systems. The metrics and architec-ture of them are not suitable for cloud systems. C. Binniget al. [17] hence suggested a new benchmark system specificfor cloud scalability, pay-per-use and fault-tolerance testingand evaluation. The benchmark uses e-commerce scenario anddefines web interactions as benchmark drivers. Particularly,following four new metrics are defined for cloud storagesystem evaluation, as shown in Fig. 8:• Scalability. Cloud services are expected to scale linearly

with a constant cost per web interaction. The papersuggests measures the deviations of response time to theperfect linear scale by using correlation coefficient R2 orby determing the parameters of a power function of theform f(x) = xb.

• Cost. The economy of cloud performance is measured as$/WIPS, where WIPS is for web interactions per secondused by conventional TPC-W benchmark. In addition, italso measures standard deviation of the cost during thescaling.

• Peaks. This is to measure how well a cloud can adapt topeak loads, including scale-up to reach peak loads andscale-down after peak loads. The adaptability is definedas the ratio between WIPS in RT (real-time) and IssuedWIPS.

• Fault tolerance. Cloud infrastructure are usually basedon huge number of commodity hardware. Hardware fail-ures are common to the infrastructure services. Hence, themetric is introduce to analyze cloud self-healing capabili-ties. Given failures in a period of time, the recoverability

8

is also defined as the ratio between WIPS in RT (real-time) and Issued WIPS.

C. Terasort

The Hadoop distribution comes with a number of bench-marks, including TestDFSIO, nnbench, mrbench and Tera-Gen/TeraSort/TeraValidate. The TeraSort benchmark is popu-lar for testing Hadoop clusters at both HDFS and MapReducelayers. It aims to compare cluster performance by sorting 1TBof data as fast as possible. Hadoop set a record in 2009 bysorting 100TB of data at 0.578 TB/minute using 2800 nodes.

The package of Hadoop Tera benchmark is composed ofthree components.• TeraGen to generate random input data.• TereSort to run MapReduce sort jobs with the generated

inputs.• TeraValidate to ensure that the outputs of TeraSort are

correctly sorted.

D. Cloudstone

Cloudstone [16] is an open source project from the RADLadat UC Berkeley. It aims to address the performance charac-teristics of new system architectures and to compare acrossvarious development stacks. A benchmark toolkit is built usingsocial applications and automated load generators that can beexecuted on different deployment environments. In accordanceto the pay-as-you-go model of storage and computing, newmetrics are introduced, called dollars-per-user-per-month, forevaluating cloud performance with regard to realistic usageand cost.

Cloudstone is built upon open source tools including Oliowhich is social-event calendar web application, Faban which isa Markov-based workload generator, and automation tools toexecute Olio following different workload patterns on variousplatforms. The benchmark has been exercised on differentconfigurations of Amazon EC2 platform.

E. MalStone

MalStone is developed by the Open Data Group [55]. Itis specially designed for performance testing of cloud mid-dlewares for data intensive computing. MalGen is developedto generate synthetic log-entity files that are used for testinginputs. It can generate tens of billions of events on cloud withover 100 nodes.

VI. COMMERCIAL TESTING TOOLS

A. SOASTA

SOASTA [56] is motivated by the necessity to test inproduction, rather than in a laboratory environment. Today’sweb applications usually follow agile practices with frequentbuilds and high change rates. Load testing with legacy tools inthe laboratory can be significantly different from testing in theproduction environment in terms of scale, configuration, userprofiles and network environment. Running tests against pro-duction websites thus can achieve higher degree of accuracyand confidence, compared with lab practices.

Fig. 9. SOASTA CloudTest architecture [56]

SOASTA CloudTest is a production performance testingtool for Web applications. It can simulate thousands of virtualusers visiting website simultaneously, using either private orpublic cloud infrastructure service. The worker nodes can bedistributed across public and private clouds to cooperate in alarge load testing. Test results from distributed test agents areintegrated for analysis. Memory-based analytic techniques areimplemented to handle, in real-time, the huge data produced bylarge-scale testing. Provisioning data are displayed via analyticdashboard on a synchronized time-line. Through an Ajax-based web UI, testers can operate and supervise the whole pro-cess including launching hundreds of load generation servers,creating and running test agents geographically distributed,and analyzing test results.

Fig. 9 shows SOASTA architecture. CloudTest is composedof three distributed services to support test creation, testexecution and test results analytics, as listed below:• SOASTA repository is responsible for delivering SOAS-

TA CloudTest objects such as test scenario recordings,test composition, and performance data.

• Analytics Dashboard is a memory-based analytic serviceto handle large sets of results and analytics from dis-tributed load tests. It is able to correlates many datastreams from distributed environments into a single onewith synchronized timeline.

• Meastro is a test engine as a massively multi-threadservice, used for test execution (including sending andvalidating responses). Multiple Meastros can execute d-ifferent parts of a test composition collaboratively, withthe ability to be geographically distributed.

B. iTKO LISA

iTKO LISA [57] aims to provide a cloud-based environmentand virtual services for composite application development,verification and validation. It claims to reduce software deliv-ery timeline by 30% or more using its innovative approachto support continuous integration for development and testing.Central to LISA architecture is its virtualizaion technology.For unavailable or inaccessible resources, LISA provides vir-tualized services by simulating the target system’s dynamicbehavior so that they can respond as live systems. In thisway, it breaks dependence constraints of system integrationand supports continuous testing.

9

Fig. 8. Metrics of Cloud Benchmark [17]

LISA quality services are provided from three perspectives:LISA Test, LISA Validate, and LISA Pathfinder.• LISA Test: The capabilities it offers to enhance testing

include coverage-based testing for heterogeneous dis-tributed architecture, codeless testing, UI testing, load andperformance testing. It brings testability to all compo-nents deployed in the cloud by LISA development toolkit.It establishes collaborative testing platform with portableand executable test cases that are shared across differentteams and environments. It provides a codeless testingenvironment that allows QA, development and others torapidly design and execute automated tests.

• LISA Validate: LISA provides continuous regressiontesting, triggered by change events, for each softwarebuild and in the production environment. Policies areenforced via governance infrastructure to ensure thatsystems conform to quality requirements at design time,change time and runtime.

• LISA Pathfinder: It traces interactions step-by-step, re-viewing the dataflow and control flow among the con-stituents in a composite application. It facilitates testers tosee through system execution process to localize defectsand identify performance bottlenecks.

C. Cloud Testing

Cloud Testing [58] is initiated by a group of architects andperformance experts from UK’s largest Website PerformanceMonitoring & Load Testing Company. It aims to support crossbrowser and functional testing of Web applications. As web-sites need to be compatible to various browsers and operatingsystems, Cloud Testing offers a shared test environment so thatusers need not set up and maintain various testing platform toensure website portability. Test scripts are recorded by usersfrom local browsers using Selenium IDE. The scripts are thensubmitted to Cloud Testing to be executed automatically in thecloud with various browsers on operating systems.

VII. DISCUSSION

Compared with conventional testing methods, cloud testingemphasizes more on system testing and online testing. This isdue to the novel design and development methods imposed bycloud computing. This is still an emerging research area withmany open problems.

To better understand the issues and needs of cloud testing,the paper looks into state-of-the-art tool implementations.TABLE II selects 13 representative tools introduced aboveand compares them from following perspectives:• Testing objectives. Large-scale performance testing, scal-

ability testing, fault-tolerance testing, recovery testing,and cost-related testing are typical objectives.

• Testing activities. Testing is process of environment set-up, inputs generation, test configuration, test executabledeployment, execution, results collection and analysis.For test tools developed on a cloud, special activities areneeded, such as external service mocking, geographicalsimulation, parallel execution, resource provisioning, andresults aggregation.

• Tool architecture. Some cloud testing tools are not de-veloped on the cloud platform. For those on the cloud,testing with simulation support can further enhance cloudtestability and observability.

The results show that most of the tools take the advantagesof cloud platform to reduce testing cost and enhance tools’scalability. However, many of the features are not well sup-ported yet. For example, fault-tolerance is one of the majorpromises of cloud computing. However, only D-Cloud andPreFail attempt to perform fault-tolerance and recovery testing.Features that need future tool support include:• Online and adaptive testing. This is to address the needs

of dynamic service composition, deployment and on-lineevolution.

• Cross-cloud testing. Services need to be able to migratebetween clouds. Hence, a unified method is necessaryto test and evaluate services on heterogeneous cloudplatforms.

• Multi-tenancy testing. Services are shared across applica-tions following the multi-tenancy architecture. That is, thesame database and code can configured to meet differentcustomers’ requirements. MTA causes new problems toinstance performance, reliability and security.

• Real-time results mining. With enhanced productivity,testing can produce huge number of data and resultsanalysis could be a bottleneck to follow-up activitiessuch as fault discovery, localization, and recovery. Timelyanalysis of the log and events data is needed.

In addition, tools’ usability can also be improved, such

10

TABLE IICOMPARISON OF CLOUD TESTING TOOLS

TestO

bjectiveTest

Activity

ToolA

rchitectureL

argescalePer-for-m

anceTest-ing

ScalabilityTesting

FaultToler-anceTest-ing

Recovery

Test-ing

Cost-

relatedTest-ing

ServiceM

ock-ing

Geographical

simulation

Parallelexecu-tion

TestR

e-sourceM

an-age-m

ent

Results

Ag-

grega-tion

Architecture

Observer-

ability

ScalabilityC

loud-B

ased

D-C

loud[45]

××

XX

××

××

X×

×X

XPreF ail

[46]×

×X

X×

××

X×

X×

X×

CloudSim

[44]×

××

×X

X×

××

×X

X×

ET

HZ

[10]X

X×

××

××

××

×X

X×

YC

SB[15]

XX

××

××

×X

××

××

×B

enchmark

[17]X

X×

×X

××

××

××

××

Cloudstone

[16]×

××

×X

××

XX

X×

XX

T estbed[13]×

××

××

XX

X×

××

XX

Testbed[14]×

××

××

XX

××

××

XX

YE

TI[11]

××

××

××

×X

×X

×X

XSO

ASTA

[56]X

××

××

××

XX

X×

XX

ITK

O[57]

X×

××

×X

×X

XX

×X

XC

loudT esting[58]

X×

××

××

××

XX

×X

X

as supervision of execution status of tools on the cloud,sharing mechanism of testing assets (test data, test cases, testscripts, and results.), and automation support (test generation,execution, and deployment.).

VIII. CONCLUSION

Cloud computing changes the way of software deployment,hosting, delivery, and maintenance, even though its long-term

impacts on software engineering are not fully analyzed yet.This paper is motivated by the following two observations:

1) The new quality issues introduced by cloud collaborativeand dynamic architecture. New techniques and imple-mentations introduce new fault models.

2) The new opportunities to enhance testing system ca-pabilities by using cloud infrastructure services andenvironment. Conventional testing techniques can beparallelized using cloud infrastructure services to over-come cost and resource limitations.

The paper shows that testing tools are emerging in recentyears from both of the above two aspects. Especially, cloudstorage testing receives special attentions with standard bench-marks and evaluation results across various implementations.New architectures are proposed to provide continuous test-ing services and large-scale system testing capabilities. Withgrowing maturity, some techniques have been incorporatedinto commercial tools, especially performance testing of webapplications.

However, many cloud concepts are not well defined yetsuch as virtualization and multi-tenancy. There are still quite afew quality issues not identified and well understood. Specifictesting techniques are badly needed to address the uniqueproblems of cloud architecture.

ACKNOWLEDGMENT

This project is supported by National Science FoundationChina (No. 61073003), National Basic Research Program ofChina (No. 2011CB302505), the Open Fund of the StateKey Laboratory of Software Development Environment (No.SKLSDE-2009KF-2-0X), U.S. National Science Foundationproject DUE 0942453, and the European Regional Develop-ment Fund and the Government of Romania under the grantno. 181 of 18.06.2010.

REFERENCES

[1] B. P. Rimal, E. Choi, and I. Lumb, “A Taxonomy and Survey of CloudComputing Systems,” in Proceedings of Fifth Int. Joint Conf. INC, IMSand IDC NCM ’09, 2009, pp. 44–51.

[2] W. Tsai, Q. Shao, Y. Huang, and X. Bai, “Towards a Scalable andRobust Multi-Tenancy SaaS,” in Proceedings of the Second Asia-PacificSymposium on Internetware, 2010, p. 8.

[3] W. Tsai, Y. Huang, and Q. Shao, “EasySaaS: A SaaS developmentframework,” in SOCA ’11, 2011.

[4] Amazon Web Service. [Online]. Available: http://aws.amazon.com/[5] “News Briefs,” Computer, vol. 44, pp. 18–20, 2011.[6] J. Gao, X. Bai, and W. T. Tsai, “Cloud-Testing:Issues, Challenges, Needs

and Practice,” Software Engineering: An International Journal, vol. 1,no. 1, pp. 9–23, 2011.

[7] G. Candea, S. Bucur, and C. Zamfir, “Automated Software Testingas a Service,” in Proceedings of the 1st ACM symposium on Cloudcomputing, 2010, pp. 155–160.

[8] L. Ciortea, C. Zamfir, S. Bucur, V. Chipounov, and G. Candea, “Cloud9:A Software Testing Service,” ACM SIGOPS Operating Systems Review,vol. 43, no. 4, pp. 5–10, 2010.

[9] T. King and A. Ganti, “Migrating Autonomic Self-Testing to the Cloud,”in Third International Conference on Software Testing, Verification, andValidation Workshops, 2010, pp. 438–443.

[10] J. Moreno, D. Kossmann, T. Kraska, and S. Loesing, “A TestingFramework for Cloud Storage Systems,” Master’s thesis, Swiss FederalInstitute of Technology Zurich, 2010.

11

[11] M. Oriol and F. Ullah, “YETI on the cloud,” in Third InternationalConference on Software Testing, Verification, and Validation Workshops,2010, pp. 434–437.

[12] L. Yu, W. Tsai, X. Chen, L. Liu, Y. Zhao, L. Tang, and W. Zhao, “Testingas a Service over Cloud,” in 2010 Fifth IEEE International Symposiumon Service Oriented System Engineering, 2010, pp. 181–188.

[13] Z. Ganon and I. E. Zilbershtein, “Cloud-based Performance Testingof Network Management Systems,” in Proceedings of IEEE 14th Int.Workshop Computer Aided Modeling and Design of CommunicationLinks and Networks (CAMAD ’09), 2009, pp. 1–6.

[14] S. Baride and K. Dutta, “A Cloud Based Software Testing Paradigmfor Mobile Aapplications,” ACM SIGSOFT Software Engineering Notes,vol. 36, no. 3, pp. 1–4, 2011.

[15] B. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears,“Benchmarking Cloud Serving Systems with YCSB,” in Proceedingsof the 1st ACM symposium on Cloud computing, 2010, pp. 143–154.

[16] W. Sobel, S. Subramanyam, A. Sucharitakul, J. Nguyen, H. Wong,S. Patil, A. Fox, and D. Patterson, “CloudStone: Multi-platform, Multi-language Benchmark and Measurement Tools for Web 2.0,” in Proceed-ings of Cloud Computing and Its Applications, 2008.

[17] C. Binnig, D. Kossmann, T. Kraska, and S. Loesing, “How is the WeatherTomorrow? Towards a Benchmark for the Cloud,” in Proceedings of theSecond International Workshop on Testing Database Systems, 2009, pp.9:1–9:6.

[18] D. Borthakur. The Hadoop Distributed File System: Architecture andDesign. [Online]. Available: http://hadoop.apache.org/common/docs/r0.18.0/hdfs design.pdf

[19] S. Ghemawat, H. Gobioff, and S. Leung, “The Google File System,”in ACM SIGOPS Operating Systems Review, vol. 37, no. 5, 2003, pp.29–43.

[20] W. Tsai, G. Qi, and X. Bai, “AgileSaaS: An agile SaaS developmentframework,” Arizona State University, Tempe, AZ, USA, 2011.

[21] W. Tsai, Q. Shao, and W. Li, “OIC: Ontology-based intelligent cus-tomization framework for SaaS,” in Proceedings of SOCA, 2010, pp.1–8.

[22] X. Bai, W. T. Tsai, Y. Gong, and J. Huang, “Massive Scalabilityand Dynamic Reconfiguration: Challenges to Software System TestingIntroduced by Service-Oriented Computing,” Communications of theCCF, vol. 9, pp. 35–41, 2010.

[23] C. Frye. Cloud Computing Creates Software Testing Challenges.[Online]. Available: http://http://searchcloudcomputing.techtarget.com/news/1355198/Cloud-computing-creates-software-testing-challenges

[24] T. Parveen and S. Tilley, “When to Migrate Software Testing tothe Cloud?” in Third International Conference on Software Testing,Verification, and Validation Workshops, 2010, pp. 424–427.

[25] L. Riungu, O. Taipale, and K. Smolander, “Research Issues for SoftwareTesting in the Cloud,” in 2010 IEEE Second International Conferenceon Cloud Computing Technology and Science (CloudCom), 2010, pp.557–564.

[26] K. Yim, D. Hreczany, and R. Iyer, “HTAF: Hybrid Testing AutomationFramework to Leverage Local and Global Computing Resources,” inComputational Science and Its Applications - ICCSA 2011. SpringerBerlin / Heidelberg, 2011, vol. 6784, pp. 479–494. [Online]. Available:http://dx.doi.org/10.1007/978-3-642-21931-3 37

[27] Taobao. [Online]. Available: http://www.Taobao.com/[28] W. Tsai, Y. Huang, Q. Shao, and M. Barrett, “Testing the scalability of

SaaS applications,” in IEEE RTSOAA ’11, 2011.[29] W. Tsai, Y. Huang, Q. Shao, and X. Bai, “Data partitioning and

redundancy management for robust multi-tenancy SaaS,” Int J SoftwareInformatics, vol. 4, no. 4, 2010.

[30] W. Tsai, X. Zhou, Y. Chen, and X. Bai, “On Testing and EvaluatingService-Oriented Software,” Computer, vol. 41, no. 8, pp. 40–46, 2008.

[31] W. Tsai, W. Li, H. Sarjoughian, and Q. Shao, “SimSaaS: simulationsoftware-as-a-service,” in Proceedings of the 44th Annual SimulationSymposium, 2011, pp. 77–86.

[32] P. Brebner and A. Liu, “Performance and Cost Assessment of CloudServices,” in Proceedings of the 2010 international conference onService-oriented computing, 2011, pp. 39–50.

[33] A. Li, X. Yang, S. Kandula, and M. Zhang, “CloudCmp: ComparingPublic Cloud Providers,” in Proceedings of the 10th annual conferenceon Internet measurement, 2010, pp. 1–14.

[34] W. Tsai, R. Paul, L. Yu, A. Saimi, and Z. Cao, “Scenario-basedWeb Services Testing with Distributed Agents,” IEICE Transactions onInformation and Systems, vol. 86, no. 10, pp. 2130–2144, 2003.

[35] W. T. Tsai, Y. Chen, R. Paul, N. Liao, and H. Huang, “Cooperative andGroup Testing in Verification of Dynamic Composite Web Services,” inProceedings of 28th Annual Int. Computer Software and ApplicationsConf. (COMPSAC 2004), vol. 2, 2004, pp. 170–173.

[36] W. T. Tsai, R. Paul, Z. Cao, L. Yu, and A. Saimi, “Verification ofWeb Services using an enhanced UDDI server,” in Proceedings of 8thInt. Workshop Object-Oriented Real-Time Dependable Systems (WORDS2003), 2003, pp. 131–138.

[37] X. Bai, Z. Cao, and Y. Chen, “Design of a Trustworthy ServiceBroker and Dependence-based Progressive Group Testing,” InternationalJournal of Simulation and Process Modelling, vol. 3, no. 1, pp. 66–79,2007.

[38] X. Bai, Y. Wang, G. Dai, W.-T. Tsai, and Y. Chen, “A Frameworkfor Contract-Based Collaborative Verification and Validation of WebServices,” in Component-Based Software Engineering. Springer Berlin/ Heidelberg, 2007, vol. 4608, pp. 258–273.

[39] L. Riungu, O. Taipale, and K. Smolander, “Software Testing as an OnlineService: Observations from Practice,” in Third International Conferenceon Software Testing, Verification, and Validation Workshops, 2010, pp.418–423.

[40] Tieto. [Online]. Available: http://www.tieto.com/[41] Sogeti. [Online]. Available: http://www.sogeti.com/testing[42] T. King, A. Ramirez, R. Cruz, and P. Clarke, “An Integrated Self-Testing

Framework for Autonomic Computing Systems,” Journal of Computers,vol. 2, no. 9, pp. 237–249, 2007.

[43] T. King, “A Self-Testing Approach for Autonomic Software,” Ph.D.dissertation, Florida International University, 2010.

[44] R. Calheiros, R. Ranjan, A. Beloglazov, C. De Rose, and R. Buyya,“CloudSim: a Toolkit for Modeling and Simulation of Cloud ComputingEnvironments and Evaluation of Resource Provisioning Algorithms,”Software: Practice and Experience, vol. 41, no. 1, pp. 23–50, 2011.

[45] T. Banzai, H. Koizumi, R. Kanbayashi, T. Imada, T. Hanawa, andM. Sato, “D-Cloud: Design of a Software Testing Environment forReliable Distributed Systems using Cloud Computing Technology,” inProceedings of the 2010 10th IEEE/ACM International Conference onCluster, Cloud and Grid Computing, 2010, pp. 631–636.

[46] P. Joshi, H. S. Gunawi, and K. Sen, “PreFail: a Programmable Tool forMultiple-Failure Injection,” in Proceedings of the 2011 ACM interna-tional conference on Object oriented programming systems languagesand applications, 2011, pp. 171–188.

[47] H. S. Gunawi, T. Do, P. Joshi, P. Alvaro, J. Yun, J.-s. Oh, J. M.Hellerstein, A. C. Arpaci-Dusseau, R. H. Arpaci-Dusseau, K. Sen, andD. Borthakur, “FATE and DESTINI: A Framework for Cloud RecoveryTesting,” EECS Department, University of California, Berkeley, Tech.Rep. UCB/EECS-2010-127, Sep 2010. [Online]. Available: http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-127.html

[48] J. Penix, “Panel presentation,” in 33rd International Conference onSoftware Engineering. Google, 2011.

[49] A. Duarte, W. Cirne, F. Brasileiro, P. Duarte, and L. Machado, “Usingthe Computational Grid to Speed Up Software Testing,” in Proceedingsof 19th Brazilian symposium on software engineering, 2005.

[50] A. Duarte, W. Cirne, F. Brasileiro, and P. Machado, “GridUnit: SoftwareTesting on the Grid,” in Proceedings of the 28th international conferenceon Software engineering, 2006, pp. 779–782.

[51] A. Duarte, G. Wagner, F. Brasileiro, and W. Cirne, “Multi-EnvironmentSoftware Testing on the Grid,” in Proceedings of the 2006 workshopon Parallel and distributed systems: testing and debugging, 2006, pp.61–68.

[52] A. Lastovetsky, “Parallel Testing of Distributed Software,” Informationand Software Technology, vol. 47, no. 10, pp. 657–662, 2005.

[53] T. Parveen, S. Tilley, N. Daley, and P. Morales, “Towards a DistributedExecution Framework for JUnit Test Cases,” in IEEE InternationalConference on Software Maintenance, 2009., sept. 2009, pp. 425 –428.

[54] R. L. Grossman, Y. Gu, M. Sabala, C. Bennett, J. Seidman, andJ. Mambretti, “The Open Cloud Testbed: A Wide Area Testbed for CloudComputing Utilizing High Performance Network Services,” CoRR, vol.abs/0907.4810, 2009.

[55] C. Bennett, R. L. Grossman, D. Locke, J. Seidman, and S. Vejcik,“MalStone: Towards a Benchmark for Analytics on Large Data Clouds,”in Proceedings of the 16th ACM International Conference on KnowledgeDiscovery and Data mining (SIGKDD ’10), 2010, pp. 145–152.

[56] SOASTA. [Online]. Available: http://www.SOASTA.com/[57] ITKO. [Online]. Available: http://www.itko.com/[58] Cloud Testing. [Online]. Available: http://www.CloudTesting.com/

12