manage performance by testing early and often

A Publication

Manage Performance ByTesting Early and Often

Building’s Not in the Cards?Minimize Risk When Buying

MMoottiivvaattee aa TTeeaamm WWiitthh SSoommee SSppaaddee WWoorrkk

The Foundation Of Good Testing

BESTPRACTICES:Change

Management

VOLUME 5 • ISSUE 3 • MARCH 2008 • $8.95 • www.stpmag.com

Featured

in this issue

Software Test & Performance MARCH 2008

Testing Early and Often Can Help

Prevent Web Applications From Crumbling

Under Pressure Like a House of Cards

Photograph by Alexey Kashin

to the discovery that the architecturedoesn’t scale well, at a time when it’s toolate to do anything about it. The earlier you start load testing dur-

ing the application life cycle, the earlierthe underlying infrastructure’s softwaredefects, design flaws and bottleneckswill be found. A methodology thatestablishes quality and perform-ance-related activities early in theapplication life cycle helps to miti-

gate the risk of project failure, reducesoverall project costs, and increasesthe application’s quality and per-formance. Despite the well-known fact that the

cost of issue correction increases ineach downstream phase, project teamsoften wait until the end of developmentto set up and integrate load testing.While it’s good practice to perform end-to-end load tests on an applicationshortly before going live with a new orupdated product to prove that the appli-cation performs and scales as expected,if the results don’t meet the expecta-tions, you can’t do much to salvagethe project at such a late stage.

Usually these activities are lim-ited to tuning the hardware orsoftware configurations, andoften, as a last resort, throw-ing more or faster hardware

at the problem. If neither ofthese activities is successful, it’s

back to development to find theroot cause of the problem in the

application code. In the worst-case

scenario, the core architecture isn’t suit-ed for scalability and performance, andyou have to redo core parts of the appli-cation.With the emergence of application

technologies such as SOA and the Web,you also need to adapt your load testingprocess to the new requirements andchallenges that new technologies bring.

What to Test EarlyDecisions about infrastructure andapplication architecture are usuallydone early in the application life cycle.Both have a strong impact on applica-tion design, implementation and opera-tion. Reverting infrastructure and archi-tectural decisions until late in the devel-opment process can be painful. If youwant to prove your architectural con-cept or different architectural alterna-tives, you often start with a prototypethat implements your major concepts.By applying the prototype to theplanned hardware/software infrastruc-ture early, you can test how well the cho-sen architecture is suited to the infra-structure it will run on. Component load testing can be done

against business logic components assoon as they’re ready, and without theneed of a fully developed UI or othersoftware components. With SOA-com-ponent load testing, early load testingbecomes even more critical. The earlier you start developing load

Ernst Ambichl is chief scientist at Borland.

By Ernst Ambichl

Many organizations wait until the end stages of applicationdevelopment to perform load testing. This practice often leads

MARCH 2008 www.stpmag.com


tests for components of your system, theearlier you can start to find regressionsof performance when these compo-nents change. By integrating load testsas part of your regression test suite, youcan avoid detecting performance prob-lems long after they are introduced.

Focus on Infrastructure AndArchitectureSome could argue that testing with afocus on infrastructure is a classic bench-marking domain and doesn’t have muchto do with load testing an application.Basic hardware/software infrastructuressuch as network switches, Web servers,firewalls, application servers, DBMSs ormessaging middleware are already wellknown and mature. Often you can evenfind standard benchmarks for mostparts of your infrastructure. But be care-ful: Standard benchmarks have downsides, as they:• Ignore your application’s individ-ual structure and workload• Exist only for discrete infrastruc-ture parts, not for the specific com-bination of infrastructure parts thatmake up your application infra-structure• Usually aren’t available for newapplication technologiesThe benefits of early load tests of

parts of the application within the tar-get infrastructure are:• Early capacity assessment of theapplication infrastructure• Early check for scalability of yourarchitecture

• Early identification of relevant per-formance indicators and configu-ration settings• Early information for infrastruc-ture tuningBy load testing the infrastructure

early, you’re able to learn about the con-figuration settings and metrics that arerelevant to performance. Knowledge ofthe relevant performance indicatorsand configuration settings is highlyvaluable, not only for later testing andtuning, but also for setting up the rightset of infrastructure monitors for yourlive application. For this kind of test, especially with-

in large IT organizations, two or more

groups often need to cooperate. Thefirst is IT operations, which is responsi-ble for the infrastructure the applica-tion will run on in production. The sec-ond is the development team, which isresponsible for the application and thescalability of the architecture. A dedi-cated performance team (perhaps partof the QA group, development or IT)can greatly facilitate these efforts andact as the bridging group betweendevelopment and IT.

Load Testing a UI PrototypeLet’s assume you need to build a highlyscalable architecture for a Web-basedapplication with a high standard forusability and speed of the user inter-face. The application will be deliveredto all locations using the existing corpo-rate intranet. As part of the application develop-

ment, you’re designing a new HTML/UIframework including third-party AJAXcomponents. You want to ensure early inthe process that the existing networkinfrastructure—as well as the company’sstandard Web server infrastructure—willdeliver the required performance andresponsiveness for the new application.To accomplish this goal, you’ll

load test a prototype of the applica-tion UI using a new UI framework.The prototype includes only a smallsubset of the planned application’sUI logic and is already using theframework’s UI controls.Since you don’t yet have the business

logic in place, you’re emulating thebusiness logic as “hard-coded” parts

WEB-APP LOAD TESTING

Web Server

UI Prototype

Web Server

UI Prototype

DatabaseServer

App. Server

App. Server

App. Server

Load Test Load Test Load Test

Intranet LoadBalancer

FIG. 1: UI PROTOTYPE

Load Test

Web Server(UI)UI

Component 1

DatabaseServer

Data AccessComponent 1

Data AccessComponent 2App. ServerWeb Server


UI Component2...n

App. Server(Business Logic)

BLComponent 1

BL Component2...n

App. Server

FIG. 2: WITH FULL-TIER PROTOTYPE


inside the UI prototype (see Figure 1).Having this UI prototype in place, youalready can test how well the UI frame-work performs on the planned infra-structure. Stepwise, you can do testsagainst a single Web server, load-bal-anced Web servers, and across the cor-porate intranet. The idea of this type of early testing

is to determine whether some UI com-ponents might not be suitable for yourintranet’s network latency, for exam-ple, or are consuming too much mem-ory on the Web server to scale well.This can and should be done before youbase your whole application on thesecomponents.

Load Testing a ‘Full-Tier’ PrototypeIn another scenario, you may want toverify that your application’s planneddistributed architecture actually runsand scales as expected on the infra-structure chosen for deployment. To accomplish this, you can use anoth-

er prototype of your application for loadtesting. The prototype needs only to con-tain a small subset of the real application;it doesn’t need to be complete in terms ofthe functionality it will deliver. It’s impor-tant that the prototype allows you to testagainst a small set of use cases that alreadytouch all tiers of the application using theproposed distributed architecture for the

application. For a typical Web-based appli-cation, these tiers are Web server, applica-tion server, database server and externalproviders, if applicable.Load tests using a “full-tier” prototype

on the target infrastructure can help you

to get answers to the following questions:What is the viability of the infrastruc-

ture? With a small subset of functionali-ty touching all tiers, you can determinewhether the different infrastructurecomponents can work together to deliv-er acceptable performance.What are the design flaws that result in

bottlenecks? Your software architecture’sscalability and performance, whichdefine how the different parts of theinfrastructure will work together, also

can be verified by a prototype that imple-ments the architectural framework usedby the application’s components. Evendifferent design alternatives (if availableas prototypes) can be tested for per-formance, scalability and reliability.

Are there incompatibilities between thedifferent technologies used? Early detectionof incompatible parts of the infrastruc-ture can be accomplished when a ”full-tier“ prototype (as in Figure 2) exists thattouches all tiers of the application. I ran into such a problem when test-

ing the servlet engine used in certain

Web-based products (in our case,Tomcat) with one of the Web servers weneeded to support (in this case, IIS).The problems occurred only under loadconditions. Testing our application onTomcat without IIS as the servlet con-tainer never showed similar problems.

Component Load TestingModern multi-user applications are usu-ally built with frameworks that allow formodular, componentized design andarchitecture. Componentizing yourapplication is the first and most impor-tant step to enable you to begin your test-ing earlier, when certain individual com-ponents are getting ready. Especiallywith components that are accessibleremotely and/or concurrently from mul-tiple clients, functional testing should beexpanded to component load testing assoon as possible. Often, functional tests for compo-

nents are already completed by devel-opers with the help of standard unit-testing frameworks like JUnit or NUnit.With the right tools in place, it’s only asmall step to extend these JUnit/NUnit-based tests to small component loadtests. My experience has shown thatmany elusive performance problemscan easily be found when exposing


UI ComponentLoad Test

Web Server(User Interface)

UIComponent 1

DatabaseServer

Data AccessComponent 1

Data AccessComponent 2App. ServerWeb Server


UI Component2...n

App. Server(Business Logic)

BLComponent 1

BL Component2...n

App. Server

BL ComponentLoad Test

DA ComponentLoad Test

FIG. 3: COMPONENTS IN A MULTI-TIER APP

App 1 App 2

Service A Service B

App 2 App 3

Consumers

Services

Providers

Services Framework

FIG. 4: EXEMPLARY SOA


remote and multi-instance componentsto moderate load conditions. What components should be load tested?

It’s important to concentrate your loadtesting on remote components and/orcomponents that are used concurrently

by multiple clients (Figure 3). From atechnology view, these are componentsthat expose their functionality via inter-faces like RMI, RCP, CORBA and(D)COM, .NET Remoting and, ofcourse, Web services. (SOA will be han-dled in more detail later in this article).I also typically include SQL-based dataaccess components in my roster of can-didates for load testing. Database per-formance remains one of the criticalelements in a distributed applicationarchitecture.With the evolution of SOA technolo-

gy, there comes the need to adapt yourload testing approaches to SOA’s newrequirements and challenges. First, let’sdefine SOA. As defined by XML.com:

SOA is an architectural style whose goalis to achieve loose coupling among interact-ing software agents. A service is a unit ofwork done by a service provider to achievedesired end results for a service consumer.Both provider and consumer are roles playedby software agents on behalf of their owners.

Loose coupling is the magic phrase inthis definition, and is the enabling fac-tor that allows us to start testing assoon as the services contract (or inter-face) between the software agents isdefined. In theory, SOA architecturesare well suited for applying “testingearly” principles, as services should bebuilt with a high degree of autonomy

and with minimal dependencies to theenvironment they run in. The termsWeb services and SOAP are purposelyomitted from this definition of SOA,which is a much broader architecturalconcept.

Factors that influence your load test-ing approach for SOA applicationsinclude:A decreased predictability of use. The

agility SOA provides for building newapplications based on existing servicesleads to more unpredictable usage pat-terns and workloads compared to classic“n-tier” applications. As a serviceprovider, you may not know who mightultimately consume your service at thetime you’re developing it. Hence, test-

ing early for the scalability of your serv-ices is important.Increased complexity. Since applica-

tions based on SOA often consume mul-tiple services (such as composite appli-cations), the services call chain to fulfillan application request can get quitelong, especially when using services thatthemselves consume services. Availability of service providers comes

late in the application life cycle. This isespecially true when your applicationdepends on a service provided by athird party, such as a business partner. Ifthis is the case, you need to ensure thatyou can test your application when notall service providers are available. Availability of service consumers comes

late in the application life cycle. You needto ensure that you can test your servicebefore the service consumers begin con-suming it.SOA facilitates distributed development.

Often, distributed teams or even differ-ent organizations work on serviceproviders and service consumers. Toavoid finger pointing when perform-ance problems are found during systemtesting, it’s important to test the servicesin isolation.Complex root-cause analysis. Due to

the complexity and the distributed

nature of SOA applications, identifyingthe root cause of SOA performanceproblems is harder than in traditional n-tier systems. The earlier you detectproblems in isolation, the easier it willbe to fix them.Impact of change increase. SOA-based

applications typically evolve over time

Consumers

Services

Providers

Services Test Framework

App 1

Service A

App 2

Simulator App 2

Service B

App 3

FIG. 5: TESTING SERVICE B IN TEST FRAMEWORK

Consumers

Services

Providers

Services Test Framework

SimulatorApp 1

Service A

App 2

App 2

MockService B

App 3

FIG. 6: TESTING SERVICE A IN TEST FRAMEWORK



and change constantly by adding newapplications on top of existing servicesand new providers for existing services,or by creating new services on top ofexisting services. A simple change in aservice can impact multiple ap pli cationsconsuming this service. This also intro-duces the need to constantly retest andcarefully monitor your services whenev-er you change the service. Different types of load tests can be

done in different stages of system devel-opment. This depends on your testingstrategy and the availability of your SOAcomponents.

Isolation Load Test Load testing should be done before youintegrate the service with your con-sumer applications or integrate it intothe services framework.Isolation load tests are the “cheap-

est” load tests because you can do themwithout having the whole infrastructurein place. In addition, you typically won’tneed a lot of virtual users to test a singleservice behavior under load conditions(synthetic workload in contrast to realis-tic loads for end-to-end load tests). This makes such tests good candi-

dates for regression testing. As soon asthe service changes, you can run isola-tion load tests to check if the behaviorof the service has changed under loadconditions. Often, a fix of a defect relat-ed to the component’s functionalbehavior just introduces a degradationof performance.

Testing Without a ConsumerWhen developing services, you oftenhave no access to the client application,or the application isn’t ready for testing.Also, if a service is consumed by multi-ple applications, you won’t reach suffi-cient test coverage when testing is donewith only one client application. In the absence of a client applica-

tion, traditional test-script creationtechniques such as recording clientinteractions aren’t possible. So, even ifyou aren’t working in an agile develop-ment shop, developing functional testsas part of service implementation is agood practice. You might even say it’s anecessity. These functional tests can alsobe reused for load testing.

Testing Without a Services TestFrameworkDevelopers usually don’t work within thedeployment infrastructure. They typically

use a small subset of the deploymentinfrastructure or are developing within atest framework (Figure 5) to execute anddebug their work, which is different fromthe target framework.

Conducting small load tests as part ofdeveloper activities (which can most oftenbe directly derived from unit tests) with-out the burden to set up big infrastruc-tures for testing helps to move load test-ing nearer to the developer and earlierinto the application life cycle. You can dosmall load tests with your nightly develop-er builds, which can signal changes in per-formance as soon as possible.

Testing Without a ProviderAlthough SOA fosters loose coupling

between components and thereforeminimizes dependencies betweencomponents, real dependency alwaysexists and can’t be reduced. Realdependency is the set of features or serv-ices that a system consumes fromother systems. So how can you test a service that

depends on another service before thatservice is available? In object-orientedprogramming, you use mock objects,which are simulated objects that mimicthe behavior of real objects in con-trolled ways. Similarly, you can createmock services for services that aren’t avail-able or that you want to factor out ofyour test (Figure 6). Factoring out services by emulating

their behavior through mock servicesoffers the advantage of allowing testersto control the behavior of the emulat-ed service. This allows you to easilybuild load testing scenarios in whichyou emulate the misbehavior ofdependent services such as servicecalls that are tardy, time out or returnincorrect data.

Integration Load Test: ServicesFramework Integration Test After isolation testing, in which youtest the service in your services testframework, you can replace your testframework with the services frame-work used for deployment. This letsyou test how well your service works inthe target environment. While thisusually adds the work of deployingyour services and providing a test envi-ronment with the target servicesframework, you can reuse the tests

Consumers

Services

Providers

Services Framework

App 1

Service A

App 2

SimulatorApp 2

Service B

App 3

FIG. 7: TESTING SERVICE B IN SERVICES FRAMEWORK

•You can

create mock

services for

services that

aren’t available.

•



you’ve already written. You won’t perform integration load

testing (Figures 7 and 8) as often asyour isolation tests (as with every check-in). But they should be done on a regu-lar basis, such as every time develop-ment passes a build to QA. This ensuresthat QA isn’t wasting time on testingbuilds that don’t pass the performancecriteria checked by your service frame-work integration tests. You’ll also most likely increase work-

load by testing the scalability of the serv-ices framework in combination withyour service. Extending your isolationtests to services framework integrationtests helps to answers questions like: • How does the service scale withinthe services framework? • How much overhead is the frame-work adding to the service? • Does the framework correctly han-dle the life cycle of the service? • What is the payload for enablingsecurity?

Integration Load Test: ServiceInteraction TestAs important as it is to test services in iso-lation as early as possible to detect per-formance problems, it’s equally crucialto test the services in combination todetect problems related to their interac-tion with other services. No isolation testwill ever give you absolute certainty thatyour system will pass even the most sim-ple integration test, even if your isola-tion tests cover almost all your code.This is especially true of the per-

formance, scalability and stabilityaspects of your SOA-based application.

Establishing integration load tests assoon as two interacting services areavailable helps to find integration prob-lems early. Rerunning integration loadtests (regression testing) as soon asdependent services change helps toidentify performance degradations atthe time they’re introduced. With service interaction tests, you’ll

extend the test infrastructure to betterreflect the target system and extend theworkload patterns to more realistic sce-narios (Figure 9). Also, your test scriptswill need to reflect that they’re testingthe integration aspect and not the isola-tion aspect of the services.

System Load Test: End-to-End TestLoosely coupled architectural implemen-tations such as those of an SOA create

additional complexities with end-to-endload testing (Figure 10). Services thatshare common a infrastructure or plat-form require coordinated load testing totruly replicate production-like states.Providing the test infrastructure, cre-

ating and setting up these tests, identi-fying production-like workloads, analyz-ing results and finding the root causefor performance problems is even moredifficult when compared with more tra-ditional n-tier systems. Everything that can be done to iden-

tify possible performance problemsbefore you actually perform your systemload test helps to lower the cost of fixingperformance problems and mitigatethe risk of project failure due to wrongarchitectural decisions you can’t redo atthe end.

Regression Load Test Every change in a system might not onlyintroduce regressions in terms of func-tionality, but also in terms of perform-ance, scalability and stability. Focusingonly on functional test automation toaddress regressions leaves performanceproblems undetected until final system-load tests. Integrating load tests as part of

your regression test suite avoids thedanger of detecting performanceproblems too long after they are intro-duced. Because it’s expensive to set upand integrate load testing into a testautomation process, not all types ofload tests are suited for regressionload tests. Some good candidates forregression load testing are: Isolation load tests. Such load tests can

Consumers

Services

Providers

Services Framework

SimulatorApp 1

Service A

App 2

App 2

MockService B

App 3

FIG. 8: TESTING SERVICE A IN SERVICE FRAMEWORK

SimulatorApp 1

SimulatorApp 2

Service A Service B

App 2 App 3

Consumers

Services

Providers

Services Framework

FIG. 9: SERVICE A AND B TESTING IN SERVICES FRAMEWORK



be done on a regular basis (rangingfrom tests per check-in to nightly sched-uled builds).Services framework integration tests.

Isolation load tests also should be exe-cuted regularly in the target servicesframework. Functional tests have simple success

conditions (usually pass/fail per testcase based on assertions in your testscript that make it easy to automateyour tests’ results analysis). This isn’tthe case for load tests, which usuallyrequire analysis of multiple metrics todetermine a pass or fail status. To auto-mate that process and flag “failed”load tests, you can use the followingmethods: • Compare performance-relevantmetrics such as response times,throughput rates and resourceconsumption to defined baselines(static thresholds) that you’ve setup for each individual load test.• Compare the change/delta of per-formance-relevant metrics to his-toric measurements of the sametest. In this case, you don’t need to

set up thresholds for each test.• Both methods have their advan-tages and disadvantages. Decidecase-by-case which one best suitsyour requirements.

Testing in Production: Application MonitoringLoad testing SOA applications underreal-life conditions is extremely com-plex (Figure 11). It’s therefore valu-able to extend your testing approachto the production phase of your appli-cation to gather feedback for your test-ing. Two techniques extend testing into

production andboth provide valu-able feedbackabout the accuracyof your load testing:

Active servicemonitors. By reusingexisting load-test-ing scripts and exe-cuting them on thelive system, you getan accurate pictureof how the per-formance of thesystem under testand the live systemcompares. Leadingload-testing toolshave integrationswith applicationperformance monitoring frameworks,which makes it easy to reuse your loadtesting assets for active monitors.

System and in-depth monitors. By usingsystem monitoring techniques, you cankeep track of services usage patterns.Input/output data can be monitored within-depth monitoring techniques. Results

for service execution counts and inputcan be fed back into the testing process tocreate more accurate workloads.

Early and IntegratedLoad testing can be done in earlystages of development and applied tovarious components of an applicationbefore the final end-to-end load test.Early infrastructure load tests can mit-igate the risk of investing in a specificinfrastructure that doesn’t scale orperform as needed. By using prototypes of the applica-

tion for load testing, you can proofarchitectural concepts before you base

your whole application code on theseconcepts. Component load tests helpto isolate performance problemsearly—before they become difficult tofind and expensive to fix.The integration of load testing

throughout the development processhas never been more important as,due to increasing complexity, we faceless predictability of usage and moredynamic changes in applications.Because of SOA’s loosely couplednature, unit and component testingapproaches can be adopted for loadtesting, delivering early results aboutthe performance and scalability ofyour services-based components.Integrating load tests into your regu-

lar regression testing suite will help youto detect performance regressions assoon as they’re introduced. You canextend your testing approach to the pro-duction phase of your application byreusing load testing assets for applicationmonitoring to gather feedback aboutreal usage and real performance. For optimal success, load testing

should be conducted throughout theproject life cycle, started soon after anapplication is conceived and continueduntil it’s retired. � REFERENCES• “What Is Service-Oriented Architecture?” Hao He, Sept. 30, 2003, O’Reilly xml.com,www.xml.com/pub/a/ws/2003/09/30/soa.html

• W3.org, Web Services Glossaryhttp://dev.w3.org/2002/ws/arch/glossary/wsa-glossary.html

• “Best Practices for Web Application Deployment”keynote, Ernst Ambichl, Segue Software, TotalPerformance Management Symposium, Mar. 18,2004

• “Choosing a Load Testing Strategy” Whitepaper,Ernst Embichl, Segue Software, 2005

• “Adjusting Testing for SOA,” David S. Linthicum,SD Times, Aug. 15, 2007

App 1 App 2

Service A Service B

App 2 App 3

Consumers

Services

Services Framework

Real Users Real Users

Active MonitorApp 1

Providers

System

Monitor

Service Performance Metrics(e.g. service response time)

Active MonitorApp 2

Service System Metrics(e.g. service execution count)

Service In-DepthMetrics(e.g. service input data)

In-depth Monitoring

FIG. 11: TESTING IN PRODUCTION


SimulatorApp 1

SimulatorApp 2

Service A Service B

App 2 App 3

Consumers

Services

Providers

Services Framework

SimulatorApp 1

SimulatorApp 2

FIG. 10: END-TO-END TESTING

manage performance by testing early and often

Documents

early load

webapp load testing

lim component load testing

end load tests

load tests early information

server application

new application

testing dur