understanding the impact of running wan emulation with load testing

Understanding the Impact of Running WAN Emulation with Load

Testing

A Shunra Software White Paper July 2, 2008

of 13

Understanding the Impact of Running WAN Emulation with Load Testing

.

© 2008 Shunra Software Ltd.

Introduction Investment in pre-deployment performance testing has become a widely adopted standard used to verify the performance and reliability characteristics of core business applications. As reflected by the robust capabilities of industry-standard load testing tools such as HP LoadRunner, the proper setup and execution of a load test is a complex undertaking. Significant effort goes into each load testing project. Organizations that are disciplined enough to properly implement performance testing into their application development lifecycle have gained significant business benefits from finding issues and properly sizing capacity prior to application roll-out. Load testing has traditionally been focused on identification of software or hardware capacity problems, without taking into account the impact of the network on application performance. In some cases the network is not an impediment to application deployment. Unfortunately, in many cases applications fail to perform to expected levels, despite load testing prior to roll-out. A load test environment that does not reflect all relevant real-world conditions can not provide an accurate projection of how the application will perform once deployed. When this happens, the analysis and decisions leading up to approval to deploy the application may be misguided. Increasingly, many pre-deployment testing labs have found that it is critical to incorporate the impact of WAN on application load testing. This white paper describes the importance of assessing network impact on application performance and the value that is obtained by integrating WAN emulation into load testing. Organizations typically have used one of four approaches to assessing the impact of the network: Approach 1 – Ignore network impact entirely Focus on the scalability of the datacenter infrastructure and target response time and throughput targets based solely on local performance at the datacenter, ignoring the WAN.

Approach 2 – Deploy agents to generate load Incorporate network impact on the application by deploying distributed agents for load generation. Approach 3 – Estimate impact of the network Separately estimate the impact of the network on an application being deployed and manually factor that into result reports and deployment readiness findings Approach 4 – Incorporate WAN emulation Incorporate WAN emulation as an integral part of application deployment automated testing

of 13


.


There are problems inherent in the first three approaches, and benefits for many organizations that are using the 4th approach, which includes a WAN emulation solution within their load testing environment. Weighing the Pro’s & Con’s Approach 1 – Ignore network impact entirely This has been a common practice, despite shortcomings that this paper highlights. Ignoring the impact of the network poses a substantial risk. Applications may be deployed without fully understanding their performance characteristics. Deployments may fail due to unidentified scalability issues that were entirely overlooked by load testing focused on the server infrastructure. Approach 2 – Deploy agents to generate load To accurately assess end user transaction response times, it seems clear that some incorporation of network performance needs to be included into test results. However, without a WAN emulation solution such as Shunra Virtual Enterprise (VE), the only effective way to accomplish this is to place physical load generators in remote end-user branch locations. This is both costly and inaccurate. First, the cost of deploying, maintaining, and administering remote generators is an obvious negative. Second, since a distributed load test is typically run during off-hours when the network is lightly loaded by other traffic, it is unlikely that the test will appropriately reflect performance that actual users will experience during working hours. Additionally, running a load test over the production network runs the risk of impacting other production applications and users. Further, in most cases it is not possible to test applications delivered over the internet to business partners and external customers using this approach. As a result, few organizations have chosen to select this approach. Approach 3 – Estimate impact of the network This approach is based on testing a single user running transactions at each remote site and measuring the overall performance. This provides partial insight into the impact of the network, but totally ignores the interrelationships that network performance has on the application logic. In addition, since this approach entails a separate activity from load testing, it is typically a manual process that is labor intensive and not repeatable. Whenever the application logic or infrastructure configuration changes, there is an ongoing risk of encountering an unanticipated adverse impact and unhappy users. Approach 4 – Incorporate WAN emulation This approach incorporates WAN emulation as an integral part of application deployment automated testing. Shunra’s customers have adopted this approach and are vocal advocates for the resulting benefits they have provided to their companies. With the most recent versions of Shunra VE, integration into load testing scenarios has been made even easier to accomplish.

of 13


.


Manage Your Risk with WAN Emulation Rigorous performance testing and optimization is a critical factor in the successful delivery of key business applications. Yet frequently the performance of deployed applications doesn’t live up to business requirements or end-user expectations despite load testing. One reason behind these unpleasant “surprises” is the fact that most performance staging labs only test the application with local users (in a local area network (LAN) environment), while the fully deployed application is used by a variety of end-users, some local and others accessing the application remotely over different network links. The different network conditions that exist between end-users and application servers have a tremendous effect on the overall performance that remote end-users experience. This deviation from performance test results obtained in the lab is further exacerbated in the case of SOA composite applications where various services may reside in different geographical locations, each posing their own unique set of network conditions.

Ensuring Application Performance to Remote End-Users A simple example of such a deviation can be seen from the results of the following test conducted at a major North American financial institution. This test included both local users, (i.e. users interacting with the database over a local, unrestricted link), as well as remote end-users located in San Antonio, connecting to the datacenter in Tuttle, Ohio over a wide area network link. The wide area network and end-user load were created using emulation technologies. The emulated wide area link was a T1 connection, which accommodated the latency (geographic distance) between Columbus, OH and San Antonio, TX as well as the expected packet loss. Furthermore, the link was 30% utilized before the test in order to replicate the exact conditions in the production environment where the same network link supported other applications. This left 70% of the network link available for our financial application in the test below. Location Max Throughput (kbps) &

Bandwidth Utilization (%) Scripts

In Out Name Max Load

Transaction Response Time

Local users in Tuttle, OH with local datacenter (LAN)

3838.29 (0%)

658.84 (0%) Financial Transaction A

25 50.2

Remote users in San Antonio, TX

1168.66 (100%)

656.86 (80%) Financial Transaction A

25 80.48

Figure 1

of 13


.


The results in Figure 1 show a substantial deviation from the performance of the local end-users on the local area network to that experienced by end-users at the remote location on the wide area network link. While increasing the load of local end-users did not seem to strain the application server or backbone network, the same gradual load increase from the remote locations negatively affected performance due to overwhelming strain on the network and the application servers as well. Now let’s drill down further to understand the end-user behavior at each location and the causes of performance problems over the wide area network. The graph in Figure 2 below shows the performance of end-user transactions over time (red line), over a local network link, as the number of end-user transaction scripts running increases. In this case, the transaction response time over the local network increases only slightly, reaching 60 seconds at 25 end-user transactions (i.e. 33% increase from the starting point of 7 users at 45 seconds). This means that while the users are running over a local link, the system can scale and support additional end-user load.

Figure 2. Results from testing end- user load over a local network. However, when we ran the same test over an emulated wide area network the results were drastically different.

of 13


.


The graph in Figure 3 below correlates end-user load with wide area network utilization. In this case, we can see that with as few as 16 end-user transactions (red line) running over an emulated wide area network, the available network bandwidth (light orange line) is already 90% utilized. Furthermore, at 25 end-user transactions the response time is up to 100 seconds (an increase of 66% over the response time over the local network link). Furthermore, nine minutes into the test, the application started generating errors (this is not shown on the chart). Obviously, this application is not ready for deployment to remote end-users over the wide area network.

Figure 3. Results from testing end-user load over a wide area network.

of 13


.


The graphs in Figure 4a and 4b below show how HP Web Page Diagnostics help pinpoint the extent to which particular web components are impacted by the network.

Figure 4a - Web Page Diagnostics with 50ms of latency

Figure 4b - Web Page Diagnostics with 100ms of latency The graph in Figure 5 (next page) shows Shunra VE can deliver a breakdown of transaction performance from a single Virtual User into Client, Network, and Server tiers during a load test. Like the Web Page Diagnostics, Shunra analysis can be used to analyze web applications. However Shunra’s technology also enables analysis of applications using other protocols such as SQL and RPC. Targeted analysis and tuning can be quickly accomplished for transactions that perform poorly over the WAN.

of 13


.


Figure 5. Shunra analysis for an application’s Customer Summary transaction By incorporating automatic analysis into load test scripts, Shunra makes it easy to see the relative impact of the network and server on transaction performance.

It’s Not Just the Network The tests above clearly show that production wide area network conditions can significantly impact the performance of distributed applications, but it doesn’t tell the whole story. In addition to the delay from the constrained network resources, the stress on the servers also increases since more resources are needed to support remote end-users. Sessions are open longer, OS resources are occupied for more time and more concurrent threads are needed. None of these factors are taken into account when testing and optimizing the performance only for local end-users on the local area network. Thus, remote business users are often unpleasantly “surprised” by poorly performing application despite load testing efforts that failed to address the impact of the WAN. Shunra’s best practices implementation guidelines for effective application deployment is based on a methodology that incorporates the following items in order to deliver comprehensive performance testing capabilities that incorporate the key impact of the network during application deployment: Latency impact on the datacenter – Latency causes the transactions and sessions to stay open longer than they would on the LAN. Without incorporating the WAN effects of delay into a load test, memory usage, thread usage, connection pool utilization, network stack, and other critical server resources can be significantly understated.

of 13


.


Bandwidth limitations – Affects transactions “in-flight,” causing them to take longer to complete their transmission, also impacting important server resources. Network impact on response time – Where specific response time goals are being targeted, it is important to incorporate the effect of the network into testing to examine the impact on total response time. Shunra enables you to accurately assess total response time for remote users prior to deployment, and can analyze the breakdown of total response time into the client, network, and server components to assist in diagnosing and resolving transactions that will not consistently meet your response time goals.

Site capacity limitations – the number of concurrent users that can use an application at a remote site is highly dependant on the network capacity to and from the remote site. Incorporating bandwidth limitations into the load test allows accurate prediction and planning for the needed capacity to support an application at a remote site. Figure 6 shows an example of how Shunra can be used to enable LoadRunner to determine the actual response times by transaction for key remote locations. Realistic expectations can be set for the end user experience as the application is deployed. If an application is not going to perform acceptably for some remote users, Shunra VE’s analysis helps to rapidly determine the underlying cause of the problem.

of 13


.


Figure 7 shows an example of how Shunra can be used to enable LoadRunner to determine the bandwidth capacity that will be needed to support expected usage for key remote locations.

Two of the primary causes of application performance shortfalls are insufficient resource capacity and shared resource contention. Capacity planning which is performed without considering the network and remote end-users lacks vital information and may result in misleading or incorrect recommendations.

Is the Network too Complex to Accurately Emulate? Many load testing organizations are not aware that network-aware load testing is readily attainable. There are 4 common misconceptions about network-aware application testing:

1. Emulating the live network is not realistic 2. I need to have a network expert involved in my testing 3. Network emulation is a costly, time-consuming process 4. Network-aware testing rarely needs to be a high priority in test planning

of 13


.


Here are the realities: 1. Emulating the live network is not realistic Shunra’s customers have consistently found that it is very easy to test with enough accuracy to properly anticipate performance as the application is rolled into production To completely model the network is truly a daunting task. There are many variables including the potential interdependencies between multiple hops of network links, router memory and configuration, changes to network routes, capacity, competing traffic, and QoS amongst others. Fortunately, you don’t need to model the entire network to perform network-aware load testing. You don’t need to gain access to network devices or to install special agents or probes to get the needed parameters for testing. In many cases latency is the key factor that will influence application performance over the WAN. Without network aware testing the critical impact of network latency is completely ignored. The reality is that it is easy to gather the end to end latency information needed for emulation testing. This provides a huge value to enabling accurate load testing with minimal effort. 2. I need to have a network expert involved in my testing The great thing about emulation is that the VUs will experience performance just as if they were physically at a remote location. There is no longer a need to deploy remote load generators to include the network into load testing. Often even just a simple ping to each remote location is adequate for setting the correct latency. Network Catcher lite is a simple (free) tool from Shunra that can immediately give you the data needed. For more advanced testing that includes bandwidth constraints and other factors, Shunra offers a more advanced version of Network Catcher that allows you to monitor network performance over a longer period of time with greater detail. Shunra also provides training, professional services, and best practices to help users properly characterize and emulate the network. Shunra’s reporting includes intuitive reports that allow analysis of application behavior without entailing the need for network expertise.

of 13


.


3. Network emulation is a costly, time-consuming process Shunra customers have seen significant and ongoing payback from their investment in WAN emulation. Shunra’s powerful technology is designed to make it quick and easy to setup and run a load test that includes network emulation. By adding our VE Desktop solution to LoadRunner you can install Shunra at 9AM and be performing network-aware load testing by noon 4. Network-aware testing rarely needs to be a high priority in test planning Shunra has evidence from many customers across a wide range of industries that prove that the network is often a critical component of application performance. Many applications are still tested only under LAN conditions and then deployed to the WAN. The cost of one failed deployment is a risk not worth taking, now that an integrated testing solution is available.

I Don’t Control the Network – Why Should I Care? Although responsibility for the network typically falls outside the responsibility of the performance testing team, even if infinite bandwidth was provisioned with no packet loss some application transactions would still fail to perform optimally for remote end-users. Add the impact of real-world provisioned networks which have capacity constraints and periodic packet loss and any transaction problems for remote users become magnified. If an application is certified for deployment yet can't realistically handle the conditions on the corporate network, there is very little anyone can do on the network side to fix it. From the business owner’s vantage point as well as the CIO, it is really the responsibility of the entire development team, and QA in particular, to verify that an application can cope with the actual conditions that will be encountered on the corporate network.

Can I Use My Load Tool's "Think Time" and Bandwidth Option to Create the Network Effects? WAN Emulation lets you test the actual behavior of an application as it will perform over the WAN. There are many underlying factors that can influence how an application will perform under real-world latency and bandwidth that prevent a simplified approach such as adding "think time" to the end-user scripts from being effective. All applications are impacted by how chatty they are on the network. Chattiness is defined as the number of application turns or times that the user needs to wait for the response to arrive back from data transmitted to a remote application service.

of 13


.


In many cases the majority of the sensitivity of an application’s performance comes from the latency, not a bandwidth constraint. The bandwidth throttling built into LoadRunner only restricts outbound total usage, not the return path. Typically web applications send small requests to the server and larger content such as images and tables are returned. This limits the value of the throttling provided by LoadRunner. With Shunra VE and LoadRunner users get a complete and accurate test that reflects how the application will behave under real-world latency and bi-directional bandwidth constraints.

Conclusion – Comprehensive testing can be done within a local Performance Testing Lab

Shunra Virtual Enterprise is tightly integrated with leading tools such as HP LoadRunner. Shunra’s training, implementation guidelines, and professional services have allowed many leading organizations to rapidly achieve the benefits of comprehensive testing that includes the WAN as an integral part of their load testing efforts. Shunra Virtual Enterprise provides an accurate replica of the production network environment within the test lab. With the Shunra Virtual Enterprise, users can quickly and easily find and resolve problems during testing, not after rollout.

About Shunra Shunra is the pioneer and market leader in predicting how business applications and network services will perform for remote end-users - before deployment. It is the only company to provide both hardware and software WAN emulation products that address the needs of all IT groups throughout the application development lifecycle. Shunra’s headquarters are in Philadelphia, Pennsylvania and the R&D center is in Kfar Saba Israel. The company also has local offices in the New York and the UK and is also supported through a global network of channel partners. For more information, visit www.shunra.com or contact Shunra at 1-877-474-8672 (U.S.) or +44 (0)207 153 9835 (outside U.S.).

understanding the impact of running wan emulation with load testing

Documents