soasta testing inproduction_whitepaper__v1.0

SOASTA, Inc. 2010

Cloud Testing Production Applications SOASTA CloudTest Methodology™ White Paper Series

Executive Summary Online application performance is critical – no one would challenge this statement. Yet the “dirty little secret” in the web world is that applications are rarely performance tested. When performance testing is done it’s usually conducted in a lab with significant resources spent on software, infrastructure, and people. The results from this investment are not delivering the answers that leadership needs about web application performance. Simply put, a lab is not a production environment. Running a few hundred virtual users in a small lab does not mean that you can support twenty times as much load in the production environment. Yet this kind of extrapolation is being done all throughout the industry. The limitations of using legacy client-‐server testing tools for web applications has delayed the introduction of load testing methodologies for production environments. Testing in production is a required component of any performance methodology that aims to achieve the highest confidence possible. This principle is at the core of the SOASTA CloudTest Methodology. Since 2007 SOASTA’s methodology has grown to become the set of patterns that address challenges that arise in testing production environments. This white paper discusses the three most frequently discussed topics about testing in production (security, test data, and potential live customer impact), and describes how the methodology helps to achieve high confidence testing in production.

CloudTest® Strategy and Approach

Table of Contents

Executive Summary .......................................................................................................1

1. Intended Audience ....................................................................................................1

2. The Past and Present of Performance Engineering.....................................................1

3. Product and Methodology Working Together in Harmony .........................................3

4. Select Topics From SOASTA’s CloudTest Methodology...............................................3 4.1 Security .....................................................................................................................................................................5 4.1.1 Sensitive Data Applications.......................................................................................................................... 6 4.1.2 Systems Monitoring Data.............................................................................................................................. 8 4.1.3 Test Data and Results Security ................................................................................................................... 9

4.2 Test Data in Production ..................................................................................................................................10 4.2.1 Using Test Data in Production..................................................................................................................10 4.2.2 Coding for Testability ...................................................................................................................................13

4.3 Potential Live Customer Impact..................................................................................................................13 5. Customer Case Studies – Testing in Production........................................................14 5.1.1 Banking and Financial Data Security....................................................................................................14 5.1.2 Results Confidentiality..................................................................................................................................15 5.1.3 Testing With Live Customers.....................................................................................................................15

About SOASTA, Inc. .....................................................................................................17

Cloud Testing Production Applications 9/27/10 Version 1.0 1

1. Intended Audience Key stakeholders in the application performance lifecycle will find this white paper valuable. For many companies, the target audience for this document is the C-‐Level executive. Web sites are often the primary, if not the only revenue channel for a company. Revenue generation aside, web sites are the most accessible consumer-‐facing outlet for most companies. Therefore, the brand impact from poor performance or an outage is far reaching and usually garners visibility at the highest levels. This means that the CEO, CIO, CFO and CTO are the primary stakeholders for performance. Anyone involved in the performance of a web site will find this information valuable, especially those involved in the decision-‐making and operations of consumer-‐facing applications.

2. The Past and Present of Performance Engineering In the late 1980’s and early 1990’s client-‐server applications were being deployed everywhere. Fueled by the proliferation of desktop computers and advances in networking technologies, information was flowing across the wire in greater quantities than ever before. The need for testing client-‐server performance was quickly identified, which gave birth to products such as LoadRunner by Mercury Interactive (founded in 1989 and now part of HP through acquisition in 2006), and SilkPerformer by Segue Software (founded in 1988 and acquired by Borland in 2006, who was then acquired by Micro Focus in 2009). Along with new products, the discipline of the “performance engineer” emerged as a critical part of the software engineering team. In the late nineties the focus for companies began to shift towards online retail and software-‐as-‐a-‐service applications, moving further and further away from client-‐server technologies. Fast forward to today. Having a web presence is the minimum bar of entry for almost any company in the world. The web is a critical source of revenue and the primary customer-‐facing outlet for advertising and brand management. Applications are running in a web browser instead of being installed. When desktop applications are installed, the software is no longer delivered on CD, it’s downloaded. TV shows and movies are streamed over the web and content is delivered to desktop computers, laptops, cell phones, gaming consoles, or one of many other flavors of desktop and mobile devices. Applications are increasingly event driven, needing to serve a larger than normal customer base for perhaps only a few weeks out of the year… for example, retail applications during the holiday season or tax preparation in tax season. Some applications may only need to support peak load for a few days, or even just a few hours, during events such as the Olympics, the World Cup, or the airing of a TV commercial. In addition to the technology changes introducing complexity in modern web apps, we are also facing unpredictability in


traffic patterns. Social media outlets such as Facebook, MySpace and Twitter can drive more traffic to a web site than it’s prepared to handle. Today’s software development practices are agile. Builds happen multiple times per day, releases to customers happen frequently, and the rate of change is high. Without agile test practices and the right tools, a team is only doing agile development… they’re not really doing agile. All processes and tools need to be able to adapt to the fast moving software development lifecycle to fully benefit an organization. Otherwise, quality and performance will suffer as agile passes them by. Currently, when performance testing is done, it is almost always done in a lab environment. This lab is usually a small subset of the production environment. As such, tests are done on a small scale and the results are extrapolated to peak production numbers. This is problematic for a number of reasons, not the least of which is the fact that the lab is significantly different from production in terms of hardware, scale, configuration, and activity at any given time (batch jobs, shared services, etc). Perhaps most critical is the fact that real users come from outside of the firewall where latency is an important factor in the customer experience. Testing in a lab, while important for extracting some types of results, cannot answer questions about production performance or capacity with a high degree of accuracy and confidence. The most common test tool in use today for web application testing was never designed for this purpose. From technology to licensing, the culture created by using the wrong tools for the job are holding organizations back from getting the right testing outcomes for their applications. HTTP was created as an add-‐on protocol for the client-‐server testing tool LoadRunner. The same methodology of testing with a few thousand users is being used for web sites today. This is not sufficient for web sites with a significant presence. The licensing models for tools like LoadRunner come from a time when expensive Enterprise License Agreements were the standard. Today we live in a world of Software-‐as-‐a-‐Service and On-‐Demand. To meet the challenges of increasing complexity and the real-‐world scale of online applications a modern methodology with the accompanying product solution is required. This solution must offer speed to keep up with agile development and accelerating change rates, it must be able to scale up to the real-‐world traffic levels applications are seeing, and it must be able to do so in a cost effective way. The highest confidence results will be achieved by using this approach on a production environment. Therefore, the final requirement is that the solution is delivered with the expertise and best practices that come from running tests against production websites to ultimately deliver high confidence results.


3. Product and Methodology Working Together in Harmony The methodology for high confidence production testing needs the right product for successful execution. Conversely, the best tool in the marketplace needs the appropriate methodology to go along with it. This harmony enables companies to use the solution to its fullest potential. The SOASTA solution includes both the product and the methodology. The focus of this white paper is on select topics from the methodology that enable production testing. However, it must be mentioned that in order to be successful at production testing having the right software in place is critical. The CloudTest product is discussed in more detail in a separately published white paper titled “The SOASTA CloudTest Solution”. A key takeaway from that work is that there are certain pieces of critical functionality that are required in a testing tool to be successful at production testing. CloudTest is a purpose-‐built production performance-‐testing tool. It has the ability to scale up to generate true concurrent user loads up into the millions of users while still delivering the analytics in real time. Throughout the next few sections there will be references to the CloudTest product as the enabling tool, and to features within it that enable production testing to take place with confidence. For a more in-‐depth understanding of how CloudTest excels over other testing tools on the market, and how it is in many cases the only way to accomplish production testing, please refer to the CloudTest Solution white paper.

4. Select Topics From SOASTA’s CloudTest Methodology The CloudTest Methodology defines the people, processes, and technologies needed for achieving a successful performance-‐engineering ecosystem. A core component of this methodology is testing in production. For years SOASTA has been enabling companies to overcome the organizational and technological hurdles that are encountered when doing production testing. The solutions to these problems have become the blueprints and patterns that should be implemented for successful testing. Every application is different, and every application has different testing needs. The methodology SOASTA has developed is a full lifecycle system. It starts with strategy and planning, and then proceeds into implementation details. The third pillar is execution, where the focus is to produce as much value from testing as possible. Finally, a full iteration of the process ends with measurement of success and changes in the processes to improve the system, ultimately looping back to the beginning to repeat again. The following diagram shows the high-‐level methodology process.


The topics in the following sections summarize the detailed content that resides in the Strategy and Implementation phases. Solving the types of problems discussed here is required to move on to the execution phase. Part of the initial Definition phase includes creating a heat-‐map of the production application and architecture, and subsequently deciding what should be tested and what shouldn’t. Not every business process must be tested in production. For the things that do require testing the methodology speaks to how customers have accomplished this with most of the common pieces of functionality seen in today web applications… things such as logins, payments, checkouts and so on. When production testing is first being integrated into existing development lifecycles and testing practices, the same three questions usually surface.

1. Is testing in production secure? 2. What about test data in the production environment?


3. What about potential impact to live customers? These excerpts from the methodology address these questions and explain the solutions used for successful execution by SOASTA customers.

4.1 Security In general, production testing a web site or application is just as secure as it is for real user traffic. This general statement applies to applications that are already consumer facing, or internal applications that are not classified in nature. These two clarifying statements about how traffic is handled usually provide a high level of assurance about what happens during testing:

1) CloudTest is simulating traffic to a web site over the same communication channels as a browser; HTTP/HTTPS over ports 80 and 443 respectively. The traffic is encrypted just as it is to an end user’s browser.

2) During a load test the CloudTest product only retains responses from an application in memory for a matter of milliseconds… there is no persistence of a response unless programmatically declared in a test. This means that none of the data that goes across the wire is saved.

Diagram: Lower left: Tests are created and controlled via web browser from anywhere. Center: Load is generated from one or more cloud environments around the world. Upper right: Browser traffic is directed at the target application wherever it resides (private datacenter, cloud, etc) over standard HTTP and HTTPS ports just like real users.


The parameters of a test are defined and controlled by the engineer so every action that virtual users are performing is tightly managed. The sensitivity of the data going across the wire can also be handled with proper use of test data techniques and documented processes (discussed in the next section). Depending on the sensitivity of an application, there are measures required to ensure that security is maintained. SOASTA has years of experience testing secured and confidential applications. Testing these kinds of applications is possible with the proper methodology and process applied.

4.1.1 Sensitive Data Applications This document is geared towards testing applications that are already accessible from the outside world, or that could be made accessible to the outside world safely for a period of time. There are some applications that are only accessed on secure networks. SOASTA has done testing for these types of applications through installation of a CloudTest Appliance on-‐site and then testing over the intranet. Sensitive data applications fall into two categories and there are different approaches to ensuring security and confidentiality for each type. The following solutions will apply whether the load generation source is inside or outside the firewall. Non-Disclosure Agreements Non-‐Disclosure Agreements (NDA’s) are commonly put in place between SOASTA, the customer and other involved parties, such as an outsourced development team. While this is common practice in the industry, and certainly common practice between SOASTA and its customers, it is worth a brief mention. This practice is important for all parties to maintain the agreed upon level of confidentiality when working with secure or unreleased information. Sensitive Data Sites These are applications that expose things such as personally identifiable information (PII), legally protected information, classified data, or other types of records that are should only be viewable by those with authority or clearance. Applications that fit this profile include customer service portals, e-‐commerce sites, banking and finance applications, access controlled government sites, law enforcement and military applications. The use of fictitious test data is a common approach to allowing for testability on applications that would typically expose sensitive data to its users. If these sites are being accessed via the outside world the encryption and communication channels are just as


secure as they are for real world users. For some companies this doesn’t alleviate the concern about what data is going across the wire during a test. It is possible to further tighten security of the test by having the virtual users access fictitious data and records. This extra level of security is gained by populating the database with test data. This population can be done either programmatically in a test (e.g. a CloudTest scenario that adds thousands of user accounts in an automated fashion), or by the application engineering team through a back channel (such as direct insertion of user accounts through queries into the database). Data that is typically created in this fashion includes dummy user accounts, events, product SKU’s, inventory counts, etc. Regardless of the data that was transferred during a test, CloudTest discards message responses in matter of milliseconds after they are received. When a response arrives from a request made to an application, it is kept in memory only long enough to analyze and do any extraction or validation on it that might be required, and then it is discarded. There is no persistence of response data unless it is done programmatically. This is often assurance enough to avoid any additional steps in locking down the test data. Pre-Release and Early Access Sites These are applications that will soon be accessible by the general public but have not yet been released. They may or may not contain sensitive data such as social security numbers, credit cards, and things of that nature. The focus is on protecting the confidentiality of the information before it goes public and restricting access so that only the test tool and clients with certain configurations can access the site for testing. Applications that fit this profile include new product releases, marketing campaigns, upcoming sporting events, concert tours, and press releases. IP Address Restriction Access to environments from the outside world during tests can be restricted by the IP address of the servers being used for load generation. A typical practice for companies is to put rules in place on the perimeter firewalls that allow access for the exact IP addresses or ranges of IPs that will be using during a test. This can be enabled right before and disabled right after a test. This practice ensures access by IP address and only during the test window. Using firewall and load balancer access rules that are based on host IP addresses will ensure that content cannot be accessed except from specified sources.


Cookies and Headers Transaction tagging is when the test engineer modifies a request to contain flags that mark it as a test transaction. A recommended practice for access restriction is to configure a load balancer or web application to check for cookie or header values in a request and then redirect the user into a hidden site. The application engineering team can create a load balancer rule that checks for a complex cookie value, for example a string that’s long and obfuscated. The team then gives this value to the test engineer to add to the relevant requests. If the load balancer detects this value, then it would send the request to a different pool of servers that are responsible for serving a hidden version of a site to be tested. Transaction tagging in conjunction with IP address restriction is a secure method of hiding content. This same approach of ‘tagging’ transactions is used for halting processing of certain test requests, such as order placements in live environments (more on this later).

4.1.2 Systems Monitoring Data Another data stream that might be present during a running cloud test is the monitoring data (CPU, Memory, etc). Whether the data is coming from SOASTA’s monitoring agents themselves or from 3rd party monitoring systems, the data is always relayed through SOASTA’s agent called ‘Conductor’. In a production test, Conductor pulls monitoring data from systems inside the firewall, and then it uses ‘push’ technology to send data to the CloudTest main instance. While this data is not generally considered sensitive, as it is only related to the health of systems under test, it can be encrypted for further security and transferred over HTTPS.


Diagram: CloudTest dashboards combine data from the test controller perspective with data from the servers inside the firewall. Connections originate from inside the firewall and the traffic is sent over port 80/443 in standard HTTP/HTTPS protocol. SOASTA’s Conductor agent can also run in a DMZ, a secured area between the servers being monitored and the outside world. This creates a secure relay point for the monitoring data to travel through. Of course, in the most secure environments, monitoring can be turned off so that this stream never leaves the datacenter. Unfortunately, by choosing this option, customers lose the ability to see monitoring data on the same timeline as the test data. Once the test is finished, monitoring data can be wiped, following properly documented procedures similar to those used for confidential test results.

4.1.3 Test Data and Results Security Test data, in the context of this paper, refers to data that resides in an environment (production, in this case) to enable testing. It can also refer to data that was created in the production environment as a result of testing, such as test orders, test accounts, and things of that nature.


Test data is sometimes used loosely to refer to data collected during a test by the CloudTest load generators, things such as result reports, dashboards, monitoring data, etc. In some cases, added security measures for managing the results of a test are required. Documented practices can ensure the confidentiality and proper handling of the results from performance tests. This sometimes applies to companies whose performance and benchmarking are critical to their business, such as platform companies and product companies that would be negatively affected by the release of confidential performance information about their software.

4.2 Test Data in Production Test data, as previously defined in this context, means data that enables testing or is created as a result of testing. The extent of the management effort for test data depends a great deal on the coverage of the test scenarios. A simple browse scenario may have absolutely no data requirements, just page click steps. A test case that logs in might need to have pre-‐populated accounts in the database. On the extreme end of spectrum, some test cases may require placing orders, creating accounts, and sending requests off to third party systems. The art of test case definition to get the right level of coverage is a separate topic, but the goal is to stress the right functionality of the application and not to create a suite of tests that do everything. If at peak a web site does 6 orders per second, the order system itself might not need to be tested in production because the volume is so low. If it does need to be tested there are best practices for enablement. Different strategies apply for different sites -‐ there isn’t a one-‐size-‐fits-‐all approach. The tools to get the job done all exist, but implementations can vary significantly depending on the maturity of the application, the real user traffic at any given time of day, and various other factors.

4.2.1 Using Test Data in Production The notion of using fictitious test data in place of confidential data for security reasons was discussed in section 4.1.1. This section is about best practices on how to maintain any type of test data in the production environment. Throughout the course of testing in production there comes a time when test data will inevitably end up in the production database, either to enable testing of certain scenarios or as a result of test scenarios generating records. If companies want to test anything other than a browsing scenario, and they usually do, this topic needs to be addressed. User Accounts Speaking in general across different categories of applications, the most common type of test data encountered in production environments is user accounts. Most applications have the notion of a login. It is very common to pre-‐populate a production environment with as many user accounts as is needed to accomplish regular performance testing.


This can be done either through direct insertion into the database or through a CloudTest test scenario written to do account creation. Best practices dictate that when test accounts are created, the usernames should be randomized and not follow any particular pattern. The reason for this is that ‘username’ is usually an indexed value in the database. Creating a large number of accounts that have a distinct pattern will cause many databases to change their query plan to optimize for looking up usernames with that pattern. An example of this would be creating 50,000 accounts that are named PERFTEST0 through PERFTEST49999. This would create a pattern in the database that might get optimized for lookups. A better approach is to create usernames that are around 12 characters in length with 3 segments that are randomized and assembled at creation time. Segments can be generated from letters and numbers, pulled from timestamps and the like, and then put together to create fairly distinct names. At creation time the usernames can be exported to a .csv file to be used in CloudTest. Sometimes, during the creation process they are inserted into a separate table, and then exported to .csv; whatever works best for the DBA and the database platform. The key is to make sure they are tracked at creation time, as significantly randomized usernames may be hard to pull out of the database after the fact. Once the test accounts are in the database, they should remain there to support ongoing testability. SOASTA has customers with hundreds of thousands of test accounts in the production database to support ongoing testing. A site that has 300,000 concurrent users logged in at any given time would statistically have a user base in the 50 to 100 million range. Therefore, the storage requirements for 300,000 test accounts are a fraction of what is being used for the entire population. Additionally, deleting that many records from a production data store may cause problems such as database fragmentation (see the section titled ‘Test Data Cleanup’ for information on how to address this). Handling Other Types of Test Data Data that needs to be read usually has requirements that are easy to meet. User accounts aside, things such as products, inventory counts, content articles, downloads, and things of that nature can be used in a test case with acceptable stress on the application, whether it’s real or fictitious data (aforementioned security concerns aside). Data that’s being inserted, updated or deleted needs special attention. Many test data situations can be handled using techniques already documented. However, to stress critical functionality in an application sometimes there doesn’t need to be a database operation or a transaction that goes past a certain point. For example, in an e-‐commerce application it is common practice to have an order object move into a queue to be picked up and processed. If getting the order into the queue is the performance concern, and the processing operation is not in the testing heat map because it’s not a concern, or because it can’t be tested for technology reasons, then ideally the processor needs to pick


up the order and discard it. Techniques like this are called ‘coding for testability’. In this example the queue processor needs to be modified to pick up the order off of the queue, recognize that it’s a test order (possibly through previously discussed transaction tagging), and discard it. This technique is discussed in more detail in section 4.2.2. Test Data Cleanup Some types of data should stay in the production database to support ongoing testing. User accounts, fictitious events or content, etc. However, database performance does get worse as a table gets larger. Depending on the size of a test and the impact that the test data can have on database performance, this data may need to be deleted from production. A common performance problem that databases encounter over time is called database fragmentation. This fragmentation is much like what occurs on a hard disk drive. Hard disks try to store files in a contiguous group of blocks and sectors. When you delete a file, it leaves a gap on the drive. If the next thing written to the drive is larger than the gap, part of it goes in the hole left by the previously deleted file and the rest goes in the next available spot. Reading this file will be slower because the drive head has to move across the drive to get all of it. The same thing happens to databases, except that this is a non-‐trivial problem because the fragmentation is in memory. Most modern databases store their data in RAM and only access the disk when they need to. If you delete large chunks of data from a database, there will be gaps in the data and the individual pages that store data will become non-‐contiguous. A side effect of this is that SQL statements will take much longer to process because they are spanning large gaps of memory and disk pages when they execute. Databases usually have ongoing nightly maintenance plans and special care should be given to them, especially during times surrounding bulk deletes. Steps should be taken to ensure that indexes are up to date (so that the database always knows where to find data without having to follow multiple data pointers), and also that the records exist in the proper order (defined by the Primary Key) with no extra space on data pages. Severe fragmentation can result in a single record being stored on an 8-‐kilobyte memory page instead of multiple records grouped together. Something to keep in mind around the first few production tests is that a regularly scheduled maintenance plan might run for much longer the night after a production test and a data delete. This should be observed to ensure that the plan does not execute into the following day and peak traffic hours.


4.2.2 Coding for Testability One benefit of being able to control the entire payload of a request in CloudTest is that an engineer can put anything in the message needed to support testability. More specifically, they have the flexibility to create messages that wouldn’t normally show up through typical use of the application. One can programmatically set cookie values, modify headers, and change query strings or post data in an HTTP request. When this is done to mark a request it is called ‘tagging’ a transaction. This has many uses ranging from access control to flagging requests so that they do not proceed past a certain point in an execution sequence. A typical scenario where this is useful is in order placement on an e-‐commerce application. A tagged transaction will get as far as it needs to go to achieve adequate test coverage, possibly moving through alternate code paths. The goal of testing is to get the right amount of coverage on the architectural heat map identified in the strategy phase, so where the transaction needs to stop depends on the criticality of what needs to be stressed. Tagging a transaction and handling it properly can ensure that payments don’t get processed, orders don’t get fulfilled, and that ultimately thousands of test orders don’t end up getting shipped to an unfortunate engineer’s doorstep.

4.3 Potential Live Customer Impact Testing in production can occur with or without live users on the environment. The majority of customers testing in production with the CloudTest Methodology are doing so on a live environment. It is possible, but not always feasible, to put up a maintenance or turn-‐away page and wait until all existing user sessions have been finished. This is actually rarely encountered, though, because the right tool and methodology working together can allow testing to take place in conjunction with real users in all but the most extreme cases. To use one customer’s exact words, “The cost of not testing is far greater than the potential impact to live customers”. You can continue to generate revenue while testing at the same time with the right approach. It is also possible to segment a portion of the live environment during a low-‐traffic period and allow for testing on this environment. Typically, a separate IP address is configured on a load balancer and servers are moved out of the live pool and placed into the test pool. Sometimes configuration changes need to be made to the application servers in this cluster to point to new databases and be taken off of other shared components. This is a more costly approach because it requires extra hardware and the associated maintenance overhead. It’s also somewhat less reliable because you start to deviate from the actual production configuration, and you cannot test to the true scale. It is still, however, a more realistic test than simply testing in a lab.


Three Requirements for Successful Live Testing The first requirement for being able to do a large-‐scale test with live customers is having real-‐time analytics in your test tool. With up to the second information about site performance, you can quickly tell if a site has started to experience poor performance or become completely unresponsive. Secondly, the product needs a good kill switch. Pressing stop or abort in a running CloudTest will stop the load immediately. Bandwidth, concurrent connections, threads in use, and other typical pinch-‐points will all drop back down to normal. With real-‐time analytics and the kill switch, you can stop a test as soon as you suspect customers may be impacted. If the test tool takes a long time to kill a test this poses a serious risk. CloudTest is specifically designed to accomplish a near immediate stop. Third, having good monitoring practices internally, preferably integrated with CloudTest, can prevent you from ever needing to abort a test because of live user impact. Being able to watch the CPU growing to high levels or the number of connections reach its maximum on a load balancer or firewall in real-‐time can help prevent those thresholds ever being violated by routine testing.

5. Customer Case Studies – Testing in Production 5.1 Security Case Studies

5.1.1 Banking and Financial Data Security A large financial software company with an online application that allows customers to see all of their financial accounts in one place uses the SOASTA solution to test in production. This is accomplished through the use of test data and alternate code paths. The customer uses the following test data:

• Test user accounts in the production database (10,000 total) • Fictitious financial institutions • Dummy transactional data

When a real user logs into this application, calls are made to their financial institutions and all of their transactions are pulled in. Test users have fictitious financial institutions set up in their profiles, so when they log in, those are loaded on the screen. Then, an asynchronous refresh of those accounts kicks off in the background. The code responsible for triggering this refresh recognizes that the financial institution is for testing purposes, and uses an alternate code path that ends the refresh, aborting any attempts to try to look up and contact an institution that doesn’t exist.


Using these mechanisms, the customer is able to test all of the major functionality of the application. This includes starting on the home page, logging in, loading transactions to the screen, building charts and graphs, and so on. No actual customer data is shown during testing as the accounts and the transactions are fictitious. All tiers of the application are stressed except the asynchronous account refresh service that contacts 3rd parties -‐ because it was not deemed necessary for testing during the strategy phase of the methodology.

5.1.2 Results Confidentiality A large provider of a highly scalable e-‐commerce platform uses SOASTA to conduct testing from the cloud. Scalability and performance are critical to their business, for ensuring that their customers have the best platform to deploy on, and ensuring that they stay ahead of the competition. As such, the results of their test are considered highly confidential. SOASTA has a special NDA in place with the customer that includes a few individually selected and named SOASTA engineers. Access to controlled test environments is allowed through IP address restriction. After a test is run, the results are exported and sent to the customer. Then the results are deleted from the CloudTest results repository in a documented process that includes customer observation of the post-‐test cleanup activities.

5.1.3 Testing With Live Customers One of America’s largest publicly traded financial software companies uses SOASTA CloudTest to test regularly in production with live customers. During daylight hours with tens of thousands of real users they will conduct tests up into the hundreds of thousands of users. This is accomplished through the use of real-‐time data provided by CloudTest, as well as a mature in-‐house monitoring solution. With a good picture of the customer experience at any given time from network, application, and end user perspectives, the team can tell with up to the moment data whether customers are being impacted by a test. If a breaking point is hit, the test will be immediately stopped with the CloudTest kill switch. At this point, load will immediately stop. Testing in production with live users for performance and failover is a routine approach used by some of the world’s largest companies. Google routinely fails over entire datacenters as part of monthly maintenance.


Conclusion Today’s fast moving online world demands that companies have the right performance testing solution to be successful. Creating a poor customer experience can be catastrophic. These events can result in lost revenue, failed customer retention, and a viral spread of poor brand visibility. The dominant players in the testing marketplace have been slow to respond and give companies what they need: high confidence that these problems won’t happen. That confidence comes through using the right testing approach and the right tools for the job. SOASTA is the leader in cloud testing and at the forefront of modern performance testing. SOASTA has received numerous awards and accolades including being listed as the Visionary Leader in the Gartner Magic Quadrant for test tools, and named a Wall Street Journal Top 50 Venture Backed Company (out of 38,000 world-‐wide). SOASTA delivers the highest confidence possible by conducting accurate and realistic testing in the right environment. The performance engineering team at SOASTA has deployed hundreds of thousands of cloud servers, run tens of thousands of hours of tests against production environments, and has run tests with millions of concurrent users. This team is uniquely qualified to do large-‐scale tests against production websites. All of this contributes to the value that SOASTA is delivering daily to leading online customers such as Chegg, Gilt Groupe, Hallmark, Intuit, Netflix, Mattel, MySpace, Symantec, and many others.


About SOASTA, Inc. SOASTA’s mission is to ensure that today’s Web applications and services perform in a high quality, scalable, and predictable manner. The company’s product, SOASTA CloudTest®, is available as an on-‐demand service in the Cloud or as an appliance (virtual and hardware), and enables developers to test and monitor their Web applications and services at an affordable price. SOASTA CloudTest® supports Load, Performance, Functional, and Web UI/Ajax testing. SOASTA is privately held and headquartered in Mountain View, California. For more information about SOASTA, please visit www.soasta.com

soasta testing inproduction_whitepaper__v1.0

Real Estate

alternate

support ongoing

ip address

soastas cloudtest

financial

application

real user

enable production