performance testing: putting cloud customers back in the driver’s seat
DESCRIPTION
Many businesses wrongly assume they will enjoy Google.com- and Amazon.com-like performance and consistency when they enlist cloud computing services from these and other major cloud providers. The truth is that businesses must conduct due diligence and insist on business-relevant performance guarantees in their service level agreements (SLAs). The keys for businesses success in working with cloud providers lies in understanding exactly why businesses are using the cloud and in testing performance levels from the realistic perspective of application end-users--both before and after a cloud service provider is enlisted.TRANSCRIPT
![Page 1: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/1.jpg)
Performance Testing:
Putting Cloud Customers Back in the Driver’s Seat
Imad Mouline, CTO, Gomez Twitter: @imadmouline
![Page 2: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/2.jpg)
Agenda
A brief overview of the performance problem in the cloud
Real-world data: how have cloud providers performed so far?
Optimization opportunities in the cloud
Takeaways
Q&A
![Page 3: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/3.jpg)
Typical Web Application Delivery
The Web Application Delivery Chain
MajorISP
Local ISP
Mobile Carrier
Internet
Content DeliveryNetworks
3rd Party/Cloud Services
Browsers and devices Users
Storage
Web Servers
App Servers
DB Servers
Mainframe
Load Balancers
Mobile Components
Network
Traditional zone of control
![Page 4: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/4.jpg)
MajorISP
Local ISP
Mobile Carrier
Internet
Content DeliveryNetworks
3rd Party/Cloud Services
Browsers and devices
Storage
Web Servers
App Servers
DB Servers
Mainframe
Load Balancers
Mobile Components
Network
Moving Web Applications to the Cloud: Benefits
The Web Application Delivery Chain
Users
Traditional zone of control
Elastic and scalableFocus on my business
while someone manages infrastructure
Cloud
![Page 5: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/5.jpg)
MajorISP
Local ISP
Mobile Carrier
Internet
Content DeliveryNetworks
3rd Party/Cloud Services
Browsers and devices UsersUsers
Storage
Web Servers
App Servers
DB Servers
Mainframe
Load Balancers
Mobile Components
Network
The Problem: The Cloud Creates Performance Concerns
The Web Application Delivery Chain
Traditional zone of control
Cloud
Cloud is opaque• Loss of visibility and control• Traditional tools don’t apply
![Page 6: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/6.jpg)
The Answer: Adopt an “Outside-In” User Point of View
The Web Application Delivery Chain
Users
MajorISP
Local ISP
Mobile Carrier
Internet
Content DeliveryNetworks
3rd Party/Cloud Services
Browsers and devices
Storage
Web Servers
App Servers
DB Servers
Mainframe
Load Balancers
Mobile Components
Network
Users
Traditional zone of control
Full understanding of performance from user perspective
“Outside-in” customer point of view
![Page 7: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/7.jpg)
MajorISP
Local ISP
Mobile Carrier
Internet
Content DeliveryNetworks
3rd Party/Cloud Services
Browsers and devices
Storage
Web Servers
App Servers
DB Servers
Mainframe
Load Balancers
Mobile Components
Network
My users
Other users
Other users
Other users
The Problem: The Cloud Creates Performance Concerns
The Web Application Delivery Chain
Cloud is shared• Others can affect my performance
My app
Otherapp
Otherapp
Otherapp
![Page 8: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/8.jpg)
MajorISP
Local ISP
Mobile Carrier
Internet
Content DeliveryNetworks
3rd Party/Cloud Services
Browsers and devices
Storage
Web Servers
App Servers
DB Servers
Mainframe
Load Balancers
Mobile Components
Network
My users
Other users
Other users
Other users
The Answer: Collective Intelligence
The Web Application Delivery Chain
Cloud is shared
Multiple contributors help diagnose issues for everyone
My app
Otherapp
Otherapp
Otherapp
“Outside-in” customer point of view
![Page 9: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/9.jpg)
Web Applications in the Cloud: Need to Be Able to Pinpoint Problems
The Web Application Delivery Chain
MajorISP
Local ISP
Mobile Carrier
Internet
Content DeliveryNetworks
3rd Party/Cloud Services
Browsers and devices
Storage
Web Servers
App Servers
DB Servers
Mainframe
Load Balancers
Mobile Components
Network
My users
Other users
Other users
Other usersIs it my
cloud provider
problem?
Is it an ISP or
Internet problem?
Is it a service
provider problem?
Is it a browser or
device problem?
![Page 10: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/10.jpg)
Introducing… CloudSleuth
![Page 11: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/11.jpg)
A Year In The Cloud
An end-user perspective on cloud performance
![Page 12: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/12.jpg)
What we measured
Cloned reference web application deployed across various IaaS and PaaS providers
Added various services such as CDN when available
Structured application to highlight performance issues
![Page 13: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/13.jpg)
How we measured
Various locations around the world
Backbone and Last Mile locations
Every 15 minutes, 24/7, for almost 1 year
Used a strict definition of availability.
![Page 14: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/14.jpg)
Not all clouds perform the same way
Average Response Time of Reference Application, as measured from US backbone locations
![Page 15: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/15.jpg)
Taking the long view – response time
3/25/0
9
4/4/0
9
4/14/0
9
4/24/0
9
5/4/0
9
5/14/0
9
5/24/0
9
6/3/0
9
6/13/0
9
6/23/0
9
7/3/0
9
7/13/0
9
7/23/0
9
8/2/0
9
8/12/0
9
8/22/0
9
9/1/0
9
9/11/0
9
9/21/0
9
10/1/0
9
10/11/0
9
10/21/0
9
10/31/0
9
11/10/0
9
11/20/0
9
11/30/0
9
12/10/0
9
12/20/0
9
12/30/0
9
1/9/1
0
1/19/1
0
1/29/1
0
2/8/1
0
2/18/1
0
2/28/1
0
3/10/1
00
1
2
3
4
5
6
7
8
9
10
Average Response Time for reference transaction on Amazon S3, as measured from US locations
© 2
010
Gom
ez –
All
Rig
hts
Res
erve
d
![Page 16: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/16.jpg)
Taking the long view - availability
99
99.1
99.2
99.3
99.4
99.5
99.6
99.7
99.8
99.9
100
100.1
Weekly availability for reference transaction on Amazon S3, as measured from US locations
![Page 17: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/17.jpg)
Geographic Latency – GoGrid
![Page 18: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/18.jpg)
Geographic Latency – S3
![Page 19: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/19.jpg)
Going International?
![Page 20: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/20.jpg)
It’s not all about network latency
![Page 21: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/21.jpg)
Enough scary news…… let’s see something good for a change
![Page 22: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/22.jpg)
Geographic Latency – S3
![Page 23: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/23.jpg)
Added services help… a lot!
![Page 24: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/24.jpg)
Unique opportunities for optimization
Better performance, for little work, at no extra cost?
![Page 25: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/25.jpg)
Best Practice: Define your goals and build a plan
Align goals across your organization Why are we moving to the cloud?
Common goals include:
1. Additional Capacity – How much capacity do we need during normal and peak times?
2. Improved End-User Experience – What performance goals are we trying to deliver against?
3. Greater Elasticity – How quickly can the provider we select ramp up to meet our needs?
4. Flexible Bursting – How fast do we need to be able to access additional capacity?
If only there was a button to push!
![Page 26: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/26.jpg)
Best Practice: Keep your end-users in mind
How was that experience for your customers?
Test your cloud applications the same way your customers use it:
What they do? Customers care about completing tasks
NOT whether the homepage is available
Where they do it from? Your customers don’t live in the cloud, test from their perspective
When they do it? Test at peak and normal traffic levels, to find all the problems
What expectations do customers have? Is 4 seconds fast enough or does it have to be quicker?
Geographic disparities
4 sec’s
22 sec’s
Find configuration, capacity, and performance problems before end-users with pre-production testing and production monitoring
![Page 27: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/27.jpg)
Best Practice: Performance testing cloud capabilities
Evaluate vendors based on your goals…
CapacityTest vendors to 15-20% past estimated capacity goals
ElasticityBaseline end-user performance before & after testing
Test during pre-deployment and in production
Ramp elasticity testing to peak levels
BurstabilityIsolate the cloud elements from other infrastructure to test
Test the “failover process”
![Page 28: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/28.jpg)
Best Practice: Set SLAs to match your goals
Set SLAs based on your goals…
End-user availability and response times
Capacity & elasticity objectives
Burstability goalsSet SLAs based on how you are using the cloud…
SaaS PaaS IaaS• Transactional Success Rate• Fast end-user experience across key geographies
• Availability and performance of key components, services, and API’s
• Hardware & OS availability / performance• Network connectivity
• Capacity and elasticity requirements
![Page 29: Performance Testing: Putting Cloud Customers Back in the Driver’s Seat](https://reader035.vdocuments.net/reader035/viewer/2022081401/5591b20e1a28ab26518b4801/html5/thumbnails/29.jpg)
Putting together a cloud management strategy
Key Steps Vendor Evaluation
Deployment Testing
Production Monitoring
Service Level Management
Validate the burstability & elasticity of cloud solutionsIdentify application performance bottlenecksMeasure performance under normal and peak conditionsBaseline and monitor service level objectivesMeasure how end-user performance impacts behaviorMonitor performance & availability from the end-users’ perspective