+ People don’t know what they want until you show it to them.

Why “Average” Response Time is not a right measure of your web application's performance?

+ For the next 15 minutes

n  Web Application Performance Testing

n  Average Response Time

n  Solution to Average Response Time

n  Beyond Average Response Time

n  Questions

+ Web Application Performance Testing

+ Web Application Performance Testing

How to do

Design Test/Scenarios

Establish Performance Acceptance

Criteria

Identify Environment

Execute Test

Analyze Reports / Tune/

Retests

What we collect

Response Time

Throughput

CPU Utilization

Memory

+ Average Response Time

n  Response Time is defined as the time it takes for each Web page to load.

n  How we capture? n  Hit the server with a specified number of user (could be in pattern)

and then capture the response time of the server.

n  Refer the plot of ART on next page.

+ Average Response Time

0

100

200

300

400

500

600

Page 1 (in ms)

Page 2 (in ms)

Page 3 (in ms)

+ Let’s take a real life scenario…

n  Consider a typical web application

n  Load Scenario (1): n  60% of requests | 300ms

n  20% of requests |10ms

n  went through real quick.

n  mostly served from cache.

n  20% of requests | 10 sec

n  Requests stuck up due to external api’s or

n  let’s say some DB locks.

n  Average Response Time: 2.182 sec

+ Fallacy - #1

What’s wrong with 2.182 sec?

n  Bad Indicator of user “satisfaction” given that 20% of your users are unhappy with your site.

n  80% of users who are facing fast response time can’t compensate the others who are leaving the website.

n  20% of users are very critical depending on which part of application they are on.

n  E.g. Sign Up, Log In, Check out

Learning: It hides important outliers.

+ Fallacy - #2

Increasing complexity of website Huh !

n  Websites are little complicated these days. Different pages have different normal response time. n  Login thru LinkedIn API taking 5 sec is ok. n  A kernel cached image taking 5 sec to serve is not ok.

n  Averaging these two together wouldn’t really let me know anything.

n  It’s unfair to compare the Average Response Time of a page over time. n  Every time the component of the traffic changes, it will have

impact on ART.

Learning: Your Average Response Time is not a fixed number.

+ Fallacy - #3

Averages are meant more for averaging the total quantity being measured, not the number of samples !

n  Load scenario (2):

n  10 request | ~300ms

n  1 request | 5 minute

n  hit triggered a cache miss, which got stuck in some almost never hit DB query.

n  Average Response Time spiked to 27.3 sec !

n  First impression “Something is seriously wrong!”

Learning: Your average response time never tells you the number of users affected by the problem.

What we need?

+ Apdex

n  Created by Peter Sevcik of Apdex Alliance (apdex.org).

n  How is it calculated?

n  ApdexT = (# Satisfied requests + # Tolerating requests / 2)

(# Total requests)

APDEX: Application Performance Index

+ How it scores over Average Response Time

n  Load Scenario(1). 60% of requests take 300ms, 20% take 10ms, and 20 percent are getting unacceptable latencies at 10 seconds.

n  Average Response Time = 2.182 sec; Wrong Confidence

n  Apdex2s is 80%; Pin points the problem

n  Load Scenario(2). 10 requests takes 300ms and 1 request takes 5 minute.

n  Average Response Time = 27.3 seconds; False Alarm

n  Apdex2s is 90.9 % ; Good health

ü  It does a good job.

ü  Indicates correct health of application.

ü  Highlights even when the small percentage of user face performance issue (so called outliers).

+ One last thing…

Load Testing

Stress Testing

Soak Testing

Performance Tuning

+