©2 016 S OA S TA , A l l r i gh t s r es e rv ed .

Tom Chavez @tomchavezMike Ostenberg @mikeostenberg

Load testing for peak season:What to test and how to test

©2 016 S OA S TA , A l l r i gh t s r es e rv ed .

• How fast is your site now?• Measure everything• Too much data? Science!• Prioritize what to fix• Optimize your site• What to test?• Test early and often – shift left and right• How to test?

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 3

Measure, Optimize, Test – Repeat

How fast is your site now?

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 5

How fast is your site now?

• Google says 1000ms!

• No single right answer

• Not all pages are equalo Some pages should be faster than others

o The slowest pages might not be the ones to fix first

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 6

How is your site now?

• mCommerce is globalo US is not the benchmark

• 39% of mobile users

are unhappy with their online experienceso Top reasons: slow pages and site freezes/crashes

• 57% have experienced problems when using mobile

• 3 out of 4 consider mobile slower than desktop

• 46% will not return to a site that performs poorly

• 27% of smartphone users exceed data cap every month

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 7

When is your Peak Season?

• Peak is year-roundo Super Bowl Sunday – 110 million viewers

o Bi-weekly or monthly payroll processing

o Valentine’s Day, President’s Day, Mother’s Day, Father’s Day

o Quarterly processing and filing

o US Tax filing

o Spring, Summer, Fall & Winter season sales

o July 4th, Christmas in July

o Back to School We are here

o Olympics

o Halloween

o Election Day

o Singles Day – world’s biggest shopping day, $9B in 12 hours

o Black Friday, Cyber Monday and Holidays

Measure Everything

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 9

Measure Real, Synthetic, and Virtual Users

• Real Users (RUM) provide deep data about your visitorso No other way to know everything about your users on your site

• Synthetic Users tell you how parts of your site are doingo When no real users are visiting, synthetic users are still coming

• Virtual Users tell you how your site will do at scaleo Can’t get a million friends to visit your site at the same time?

• Collect all the data and save it in the cloud!o Storage is cheap in the cloud, plus easier to analyze there

• Add App Performance Management (APM) for a complete picture

The Three Musketeers of Performance

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 10

Other Things to Measure

• Session pathso How are your users going through your site

• Mobile App Performanceo How are your mobile apps performing?

o What are users doing in your mobile apps?

• Revenueo How much money are you making from each user and where?

• How fast are your competitors’ pages?o You can’t insert code on their sites but you can measure

Your company may be measuring these already

Too much data?Science!

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 12

Analyze

• Benchmark and baseline your siteo How fast is your site now? How about the competition?

• Correlate performance and outcomeo Is poor performance affecting the business?

• Find root causes for performance issueso Use RUM and APM data

• Fix pages that impact mosto Make a prioritized punch list of the pages to fix

Use the data you’ve collected to gain insights

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 13

Baseline Your Performance

How is your site doing? How does it compare to the competition?

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 14

Find Root Causes for Poor Performance

Not all problems are from your app or back end systems

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 15

Page Visits vs. Performance

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 16

Page Visits vs. Performance by Device

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 17

Prioritize What to Fix – Greatest Impact

The poorest performing page might not be the most important page

Fix and Optimize

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 19

Page Bloat

The average web page is more than 2MB in size

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 20

Page Bloat

Where are all those kilobytes going

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 21

Doesn’t Bandwidth Solve the Problem?

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 22

Comparing Network Performance

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 23

Comparing Network Performance

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 24

Doesn’t Bandwidth Solve the Problem?

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 25

Doesn’t Bandwidth Solve the Problem?

25x Faster Bandwidth != 25x Faster Load Time

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 26

Prepare for Peak Events

Optimize content for peak load events

• Simplify website contento Keep only the most important content on the page

• Reduce infrastructure requestso Block unnecessary requests, out-of-region visitors

• Leverage historical comparisonso Review past traffic levels to predict future levels

What Should I Test?

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 28

Creating a Test Plan – Session Analysis

Which paths are the most important to users?

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 29

Creating a Test Plan – Session Analysis

Which paths are the most important to users?

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 30

Creating a Test Plan – Peak Load

What was the busiest Day, Hour, Minute, Second?

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 31

Creating a Test Plan – Page Flows

What were most popular page flows with think times?

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 32

Creating a Test Plan -

Which paths should I test, what think times, at what load, from where?

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 33

Network Emulation

• Testing to match realistic situationso Users are not all from one location on a fast network

o Packets get delayed, lost, re-ordered

• Apps perform differently as network conditions worseno Reduced throughput

o Timeouts occur, connections are lost

o Transactions take longer, longer sessions

Adding real network modeling to load and performance testing

Test Earlyand Often

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 35

Start Testing Yesterday

• How long will it take?o 300 test cases x 2 browsers = 600 cases

o 600 cases x 30-minutes/case = 300 hours

o 300 hours / 6 hours/day = 50 days

o 50 days / 5 people = 10 days

• And for mobile:o 300 text cases x 26 devices = 7800 cases

o 7800 cases x 30 minutes/case = 3900 hours

o 3900 hours / 6 hours/day = 650 days

o 650- days/5 people = 130 days of test – 26 weeks!

And even earlier for Mobile

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 36

Don’t Wait Til the Last Minute to Test!

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 37

Test Right and Test Left, even into Production

• Performance Testing in developmento Set performance SLAs for new code and test in development,

same as unit and functional tests which code must pass

• Performance Testing in stagingo Higher loads in near-production systems

• Performance Testing in Productiono Full scale load tests to expected load and beyond

o Find weak and stress points and prioritize fixes into project

Performance Testing is Continuous

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 38

Test Continuously

Test from Development to Integration to Staging to Production

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 39

Test in Production

Some issues will only be found at scale load on production systems

Don’t forget your third party providers: CDNs, external scripts

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 40

Measure Mobile App Performance Under Load

All mobile apps perform well when there is no load

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 41

Automate Your Testing

Use Continuous Integration tools to automate testing

Demo

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 43

Now That the Test is Over

• Analyze results for future areas to addresso Weak or stress areas that aren’t likely this time but …

• Identify third parties that were affected by loado Coordinate with suppliers to address issues

o Align schedules or at least be aware of their changes

Even if the test passed, you’re not done

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 44

Issues Will Happen

• “I am 100% confident that everything we tested will work

just fine.” – Nordstrom Performance Leadero When loads are different than what you modeled, may have

problems

o E.g. At Nordstrom, during a recent anniversary sale, mobile sign-

ins were 9x greater than anticipated

o Used RUM to get to the root cause of the issue

Realize that Issues Will Happen Where You Can’t Predict Them

Q & AResourcesNext Steps

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 46

Questions, Resources & Next Steps

• Holiday Readinesso soasta.com/holiday-readiness

• The Performance Beacono soasta.com/blogo soasta.com/blog/2015-popular-posts-web-performance

• Upcoming Webinarso soasta.com/webinars: RUM-Based Testing

o Recordings of past webinars

• CloudTest on-Demando soasta.com/cloudtest: SOASTA will build & run tests for you

Resources

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 47

“The leader in performance analytics.”

www.soasta.com

SOASTA

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 48

CloudTest

• Load testing from development to productiono Continuous testing at any scale, on-prem or in the cloud

• Real-time analytics with customizable dashboardso Safely test in production with full visibility

• Run tests live or automaticallyo Full hands-on control or hands-off automation

• Full end-to-end viewo Front-end performance to back-end systems

Fast, scalable, affordable load testing for performance optimization