synthetic and rum - best of bo
Post on 02-Jul-2015
649 Views
Preview:
DESCRIPTION
TRANSCRIPT
B E S T O F B O T H W O R L D SS Y N T H E T I C A N D R U M
Mark Zeman @SpeedCurve
Cliff Crocker @cliffcrocker
A C T I V E M O N I T O R I N G
• Simulated health-checks of the system and it’s parts
• Scripted journeys through an application
• “Creation” of a singular or set of events and/or actions
• Synthetic is a form of active monitoring AP Photo/ Eric Risberg
PA S S I V E M O N I T O R I N G
• Listening to actual traffic as it moves through a system
• Observing behavior of the system and/or its operator
• “Measurement” of actions or events
• RUM is a form of passive monitoring
What is RUM?Real User Measurement (traditional definition)RUM 101Real User Measurement (RUM) is a technology for collecting performance metrics directly from the browser of an end user.
• Involves instrumentation of your website via JavaScript
• Measurements are fired across the network to a collection point through a small request object (beacon)
<JS> <beacon>
T H E G O O D PA R T S - R U M
• Always on
• Every user, every browser, every network, anywhere in the world
• Ability to capture human behavior/events with performance data
• Only getting better (waterfalls)
What is RUM?Real User Measurement (traditional definition)Synthetic 101Synthetic monitoring (for purposes of this discussion) refers to the use of automated agents (bots) to measure your website from different physical locations.
• A set ‘path’ or URL is defined• Tests are run either adhoc or scheduled and data is collected
T H E G O O D PA R T S - S Y N T H E T I C
• Rich data collected (waterfall, videos/filmstrip, HTTP Headers)
• Consistent “clean room” baseline
• Nothing to install
• Doesn’t require users/ability to measure pre-production, competition
C O M M O N T H I N G S W E H E A R A B O U T B O T H R U M A N D S Y N T H E T I C
• “Why are these numbers so different?”
• “I don’t trust your data. Your numbers are wrong.”
• “How are you calculating Page Load time?”
• “I can’t share two sets of numbers with the business.”
Real users are not normal
W H Y A R E T H E T W O N U M B E R S S O FA R O F F ?
Page Load TimesAll Countries, User Agents, Carriers, Operating Systems
3.23s – Median
Page Load TimesAll Countries, User Agents, Carriers, Operating Systems
10.45s – p95
Page Load TimesAll Countries, User Agents, Carriers, Operating Systems
17.26s – p98
Page Load TimesBy Operating System
OS Median 95th Percentile 98th Percentile
Windows 7 2.41s 9.29s 15.89s
Mac OS X/10 2.30s 8.11s 13.45s
iOS7 3.27s 10.64s 15.79s
Android 4 4.06s 14.30s 27.93s
iOS8 3.53s 11.54s 19.72s
Windows 8 2.67s 10.75s 18.74s
0 1 2 3 5 6 7 8 9 10 11 12 14 15 16 17 18 19
Median: 3.23s
R U MS Y N T H E T I C
F I R S TS Y N T H E T I C
R E P E AT
3 . 2 3 4 . 1 2 . 2
W H Y A R E T H E T W O N U M B E R S S O FA R O F F ?
“ B U T, I T L O A D S S O M U C H FA S T E R F O R M E ? ? ! ! ”
• 2015 Macbook Pro
• Warm browser cache
• FIOS
vs.
• x86 - Windows 7 VM
• Completely cold cache/dns
• Throttled bandwidth
4 seconds 20 seconds
R U M S Y N T H E T I C
• Primary source of truth
• User behavior/conversion
• Complete coverage
• Diagnostics
• Page Construction
• Pre-production & Competitive benchmarking
R U M S Y N T H E T I C
• Primary source of truth
• User behavior/conversion
• Complete coverage
• Diagnostics
• Page Construction
• Pre-production & Competitive benchmarking
You need both
S TA R T R E N D E R 1 . 5 8 S
D N S 1 1 0 M S
T C P 4 0 M S
T T F B 6 5 0 M S
D O M L O A D I N G 6 8 0 M S
D O M R E A D Y 2 . 7 5 S
PA G E L O A D 3 . 0 5
F U L LY L O A D E D 1 9 . 6 8 S
U S E R T I M I N G 1 . 9 5 S
R E S O U R C E T I M I N G 2 3 7 M S
R E Q U E S T S 1 1 6
B Y T E S I N 2 , 1 4 3 K B
S P E E D I N D E X 3 0 8 6
PA G E S P E E D S C O R E
8 31 S = $ 2 7 M
D O M E L E M E N T S
3 9 3 1
D O M S I Z E 3 , 1 4 3 K B
V I S U A L LY C O M P L E T E
1 5 . 2 S
R E D I R E C T 0 S
S S L N E G O T I AT I O N
9 3 M S
T H I S I S Y O U R “ O N E N U M B E R ”
T H E M A N Y FA C E S O F P E R F O R M A N C E
C-Level Operations Developer Designer
thenounproject.com
S TA R T R E N D E R D N S T C P T T F B
D O M L O A D I N G D O M R E A D Y PA G E L O A D F U L LY L O A D E D
U S E R T I M I N GR E S O U R C E
T I M I N GR E Q U E S T S B Y T E S I N
S P E E D I N D E XPA G E S P E E D
S C O R E1 S = $ $
D O M E L E M E N T S
D O M S I Z EV I S U A L LY
C O M P L E T ER E D I R E C T
S S L N E G O T I AT I O N
S TA R T R E N D E R D N S T C P T T F B
D O M L O A D I N G D O M R E A D Y PA G E L O A D F U L LY L O A D E D
U S E R T I M I N GR E S O U R C E
T I M I N GR E Q U E S T S B Y T E S I N
S P E E D I N D E XPA G E S P E E D
S C O R E1 S = $ $
D O M E L E M E N T S
D O M S I Z EV I S U A L LY
C O M P L E T ER E D I R E C T
S S L N E G O T I AT I O N
S TA R T R E N D E R D N S T C P T T F B
D O M L O A D I N G D O M R E A D Y PA G E L O A D F U L LY L O A D E D
U S E R T I M I N GR E S O U R C E
T I M I N GR E Q U E S T S B Y T E S I N
S P E E D I N D E XPA G E S P E E D
S C O R E1 S = $ $
D O M E L E M E N T S
D O M S I Z EV I S U A L LY
C O M P L E T ER E D I R E C T
S S L N E G O T I AT I O N
S TA R T R E N D E R D N S T C P T T F B
D O M L O A D I N G D O M R E A D Y PA G E L O A D F U L LY L O A D E D
U S E R T I M I N GR E S O U R C E
T I M I N GR E Q U E S T S B Y T E S I N
S P E E D I N D E XPA G E S P E E D
S C O R E1 S = $ $
D O M E L E M E N T S
D O M S I Z EV I S U A L LY
C O M P L E T ER E D I R E C T
S S L N E G O T I AT I O N
S TA R T R E N D E R 1 . 5 8 S
D N S 1 1 0 M S
T C P 4 0 M S
T T F B 6 5 0 M SD O M L O A D I N G
6 8 0 M SD O M R E A D Y
2 . 7 5 SPA G E L O A D
3 . 0 5F U L LY L O A D E D
1 9 . 6 8 SU S E R T I M I N G 1 . 9 5 S
R E S O U R C E T I M I N G 2 3 7 M S
R E Q U E S T S 1 1 6
B Y T E S I N 2 , 1 4 3 K BS P E E D I N D E X
3 0 8 6PA G E S P E E D S C O R E
8 31 S = $ 2 7 M D O M E L E M E N T S
3 9 3 1D O M S I Z E 3 , 1 4 3 K B
V I Z C O M P L E T E 1 5 . 2 S
R E D I R E C T 0 S
S S L N E G O T I AT I O N 9 3 M S
Analysis
S TA R T R E N D E R 1 . 5 8 S
D N S 1 1 0 M S
T C P 4 0 M S
T T F B 6 5 0 M SD O M L O A D I N G
6 8 0 M SD O M R E A D Y
2 . 7 5 SPA G E L O A D
3 . 0 5F U L LY L O A D E D
1 9 . 6 8 SU S E R T I M I N G 1 . 9 5 S
R E S O U R C E T I M I N G 2 3 7 M S
R E Q U E S T S 1 1 6
B Y T E S I N 2 , 1 4 3 K BS P E E D I N D E X
3 0 8 6PA G E S P E E D S C O R E
8 31 S = $ 2 7 M D O M E L E M E N T S
3 9 3 1D O M S I Z E 3 , 1 4 3 K B
V I Z C O M P L E T E 1 5 . 2 S
R E D I R E C T 0 S
S S L N E G O T I AT I O N 9 3 M S
Analysis
– C T O
“What is the right number for me to communicate to the organization?”
R E A L I S T I C A L LY, T H E R E I S N O T O N E N U M B E R
1s=$27M Page Load 3.05s
Start Render 1.58s
Speed Index 3086
– C E O
“How is site performance impacting revenue?”
N O T A L L PA G E S A R E C R E AT E D E Q U A LT O P O F F U N N E L I M PA C T ( B R O W S E PA G E S )
For a typical eCommerce site, conversion rate
drops by up to 50% when “browse”
pages increase from 1 to 6 seconds
N O T A L L PA G E S A R E C R E AT E D E Q U A LB O T T O M O F F U N N E L I M PA C T ( C H E C K O U T PA G E S )
However, there is much less impact
to conversion when “checkout” pages degrade
H O W FA S T S H O U L D I B E ?
1s = $27M
– C E O
“How do we compare to the competition?”
– C E O
“How is our site doing in Spain? Are we winning there?”
S I T E P E R F O R M A N C E - S PA I N
• Represents 1% of our overall traffic
• 3rd overall in our competitive index
3.91s Competitive Index
Site 1 Site 2 Site 3 Site 4 Site 5
– V P E C O M M E R C E
“How are third parties impacting my site?”
Dom
ains
Alexa Top 25 Retail
31
10
35
23
86
41
20
37
303131
4
48
3842
16
11
6
28
68
31
36
4
198
2125133511225324323517
37
17
Primary domains 3rd Party domains
3 R D PA R T Y S T U D Y - U N I Q U E D O M A I N S P E R PA G E
Domains: Median - 32 Min - 25 Max - 87
91%
9%
91% of domains are 3rd parties
3 R D PA R T Y S T U D Y
H U F F I N G T O N P O S T - T H I R D PA R T Y I M PA C T U S I N G P E T T Y C A S H
S E T T I N G M E A N I N G F U L S L A S
Example SLA: Response time measured using resource timing from Chrome browsers in the United States should not exceed a median (50th percentile) of 100ms or a 95th percentile of 500ms for a population of more than 500 users in a 24 hour period
– F R O N T E N D D E V E L O P E R
“How should I measure my site in pre-production?”
– V P E C O M M E R C E
“Should I optimize my site for mobile?”
Platform Median 95th Percentile 98th Percentile
Mobile 3.62s 12.53s 20.04s
Desktop 2.44s 9.31s 15.86s
Page Load TimesDesktop vs. Mobile (Mobile and Tablet Optimized)
0 1.5 3 4.5 6 7.5 9 10.5 12 13.5 15 16.5 18 19.5
Platform Median 95th Percentile 98th Percentile
Mobile 4.70s 17.87s 34.43s
Desktop 3.81s 25.15 55.34s
Page Load TimesDesktop vs. Mobile (Full site on Tablet)
1024 1366320 1920
IMG
JSCSS
35303530
768 1280
1770
637
320 1920
IMG
JS
– O P E R AT I O N S
“How can I determine the effectiveness of my CDN provider?”
W H I C H S I T E I S U S I N G A C D N ?
Site A Site B
– F R O N T E N D D E V E L O P E R
“How has page construction changed over the last month, and what impact has that had on performance?”
– F R O N T E N D D E V E L O P E R
“What is the performance of my single page web application?”
T H E C H A L L E N G E O F S I N G L E - PA G E A P P L I C AT I O N S
• Challenges with synthetic: Only measure first page load, not feasible to measure secondary loads via XHR
• Challenges with RUM: Determining whether call is a “page” or service call;Risk of diluting overall numbers
– D E S I G N E R
“How do I know how fast my page becomes usable?”
Start Render Loaded
Usable
• Other questions?
S U M M A R Y
• You need rich sets of data
• There is not one number
• Choose the right number(s) for the right audience
top related