Download - Synthetic and RUM - Best of bo
![Page 1: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/1.jpg)
B E S T O F B O T H W O R L D SS Y N T H E T I C A N D R U M
![Page 2: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/2.jpg)
Mark Zeman @SpeedCurve
Cliff Crocker @cliffcrocker
![Page 3: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/3.jpg)
A C T I V E M O N I T O R I N G
• Simulated health-checks of the system and it’s parts
• Scripted journeys through an application
• “Creation” of a singular or set of events and/or actions
• Synthetic is a form of active monitoring AP Photo/ Eric Risberg
![Page 4: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/4.jpg)
PA S S I V E M O N I T O R I N G
• Listening to actual traffic as it moves through a system
• Observing behavior of the system and/or its operator
• “Measurement” of actions or events
• RUM is a form of passive monitoring
![Page 6: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/6.jpg)
What is RUM?Real User Measurement (traditional definition)RUM 101Real User Measurement (RUM) is a technology for collecting performance metrics directly from the browser of an end user.
• Involves instrumentation of your website via JavaScript
• Measurements are fired across the network to a collection point through a small request object (beacon)
<JS> <beacon>
![Page 7: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/7.jpg)
T H E G O O D PA R T S - R U M
• Always on
• Every user, every browser, every network, anywhere in the world
• Ability to capture human behavior/events with performance data
• Only getting better (waterfalls)
![Page 8: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/8.jpg)
What is RUM?Real User Measurement (traditional definition)Synthetic 101Synthetic monitoring (for purposes of this discussion) refers to the use of automated agents (bots) to measure your website from different physical locations.
• A set ‘path’ or URL is defined• Tests are run either adhoc or scheduled and data is collected
![Page 9: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/9.jpg)
T H E G O O D PA R T S - S Y N T H E T I C
• Rich data collected (waterfall, videos/filmstrip, HTTP Headers)
• Consistent “clean room” baseline
• Nothing to install
• Doesn’t require users/ability to measure pre-production, competition
![Page 10: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/10.jpg)
C O M M O N T H I N G S W E H E A R A B O U T B O T H R U M A N D S Y N T H E T I C
• “Why are these numbers so different?”
• “I don’t trust your data. Your numbers are wrong.”
• “How are you calculating Page Load time?”
• “I can’t share two sets of numbers with the business.”
![Page 11: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/11.jpg)
Real users are not normal
W H Y A R E T H E T W O N U M B E R S S O FA R O F F ?
![Page 12: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/12.jpg)
Page Load TimesAll Countries, User Agents, Carriers, Operating Systems
3.23s – Median
![Page 13: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/13.jpg)
Page Load TimesAll Countries, User Agents, Carriers, Operating Systems
10.45s – p95
![Page 14: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/14.jpg)
Page Load TimesAll Countries, User Agents, Carriers, Operating Systems
17.26s – p98
![Page 15: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/15.jpg)
Page Load TimesBy Operating System
OS Median 95th Percentile 98th Percentile
Windows 7 2.41s 9.29s 15.89s
Mac OS X/10 2.30s 8.11s 13.45s
iOS7 3.27s 10.64s 15.79s
Android 4 4.06s 14.30s 27.93s
iOS8 3.53s 11.54s 19.72s
Windows 8 2.67s 10.75s 18.74s
![Page 16: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/16.jpg)
0 1 2 3 5 6 7 8 9 10 11 12 14 15 16 17 18 19
Median: 3.23s
R U MS Y N T H E T I C
F I R S TS Y N T H E T I C
R E P E AT
3 . 2 3 4 . 1 2 . 2
W H Y A R E T H E T W O N U M B E R S S O FA R O F F ?
![Page 17: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/17.jpg)
“ B U T, I T L O A D S S O M U C H FA S T E R F O R M E ? ? ! ! ”
• 2015 Macbook Pro
• Warm browser cache
• FIOS
vs.
• x86 - Windows 7 VM
• Completely cold cache/dns
• Throttled bandwidth
4 seconds 20 seconds
![Page 18: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/18.jpg)
R U M S Y N T H E T I C
• Primary source of truth
• User behavior/conversion
• Complete coverage
• Diagnostics
• Page Construction
• Pre-production & Competitive benchmarking
![Page 19: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/19.jpg)
R U M S Y N T H E T I C
• Primary source of truth
• User behavior/conversion
• Complete coverage
• Diagnostics
• Page Construction
• Pre-production & Competitive benchmarking
You need both
![Page 20: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/20.jpg)
S TA R T R E N D E R 1 . 5 8 S
D N S 1 1 0 M S
T C P 4 0 M S
T T F B 6 5 0 M S
D O M L O A D I N G 6 8 0 M S
D O M R E A D Y 2 . 7 5 S
PA G E L O A D 3 . 0 5
F U L LY L O A D E D 1 9 . 6 8 S
U S E R T I M I N G 1 . 9 5 S
R E S O U R C E T I M I N G 2 3 7 M S
R E Q U E S T S 1 1 6
B Y T E S I N 2 , 1 4 3 K B
S P E E D I N D E X 3 0 8 6
PA G E S P E E D S C O R E
8 31 S = $ 2 7 M
D O M E L E M E N T S
3 9 3 1
D O M S I Z E 3 , 1 4 3 K B
V I S U A L LY C O M P L E T E
1 5 . 2 S
R E D I R E C T 0 S
S S L N E G O T I AT I O N
9 3 M S
T H I S I S Y O U R “ O N E N U M B E R ”
![Page 21: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/21.jpg)
T H E M A N Y FA C E S O F P E R F O R M A N C E
C-Level Operations Developer Designer
thenounproject.com
![Page 22: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/22.jpg)
S TA R T R E N D E R D N S T C P T T F B
D O M L O A D I N G D O M R E A D Y PA G E L O A D F U L LY L O A D E D
U S E R T I M I N GR E S O U R C E
T I M I N GR E Q U E S T S B Y T E S I N
S P E E D I N D E XPA G E S P E E D
S C O R E1 S = $ $
D O M E L E M E N T S
D O M S I Z EV I S U A L LY
C O M P L E T ER E D I R E C T
S S L N E G O T I AT I O N
S TA R T R E N D E R D N S T C P T T F B
D O M L O A D I N G D O M R E A D Y PA G E L O A D F U L LY L O A D E D
U S E R T I M I N GR E S O U R C E
T I M I N GR E Q U E S T S B Y T E S I N
S P E E D I N D E XPA G E S P E E D
S C O R E1 S = $ $
D O M E L E M E N T S
D O M S I Z EV I S U A L LY
C O M P L E T ER E D I R E C T
S S L N E G O T I AT I O N
S TA R T R E N D E R D N S T C P T T F B
D O M L O A D I N G D O M R E A D Y PA G E L O A D F U L LY L O A D E D
U S E R T I M I N GR E S O U R C E
T I M I N GR E Q U E S T S B Y T E S I N
S P E E D I N D E XPA G E S P E E D
S C O R E1 S = $ $
D O M E L E M E N T S
D O M S I Z EV I S U A L LY
C O M P L E T ER E D I R E C T
S S L N E G O T I AT I O N
S TA R T R E N D E R D N S T C P T T F B
D O M L O A D I N G D O M R E A D Y PA G E L O A D F U L LY L O A D E D
U S E R T I M I N GR E S O U R C E
T I M I N GR E Q U E S T S B Y T E S I N
S P E E D I N D E XPA G E S P E E D
S C O R E1 S = $ $
D O M E L E M E N T S
D O M S I Z EV I S U A L LY
C O M P L E T ER E D I R E C T
S S L N E G O T I AT I O N
![Page 23: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/23.jpg)
S TA R T R E N D E R 1 . 5 8 S
D N S 1 1 0 M S
T C P 4 0 M S
T T F B 6 5 0 M SD O M L O A D I N G
6 8 0 M SD O M R E A D Y
2 . 7 5 SPA G E L O A D
3 . 0 5F U L LY L O A D E D
1 9 . 6 8 SU S E R T I M I N G 1 . 9 5 S
R E S O U R C E T I M I N G 2 3 7 M S
R E Q U E S T S 1 1 6
B Y T E S I N 2 , 1 4 3 K BS P E E D I N D E X
3 0 8 6PA G E S P E E D S C O R E
8 31 S = $ 2 7 M D O M E L E M E N T S
3 9 3 1D O M S I Z E 3 , 1 4 3 K B
V I Z C O M P L E T E 1 5 . 2 S
R E D I R E C T 0 S
S S L N E G O T I AT I O N 9 3 M S
Analysis
![Page 24: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/24.jpg)
S TA R T R E N D E R 1 . 5 8 S
D N S 1 1 0 M S
T C P 4 0 M S
T T F B 6 5 0 M SD O M L O A D I N G
6 8 0 M SD O M R E A D Y
2 . 7 5 SPA G E L O A D
3 . 0 5F U L LY L O A D E D
1 9 . 6 8 SU S E R T I M I N G 1 . 9 5 S
R E S O U R C E T I M I N G 2 3 7 M S
R E Q U E S T S 1 1 6
B Y T E S I N 2 , 1 4 3 K BS P E E D I N D E X
3 0 8 6PA G E S P E E D S C O R E
8 31 S = $ 2 7 M D O M E L E M E N T S
3 9 3 1D O M S I Z E 3 , 1 4 3 K B
V I Z C O M P L E T E 1 5 . 2 S
R E D I R E C T 0 S
S S L N E G O T I AT I O N 9 3 M S
Analysis
![Page 25: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/25.jpg)
– C T O
“What is the right number for me to communicate to the organization?”
![Page 26: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/26.jpg)
R E A L I S T I C A L LY, T H E R E I S N O T O N E N U M B E R
1s=$27M Page Load 3.05s
Start Render 1.58s
Speed Index 3086
![Page 27: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/27.jpg)
– C E O
“How is site performance impacting revenue?”
![Page 28: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/28.jpg)
![Page 29: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/29.jpg)
![Page 30: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/30.jpg)
N O T A L L PA G E S A R E C R E AT E D E Q U A LT O P O F F U N N E L I M PA C T ( B R O W S E PA G E S )
For a typical eCommerce site, conversion rate
drops by up to 50% when “browse”
pages increase from 1 to 6 seconds
![Page 31: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/31.jpg)
N O T A L L PA G E S A R E C R E AT E D E Q U A LB O T T O M O F F U N N E L I M PA C T ( C H E C K O U T PA G E S )
However, there is much less impact
to conversion when “checkout” pages degrade
![Page 32: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/32.jpg)
H O W FA S T S H O U L D I B E ?
1s = $27M
![Page 33: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/33.jpg)
– C E O
“How do we compare to the competition?”
![Page 34: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/34.jpg)
![Page 35: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/35.jpg)
![Page 36: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/36.jpg)
– C E O
“How is our site doing in Spain? Are we winning there?”
![Page 37: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/37.jpg)
S I T E P E R F O R M A N C E - S PA I N
• Represents 1% of our overall traffic
• 3rd overall in our competitive index
3.91s Competitive Index
Site 1 Site 2 Site 3 Site 4 Site 5
![Page 38: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/38.jpg)
– V P E C O M M E R C E
“How are third parties impacting my site?”
![Page 39: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/39.jpg)
Dom
ains
Alexa Top 25 Retail
31
10
35
23
86
41
20
37
303131
4
48
3842
16
11
6
28
68
31
36
4
198
2125133511225324323517
37
17
Primary domains 3rd Party domains
3 R D PA R T Y S T U D Y - U N I Q U E D O M A I N S P E R PA G E
Domains: Median - 32 Min - 25 Max - 87
![Page 40: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/40.jpg)
91%
9%
91% of domains are 3rd parties
3 R D PA R T Y S T U D Y
![Page 41: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/41.jpg)
H U F F I N G T O N P O S T - T H I R D PA R T Y I M PA C T U S I N G P E T T Y C A S H
![Page 42: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/42.jpg)
S E T T I N G M E A N I N G F U L S L A S
Example SLA: Response time measured using resource timing from Chrome browsers in the United States should not exceed a median (50th percentile) of 100ms or a 95th percentile of 500ms for a population of more than 500 users in a 24 hour period
![Page 43: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/43.jpg)
– F R O N T E N D D E V E L O P E R
“How should I measure my site in pre-production?”
![Page 44: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/44.jpg)
![Page 45: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/45.jpg)
![Page 46: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/46.jpg)
– V P E C O M M E R C E
“Should I optimize my site for mobile?”
![Page 47: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/47.jpg)
Platform Median 95th Percentile 98th Percentile
Mobile 3.62s 12.53s 20.04s
Desktop 2.44s 9.31s 15.86s
Page Load TimesDesktop vs. Mobile (Mobile and Tablet Optimized)
![Page 48: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/48.jpg)
0 1.5 3 4.5 6 7.5 9 10.5 12 13.5 15 16.5 18 19.5
Platform Median 95th Percentile 98th Percentile
Mobile 4.70s 17.87s 34.43s
Desktop 3.81s 25.15 55.34s
Page Load TimesDesktop vs. Mobile (Full site on Tablet)
![Page 49: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/49.jpg)
1024 1366320 1920
IMG
JSCSS
35303530
![Page 50: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/50.jpg)
768 1280
1770
637
320 1920
IMG
JS
![Page 51: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/51.jpg)
– O P E R AT I O N S
“How can I determine the effectiveness of my CDN provider?”
![Page 52: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/52.jpg)
W H I C H S I T E I S U S I N G A C D N ?
Site A Site B
![Page 53: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/53.jpg)
– F R O N T E N D D E V E L O P E R
“How has page construction changed over the last month, and what impact has that had on performance?”
![Page 54: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/54.jpg)
![Page 55: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/55.jpg)
– F R O N T E N D D E V E L O P E R
“What is the performance of my single page web application?”
![Page 56: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/56.jpg)
T H E C H A L L E N G E O F S I N G L E - PA G E A P P L I C AT I O N S
• Challenges with synthetic: Only measure first page load, not feasible to measure secondary loads via XHR
• Challenges with RUM: Determining whether call is a “page” or service call;Risk of diluting overall numbers
![Page 57: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/57.jpg)
– D E S I G N E R
“How do I know how fast my page becomes usable?”
![Page 58: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/58.jpg)
Start Render Loaded
Usable
![Page 59: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/59.jpg)
• Other questions?
![Page 60: Synthetic and RUM - Best of bo](https://reader030.vdocuments.net/reader030/viewer/2022020306/559434a21a28ab911a8b468f/html5/thumbnails/60.jpg)
S U M M A R Y
• You need rich sets of data
• There is not one number
• Choose the right number(s) for the right audience