make your data talk analytic techniques for identifying h idden p erformance i ssues
DESCRIPTION
Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues. Steve Fox: Foglight Product Manager Rob Dickinson: Dir. Software Development. Contextual relevance of performance. Application Performance Monitoring. Experiment. Experiment: Group A. Experiment. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/1.jpg)
1
Make Your Data TalkAnalytic Techniques for Identifying Hidden Performance IssuesSteve Fox: Foglight Product ManagerRob Dickinson: Dir. Software Development
![Page 2: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/2.jpg)
2
Contextual relevance of performance
Application Performance Monitoring
![Page 3: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/3.jpg)
3
Experiment
Application Performance Monitoring
![Page 4: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/4.jpg)
4
Experiment: Group A
Application Performance Monitoring
![Page 5: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/5.jpg)
5
Experiment
Application Performance Monitoring
![Page 6: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/6.jpg)
6
Experiment: Group B
Application Performance Monitoring
![Page 7: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/7.jpg)
7
Experiment
Application Performance Monitoring
![Page 8: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/8.jpg)
8
Experiment: What do you see?
Application Performance Monitoring
![Page 9: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/9.jpg)
9
Experiment: Why such different answers?
Application Performance Monitoring
![Page 10: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/10.jpg)
10
Experiment #2
Application Performance Monitoring
![Page 11: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/11.jpg)
11
Experiment #2
Application Performance Monitoring
Velocity Santa Clara is cool!
![Page 12: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/12.jpg)
12
Experiment #2
Application Performance Monitoring
![Page 13: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/13.jpg)
13
Lesson: We are not always as objective as we think
Application Performance Monitoring
Hypotheses
Collect Data
Analyze
Conclusions
Scientific Method
![Page 14: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/14.jpg)
14 Dell Restricted Software
See in Diagnosing Performance Problems
Who is having Issue?
How bad is it?
Are all the users from a particular location?
Are other users using the app ok?
A few dozen users called complaining
From 15 to 60 seconds
No, scattered about the country
Yes, average response time is 1.2 seconds
![Page 15: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/15.jpg)
15
Division of Labor
Application Performance Monitoring
Hypotheses
Collect Data
Analyze
Conclusions
![Page 16: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/16.jpg)
16
Division of Labor
Application Performance Monitoring
Hypotheses
Collect Data
Analyze
Conclusions
What if?
![Page 17: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/17.jpg)
17
Hypotheses
Collect Data
Analyze
Conclusions
Division of Labor
Application Performance Monitoring
What if?
![Page 18: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/18.jpg)
18
Challenge #1: Problems Hide in Aggregations
• Averages misleading• Majority of users fine, but
problem still exists• Often outliers give more clues
than averages or aggregates• Solution
• Capture and store discrete transactional data
Application Performance Monitoring
![Page 19: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/19.jpg)
19
Challenge #2: Complexity and Volume of Data
• Saving everything creates lots of “big data”
• Entire data set too large to look for patterns manually
• Solution: Only look at problem transactions
• Look for patterns and commonalities in the problem transactions
Application Performance Monitoring
![Page 20: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/20.jpg)
Real World Example• VP reports many user complaints. Scanning
problem transaction data shows every one originates from an IE 9 browser
Solved! A browser version issue!Or is it?
• Turns out IE 9 is a corporate enforced standard and this application is only accessed by employees
• Did the scientific method fail us?• Where did we go wrong?
Software
![Page 21: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/21.jpg)
21
What went wrong?
• Our test of “IE 9 is the determining factor of the problem” was flawed
• We made assumptions about distribution of traffic that were incorrect
• Easy to do, especially under crisis-mode pressure• Even easier to do if only looking at the problem data• Happens in other operations disciplines too
Application Performance Monitoring
![Page 22: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/22.jpg)
22
Interpreting Causality from Data Analysis
• Causality and Correlation are related but different things
• Have to be very careful about hidden reasons behind otherwise interesting patterns
• In the heat of crisis, easy to fall into trap of “Believing what feels to be true”
Application Performance Monitoring
![Page 23: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/23.jpg)
23
Mexican Lemons Prevent Highway Death
Application Performance Monitoring
![Page 24: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/24.jpg)
Heart Disease• Studies showed women on hormone
replacement therapy (HRT) have lower than average incidence of heart disease.
• Randomized control trials showed HRT actually slightly increased incidence of heart disease
• Analysis showed women on HRT are more likely to be from higher social-economic group with better than average diet, exercise, and medical care
• Termed “Lurking variable”: a hidden 3rd variable that causes the correlation
Software
![Page 25: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/25.jpg)
25
So back to our performance problem
• What if it had been a public facing site with normal browser distribution?• Much more compelling correlation
• We discovered a more accurate indicator of likely performance issues was to analyze ratios in the problem set and compare to ratios in the total set of traffic• We call it: Differential Analytics
Application Performance Monitoring
![Page 26: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/26.jpg)
26
Differential Analytics Definition
Application Performance Monitoring
Performance Differential : 𝑃𝐷=𝑅𝐷−𝑅𝑇
𝑅𝑇=% 𝑜𝑓 𝑟𝑒𝑞𝑢𝑒𝑠𝑡𝑠 h𝑡 𝑎𝑡 h𝑎𝑑 h𝑡 𝑖𝑠𝑑𝑖𝑚𝑒𝑛𝑠𝑖𝑜𝑛𝑎𝑙𝑣𝑎𝑙𝑢𝑒𝑅𝐷=% 𝑜𝑓 𝑝𝑜𝑜𝑟 𝑝𝑒𝑟𝑓𝑜𝑟𝑚𝑖𝑛𝑔𝑟𝑒𝑞𝑢𝑒𝑠𝑡𝑠 h𝑡 𝑎𝑡 h𝑡 𝑖𝑠 𝑑𝑖𝑚𝑒𝑛𝑠𝑖𝑜𝑛𝑎𝑙𝑣𝑎𝑙𝑢𝑒
The higher the performance differential the more statistically likely having this value in this dimension is strongly correlated to the problem
![Page 27: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/27.jpg)
27
Differential Analytics: IE 9 Internal Example
Application Performance Monitoring
Performance Differential for IE 9 : 𝑃𝐷=100−100
𝑅𝑇=100% 𝑜𝑓 𝑟𝑒𝑞𝑢𝑒𝑠𝑡𝑠 h𝑡 𝑎𝑡 h𝑎𝑑 𝐼𝐸9 𝑎𝑠𝑏𝑟𝑜𝑤𝑠𝑒𝑟𝑅𝐷=100% 𝑜𝑓 𝑝𝑜𝑜𝑟 𝑝𝑒𝑟𝑓𝑜𝑟𝑚𝑖𝑛𝑔𝑟𝑒𝑞𝑢𝑒𝑠𝑡𝑠h𝑎𝑑 𝐼𝐸9𝑎𝑠𝑏𝑟𝑜𝑤𝑠𝑒𝑟
No correlation at all.
𝑃𝐷=0
![Page 28: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/28.jpg)
28
Differential Analytics: IE 9 External Example
Application Performance Monitoring
Performance Differential for IE 9 : 𝑃𝐷=100−6
𝑅𝑇=6% 𝑜𝑓 𝑟𝑒𝑞𝑢𝑒𝑠𝑡𝑠 h𝑡 𝑎𝑡 h𝑎𝑑 𝐼𝐸 9𝑎𝑠𝑏𝑟𝑜𝑤𝑠𝑒𝑟𝑅𝐷=100% 𝑜𝑓 𝑝𝑜𝑜𝑟 𝑝𝑒𝑟𝑓𝑜𝑟𝑚𝑖𝑛𝑔𝑟𝑒𝑞𝑢𝑒𝑠𝑡𝑠h𝑎𝑑 𝐼𝐸9𝑎𝑠𝑏𝑟𝑜𝑤𝑠𝑒𝑟
Highly correlated.
𝑃𝐷=94
![Page 29: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/29.jpg)
29
Differential Analytics: IE 9 External Realistic Example
Application Performance Monitoring
Performance Differential for IE 9 : 𝑃𝐷=22−6
𝑅𝑇=6% 𝑜𝑓 𝑟𝑒𝑞𝑢𝑒𝑠𝑡𝑠 h𝑡 𝑎𝑡 h𝑎𝑑 𝐼𝐸 9𝑎𝑠𝑏𝑟𝑜𝑤𝑠𝑒𝑟𝑅𝐷=22% 𝑜𝑓 𝑝𝑜𝑜𝑟 𝑝𝑒𝑟𝑓𝑜𝑟𝑚𝑖𝑛𝑔 𝑟𝑒𝑞𝑢𝑒𝑠𝑡𝑠h𝑎𝑑𝐼𝐸 9𝑎𝑠𝑏𝑟𝑜𝑤𝑠𝑒𝑟
Suspicious, but there are likely other factors or multiple problems active at the same time
𝑃𝐷=18
![Page 30: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/30.jpg)
• Powerful technique to apply manually
• Automation takes the power to a new level
• What if we calculated automatically for:• Device types, OS• Locations, ISP’s• For individual URL’s
• What if we included back end dimensions?• Method calls, SQL Queries
• What if we did the same set of dimensions for errors too?
Software
![Page 31: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/31.jpg)
31 Software
In a matter of seconds we evaluate 1000’s of possible correlations
For both performance issues and errors
Rank and display top correlations
Automates and streamlines the hypothesis phase of troubleshooting
Guessing / Disproving 1000’s of possibilities at a time
31
Differential Analytics
Software
![Page 32: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/32.jpg)
32
Demo
Application Performance Monitoring
![Page 33: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/33.jpg)
33
Demo
Application Performance Monitoring
![Page 34: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/34.jpg)
34 Application Performance Monitoring
![Page 35: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/35.jpg)
35
Technology to Support Differential Analytics
• Only works if you have “Good” transactions to compare against• Equivalent of “randomized control study” in medicine• Heavy big data processing requirements, difficult to do
• Simplification option of only storing problem transactions falls short• No “good” transactions to compare dimensional data on• Analysis cannot reach “randomized control study” quality
• If only storing aggregates, you are losing valuable outlier data
Application Performance Monitoring
![Page 36: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/36.jpg)
36
Differential Analytics
Application Performance Monitoring
Hypotheses
Collect Data
Analyze
Conclusions
![Page 37: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/37.jpg)
37
Investigative Analytics
• Differential Analytics answers a specific question:• What are the most correlated characteristics of my problem
requests?• What about other questions we might be interested in?
• Which requests are consuming the most database resources?
• How much time is consumed in front-end vs. back end?• Which app tier method is called the most frequently?
• from a mobile device?• In the US?• By which browser type?
Application Performance Monitoring
![Page 38: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/38.jpg)
38
Investigative Analytics:
• Differential Analytics across multiple dimensions• Many analytics questions span multiple dimensions• Traditional presentation styles limited to 2 or 3 dimensions
Application Performance Monitoring
![Page 39: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/39.jpg)
39
Investigative Analytics: Scatterplot
• 2 Dimensions: Response Time vs Time
Application Performance Monitoring
![Page 40: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/40.jpg)
40
Investigative Analytics: Geographic Map
• 2 Dimensions: Hit Count by City
Application Performance Monitoring
![Page 41: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/41.jpg)
41
Investigative Analytics: Scatterplot
• How add more dimensions?
Application Performance Monitoring
![Page 42: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/42.jpg)
42
Investigative Analytics: Scatterplot + Color
• 3 Dimensions: Time, Response Time, Device Type
Application Performance Monitoring
![Page 43: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/43.jpg)
43
Investigative Analytics: Visual Filter Chain
• How traditionally handle?• Screen to screen navigation of 2 dimensional views• Or many independent 2 dimensional views in portal style
• Visual Filter Chain• Chain together 2 dimensional visualizations on one screen• Each one can provide dynamic selection• The resulting data set is fed to downstream visualizations• No limit to the number of dimensions represented
Application Performance Monitoring
![Page 44: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/44.jpg)
44 Application Performance Monitoring
Investigative Analytics: Multiple Dimension Analytics
![Page 45: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/45.jpg)
45 Application Performance Monitoring
![Page 46: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/46.jpg)
46
Investigative Analytics: Example Analysis
Application Performance Monitoring
![Page 47: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/47.jpg)
47 Application Performance Monitoring
![Page 48: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/48.jpg)
48 Application Performance Monitoring
![Page 49: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/49.jpg)
49 Application Performance Monitoring
![Page 50: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/50.jpg)
50 Application Performance Monitoring
![Page 51: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/51.jpg)
51 Application Performance Monitoring
![Page 52: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/52.jpg)
52 Application Performance Monitoring
![Page 53: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/53.jpg)
53 Application Performance Monitoring
![Page 54: Make Your Data Talk Analytic Techniques for Identifying H idden P erformance I ssues](https://reader036.vdocuments.net/reader036/viewer/2022062501/56815fe8550346895dceec21/html5/thumbnails/54.jpg)
54
Ad Hoc Analytics to Investigate Anything
Application Performance Monitoring
Hypotheses
Collect Data
Analyze
Conclusions