![Page 1: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/1.jpg)
Mining Performance Regression Testing Repositories for Automated Performance AnalysisKing Chun Foo
Zhen Ming Jiang
Bram Adams
Ahmed E. Hassan
Ying Zou
Queen’s University, Canada
1
Parminder Flora
Research In Motion, Canada
![Page 2: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/2.jpg)
Performance Regression
• Software changes over time– New features, bug fixes– May slow down the system
• Performance regressions describe situations where the performance degrades compared to previous releases
2
![Page 3: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/3.jpg)
Performance Regression Testing
• Part of performance testing effort• A process of putting load on a system to test its
performance– Mimic the field usage which contains a mix of scenarios– Span from a few hours to a few days
• During a test, various data are collected– Execution logs– Performance metrics, e.g., CPU utilizations, response time
3
![Page 4: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/4.jpg)
Current Practice of Performance Regression Testing
• Analysts conduct a performance regression test for a new software release
1. Conduct Performance Regression Test
1. Conduct Performance Regression Test
2. Visualize Performance Metrics
2. Visualize Performance Metrics
3. Analyze Test Result3. Analyze Test Result
4. Report and Store Analysis
4. Report and Store Analysis
4
![Page 5: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/5.jpg)
Current Practice of Performance Regression Testing
• Tools check the metric averages against pre-defined thresholds
• Generate a plot for each metric
1. Conduct Performance Regression Test
1. Conduct Performance Regression Test
2. Visualize Performance Metrics
2. Visualize Performance Metrics
3. Analyze Test Result3. Analyze Test Result
4. Report and Store Analysis
4. Report and Store Analysis
5
![Page 6: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/6.jpg)
Current Practice of Performance Regression Testing
1. Conduct Performance Regression Test
1. Conduct Performance Regression Test
2. Visualize Performance Metrics
2. Visualize Performance Metrics
3. Analyze Test Result3. Analyze Test Result
4. Report and Store Analysis
4. Report and Store Analysis
• Compare plots of the new test to those from a prior test
• look for evidence of performance regressions
6
![Page 7: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/7.jpg)
Current Practice of Performance Regression Testing
• File bug report if necessary
• Archive all data and result in a central repository
1. Conduct Performance Regression Test
1. Conduct Performance Regression Test
2. Visualize Performance Metrics
2. Visualize Performance Metrics
3. Analyze Test Result3. Analyze Test Result
4. Report and Store Analysis
4. Report and Store Analysis
7
![Page 8: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/8.jpg)
Challenge 1Lots of Metrics
• 2000+ metrics for our industrial case study
• Time consuming to analyze
• Difficult to consider all correlating metrics together
• Hard to compare more than 2 tests at once
8
![Page 9: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/9.jpg)
Challenge 2No Up-to-date Performance Objectives
9
• Metric averages of the most recently passed test are used as thresholds– Miss common problems
![Page 10: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/10.jpg)
Challenge 3 Analysts’ Subjectivity
• Analysts have different perceptions of performance regressions
• Analysis may be influenced by– Analyst’s knowledge– Deadline
10
![Page 11: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/11.jpg)
Research Goal
1. Conduct Performance Regression Test
1. Conduct Performance Regression Test
2. Visualize Performance Metrics
2. Visualize Performance Metrics
4. Report and Store Analysis
4. Report and Store Analysis
11
Automate the analysis of
performance regression test
Automate the analysis of
performance regression test
3. Analyze Test Result3. Analyze Test Result
![Page 12: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/12.jpg)
Performance Signatures
Intuition: Metric correlations are the same across tests
12
RepositoryRepository
Arrival RateMedium
Arrival RateMedium
CPU Utilization Medium
CPU Utilization Medium
Throughput Medium
Throughput Medium
RAM Utilization Medium
RAM Utilization Medium
Job Queue Size Low
Job Queue Size Low
…
Performance Signatures
![Page 13: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/13.jpg)
Performance Signatures
• Performance signatures represents the expected metric correlations– Frequent violations mean abnormal behavior
• We derive performance signatures with Association Rule Mining
13
Throughput Medium
Throughput Medium
RAM Utilization Medium
RAM Utilization Medium
Job Queue Size Low
Job Queue Size Low
Premise Consequent
![Page 14: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/14.jpg)
Performance Signatures
• Confidence– Probability that the premise leads to the
consequent
• Confidence will change if the new release behaves differently from past releases– Big changes represent performance regressions
14
![Page 15: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/15.jpg)
Technique Overview
1. MetricNormalization
1. MetricNormalization
2. Metric Discretization
2. Metric Discretization
3. Performance Signature Derivation and Comparison3. Performance Signature
Derivation and Comparison4. Report
Generation4. Report
Generation
New RunNew Run
Performance Regression
Report
Performance Regression
Report
Deviation flagged?No – add to historical dataset for analysis of
future testsRepositoryRepository
15
![Page 16: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/16.jpg)
Technique Overview
1. Metric Normalization– Remove test dependent information (e.g., lab
name)
1. Metric Discretization– Put metric values into discrete levels for our data-
mining technique
16
![Page 17: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/17.jpg)
Technique Overview
3. We derive the performance signatures and measure the confidence change for each rule in the new test– Rules with large confidence change is reported
17
![Page 18: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/18.jpg)
Technique Overview
4. Generate a report of performance regressions– Contains the metrics that show performance
regressions and violated signatures– Metrics are sorted by level of severity
18
![Page 19: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/19.jpg)
Performance Regression ReportStep 1. Overview of Metrics with
Performance Regressions
19
![Page 20: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/20.jpg)
Performance Regression ReportStep 2. Explanation of Performance
Regressions
20
![Page 21: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/21.jpg)
Performance Regression ReportStep 3. Regressions Over Time
21
![Page 22: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/22.jpg)
Case Studies
• 2 Open Source Systems– Dell DVD Store and JPetStore– Manually injected bugs to the systems to simulate
performance regressions
• Enterprise System– Compare metrics flagged by our technique against
analyst’s reports– Verify with the organization metrics that are not
present in the analyst’s reports
22
![Page 23: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/23.jpg)
Case Studies Result
• Open source systems:– Precision: 75% - 100%– Recall: 52% - 67%
• Enterprise System:– Precision: 93%– Discovered new regressions that were not
included in the analysis reports
23
![Page 24: Mining Performance Regression Testing Repositories for Automated Performance Analysis](https://reader031.vdocuments.net/reader031/viewer/2022022419/5878d7da1a28ab917a8b671f/html5/thumbnails/24.jpg)
Conclusion
24