automated discovery of performance regressions in enterprise applications
TRANSCRIPT
![Page 1: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/1.jpg)
Automated Discovery of Performance Regressions in Enterprise Applications
King Chun (Derek) FooSupervisors: Dr. Jenny Zou and Dr. Ahmed E. HassenDepartment of Electrical and Computer Engineering
![Page 2: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/2.jpg)
Performance Regression
• Software changes over time– Bug fixes– Features enhancements– Execution environments
• Performance regressions describe situations where the performance degrades compared to previous releases
2
![Page 3: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/3.jpg)
Example of Performance Regression
3
Application Server Data Store
SP1 introduces a new default policy to throttle the “# of RPC/min”
• Significant increase of job queue and response time
• CPU utilization decreases
• Certification of 3rd party component
Load Generator
Application Server
Data StoreSP 1
Load Generator
![Page 4: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/4.jpg)
Current Practice of Performance Verification
Requirements
Design
Implementation
Verification
Maintenance
4
![Page 5: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/5.jpg)
FOUR CHALLENGESThe Current Practice of Performance Verification
5
![Page 6: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/6.jpg)
1. Too Late in the Development Lifecycle
• Design changes are not evaluated until after code is written
– Happens at the last stage of a delayed schedule
6
Requirements
Design
Implementation
Verification
Maintenance
![Page 7: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/7.jpg)
2. Lots of Data
• Industrial case studies have 2000> counters
• Time consuming to analyze
• Hard to compare more than 2 tests at once
7
![Page 8: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/8.jpg)
3. No Documented Behavior
8
• Analysts have different perceptions of performance regressions
• Analysis may be influenced by– Analyst’s knowledge– Deadline
![Page 9: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/9.jpg)
4. Heterogeneous Environments
• Multiple labs to parallelize test executions– Hardware and software may differ– Tests from one lab may not be used to analyze
tests from another lab
9
![Page 10: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/10.jpg)
Categorize Each Challenge
10
Design Implementation
ImplementationImplementation
![Page 11: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/11.jpg)
AT THE DESIGN LEVELPerformance Verification
11
![Page 12: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/12.jpg)
Evaluate Design Changes through Performance Modeling
• Analytical models are often not suitable for all stakeholders– Abstract mathematical and statistical concepts
• Simulation models can be implemented with support of existing framework– Visualization– No systematic approach to construct models that
can be used by different stakeholders
12
![Page 13: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/13.jpg)
Layered Simulation Model
13
Physical layerComponent layer
World view layer
Can the current infrastructure support the projected growth of users?
Investigate threading model
Hardware resource utilization
![Page 14: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/14.jpg)
Case Studies
• We conducted two case studies– RSS Cloud
• Show the process of constructed the model• Derive the bottleneck of the application
– Performance monitor for ULS systems• Evaluate whether or not an organization should re-
architect the software
• Our model can be used to extract important information and aid in decision making
14
![Page 15: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/15.jpg)
AT THE IMPLEMENTATION LEVELPerformance Verification
15
![Page 16: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/16.jpg)
Challenges with Analyzing Performance Tests
• Lots of data– Industrial case studies have 2000> counters– Time consuming to analyze– Hard to compare more than 2 tests at once
• No documented behavior– Analyst’s subjectivity
16
![Page 17: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/17.jpg)
Performance Signatures
Intuition: Counter correlations are the same across tests
17
Repository
Arrival RateMedium
CPU Utilization Medium
Throughput Medium
RAM Utilization Medium
Job Queue Size Low
…
Performance Signatures
![Page 18: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/18.jpg)
Approach Overview
18
![Page 19: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/19.jpg)
Case Studies
• 2 Open Source Applications– Dell DVD Store and JPetStore– Manually injected bugs to simulate performance
regressions• Enterprise Application
– Compare counters flagged by our technique against analyst’s reports
19
![Page 20: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/20.jpg)
Case Studies Result
• Open source applications:– Precision: 75% - 100%– Recall: 52% - 67%
• Enterprise application:– Precision: 93%– Recall: 100% (relative to the organization’s report)– Discovered new regressions that were not
included in the analysis reports
20
![Page 21: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/21.jpg)
HETEROGENEOUS ENVIRONMENTSAnalyzing Performance Tests conducted in
21
![Page 22: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/22.jpg)
Heterogeneous Environments
• Different hardware and software configurations
• Performance tests conducted in different lab exhibits different behaviors
• Must identify performance regressions from performance difference caused by heterogeneous environments
22
![Page 23: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/23.jpg)
Ensemble-based Approach
• Build a collection of models from the repository
– Each model specializes in detecting the performance regressions in a specific environment
• Reduces risks of following a single model which may contain conflicting behaviors
23
![Page 24: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/24.jpg)
Case Studies
• 2 Open Source Applications– Dell DVD Store and JPetStore– Manually injected bugs and varies
hardware/software resources• Enterprise Application
– Use existing tests conducted from different lab
24
![Page 25: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/25.jpg)
Case Studies Result
• Original approach– Precision: 80%– Recall: 50% (3-level discretization) - 60% (EW)
• Ensemble-based approach:– Precision: 80% (Bagging) - 100% (Stacking)– Recall: 80%
• Ensemble-based approach with stacking produces the best result in our experiments
25
![Page 26: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/26.jpg)
Major Contributions
• An approach to build layered simulation models to evaluate design changes early
• An automated approach to detect performance regression, allowing analysts to analyze large amount of performance data while limiting subjectivity
• An ensemble-based approach to deal with performance tests conducted in heterogeneous environments, which is common in practice
26
![Page 27: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/27.jpg)
Conclusion
27
![Page 28: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/28.jpg)
Publication
K. C. Foo, Z. M. Jiang, B. Adams, A. E. Hassan, Y. Zou, P. Flora, "Mining Performance Regression Testing Repositories for Automated Performance Analysis," Proc. Int’l Conf. on Quality Softw. (QSIC), 2010
28
![Page 29: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/29.jpg)
Future Work
• Online analysis of performance test• Compacting the performance regression
report• Maintaining the training data for our
automated analysis approach• Using performance signatures to build
performance models
29
![Page 30: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/30.jpg)
30
Logical view Development view
Process view Physical view
Scenarios view
Figure 2‑1: The "4+1" view model
![Page 31: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/31.jpg)
31
Execution of performance
regression test
Threshold-based analysis of test result
Manual analysis of test
result
Report generation
Figure 2‑2: The process of performance verification
![Page 32: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/32.jpg)
32
QN models Types of application suitable to be modeled
Open QN Applications with jobs arriving externally; these jobs will eventually depart
from the applications.
Closed QN Applications with a fixed number of jobs circulating within the applications.
Mixed QN Applications with jobs that arrive externally and jobs that circulate within
the applications.SQN-HQN
SRN Distributed applications with synchronous communication.
LQN Distributed applications with synchronous or asynchronous communication.
Table 3 1: Summary of approaches based on QN model‑
![Page 33: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/33.jpg)
33
Figure 3‑1: Open queueing network model
Figure 3‑2: Closed queueing network model
![Page 34: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/34.jpg)
34
Stakeholder Performance Concerns
End user Overall system performance for various deployment scenarios
Programmer Organization and performance of system modules
System Engineer Hardware resource utilization of the running application
System Integrator Performance of each high-level component in the application
Table 4‑1: Performance concerns of stakeholders
![Page 35: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/35.jpg)
35
Stakeholder Layer in Our Simulation Model 4+1 View ModelArchitects, Managers
End users
Sales Representatives
World View Layer Logical view
Programmers
System IntegratorsComponent Layer
Development View
Process ViewSystem Engineers Physical Layer Physical ViewAll Stakeholders Scenario Scenario
Table 4‑2: Mapping of our simulation models to the 4+1 view model
![Page 36: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/36.jpg)
36
•Physical layer•Component layer
•World view layer
Figure 4‑1: Example of layered simulation model for an RSS cloud
![Page 37: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/37.jpg)
37
Layer Component Has connection to
World view layerUsers, blogs RSS server
RSS server Users, blogs
Component layer
In Queues, out queues Application logic
Application logicInput queues, output queues,
hardware
Hardware Application logic
Physical layerHardware allocator CPU, RAM, disk
CPU, RAM, disk Hardware allocator
Table 4‑3: Components and connections in Figure 4‑1
![Page 38: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/38.jpg)
38
Resource Requirement
CPU 2 unit
RAM 5 KB
Thread 1
Processing time 2 seconds
Table 4‑4: Processing requirement for an RSS notification
![Page 39: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/39.jpg)
39
Figure 4‑2: Plot of the throughput of the RSS server at various request arrival rates
![Page 40: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/40.jpg)
40
Figure 4‑3: Plot of the response time of the RSS server at various request arrival rates
![Page 41: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/41.jpg)
41
Figure 4‑4: Plot of the hardware utilization of the RSS server at various request arrival rates
![Page 42: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/42.jpg)
42
Figure 4-5: World view layer of the performance monitor for ULS applications
![Page 43: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/43.jpg)
43
Layers Performance Data
World view layer Response time, Transmission Cost
Component layer Thread Utilization
Physical layer CPU and RAM utilization
Table 4‑5: Performance data collected per layer
CPU Util. Low OK High Very HighRange (s) < 30 30 – 60 60 – 75 > 75
Discretization 0.25 0.5 0.75 1
Table 4‑6: Categorization of CPU utilization
RAM Util. Low OK High Very HighRange (%) < 25 25 – 50 50 – 60 > 60
Discretization 0.25 0.5 0.75 1
Table 4‑7: Categorization of RAM utilization
![Page 44: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/44.jpg)
44
Data Collection Frequency
(Hz)
LayersData
Broadcast Period (s)
Response time
(s)Cost ($)
Central Monito
r Thread
Util. (%)
Central Monitor CPU Util. (%)
Central Monitor RAM
Util. (%)
0.1
World View 1 6.8 5.0 1.6 15.6 6.1
Component 1 6.8 5.0 1.6 15.6 6.1Physical 1 6.8 5.0 1.6 15.6 6.1
0.2
World View 1 7.7 5.0 4.0 40.3 15.7
Component 1 7.7 5.0 4.0 40.3 15.7Physical 7 8.9 5.3 2.3 23.4 9.2
0.3
World View 1 8.9 5.0 6.4 64.4 25.3
Component 1 8.9 5.0 6.4 64.4 25.3Physical 3 9.2 5.0 5.6 56.0 21.9
Table 4‑8: Simulation result for the performance monitor case study
![Page 45: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/45.jpg)
45
(b) Details of performance regressions
Time series plots show the periods whereperformance regressionsare detected.Box plots give a quick visual
comparison between prior tests and the new test.
Counters with performance regressions (underlined) are annotated with expected counter correlations.
(a) Overview of problematic regressions
![Page 46: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/46.jpg)
46
Figure 5‑2: Overview of performance regression analysis approach
![Page 47: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/47.jpg)
47
(a) Original counter data
(b) Counter discretization(Shaded area corresponds to the medium Discretization level)
Figure 5‑3: Counter normalization and discretization
![Page 48: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/48.jpg)
48
For each counter,
High = All values above the medium level
Medium = Median +/- 1 standard deviation
Low = All values below the medium level
Figure 5‑4: Definition of counter discretization levels
![Page 49: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/49.jpg)
49
Figure 5‑5: Example of an association rule
![Page 50: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/50.jpg)
50
# of test
scenarios
Duration per
test (hours)
Average
precisionAverage recall
DS2 4 1 100% 52%
JPetStore 2 0.5 75% 67%
Enterprise
Application13 8 93%
100%
(relative to organization’s
original analysis)
Table 5‑1: Average precision and recall
![Page 51: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/51.jpg)
51
Load generator
% Processor Time
# Orders/minute
# Network Bytes Sent/second
# Network Bytes Received/Second
Tomcat
% Processor Time
# Threads
# Virtual Bytes
# Private Bytes
MySQL
% Processor Time
# Private Bytes
# Bytes written to disk/second
# Context Switches/second
# Page Reads/second
# Page Writes/second
% Committed Bytes In Use
# Disk Reads/second
# Disk Writes/second
# I/O Reads Bytes/second
# I/O Writes Bytes/second
Table 5‑2: Summary of counters collected for DS2
![Page 52: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/52.jpg)
52
![Page 53: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/53.jpg)
53
Figure 5‑6: Performance Regression Report for DS2 test D_4 (Increased Load)
![Page 54: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/54.jpg)
54
Test
Summary of the report
submitted by the performance
analyst
Our findings
E_1 No performance problem found.
Our approach identified
abnormal behaviors in system
arrival rate and throughput
counters.
E_2
Arrival rates from two load
generators differ significantly.
Abnormally high database
transaction rate.
High spikes in job queue.
Our approach flagged the same
counters as the performance
analyst’s analysis with one false
positive.
E_3Slight elevation of # database
transactions/second.No counter flagged.
Table 5‑4: Summary of analysis for the enterprise application
![Page 55: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/55.jpg)
55
Model Counters flagged as Violation
R1 CPU utilization, throughput
R2 Memory utilization, throughput
R3 Memory utilization, throughput
R4 Database transactions/second
Table 6‑1: Counters flagged in T5 by multiple rule sets
![Page 56: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/56.jpg)
56
Counters flagged as Violation # of times flagged
Throughput 3
Memory utilization 2
CPU utilization 1
# Database transactions / second 1
Table 6‑2: Count of counters flagged as violations by individual rule set
![Page 57: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/57.jpg)
57
Performance Testing Repository New Test
T1 T2 T5
CPU 2 GHz, 2 cores 2 GHz, 2 cores2 GHz, 2
cores
Memory 2 GB 1 GB 2 GB
Database Version 1 2 1
OS Architecture 32 bit 64 bit 64 bit
Table 6‑3: Test Configurations
![Page 58: Automated Discovery of Performance Regressions in Enterprise Applications](https://reader031.vdocuments.net/reader031/viewer/2022030304/5878d79d1a28ab917a8b6673/html5/thumbnails/58.jpg)
58
Table 6‑4: Summary of performance of our approachesP represents precision, R represents recall, and F represents F-measure
(Values are rounded up to 1 significant digit)