msr2016 tarek
TRANSCRIPT
1
Studying the Effectiveness of Application Performance Management (APM) Tools for
Detecting Performance Regressions for Web Applications
Tse-Hsun Chen Ahmed E. Hassan Weiyi ShangTarek M. Ahmed Cor-Paul Bezemer
An Experience Report
2
3
$1.6 billion loss for a one-second slowdown
4
What is a performance regression?
Version 1 Version 2
Does version 2 have worse performance than the version 1?
5
Performance regression:changed source code + unchanged workload
change in performance?
Version 1
Version 2
requests
requests
requests
Performance metrics
Performance metrics
Unchanged workload Testing environment
Testing environment
Changed source code Change in
performance?
6
MSR techniques are widely used to detect performance regressions
7
MSR techniques are widely used to detect performance regressions
Building statistical models Mining data associationsComparing distributions
8
An example of using MSR techniques to detect performance regressions
Version 1 Version 2
Build model
Verify model
Model Prediction error
< threshold
> threshold
Shang et al. ICPE 2015
9
The extensive performance regression detection research is hardly adopted in practice
Too complex to implement by practitionersNot easily accessible
10
Application performance management (APM) tools are well adopted in practice
11
APM tools work with:unchanged source code + changing workload
requests
requests
requests
Real users
Field environment
Slow transactionsChanging workload
Unchanged source code
12
An example report from an APM tool
Most time consuming transactions
Time breakdown for the transaction
13
Performance regressions detection research
+ unchanged workload
changed source code
Not well adopted
unchanged source code +
changing workload
Widely adopted
14
Research Questions:
Can APM tools detect performance regressions?
Can APM tools be used as a vehicle to deploy performance regression detection research in practice?
15
Case study: subject systems and injected performance regressions
CPU overhead
Memory overhead
Inefficient use of database
• Unreleased resources• Inefficient use of data streams
• Unnecessary loop• Loop with a missing break
condition
• One-by-one processing• Excessive data
16
We leverage APM tools to detect injected performance regressions
Slow transactions in new version
Slow transactions in old version
Version 1
Version 2
requests
requests
requests
Unchanged workload Testing environment
Testing environment
Changed source code
17
Series10
1
2
3
4
5
6 Series1 Series2
APM tools can detect performance regressions
# of
det
ecte
d iss
ues d
etec
ted
Commercial Open source
Total 6 issues injected
18
APM tools have rooms of improvement
APM tools are a great vehicle to push performance regression detection research into practice.
Large amounts of reports for manual exploration Basic analysis
Lack of actionable suggestion
19
$1.6 billion loss for a one-second slowdown
20
21
unchanged source code +
changing workload
Widely adopted
Performance regressions detection research
changed source code +
unchanged workload
Not well adopted
22
23
Case study: subject systems and injected performance regressions
CPU overhead
Memory overhead
Inefficient use of database
• Unreleased resources• Inefficient use of data streams
• Unnecessary loop• Loop with a missing break
condition
• One-by-one processing• Excessive data
24
25
Series10
1
2
3
4
5
6 Series1 Series2
APM tools can detect performance regressions
# of
det
ecte
d iss
ues d
etec
ted
Commercial Open source
Total 6 issues injected
