ratio games and designing experiments andy wang cis 5930-03 computer systems performance analysis
TRANSCRIPT
![Page 1: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/1.jpg)
Ratio Gamesand
Designing ExperimentsAndy WangCIS 5930-03
Computer SystemsPerformance Analysis
![Page 2: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/2.jpg)
2
Ratio Games
• Choosing a base system• Using ratio metrics• Relative performance enhancement• Ratio games with percentages• Strategies for winning a ratio game• Correct analysis of ratios
![Page 3: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/3.jpg)
3
Choosing a Base System
• Run workloads on two systems• Normalize performance to chosen
system• Take average of ratios• Presto: you control what’s best
![Page 4: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/4.jpg)
4
Example of Choosinga Base System
• (Carefully) selected Ficus results:
1 2 1/2 2/1cp 179.5 168.6 1.06 0.94rcp 286.8 338.3 0.85 1.18Mean 233.2 253.5 0.96 1.06
![Page 5: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/5.jpg)
5
Why Does This Work?
• Expand the arithmetic:
abbi
ai
b
a
b
abaiba
bbb
aba
Pyn
yn
y
y
y
y
nR
nP
Ry
yR
;;
;
;1
;1
;0
;0;;;
;;
11
1
11
0.1
![Page 6: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/6.jpg)
6
Using Ratio Metrics
• Pick a metric that is itself a ratio– E.g., power = throughput response time– Or cost/performance
• Handy because division is “hidden”
![Page 7: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/7.jpg)
7
Relative Performance Enhancement• Compare systems with incomparable bases
• Turn into ratios• Example: compare Ficus 1 vs. 2 replicas
with UFS vs. NFS (1 run on chosen day):
• “Proves” adding Ficus replica costs less than going from UFS to NFS
"cp" Time RatioFicus 1 vs. 2 197.4 246.6 1.25UFS vs. NFS 178.7 238.3 1.33
![Page 8: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/8.jpg)
8
Ratio Games with Percentages
• Percentages are inherently ratios– But disguised– So great for ratio games
• Example: Passing tests
• A is worse, but looks better in total line!
Test A RunsA Passes A % B RunsB PassesB %1 300 60 20 32 8 252 50 2 4 500 40 8
Total 350 62 18 532 48 9
![Page 9: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/9.jpg)
9
More on Percentages
• Psychological impact– 1000% sounds bigger than 10-fold (or 11-
fold)– Great when both original and final
performance are lousy• E.g., salary went from $40 to $80 per week
• Small sample sizes generate big lies• Base should be initial, not final value
– E.g., price can’t drop 400%
![Page 10: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/10.jpg)
10
Strategies for Winning
a Ratio Game• Can you win?• How to win
![Page 11: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/11.jpg)
11
Can You Winthe Ratio Game?
• If one system is better by all measures, a ratio game won’t work– But recall percent-passes example– And selecting the base lets you change the
magnitude of the difference
• If each system wins on some measures, ratio games might be possible (but no promises)– May have to try all bases
![Page 12: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/12.jpg)
12
How to WinYour Ratio Game
• For LB metrics, use your system as the base
• For HB metrics, use the other as a base• If possible, adjust lengths of
benchmarks– Elongate when your system performs best– Short when your system is worst– This gives greater weight to your strengths
![Page 13: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/13.jpg)
13
Correct Analysis of Ratios
• Previously covered in Lecture 2• Generally, harmonic or geometric
means are appropriate– Or use only the raw data
![Page 14: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/14.jpg)
14
Introduction To Experimental Design
• You know your metrics• You know your factors• You know your levels• You’ve got your instrumentation and
test loads• Now what?
![Page 15: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/15.jpg)
15
Goals in Experiment Design
• Obtain maximum information with minimum work– Typically meaning minimum number of
experiments
• More experiments aren’t better if you’re the one who has to perform them
• Well-designed experiments are also easier to analyze
![Page 16: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/16.jpg)
16
Experimental Replications
• System under study will be run with varying levels of different factors, potentially with differing workloads
• Run with particular set of levels and other inputs is a replication
• Often, need to do multiple replications with each set of levels and other inputs– Usually necessary for statistical validation
![Page 17: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/17.jpg)
17
Interacting Factors
• Some factors have completely independent effects– Double the factor’s level, halve the
response, regardless of other factors• But effects of some factors depends on
values of others– Called interacting factors
• Presence of interacting factors complicates experimental design
![Page 18: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/18.jpg)
18
The Basic Problemin Designing Experiments
• You’ve chosen some number of factors– May or may not interact
• How to design experiment that captures full range of levels?– Want minimum amount of work
• Which combination or combinations of levels (of factors) do you measure?
![Page 19: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/19.jpg)
19
Common Mistakesin Experimentation
• Ignoring experimental error• Uncontrolled parameters• Not isolating effects of different factors• One-factor-at-a-time experimental
designs• Interactions ignored• Designs require too many experiments
![Page 20: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/20.jpg)
20
Types ofExperimental Designs
• Simple designs• Full factorial design• Fractional factorial design
![Page 21: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/21.jpg)
21
Simple Designs
• Vary one factor at a time• For k factors with ith factor having ni
levels, no. of experiments needed is:
• Assumes factors don’t interact– Even then, more effort than required
• Don’t use it, usually
k
iinn
1
11
![Page 22: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/22.jpg)
22
Full Factorial Designs
• Test every possible combination of factors’ levels
• For k factors with ith factor having ni levels:
• Captures full information about interaction• But a huge amount of work
k
iinn
1
![Page 23: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/23.jpg)
23
Reducing the Work inFull Factorial Designs
• Reduce number of levels per factor– Generally good choice– Especially if you know which factors are
most important• Use more levels for those
• Reduce number of factors– But don’t drop important ones!
• Use fractional factorial designs
![Page 24: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/24.jpg)
24
Fractional Factorial Designs
• Only measure some combination of levels of the factors
• Must design carefully to best capture any possible interactions
• Less work, but more chance of inaccuracy
• Especially useful if some factors are known to not interact
![Page 25: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/25.jpg)
25
2k Factorial Designs
• Used to determine effect of k factors– Each with two alternatives or levels
• Often used as preliminary to larger performance study– Each factor measured at its maximum and
minimum level– Might offer insight on importance and
interaction of various factors
![Page 26: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/26.jpg)
26
Unidirectional Effects
• Effects that only increase as level of a factor increases– Or vice versa
• If system known to have unidirectional effects, 2k factorial design at minimum and maximum levels is useful
• Shows whether factor has significant effect
![Page 27: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/27.jpg)
27
22 Factorial Designs
• Two factors with two levels each• Simplest kind of factorial experiment
design• Concepts developed here generalize• Regression can easily be used
![Page 28: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/28.jpg)
28
22 Factorial Design Example
• Consider parallel operating system• Goal is fastest possible completion of a
given program• Quality usually expressed as speedup• We’ll use runtime as metric (simpler but
equivalent)
![Page 29: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/29.jpg)
29
Factors and Levelsfor Parallel OS
• First factor: number of CPUs– Vary between 8 and 64
• Second factor: use of dynamic load management– Migrates work between nodes as load
changes
• Other factors possible, but ignore them for now
![Page 30: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/30.jpg)
30
Defining Variables for22 Factorial OS Example
1
1Ax if 8 nodes
if 64 nodes
1
1Bx if no dynamic load mgmt
if dynamic load mgmt
![Page 31: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/31.jpg)
31
Sample Datafor Parallel OS
• Single runs of one benchmark
DLM
NO
DLM
8 Nodes 64 Nodes
820
776 197
217
![Page 32: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/32.jpg)
32
Regression Modelfor Example
• y = q0 + qAxA + qBxB + qABxAxB
• Note that model is nonlinear!
820 = q0 – qA – qB + qAB
217 = q0 + qA – qB – qAB
776 = q0 – qA + qB – qAB
197 = q0 + qA + qB + qAB
![Page 33: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/33.jpg)
33
Solving the Equations
• 4 equations in 4 unknowns• q0 = 502.5• qA = -295.5• qB = -16• qAB = 6• So y = 502.5 – 295.5xA – 16xB + 6xAxB
![Page 34: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/34.jpg)
34
The Sign Table Method
• Write problem in tabular formI A B AB y1 -1 -1 1 8201 1 -1 -1 2171 -1 1 -1 7761 1 1 1 1972010 -1182 -64 24
Total502.5 -295.5 -16 6 Total/4
![Page 35: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/35.jpg)
35
Allocation of Variationfor 22 Model
• Calculate the sample variance of y:
• Numerator is SST: total variation
SST = 22qA2 + 22qB
2 + 22qAB2
• SST explains causes of variation in y
122
2
1
2
2
2
i i
y
yys
![Page 36: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/36.jpg)
36
Terms in the SST
• 22qA2 is variation explained by effect of
A: SSA• 22qB
2 is variation explained by effect of B: SSB
• 22qAB2 is variation explained by
interaction between A and B: SSABSST = SSA + SSB + SSAB
![Page 37: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/37.jpg)
37
Variations in Our Example
• SST = 350449• SSA = 349281• SSB = 1024• SSAB = 144• Now easy to calculate fraction of total
variation caused by each effect
![Page 38: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/38.jpg)
38
Fractions of Variation in Our Example
• Fraction explained by A is 99.67%• Fraction explained by B is 0.29%• Fraction explained by interaction of A
and B is 0.04%• So almost all variation comes from
number of nodes• If you want to run faster, apply more
nodes, don’t turn on dynamic load management
![Page 39: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/39.jpg)
39
General 2k
Factorial Designs• Used to explain effects of k factors,
each with two alternatives or levels• 22 factorial designs are a special case• Same methods extend to more general
case• Many more interactions between pairs
(and trios, etc.) of factors
![Page 40: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/40.jpg)
40
Sample 23 Experiment• Sign table A, B, C are binary count;
interactions are products of columns:I A B C AB AC BC ABC y1 -1 -1 -1 1 1 1 -1 141 1 -1 -1 -1 -1 1 1 221 -1 1 -1 -1 1 -1 1 101 1 1 -1 1 -1 -1 -1 341 -1 -1 1 1 -1 -1 1 461 1 -1 1 -1 1 -1 -1 581 -1 1 1 -1 -1 1 -1 501 1 1 1 1 1 1 1 86
40 10 5 20 5 2 3 1 T/818 4.4 71 4.4 0.7 1.6 0.2 %
![Page 41: Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e735503460f94b72285/html5/thumbnails/41.jpg)
White Slide