introduction to experimental design and analysis - department of
TRANSCRIPT
Dr. John Mellor-Crummey
Department of Computer ScienceRice University
Introduction to ExperimentalDesign and Analysis
COMP 528 Lecture 11 22 February 2005
2
Experimental Design and Analysis
Understand how to• Design a experiments for measurement or simulation• Develop a model that describes the data obtained• Estimate the contribution of each factor to performance• Isolate measurement errors• Estimate confidence intervals for model parameters• Check if alternatives are significantly different• Check if a model is adequate
3
Goals for Today
Understand• What are the benefits of experimental design• Terms• Avoiding mistakes in experimental design• Basic taxonomy of experimental designs
—simple design, factorial design, fractional factorial design
• Understand 22 factorial design and its analysis—sign table method—properties—analysis—allocating variation
4
Why Experimental Design?
• Obtain maximum information from fewest experiments— minimize time spent gathering data
• Quantify effects from different factors using analysis• Determine if a factor’s effect is significant
—differences might be random variations caused by– measurement errors– parameters not controlled
5
Terms - I
• Response variable: outcome of an experiment—generally represents measured performance of the system—e.g. throughput: transactions/second, round-trip latency, etc.
• Factors: variables with alternatives that affect response—also called predictor variables or predictors—e.g. CPU type, memory size, # disk drives, workload used, etc.
• Levels: values a factor can assume—also called “treatment” in experimental design literature—e.g. CPU type levels: Itanium2, Alpha 21264, Opteron—e.g. memory size levels: 512MB, 1GB, 2GB, 4GB
• Primary factors: factors whose effects need to be quantified• Secondary factors: factors that are not being quantified
—they impact performance; we may not be interested in how much
6
Terms - II
• Replication: repetition of some or all experiments—if all experiments repeated 3x, experiment is said to have 3 replications
• Experimental design: plan for experimentation—number of experiments, factor level combinations for each, replications
• Experimental unit: any entity used for experiments—workstations, patients, land in agriculture expts—goal of experimental design: minimize impact of variation among units
• Interaction: if level of A changes effect of level change of B
86B253B1A2A1
Non-interacting
96B253B1A2A1
Interacting
lines parallel in graph of A vs. B
lines not parallel in graph of A vs. B
7
Common Mistakes in Experimentation - I
• Ignore variation due to experimental error—every measured value is a random variable—measured values change even if controllable vars kept constant—must compare variation due to factor vs. experimental error
– don’t make decision about factor’s effect without this comparison!
• Fail to control important parameters—only some parameters selected as factors and varied—must control other variables expected to have a significant effect
– e.g. if measuring impact of memory speed on performance,• use same CPU type/speed for experiments if CPU not a factor
• Fail to isolate effects of different factors—if varying several factors are being varied simultaneously
– design experiments so that effects of factors can be separated
8
Common Mistakes in Experimentation - II
• Use simple one-factor-at-a-time designs—leads to too many experiments—yields too little information per experiment—proper design ⇒narrower confidence intervals with same # expts
• Ignore interactions between factors—cannot estimate interactions with one-factor-at-a-time
experiments
• Conduct too many experiments—# experiments needed depends upon # factors, # factor levels—enormous single-step design vs. several steps
– better to use multiple steps• each with small designs and # levels
– first step: test analysis assumptions and whether transformationsrequired
– remaining steps: more factors and levels
9
Types of Experimental Designs
• Simple designs• Full factorial design• Fractional factorial design
10
• What is a simple design?— start with a typical configuration— vary one factor at a time to see how performance changes
• Example: comparing workstation configurations1. run typical configuration using a benchmark2. run experiments to pick the best CPU by varying CPU only3. using the best CPU, run a set of experiments to find the
minimum memory size yielding good performance4. using the best (CPU,memory) configuration, examine the
impact of disk RPM on the benchmark’s performance• Given: k factors; ith factor has ni levels
• Drawbacks— if factors interact, may yield wrong conclusions— fails to make best use of # experiments: statistically inefficient
Simple Designs (Not recommended)
number of experiments
!
n =1+ (ni"1)
i=1
k
#
11
Full Factorial Designs
• Explores every possible combination at all levels of factors
• Example n = (5 CPU types)(4 memory sizes)(2 disk RPMs)(4 workloads) = 160 experiments
• Advantages—thorough: every possible configuration of workload is examined
– can find effect of every factor, secondary factors, and interactions
• Disadvantages—cost: too many experiments, especially with repetitions
• Ways to reduce cost—reduce number of levels per factor (2 is very popular)—reduce # factors:
– initially only examine a few levels of each factor– prune unimportant factors, then try more factors per level
—use fractional factorial designs
number of experiments
!
n = ni
i=1
k
"
12
Fractional Factorial Designs
• A full factorial design may require many experiments• How can we get by with less: fractional factorial design• Example
—full factorial design (here, a 24 design) n = (2 CPU types)(2 memory sizes)(2 disk RPMs)(2 workloads) = 16 experiments
—fractional factorial design (here a 24-1 design)
WorkloadWorkloadWorkloadWorkloadWorkloadWorkloadWorkloadWorkload
CPUCPUCPUCPUCPUCPUCPUCPU
DiskMemoryDiskMemoryDiskMemoryDiskMemoryDiskMemoryDiskMemoryDiskMemoryDiskMemory
“half replicate design”
13
In Class Exercise
• A system’s performance depends upon the following factors—CPU type: Pentium4, Opteron, Athlon—OS type: Linux, Windows, Solaris—file compression utility: bzip2, gzip, zip
• How many experiments are necessary if:—there is significant interaction among the factors—the interactions are small compared to the main effects—there is no interaction among the factors
14
Beginning2k Factorial Designs
15
2k Factorial Designs
• What are they?—design to determine effect of k factors, each with 2 levels
• Why consider them?—easy to analyze—helps order factors based on impact—useful to identify
– factors that have significant impact ⇒ study with full factorial design– factors have little impact ⇒ not of interest for quantitative study
• How to select 2 levels—if factor effect is expected to be unidirectional ⇒ select min, max
– performance increases or decreases with factor– e.g. performance improves with more memory
16
22 Factorial Designs
• Special case of 2k factorial designs, k = 2—two factors at two levels
• Utility—22 designs are simple to analyze with regression
17
Example: a 22 Design
Consider impact of memory & cache sizes on performance
Define 2 categorical variables
Model performance using a non-linear equation in xA and xB
Solve using regression
5.2 GFLOPS2.5 GFLOPS6
3.2 GFLOPS1.5 GFLOPS4
Memory Size - 2GBMemory Size - 1GBL3 Cache Size (MB)
xA=-1 if 1GB memory+1 if 2GB memory xB=
-1 if 4MB cache+1 if 6MB cache
!
y = q0
+ qA xA + qB xB + qAB xA xB
18
Example: a 22 Design
Regress performance in GFLOPS on xA and xB
Substitute y, xA and xB
4 equations, 4 unknowns: unique solution
Interpretation—mean performance: 3.1 GFLOPS—effect of memory: 1.1GFLOPS—effect of cache: .75 GFLOPS—cache and memory interaction: .25 GFLOPS
!
y = q0
+ qA xA + qB xB + qAB xA xB
!
1.5 = q0" qA " qB + qAB
3.2 = q0
+ qA " qB " qAB
2.5 = q0" qA + qB " qAB
5.2 = q0
+ qA + qB + qAB
!
y = 3.1+1.1xA + .75xB + .25xA xB
19
!
q0
=1
4(y1
+ y2
+ y3
+ y4)
qA =1
4("y
1+ y
2" y
3+ y
4)
qB =1
4("y
1" y
2+ y
3+ y
4)
qAB =1
4(+y
1" y
2" y
3+ y
4)
Solving the equations for the qi’s
Computing Effects for 22 Design
!
y = q0
+ qA xA + qB xB + qAB xA xB
!
y1
= q0" qA " qB + qAB
y2
= q0
+ qA " qB " qAB
y3
= q0" qA + qB " qAB
y4
= q0
+ qA + qB + qAB
Substituting 4 observations into the model
Notice: expressions for qA, qB, qAB (contrasts)• are linear combinations of responses• sum of coefficients is 0
y4114y31-13y2-112y1-1-11yBAExpt
20
Calculating Effects with a Sign Table
total/4.25.751.13.1total134.412.45.22.53.21.5y
1111-11-11-1-1111-1-11
ABBAI
All possible combinations of -1, 1AB is product of columns A and By is set of observationscompute product of each column and y;write product underneathdivide through totals by 4 to compute regression coefficients
21
1111-11-11-1-1111-1-11
ABBAI
Sign Table Properties
• Sum of entries in columns A, B, AB is 0
• Sum of squares of entries in each column is 4
• Columns are orthogonal since inner product of column pairs is 0!
xAi
= 0i=1
4
"
!
xBi
= 0i=1
4
"
!
xAixBi
= 0i=1
4
"
!
xAi
2= 4
i=1
4
"
!
xAixBi( )
2
= 4i=1
4
"
!
xBi
2= 4
i=1
4
"
!
xAixAixBi( ) = 0
i=1
4
"
!
xAixBi
= 0i=1
4
"
!
xBixAixBi( ) = 0
i=1
4
"
22
Computing Sample Mean with Sign Table
!
y =1
4yi
i=1
4
"
!
y =1
4q0
+ qA xAi + qB xBi + qAB xAixBi( )i=1
4
"
sample mean
!
=1
4q0
i=1
4
" +1
4qA xAi
i=1
4
" +1
4qB xBi
i=1
4
" +1
4qAB xAixBi
i=1
4
"
!
= q0
23
Computing Total Variation with Sign Table
!
= yi " y ( )2
i=1
4
#
!
= qA xAi + qB xBi + qAB xAixBi( )2
i=1
4
"
Total variation
!
= qA xAi( )2
i=1
4
" + qB xBi( )2
i=1
4
" + qAB xAixBi( )2
i=1
4
" + product terms
!
= qA2
xAi( )2
i=1
4
" + qB2
xBi( )2
i=1
4
" + qAB2
xAixBi( )2
i=1
4
"
!
= 4qA2
+ 4qB2
+ 4qAB2
SST = SSA + SSB + SSAB =
SSA = SSB = SSAB =
!
4qB2
!
4qA2
!
4qAB2
!
4qA2
+ 4qB2
+ 4qAB2
24
Allocating Variation
Importance of a factor = how much variation it explains
SST = SSA + SSB + SSAB = 4(1.1)2 + 4(.75)2 + 4(.25)2 = 7.34
SSA 4(1.1)2
SST 7.34 Variation due to A = = = .66
SSB 4(.75)2
SST 7.34Variation due to B = = = .31
SSAB 4(.25)2
SST 7.34Variation due to interaction A & B = = = .03