VLSI Systems Design—Experiments

Necessary steps:

• Explore the problem space• Design experiment(s)• Carry out experiment(s)• Analyze results

software packages: R, Matlab, …• Report results

Example: design a “better” transistor

What do we mean by “better”?

What FACTORS influence design?

--fabrication

--design

--environmental

For which of these is there random variation?

Which “factors” do we want to investigate?

SUMMARY—15 IMPORTANT POINTS FOR EXPERIMENTERS:1. Even careful experimentation and observation may miss important facts; new experiments may cause old conclusions to be discarded; EXPERIMENTS ARE NOT PROOFS.

2. It is just as important to report NEGATIVE results as to report POSITIVE results. The experimenter must always accurately record and thoroughly report ALL results.

3. IGNORING IMPORTANT FACTORS CAN LEAD TO ERRONEOUS CONCLUSIONS, SOMETIMES WITH TRAGIC RESULTS.

4. YOUR RESULTS ARE ONLY VALID FOR THE PART OF THE DATA-TREATMENT SPACE YOU HAVE EXPLORED; YOU CANNOT CLAIM KNOWLEDGE OF WHAT YOU HAVE NOT EXPLORED

5. An experiment is worthless unless it can be REPEATED by other researchers using the same experimental setup; experimenters have a duty to the research community to report enough about their experiment and data so that other researchers can verify their claims6. YOU ONLY GET ANSWERS TO THE QUESTIONS YOU ASK

7. if your are going to use a (pseudo-)RANDOM NUMBER GENERATOR, make sure the output behaves enough like a sequence of TRUE RANDOM NUMBERS

8. An experiment must be repeated a SUFFICIENT NUMBER OF TIMES for the results to be attributed to more than random error

9. Choosing the CORRECT MEASURE for the question you are asking is an important part of the experimental design

10. Reporting CORRECT results, PROPERLY DISPLAYED, is an integral part of a well-done experiment

11. MISUSE OF GRAPH LABELING can lead to MISLEADING RESULTS AND INCORRECT CONCLUSIONS

12. INTERPOLATING your results to regions you have not explored can lead to INCORRECT CONCLUSIONS

13. IGNORING the “NULL HYPOTHESIS” when reporting your results can be very misleading

14. Don’t mistake CORRELATION for DEPENDENCE

15. Justify your choice of CURVE using VALID STATISTICS, not “appearance”

Topics

• Analyzing and Displaying Data– Simple Statistical Analysis– Comparing Results– Curve Fitting

• Designing Experiments: Factorial Designs– 2K Designs Including Replications– Full Factorial Designs

• Ensuring Data Meets Analysis Criteria• Presenting Your Results; Drawing Conclusions

Example: A System

System (“Black Box”)

SystemInputs

SystemOutputs

Factors(Experimental Conditions)

Responses(Experimental Results)

Experimental ResearchDefineSystemDefineSystem

IdentifyFactors

and Levels

IdentifyFactors

and Levels

IdentifyResponse(s)

IdentifyResponse(s)

● Define system outputs first● Then define system inputs● Finally, define behavior (i.e., transfer function)

● Identify system parameters that vary (many)● Reduce parameters to important factors (few)● Identify values (i.e., levels) for each factor

● Identify time, space, etc. effects of interest

DesignExperiments

DesignExperiments

● Identify factor-level experiments

Create and Execute System; Analyze Data

DefineWorkloadDefine

Workload

CreateSystemCreateSystem

ExecuteSystem

ExecuteSystem

● Workload can be a factor (but often isn't)● Workloads are inputs that are applied to system

● Create system so it can be executed● Real prototype● Simulation model● Empirical equations

● Execute system for each factor-level binding● Collect and archive response data

Analyze &Display

Data

Analyze &Display

Data

● Analyze data according to experiment design● Evaluate raw and analyzed data for errors● Display raw and analyzed data to draw conclusions

Some ExamplesAnalog Simulation

– Which of three solvers is best?

– What is the system?– Responses

• Fastest simulation time

• Most accurate result• Most robust to types

of circuits being simulated

– Factors• Solver• Type of circuit model• Matrix data structure

Epitaxial growth– New method using non-

linear temp profile

– What is the system?

– Responses

• Total time

• Quality of layer

• Total energy required

• Maximum layer thickness

– Factors

• Temperature profile

• Oxygen density

• Initial temperature

• Ambient temperature

Basic Descriptive Statistics for a Random Sample X

• Mean• Median• Mode• Variance / standard deviation• Z scores: Z = (X – mean)/ (standard deviation) • Quartiles, box plots• Q-Q plot

Note: these can be deceptive. For example, ifP (X = 0) = P(X = 100) = 0.5 and P (Y = 50 ) = 1,Then X and Y have the same mean (and nastier examples can be constructed)

home.oise.utoronto.ca/~thollenstein/Exploratory%20Data%20Analysis.ppt

SIMPLE MODELS OF DATA

Ms. # Latency1 222 233 194 185 156 207 268 179 1910 17

Data file “latency.dat”

Example: Evaluation of a new wireless network protocolSystem: wireless network with new protocol Workload:

10 messages applied at single sourceEach message identical configuration

Experiment output:Roundtrip latency per message (ms)

Mean: 19.6 msVariance: 10.71 ms2

Std Dev: 3.27 ms

Hypothesis: Distribution is N(2)

Verify Model PreconditionsCheck randomness

Use plot of residuals around meanResiduals “appear” random

Check normal distribution Use quantile-quantile (Q-Q) plot

Pattern adheres consistently alongideal quantile-quantile line

http://itl.nist.gov/div898/software/dataplot/refman1/ch2/quantile.pdf

Confidence Intervals

)/,/( ]2/1[]2/1[ nszxnszx aa

)/,/ ]1;2/1[]1;2/1[ nstxnstx nana

Sample mean vs Population meanIf many samples are collected, about 1 - will contain the“true mean”

CI: > 30 samples

CI: < 30 samples

For the latency data, = 10, a = 0.05:

(17.26, 21.94)

Raj Jain, “The Art of Computer Systems Performance Analysis,” Wiley, 1991.

Scatter and Line PlotsDepth Resistance

1 1.6890152 4.4867223 7.9152094 6.3623885 11.8307396 12.3291047 14.0113968 17.6000949 19.02214610 21.513802

Resistance profile of doped silicon epitaxial layer

Expect linear resistance increase as depth increases

Linear Regression Statistics(hypothesis: resistance = 0 + 1*depth + error)

model = lm(Resistance ~ Depth)summary(model)

Residuals: Min 1Q Median 3Q Max-2.11330 -0.40679 0.05759 0.51211 1.57310 Coefficients: Estimate Std. Error t value Pr(>|t|)(Intercept) -0.05863 0.76366 -0.077 0.94Depth 2.13358 0.12308 17.336 1.25e-07 ***---Signif. codes: 0 `***' 0.001 `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1 Residual standard error: 1.118 on 8 degrees of freedom “variance of error: (1.118)2”Multiple R-Squared: 0.9741, Adjusted R-squared: 0.9708 F-statistic: 300.5 on 1 and 8 DF, p-value: 1.249e-07 “evidence this estimate valid”

“reject hypotheses 0 = 0, 1 = 0”

(Using R system; based on http://www.stat.umn.edu/geyer/5102/examp/reg.html

Validating ResidualsErrors are marginally normally distributed due to “tails”

Comparing Two Sets of DataExample: Consider two different wireless access points. Which one is faster?

Inputs: same set of 10 messages communicated through both access points.

Response (usecs):

Latency1 Latency2 22 19 23 20 19 24 18 20 15 14 20 18 26 21 17 17 19 17

17 18

Approach: Take difference of data

and determine CI of difference.

If CI straddles zero, cannot tell which access point is faster.

CI95% = (-1.27, 2.87) usecs

Confidence interval straddles zero. Thus, cannot determine which is faster with 95% confidence

Plots with error bars

Execution time of SuperLU linear system solution on parallel computer

Ax = b For each p, ran problem

multiple times with same matrix size but different values Determined mean and CI

for each p to obtain curve and error intervals

Matrix density p

Curve Fitting> model = lm(t ~ poly(p,4))> summary(model) Call:lm(formula = t ~ poly(p, 4)) Residuals: 1 2 3 4 5 6 7 8 9-0.4072 0.7790 0.5840 -1.3090 -0.9755 0.8501 2.6749 -3.1528 0.9564 Coefficients: Estimate Std. Error t value Pr(>|t|)(Intercept) 236.9444 0.7908 299.636 7.44e-10 ***poly(p, 4)1 679.5924 2.3723 286.467 8.91e-10 ***poly(p, 4)2 268.3677 2.3723 113.124 3.66e-08 ***poly(p, 4)3 42.8772 2.3723 18.074 5.51e-05 ***poly(p, 4)4 2.4249 2.3723 1.022 0.364---Signif. codes: 0 `***' 0.001 `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1 Residual standard error: 2.372 on 4 degrees of freedomMultiple R-Squared: 1, Adjusted R-squared: 0.9999F-statistic: 2.38e+04 on 4 and 4 DF, p-value: 5.297e-09

Model Validation: y’ = ax + bR2 – Coefficient of Determination

“How well does the data fit your model?”What proportion of the “variability” is accounted for by the statistical model? (what is ratio of explained variation to total variation?)

Suppose we have measurements y1, y2, …, yn with mean m

And predicted values y1’, y2’, …, yn’ (yi’ = axi + b = yi + ei)

SSE = sum of squared errors = ∑ (yi – yi’)2 = ∑ei2

SST = total sum of squares =∑ (yi – m)2

SSR = SST – SSE = residual sum of squares = ∑ (m – yi’)2

R2 = SSR/SST = (SST-SSE)/SSTR2 is a measure of how good the model is.The closer R2 is to 1 the better.

Example: Let SST = 1499 and SSE = 97.

Then R2 = 93.5%http://www-stat.stanford.edu/~jtaylo/courses/stats191/notes/simple_diagnostics.pdf

Using the t-test

extra group1 0.7 12 -1.6 13 -0.2 14 -1.2 15 -0.1 16 3.4 17 3.7 18 0.8 19 0.0 110 2.0 111 1.9 212 0.8 213 1.1 214 0.1 215 -0.1 216 4.4 217 5.5 218 1.6 219 4.6 220 3.4 2

Consider the following data (“sleep.R”)

From “Introduction to R”, http://www.R-project.org

T.test result

> t.test(extra ~ group, data = sleep)

Welch Two Sample t-test data: extra by groupt = -1.8608, df = 17.776, p-value = 0.0794 alternative hypothesis: true difference in means is not equal to 095 percent confidence interval: -3.3654832 0.2054832sample estimates:mean of x mean of y 0.75 2.33

p-value is smallest 1- confidence where null hypothesis. not true. p-value = 0.0794 means difference not 0

above 92%

Factorial DesignWhat “factors” need to be taken into account?How do we design an efficient experiment to test all these factors?How much do the factors and the interactions among the factors contribute to the variation in results?

Example: 3 factors a,b,c, each with 2 values: 8 combinations

But what if we want random order of experiments?What if each of a,b,c has 3 values?Do we need to run all experiments?

http://www.itl.nist.gov/div898/handbook/pri/section3/pri3332.htm

Standard Procedure-Full Factorial Design

(Example) Variables A,B,C: each with 3 values, Low, Medium, High (coded as -1,0,1)“Signs Table”:

A B C

1 -1 -1 -1

2 +1 -1 -1

3 -1 +1 -1

4 +1 +1 -1

5 -1 -1 +1

6 +1 -1 +1

7 -1 +1 +1

8 +1 +1 +1

1.Run the experiments in the table (“2 level, full factorial design”)

2.Repeat the experiments in this order n times by using rows 1,…,8,1,…,8, … (“replication”)

3.Use step 2, but choose the rows randomly (“randomization”)

4.Use step 4, but add some “center point runs”, for example, run the case 0,0,0, then use 8 rows, then run 0,0,0, …finish with a 0,0,0 case

In general, for 5 or more factors, use a “fractional factorial design”

2k Factorial DesignExample: k = 2, factors are A,B, and X’s are computed from the signs table: y = q

0 + q

Ax

A + q

Bx

B + q

ABx

AB

SST = total variation around the mean = ∑ (y

i – mean)2

= SSA+SSB+SSAB where SSA = 22q

A2 (variation allocated to A), and SSB, SSAB

are defined similarly

Note: var(y) = SST/( 2k – 1)

Fraction of variation explained by A = SSA/SST

Example: 2k Design

Are all factors needed?If a factor has little effect on the variability of the output, why study it further?

Method? a. Evaluate variation for each factor using only two levels each b. Must consider interactions as well

Interaction: effect of a factor dependent on the levels of another

L K C Misses 32 4 mux 512 4 mux 32 16 mux 512 16 mux 32 4 lin 512 4 lin 32 16 lin 512 16 lin

Factor LevelsLine Length (L) 32, 512 wordsNo. Sections (K) 4, 16 sectionsControl Method (C) multiplexed, linear

Experiment Design

CacheAddress Trace

Misses

L K C Misses -1 -1 -1 1 -1 -1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 1 -1 1 1 1 1 1

Encoded Experiment Design

www.stat.nuk.edu.tw/Ray-Bing/ex-design/ex-design/ExChapter6.ppt

I L K C LK LC KC LKC Miss.Rate (yj) 1 -1 -1 -1 1 1 1 -1 14 1 1 -1 -1 -1 -1 1 1 22 1 -1 1 -1 -1 1 -1 1 10 1 1 1 -1 1 -1 -1 -1 34 1 -1 -1 1 1 -1 -1 1 46 1 1 -1 1 -1 1 -1 -1 58 1 -1 1 1 -1 -1 1 -1 50 1 1 1 1 1 1 1 1 86

Analyze Results (Sign Table)

qi: 40 10 5 20 5 2 3 1

= 1/∑(signi*Response

i)

SSL = 23q2L = 800

SST = SSL+SSK+SSC+SSLK+SSLC+SSKC+SSLKC = 800+200+3200+200+32+72+8 = 4512

%variation(L) = SSL/SST = 800/4512 = 17.7%

Effect % VariationL 17.7C 4.4K 70.9

LC 4.4LK 0.7CK 1.6

LCK 0.2

L K C Misses -1 -1 -1 14 1 -1 -1 22 -1 1 -1 10 1 1 -1 34 -1 -1 1 46 1 -1 1 58 -1 1 1 50 1 1 1 86

Obtain Reponses

Example: 2k Design (continued)

Ex: y1 = 14 = q0 – qL –qK –qC

+ qLK + qLC + qKC – qLKC

Solve for q’s

http://www.cs.wustl.edu/~jain/cse567-06/ftp/k_172kd/sld001.htm

Full Factorial DesignModel: y

ij = m+a

i + b

j + e

ij

Effects computed such that ∑ai = 0 and ∑b

j = 0

m = mean(y..)aj = mean(y.j) – mbi = mean(yi.) – m

Experimental ErrorsSSE = ei

2j

SS0 = abm2

SSA= b∑a2

SSB= a∑b2

SST = SS0+SSA+SSB+SSE

Full-Factorial Design ExampleDetermination of the speed of light

Morley Experiments

Factors: Experiment No. (Expt) Run No. (Run)

Levels: Expt – 5 experiments Run – 20 repeated runs

Expt Run Speed001 1 1 850002 1 2 740003 1 3 900004 1 4 1070 <more data>019 1 19 960020 1 20 960021 2 1 960022 2 2 940023 2 3 960 <more data>096 5 16 940097 5 17 950098 5 18 800099 5 19 810100 5 20 870

Box Plots of Factors

Two-Factor Full Factorial

> fm <- aov(Speed~Run+Expt, data=mm) # Determine ANOVA> summary(fm) # Display ANOVA of factors Df Sum Sq Mean Sq F value Pr(>F)Run 19 113344 5965 1.1053 0.363209Expt 4 94514 23629 4.3781 0.003071 **Residuals 76 410166 5397---Signif. codes: 0 `***' 0.001 `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1

Conclusion: Data across experiments has acceptably small variation, but variation within runs is significant

Visualizing Results: Tufte’s Principles

• Have a properly chosen format and design

• Use words, numbers, and drawing together

• Reflect a balance, a proportion, a sense of relevant scale

• Display an accessible complexity of detail

• Have a story to tell about the data

• Draw in a professional manner

• Avoid content-free decoration, including “chart junk”

Back to the transistor:

• What factors are there?

• Which ones do we want to investigate?

• How should we define our experiments?

• What role will randomness play? (simulation/actual)

• How should we report the results?