Comparing Systems Using Sample Data

Manijeh KeshtgaryChapter 13

2

How to report the performance as a single number? Is specifying the mean the correct way?

How to report the variability of measured quantities? What are the alternatives to variance and when are they appropriate?

How to interpret the variability? How much confidence can you put on data with a large

variability? How many measurements are required to get a desired

level of statistical confidence? How to summarize the results of several different

workloads on a single computer system? How to compare two or more computer systems using

several different workloads? Is comparing the mean sufficient?

Part III: Probability Theory and Statistics

3

Sample Versus Population Confidence Interval for The Mean Approximate Visual Test One Sided Confidence Intervals Sample Size for Determining Mean

Overview

4

Old French word `essample' `sample' and `example'

One example theory One sample Definite statement

Sample

5

x

Sample Versus Population Generate several million random numbers

with mean m and standard deviation s Draw a sample of n observations: {x1, x2, …,

xn}

◦ Sample mean (x) ¹ population mean (m) Parameters: population characteristics

◦ Unknown, Use Greek letters ( , )m s Statistics: Sample estimates

◦ Random, Use English letters (x, s)

it is not possible to get a perfect estimate of the population mean from any finite number of finite size samples.

The best we can do is to get probabilistic bounds. Thus, we may be able to get two bounds, for instance, c1 and c2,

Estimate of mean

7

k samples k Sample means◦ Can't get a single estimate of m

◦ Use bounds c1 and c2: Probability{c1 m c2} = 1- ( is very small)

Confidence interval: [(c1, c2)] Significance level: ( 0.1 )a Confidence level: 100(1-a) Confidence coefficient: 1-a

Confidence Interval for The Mean

mc1 c2

8

Use 5-percentile and 95-percentile of the sample means to get 90% Confidence interval Need many samples (n > 30)

Central limit theorem: Sample mean of independent and identically distributed observations:

Where m = population mean, s = population standard deviation Standard Error: Standard deviation of the sample mean

100(1-)% confidence interval for m:

z1-a/2 = (1-a/2)-quantile of N(0,1)

Determining Confidence Interval

0-z1-a/2 z1-a/2

Example (Table A.2) For example, for a

two–sided confidence interval at 95%, α= 0.05 and p = 1 – α/2 = 0.975. The entry in the row labeled 0.97 and column labeled 0.005 gives zp = 1.960.

10

x = 3.90, s = 0.95 and n = 32 A 90% confidence interval for the mean

= We can state with 90% confidence that the

population mean is between 3.62 and 4.17.The chance of error in this statement is 10%.

Example 13.1

11

If we take 100 samples and construct confidence interval for each sample, the interval would include the population mean in 90 cases.

Confidence Interval: Meaning

mc1 c2

Total yes > 100(1-)

The preceding confidence interval applies only for large samples, that is, for samples of size greater than 30.

For smaller samples, confidence intervals can be constructed only if the observations come from a normally distributed population

100(1-a) % confidence interval for n < 30

Confidence Interval for Small Samples

13

◦ Note: can be constructed only if observations come from a normally distributed population

t[1-a/2; n-1] = (1-a/2)-quantile of a t-variate with n-1 degrees of freedom◦ Listed in Table A.4 in the Appendix

Confidence Interval for Small Samples

Table A.4 lists t[p;n]. For example, the t[0.95;13] required for a two–sided 90% confidence interval of the mean of a sample of 14 observation is 1.771

15

Sample◦ -0.04, -0.19, 0.14, -0.09, -0.14, 0.19, 0.04, and

0.09. Mean = 0, Sample standard deviation =

0.138. For 90% interval: t[0.95;7] = 1.895 Confidence interval for the mean

Example 13.2

16

Testing For A Zero Mean

17

Difference in processor times: {1.5, 2.6, -1.8, 1.3, -0.5, 1.7, 2.4}

Question: Can we say with 99% confidence that one is superior to the other?◦ Sample size = n = 7◦ Mean = 7.20/7 = 1.03◦ Sample variance = (22.84 - 7.20*7.20/7)/6 = 2.57◦ Sample standard deviation = = 1.60

t[0.995; 6] = 3.707

99% confidence interval = (-1.21, 3.27)

Example 13.3

18

Opposite signs we cannot say with 99% confidence that the mean difference is significantly different from zero

Answer: They are same Answer: The difference is zero

Example 13.3 (cont’d)

19

Difference in processor times◦ {1.5, 2.6, -1.8, 1.3, -0.5, 1.7, 2.4}.

Question: Is the difference 1? 99% Confidence interval = (-1.21, 3.27)

◦ The confidence interval includes 1 => Yes: The difference is 1 with 99% of

confidence

Example 13.4

20

Paired: one-to-one correspondence between the ith test of system A and the ith test on system B◦ Example: Performance on ith workload◦ Straightforward analysis: the two samples are

treated as one sample of n pairs◦ Use confidence interval of the difference

Unpaired: No correspondence◦ Example: n people on System A, n on System BNeed more sophisticated method

◦ t-test procedure

Paired vs. Unpaired Comparisons

21

Performance: {(5.4, 19.1), (16.6, 3.5), (0.6, 3.4), (1.4, 2.5), (0.6, 3.6), (7.3, 1.7)}. Is one system better?

Differences: {-13.7, 13.1, -2.8, -1.1, -3.0, 5.6}.

Answer: No. They are not different (the confidence interval includes zero)

Example 13.5; Paired Observations

22

1. Compute the sample means

2. Compute the sample standard deviations

Unpaired Observations

23

3. Compute the mean difference 4. Compute the standard deviation of the mean difference

5. Compute the effective number of degrees of freedom

6. Compute the confidence interval for the mean difference

7. If the confidence interval includes zero, the difference is not significant

Unpaired Observations (cont’d)

24

Times on System A: {5.36, 16.57, 0.62, 1.41, 0.64, 7.26}Times on system B: {19.12, 3.52, 3.38, 2.50, 3.60, 1.74}

Question: Are the two systems significantly different? For system A:

For System B:

Example 13.6

25

The confidence interval includes zero the two systems are not different

Example 13.6 (cont’d)

26

Approximate Visual Test

27

Times on System A: {5.36, 16.57, 0.62, 1.41, 0.64, 7.26}Times on system B: {19.12, 3.52, 3.38, 2.50, 3.60, 1.74}t[0.95, 5] = 2.015

The 90% confidence interval for the mean of A = 5.31 (2.015) = (0.24, 10.38)

The 90% confidence interval for the mean of B = 5.64 (2.015) = (0.18, 11.10)

Confidence intervals overlap and the mean of one falls in the confidence interval for the other◦ Two systems are not different at this level of confidence

Example 13.7

28

Need not always be 90% or 95% or 99% Based on the loss that you would sustain if the

parameter is outside the range and the gain you would have if the parameter is inside the range

Low loss Low confidence level is fine◦ E.g., lottery of 5 Million, one dollar ticket cost,

with probability of winning 10-7 (one in 10 million)◦ 90% confidence buy 9 million tickets (and pay $9M)◦ 0.01% confidence level is fine

50% confidence level may or may not be too low 99% confidence level may or may not be too high

What Confidence Level To Use?

One Sided Confidence Intervals

Time between crashes was measured for two systems A and B. The mean and standard deviations of the time are listed in Table 13.1. To check if System A is more susceptible to failures than System B,

Example 13.8

33

Larger sample Narrower confidence interval resulting in higher confidence

Question: How many observations n to get an accuracy of § r% and a confidence level of 100(1-)%?

We know that for a sample of size n, the 100(1 - α)% confidence interval of the population mean is

r% accuracy implies that confidence interval should be

Sample Size for Determining Mean

34

Sample mean of the response time = 20 secondsSample standard deviation = 5Question: How many repetitions are needed to get the response time accurate within 1 second at 95% confidence?

Required accuracy = 1 in 20 = 5%Here, = 20, s= 5, z= 1.960, and r=5,

n =

A total of 97 observations are needed.

Example 13.11

35

All statistics based on a sample are random and should be specified with a confidence interval

If the confidence interval includes zero, the hypothesis that the population mean is zero cannot be rejected

Paired observations Test the difference for zero mean

Unpaired observations More sophisticated t-test

Summary