04/22/23

Discrete Event Simulation - 8Design of Simulation Experiments

04/22/23

Discrete Event Simulation - 8How do we design a simulation experiment? This is not very different from designing any other kind of experiment: 1) determine what variables you want to examine, 2) determine what tests are most suitable for the desired variables,3) set up the experiment: create a controlled environment that will allow for replication,4) run the experiment, 5) collect the data, 6) analyze the data.

04/22/23

Discrete Event Simulation - 8You generally wish to determine some values and some "confidence intervals" for those values. For example, you may want to estimate the size of some queues under certain operating conditions: two quantities of interest are the queue means and the queue variations. The simulation experiment may introduce some experimental error itself: numerical operations tend to propagate numerical errors; subtle timing errors may be incorporated into the design, etc.

04/22/23

We are going to look at several types of experiments, increasing in complexity. Terminology is - sometimes - used from some old applications.Assume we have a number of observations Yij:

Where is the common mean and ij is an error term from a population with mean 0 and variance 2. Tj is usually interpreted as the effect of a treatment - the j-th treatment (in our context it could be the j-th pair of (voice, silence) values tried in our simulation of voice traffic). One often assumes that and the null hypothesis usually tested is H0: Tj = 0, for all j - the treatments make no difference.

Discrete Event Simulation - 8

Yij =μ +Tj +εij, i =0K nj , j =0K k.

Ti =0j∑

04/22/23

Discrete Event Simulation - 8Recall that we only have the observed Yij. The existence of the quantities , Tj and ij is simply assumed, to account for the of the effects of the changes on the observables. The further hypothesis - implicit in the formula - that the relationships among the variables is linear, is also just a “working hypothesis”. It may be true, it may be almost true, or it may be quite wrong. Any further refinement would require a better understanding of the theoretical model and, probably, trial and error with several data sets.

04/22/23

How can we test it? What kind of test can we employ?One thing we may observe is that we can try to separate the various (hypothesized) contributions and that most of the tests we are familiar with require variances or 2 variables (which are, most easily, sums of variances).

What we need to do is extract the contribution of the Tj terms and compare it to the contribution of the error terms. If both of them can be written in the form of 2 variables with appropriate degrees of freedom, we can use them in the quotient of the F-test.

Discrete Event Simulation - 8

04/22/23

Discrete Event Simulation - 8We start - sum of squares:

Yij( )2 =i=1

n j∑j=1

k∑ Yij −Y •j +Y •j −Y +Y ( )2 =i =1

n j∑j=1

k∑Yij −Y •j( )2 + Y • j −Y ( )2 + Y ( )2

i=1

n j∑j=1

k∑where

The "cross product" terms vanish: we show it in one case:

Yij −Y •j( ) Y •j −Y ( )i=1

n j∑j=1

k∑ = Yij −Y •j( )i=1

n j∑⎛ ⎝ ⎜ ⎞

⎠ ⎟

j=1

k∑ Y • j −Y ( )= 0( )j=1

k∑ Y • j −Y ( ) =0

Y • j = 1nj

Yiji=1

n j∑ = 1nj

μ +Tj +εij( )i=1

nj∑ =μ +Tj +ε • j

04/22/23

Discrete Event Simulation - 8 At this point we need two older theorems for guidance:Central Limit Theorem: if x is normally distributed with mean and standard deviation , and a random sample of size n is drawn, then the sample mean will be normally distributed with mean and standard deviation /√n.2 Theorem: if x is normally distributed with variance 2 and s2 is the sample variance based on a random sample of size n, then ns2/2 has a 2 distribution with n-1 degrees of freedom.

x

04/22/23

Discrete Event Simulation - 8 The two useful terms are Yij −Y •j( )2

and Y • j −Y ( )2

i=1

nj∑j=1

k∑i=1

n j∑j=1

k∑We start with the second one: Y •j −Y ( )2 = nj Y •j −Y ( )2

j=1

k∑i=1

n j∑j=1

k∑Recall that Y • j = 1

njμ +Tj +εij( )

i=1

n j∑ =μ +Tj +ε • j

By hypothesis, the variance depends strictly on the ij terms, which have mean 0 and variance 2. Then the Null Hypothesis (Tj = 0) implies that all these variables have the same mean () and (by the Central Limit Theorem) variances given by 2/nj. If all the nj are the same, then elementary standard theory applies (if they are different - text claims also … I have no reference) and we can claim:

04/22/23

Discrete Event Simulation - 8 The expression

Y •j −Y ( )2

σ 2 = Y •j −Y ( )2

σ 2nj =1

k∑i=1

n∑j=1

k∑possesses a 2 distribution with k - 1 degrees of freedom. One now needs to examine the first sum:

Yij −Y •j( )2

i=1

n j∑j=1

k∑We will not go into the details of the derivation (not even the plausibility argument given above) and conclude that possesses a 2 distribution with k(n - 1) = kn - k degrees of freedom.

Yij −Y •j( )2

σ 2i=1

n∑j=1

k∑

04/22/23

Discrete Event Simulation - 8

How should we interpret ?We can expandWe observe that this term contains only error contributions, and is completely independent of the Tj terms. The expression

Yij −Y •j( )2

σ 2i=1

n∑j=1

k∑

Y •j −Y ( )2

σ 2i=1

n∑j=1

k∑k−1

Yij −Y • j( )2

σ 2i=1

n∑j=1

k∑nk−k

=nk−k( ) Y •j −Y ( )

2

i=1

n∑j=1

k∑k −1( ) Yij −Y •j( )

2

i=1

n

∑j=1

k

∑

is the desired F variable: contribution from the Tj terms divided by the contribution of the errors alone.

04/22/23

Discrete Event Simulation - 8An example: how do we use this??? 10.1-10.2Suppose we simulate a single-channel queuing system in which arrivals occur according to a Poisson process with rate and service times are exponentially distributed with service rate . Suppose further that we want to test the effect of the queuing disciplines FIFO, SIRO(Serially_In_Randomly_Out??? I have NO idea), and LIFO on the number of customers in the system at the end of one hour. The response variable (i.e. the Tj) is the number of customers in the system at the end of one hour; the single factor influencing the variable is the queuing discipline.

04/22/23

Discrete Event Simulation - 8Setting up the experiment: choose one discipline at random, run for one hour, count. Choose second discipline at random, run, count. Choose third discipline, run count. You may want to repeat this process several times with different runs of (= seeds for) the random number generator. Let's say you have executed four runs for each of the disciplines and you have:

jFIFO SIRO LIFO

Y11= 50 Y12= 32 Y13= 8048 18 7361 64 5458 72 63

i1234

04/22/23

Discrete Event Simulation - 8We now compute the various sums:

Y •1 =14 50+48+61+58( ) =54.25

Y •2 =14 32+18+64+72( ) =46.50

Y •3 =14 80+73+54+63( ) =67.50

Y •• =13 54.25+46.50+67.50( )=56.08

SSTRT = Y •j −Y ( )2 =4i=1

4∑j=1

3∑ Y •1 −Y ( )2 +4 Y •2 −Y ( )2 +4 Y •3 −Y ( )2

=902.1668 (courtesyof Maple)

SSERR = Yij −Y • j( )2 =2489.23

i=1

4∑j=1

3∑

04/22/23

Discrete Event Simulation - 8

The number of degrees of freedom for the numerator is 3-1=2, and for the denominator 3*4-3=9.

F =902.1722489.939

=1.63

From the tables F0.05,(2,9)=4.26, and we cannot reject the null hypothesis: we cannot show (at least at the 5% level) that the queuing discipline has any effect.

04/22/23

Discrete Event Simulation - 8

A more complex simulation experiment would add the possibility of using different service rates at different times - say four different ones - each service rate remaining constant at least long enough for the one-hour experiment to be conducted. We now have the possibility of determining some of the interactions between service rates and queuing discipline, but let's assume for the moment that we are only interested in the effects of queuing discipline. We have each run:

Yij =μ +Bi +Tj +εij

Where the Bj terms correspond to the "blocks" of different service rates. You need to run at least one experiment with each rate and with each discipline.

04/22/23

Discrete Event Simulation - 8How do we set up the appropriate 2 variables and the F variable?

Yij( )2

i =1

n

∑j=1

k

∑ = Yij −Y i• +Y i• −Y • j +Y • j −Y +Y −Y +Y ( )2

i=1

n

∑j=1

k

∑ =

Yij −Y i• −Y • j +Y ( ) + Y •j −Y ( )+ Y i• −Y ( )+Y ( )2

i =1

n∑j=1

k∑ =

Yij −Y i• −Y • j +Y ( )2 + Y •j −Y ( )2 + Y i• −Y ( )2 + Y ( )2( )

i =1

n∑j=1

k∑Since the cross product terms will cancel out. The terms of interest are:

04/22/23

Discrete Event Simulation - 8

Yij −Y i• −Y • j +Y ( )2

i =1

n

∑j=1

k

∑Y •j −Y ( )2

i =1

n∑j=1

k∑Y i• −Y ( )2

i =1

n∑j=1

k∑

For the error term.

For the discipline term.

For the block (rate) term.Each value of i now corresponds to a different service rate. Proof that we will end up with the appropriate 2 variables is omitted - although it is possible.

04/22/23

Discrete Event Simulation - 8We now attempt to reconcile the formulae in the text with the formulae obtained here.

SSTRT = Y •j −Y ( )2

i=1

n∑j=1

k∑ = Y •j( )2 −2Y Y • j +Y 2( )i =1

n∑j=1

k∑ =

nY •j( )2 −2nY Y •jj =1

k∑j=1

k∑ +nkY 2 =n Y •j( )2

j=1

k∑ −nkY 2

The remainder of the formulae now follow in the same way.One more unexplained term is given by the formula for the number of degrees of freedom for SSERR, which differs from the one in the previous experiment.

04/22/23

Discrete Event Simulation - 8The general derivation (without proof) goes as follows:

Yij( )2

i=1

n∑j=1

k∑ =

Yij −Y i• −Y • j +Y ( )2

i=1

n∑j=1

k∑ + Y • j −Y ( )2

i=1

n∑j=1

k∑ + Y i• −Y ( )2

i=1

n∑j=1

k∑ +nkY ( )2

The left hand side of the equality has kn degrees of freedom; the terms on the right hand side have, respectively:

nkY ( )2 → 1; Y •j −Y ( )2

i=1

n∑j=1

k∑ → k−1; Y i• −Y ( )2

i=1

n∑j=1

k∑ → n−1;

Yij −Y i• −Y • j +Y ( )2

i=1

n∑j=1

k∑ → kn−(k−1) −(n−1)−1=(k−1)(n−1)

04/22/23

Discrete Event Simulation - 8The example in the text considers each of the four distinct rows as corresponding to different rates. If one wants to take into account the interactions between the disciplines and the rate changes, one must introduce appropriate "interaction terms". For example

Yijk =μ +Ai +Bj +ABij +εijk

Where one carries out k = 1…m observations for each of the choices of Ai and Bj factors (or treatments), i = 1…a and j = 1…b. By analogy with the previous treatment, the contribution of the Ai terms will come through that of the Bj ones via , and that of the ABij by means of .

Y i•• −Y Y • j• −Y

Y ij • −Y i•• −Y •j• +Y

04/22/23

Discrete Event Simulation - 8

Before continuing, let’s examine the idea of interactions a little more closely. Assume that we change TWO variables, but make only ONE observation for each parameter configuration. Then we would have

Yij =μ +Ai +Bj +??where the ?? term must contain both interaction (ABij) and error (ij)effects. The expansion (from a previous slide) leads us to:

Yij( )2

i=1

n∑j=1

k∑ =

Yij −Y i• −Y • j +Y ( )2

i=1

n∑j=1

k∑ + Y • j −Y ( )2

i=1

n∑j=1

k∑ + Y i• −Y ( )2

i=1

n∑j=1

k∑ +nkY ( )2

04/22/23

Discrete Event Simulation - 8Where no interaction effects - other than “error terms” - can be extracted. In order for us to have explicit access to interaction effects with just TWO parameters being changed we need to perform at least two observations for each parameter setting. This will lead us back to the discussion of Yijk - as a special case of the situation where the observable depends on THREE parameters - one of which affects only the error terms ijk.

04/22/23

Discrete Event Simulation - 8In general, if we have three variables, we have the expansion:

Yijk( )2∑∑∑ = Y i•• −Y ( )2∑∑∑ + Y •j• −Y ( )2∑∑∑ + Y ••k −Y ( )2∑∑∑ +

Y ij• −Y i•• −Y •j• +Y ( )2∑∑∑ + Y i•k −Y i•• −Y ••k +Y ( )2∑∑∑ +

Y • jk −Y •j• −Y ••k +Y ( )2∑∑∑ +

Y ijk −Y ij• −Y i•k −Y • jk +Y i•• +Y • j• +Y ••k −Y ( )2∑∑∑ + Y ( )2∑∑∑In the textbook example we have two variables with a and b levels, respectively, and m observations for each variable configuration. Thus the identity above can be rewritten without the terms that correspond to segregation of the third variable:

04/22/23

Discrete Event Simulation - 8Yijk( )2∑∑∑ = Y i•• −Y ( )2∑∑∑ + Y •j• −Y ( )2∑∑∑ +

Y ij • −Y i•• −Y •j• +Y ( )2∑∑∑ + Y ijk −Y ij•( )2∑∑∑ + Y ( )2∑∑∑Which can be verified to match the figure on p.229 of our text. The computation of the degrees of freedom follows as before:a*b*m on the left, with (a - 1) + (b - 1) + (a - 1)*(b - 1) + a*b*(m - 1) + 1 on the right. The values can be obtained by the observation that an index which has not been averaged in the formula counts for the full amount, while one averaged counts for one less (there is a relation already in the formula).