using sensitivity analysis to identify significant

Using Sensitivity Analysis to Identify Significant P t i N t k Si l tiParameters in a Network Simulation

image generated using http://www wordle net/image generated using http://www.wordle.net/

Kevin Mills & Jim Filliben, NISTKevin Mills & Jim Filliben, NISTWinter Simulation Conference – Dec 6, 2010

Outline• Goal – Problem – Solution• Scale Reduction: Theory and Practice

O i f th 20 P t i M N t• Overview of the 20 Parameters in MesoNet, a TCP\IP Network Simulator

• 2-Level Per Parameter Experimental DesignTh– Theory

– Application to MesoNet• Selected Analysis Techniques

M i Eff t A l i– Main Effects Analysis– Two Factor Interaction Analysis– Tabular Summary Analysis

Relati e Importance of MesoNet Parameters• Relative Importance of MesoNet Parameters• Findings and Conclusions

2

Goal – Problem – Solution• Goal – compare proposed Internet congestion control

algorithms under a wide range of controlled, repeatable conditions as simulated by selecting repeatable conditions, as simulated by selecting combinations of parameter values for MesoNet, a mesoscopic network model

• Problem – how to determine key parameters influencing behavior in MesoNet, a 20-parameter

k d lnetwork model?

• Solution – apply 2-level-per-factor orthogonal Solution apply 2 level per factor orthogonal fractional factorial (OFF) experimental design and related data analysis techniques to identify the relative importance of model parametersrelative importance of model parameters

3

Simulating large, fast networks across many conditions and congestion control algorithms requires scale reduction in both model parameters & responses

Scale Reduction: Theory & Practice

algorithms requires scale reduction in both model parameters & responses

y1, …, yz = f(x1|[1,…,l] …, xp|[1,…,l])

Response State‐Space Stimulus State‐Space

(232)1000 O(109633) [ 1080 = atoms in visible universe]

Parameter ReductionMultidimensional ResponseReduction (2 )

Discard parameters not germane to study – reduce by 944 parameters

O(10 ) [ ]

(232)56 O(10539)

20Group related remaining parameters– reduce by 36 parameters

22 Responses

CorrelationAnalysis &Clustering

PrincipalComponentsAnalysis

Talk on Wed. 8:30 AM

Use experiment design theory to reducebi i 256

220

(232)20 O(10192)

O(106)

Model ReductionSelect only 2 values for each parameter

Level Reduction

g

7 Responses 4 ResponsesDomainAnalysis

parameter combinations to 256

Use sensitivity analysisto identity six mostsignificant parameters

220‐12 256ExperimentDesign Theory

26‐1 32

SensitivityAnalysis7 Responses

This TalkNext Talk

4

Use experiment design theory again to reduceparameter combinations to 32

This Talk

2-Level Per Parameter Orthogonal Fractional Orthogonal Fractional

Factorial Experimental Design p gTheory

5

Sample & PopulationY = f(X1, X2, X3, …, Xk)

h f i ( ) l i ?

f( , , 3, …, )

?Key: What factors are in (my) population?

What scope (robustness) do I want my conclusions to be valid over?

Sample {…}n =

Population {…}N =

Representative

1. random sampling2. blockingg3. proportional sampling4. orthogonal fractionation

S i i

6

Summarization Inference

What is a 2-Level Per Parameter Design?Each experimental parameter p is assigned only 2 of its possible values

What is a 2-Level Full Factorial Design?

Each experimental parameter, p, is assigned only 2 of its possible values

What is a 2 Level Full Factorial Design?An experiment is conducted for each of the 2p parameter combinations

What is a 2-Level Fractional Factorial (FF) Design?An experiment is conducted for a 2p-m subset of parameter combinations

What is a 2-Level Orthogonal FF (OFF) Design?The choice of the 2p-m subset of parameter combinations for experimentsThe choice of the 2 subset of parameter combinations for experiments

is made in a fashion that achieves balance and orthogonality,minimizing confounding of interactions between main effects and

also between main effects and 2-term interactions and minimizing thegvariance in the estimation of effects

7

Why 2 Levels Per Factor?Pros• Requires relatively few runs per factor

F ilit t s i t t ti f s s d t• Facilitates interpretation of response data• Identifies promising directions for future

experiments, and may be augmented with thorough exper ments, and may be augmented w th thorough local exploration

• Forms basis for 2-level fractional factorial designs• Fits naturally into a sequential strategy, which

supports the scientific methodConsCons• Limited exploration of parameter values• Assumes monotonicity in range between chosen valuesm m y g

8

Why Orthogonal Fractional Factorial Design?2-Level Design for MesoNet requires 220 = 1 048 576 runs

At 28 processor hours per run and with 48 available processors, theseruns would require about 612 000 hours (70 years)

Adopting a 220-12 OFF experimental design would reduce the resource requirement to only 256 runs, which could be completed in about 150 hours (1 week)

Cost: misses 220 - 28 parameter combinations

OFF Benefit #1: Superior Coverage & Robustness when compared with1-Factor-at-a-Time Designs (k = 7)

+ + + +

X4

X5

+

X1

X2X3

+

+

__

_

_

+ X4

X5

+

X1

X2X3

+

+

__

_

_

+ X4

X5

+

X1

X2X3

+

+

__

_

_

+ X4

X5

+

X1

X2X3

+

+

__

_

_

+

OFF Design 1-FAT DesignX7

X4

X5

+

X1

X2X3

+

+

__

_

_

+

+ X4

X5

+

X1

X2X3

+

+

__

_

_

+

+ X4

X5

+

X1

X2X3

+

+

___

_

_

+

+ X4

X5

+

X1

X2X3

+

+

___

_

_

+

+

X5

X4

9

X4

X6

What is the minimum number of required runs?

Minimally strive for a resolution IV design, i.e., a design where thereis no confounding among parameters and between parameters and2-parameter interactions2 parameter interactions

Requires sufficient runs, n, to resolve a leading constant, the parameters and2-parameter interactions: n = 1 + p + C(p, 2)

MesoNet example – parameters, p = 20

Minimum runs n = 1 + 20 + C(20, 2) = 1 + 20 + 190 = 211

Given 2-levels per factor, we can choose the first power of 2 above 211

n = 256 = 220-12 – this is a resolution IV design n = 2p-r, where the reductionfactor is r

10

2**(20-12) Orthogonal Fractional Factorial Design (k=20, n=256, l=2) Resolution = 4 2to20m12.xlsSpecifying Parameter Combinations

x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 x14 x15 x16 x17 x18 x19 x201 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -12 1 -1 -1 -1 -1 -1 -1 -1 1 1 1 1 1 1 -1 -1 -1 -1 -1 -13 -1 1 -1 -1 -1 -1 -1 -1 1 1 1 1 1 1 1 1 1 1 1 -14 1 1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 1 1 1 1 1 -15 -1 -1 1 -1 -1 -1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 1 1 16 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 16 1 -1 1 -1 -1 -1 -1 -1 -1 1 1 1 1 1 1 1 1 1 1 17 -1 1 1 -1 -1 -1 -1 -1 -1 1 1 1 1 1 -1 -1 -1 -1 -1 18 1 1 1 -1 -1 -1 -1 -1 1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 19 -1 -1 -1 1 -1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1

10 1 -1 -1 1 -1 -1 -1 -1 1 -1 1 1 1 1 1 -1 -1 -1 -1 111 -1 1 -1 1 -1 -1 -1 -1 1 -1 1 1 1 1 -1 1 1 1 1 112 1 1 -1 1 -1 -1 -1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 1 113 -1 -1 1 1 -1 -1 -1 -1 1 1 -1 -1 -1 -1 -1 1 1 1 1 -114 1 -1 1 1 -1 -1 -1 -1 -1 -1 1 1 1 1 -1 1 1 1 1 -115 -1 1 1 1 -1 -1 -1 -1 -1 -1 1 1 1 1 1 -1 -1 -1 -1 -116 1 1 1 1 -1 -1 -1 -1 1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 -117 -1 -1 -1 -1 1 -1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 -1 118 1 -1 -1 -1 1 -1 -1 -1 1 1 -1 1 1 1 -1 1 -1 -1 -1 119 -1 1 -1 -1 1 -1 -1 -1 1 1 -1 1 1 1 1 -1 1 1 1 119 -1 1 -1 -1 1 -1 -1 -1 1 1 -1 1 1 1 1 -1 1 1 1 120 1 1 -1 -1 1 -1 -1 -1 -1 -1 1 -1 -1 -1 1 -1 1 1 1 121 -1 -1 1 -1 1 -1 -1 -1 1 -1 1 -1 -1 -1 1 -1 1 1 1 -122 1 -1 1 -1 1 -1 -1 -1 -1 1 -1 1 1 1 1 -1 1 1 1 -123 -1 1 1 -1 1 -1 -1 -1 -1 1 -1 1 1 1 -1 1 -1 -1 -1 -124 1 1 1 -1 1 -1 -1 -1 1 -1 1 -1 -1 -1 -1 1 -1 -1 -1 -125 -1 -1 -1 1 1 -1 -1 -1 -1 1 1 -1 -1 -1 1 1 -1 -1 -1 -126 1 -1 -1 1 1 -1 -1 -1 1 -1 -1 1 1 1 1 1 -1 -1 -1 -127 -1 1 -1 1 1 -1 -1 -1 1 -1 -1 1 1 1 -1 -1 1 1 1 -128 1 1 -1 1 1 -1 -1 -1 -1 1 1 -1 -1 -1 -1 -1 1 1 1 -129 -1 -1 1 1 1 -1 -1 -1 1 1 1 -1 -1 -1 -1 -1 1 1 1 130 1 -1 1 1 1 -1 -1 -1 -1 -1 -1 1 1 1 -1 -1 1 1 1 131 -1 1 1 1 1 -1 -1 -1 -1 -1 -1 1 1 1 1 1 -1 -1 -1 132 1 1 1 1 1 -1 -1 -1 1 1 1 -1 -1 -1 1 1 -1 -1 -1 1

220-12 Experiment Design Template256

11

32 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Design Properties: Balance & Orthogonality

(p = 20, n = 256)

- +128 128

XAll 20:Balance

Xi

+ 64 64XjAll :

202Orthogonality

- +- 64Xi

64

12

(p = 20 n = 256)

FAT = 1.4142…

OFF Design Benefit #2: Minimizes Variation in Effect Estimates

(p = 20, n = 256)

112

11

1)(nn

BSD i += σ)

0.125

128

13

2-Level Per Factor OFF Design gApplied to

MesoNet Sensitivity AnalysisMesoNet Sensitivity Analysis

14

Overview of the 20 Parameters* in MesoNet, a TCP\IP Network Simulator

Network Parameters

x1 Network Speedx2 Propagation Delayx3 Buffer Provisioningx4 Topology

User Behavior

x4 Topologyx5 Web Browsing File Sizesx6 Larger File Download Probability & Sizesx7 User Think Timex8 User Patience

S & R i

x9 Spatiotemporal Congestion on Very Fast Pathsx10 Number, Location and Start Time for Long‐Lived Flowsx11 Speed of Interfaces Connecting Sources & Receivers to Networkx12 Number of Sources & Receivers

Sources & Receivers

Protocols

x13 Distribution of Sourcesx14 Distribution of Receiversx15 Probability of Source using a specific Congestion Control Algorithmx16 Initial Size of Congestion Window (cwnd )

Simulation & Measurement Control

x17 Initial Slow Start Threshold (sst )x18 Measurement Interval Sizex19 Simulation Durationx20 Startup Pattern for Sources

15*For more details, attend related talk on Wed. 8:30 AM

Abilene-based Topology: (-) LevelA1a

A2bA2a

I2cI2bI2a K1bK1a

I1e I1f I1gA2c K1c

AI

A1

A2

A1b

A1c

A1d

A1e C2

C2f

C2eC2dC2c

C2b

C2a E1

E1bE1a G2e

G2c

G2b

G2a

G2

G2dG2f

I1c

I1bI1a

I1

I1d

I2

I2c

I2d

K1

I2f

K2b

G2g K2a

I2eE0a

I1e

I2g

A2d E1c E1d K1d

K2dK2c

C

G K

J

A1e

EA2gA1f

B0a

B1b

B1a

C1

C2C2g

E2E2aE2bE2c

G1b

G1c G1d

G2a

C1G1G1a

G1f J2

J2f

J2c

J2d

J2eI0a

J2g

K2

K0a

G1g

C0a

B

D

H

JB1

B2g

B2

B2aB2b

B2c

C1a

C1bC1c C1d

C1e

C1f E2f

E2c

E2d E2e

G1c G1dG1e

J1

J1e

J1dJ1cJ1b

J1aJ1f

J2a J2bJ1g

C1g E2gB1c

B1d

D

F

B2fB2

B2d

B2e D1

D1b

D2g

D1a

D2

D2a

D2b D2e

F0a

F1

F1b

F1a F2

H2

H2a

H2b

H2c H2dH2e

H1

H1b

H2g

H1a

F2f

H2fD2f

F2g

F1g H1c H1d

Backbone Routers 11

Backbone Links 14

Backbone Routes 110

PoP Routers 22

l

Access Router n x Normal Speed

Access Router m x Normal Speed

F1d

D2c D2dF1b

F1c

F2aF2b F2c F2d

F2e

g

F1fF1e

D1dD1c

Access Router Normal Speed102 to 103 Sources (not shown) and Receivers (not shown) under each Access Router

PoP Routers 22

Access Routers 139

Directly Connected 6

Fast 28

Typical 105Routes are shortest-path based on propagation delay

16

yp

Commercial ISP-based Topology: (+) LevelA2b A2cA2a

K1bK1c K1d

K2a K2bI2aI1b I1c I1d I2dI2cI2b

A

C

K

A1

A1b

A1c

A1a

C1

C1b

C1c

C1a

H1

H1bH1cH1a

H1d

H1e

H1f

K1

K2K0a

K1a

K1b K2a K2b

K2c

K2d

L1

J1aJ1b J1c

J1d

J1eJ2a

I2I1

II0a

I1a

I2gI2f

I2e

B2cB2d

B2eB2f H

A2

J

B

L

C2

C2bC2c

C2a

H2b

H2c

H2a

H2fH2e

H2d

L2

L1

L1a L1bL1c

L1d

L2a

L2b

L2c

L0aL0b

J1

J2

J1f

J2a

J2b

J2c

J2dJ2e J2f

M2d M2eM2f

M2g

B2

B2a

B2b B2g

H

H2

D

F

M

B

N

O

L2c

L2dN1aN1b

N1c

N1d

N1e

N1fN2

N2a N2f

M2b

M2cM2

g

M2aM1a

M1M1b

M1cM1dM1e M1f M1g

F2aF1

F1a

F1b F1c F1d

F2c

F2b

D2

D2a

D1D1a

D2gD2f

D2dD2cD2b

D2eB0aB1B1a

B1b

B1c B1dN1

B kb R t 16

E

G

O

E1

E1b

E1a

G1b

G1a

N2aN2b

N2c N2dN2e

N2fO0a

O1aO1b

O1cO1c

O1

O2a

O2

O2b O2c O2d

O2e

O2f

O2g

F2

F2g F2f

F2dF0a

F2e

D1D1a

D1b

D1c D1d

D0a

E2

Backbone Routers 16

Backbone Links 24

Backbone Routes 240

PoP Routers 32

Access Routers 170PE1c

E2bE2cE2a

G1b

G1c

G1eG1d

G2b G2cG2a

G2e

G2dG1f

G2fP1 P2

P1a

P1b

P1c

P1dP2a

P2bP2c P2d

P2e

P2f

P2gG1 G2 Directly Connected 8

Fast 40

Typical 122

Routes are shortest-path based on traffic engineering goals17

2 Levels Per Factor Used in Sensitivity AnalysisCategory ID Name Minus ( 1) Level Plus (+1) LevelCategory ID Name Minus (-1) Level Plus (+1) Level

Network Configuration

X1 Network Speed 800 packets/ms 1600 packets/ms X2 Propagation Delay 1 2 X3 Buffer Provisioning RTT x C/sqrt(n) RTT x C X4 Topology Abilene - SPF prop.

delayISP - SPF assigned costsy

User Behavior

X5 Web Object Size ( = 1.5) 75 packets 150 packets X6 Proportion/Size of Larger Files

(Fx = 10 Sx = 1000 Mx = 10000) Fp = 0.02 Sp = 0.002 Mp = 0.0002

Fp = 0.04 Sp = 0.004 Mp = 0.0004

X7 User Think Time 2 s 5 s X8 User Patience 0.0 (Infinite) 1.0 (Finite) X9 Selected Spatiotemporal 4th Time Period NoneX9 Selected Spatiotemporal

Congestion 4 Time Period None

X10 Long-lived Flows 3 Start in 3rd Time Period

None

Sources &

X11 Probability of Fast Interface 0.2 0.8 X12 Number of Sources & 2 3 Sources &

Receivers ReceiversX13 Distribution of Sources Web Centric Peer-to-Peer Centric X14 Distribution of Receivers Web Centric Peer-to-Peer Centric

Protocols X15 Probability of Algorithms TCP = 0.8 CTCP = 0.2 TCP = 0.2 CTCP = 0.8 X16 Initial Congestion Window Size 2 packets 8 packets X17 Initial Slow Start Threshold 43 packets 1 073 741 823 packetsX17 Initial Slow Start Threshold 43 packets 1 073 741 823 packets

Simulation & Measurement Control

X18 Measurement Interval Size 200 ms 1 s X19 Simulation Duration 25 minutes 50 minutes X20 Source Startup Pattern Exponential (mean =

X7) 50 % start early

18

Traffic Scenario(s)

19

M i R Fl G f Th h t A18 Macroscopic Response Variables Throughput in each of 24 Flow Groups

102 Response VariablesMacroscopic Responses Flow Groups for Throughput Averages

Category Identity Definition Number File Size Path Class

Max. Rate

Flow

Y1 Average # sources connecting 1 VF F Y2 Average # sources sending 2 VF N Y3 % sending flows in initial slow start 3 F F Y4 % di fl i d d i F NFlow

State Movie Y4 % sending flows in standard congestion avoidance 4 F N

Y5 % sending flows in alternate congestion avoidance 5 T F

6 T N

Congesti Y6 Retransmission rate 7 VF F Y7 A i i d i ( k ) 8 VF NCongesti

on Software Service Pack

Y7 Average congestion window size (packets) 8 VF NY8 Aggregate # connection failures 9 F F

10 F N

Delay Y9 Average round-trip time (ms) 11 T F Y10 Average queuing delay (ms) 12 T N

13 VF F

Document

Work Y11 Average # flows completed per second 14 VF NY12 Average # packets output per second 15 F F

16 F N Long-Lived Flows

Y13 Average throughput on long-lived flow #1 17 T F Y14 Average throughput on long-lived flow #2 18 T N Y15 Average throughput on long-lived flow #3 19 VF Fg g p g

Web Object

20 VF N

Flows by Path Class

Y16 Average throughput on flows transiting VF paths 21 F F

Y17 Average throughput on flows transiting F paths 22 F N Y18 Average throughput on flows transiting T paths 23 T F

24 T N

20

24 T N Averaged over each of three time periods (3 x 18 = 54 responses)

Average Computed Separately for TCP Flows and CTCP Flows ( 2 x 24 = 48 responses)

Selected Analysis Techniques1. Main Effects Analysis

2. Two Factor Interaction Analysis

3. Tabular Summary Analysis

21

Sample: 1 of 102 Main Effects Analyses

Larger InitialSlow Start ThresholdLarger

Fil

FasterNetwork

ShorterPropagationDelay

Files

Throughput (pps) for Movies transferred over Very Fast Paths with Fast Interfaces using CTCP22

Another Sample: 1 of 102 Main Effects Analyses

Shorter Think Time More Sources Distributed in a Peer-to-Peer Pattern

Larger Files

BiggerSlowerNetwork Bigger

TopologyNetwork

Y2 – Average Number of Sending Sources in Time Period #2 23

Sample: 1 of 102 Two Factor Interaction Analyses

Two Factor Interaction Plot for Y2 – Avg. Number of Sending Sources in Time Period #2(not much in the way of significant 2 factor interactions)

24

Sample: 1 of 5 Tabular Summary Analyses

Metric Class Y# Name

Network User Behavior Source/Receiver Protocol Sim. Control &

Meas.

X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 NSp PrD Buf Top FS LFS ThT UP CVF LLF SSR NSR DiS DiR CCA ICW IST MIS DUR StP

Y1 # Connecting +** +** +* ‐** +** +** +**

Y2 # Active +** +** +** ‐** +** +**

Flows

Y2 # Active + + + ‐ + +

Y3 % ISS +** +** +** ‐** +** ‐** ‐** +**

Y4 % NCA ‐** ‐** ‐** +** ‐** +** +** ‐*

Y5 % ACA +** +** +** ‐** ‐** +** +* ‐**

Y6 Retrans. Rate ‐** ‐** ‐** +** ‐** +** +** +**

Congestion Y7 cwnd Size +*

Y8 # conn. fails ‐** ‐** ‐** +** +** +** +**

Delay Y9 SRTT ‐** +** +** +* +*

Y10 Queue Delay ‐** +** +** ‐* +** +**

Aggregate Y11 Flows/sec +** * +** ** ** +** +**Aggregate TP

Y11 Flows/sec +** ‐* +** ‐** ‐** +** +**

Y12 Packets/sec +** +** +** +** ‐** ‐** +**

Long‐Lived Flow TP

Y13 LLF 1 +** +* +* +** ‐**

Y14 LLF 2 +* +** +** ‐**

Y15 LLF 3 +** +** ‐* ‐*

Other Flow TP

Y16 VF Paths +** ‐** ‐** +* +** ‐** ‐* +** ‐*

Y17 F Paths +** ‐** +* +** +** ‐** +** ‐** +** ‐*

Y18 N Paths +** ‐** ‐** +** ‐** ‐** +**

Significant Influence of each Factor on each Macroscopic Response in Time Period #2

25


Meas

Another Sample: 1 of 5 Tabular Summary Analyses

File Type

Path Class

Connection Speed

Meas.X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 NSp PrD Buf Top FS LFS ThT UP CVF LLF SSR NSR DiS DiR CCA ICW IST MIS DUR StP

Movies

VF Fast +** ‐** +** +**

VF Normal +** ‐** +** ‐* +**

F Fast +** +** ‐** ‐** +** ‐* +** ‐*Movies

F Normal +** +** +** ‐** +** ‐** +** ‐*

T Fast +** +** ‐** +** ‐** ‐**

T Normal +** +** ‐** +** ‐** ‐**

VF Fast +** ‐** +** +**

VF Normal ‐** ‐* +** +**

Service Packs

F Fast +** ‐** +** +** +* +** ‐* +**

F Normal +** ‐** +** +** +** ‐** +** ‐* +**

T Fast +** +** ‐** +** ‐* ‐**

T Normal +** +** ‐** +** ‐* ‐**

VF Fast ‐** +* +* +**

l ** ** ** **

Documents

VF Normal ‐** +** +** +**

F Fast +** ‐** +** +* +* ‐* +** +**

F Normal +** ‐** +** +** +** ‐* +** +**

T Fast +** ‐** +** +** ‐** ‐** +**

T Normal +** ‐** +** +** ‐** ‐** +**

VF Fast ** +** +** +**

Web Objects

VF Fast ‐** +** +** +**

VF Normal ‐** +* +** +**

F Fast +** ‐** ‐* +** +** +* +** +**

F Normal +** ‐** ‐* +** +** +** +** +**

T Fast +** ‐** +** ‐** ‐** +**

T Normal +** ‐** +** ‐** ‐** +**

Significant Influence of Each of 20 Factors on Throughput for Each of 24 Flow Groups when using CTCP 26

Relative Importance ofRelative Importance ofMesoNet Parameters

27

i l &

Summary of Influence of Each Factor on All Responses

Protocol T‐test Statistic


Meas. X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20NSp PrD Buf Top FS LFS ThT UP CVF LLF SSR NSR DiS DiR CCA ICW IST MIS DUR StP

Time P i d #1

>0.99 17 9 10 8 8 0 11 0 0 3 0 12 11 2 1 7 2 1 0 0

>0.95<0.99 1 1 3 2 2 0 2 0 0 0 0 3 2 1 0 2 1 1 0 0Period #1

Total 18 10 13 10 10 0 13 0 0 3 0 15 13 3 1 9 3 2 0 0

Time Period #2

>0.99 16 9 9 6 7 0 10 0 6 2 0 13 11 1 1 5 3 0 0 0

>0.95<0.99 2 1 1 2 0 0 2 0 0 1 0 1 1 2 1 1 0 2 0 0

Total 18 10 10 8 7 0 12 0 6 3 0 14 12 3 2 6 3 2 0 0

>0 99 17 9 11 6 9 0 12 0 4 3 0 12 11 2 1 5 3 1 0 0Time

Period #3

>0.99 17 9 11 6 9 0 12 0 4 3 0 12 11 2 1 5 3 1 0 0

>0.95<0.99 1 2 0 3 1 0 1 0 1 0 0 3 2 0 0 0 0 2 0 0

Total 18 11 11 9 10 0 13 0 5 3 0 15 13 2 1 5 3 3 0 0

TCP >0.99 19 16 12 8 11 0 10 0 1 0 0 4 16 0 0 8 16 1 0 0

>0.95<0.99 0 2 3 3 5 0 4 0 0 0 0 2 0 0 0 0 0 0 0 0

Total 19 18 15 11 16 0 14 0 1 0 0 6 16 0 0 8 16 1 0 0Total 19 18 15 11 16 0 14 0 1 0 0 6 16 0 0 8 16 1 0 0

CTCP >0.99 19 18 10 9 15 0 13 0 1 0 0 8 16 0 0 7 12 0 0 0

>0.95<0.99 0 0 2 3 1 0 3 0 0 0 0 6 0 4 0 1 0 0 0 0

Total 19 18 12 12 16 0 16 0 1 0 0 14 16 4 0 8 12 0 0 0

Total>0.99 88 61 52 37 50 0 56 0 12 8 0 49 65 5 3 32 36 3 0 0

>0 95<0 99 4 6 9 13 9 0 12 0 1 1 0 15 5 7 1 4 1 5 0 0Total >0.95<0.99 4 6 9 13 9 0 12 0 1 1 0 15 5 7 1 4 1 5 0 0

Total 92 67 61 50 59 0 68 0 13 9 0 64 70 12 4 36 37 8 0 0

l E E

90% 66%% of responses influenced 60% 49% 58% 67% 63% 69%13% 12% 4% 35% 36% 8%9%

Significant Influence of Each of 20 Factors on Each of 18 Macroscopic Responses

28

Findings: 7 main factors drive MesoNet Response

• Capacity (network speed)

• Demand (number, distribution and activity of sources)

• Physics (propagation delay)

• Buffer sizing

• File sizes

29

Conclusions• 2-Level-per-Factor Orthogonal Fractional Factorial

(OFF) experimental designs can reveal significant information about simulation modelsf m u mu m

• MesoNet simulation appears to be driven by the same key factors that influence behavior in real networkskey factors that influence behavior in real networks

• Appears feasible to compare proposed Internet pp p p pcongestion control algorithms while varying only 7 MesoNet parameters

Entire Study Report NIST Special Publication 500-282:Study of Proposed Internet Congestion Control MechanismsAvailable online at h // i /i l/ d/C i C l S d f

30

Available online at http://www.nist.gov/itl/antd/Congestion_Control_Study.cfm