measurement science for complex information systems...measurement science for complex information...

March 27, 2009Innovations in Measurement Science

1

Measurement Science for Complex Information Systems

Measurement Science for Measurement Science for Complex Information SystemsComplex Information Systems

D. Cho, C. Dabrowski, J. D. Cho, C. Dabrowski, J. FillibenFilliben, J. , J. HagedornHagedorn, C. , C. HouardHouard, F. Hunt, D. , F. Hunt, D. GeninGenin, V. , V. MarbukhMarbukh and and K. MillsK. Mills

and various studentsand various students

Innovations in Measurement ScienceInnovations in Measurement Science MarMar 27, 200927, 2009


2


Plan for Presentation

• Introduce NIST project to develop Measurement Science for Complex Information Systems

• Show an application of measurement science to compare seven alternate congestion-control algorithms for the Internet


3


What are complex systems?Large collections of interconnected components whose interactions lead to macroscopic behaviors

http://autoinfo.smartlink.net/quake/quake.htmhttp://autoinfo.smartlink.net/quake/quake.htm

http://www.sover.net/~kenandeb/fire/hotshot.htmlhttp://www.sover.net/~kenandeb/fire/hotshot.html

http://www.avalanche.org/http://www.avalanche.org/

– Physical systems (e.g., earthquakes, avalanches, forest fires)

© http://www.nationalgeographic.com/© http://www.nationalgeographic.com/

©http://emergent.brynmawr.edu 2003©http://emergent.brynmawr.edu 2003

– Biological systems (e.g., slime molds, ant colonies, embryos)

http://www.wtopnews.com/http://www.wtopnews.com/

http://www.english.uiuc.edu/maps/depression/photoessay.htmhttp://www.english.uiuc.edu/maps/depression/photoessay.htm

©http://www.waag.org/realtime/©http://www.waag.org/realtime/

– Social systems (e.g., transportation networks, cities, economies)

http://www.emulab.net/pix/pc3k-back.jpghttp://www.emulab.net/pix/pc3k-back.jpg

http://www.kk.org/thetechnium/images/aol-server-farm.jpghttp://www.kk.org/thetechnium/images/aol-server-farm.jpg

– Information systems (e.g., Internet, Web services, compute grids)


4


What is the problem? No one understands how to measure, predict or control macroscopic behavior in complex information systems

“[Despite] society’s profound dependence on networks, fundamental knowledge about them is primitive. [G]lobal communication …networks have quite advanced technological implementations but their behavior under stress still cannot be predicted reliably.… There is no science today that offers the fundamental knowledge necessary to design large complex networks [so] that their behaviors can be predicted prior to building them.”— Network Science, NRC report released in 2006


5


Technical Approach• Evaluate models and analysis methods

– Computationally tractable?– Reveal macroscopic behavior?– Establish causality?

• Evaluate distributed control techniques– Can economic mechanisms elicit desired behaviors?– Can biological mechanisms organize elements?

Leverage models and mathematics from the physical sciences to define a systematic method to measure, understand and control macroscopic behavior in the Internet and distributed software systems built on the Internet

Internet

Distributed Systems

Scheduler

TaskControl

NegotiationControl

DSI

Grid Processor

Service Negotiator

Agreement

Grid ProcessorGrid Processor

DSIService

NegotiatorAgreement

Execution Control

CLIENT

Application

Client Negotiator

Task 1

DiscoveryControl

Task 2

Grid Processor

DSF

DSIService

NegotiatorAgreement

Client Negotiator

Task 3

TaskControl

NegotiationControl

ApplicationTask 1

DiscoveryControl

Task 2

Scheduler

DRMS Front-End

DRMS Front-End

DRMS Front-End

DRMS Front-End

DSF

spawnsspawns

negotiatesnegotiates

monitors

monitors

requests reservation

spawns

Supervisory Process Supervisory Process

spawns

GIIS

GRIS

GIIS

GRIS

GIIS

ProviderSite

ClientSite

ProviderSite

Scheduler

TaskControl

NegotiationControl

DSI

Grid Processor

Service Negotiator

Agreement

Grid ProcessorGrid Processor

DSIService

NegotiatorAgreement

Execution Control

CLIENT

Application

Client Negotiator

Task 1

DiscoveryControl

Task 2

Grid Processor

DSF

DSIService

NegotiatorAgreement

Client Negotiator

Task 3

TaskControl

NegotiationControl

ApplicationTask 1

DiscoveryControl

Task 2

Scheduler

DRMS Front-End

DRMS Front-End

DRMS Front-End

DRMS Front-End

DSF

spawnsspawns

negotiatesnegotiates

monitors

monitors

requests reservation

spawns

Supervisory Process Supervisory Process

spawns

GIIS

GRIS

GIIS

GRIS

GIIS

ProviderSite

ClientSite

ProviderSite

What is the new idea?


6


Previous NIST Groundwork (2000-2005)

Complex Dynamics in Communications Networks, December 2005(including Macroscopic Dynamics in Large-Scale Data Networks by Yuan and Mills)

Yuan and Mills, A Cross-Correlation-based Method forSpatial-Temporal Traffic Analysis, July 2005

Yuan and Mills, Monitoring the Macroscopic Effects of DistributedDenial of Service (DDoS) Flooding Attacks, October 2005

Yuan and Mills, Simulating Timescale Dynamics of NetworkTraffic Using Homogeneous Modeling, May-June 2006

Preliminary investigation to identify hard technical issues


7


Why is this hard? Why can we succeed?

A5. Evaluate distributed control regimesH5. Controlling behavior

A4. Evaluate analysis techniquesH4. Causal analysis

A3. Cluster analysis and statistical analysesH3. Tractable analysis

A2. Sensitivity analysis & key comparisonsH2. Model validation

A1. Scale-reduction techniquesH1. Model scalePlausible ApproachesHard Issues

Project Start Date: October 2006


8


Multidisciplinary Project

V. Marbukh

K. MillsF. Hunt

C. DabrowskiD. Genin

SimulationAnalytical

Modeling Methods

J. Filliben

Experiment Design Methods

D. Y. Cho

J. Filliben

Data Analysis Methods

C. Houard

J. Hagedorn

Visualization Methods

Disciplinary Expertise Problem Domains

A

B

D

C

F

G

H

I

K

J

A1

A2

A1a

A1b

A1c

A1d

A1e

E

A2b

A2gA1f

A2a

B0a

B1

B1b

B2g

B2f

B1a

B2

B2aB2b

B2c

B2d

B2e D1

D1b

D2g

F1d

D1a

D2

D2a

D2bD2c D2d

D2e

C1

C1a

C1bC1c C1d

C1e

C2

C1f

C2f

C2eC2dC2c

C2b

C2a E1

E1b

C2g

E2f

E1a

E2E2aE2bE2c

E2d E2e

G1b

G1c G1dG1e

G2e

G2c

G2b

G2a

C1G1G1a

G1f

G2

G2dG2f

F0a

F1

F1b

F1a

F1c

F2

F2aF2b F2c F2d

F2e

H2

H2a

H2b

H2c H2dH2e

H1

H1b

H2g

H1a

F2f

J1

J1e

J1dJ1cJ1b

J1aJ1f

J2

J2a

J2f

J2b

J2c

J2d

J2eI0a

I1c

I1bI1a

I1

I1d

I2

I2cI2bI2a

I2d

K1

K1b

J2g

K1a

I2f

K2

K2b

G2g

H2f

K2a

I2e

D2f

E0a

K0a

J1g

I1e I1f

G1g

I1g

I2g

F2g

C1g E2g

F1g

F1fF1e

A2d

A2c

B1c

B1d

D1dD1cH1c

E1c E1d

H1d

K1dK1c

K2dK2c

C0a

Internet

Problem Approaches

Mesoscopic Modeling

Fluid-Flow Modeling

Markov Modeling

Perturbation Analysis

Mean-Field Approximation

Orthogonal FractionalFactorial Design

Sensitivity Analysis

Clustering Analysis

Principal Components Analysis

Correlation Analysis

Multidimensional Data Visualization

Computational Grids


9


Sample Challenge Problems Under Investigation• Predict effect on global behavior and user experience from

adopting proposed replacement congestion-control algorithms for the Internet* (Filliben, Cho, Houard & Mills)

• Evaluate accuracy of proposed fluid-flow models for TCP, characterize limits of applicability of such models and propose improved analytical models (Genin & Marbukh)

• Devise efficient Markov models to accurately simulate large-scale systems and apply perturbation analysis to predict system changes that could lead to undesired behaviors (Dabrowski & Hunt)

• Investigate the use of economic methods for resource allocation in large distributed systems (e.g., computational grids and networks) (Dabrowski, Marbukh & Mills)

*Today I use this challenge problem to illustrate some of our approaches


10


Sample Artifacts Produced by the ProjectMesoNet – a medium scale network simulator that includes seven

congestion control algorithms: BIC, CTCP, FAST, HSTCP,H-TCP, Scalable TCP and TCP Reno

EconoGrid – a detailed simulation model of a standards-based Gridcompute economy

Flexi-Cluster – a flexible simulation model of a compute cluster thatincludes alternate, replaceable functions forpricing, admission control, scheduling and queuemanagement

Markov-Model Rewriter – software to systematically perturb a Markovmodel with bounds defined by a user

DiVisa – software for interactive exploration of multidimensional data


11


Challenge Problem: Study of Proposed Replacement Congestion-Control Algorithms for the Internet

• Modeling the network– Parameter state-space reduction techniques– Response state-space reduction techniques– Orthogonal fractional-factorial experiment design– Sensitivity analysis

• Modeling congestion-control algorithms– Unified model with phase and procedure alignment– Validation against empirical measurements

• Comparing congestion-control algorithms– Cluster analysis– Detailed analysis of individual responses– Condition-response summary analysis– Causality analysis – through domain expertise


12


The Modeling State-Space Problem

y1, …, ym = f(k.x1, …, k.xn)

Response State‐Space Stimulus State‐Space

n Number of inputs (i.e., stimulus factors)k Factor range (i.e., number of values each factor can assume)m Number of outputs (i.e., responses)


13


Parameter State-Space Reductionkn

k(n‐r1)

k(n‐r1‐r2)

2(n‐r1‐r2)

2(n‐r1‐r2‐r3)

2‐levelexperiment

design

modelingbrilliance

factorclustering

OFF (orthogonal fractional factorial)experiment design

SCIENTIFIC DOMAIN EXPERTISE

STATISTICALEXPERIMENT

DESIGNEXPERTISE

kn

k(n‐r1)

k(n‐r1‐r2)

2(n‐r1‐r2)

2(n‐r1‐r2‐r3)

2‐levelexperiment

design

modelingbrilliance

factorclustering

OFF (orthogonal fractional factorial)experiment design



DESIGNEXPERTISE


DESIGNEXPERTISE


14


Parameter State-Space Reduction: MesoNet Example

(232)1000

220

28

k = 2

r1 = 944

O(109633) [ 1080 = atoms in visible universe]

(232)56 O(10539)

(232)20r2 = 36

O(10192)

O(106)

r3 = 12 256

Kevin

Jim

(232)1000

220

28

k = 2

r1 = 944


(232)56 O(10539)

(232)20r2 = 36

O(10192)

O(106)

r3 = 12 256

(232)1000

220

28

k = 2

r1 = 944


(232)56 O(10539)

(232)20r2 = 36

O(10192)

O(106)

r3 = 12 256

Kevin

Jim

Kevin

Jim


15


220-12 OFF Design Improves Computational Feasibility of Searching Parameter State Space

Assumptions

Processing Time for One Run 8 CPU‐Hours

Number of Available CPUs 48

Assumptions

Processing Time for One Run 8 CPU‐Hours

Number of Available CPUs 48

(220 runs x 8 CPU‐Hours Per Run)/48 CPUs = 174,762.67 Hours

2‐Level Experiment Design Requires 20 Years*

OFF Experiment Design Reduces Requirement To Under 2 Days(28 runs x 8 CPU‐Hours Per Run)/48 CPUs = 42.67 Hours

*If we had 1,000 CPUs to dedicate to this problem, then we could compute the 220 runs in just under 1 year


16


Response State-Space Reduction

m responses (y1, … ym)

Principal Components

Analysis

m – d1 responses

CorrelationAnalysis

m – d2 responses

DomainExpertise

m – d3 responses


DATAANALYSISEXPERTISE



Analysis

m – d1 responses

CorrelationAnalysis

m – d2 responses

DomainExpertise

m – d3 responses



Analysis

m – d1 responses

CorrelationAnalysis

m – d2 responses

DomainExpertise

m – d3 responses






17


Response State-Space Reduction: MesoNet Example

22 responses (y1, … y22)


Analysis

22 – 17 responses

CorrelationAnalysis

22 – 15 responses

DomainExpertise

22 – 14 responses

Kevin

Jim

Jim



Analysis

22 – 17 responses

CorrelationAnalysis

22 – 15 responses

DomainExpertise

22 – 14 responses



Analysis

22 – 17 responses

CorrelationAnalysis

22 – 15 responses

DomainExpertise

22 – 14 responses

Kevin

Jim

Jim


18


22-Dimension Response State Space from MesoNet

Macroscopic Network Behavior

UserExperience


19


Correlation Analysis Identifies 7 Dimensions

Flow ThroughputTypical Flows

Congestion

Flow ThroughputAdvantaged Flows

Delay

AggregatePacket

Throughput

Flow ThroughputVery Advantaged Flows

Aggregate Rate ofFlow Completions

Flow ThroughputTypical Flows

Congestion

Flow ThroughputAdvantaged Flows

Delay

AggregatePacket

Throughput

Flow ThroughputVery Advantaged Flows

Aggregate Rate ofFlow Completions


20


Principal Components Analysis Suggests 4 Dimensions

PC1 – Congestion

PC3 – Throughput for Advantaged Flows

PC2 – Delay

PC4 – Aggregate Throughput


21


Domain Expert Selects 8 Dimensions

Domain Expert Sides with the Correlation Analysis and Adds One Response (y1)


22


Sensitivity Analysis as a Model Validation Step

What parameters (or combinations of parameters)determine a model’s responses?

Does the influence of parameters on responsesmake sense to a domain expert?

Can any unexpected responses be explained anddo the explanations make sense?


23


11 of 20 MesoNet Parameters Selected for Sensitivity Analysis* selected parameter

other parameters fixed Network Parameters

Topology

Propagation Delay

Speed

Buffer Sizing

Network Parameters

Topology

Propagation Delay

Speed

Buffer Sizing

Simulation Control

Measurement‐Interval Size

Number of Measurement Intervals

Random‐Number Seed

Simulation Control

Measurement‐Interval Size

Number of Measurement Intervals

Random‐Number Seed

User Behavior

Think Time

File‐Size Distribution

Pattern of Long‐Lived Flows

User Behavior

Think Time

File‐Size Distribution

Pattern of Long‐Lived Flows

Protocol Parameters

Initial Congestion Window

Initial Slow‐Start Threshold

Type of Slow Start Regime

Protocol Parameters

Initial Congestion Window

Initial Slow‐Start Threshold

Type of Slow Start Regime

Characteristics of Sources & Receivers

Number of Sources

Distribution of Sources

Number of Receivers

Distribution of Receivers

Distribution of Network‐Interface Speeds

Distribution of Congestion‐Control Mechanisms

Startup Pattern for Sources

Characteristics of Sources & Receivers

Number of Sources


Number of Receivers


Distribution of Network‐Interface Speeds

Distribution of Congestion‐Control Mechanisms

Startup Pattern for Sources

***

*****

*

**None

Next slide

200 ms

6000

200000

2 packets

Limited slow start

All TCP

25% at a time


24


Representative Heterogeneous Four-Tier Topology SelectedFast access routers

Directly-connected access routers

Normal access routers


25


211-5 Orthogonal Fraction Factorial (OFF) Experiment Design

Design Template

Full Factorial Design Requires 211 = 2048 runs

We can only afford 26 = 64 runs; thus,

we require a 211-5 OFF Experiment Design

No confounding of main effects with two-terminteractions

Main effects may be confounded with three-termand higher interactions


26


Values Selected to Replace +1 and -1 in Design Template

Initial Slow-Start Threshold (packets)



Multiplier on Number Sources Per Access Router

Probability of Fast Network Interface*

Probability of a Larger File*

Average Think Time (ms)

Average File Size (packets)

Buffer Sizing Algorithm

Backbone Router Speed (ppms)*

Multiplier for Propagation Delay

Parameter

431.07x109x11Protocol Factors

WEBP2Px10

WEBP2Px9

23x8

0.40.2x7

Source &

Receiver Factors

0.020.01x6

20005000x5

50100x4UserFactors

RTTxC/SQR(n)RTTxCx3

800400x2

12x1NetworkFactors

-1+1Factor

*By convention +1 is coded with larger value and -1 is coded with smaller value. Here, coding was reversed in three cases.This makes no difference in the construction of the experiment, but must be managed during data analysis


27


Speed of All Routers Derived from Speed of Backbone Routers(Topology Tiers 1 to 3)

= x2/R2/R4xDC

= x2/R2/R4xFA

= x2/R2/R4

= x2/R2

= x2

Equation

DC (= 10)

FA (= 2)

R3 (= 10)

R2 (= 4)

x2

Parameter

200 ppms100 ppmsDirectly Connected Access

40 ppms20 ppmsFast Access

20 ppms10 ppmsTypical Access

200 ppms100 ppmsPOP

800 ppms400 ppmsBackbone

-1+1Router Type


28


Number & Distribution of Sources & Receivers (Tier 4)

45.2548.276.4627,840P2PWEB3

45.2548.276.4618,560P2PWEB2

75.5420.144.3241,700WEBP2P3

75.5420.144.3227,800WEBP2P2

45.2548.276.4627,840WEBWEB3

45.2548.276.4618,560WEBWEB2

75.5420.144.3241,700P2PP2P3

75.5420.144.3227,800P2PP2P2

% underN Routers

% underF Routers

% underD Routers

TotalSourcesx10x9x8

Sources

75.5420.144.32166,800P2PWEB3

75.5420.144.32111,200P2PWEB2

86.0611.472.45219,600WEBP2P3

86.0611.472.45146,400WEBP2P2

86.0611.472.45219,600WEBWEB3

86.0611.472.45146,400WEBWEB2

75.5420.144.32166,800P2PP2P3

75.5420.144.32111,200P2PP2P2

% underN Routers

% underF Routers

% underD Routers

Total Receiversx10x9x8

Receivers


29


Resulting Distribution of Flow Classes

Flow classes defined by relative locations of source & receiver

34.1845.589.726.833.380.279P2PWEB3

34.1845.589.726.833.380.279P2PWEB2

65.0126.002.315.570.990.106WEBP2P3

65.0126.002.315.570.990.106WEBP2P2

38.9546.745.536.671.920.159WEBWEB3

38.9546.745.536.671.920.159WEBWEB2

57.0630.434.056.521.740.186P2PP2P3

57.0630.434.056.521.740.186P2PP2P2

% NNFlows

% FNFlows

% FFFlows

% DNFlows

% DFFlows

% DDFlowsx10x9x8


30


Influence of other Factors on Average Buffer Sizes

258816277651068002

12948139325538001

12948139325534002

6474070162774001

Access RouterBuffers (avg.)

POP RouterBuffers (avg.)

Backbone Router Buffers

(avg.)x2x1

RTTxC

1052707288002

531353648001

531353644002

27681824001

Access RouterBuffers (avg.)

POP RouterBuffers (avg.)

Backbone Router Buffers

(avg.)x2x1

RTTxC/SQR(n)


31


Sensitivity Analysis Relies on Main Effects Plots


32


Sensitivity Analysis Driven by Correlation Analysis

Active Flows Retransmission Rate

Avg. TP NN Flows SRTT

Aggregate TP Flow Completion Rate Avg. TP DD Flows Avg. TP FF Flows

Seven Responses Chosen by Correlation Analysis + 1


33


Optional mode switch between TCP & alternate procedures:BIC, CTCP, HSTCP, H-TCP, Scalable TCP

Current timetime

Timeout for current SYNsynTO

Number of SYNs that have been sentsynSENT

Maximum number of SYNs to sendsynMAX

Timeout interval for initial SYNsynINT

DefinitionSymbol

Current timetime

Timeout for current SYNsynTO

Number of SYNs that have been sentsynSENT

Maximum number of SYNs to sendsynMAX

Timeout interval for initial SYNsynINT

DefinitionSymbol

Threshold to terminate initial slow start (varies with experiment)

sstINT

Threshold to switch from exponential to logarithmic increase (varies with experiment)

sstMAX

Current slow‐start thresholdsst

Initial congestion window (we use cwndINT = 2)cwndINT

Current congestion windowcwnd

DefinitionSymbol

Threshold to terminate initial slow start (varies with experiment)

sstINT

Threshold to switch from exponential to logarithmic increase (varies with experiment)

sstMAX

Current slow‐start thresholdsst

Initial congestion window (we use cwndINT = 2)cwndINT

Current congestion windowcwnd

DefinitionSymbol

Window increase procedures (ACK)

Window decrease procedures (Loss)

Timeout procedures

Optional periodic procedures:CTCP, FAST, H-TCP

Standard or Limited

MS Windows® TCP

Unified Model for Congestion-Control Algorithms


34


Sample: CTCP Congestion-Avoidance Model

Average Smoothed Round‐Trip Time experienced on CTCP flowSRTTC

Minimum round‐trip time experienced on CTCP flowminRTTC

Low‐window threshold ( LWC = 41) for applying CTCP proceduresLWC

Exponent (kC = 0.8) for CTCP window‐increase procedureskC

CTCP zeta parameter ( C = 0.1) defining reduction speed in delay windowC

Expected throughput (cwnd/minRTTC) on CTCP flowEC

CTCP delay windowdwnd

Difference between expected and actual throughput experienced on CTCP flowDC

CTCP gamma threshold ( C = 30) for detecting early congestionC

Boolean denoting whether early congestion has been detected (true) or not (false)CDC

Window decrease ( C = 0.5) weight for CTCPC

Actual throughput (cwnd/SRTTC) experienced on CTCP flowAC

Window increase ( C = 0.125) weight for CTCPC

DefinitionSymbol

Average Smoothed Round‐Trip Time experienced on CTCP flowSRTTC

Minimum round‐trip time experienced on CTCP flowminRTTC

Low‐window threshold ( LWC = 41) for applying CTCP proceduresLWC

Exponent (kC = 0.8) for CTCP window‐increase procedureskC

CTCP zeta parameter ( C = 0.1) defining reduction speed in delay windowC

Expected throughput (cwnd/minRTTC) on CTCP flowEC

CTCP delay windowdwnd

Difference between expected and actual throughput experienced on CTCP flowDC

CTCP gamma threshold ( C = 30) for detecting early congestionC

Boolean denoting whether early congestion has been detected (true) or not (false)CDC

Window decrease ( C = 0.5) weight for CTCPC

Actual throughput (cwnd/SRTTC) experienced on CTCP flowAC

Window increase ( C = 0.125) weight for CTCPC

DefinitionSymbol

Mode Switch Procedures

Periodic Procedures

IncreaseProcedures

DecreaseProcedures

TimeoutProcedures


35


Flow 1 Flow 1

Flow 2 Flow 2

PATH

Flow 1 Flow 1

Flow 2 Flow 2

PATH

PARAMETERS• Number of flows sharing path• Start time of each flow• Bottleneck bandwidth• Round-trip propagation delay• Number of buffers

1. Yee-Ting, Leith and Shorten, “Experimental Evaluation of TCP Protocols for High-Speed Networks”, IEEE/ACM Transactions on Networking, (15)5, October 2007, pp. 1109 – 1122.– Covers BIC, FAST, HSTCP, H-TCP, Scalable TCP– Uses implementations within Linux kernel

2. Leith, Andrew, Quetchenbach, Shorten and Lavi, “Experimental Evaluation of Delay/Loss-based TCP Congestion Control Algorithms”, Proceedings of the 6th International Workshop on Protocols for Fast Long-Distance Networks (PFLDnet 2008), March 5-7,2008, Manchester, UK.– Covers CTCP and TCP Illinois– Uses CTCP implementation in Windows® VISTA and TCP Illinois

implementation within Linux kernel

Empirical Studies of Six Alternate Congestion-Control Algorithms


36


Simulated Behavior of MesoNet CTCP Model

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 1 .1040

100

200

300

400

500

600

700

800

900

1000

1100

1200

Time

Con

gest

ion

Win

dow

avg. cwnd = 798 avg. cwnd = 836

avg. red cwnd = 419avg. blue cwnd = 417

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 1 .1040

100

200

300

400

500

600

700

800

900

1000

1100

1200

Time

Con

gest

ion

Win

dow



0 1000 2000 3000 4000 5000 6000 7000 8000 9000 1 .1040

500

1000

1500

2000

2500

3000

3500

4000

4500

Time

Con

gest

ion

Win

dow



0 1000 2000 3000 4000 5000 6000 7000 8000 9000 1 .1040

500

1000

1500

2000

2500

3000

3500

4000

4500

Time

Con

gest

ion

Win

dow



rtt = 162 ms

rtt = 42 ms

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 1 .1040

1000

2000

3000

4000

5000

6000

7000

8000

9000

Time

Con

gest

ion

Win

dow



0 1000 2000 3000 4000 5000 6000 7000 8000 9000 1 .1040

1000

2000

3000

4000

5000

6000

7000

8000

9000

Time

Con

gest

ion

Win

dow



rtt = 324 ms

MesoNet CTCP simulation behavioragrees with empirical results

Other MesoNet congestion-controlalgorithms also agree with empiricalresults


37


Comparing Alternate Congestion-Control Algorithms in a Large (up to 278,000 sources), Fast (up 192 Gbps) Homogeneous Network

Transmission Control Protocol (Reno)TCP7Scalable Transmission Control ProtocolScalable6Hamilton Transmission Control ProtocolHTCP5

High-Speed Transmission Control ProtocolHSTCP4

Fast Active-Queue Management Scalable Transmission Control ProtocolFAST3

Compound Transmission Control ProtocolCTCP2Binary Increase Congestion ControlBIC1Name of Congestion-Avoidance AlgorithmLabelIdentifier

Transmission Control Protocol (Reno)TCP7Scalable Transmission Control ProtocolScalable6Hamilton Transmission Control ProtocolHTCP5

High-Speed Transmission Control ProtocolHSTCP4

Fast Active-Queue Management Scalable Transmission Control ProtocolFAST3

Compound Transmission Control ProtocolCTCP2Binary Increase Congestion ControlBIC1Name of Congestion-Avoidance AlgorithmLabelIdentifier

Source & receiver under normal access routerNN

Source or receiver under fast access router and correspondent under normal access routerFN

Source or receiver under directly connected access router and correspondent under normal access routerDN

Typical

Source & receiver under fast access routerFF

Source or receiver under directly connected access router and correspondent under fast access routerDF

Fast

Source & receiver under directly connected access routerDDVery Fast

DefinitionFlow TypePath Class

Source & receiver under normal access routerNN

Source or receiver under fast access router and correspondent under normal access routerFN

Source or receiver under directly connected access router and correspondent under normal access routerDN

Typical

Source & receiver under fast access routerFF

Source or receiver under directly connected access router and correspondent under fast access routerDF

Fast

Source & receiver under directly connected access routerDDVery Fast

DefinitionFlow TypePath Class

7 Algorithms

3 Path Classes


38


Input Factors & Network Parameters

6 Robustness Factors

RTTxCapacity/SQR(N)RTTxCapacityBuffer Sizing Algorithmx650100File Sizex5

12Propagation Delayx4Skewed (.1/.6/.3)Uniform (.33/.33/.33)Source Distributionx3

25005000Think Timex240008000Network Speedx1

Minus (-1) ValuePLUS (+1) ValueDefinitionIdentifier

RTTxCapacity/SQR(N)RTTxCapacityBuffer Sizing Algorithmx650100File Sizex5

12Propagation Delayx4Skewed (.1/.6/.3)Uniform (.33/.33/.33)Source Distributionx3

25005000Think Timex240008000Network Speedx1

Minus (-1) ValuePLUS (+1) ValueDefinitionIdentifier

Router Speeds12 Gbps24 GbpsDirectly Connected Access

2.4 Gbps4.8 GbpsFast Access1.2 Gbps2.4 GbpsNormal Access12 Gbps24 GbpsPOP96 Gbps192 GbpsBackbone

Minus (-1)PLUS (+1)Router

12 Gbps24 GbpsDirectly Connected Access

2.4 Gbps4.8 GbpsFast Access1.2 Gbps2.4 GbpsNormal Access12 Gbps24 GbpsPOP96 Gbps192 GbpsBackbone

Minus (-1)PLUS (+1)Router

Propagation Delays (ms)100416Minus (-1)2008112PLUS (+1)

MaxAvgMin

100416Minus (-1)2008112PLUS (+1)

MaxAvgMin

Buffer Sizes (packets)

Number of Sources

174,600278,000Minus (-1)PLUS (+1)

174,600278,000Minus (-1)PLUS (+1)

25,879162,764

1,302,110Max

14,55791,555

732,437Avg

6,47040,691

325,528Min

PLUS (+1)

36920791Access908505221POP

4,6542,6061,153BackboneMaxAvgMinRouter

Minus (-1)

25,879162,764

1,302,110Max

14,55791,555

732,437Avg

6,47040,691

325,528Min

PLUS (+1)

36920791Access908505221POP

4,6542,6061,153BackboneMaxAvgMinRouter

Minus (-1)


39


Simulation Scenario & Long-Lived Flows

F0aI0a

K0a

Receiver Router

0.4 x 25 mins.E0aShort-distance flowL30.4 x 25 mins.C0aMedium-distance flowL2

0.4 x 25 mins.B0aLong-distance flowL1

Start TimeSource RouterDefinitionIdentifier

F0aI0a

K0a

Receiver Router

0.4 x 25 mins.E0aShort-distance flowL30.4 x 25 mins.C0aMedium-distance flowL2

0.4 x 25 mins.B0aLong-distance flowL1

Start TimeSource RouterDefinitionIdentifier

Long-Lived Flows

Scenario


40


32 Conditions Simulated (26-1 OFF Design)Factor-> X1 X2 X3 X4 X5 X6

Condition -- -- -- -- -- --1 4000 2500 .1/.6/.3 1 50 RTTxCapacity/SQR(N )2 8000 2500 .1/.6/.3 1 50 RTTxCapacity3 4000 5000 .1/.6/.3 1 50 RTTxCapacity4 8000 5000 .1/.6/.3 1 50 RTTxCapacity/SQR(N )5 4000 2500 .3/.3/.3 1 50 RTTxCapacity6 8000 2500 .3/.3/.3 1 50 RTTxCapacity/SQR(N )7 4000 5000 .3/.3/.3 1 50 RTTxCapacity/SQR(N )8 8000 5000 .3/.3/.3 1 50 RTTxCapacity9 4000 2500 .1/.6/.3 2 50 RTTxCapacity

10 8000 2500 .1/.6/.3 2 50 RTTxCapacity/SQR(N )11 4000 5000 .1/.6/.3 2 50 RTTxCapacity/SQR(N )12 8000 5000 .1/.6/.3 2 50 RTTxCapacity13 4000 2500 .3/.3/.3 2 50 RTTxCapacity/SQR(N )14 8000 2500 .3/.3/.3 2 50 RTTxCapacity15 4000 5000 .3/.3/.3 2 50 RTTxCapacity16 8000 5000 .3/.3/.3 2 50 RTTxCapacity/SQR(N )17 4000 2500 .1/.6/.3 1 100 RTTxCapacity18 8000 2500 .1/.6/.3 1 100 RTTxCapacity/SQR(N )19 4000 5000 .1/.6/.3 1 100 RTTxCapacity/SQR(N )20 8000 5000 .1/.6/.3 1 100 RTTxCapacity21 4000 2500 .3/.3/.3 1 100 RTTxCapacity/SQR(N )22 8000 2500 .3/.3/.3 1 100 RTTxCapacity23 4000 5000 .3/.3/.3 1 100 RTTxCapacity24 8000 5000 .3/.3/.3 1 100 RTTxCapacity/SQR(N )25 4000 2500 .1/.6/.3 2 100 RTTxCapacity/SQR(N )26 8000 2500 .1/.6/.3 2 100 RTTxCapacity27 4000 5000 .1/.6/.3 2 100 RTTxCapacity28 8000 5000 .1/.6/.3 2 100 RTTxCapacity/SQR(N )29 4000 2500 .3/.3/.3 2 100 RTTxCapacity30 8000 2500 .3/.3/.3 2 100 RTTxCapacity/SQR(N )31 4000 5000 .3/.3/.3 2 100 RTTxCapacity/SQR(N )32 8000 5000 .3/.3/.3 2 100 RTTxCapacity


41


Processor Time Requirements

6.61

1.51

3.46

110.5

Scalable

26.37(13.18x2)

6.5728.425.635.845.175.855.94Max. CPU time

(one run)

1.611.281.401.441.331.16Min. CPU time

(one run)

13.70(6.85x2)

3.3914.772.943.012.923.042.86Avg. CPU time

(per run)

219.1108.6472.594.296.493.497.291.5CPU time (32 runs)

TotalsHSTCPTotalsTCPHTCPFASTCTCPBIC

Compute Servers ws9-ws10Compute Servers ws11-ws14

6.61

1.51

3.46

110.5

Scalable

26.37(13.18x2)

6.5728.425.635.845.175.855.94Max. CPU time

(one run)

1.611.281.401.441.331.16Min. CPU time

(one run)

13.70(6.85x2)

3.3914.772.943.012.923.042.86Avg. CPU time

(per run)

219.1108.6472.594.296.493.497.291.5CPU time (32 runs)

TotalsHSTCPTotalsTCPHTCPFASTCTCPBIC

Compute Servers ws9-ws10Compute Servers ws11-ws14

(Units are CPU days)

Experiment required about 15 days of wall-clock time spread over 48 processors

1,548,371,719,08416,583,418,069Total All Runs11,917,420,154154,914,953Max. Per Condition3,146,870,57140,966,013Min. Per Condition6,912,373,74674,033,116Avg. Per Condition

Data Packets SentFlows CompletedStatistic

1,548,371,719,08416,583,418,069Total All Runs11,917,420,154154,914,953Max. Per Condition3,146,870,57140,966,013Min. Per Condition6,912,373,74674,033,116Avg. Per Condition

Data Packets SentFlows CompletedStatistic

Simulated Workload


42


Characterizing the 32 Conditions Simulated

0

0.1

0.2

0.3

0.4

0.5

0.6

12 8 20 2 32 26 3 14 4 15 27 9 16 10 28 11 18 24 22 19 17 6 1 29 7 30 5 25 23 13 31 21

Condition

Ret

rans

mis

sion

Rat

e

0

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

0.0014

0.0016

0.0018

0.002

12 8 20 2 32 26 3 14 4 15 27 9 16 10 28 11

Condition

Ret

rans

mis

sion

Rat

e

Max. = 0.0018

LN M

C

0

0.1

0.2

0.3

0.4

0.5

0.6

12 8 20 2 32 26 3 14 4 15 27 9 16 10 28 11 18 24 22 19 17 6 1 29 7 30 5 25 23 13 31 21

Condition

Ret

rans

mis

sion

Rat

e

0

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

0.0014

0.0016

0.0018

0.002

12 8 20 2 32 26 3 14 4 15 27 9 16 10 28 11

Condition

Ret

rans

mis

sion

Rat

e

Max. = 0.0018

0

0.1

0.2

0.3

0.4

0.5

0.6

12 8 20 2 32 26 3 14 4 15 27 9 16 10 28 11 18 24 22 19 17 6 1 29 7 30 5 25 23 13 31 21

Condition

Ret

rans

mis

sion

Rat

e

0

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

0.0014

0.0016

0.0018

0.002

12 8 20 2 32 26 3 14 4 15 27 9 16 10 28 11

Condition

Ret

rans

mis

sion

Rat

e

Max. = 0.0018

LN M

C

16 Uncongested and 16 Congested Conditions

3000 3500 4000 4500 5000 5500 6000 6500 7000 75000

1000

2000

3000

4000

5000

6000

7000

8000Evolution of Flows Condition 4 - TCP

Time

Act

ive

Flow

s

Sending Flows

Normal Congestion Control

Connecting Flows

Flows in Initial Slow Start

3000 3500 4000 4500 5000 5500 6000 6500 7000 75000

1000

2000

3000

4000

5000

6000

7000

8000Evolution of Flows Condition 4 - TCP

Time

Act

ive

Flow

s

Sending Flows


Connecting Flows


Evolution of Flow States in Uncongested Condition 4

3000 3500 4000 4500 5000 5500 6000 6500 7000 75000

2 .104

4 .104

6 .104

8 .104

1 .105

1.2 .105

1.4 .105Evolution of Flows Condition 5 - TCP

Time

Act

ive

Flow

s

Sending Flows


Connecting Flows


3000 3500 4000 4500 5000 5500 6000 6500 7000 75000

2 .104

4 .104

6 .104

8 .104

1 .105

1.2 .105

1.4 .105Evolution of Flows Condition 5 - TCP

Time

Act

ive

Flow

s

Sending Flows


Connecting Flows


Evolution of Flow States in Congested Condition 5

Cluster Analysis Annotated with Congestion Level


43


Cluster Analyses – Algorithm 3 (FAST) Stands OutTime Period 1 Time Period 2

Time Period 3 Aggregate Responses over 25 minutes


44


Approach to Detailed Analysis of Individual Responses

Time Period 1Retransmission Rate


45


Approach to Summarizing Detailed Analyses of Responses

Condition-Response Summary Time Period 1

Statistically significant and >10% Effect Filter


46


Condition-Response Summaries – Algorithm 3 (FAST) Stands OutTime Period 1 – 10% Filter Time Period 2 – 30% Filter

Time Period 3 – 30% Filter Aggregate Responses – No Filter


47


Why does Algorithm 3 (FAST) Stand Out?

FAST does not respond well to congestion, where too many flows compete for insufficient buffers – leading to rapid oscillation in flow congestion windows

0

50

100

150

200

250

300

350

4600 4700 4800 4900 5000 5100

Time

CW

ND

Evolution of congestion window under FAST for long-lived flow L2 during 500 measurement intervals within Time Period 2 under (the most congested) condition 21


48


Other Congestion-Control Algorithms Adjust Less Rapidly

Evolution of congestion window under other congestion-control algorithms for long-lived flow L2 during the same500 measurement intervals within Time Period 2 under (the most congested) condition 21

0

50

100

150

200

250

300

350

4600 4700 4800 4900 5000 5100

Time

CW

ND

0

50

100

150

200

250

300

350

4600 4700 4800 4900 5000 5100

TimeC

WN

D

TCP Reno BIC

0

50

100

150

200

250

300

350

4600 4700 4800 4900 5000 5100

Time

CW

ND

CTCP

0

50

100

150

200

250

300

350

4600 4700 4800 4900 5000 5100

Time

CW

ND

HSTCP

0

50

100

150

200

250

300

350

4600 4700 4800 4900 5000 5100

Time

CW

ND

HTCP

0

50

100

150

200

250

300

350

4600 4700 4800 4900 5000 5100

Time

CW

ND

Scalable


49


Summary of FAST Behavior under Congestion• Rapid oscillatory adjustment of congestion window

leads to:– Larger congestion-window increase rate– Higher retransmission rate (including for SYNs)– Larger number of flows pending in the connecting state

• Practical implications include:– Flows take longer to connect– Flows take longer to complete– Goodput is lower for flows transiting congested areas– Fewer (105 to 107, depending on condition) flows complete in a 25

minute period


50


Future Work on Congestion-Control Algorithms

• Does MesoNet reveal the same behavior when modeling a smaller, slower network with a much lower initial slow-start threshold?

• How do the congestion-control algorithms compare in a relatively uncongested, heterogeneous network with a wider range of traffic classes (e.g., web objects, documents, service packs and movie downloads)?

• Does the sensitivity analysis change when considering all 20 MesoNet parameters?


51


Summary of Presentation• Introduced NIST project to develop Measurement

Science for Complex Information Systems

• Showed an application of measurement science to compare seven alternate congestion-control algorithms for the Internet

• Identified a potential for FAST algorithm to behave undesirably under congested conditions– Explained the root cause for the potential undesirable behavior– Explained why other congestion-control algorithms are not

likely to exhibit the same potential undesirable behavior


52


Backup Information


53


Interactive Exploration of Multidimensional Data

Download from – http://math.nist.gov/mcsd/savg/software/divisa/


54


Sample Papers Produced by the Project (1 of 2)

C. Dabrowski, “Reliability in Grid Computing Systems”, in Concurrency and Computation: Practice and Experience,Wiley-Blackwell, in press.

D. Genin and V. Marbukh, "Do Current Fluid Approximation Models Capture TCP Instability?“, submitted to ICC 2009,Dresden, Germany, June 14 -18.

C. Dabrowksi and F. Hunt, “Using Markov Chain Analysis to Study Dynamic Behavior in Large-Scale Grid Systems”,Proceedings of the 7th Australasian Symposium on Grid Computing and e-Research, Wellington, New Zealand, Jan. 2009.

C. Dabrowski and F. Hunt, Markov Chain Analysis for Large-Scale Grid Systems, NIST Technical Report (under review).

D. Genin and V. Marbukh, "Metastability in cellular networks with migrating users: emergence and implications for performance.“ GLOBECOM 2008, New Orleans, Nov. 31 - Dec. 4.

K. Mills and C. Dabrowski, “Can Economics-based Resource Allocation Prove Effective in a Computation Marketplace?", Journal of Grid Computing, Vol. 6, No. 3, September 2008, pp. 291-311.

F. Hunt and V. Marbukh, “Dynamic Routing and Congestion Control Through Random Assignment of Routes”, Proceedings of the 5th International Conference on Cybernetics and Information Technologies, Systems and Applications: CITSA 2008, Orlando FL, July 2008.

V. Marbukh and K. Mills, "Demand Pricing & Resource Allocation in Market-based Compute Grids: A Model and Initial Results", Proceedings of the 7th International Conference on Networking, IEEE, April 2008, pp. 752-757.


55


Sample Papers Produced by the Project (2 of 2)

Marbukh and S. Klink, "Decentralized Control of Large-Scale Networks as a Game with Local Interactions: Cross-Layer TCP/IP Optimization", 2nd International Conference on Performance Evaluation Methodologies and Tools, Nantes, France, October 23-25, 2007.

V. Marbukh, "Utility Maximization for Resolving Throughput/Reliability Trade-offs in an Unreliable Network with Multipath Routing", 2nd International Conference on Performance Evaluation Methodologies and Tools, Nantes, France, October 23-25, 2007.

K. Mills, "A Brief Survey of Self-Organization in Wireless Sensor Networks", Wireless Communications and Mobile Computing, Wiley Interscience, Vol. 7, No. 7, September 2007, pp. 823-834.

V. Marbukh and K. Mills, "On Maximizing Provider Revenue in Market-Based Compute Grids", Proceedings of the 3rd International Conference on Networking and Services, Athens, Greece, June 19-25, 2007.

K. Mills and C. Dabrowski, "Investigating Global Behavior in Computing Grids", Self-Organizing Systems, Lecture Notes in Computer Science, Volume 4124 ISBN 978-3-540-37658-3, pp. 120-136.


56


Sensitivity Analysis Driven by Principal Components AnalysisPC1 - Congestion PC2 - Delay

PC3 - Avg. Throughput onAdvantaged Flows

PC4 - Aggregate Throughput


57


Major Factors Influencing Model Behavior

(1) Network Speed

(2) File Size

(3) Think Time

(4) Number of Sources

(5) Propagation Delay

(7) Buffer Size – small buffer size reduces delay variability

& larger buffer size has greater effect under high network speed

Correlation-based Analysis

Principal Components-based Analysis(6) Distribution of Sources

8964111032715Ordinal Rank

7.678.175.424.679.508.584.53.55.832.925.25Average Rank

7926111045831y20

5810.53558210.581y17

10107610854132y15

108351010551.51.57y10

7.57.554109326111y6

6.56.554119.523819.5y4

x11x10x9x8x7x6x5x4x3x2x1


7.678.175.424.679.508.584.53.55.832.925.25Average Rank

7926111045831y20

5810.53558210.581y17

10107610854132y15

108351010551.51.57y10

7.57.554109326111y6

6.56.554119.523819.5y4



7.258.636.755.509.759.253.254.004.002.505.13Average Rank

6897111023415PC4

10.594710.5861523PC3

5101148.58.537162PC2

7.57.534910.5256110.5PC1



7.258.636.755.509.759.253.254.004.002.505.13Average Rank

6897111023415PC4

10.594710.5861523PC3

5101148.58.537162PC2

7.57.534910.5256110.5PC1


measurement science for complex information systems...measurement science for complex information...

Documents