traffic matrix estimation in non-stationary environments

Advanced Technology LaboratoriesAdvanced Technology Laboratories

Traffic Matrix Estimation in Non-Stationary Environments

Presented by R. L. Cruz

Department of Electrical & Computer EngineeringUniversity of California, San Diego

Joint work with

Antonio NucciNina Taft

Christophe Diot

NISS Affiliates Technology Day on Internet TomographyMarch 28, 2003

page 2Advanced Technology LaboratoriesAdvanced Technology Laboratories

The Traffic Matrix Estimation Problem

• Formulated in

Y. Vardi, “Network Tomography: Estimating Source-Destination Traffic From Link Data,” JASA, March 1995, Vol. 91, No. 433, Theory & Methods

page 3Advanced Technology LaboratoriesAdvanced Technology Laboratories

The Traffic Matrix Estimation Problem

ingress

egress

Xj

Xj

Yi

PoP (Point of Presence)

Y = A X

Link Measurement Vector

Routing Matrix

“Traffic Matrix”

page 4Advanced Technology LaboratoriesAdvanced Technology Laboratories

The Traffic Matrix Estimation Problem

• Importance of Problem: capacity planning, routing protocol configuration, load balancing policies, failover strategies, etc.

• Difficulties in Practice– missing data – synchronization of measurements (SNMP)– Non-Stationarity (our focus here)

• long convergence time needed to obtain estimates

page 5Advanced Technology LaboratoriesAdvanced Technology Laboratories

What is Non-Stationary?

•Traffic Itself is Non-Stationary

page 6Advanced Technology LaboratoriesAdvanced Technology Laboratories

What is Non-Stationary?

• Also, Routing is Non-Stationary– e.g. Due to Link Failures– Essence of Our Approach

• Purposely reconfigure routing in order to help estimate traffic matrix

– More information leads to more accurate estimates

• Effectively increases rank of A• We have developed algorithms to

reconfigure the routing for this purpose (beyond the scope of this talk)

page 7Advanced Technology LaboratoriesAdvanced Technology Laboratories

Outline of Remainder of Talk

• Describe the “Stationary” Method– Stationary traffic, non-stationary routing– Stationary traffic assumption is reasonable if we

always measure traffic at the same time of day (e.g. “peak period” of a work day)

• Briefly Describe the “Non-Stationary” Method– Both non-stationary traffic and non-stationary

routing– More complex but allows estimates to be obtained

much faster

page 8Advanced Technology LaboratoriesAdvanced Technology Laboratories

Network and Measurement Model

• Network with L links, N nodes, P=N(N-1) OD pair flows

– K measurement intervals, 1 ≤ k ≤ K – Y(k) is the link count vector at time k: (L x 1)– A(k) is the routing matrix at time k: (L x P)– X(k) is the O-D pair traffic vector at time k: (P

x 1)• X(k) = (x1(k) , x2(k) , … xP(k))T

Y(k) = A(k) X(k)

k [1,K]

Y(k) and A(k) can be truncated to reflect missing and redundant data

page 9Advanced Technology LaboratoriesAdvanced Technology Laboratories

Traffic Model: Stationary Case

x i(k) x i wi(k) k [1,K]

k

)(kxi

ix

)(kwi

• X(k) is the O-D pair traffic vector at time k: (P x 1)

X(k) = (x1(k) , x2(k) , … xP(k))T

X(k) = X + W(k)

W(k) : “Traffic Fluctuation Vector• Zero mean, covariance matrix B• B = diag(X)

page 10Advanced Technology LaboratoriesAdvanced Technology Laboratories

Matrix Notation

CWAXY

KkkWkAXkAkY

],1[)()()()(

)(

...

)1(

...

)(000

0...00

00)2(0

000)1(

)(

...

)1(

)(

...

)1( 1

KW

W

W

x

x

X

kA

A

A

C

kA

A

A

kY

Y

Y

P

where:

Linear system of equations:

[LK] [LK][P] [LK][KP] [KP][P]

Choose Routing Configurations such that

Rank(A) = P

page 11Advanced Technology LaboratoriesAdvanced Technology Laboratories

Traffic matrix Estimation-Stationary Case

• Initial Estimate: Use Psuedo-Inverse of A:

- does not require statistics of W (covariance B)• Gauss-Markov Theorem: Assume B is known

- Unbiased, minimum variance estimate- Coincides with Maximum Likelihood Estimate if W is Gaussian

Y = AX + CW

ˆ X (0) (AT A) 1 ATY

ˆ X (AT (CBCT ) 1 A) 1 AT (CBCT ) 1Y

page 12Advanced Technology LaboratoriesAdvanced Technology Laboratories

Traffic matrix Estimation-Stationary Case

Y = AX + CW

• Minimum Estimation Error:

(assumes B is known)

11 ))((])ˆ)(ˆ[( ACBCAXXXXE TTT

ˆ X (AT (CBCT ) 1 A) 1 AT (CBCT ) 1Y

page 13Advanced Technology LaboratoriesAdvanced Technology Laboratories

Traffic matrix Estimation-Stationary Case

• Recall we assume B = cov(W) satisfies B = diag(X)• Set

ˆ B (k ) diag( ˆ X (k ))

ˆ X (k1) (AT (C ˆ B (k )CT ) 1 A) 1 AT (C ˆ B (k )CT ) 1Y

• Recursion for Estimates:

page 14Advanced Technology LaboratoriesAdvanced Technology Laboratories

Traffic matrix Estimation-Stationary Case

• Our estimate is a solution to the equation:

ˆ X (AT (C diag( ˆ X ) CT ) 1 A) 1 AT (C diag( ˆ X ) CT ) 1Y

• Open questions for fixed point equation:- Existence of Solution?- Uniqueness?- Is solution an un-biased estimate?

page 15Advanced Technology LaboratoriesAdvanced Technology Laboratories

Numerical Example-Stationary case

N=10 nodes, L=24 links and P=90 connections.

Three set of OD pairs with mean x equal to:

– 500 Mbps, 2 Gbps and 4 Gbps.

Gaussian Traffic Fluctuations:

bx 20

page 16Advanced Technology LaboratoriesAdvanced Technology Laboratories

Stationary case: b=1 Samples/Snapshot=1

page 17Advanced Technology LaboratoriesAdvanced Technology Laboratories

Stationary case: b=1 Samples/Snapshot=1

page 18Advanced Technology LaboratoriesAdvanced Technology Laboratories

Stationary case: b=1 Samples/Snapshot=15

page 19Advanced Technology LaboratoriesAdvanced Technology Laboratories

Stationary case: b=1 Samples/Snapshot=15

page 20Advanced Technology LaboratoriesAdvanced Technology Laboratories

Stationary case: b=1.4 Samples/Snapshot=1

page 21Advanced Technology LaboratoriesAdvanced Technology Laboratories

Stationary case: b=1.4 Samples/Snapshot=1

page 22Advanced Technology LaboratoriesAdvanced Technology Laboratories

Stationary case: b=1.4 Samples/Snapshot=15

page 23Advanced Technology LaboratoriesAdvanced Technology Laboratories

Stationary case: b=1.4 Samples/Snapshot=15

page 24Advanced Technology LaboratoriesAdvanced Technology Laboratories

Stationary and Non-Stationary traffic

20 snapshots / 4 samples per snapshot / 5 min per sample

• Stationary Approach: 20 min per day (same time) / 20 days• Non-Stationary Approach: aggregate all the samples in one window time large 400 min (7 hours)

page 25Advanced Technology LaboratoriesAdvanced Technology Laboratories

Traffic Model: Non-Stationary Case

)()( kTxkx ii

• Each OD pair is cyclo-stationary:

• Each OD pair is modeled as:

• Fourier series expansion:

],1[)()()( Kkkwkmkx iii

)/2sin(

)/2cos()(

)(ˆ)(ˆ2

0

Nkn

Nknkb

kbmkm

n

N

nnini

b

]2,1[],0[

],1[],0[

bb

b

NNnNk

NnNk

)(kxi )(kxi

kk

)(kwi )(kwi

)(kmi)(kmi

page 26Advanced Technology LaboratoriesAdvanced Technology Laboratories

Mean estimation Results-Non Stationary case

Three set of OD pairs

where are linear independent Gaussian variables with:

)()2sin()( twfaxtx iiiiii

bix 20

3/2

3/

0

4

2

500

ii

Gbps

Gbps

Mbps

x

)(twi

page 27Advanced Technology LaboratoriesAdvanced Technology Laboratories

Non Stationary case: b=1 Link Count

page 28Advanced Technology LaboratoriesAdvanced Technology Laboratories

Non Stationary case: b=1 Mean estimation