building a campus network monitoring system for research

Building a Campus Network Monitoring System

for Research

Sue B. MoonEECS, Division of CS

2

Is Campus Network a Good Place to Monitor?

1GE/10GE/100GE link speedcomparable to backbone networks

•BcN (Broadband convergence Network) will turn access networks to backbone networks.

•B/W distinction between access and backbone may no longer exist.

Source of “innovation” research communities “invent” new things

•first users of new applications•new attacks / vulnerable machines•extreme types of usage

3

Speed Comparison

Last hop

LAN/MAN Long-Haul

1980 T1/T3

1990 64Kbps

10/100M EthernetFDDI rings

OC-3 ~ OC-12

2000 10 Mbps

100M/1GE/10GE

OC-48/192/768 (2.5/10/40G)

4

Is Campus Network a Good Place to Monitor?

Bureacratic overheadLower bar to tap (or so I believe)

Less sensitive to business

5

Goals

Share data with researchersGigascope with AT&T, UMass, ...KISTI

6

Data to Collect

Data PlanePacket tracesNetFlow dataSink hole data

Control PlaneRouting protocol tables/updatesRouter configurationSNMP statistics

7

Monitoring System Infrastructure

ComponentsDAGMONPCsStorageAnalysis platform

8

Projects in Mind

Port scanning activities

General study on security attacks

9

Overview

Definition and implications of small-time scaling behaviors

Queueing delay vs. Hurst parameter Observations from high-speed links Flow composition

Large vs. smallDense vs. sparse

Summary Future directions

10

Scaling Behaviors of Backbone Traffic

What does it mean? Fluctuations in traffic volume over time

• e.g. measured in 10ms, 1s or 1min intervals

Large-time scale (> 1 sec): Hurst parameter 0.5 <= H < 1, measure of “correlation” over

time H > 0.5, long-range dependent or asym. self-

similar

Small-time scale (1-100 ms): Important to queueing performance, router

buffer dimensioning

11

How to Represent Time Scales

Dyadic time index system Fixing a reference time scale T0

At scale j (or –j): Tj = T0 / 2 t j,k = (k Tj, (k+1) Tj) W j,k = 2j/2 (Tj+1,2k - Tj+1,2k+1)

j

12

Scaling Exponent and Wavelet Analysis

Energy function: Energy Plot: Second-order (local) scaling exponent: h

Suppose spectrum density function has the form

Long range dependence (asym. self-similar) process:

Fractional Brownian Motion: single h for all scales

][ ,2

kjj WE E

][,||~)( 2121 ,νν ν range frequency in ν νΓ h

],[)21(~ 12 jjj constant, hj Elog then j2

-j vs. Elog j2

5.0)21(~ H withj constant, Hj Elog j2

13

Hurst Parameter & (Avg.) Queueing Delay

Poisson model

FBM model(Fractional Brownian Motion)

H: Hurst parameter

H1H

ρ1(D ~

H =0.5 => Poisson

D ~11( ρ

22)( ~)( Hm mXVar

14

Traces

Collected from IPMON systemsOC3 to OC48 linksPeer, customer, intra-POP inter-router, inter-POP inter-router links

GPS timestamps40 bytes of header per packetTrace 1: domestic tier-2 ISP (OC12-tier2-dom)

Trace 2: large corporation (OC12-corp-dom)

15

Energy Plots

Trace 1 Trace 2

16

Observations

Large time scale Long-range dependent asymptotically “self-similar”

Small time scale: more “complex” Majority traces: uncorrelated or nearly

uncorrelated• Fluctuations in volume tend to be

“independent” Some traces: moderately correlated

17

Traffic Composition

How is traffic aggregated?By flow size

•Large vs. smallBy flow density

•Dense vs. sparse

18

Flow Composition: Large vs. Small

19

Byte Contribution

20

Impact of Large vs. Small Flows on Scalings

Flow size alone does not determine small-time scaling behaviors(cf. large-time scaling behaviors)

large: flow size > 1MB; small: flow size < 10KB

21

Dense vs. Sparse Flows

Density defined by inter-arrival times

22

PDF of packet inter-arrival times

23

Impact of Dense vs. Sparse Flows on Scalings

Flow density is a key factor in influencing small-time scalings!

dense: dominant packet inter-arrival time 2ms; sparse: > 2ms

24

Effect of Dense vs. Sparse Flow Traffic Composition

Semi-experiments using traces: vary mixing of dense/sparse flows

OC12-tier2-dom OC12-corp-dom

25

Where Does Correlation in Traffic Come From?

Effect of TCP window-based feedback control Sparse flows:

packets from small flows arrive “randomly”

Dense flows: Packets injected into network in bursts (window) Burst of packets arrive every round-trip-time(RTT)

Speed and location of bottleneck links matters! Larger bottleneck link => larger bursts Deeper inside the network => more corr. flows

26

So Within Internet Backbone Network …

Facts about today’s Internet backbone networks bottleneck links reside outside backbone networks bottleneck link speeds small relative to backbone linksHigh degree of aggregation of mostly independent

flows! Consequences:

Queueing delay likely negligible!• And easier to model and predict • More so with higher speed links (e.g., OC192)

Can increase link utilization Only higher degree of aggregation of independent

flowsBe cautious with high-speed “customer” links!

27

Will Things Change in the Future?

But what happens if More hosting/data centers and VPN customers

directly connected to the Internet backbone?• have higher speed links, large-volume data transfers

User access link speed significantly increased?• e.g., with more DSL, cable modem users

Larger file transfer? • e.g. distributed file sharing (of large music/video files)

UDP traffic increases significantly? • e.g. Video-on-Demand and other real-time applications

28

Status Quo of IP Backbone

Backbone network well-provisioned High-level of traffic aggregation

•Negligible delay jitter Low average link utilization

•< 30% Protection in layer 3

QoS? Not needed inside the backbone Is it ready for VoIP/Streaming media?

•Yet to be decided

29

Future Directions in Networking Research

RoutingNo QoS with current routing protocols

Performance issuesBcN: bottleneck moves closer to you!

Wired/wireless integrationSensitivity to lossE2e optimization

Security IPv6 vs NAT

30

Fraction of Packets in Loops

31

Single-Hop Queueing Delay PDF

32

Multi-Hop Queueing Delay CCDF

Data Set 3, Path 1

33

Multi-Hop Queueing Delay

Data Set 3

34

Impact of Bottleneck Link Load

90

35

Variable Delay Revisited: Tail

Data Set 3, Path 1

36

Peaks in Variable Delay

37

Closer Look

Queue Build up &Drain

Backup Slides

39

Impact of RTT

40

Impact of Traffic Composition

Trace 1 Trace 2

41

Small-Time Scalings ofLarge vs. Small Flows

42

Small-Time Scalings ofDense vs. Sparse Flows

43

Small-Time Scalings ofDense/Sparse Large Flows

44

Small-Time Scalings ofDense/Sparse Small Flows

45

Fourier Transform Plots

Trace 1 Trace 2

46

Gaussian?

Backbone traffic close to Gaussian due to high-level of aggregation

Kurtosis Close to 3

Skewness Close to 0

Trace 1

47

Illustrations of Small Time Scale Behaviors

(Nearly) Uncorrelated Moderately Correlated

NYC Nexxia (OC12) @Home PEN (OC-12)

48

What Affect the Small-Time Scalings?

composition of small vs. large flows “correlation structure” of large flows

49

Flow (/24) Size & Byte Distribution in 1-min Time Span

50

Where Does Correlation in Traffic Come From?

Effect of TCP window-based feedback control Small flows:

packets from small flows arrive “randomly” Large flows:

Packets injected into network in bursts (window) Burst of packets arrive every round-trip-time(RTT)

Speed and location of bottleneck links matters! Larger bottleneck link => larger bursts Deeper inside the network => more corr. flows

51

Three Distinct Time Scales: HTTP TCP Flows

52

Avg. Rate Distribution of Large TCP Flows

53


Facts about today’s Internet backbone networks bottleneck links reside outside backbone networks bottleneck link speeds small relative to backbone linksHigh degree of aggregation of (mostly) independent flows!

Consequences: Queueing delay likely negligible!

•And easier to model and predict •More so with higher speed links (e.g., OC192)

Can increase link utilization (while ensure little queueing)•Only higher degree of aggregation of independent flows

Be cautious with high-speed “customer” links!

54






UDP traffic increases significantly? • e.g. Video-on-Demand and other real-time applications

55

How Large Flows Affect Small Time Scalings?

56

Degree of Aggregation & Burst Sizes over Time Scales

57

Autocovariance of “Active” Flows over 1ms

58

Effect of TCP: Large vs. Small Flows

Three Distinct Time Scales Session time scale: on-off sessions

• file sizes, applications RTT Time Scale:

• TCP window-based feedback control• window size: burst of packets • RTT: prop. delay (+ random

variable) Inter-packet time scale

• packet sizes• TCP: ack-paced packet injection

Bottleneck Link & Queueing session duration clustered bursts, RTT inter-packet arrival times

59

Effect of Aggregation: (In-)dependence?

aggregating different (presumably independent) flows intermixing bursts and packets from different flows

Introduce independence (randomness) in the aggregate,

but also can induce “correlation” (due to TCP)! depending on where bottleneck link is!

different effects may manifest in different time scales!

60

Summary: Time and Space of Observation

What time scale we observe traffic matters! Where we observe traffic also matters! Large vs. small time scale behaviors

Large time scale:•superposition of many independent on-off sessions•heavy-tail file size distribution => self-similar scaling

Small time scale: more “complex”!• degree of aggregation•composition of large vs. small flows• correlation structure of bursts (of large flows)

Small-Time Scaling Behaviors of

Internet Backbone TrafficZhi-Li Zhang

U. of MinnesotaJoint work with

Vinay Ribeiro (Rice U.), andSue Moon, Christophe Diot (Sprint ATL)

62

Scaling Exponent and Wavelet Analysis

Energy function: Energy Plot: Second-order (local) scaling exponent: h

Suppose spectrum density function has the form

Long range dependence (asym. self-similar) process:

Fractional Brownian Motion: single h for all scales

Multi-scale Fractional Brownian: multiple h’s

][ ,2

kjj WE E

][,||~)( 2121 ,νν ν range frequency in ν νΓ h

],[)21(~ 12 jjj constant, hj Elog then j2

-j vs. Elog j2

time)-(large Jj for H and time),-(small Jj for h e.g.,

5.0)21(~ H withj constant, Hj Elog j2

63

Importance of Scaling Exponents

Poisson model

FBM model (Fractional Brownian

Motion) H: scaling exponent Var(t) ~

H1H

ρ1(D ~

H =0.5 => Poisson

2Ht

D ~11( ρ

64

Observations on OC3/OC12/OC48 Links

Large time scale Long-range dependent, asymptotically self-similar

Small time scale: more “complex” behavior Majority traces: (nearly) uncorrelated

• fluctuations in volume almost “independent” Some traces: moderately correlated

Small time scaling behavior: link specific (mostly) independent of link utilization observed

65

Illustrations of Scaling Behaviors

(Nearly) Uncorrelated Slightly Correlated

OC3-tier1-dom OC48-bb-1

66

Illustrations of Scaling Behaviors (cont’d)

(Nearly) Uncorrelated Moderately Correlated


67

Relation between SDF and Scaling Exponent

OC12-tier2-dom

OC12-corp-dom

68

Multi-Fractal Scaling Analysis

Linearity of => Monofractal scaling

Based on wavelet partition functions:


q

qh q constantqj~ qSlog qqqqqj /,2/,)(2

|| )( ,q

kjj WEqS

69

Multi-Fractal Scaling Analysis (cont’d)

Gaussian marginals => Monofractal scaling

Marginal distributions over 4 ms time scale

OC12-Tier2-Dom OC12-Corp-Dom

Kurtosis: 3.04Skew: 0.2

Kurtosis: 2.86Skew: 0.24

70

What affect the small-time scalings?

Internet traffic comprised of many individual flows e.g., 5-tuple flows

Flow classifications, based on Flow size: total bytes belonging to a flow in a time span

• small vs. large flows Flow density: dominant inter-packet arrival times of a

flow• dense vs. sparse flows

Traffic composition analysis Separate aggregate into large/small, dense/sparse flows Understand composition of large/small, dense/sparse

flows

71

Large vs. Small Flows

Based on 5 1-min segment of packet traces, each one hour apart

72

Dense vs. Sparse Flows

a dense flow

a sparse flow

“cumulative” packet inter-arrival times of all flows

73

Impact of Large vs. Small Flows on Scalings

Flow size alone does not determine small-time scaling behaviors(cf. large-time scaling behaviors)

large: flow size > 1MB; small: flow size < 10KB

74

Impact of Dense vs. Sparse Flows on Scalings

Flow density is a key factor in influencing small-time scalings!

dense: dominant packet inter-arrival time 2ms; sparse: > 2ms

75

Effect of Dense vs. Sparse Flow Traffic Composition

Semi-experiments using traces: vary mixing of dense/sparse flows


76

Where does correlation in traffic come from?

Aggregation of relatively large proportion of dense flows OC12-corp-dom: >2% dense flows, >15% total

bytes OC12-corp-dom: <1% dense flows, < 4% total

bytes Density of flows:

likely due to bottleneck link speed coupled with TCP window-based feedback control “fatter” bottleneck links => more dense flows

OC12-corp-dom: connect more high-speed users

OC12-tier2-dom: connect more diverse users

77


Facts about today’s Internet backbone networks bottleneck links reside outside backbone networks bottleneck link speeds small relative to backbone linksHigh degree of aggregation of (mostly) independent flows!

Consequences: queueing delay likely negligible!

• and (relatively) easier to model and predict • more so with higher speed links (e.g., OC192)

can increase link utilization (while ensure little queueing)• only higher degree of aggregation of independent flows

Be cautious with high-speed “customer” links!

78






UDP traffic increases significantly? • e.g. video-on-Demand and other real-time applications

building a campus network monitoring system for research

Documents