shivkumar kalyanaraman rensselaer polytechnic institute 1 slides adapted from paul barford...
Post on 18-Dec-2015
218 Views
Preview:
TRANSCRIPT
Shivkumar KalyanaramanRensselaer Polytechnic Institute
1Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Internet Measurement(and some inference & modeling)
Shivkumar (“Shiv”) Kalyanaraman
Rensselaer Polytechnic Institute
shivkuma@ecse.rpi.eduhttp://www.ecse.rpi.edu/Homepages/shivkuma/
GOOGLE: “Shiv RPI”
Shivkumar KalyanaramanRensselaer Polytechnic Institute
2Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Topics Measurement philosophy: why, what, when, where, how? Some measurement projects & results Techniques: passive & active
Packet tracing SNMP Probing
Inference and Modeling Tomography & Traffic Matrix Estimation for network
engineering Traffic modeling Rocketfuel: inferring topologies from outside ISP
networks
Shivkumar KalyanaramanRensselaer Polytechnic Institute
3Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Why Measurement? We built it, we depend on it, so we must try to understand
it … as it works in reality... Measurement gives us the data and basis for this
understanding.Modeling, Inference etc to get new understanding &
learning from data Complex interactions between protocols not well
modeled during their design. Need support for troubleshooting and network
management Wide area behavior unpredictable Change is normal
Shivkumar KalyanaramanRensselaer Polytechnic Institute
4Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Characteristics of the Internet The Internet is
Decentralized (loose confederation of peers) Self-configuring (no global registry of topology) Stateless (limited information in the routers) Connectionless (no fixed connection between hosts)
These attributes contribute To the success of Internet To the rapid growth of the Internet … and the difficulty of controlling the Internet!
ISPsender receiver
Shivkumar KalyanaramanRensselaer Polytechnic Institute
5Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Internet Measurement Challenges Size of the Internet
O(100M) hosts, O(1M) routers, O(10K) networks Complexity of the Internet
Components, protocols, applications, users Constant change is the norm
Web, e-commerce, peer-to-peer, wireless, next? The Internet was not developed with measurement as a
fundamental feature Nearly every network operator would like to keep most
data on their network private Floyd and Paxson, “Difficulties in Simulating the Internet”,
IEEE/ACM Transactions on Networking, 2000.
Shivkumar KalyanaramanRensselaer Polytechnic Institute
6Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Themes
Measurement has been the basis for critical improvements Without measurement, what do you know?
Measurement capability in the Internet is limited The systems not designed to support measurement
Measurement tools and infrastructures are few and limited Size, diversity, complexity and change
Measurement data presents many challenges Networking researchers need better connections with
experts in other domains
Shivkumar KalyanaramanRensselaer Polytechnic Institute
7Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Operator Philosophy: Tension With IP
Accountability of network resources But, routers don’t maintain state about transfers But, measurement isn’t part of the infrastructure
Reliability/predictability of services But, IP doesn’t provide performance guarantees But, equipment is not especially reliable (no “five-9s”)
Fine-grain control over the network But, routers don’t do fine-grain resource allocation But, network automatically re-routes after failures
End-to-end control over communication But, end hosts and applications adapt to congestion But, traffic may traverse multiple domains of control
Shivkumar KalyanaramanRensselaer Polytechnic Institute
8Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Network Operations: Measure, Model, and Control
Topology/Configuratio
n
Offeredtraffic
Changes tothe network
Operational network
Network-wide“what-if”
model
measure
control
Shivkumar KalyanaramanRensselaer Polytechnic Institute
9Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
“Operations” Research: Detect, Diagnose, and Fix
Detect: note the symptoms of a problem Periodic polling of link load statistics Active probes measuring performance Customer complaining (via the phone network?)
Diagnose: identify the illness Change in user behavior? Router/link failure or policy change? Denial of service attack?
Fix: select and dispense the medicine Routing protocol reconfiguration Installation of packet filters
Network measurement plays a key role in each step!
Shivkumar KalyanaramanRensselaer Polytechnic Institute
10Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
11Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Traffic Measurement: Control vs. Discovery
Discovery: characterizing the network End-to-end characteristics of delay, throughput, and
loss Verification of models of TCP congestion control Workload models capturing the behavior of Web users Understanding self-similarity/multi-fractal traffic
Control: managing the network Generating reports for customers and internal groups Diagnosing performance and reliability problems Tuning the configuration of the network to the traffic Planning outlay of equipment (routers, proxies, links)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
12Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
13Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
14Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
15Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
16Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
17Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
18Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
19Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
20Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
21Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
22Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
23Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
24Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
25Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
26Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
27Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
28Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
29Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
30Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
31Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
32Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
33Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
34Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
35Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
36Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
37Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
38Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
39Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
40Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
41Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
42Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
43Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Measurement Techniques
Shivkumar KalyanaramanRensselaer Polytechnic Institute
44Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Time Scales for Network Operations Minutes to hours
Denial-of-service attacks Router and link failures Serious congestion
Hours to weeks Time-of-day or day-of-week engineering Outlay of new routers and links Addition/deletion of customers or peers
Weeks to years Planning of new capacity and topology changes Evaluation of network designs and routing protocols
Shivkumar KalyanaramanRensselaer Polytechnic Institute
45Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Traffic Measurement: SNMP Data Simple Network Management Protocol (SNMP)
Router CPU utilization, link utilization, link loss, … Collected from every router/link every few minutes
Applications Detecting overloaded links and sudden traffic shifts Inferring the domain-wide traffic matrix
Advantage Open standard, available for every router and link
Disadvantage Coarse granularity, both spatially and temporally
Shivkumar KalyanaramanRensselaer Polytechnic Institute
46Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
47Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Traffic Measurement: Packet-Level Traces
Packet monitoring IP, TCP/UDP, and application-level headers Collected by tapping individual links in the network
Applications Fine-grain timing of the packets on the link Fine-grain view of packet header fields
Advantages Most detailed view possible at the IP level
Disadvantages Expensive to have in more than a few locations Challenging to collect on very high-speed links Extremely high volume of measurement data
Shivkumar KalyanaramanRensselaer Polytechnic Institute
48Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
49Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Extracting Data from IP Packets
IPTCP
IPTCP
IPTCP
Application message (e.g., HTTP response)
Many layers of information– IP: source/dest IP addresses, protocol (TCP/UDP), …
– TCP/UDP: src/dest port numbers, seq/ack, flags, …
– Application: URL, user keystrokes, BGP updates,…
Shivkumar KalyanaramanRensselaer Polytechnic Institute
50Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
flow 1 flow 2 flow 3 flow 4
Aggregating Packets into Flows
Set of packets that “belong together” Source/destination IP addresses and port numbers Same protocol, ToS bits, … Same input/output interfaces at a router (if known)
Packets that are “close” together in time Maximum inter-packet spacing (e.g., 15 sec, 30 sec) Example: flows 2 and 4 are different flows due to time
Shivkumar KalyanaramanRensselaer Polytechnic Institute
51Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
52Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
53Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
54Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
55Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
56Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
57Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
58Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
59Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
60Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
61Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
62Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
63Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
64Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
65Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
66Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
67Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
68Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
69Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
70Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
71Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Summary: Traffic Measurement: Flow-Level Traces
Flow monitoring (e.g., Cisco Netflow) Measurements at the level of sets of related packets Single list of shared attributes (addresses, port #s, …) Number of bytes and packets, start and finish times
Applications Computing application mix and detecting DoS attacks Measuring the traffic matrix for the network
Advantages Medium-grain traffic view, supported on some routers
Disadvantages Not uniformly supported across router products Large data volume, and may slow down some routers Memory overhead (size of flow cache) grows with link speed
Shivkumar KalyanaramanRensselaer Polytechnic Institute
72Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Summary: Reducing Packet/Flow Measurement Overhead
Filtering: select a subset of the traffic E.g., destination prefix for a customer E.g., port number for an application (e.g., 80 for Web)
Aggregation: grouping related traffic E.g., packets/flows with same next-hop AS E.g., packets/flows destined to a particular service
Sampling: subselecting the traffic Random, deterministic, or hash-based sampling 1-out-of-n or stratified based on packet/flow size
Combining filtering, aggregation, and sampling
Shivkumar KalyanaramanRensselaer Polytechnic Institute
73Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Summary: Comparison of Techniques
SamplingSamplingFilteringFiltering AggregationAggregation
Generality
LocalProcessing
Local memory
Compression
Precision exact exact approximate
constraineda-priori
constraineda-priori
general
filter criterionfor every object
table updatefor every object
only samplingdecision
none one bin pervalue of interest
none
dependson data
dependson data controlled
Shivkumar KalyanaramanRensselaer Polytechnic Institute
74Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
75Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
76Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
77Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
78Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
79Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Inference and Modeling…
Shivkumar KalyanaramanRensselaer Polytechnic Institute
80Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
DATA-DRIVEN…
Shivkumar KalyanaramanRensselaer Polytechnic Institute
81Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
82Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Eg: The Network Design Problem
200
65
258
134
30
42Düsseldorf
Frankfurt
Berlin
Hamburg
München
Communication Demands
Düsseldorf
Frankfurt
Berlin
Hamburg
München
Potential topology &
Capacities
Shivkumar KalyanaramanRensselaer Polytechnic Institute
83Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
84Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
85Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Traffic Modeling …
Shivkumar KalyanaramanRensselaer Polytechnic Institute
86Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
87Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
88Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
89Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
90Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
91Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
92Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
93Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
94Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
95Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
96Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Mandelbrot’s Construction
Renewal reward processes and their aggregates Aggregate is made up of many constituents Each constituent is of the on/off type On/off periods have a “duration” Constituents make contributions (“rewards”) when “on” Constituents make no contributions when “off”
What can be said about the aggregate? In terms of assumed type of “randomness” for
durations and rewards In terms of implied type of “burstiness”
Shivkumar KalyanaramanRensselaer Polytechnic Institute
97Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Mandelbrot’s Types of “Randomness” Distribution functions/random variables
“Mild” → finite variance (Gaussian) “Wild” → infinite variance
Correlation function of stochastic process None => “IID” (independent, identically distributed) “Mild” → short-range dependence (SRD, Markovian) “Wild” → long-range dependence (LRD)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
98Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Mandelbrot’s Types of “Burstiness”
Bursty BURSTY
smooth bursty
Dis
trib
utio
n fu
nctio
n
M
ild
Wild
Mild Wild
Correlation structure
• Tail-driven burstiness (“Noah effect”)• Dependence-driven burstiness (“Joseph effect”)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
99Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Type of Burstiness: “Smooth”
White Noise
Correlation Function r(n)
lag n on linear scale
r(n
) o
n l
og
sca
le
CCDF Function 1-F(x)
x on linear scale
1-F
(x)
on
lo
g s
cal
e
Log-linear scales
Shivkumar KalyanaramanRensselaer Polytechnic Institute
100Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Type of Burstiness: “bursty”
Colored Noise
Correlation Function r(n)
lag n on log scale
r(n
) o
n l
og
sca
le
CCDF Function 1-F(x)
x on linear scale
1-F
(x)
on
lo
g s
cal
eLog-linear scale
Log-log scale
Shivkumar KalyanaramanRensselaer Polytechnic Institute
101Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Type of Burstiness: “Bursty”
Stable Noise
Correlation Function r(n)
lag n on linear scale
r(n
) o
n l
og
sca
le
CCDF Function 1-F(x)
x on log scale
1-F
(x)
on
lo
g s
cal
eLog-log scale
Log-linear scale
?
Shivkumar KalyanaramanRensselaer Polytechnic Institute
102Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Colored Stable Noise
Type of Burstiness: “BURSTY”
CCDF Function 1-F(x)
x on log scale
1-F
(x)
on
lo
g s
cal
e
Correlation Function r(n)
lag n on log scale
r(n
) o
n l
og
sca
le
?
?
Log-log scales
Shivkumar KalyanaramanRensselaer Polytechnic Institute
103Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Mandelbrot’s Types of “Burstiness”
Bursty BURSTY
smooth bursty
Dis
trib
utio
n fu
nctio
n
M
ild
Wild
Mild Wild
Correlation structure
• Tail-driven burstiness (“Noah effect”)• Dependence-driven burstiness (“Joseph effect”)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
104Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Inference For Network Engineering: Traffic Matrix Estimation…
(Truncated: Detailed slides in 2005 class)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
105Slides adapted from Paul Barford (UWisconsin), Matt Roughan (U Adelaide), Jennifer Rexford (Princeton)
Summary
Internet measurement is a fast growing and complex field
We saw brief glimpses of the area: a full course required to do justiceActive vs passive probingData management and miningModeling and inference
A new book also available! Crovella and Krishnamurthy.
top related