can drop but you can’t hide: persistent estimation in high...
TRANSCRIPT
You Can Drop but You Can’t Hide: ‐persistent Spread Estimation in High‐speed Networks
Presenter: Prof. Shigang Chen
He Huang1, Yu-E Sun2, Shigang Chen3, Shaojie Tang4,
Kai Han5, Jing Yuan6, Wenjian Yang1
1School of Computer Science and Technology, Soochow University, China2School of Rail Transportation, Soochow University, China
3Department of Computer and Information of Science and Engineering, University of Florida, US4Naveen Jindal School of Management, University of Texas at Dallas, US
5School of Computer Science and Technology, University of Science and Technology of China, China6Department of Computer Science, University of Texas at Dallas, US
19th April, 2018IEEE INFOCOM 2018
Traffic Measurement in High Speed Networks
2
Generalized Flow Size Measurement
Number of packets, number of bytes
Netflow
Generalized Flow Spread Measurement
Number of distinct elements in each flow, i.e. flow cardinality.
Scan detection, worm monitoring, proxy caching and content
access profiling, etc
Flow size v.s. Flow spread
3
1000000 packets
Size = 1000000, Spread = 1
……
1 packet
1 packet
1 packet
Size = 100Spread = 100
Persistent Spread
4
Stealthy DDoS attack, , , , ,
, , , , , Persistent element (source IP)
Limitation of Prior Art
5
Stealthy DDoS attack, , , , ,
, , , , , Persistent element (source IP)
Limitation 1: Only count persistent elements that appear in all periods
Limitation 2: Assume transient elements appear in one period
Problem Definition
6
We study a new problem called -persistent spreadestimation, which measures persist traffic elements ineach flow that appear during at least out of periods.
Other applications Identifying popular web files that are persistently accessed by
users over at least out of periods.
Profiling Internet access patterns
Monitoring scan activities
Online Persistent Traffic Measurement
7
Extremely high line speed
On-chip memory shared by Routing
Packet scheduling
Access control
Quality of service
Packet inspection and classification
Intrusion detection
Traffic measurement
How to fit in an extremely tight memory space!
Online Recoding
8
a bitmap for each flow f
0 0 0 0 0 0 0 0
0 1 2 3 4 5 6 71 11 1
Offline Operation: Bitwise SUM
9
1 0 0 1 0 0 0 1
0 1 2 3 4 5 6 7
,
,
,
, ,
,
0 0 0 1 0 1 0 1,
1 0 0 0 0 0 0 1,
Bitwise SUM0 0 2 0 1 0 32
Basic Idea
Known: , , fraction of counters whose valuesare
Unknown: , , number of elements that appearin out of measurement period.
Perdistent spread ∑ .
We derive the functional relationship between knownand unknown. , , provides T+1 equations to
solve for , .
0 0 2 0 1 0 32 V2 = 2 / 8
Per DestinationFlow
Recording Many Flows with Virtual Bitmaps
11
One physical bitmap for all
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1
1
1 11 11 111 1
1 0 1 1 0 1 0 1 0 1 0 1 1 0 1 1 1 1 0 1 1 0 0 1
One virtual bitmap for each flow
Virtual Bitmaps
12
Space saving
Implicit indexing
Noise in virtual bitmap
12
0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0
1
1 10 11 1 1
1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 1 1 0 0 1
Experiment Results
Experiment setup: Dataset
One hour of data downloaded from CAIDA
38963 distinct flows, and 7179130 distinct elements
General setWe set 5 minutes as one measurement period.
Each study incoudes 8 measurement periods, i.e. T .
Memory ranges from 0.25MB ∼ 1MB.
13
Experiment Results (cont.)
14
Experiment Results (cont.)
15
Base Station
Conclusion
A new traffic measurement problem that measuresnumber of persistent elements appearing in at leastout of predefined measurement periods.
A space-efficient solution for the problem
16
