goprobe: a scalable distributed network monitoring solution · 2020. 8. 19. · netflow netflow...

goProbe: A Scalable Distributed Network Monitoring Solution

Christian DeckerLennart Elsen

Fabian KohnRoger Wattenhofer

Goal Enable quick and efficient retrieval of key pieces of information about traffic patterns in global networks

Goal Enable quick and efficient retrieval of key pieces of information about traffic patterns in global networks

Scalability

ReportingDebugging/Operations

? ??

StoragePacket Capture

Acquisition of Traffic Data

StoragePacket Capture

Grouping

Information Reduction

Acquisition of Traffic Data

NetFlow

Source IPDestination IPNext Layer ProtocolIPv4/6 Next Hop …

Source PortDestination Port…

Packet Size Number of PacketsSampling IntervalTTLInterface Name…

Field N Length

Field N Type

…

…

Field 2 Length

Field 2 Type

Field 1 Length

Field 1 Type

Count

…

System Uptime

Sequence #

NetFlow Version

Net

wor

kTr

ansp

ort

Met

a In

fo

NetFlow Packet

Packet aggregation by set of shared attributes

Network packet headers & packet counters

Expiry time

NetFlow

NetFlow Exporter

NetFlow Exporter

Network A

Network BNetFlow Collector

Source IPDestination IPNext Layer ProtocolIPv4/6 Next Hop …

Source PortDestination Port…

Packet Size Number of PacketsSampling IntervalTTLInterface Name…

Field N Length

Field N Type

…

…

Field 2 Length

Field 2 Type

Field 1 Length

Field 1 Type

Count

…

System Uptime

Sequence #

NetFlow Version

Net

wor

kTr

ansp

ort

Met

a In

fo

NetFlow Packet

d

Analysts

Current Network Monitoring System

Single Host

ExporterDB

Query Tool

Queries

Aggregated Results

Flow Data

Request Traffic Metadata

Formatted Results

FastBitnProbe

nProbeFastBit

Query Tool

Challenges Capturing Process

nProbeFastBit

Query Tool


Immense memory footprint


FastBitQuery Tool

One process per capture interfacenP

robe

nPro

be

nPro

be

FastBitQuery ToolnP

robe

nPro

be

nPro

be

Challenges Storage Backend


FastBitQuery ToolnP

robe

nPro

be

nPro

be

Inefficient memory management


FastBit

Query ToolnP

robe

nPro

be

nPro

be

No data compression


FastBit

Query ToolnP

robe

nPro

be

nPro

be

Long query execution times

Challenges

FastBit

Query ToolnP

robe

nPro

be

nPro

be

Poor Scalability

Reduced Flow Format

Src IP Dst IP IP Protocol Src Port Dst Port Packets

RcvdPackets

SentBytes Rcvd

Bytes Sent

Shared Attributes Counters

Reduced Flow Format

Src Port Dst Port

Shared Attributes Counters

Appl. Layer

Protocol

Deep Packet

Inspection

Reduced Flow Format

Src Port Dst Port

Deep Packet

Inspection

Appl. Layer

ProtocolDst Port

Source Port Aggregation

✗

Appl. Layer

ProtocolFlow in goProbe

Stored Flow

Collection of Flow Information — goProbe

goProbe

Written in Google Go

One capture routine per interface

Packet capture using modified libpcap

Database flush in regular intervals

TimerData Channel

Data Prepare

Local Database

Aggregation…

goProbe – Concept (Multiple Interfaces)

DB

Flow Table

Interface

How does it Compare?

Database Performance Evaluation

Reference DB

Runtime

CPU utilization

Disk I/O

Memory

7.8 GB

Aggregation Queries

Conditional Queries120 Million Entries

Data Read From Disk [MB]

FastBit

InfoBright EE

InfiniDB 1405

105

5617

350

74

2200

AggregationConditional

Runtime [s]

FastBit

InfoBright EE

InfiniDB 23

10

63

17

9

60

Reserved Memory [MB]

FastBit

InfoBright EE

InfiniDB 668

387

1399

630

351

3300

CPU Utilization [%]

FastBit

InfoBright EE

InfiniDB 83

213

17

302

352

23

Results

InfiniDBInfobright EE

$

File Based

Compression

Concurrency

Independent Processing

Tailored Column Store

goDB

Tailored Column Store — goDB

File Based

Compression

Concurrency

Independent Processing

Day 1

Destination IP

Source IP

Destination Port

IP Protocol

Appl. Layer Protocol

Bytes Received

Bytes Sent

Packets Received

Packets Sent

One File per Attribute

64

64

64

Day 1

Destination IP

Source IP

Destination Port

IP Protocol

Appl. Layer Protocol

Bytes Received

Bytes Sent

Packets Received

Packets Sent

One File per Attribute

64

64

64

172.0.50.4 | 10.30.0.3 | 8145 | 6 | 128 | 1024 | 1 | 8

Block-wise Writing and Reading

5 min 5 min 5 min

Attribute File

Block Timestamps

Length of Uncompressed Block

Position

Header

Compressed Block

Day

1D

ay d

…

Full

Dat

abas

eConcurrent Processing

Day

1D

ay d

…

Concurrent Processing

Day

1D

ay d

Worker 1

Worker dsip dip counters

sip dip counters

Partial Result Block i, Day 1

Partial Result Block j, Day d


Decompress Aggregate

Day

1D

ay d

Worker 1

Worker d



sip dip counters

sip dip counters

sip dip counters

Combined Result

Merge Routine



Day

1D

ay d

Worker 1

Worker d



sip dip counters

sip dip counters

sip dip counters

Combined Result

Merge Routine

Format Sort Limit



Data Read From Disk [MB]

FastBit

goDB760

5617

494

2200

AggregationConditional

Runtime [s]

FastBit

goDB20

63

13

60

Reserved Memory [MB]

FastBit

goDB50

1399

47

3300

CPU Utilization [%]

FastBit

goDB123

17

237

23

How does it Compare?

Traffic Portfolio of an NGO Customer

Global Breakdown of PortsEx

tern

al T

raffi

cIn

tern

al T

raffi

c

Global Breakdown of PortsEx

tern

al T

raffi

cIn

tern

al T

raffi

c

HTTPS

HTTP

SMBDNS

Global Breakdown of Ports European Hub Traffic UsageEx

tern

al T

raffi

cIn

tern

al T

raffi

c

https://github.com/open-ch/

Conclusion

Improved capturing and flow logic

High performance DB written from scratch

Global deployment

Open source:

goprobe: a scalable distributed network monitoring solution · 2020. 8. 19. · netflow netflow...

Documents