FRNOG31,Paris–Oct2018

pmacctandnetworkanaly?cs

PaoloLucentepmacct

whoami

PaoloLucenteGitHub:paololucenteLinkedIn:plucenteDiggingdataoutofnetworksworldwideforfunandprofitformorethan10years

Introduc?on

FRNOG31,Paris–Oct2018

libpcap

pmacctisopen-source,free,GPL’edsoMware

maps

IGP

MySQL PgSQL SQLite

MongoDB BerkeleyDB

flat-files

RabbitMQ Kafka

memory tables

sFlow

tee

NetFlow IPFIX

NetFlow IPFIX

hNp://www.pmacct.net/

Streaming Telemetry

GeoIP

BGP

sFlow

BMP

pmacct:afewsimpleuse-cases

BMP

flat-files

tee

NetFlow IPFIX

sFlow

Kafka

IPFIX libpcap

pmacct:aslightlymorecomplexuse-case

BGP

flat-files

tee

NetFlow IPFIX

Kafka

MySQL

aggregation method #1

aggregation method #2

nfacctd

Theuse-caseformessagebrokers

Keypmacctnon-technicalfacts

§  15+yearsoldproject§  Can’tspellthenameaMertheseconddrink§  Free,open-source,independent§  Underac?vedevelopment§  Innova?onbeingintroduced§ Welldeployedaround,alsoinlargeSPs/IXPs§  ClosetotheSP/IXPcommunityneeds

BuildingaNetworkAnaly?cspipeline

FRNOG31,Paris–Oct2018

TypicalgoalsforNetworkAnaly?cs

§  BusinessIntelligence§  InsightintrafficpaNerns§  Supportpeeringdecisions§  Inves?ga?onofnetworkevents§  Capacityplanning§  TrafficEngineering

SamplepipelineforNetworkAnaly?cs

§  Inputdata(BGP,NetFlow,StreamingTelemetry,SNMP,…)

§  Collec?on(pmacct,homegrownSNMPpoller)§  Dataencoding(JSON,ApacheAvro,etc.)§  Distribu?on(Kafa)§  Enrichment(homegrowngluein$language)§  Inges?on(RDBMS,TSDB)§  Visualiza?on

SamplepipelineforNetworkAnaly?cs(cont.d)

ForwardingPlane

ControlPlane

Topology

DataCollection

Network

BigData&Analytics

BMP IPFIX

InSituOAM

sFlow

Netstream

Netflow

gRPC

gNMI

OpenConfigYANG

NetworkDevice Human/Machine

Collector Analytics

MessageBroker DataStorage

DataProcessing MessageBroker

Creditsto:T.Graf(Swisscom)@UBBF2018

GekngBGPtothecollector

§  LetpmacctcollectorBGPpeerwithallPEdevices:facingpeers,transitandcustomers•  Nobest-pathcomputa?onatthecollector:scalabilitypreferredtoop?mizingmemoryusage

•  Countsome50MBofmemoryperfull-rou?ngtable

§  SetthecollectorasiBGPpeeratthePEdevices:•  ConfigureitasaRRclient•  CollectoractsasiBGPpeeracross(sub-)ASboundaries

Gekngflowtelemetrytothecollector

§  Exportingress-onlymeasurementsatallPEdevices:facingpeers,transitandcustomers.•  Trafficisroutedtodes?na?on,soplentyofinforma?ononwhereit’sgoingto

•  It’scrucialinsteadtogetasmuchaspossibleaboutwheretrafficiscomingfrom,ie.:§  inputinterfaceatingressrouter§  sourceMACaddress

§  Performdatareduc?onatthePE(ie.sampling)

Forwardingplane/controlplanecorrela?on

§  AscalablereplacementforSNMP:•  Pushtechnology•  Subscribingtodataofinterest

§  Alongjourneytostandardiza?onahead:•  Models:Openconfigandvendor-specific•  Transport:tradi?onal,NetconfandgNMI•  RPC:Netconf(YANGPush)andgNMI•  Encoding:JSONandGPB

StreamingTelemetry

flat-files Streaming Telemetry Kafka

Telemetry dump at regular time intervals

pmtelemetryd

Telemetry real-time log

pmacct&StreamingTelemetry(1/2)

Streaming telemetry

flat-files

NetFlow IPFIX

Kafka

MySQL

aggregation method #1

aggregation method #2

nfacctd

pmacct&StreamingTelemetry(2/2)

DataEncoding

§  JSON•  Schemaless•  Canbecompressedsuccessfullyend-to-end•  Simple,easytotroubleshootanddebug•  OMenthatistheencodingsupportedatinges?on?me•  Similars:BSON,MsgPack

§  ApacheAvro•  Withschema•  Binaryformat(whenthingsgowrong..)•  Similars:ThriM,GPB,Capt’nProto

Distribu?on

§  Kafa:de-factostandardfordatashipping•  Easytomodeldifferentproducer-consumerarchitectures

•  pmaccthasaplugintoproducetoKafa•  MostTSDBscanconsumefromKafa

§  Peopleintheneedforrawdatacantapintothislayertoconsumedirectly

§  Intui?ve(thatdoesnotmeanstraighporward..)toscale-out,balanceandreplicate

Storingdatapersistently

§  IfyourcompanyrunsaBigDatashop,youmaywanttos?cktooneoftheirop?ons

§  Asyoumayverywellbetryingtoingestmillionsoftuplesperminute:•  Ifinhouse,discussdimensioning•  Ifincloud,thinkaboutcostsanddataprivacy

Storingdatapersistently(cont.d)

§  Otherwise,selectafewtechnologies:•  Inges?onmethodsandperformance•  Querymethodsandperformance•  SoMwarelanguage•  Supportop?ons

§  Testthem§  Chooseone§  CongratsonbecomingtheBigDatashopofyourcompany!J

Storingdatapersistently(cont.d)§  “noSQL”databases(BigDataJ):•  Abletohandlelarge?me-seriesdata-sets•  MeaningfulsubsetofSQLquerylanguage•  Innova?vestorageandindexingengines•  Scalable:clustering,spa?alandtemporalpar??oning•  UI-ready:ie.ELKandTICKstacks

§  Open-sourceRDBMS:•  Abletohandlelargedata-sets•  FlexibleandstandardizedSQLquerylanguage•  Solidstorageandindexingengines•  Scalable:clustering,spa?alandtemporalpar??oning

UIexample

PaoloLucentepaolo@pmacct.nethNp://www.pmacct.net/|hNps://github.com/pmacct/pmacct

pmacctandnetworkanaly?cs

FRNOG31,Paris–Oct2018

Bonusslides

FRNOG31,Paris–Oct2018

Telemetrydatacorrec?on

§  Telemetrydatamaygetimprecise(ie.duetosampling)

§  Useinterfacestatsasgoldstandard§ Moldtelemetrydata..tomatchinterfacestats:•  BuildsonTrafficMatrixes?ma?onmethods:§  Tutorial:BestPrac?cesforDeterminingtheTrafficMatrixin

IPNetworks,NANOG43•  Addstelemetrydatatolinearsystemtosolve•  Solvesystemsuchthatthereisstrictconformance

withlinkstatvalues,withothermeasurementsmatchedasbestpossible

Brieflyonscalability§  Asinglecollectormightnotfititall:•  Memory:can’tstoreallBGPfullrou?ngtables•  CPU:can’tcopewiththepaceoftelemetryexport

§  Divide-et-imperaapproachisvalid:•  AssignPEs(bothtelemetryandBGP)tocollectors•  Ifna?velysupportedDB:

§ AssigncollectorstoDBnodes§ ClustertheDB

•  Ifnot-na?velysupportedDB:§ Assigncollectorstomessagebrokers§ Clusterthemessaginginfrastructure