Network Management Automated Intelligence

GZ KabirBDCOM ONLINE LTD.

Open Source Based NMS solution

p Network Managementn Parametersn Componentsn Open Source Toolsn Demonstrationn Q & A

Parametersp Operation:

keeping the network (and the services that the network provides) up and running smoothly. It includes monitoring the network to spot problems as soon as possible, ideally before users are affected.

p Administration:

deals with keeping track of resources in the network and how they are assigned.

p Maintenance: concerned with performing repairs and upgrades. Maintenance also involves corrective and preventive measures to make the managed network run "better”.

p Provisioning:is concerned with configuring resources in the network to support a given service.

So … Network Management is the use of a system thatconstantly monitors a computer network forslow or failing systems and that notifies thenetwork administrator in case of outages viaemail, SMS or other alarms.

subset of the functions involved in networkmanagement.

Network Managementp System & Service monitoring

n Reachability, availabilityp Resource measurement/monitoring

n Capacity planning, availabilityp Performance monitoring (RTT, throughput)p Stats & Accounting/Meteringp Fault Management

n Fault detection, troubleshooting, and trackingn HIDS/HIPS

p Configuration/Change Managementp Coordination

A network in operation needs to be monitored in order to:-Deliver projected SLAs (Service Level

Agreements)-SLAs depend on policy

p What does your management expect?p What do your users expect?p What do your customers expect?p What does the rest of the Internet expect?

- What’s good enough? 99.999% Uptime?è There's no such thing as 100% uptime (as we’ll see) à

Expectations

What does it take to deliver 99.9 % uptime?30.5 days x 24 hours = 732 hours a month(732– (732 x .999)) x 60 = 44 minutes

only 44 minutes of downtime a month!

Need to shutdown 1 hour / week?(732 – 4) / 732x 100 = 99.4 %Remember to take planned maintenance into account in your

calculations, and inform your users/customers if they are included/excluded in the SLA

How is availability measured?In the core? End-to-end? From the Internet?

�Uptime�Expectations

BaseliningWhat is normal for your network?If you�ve never measured or monitored your network

you will need to know things like:n Typical load on links (è Cacti)n Level of jitter between endpoints (è Smokeping)n Typical percent usage of resources (è LibreNMS)

p Typical amounts of �noise�:p Network scansp Dropped datap Reported errors or failures

Know when to upgrade- Is your bandwidth usage too high?

- Where is your traffic going?

- Do you need to get a faster line, or more providers?

- Is the equipment too old?

Keep an audit trace of changes- Record all changes

- Makes it easier to find cause of problems due to

upgrades and configuration changes

Maintain history of network operationsn Using a ticket system lets you keep a history of events.

n Allows you to defend yourself and verify what happened

Why do all this?

ComponentspAvailabilitypReliabilitypPerformancepConfiguration Mgmt & MonitoringpNetwork Forensicp Intrusion Detection …p …. p …..pCoordination

Toolsn Diagnostic tools – used to test connectivity, ascertainthat a location is reachable, or a device is up – usuallyactive tools

n Monitoring tools – tools running in the background(”daemons” or services), which collect events, but canalso initiate their own probes (using diagnostic tools),and recording the output, in a scheduled fashion.

Toolsp Active tools

n Ping – test connectivity to a hostn Traceroute – show path to a hostn MTR – combination of ping + tracerouten SNMP collectors (polling)

p Passive toolsn log monitoring, SNMP trap receivers

p Automated toolsn SmokePing – record and graph latency to a

set of hosts, using ICMP (Ping) or other protocols

n MRTG – record and graph bandwidth usage on a switch port or network link, at regular intervals

n So MANY More .....

Tools … Reliabilityp SmokePing

n Keeps track of your network latency:n Best of breed latency visualisation.

n Interactive graph explorer.n Wide range of latency measurment plugins.

n Master/Slave System for distributed measurement.n Highly configurable alerting system.n Live Latency Charts with the most 'interesting' graphs.

n Free and OpenSource Software written in Perl

Tools … SmokePing

Tools … SmokePing

Tools … Performancep Cacti/MRTG

n A tool to monitor, store and present network and system/server statistics

n Designed around RRDTool with a specialemphasis on the graphical interface

n Almost all of Cacti's functionality can be configured via the Web.

n Uses RRDtool, PHP and stores data in MySQLn Supports the use of SNMP and graphics with

MRTGn Authentication Schemen Large Network Deployment

Tools … Cacti

Tools … Weathermap

Tools … Availabilityp Nagios

n server and service availability monitoringp Can monitor pretty much anythingp HTTP, SMTP, DNS, Disk space, CPU usage, ...p BGP, OSPF, Switch Port, room temperature, ..p Easy to write new plugins (extensions)

p Zabbix, ZenOSS, Hyperic, ... Many more Open Source...

Ø Log, Log, Log ....Ø Notification mechanism

Tools …. Nagios

Tools …. Nagios

Tools …. Nagios XI

Tools …. Nagios

Tools – Availability+ ReliabilityLibreNMSl SNMP-based auto-discover network monitoringl Derived from another project (Observium)l Written in PHP as a web applicationl Includes support for a wide range of hardware:

- Cisco, Linux, FreeBSD, Juniper, Brocade, Foundry, HP and many more- See http://docs.librenms.org/Support/Features/

- Over 100 supported!- Routers, Switches, Access Points, Security gateways, Hosts,

Printers, …

LibreNMS – Available metricsl CPU, memory and storage statisticsl Interface traffic, packet and detailed error

statistics (L2 and L3)l Temperature, fan speed, voltage, amperage,

power humidity and frequency sensorsl Users, processes, load average and uptime

statistics

LibreNMS – Available metrics cont.l Linux distribution detectionl Real-time interface traffic graphingl Device inventory collection (useful!)l Detailed IPv4, IPv6, TCP and UDP stack

statisticsl BGP and OSPF informationl MAC <-> IP address lookup

l Find which port an IP/MAC was last seen on

LibreNMS

LibreNMS

Tools … Configuration Mgmt & Monitoring

The ”Really Awesome New Cisco config Differ”pRancid

n Rancid is a configuration management tool that keepstrack of changes in the configurations of any sizenetwork equipment (Cisco, HP, Juniper, Foundry, etc.).Works on routers and switches. Automates retrieval ofthe configurations and archives them as backup tool,audit tool, blame allocation.

Tools … RANCID

The ”Really Awesome New Cisco configDiffer”

The data is stored in a VCS (Version Control System) which keepsn Track changes in the equipment configurationn Track changes in the hardware (S/N, modules)n Track version changes in the OS (IOS, CatOS

versions)n Find out what your colleagues have done

without telling you!n Recover from accidental configuration errors .

Tools … RANCID

RANCID

Tools … Net Forensicp Network Flow Analysis Tool

n NetFlow (C),n cflowd (F), n FlowScan (F), n Sniffer Pro (C), n argus (F),n i-Flow (C)n NFSen (F)n AS-STATS

Tools … NFSenp Network Flow Analysis Tool

n NFSenp Display netflow data: Flows, Packets and Bytes

using RRD (Round Robin Database).p Easily navigate through the netflow data.p Process the netflow data within the specified

time span.p Create history as well as continuous profiles.p Set alerts, based on various conditions.

Tools … NFSen

Tools … NFSen

Tools … NFSenTCP UDP

Flow s Packets Bytes Flow s Packets Bytes

Rank Port Count Port Count Port Count Port Count Port Count Port Count

1 80 39029 80 570630 80 111021671 53 116671 53 150335 12610 142186426

2 445 27833 25 83140 40936 88004359 6881 2388 12610 99433 28712 101344390

3 135 24572 40936 66203 25 52612168 39792 2276 28712 70901 40493 93146942

4 25 7881 445 53175 55893 43525223 15507 1904 40493 65155 46886 27824516

5 23 6761 135 49066 46395 39079355 43040 1611 15699 46682 57563 26436088

6 3128 4786 55893 37615 2889 30261886 60928 1588 1416 40540 62390 25767022

7 443 2999 46395 35068 1317 24692504 51012 1573 57563 37794 54505 25550351

8 22 2517 22 27489 49674 23472247 61295 1447 34018 37747 55893 23548341

9 9415 1275 443 26468 54311 23342821 5060 1309 21694 24942 40633 22940400

10 8080 1081 21651 25614 44879 23306526 49665 1225 46886 19468 40403 19544859

AS-STATSp A netflow/sflow collector

n Storing data in RRD filesp A cron program

n order the ASNs by level of traffic exchangedp A web interface to :

n See your traffic per ASN n See your traffic per LINK

How as-stats looks ? (Peers)

How as-stats looks ? (Link)

What it is good for ?

p BGP Traffic Engineering p Finding out who you should (try to) peer with p Knowing what's going on in your network p Planning for future expansion

Tools … IDS & IPSComputer Security is not something that you

can just add on when you need it.Proper planning, installation, monitoring and

maintenance all become part of a successful IDS/IPS implementation.

p OSSEC/WAZUHp Tri-Sentry (Host Sentry, NetSentry, Service

Sentry)p Nessus, Snortp Checkpoint, Cisco IPS, UTM (Cyberoam,

Barracuda) $$$ARE YOU AWARE OF YOUR NETWORK POLLUTION ….

Malicious Traffic Detection Tool

Host Event Detection

AIDE(Advanced Intrusion Detection Environment)

45

Network Detection Systems

46

47

Event Management

Logstash Kibana

p Open Source SECurityp Open Source Host-based Intrusion

Detection Systemp Provides protection for Windows, Linux,

Mac OS, Solaris and many *nix systemsp http://www.ossec.netp Founded by Daniel Cidp Current project managers – JB Cheng and

Vic Hargrave48

OSSEC Capabilitiesp Log analysisp File Integrity checking (Unix and

Windows)p Registry Integrity checking (Windows)p Host-based anomaly detection (for Unix –

rootkit detection)p Active Responsep PCI DSS Compliance

49

HIDS Advantagesp Monitors system behaviors that are not

evident from the network trafficp Can find persistent threats that penetrate

firewalls and network intrusion detection/prevention systems

50

OSSEC Architecture

OSSEC UI Screen

File Integrity Alert Sample

** Alert 1365550297.8499: mail - ossec,syscheck,2013 Apr 09 16:31:37 ubuntu->syscheckRule: 551 (level 7) -> 'Integrity checksum changed again

(2nd time).'Integrity checksum changed for:

'/etc/apt/apt.conf.d/01autoremove-kernels'

53

Log Analysis Alert Sample

** Alert 1365514728.3680: mail -syslog,dpkg,config_changed,

2013 Apr 09 06:38:48 ubuntu->/var/log/dpkg.logRule: 2902 (level 7) -> 'New dpkg (Debian Package)

installed.'2013-04-09 06:38:47 status installed linux-image-3.2.0-40-generic-

pae 3.2.0-40.64

54

PCI DSS Requirementp 10.5.5 - Use file-integrity monitoring or

change-detection software on logs to ensure that existing log data cannot be changed without generating alerts (although new data being added should not cause an alert)

p 11.5 - Deploy file-integrity monitoring software to alert personnel to unauthorized modification of critical system files, configuration files, or content files; and configure the software to perform critical file comparisons at least weekly

55

Enhancing OSSEC

Tools – Log ManagementElasticsearch

“flexible and powerful open source, distributed real-time search and analytics engine for your log”(http://www.elasticsearch.org)

n Easy to scale (Distributed)n Everything is one JSON call away (RESTful API)n Unleashed power of Lucene under the hood n Excellent Query DSLn Multi-tenancyn Support for advanced search features (Full Text)n Configurable and Extensible n Document Oriented n Schema free n Conflict managementn Active community

Tools – Log Management

Logstash Kibana

commercial

SIEM

Elastic Elasticity

Tools – Log ManagementElasticsearch, Logstash & Kibana (ELK)

New Kid on the Block -NLS

Tools … CollaborationSo, we have many Open Source/Commercial deployments

already to monitor our network.All the programs can generate alert/alarm on fault

detection.Need to centralize all the information.We need to collaborate these programsNeed NOCIts not a big Room/House – it’s a softwareIts –RT (the ticketing system)

Tools … RTRequest Trackerp RT is a battle-tested issue tracking system

which thousands of organizations use for n bug tracking, n help desk ticketing, n customer service, n workflow processes, n change management, n network operations, n And so on ..

Tools … RTRequest TrackerWhenever, wherever and however there is aproblem in the network the relevantmonitoring software will send a ticketdirectly to RT system and system adminswill know immediately via email or SMS.This automation will keep track of theSLA. RT has its own Help Desk system andescalation procedure.

Tools ... RTn Why are they important?

pTrack all events, failures and issuesn Focal point for help desk communicationn Use it to track all communications

pBoth internal and externaln Events originating from the outside:

pcustomer complaintsn Events originating from the inside:

pSystem outages (direct or indirect)pPlanned maintenance, upgrades, etc.

Tools ... RTl Why are they important?

- Track all events, failures and issuesl Focal point for help desk

communicationl Use it to track all communications

- Both internal and externall Events originating from the outside:

- customer complaintsl Events originating from the inside:

- System outages (direct or indirect)- Planned maintenance, upgrades, etc.

Tools … RT

Tools … The Big Cycle

Conclusion RT

Weather Map

NFSenNagiosSmokePingCacti

ALL IN ONE NETWORK MANAGEMENT SYSTEM

Rancid

Query Response - Conclusionp Link/Port wise Network bandwidth utilization. – Cacti, LibreNMS,

Weathermapp UPlink/Downlink graph with multi color option – Cacti, LibreNMSp Historical graph option (real time, 5 mins, day, week, month, year option) –

Cacti, LibreNMSp User group for access the MRTG sites. – Cactip Continuous rtt/jitter/packet loss monitoring of specific destinations –

Smokepingp A powerful tools which is simple and intuitive interface to the health and status

of your network. – Nagios, Cacti, LibreNMSp BGP down, Physical down, CPU, Memory all available with notification via mail &

SMS. – Nagios p Provide visibility for your TOP as destinations. – Netflow, NFSen, AS-STATSp Collects (Netflow collector) all the flows from all of our routers – nfDump,

NFSenp Tools/Application for link utilization monitoring for the whole network at a

glance -- Weathermapp Tools/Application for router configuration auto backup – RANCID p Syslog server. – Logstash + ElasticSearch + Kibanap Collaboration – RT p IDS --- OSSEC ++p Security ---- ??? Malware traffic detection

Tools USEDp Cactip Smokepingp Weathermapp Nagiosp LibreNMSp RANCIDp NFSENp AS-STATSp ELKp RTp Maltrailp OSSEC/WAZUH

???