system integration and analysis
TRANSCRIPT
![Page 1: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/1.jpg)
System integration and analysisBy José M. Román
FiberCli
We are pure fiber
![Page 2: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/2.jpg)
• (Now) CEO at Fibercli
• (2015 –Now) CEO @ WISP Cloud Networking Spain
• (2008 –Now) Security Consultant and Analyst
• (2000 –2007) Networking, security and itil teacher .
17 years experience, Mikrotik Certified Consultant and Trainer.MTCNA,
MTCRE,MTCTCE, MTCUME,MTCWE,MTCINE,CISA,CISSP,MasterITIL
@ M A F I A S O L E H
M A D R I D / P R A G U E
J o s e . r o m a n @ f i b e r c l i . c o m
+ 3 4 6 5 2 2 4 1 4 3 1
JOSE MANUEL ROMAN
2José Manuel Román para FiberCli
![Page 3: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/3.jpg)
F A J A R N U G R O H O
• (2015 –2016) Infrastucture (System, Network & Security) Engineer. @
Technology and Information Department of Jakarta Capital
Cityand Jakarta SmartCity
• (2012 –Now) Freelancer @ SMB to Enterprise customers
• (2008 –2012) Helpdesk, NOC (Network Operator Center). @ Wireless
Internet Service Provider and Triple Play (CaTV, VoIP and
Internet) Service Provider
Network Engineer by Job and Troublemaker by Act, currently
focusing on MikroTik, Juniper, Arista, UBNT, Vmware Virtualization,
Linux/Unix (Debian & FreeBSD). CCNA, MTCNA, MTCRE, MTCTCE,
JNCIA, JNCIS-ENT, JNCIS-SP, JNCIP-SP,MikroTikCertifiedTrainer.
@ M A F I A S O L E H
T O L E D O / J A K A R T A
f a j a r @ f i b e r c l i . c o m
+ 6 2 8 1 3 1 7 7 7 1 4 5 5
3José Manuel Román para FiberCli
![Page 4: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/4.jpg)
Fiber optic key projectsLevel 3 support 24 x 7 for ISP’s
Mikrotik certificationsSystem and software integration
4José Manuel Román para FiberCli
![Page 5: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/5.jpg)
20% Disccount forMUM assistant
5José Manuel Román para FiberCli10/17/2016
![Page 6: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/6.jpg)
Problem
Multiple events on the network as a system administrator or network administrator we don’t know locate the source.
6José Manuel Román , Fibercli
![Page 7: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/7.jpg)
Symptom
Multiple incidents that are not managed.
Feeling of lack of control over the network.
7José Manuel Román para FiberCli
![Page 8: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/8.jpg)
Solution
Centralized system to collect, normalize,
visualization and analysis
8José Manuel Román para FiberCli
![Page 9: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/9.jpg)
AGENDA Intro Architecture ELK (ElasticSearch, Logstash, Kibana) Mkt + AAA+ with Freeradius and DB centralized Mkt + Centralized Log and ELK Mkt + Monitoring and ELK Mkt + Netflow and ELK Q and A
9José Manuel Román para FiberCli
![Page 10: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/10.jpg)
¿What is?ELK
10José Manuel Román para FiberCli
![Page 11: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/11.jpg)
Elast icsearchElasticsearch is a search engine based on Lucene. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.
11José Manuel Román para FiberCli
![Page 12: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/12.jpg)
Elast icsearchElasticsearch is developed in Java and is released as open source under the terms of the Apache License
12José Manuel Román para FiberCli
![Page 13: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/13.jpg)
Elast icsearch
• Distributed, scalable, and highly available
• Real-time search and analytics capabilities
• Sophisticated RESTful API
•https://www.elastic.co/products/elasticsearch
13José Manuel Román para FiberCli
![Page 14: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/14.jpg)
Elast icsearch
14José Manuel Román para FiberCli
• Schema-free, REST & JSON based
distributed search engine
• Open Source: Apache License 2.0
• Easy to understand, yet very powerful query
languageFull text search (phrase, fuzzy)
Numeric search (support ranges, dates, ipv4
addresses)
Highlighting
Aggregations
Suggestions
![Page 15: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/15.jpg)
LogstashLogstash is a tool to collect, process, and forward events and log messages. Collection is accomplished via configurable input plugins including raw socket/packet communication, file tailing, and several message bus clients.
15José Manuel Román para FiberCli
![Page 16: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/16.jpg)
Logstash
• Centralize data processing of all types
• Normalize varying schema and formats
• Quickly extend to custom log formats
• Easily add plugins for custom data sources
•https://wikitech.wikimedia.org/wiki/Logstash
16José Manuel Román para FiberCli
![Page 17: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/17.jpg)
Logstash
• Inputs: collect data from variety of sources
• Filters: parse, process and enrich data
• Outputs: push data to a variety of destinations
17José Manuel Román para FiberCli
![Page 18: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/18.jpg)
Kibana
• Execute queries on your data & visualize results
• Add/remove widgets
• Share/Save/Load dashboards
• Open Source: Apache License 2.0
18José Manuel Román para FiberCli
![Page 19: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/19.jpg)
Big Picture
•
19José Manuel Román para FiberCli
KIBANAELASTICSEARCHLOGSTASHRSYSLOG
RADIUS
DATABASE BACKEND
![Page 20: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/20.jpg)
20José Manuel Román para FiberCli
![Page 21: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/21.jpg)
AAA System
21José Manuel Román para FiberCli
![Page 22: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/22.jpg)
Radius
RADIUS is an application level protocol that carries Authentication, Authorization and Accounting (AAA)
configuration information between a Network Access Server (NAS) and a Shared Authentication Server. Radius defined in RFC
2865
22José Manuel Román para FiberCli
![Page 23: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/23.jpg)
Radius
In MikroTik RouterOS itself support RADIUS for Hotspot, PPP, DHCP, Wireless and Login. RADIUS using transport protocol
UDP.UDP Port 1812 – AuthenticationUDP Port 1813 – Accounting
23José Manuel Román para FiberCli
![Page 24: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/24.jpg)
Radius
RADIUS operation typically split into three type : Dial-In User : User who requesting for login and password
Network Access Server (NAS) / RADIUS Client : Device who accept the request from dial-in user
and
: Forward into RADIUS Server.
Shared Authentication Server / RADIUS Server : Device who make a decision for request (Accept,
: Reject or Challenge)
24José Manuel Román para FiberCli
![Page 25: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/25.jpg)
25José Manuel Román para FiberCli
Link-Establishment Link-Establishment
Dial-In User try to connect (username & password) RADIUS Access-Request
RADIUS Access-Challange
RADIUS Access-Request
RADIUS Access-Reject
RADIUS Access-Accept
Accounting-Request (Start)
Accounting-Response
OROR
Disconnect
Session Start
Accounting-Request (Stop)
Accounting-Response
Disconnect
![Page 26: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/26.jpg)
Topology
26José Manuel Román para FiberCli
![Page 27: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/27.jpg)
27José Manuel Román para FiberCli
MIKROTIK SITE 1
RADIUSDATABASE BACKEND
MIKROTIK SITE 3
MIKROTIK SITE 2
![Page 28: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/28.jpg)
28José Manuel Román, FiberCli
![Page 29: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/29.jpg)
29José Manuel Román para FiberCli
service (ppp|login|hotspot|wireless|dhcp; Default: )
• hotspot - HotSpot authentication service
• login - router's local user authentication
• ppp - Point-to-Point clients authentication
• wireless - wireless client
• dhcp - DHCP protocol client authentication
Address (IPv4/IPv6 address; Default: 0.0.0.0) IPv4 or IPv6
address of RADIUS server.
Secret (string; Default: ) Shared secret used to access
the RADIUS server.
![Page 30: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/30.jpg)
Freeradius produces several logsthat we can process with logstash
http://code.metager.de/source/xref/freeradius/server/doc/schemas/logstash/
José Manuel Román para FiberCli 30
![Page 31: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/31.jpg)
# Tue Mar 10 15:32:24 201512# Packet-Type = Access-Request13# User-Name = "[email protected]"14# Calling-Station-Id = "01-02-03-04-05-06"15# Called-Station-Id = "aa-bb-cc-dd-ee-ff:myssid"16# NAS-Port = 1017# NAS-IP-Address = 10.9.0.418# NAS-Identifier = "Wireless-Controller-1"19# Service-Type = Framed-User20# NAS-Port-Type = Wireless-802.1121#
http://code.metager.de/source/xref/freeradius/server/doc/schemas/logstash/
José Manuel Román para FiberCli 31
![Page 32: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/32.jpg)
26input {27 stdin {28 type => radiusdetail29 }30}
http://code.metager.de/source/xref/freeradius/server/doc/schemas/logstash/
José Manuel Román para FiberCli 32
![Page 33: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/33.jpg)
filter {3435 if [type] == "radiusdetail" {3637 # join all lines of a record together38 multiline {39 pattern => "^[^\t]"40 negate => true41 what => "previous"42 }4344 # pull off the timestamp45 grok {46 match => [ "message", "^(?<timestamp>[^\n\t]+)[\n\t]" ]47 }4849 # create the timestamp field50 date {51 match => [ "timestamp", "EEE MMM dd HH:mm:ss yyyy",52 "EEE MMM d HH:mm:ss yyyy" ]53 }5455 # split the attributes and values into fields56 kv {57 field_split => "\n"58 source => "message"59 trim => "\" "60 trimkey => "\t "61 }62 }63}
http://code.metager.de/source/xref/freeradius/server/doc/schemas/logstash/
José Manuel Román para FiberCli 33
![Page 34: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/34.jpg)
65output {66 if [type] == "radiusdetail" {67 elasticsearch {68 host => localhost69 protocol => http70 cluster => elasticsearch71 index_type => "detail"72 index => "radius-%{+YYYY.MM.dd}"73 flush_size => 100074 }75 }76}
http://code.metager.de/source/xref/freeradius/server/doc/schemas/logstash/
José Manuel Román para FiberCli 34
![Page 35: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/35.jpg)
Centralize Log
35José Manuel Román para FiberCli
![Page 36: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/36.jpg)
RSYSLOGRSYSLOG stand for "the rocket-fast system for log processing“ is an open-source software utility used on UNIX and Unix-like computer
systems for forwarding log messages in an IP network. It implements the basic syslog protocol, extends it with content-based filtering, rich filtering capabilities, flexible configuration
options and adds features such as using TCP for transport
36José Manuel Román para FiberCli
http://www.rsyslog.com/rsyslog-8-19-0-v8-stable-released/
![Page 37: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/37.jpg)
RSYSLOG• Protocol supported by rsyslog are:
• ISO 8601 timestamp with millisecond granularity and timezoneinformation
• The addition of the name of relays in the host fields to make it possible to track the path a given message has traversed
• Reliable transport using TCP
• Support GSS-API and TLS
37José Manuel Román para FiberCli
![Page 38: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/38.jpg)
RSYSLOG
•Logging directly into various database engines.
• Support for RFC 5424, RFC 5425, RFC 5426
• Complete input/output support for systemd journal
38José Manuel Román para FiberCli
![Page 39: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/39.jpg)
Topology
39José Manuel Román para FiberCli
![Page 40: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/40.jpg)
40José Manuel Román para FiberCli
![Page 41: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/41.jpg)
41José Manuel Román para FiberCli
KIBANAELASTICSEARCHLOGSTASHRSYSLOG
![Page 42: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/42.jpg)
MikroTik log Configuration
42José Manuel Román para FiberCli
![Page 43: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/43.jpg)
43José Manuel Román para FiberCli
![Page 44: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/44.jpg)
44José Manuel Román para FiberCli
![Page 45: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/45.jpg)
45José Manuel Román para FiberCli
![Page 46: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/46.jpg)
46José Manuel Román para FiberCli
Monitoring
![Page 47: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/47.jpg)
47José Manuel Román para FiberCli
/snmp set enabled=yes contact="[email protected]" location="Mum Madrid" trap-community=public trap-version=2
![Page 48: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/48.jpg)
48José Manuel Román para FiberCli
![Page 49: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/49.jpg)
49José Manuel Román para FiberCli
There are clients to export data to databases like fluentdb.
https://github.com/jojohappy/zabbix-relay
![Page 50: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/50.jpg)
50José Manuel Román para FiberCli
We can integrate Zabbix events as input in Logstach, with thegoal to have a decoupled monitorization.
![Page 51: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/51.jpg)
Netflow
51José Manuel Román para FiberCli
![Page 52: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/52.jpg)
NetFlow is a network protocol created by Cisco Systems tocollect information about ip traffic.
52José Manuel Román para FiberCli
![Page 53: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/53.jpg)
53José Manuel Román para FiberCli
![Page 54: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/54.jpg)
/ip traffic-flow set active-flow-timeout=30m cache-entries=1M
\enabled=yes inactive-flow-timeout=15s interfaces=all
54José Manuel Román para FiberCli
![Page 55: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/55.jpg)
/ip traffic-flow target add dst-address=ip.server port=5055 disabled=no \v9-template-refresh=20 v9-template-timeout=30m version=9
55José Manuel Román para FiberCli
![Page 56: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/56.jpg)
To collect the output we need a netflow collector, for example pmacctsonda.
56José Manuel Román para FiberCli
![Page 57: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/57.jpg)
When we collect the logs with pcmacct we’ll send the output in jsonformat to ElasticSearch.
https://github.com/pierky/pmacct-to-elasticsearch/blob/master/CONFIGURATION.md
57José Manuel Román para FiberCli
![Page 58: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/58.jpg)
https://github.com/pierky/pmacct-to-elasticsearch/blob/master/CONFIGURATION.md
58José Manuel Román para FiberCli
![Page 59: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/59.jpg)
59José Manuel Román para FiberCli
![Page 60: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/60.jpg)
60José Manuel Román para FiberCli
![Page 61: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/61.jpg)
Additional resources
61José Manuel Román para FiberCli
![Page 62: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/62.jpg)
Grafana
62José Manuel Román para FiberCli
![Page 63: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/63.jpg)
Influxdb
63José Manuel Román para FiberCli
![Page 64: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/64.jpg)
Fluentd
64José Manuel Román para FiberCli
![Page 65: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/65.jpg)
65José Manuel Román para FiberCli
![Page 66: System integration and analysis](https://reader030.vdocuments.net/reader030/viewer/2022013120/58a046001a28ab584a8c7fa8/html5/thumbnails/66.jpg)
Thank you
66José Manuel Román para FiberCli