monitoring linux and windows logs with graylog collector · monitoring linux and windows logs with...
TRANSCRIPT
Bernd Ahlers – Graylog, Inc. [email protected]
Monitoring Linux and Windows Logs with Graylog Collector
Bernd AhlersGraylog, Inc.
Bernd Ahlers – Graylog, Inc. [email protected]
Structured Logging & Introduction to Graylog Collector
Bernd AhlersGraylog, Inc.
Bernd Ahlers – Graylog, Inc. [email protected]
Introduction: Graylog
● Open source log management platform● Collect, index and analyze structured and
unstructured log data● Alerts based on log data● Extensible via custom plugins
Bernd Ahlers – Graylog, Inc. [email protected]
Bernd Ahlers – Graylog, Inc. [email protected]
Bernd Ahlers – Graylog, Inc. [email protected]
Bernd Ahlers – Graylog, Inc. [email protected]
Bernd Ahlers – Graylog, Inc. [email protected]
Bernd Ahlers – Graylog, Inc. [email protected]
Bernd Ahlers – Graylog, Inc. [email protected]
Bernd Ahlers – Graylog, Inc. [email protected]
Bernd Ahlers – Graylog, Inc. [email protected]
More about Graylog
● www.graylog.org● marketplace.graylog.org● docs.graylog.org● github.com/Graylog2
Bernd Ahlers – Graylog, Inc. [email protected]
Why are we writing logs?
● Getting insight & collecting business metrics● Debugging problems● Building an audit trail● Monitoring
Bernd Ahlers – Graylog, Inc. [email protected]
How do we access our logs?
● Applications write to local files● SSH into machines● tail, grep, awk● If lucky: central log management
Bernd Ahlers – Graylog, Inc. [email protected]
What do they look like?
● Syslog RFC 3164 (BSD)● Syslog RFC 5424
Bernd Ahlers – Graylog, Inc. [email protected]
Syslog RFC 3164 (BSD)
Nov 10 15:55:01 tumbler CRON[2684]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Bernd Ahlers – Graylog, Inc. [email protected]
Syslog RFC 5424
2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...
Bernd Ahlers – Graylog, Inc. [email protected]
Apache
127.0.0.1 - bernd [28/Dec/2014:06:43:15 +0100] "PROPFIND /remote.php/webdav/ HTTP/1.1" 207 910 "-" "Mozilla/5.0 (Linux) mirall/1.7.1"
Bernd Ahlers – Graylog, Inc. [email protected]
Postfix
Aug 5 17:05:26 hostname postfix/qmgr[308]: A44F828C71: from=<[email protected]>, size=153136, nrcpt=1 (queue active)
Bernd Ahlers – Graylog, Inc. [email protected]
Squid
sq18.wikimedia.org 1715898 2010-12-01T21:57:22.331 0 1.2.3.4 TCP_MEM_HIT/20013208 GEThttp://en.wikipedia.org/wiki/Main_Page NONE/-text/html - - Mozilla/4.0%20(compatible;%20MSIE%206.0;%20Windows%20NT%205.1;%20.NET%20CLR%201.1.4322) en-US -
Bernd Ahlers – Graylog, Inc. [email protected]
log4j
0 [main] INFO MyApp - Entering application.36 [main] DEBUG com.foo.Bar - Did it again!51 [main] INFO MyApp - Exiting application.
Bernd Ahlers – Graylog, Inc. [email protected]
Ruby Logger
I, [2015-11-18T00:16:27.723972 #3609] INFO -- : Hello world!
Bernd Ahlers – Graylog, Inc. [email protected]
#1 Problem: Timestamps
● Everyone likes to invent one● Missing most of the time: timezone, year
Bernd Ahlers – Graylog, Inc. [email protected]
How to get value out of unstructured logs?
● Regex● More regex● Even more regex
Bernd Ahlers – Graylog, Inc. [email protected]
((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?
Bernd Ahlers – Graylog, Inc. [email protected]
GrokIPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9...
USERNAME [a-zA-Z0-9._-]+USER %{USERNAME}HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)EMAILLOCALPART [a-zA-Z][a-zA-Z0-9_.+-=:]+EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME}...COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
Bernd Ahlers – Graylog, Inc. [email protected]
Graylog: Extractors
● Regular expressions based● Extracts data into message fields
Bernd Ahlers – Graylog, Inc. [email protected]
Bernd Ahlers – Graylog, Inc. [email protected]
How to fix this?
● Central log collection (Graylog, ELK, others)● Use structured log formats
– Structured Syslog RFC 5424
– CEF Format
– GELF
– JSON
Bernd Ahlers – Graylog, Inc. [email protected]
Structured Syslog RFC 5424
2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...
Bernd Ahlers – Graylog, Inc. [email protected]
CEF by ArcSight/HP
Sep 19 08:26:10 host CEF:0|HP|siem|1.0|100|service
successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232
Bernd Ahlers – Graylog, Inc. [email protected]
GELF
{ "version": "1.1",
"timestamp": 1385053862.3072,
"host": "example.org",
"short_message": "A short message",
"full_message": "Backtrace here\n\nmore stuff",
"level": 1,
"_user_id": 9001,
"_some_info": "foo",
"_some_env_var": "bar"}
Bernd Ahlers – Graylog, Inc. [email protected]
JSON
{ "source": "example.org",
"message": "A log message",
"timestamp": "2015-11-15T10:43:21Z",
"user_id": 9001,
"http_method": "GET"}
Bernd Ahlers – Graylog, Inc. [email protected]
How we try to improve the ecosystem
● Icinga2 GELF output for events● Docker GELF logging driver (since Docker 1.8)● apache-mod_log_gelf (beta)● log4j2-gelf● gelfclient Java library● svloggelfd (log forwarding for runit)
Bernd Ahlers – Graylog, Inc. [email protected]
We at Graylog <3 structured data and you should too!
Bernd Ahlers – Graylog, Inc. [email protected]
Introduction: Graylog Collector
● Reads local log files and ships them to Graylog● Windows EventLog support (limited for now)● Transport encryption via TLS● Runs on Linux, Windows, Mac OS X and AIX
Bernd Ahlers – Graylog, Inc. [email protected]
Why another Collector?
● There are lots of others: nxlog, fluentd, heka, filebeat, rsyslog, syslog-ng
● We want integration and centralized management of collectors in Graylog
Bernd Ahlers – Graylog, Inc. [email protected]
Bernd Ahlers – Graylog, Inc. [email protected]
Collector Installation
● OS packages for Linux distributions● Manual installation on Windows via ZIP file
(MSI upcoming)
Runs as Windows service
Bernd Ahlers – Graylog, Inc. [email protected]
Collector Configuration
server-url = "http://your-graylog-server:12900"
inputs {
windows-application-log {
type = "windows-eventlog"
source-name = "Application"
}
}
outputs {
gelf-tcp {
type = "gelf"
host = "your-graylog-server"
port = 12201
}
}
Bernd Ahlers – Graylog, Inc. [email protected]
Collector: Current State
● Windows EventLog support needs update to support new Windows APIs
● File reading needs improvement● Centralized management needs to be
implemented● :-(
Bernd Ahlers – Graylog, Inc. [email protected]
Tomorrow: Hackathon
Bernd Ahlers – Graylog, Inc. [email protected]
QA
Ask me anything!
Bernd Ahlers / Graylog, [email protected]
@berndahlerswww.graylog.org
github.com/Graylog2