application logging with logstash
TRANSCRIPT
Application Logging With Logstash
Ben Waine
• Worked With PHP For 5 Years
• Software Engineer -Sainsbury’s
• Dabbles in devops
System Logs
Application Log
Debug Information - Errors (connections, uncaught exceptions, resource exhaustion)
Narrative Information - Methods Calls, Event Triggers
Business Events - Purchases, Logins, Registrations, Unsubscribes
Keeping Track Of All This....ssh [email protected] -f /var/log/nginx/my-site.access.logtail -f /var/log/my.application.log
ssh [email protected] -f /var/log/mysql/mysql.log
ssh [email protected] -f /var/log/rabbitmq/nodename.log
The Elk Stack
Visualizing Log Data
PHP Logging Tools
1) Monolog2) Everything else....
Basic Logging Examples
1) Monolog: Loggers And Handlers2) Monolog: Tags & Formatters3) Logging business events
use Monolog\Logger;use Monolog\Handler\FingersCrossedHandler;use Monolog\Handler\StreamHandler;
$logEnv = getenv('LOG_LEVEL');$level = empty($logLevel) ? $logEnv : Logger::WARNING;
$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG); $fcHandler = new FingersCrossedHandler($strHandler, $level);
$appLog−>pushHandler($fcHandler);$appLog−>debug('LOGGING!');
EG1: Loggers And Handlers
// Set A Log Level$logEnv = getenv('LOG_LEVEL');$level = empty($logLevel) ? $logEnv : Logger::WARNING;
// Create A Logger$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG);
$fcHandler= new FingersCrossedHandler($strHandler, $level);
// Create Handlers
$appLog−>pushHandler($fcHandler);
$appLog−>debug('Start Logging!');$appLog−>emergency('Something Terrible Happened');
// Push The Handler And Start Logging
EG 2: Tagging Formatting
$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/lg.lg', $level);$formatter = new LogstashFormatter("helloapp", "application");
$strHandler−>setFormatter($formatter); $appLog−>pushHandler($strHandler));
$id = $_SERVER('X_VARNISH');$tag = new TagProcessor(['request−id' => $id])
$appLog−>pushProcessor($tag); $appLog−>debug("LOGGING!");
// Create A Logger$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/lg.lg', $level);$formatter = new LogstashFormatter("helloapp", "app");
// Create A Handler & Formatter
// Set Formatter Onto Handler$strHandler−>setFormatter($formatter);
$appLog−>pushHandler($strHandler));
//Push Handler Onto Logger
$id = $_SERVER('X_VARNISH');$tag = new TagProcessor(['request−id' => $id])$appLog−>pushProcessor($tag); $appLog−>debug("LOGGING!");
// Capture A Unique Id, Create A Tag Processor, Push
Log Levels2009 - RFC 5424 - Syslog Protocol
Code / Severity
0 Emergency: system is unusable1 Alert: action must be taken immediately2 Critical: critical conditions3 Error: error conditions4 Warning: warning conditions5 Notice: normal but significant condition6 Informational: informational messages7 Debug: debug-level messages
https://tools.ietf.org/html/rfc5424
Log Levels2013 - PSR03 - PHP Logging Interface Standard
Phrase / Severity
emergency Emergency: system is unusablealert Alert: action must be taken immediatelycritical Critical: critical conditionserror Error: error conditionswarning Warning: warning conditionsnotice Notice: normal but significant conditioninfo Informational: informational messagesdebug Debug: debug-level messages
http://www.php-fig.org/psr/psr-3/
http://imgs.xkcd.com/comics/standards.png
EG 3: Event Logginguse Monolog\Logger;use Symfony\Component\EventDispatcher\EventDispatcher;
$dispatcher = new EventDispatcher();
$dispatcher−>addListener( "business.registration.post", function () use ($busLog) { $busLog−>info("Customer registered"); });
$dispatcher−>dispatch("business.registration.post");
Logstash Architecture
1. Logstash Shipper ships logs to logstash
2. Logstash processes them
3. Logstash Inserts Into Elastic Search
4. Kibana exposes a web interface to Elastic Search data
Logstash Architecture
https://joind.in/talk/view/13369
Why not rate the talk now BEFORE the demo?
Logstash Demo
https://github.com/LoveSoftware/application-logging-with-logstash
Logstash Config
Logstash Collecting{ "network": { "servers": [ "logs.logstashdemo.com:5000" ], "timeout": 15, "ssl ca": "/etc/pki/tls/certs/logstash−forwarder.crt" }, "files": [ { "paths": [ "/var/log/nginx/helloapp.access.log" ], "fields": { "type": "nginx−access" } } ] }
Logstash Processing
input { lumberjack { port => 5000 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash−forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash−forwarder.key"}
}
Input
Logstash ProcessingFilteringfilter { if [type] == "nginx−access" { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } date { match => [ "logdate", "dd/MMM/yyyy:HH:mm:ss Z" ] } } }
Logstash ProcessingOutput
output { elasticsearch { host => localhost }}
Groking grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
https://github.com/elasticsearch/logstash/blob/v1.4.2/patterns/grok-patterns
http://grokdebug.herokuapp.com/
55.3.244.1 GET /index.html 15824 0.043
%{IP:client}%{WORD:method}%{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}
Logging IdeasRelease MarkerError rates of various applications over timeLatency in various percentiles of each application tierHTTP Responses: 400 series responsesHTTP Responses: 500 series responsesAuto git blame production errorsAuth and Syslogs
Go Forth And Log....BUT
Remember log rotation
Beware running out of space
Beware file logging on NFS
Questions?
