About me..
Devops engineer at RetailMeNot Obsessed with making things betterHusband, father, frequently busy
Why Logstash?
Jordan Sissel
Why Logstash?
Jordan SisselNoticed a problem
Why Logstash?
Jordan SisselNoticed a problem
Why Logstash?
Jordan SisselNoticed a problem
Why Logstash?
Jordan SisselNoticed a problem
Why Logstash?
Jordan SisselNoticed a problem
What does Logstash provide?
Empowerment
What does Logstash provide?
EmpowermentAggregation
What does Logstash provide?
EmpowermentAggregationSearch
What does Logstash provide?
EmpowermentAggregationSearch
What is Logstash (ELK stack)?
Centralization
What is Logstash (ELK stack)?
CentralizationVisualization
What is Logstash (ELK stack)?
CentralizationVisualizationSegregation
Components
httpd Logstashshipper
redis Logstashindexer
Elasticsearch
Kibana
MySQL Logstashshipper
Logstash-forwarder shipper
Small package (java) (used to be “lumberjack”)
Logstash-forwarder shipper
Small package (java) (used to be “lumberjack”)Grok processing can happen on client
Logstash-forwarder shipper
Small package (java) (used to be “lumberjack”)Grok processing can happen on clientOne shipper, multiple logfiles (perms)
Logstash-forwarder shipper
Small package (java) (used to be “lumberjack”)Grok processing can happen on clientOne shipper, multiple logfiles (perms)Many log formats, one output - JSON
Redis
Message broker
Redis
Message brokerSSL encryption for non-VPN networks
Redis
Message brokerSSL encryption for non-VPN networksScalable
Logstash Indexer
Java app
Logstash Indexer
Java appGrok processing of logs off app-servers
Logstash Indexer
Java appGrok processing of logs off app-serversOutputs to multiple endpoints, if required
Logstash Indexer
Java appGrok processing of logs off app-serversOutputs to multiple endpoints, if requiredScalable
Elasticsearch
Mature, scalable db
Elasticsearch
Mature, scalable dbDocument-based index, Query DSL API
Elasticsearch
Mature, scalable dbDocument-based index, Query DSL APISharded, clustered
Elasticsearch
Mature, scalable dbDocument-based index, Query DSL APISharded, clusteredIndices are stored in date format
Kibana
Standalone (used to be simple website)
Kibana
Standalone (used to be simple website)Points directly at the Elasticsearch db
Kibana
Standalone (used to be simple website)Points directly at the Elasticsearch dbSolr Lucene syntax
Kibana
Standalone (used to be simple website)Points directly at the Elasticsearch dbSolr Lucene syntaxScalable
Kibana
Standalone (used to be simple website)Points directly at the Elasticsearch dbSolr Lucene syntaxScalable
Demo
Tricks
Elasticsearch Curator, single indices
Tricks
Elasticsearch Curator, single indicesOutputs to Graphite, Nagios, Cloudwatch,
Datadog, GELF, Slack, Hipchat, PagerDuty etc.
Tricks
Elasticsearch Curator, single indicesOutputs to Graphite, Nagios, Cloudwatch,
Datadog, GELF, Slack, Hipchat, PagerDuty etc.Kibana dashboards
Gotchas
Logstash Elasticsearch client (versions, AWS)
Gotchas
Logstash Elasticsearch client (versions, AWS)Enterprise ready? (utter lack of users/security,
streams, alerting)
Gotchas
Logstash Elasticsearch client (versions, AWS)Enterprise ready? (utter lack of users/security,
streams, alerting)Graylog – Alternative log engine
Links
Jordan Sissels Github - https://github.com/jordansisselJordan Sissel pres - http://semicomplete.com/presentations/logstash-scale11x/#/33 Logstash book - http://www.logstashbook.com/ Enterprise ready? (users, streams, alerting)
Cheers!
