log analysis using logstash,elasticsearch and kibana

Download Log analysis using Logstash,ElasticSearch and Kibana

If you can't read please download the document

Post on 27-Aug-2014




6 download


Embed Size (px)


Log analysis using Logstash, ElasticSearch and Kibana


  • Log Analysis Logstash, Elastic Search, Kibana Avinash Ramineni Shantanu Mirajkar
  • Logging Pains of Log Management Introducing Logstash Elasticsearch Kibana Demo Installing Logstash, Elasticsearch Kibana Questions Agenda
  • Why do we need Logging ? Troubleshoot Issues Security Analyze logs to detect patterns Detect Malware Activity - Intrusion Detection, Denial of Service Unauthorized Resource Usage Monitoring Monitor Resource Usage Developers and Logging Logging Aids in Development ? Forget about Production !!!!! Logging
  • Capture-it-all Approach What to Log? Everything DevOps Movement Logs are archived for years Big Data Application Usage Statistics Logging
  • Searching the logs Command line, cat, tail, sed, grep, awk Regular Expressions Multiple Servers behind the load balancer Multi-Tier Architecture Web Application Service Layer Correlation between various components in a System Geographically distributed Timestamps Log management
  • Centralize all the Logs Too much information to go through Increasingly hard to correlate the contextual Data Add Searching and Indexing Technology grep Custom logging frameworks , custom integration of logging, searching technologies Monitor the Logs Log management
  • Logstash to the Rescue Integration Framework Log Collection Centralization Parsing Storage and Search Logstash
  • JRuby Run on Java Virtual Machine (JVM) Simple Message Based Architecture Single Agent that can be configured for multiple things OPEN SOURCE Four Components Shipper Broker and Indexer Search and Storage Web Interface Logstash
  • Architecture Image courtesy of Logstashbook
  • Architecture - Broker Acts as Temp Buffer between Logstash Agents and the Central server Enhance Performance by providing caching buffer for log events Adds Resiliency Incase the Indexing fails, the events are held in a queue instead of getting lost AMQP,0MQ, Redis
  • Indexing and Searching Tool Built on Lucene Search and Index data available Restfully as JSON over HTTP Comes bundled with Logstash embedded Text indexing Search Engine Searches on the Index rather than on the content Creates Indexes of the incoming content Uses Apache Lucene to create Indexes ElasticSearch can have a schema Fields on which Indexes are created ElasticSearch
  • Indexes are stored in Lucene Instances called Shards ElasticSearch can have multiple nodes Two Types of Shards Primary Replica Replicas of Primary Shards Protect the data Make Searches Faster ElasticSearch
  • Wouldnt it be good to have a webpage to do search on ElasticSearch instead of searching it through a Service Kibana provides a Simple but Powerful web Interface Customizable Dashboards Search the log events Support Lucene Query Syntax Creation of tables, graphs and sophisticated visualizations Kibana
  • Kibana
  • Kibana
  • Demo
  • Send Alerts Emails Instant Messaging Other Monitoring System Collect and Deliver Metrics to metric engine Alerts / Monitoring Support
  • Small VMs with limited memory Outsourced managed servers Java not installed Alternatives Syslog Rsyslog Syslogd Syslog-NG Logstash Forwarder (Lumber Jack) Shipping Logs with Logstash Agent
  • Scale each component as needed Can be built into using chef and puppet scripts Scaling / Deployment
  • Industry ExperienceQuestions ? avinash@clairvoyantsoft.com Twitter:@avinashramineni shantanu@clairvoyantsoft.com


View more >