logging for openstack - elasticsearch, fluentd, logstash, kibana

Download Logging for OpenStack - Elasticsearch, Fluentd, Logstash, Kibana

Post on 21-Jan-2018

1.051 views

Category:

Technology

1 download

Embed Size (px)

TRANSCRIPT

  1. 1. OPENSTACK & LOGGING
  2. 2. ABOUT ME Md Safiyat Reza Fresh out of college! Open-source enthusiast An EMACS and KDE user. Software Engineer at Snapdeal.com safiyat @reza_safiyat reza.safiyat@acm.org
  3. 3. Logs are boring!
  4. 4. nova-compute nova-network nova-manage nova-conductor nova-scheduler nova-api nova-cert nova-console nova-consoleauth nova-dhcpbridge apache logs cinder-api cinder-scheduler cinder-volume syslog keystone glance-api glance-registry dhcp-agent l3-agent metadata-agent openvswitch-agent server.log openvswitch-server Lots of logs!
  5. 5. What logs contain... TIMESTAMP PID NAME REQUESTID USERID TENANTID [INSTANCE INSTANCEID] MESSAGE
  6. 6. All that is fine, How to use them?
  7. 7. When shit happens... DEBUG INFO AUDIT WARNING ERROR CRITICAL TRACE
  8. 8. Use logs for a good cause.
  9. 9. Log Collection, Aggregation & Visualization
  10. 10. Log Collectors and Aggregators
  11. 11. The flow of logs
  12. 12. Open source Centralized logging Collection & Transport High availability Written in Ruby Requires Ruby (alt. td-agent) XML-styled configuration Plugins available Owned by Treasure Data Written in JRuby Requires JVM JSON-styled configuration A lot and lot of plugins Owned by Elastic
  13. 13. path/var/log/syslog typesyslog formatgrok grok_pattern%{SYSLOGLINE} tagsystem typerecord_transformer enable_ruby timestamp${timestamp.gsub!("","T"); timestamp.gsub(/.d*/,"+05:30"} typeforest subtypeelasticsearch hostelasticsearchhost index_namesomeindexname type_namesometypename Fluentd: syslog to elasticsearch
  14. 14. input{ file{ path=>"/var/log/syslog" start_position=>beginning type=>syslog sincedb_path=>"/dev/null" } } filter{ Grok{ match=>["message","%{SYSLOGLINE:log}"] } } output{ file{ path=>"/home/safiyat/kafkaop" } elasticsearch{ protocol=>"http" } } Logstash: syslog to elasticsearch
  15. 15. Kibana
  16. 16. Kibana, again.
  17. 17. Thank you for bearing this!

Recommended

View more >