Logstash!Get to know your logs

Dan Ivovich

BMore on Rails4/9/13

Dan Ivovich

http://smartlogicsolutions.comTwitter - @danivovich

SmartLogic Solutions

What is the goal?

● Collect, Parse, and Store your log events

● Make log events searchable

● Analyze log events

Why bother?● Got logs?

○ syslog○ nginx access log○ application logs○ database logs

Are they all formatted the same?

3 Parts

● Inputs

● Filters

● Outputs

Inputs

● Files● TCP/UDP● Redis● AMQP● rsyslog● xmpp

http://logstash.net/docs/1.1.9/ - Full list

Filters

● grep● mutate● anonymize● date● grok

http://logstash.net/docs/1.1.9/ - Full list

Outputs

● Files● TCP/UDP● Redis● AMQP● elasticsearch

http://logstash.net/docs/1.1.9/ - Full list

Getting Startedinput { stdin { type => "stdin-type"} }

output { stdout { debug => true debug_format =>

"json"} }

java -jar logstash-1.1.9-monolithic.jar agent -f

logstash-simple.conf

Type something!

See our message!

Parse something!input { stdin { type => "stdin-type"} }

filter { grok { type => "stdin-type" pattern =>

"Hello %{DATA:message}!" } }

output { stdout { debug => true debug_format =>

"json"} }

java -jar logstash-1.1.9-monolithic.jar agent -f

logstash-simple.conf

Say Hello!

See our message in a field!

Life is better with searchinput { stdin { type => "stdin-type" } }

output {

stdout { debug => true debug_format => "json" }

elasticsearch { embedded => true }

}

java -jar logstash-1.1.9-monolithic.jar agent -f

logstash-search.conf

cURL for it!

Search for the data

Well that isn't pretty

Enter Kibana

Kibana is a friendly interface for your logs

Kibana Connects to Elasticsearch

How do we put it together?

● Logstash parses and structures data into Elasticsearch

● Kibana makes that data available● Apache Lucene Query Syntax (from elasticsearch)● Field statistics● Range searches

It Was Simple to Startinput { stdin { type => "stdin-type" } }

output {

stdout { debug => true debug_format => "json" }

elasticsearch { embedded => true }

}

java -jar logstash-1.1.9-monolithic.jar agent -f

logstash-search.conf

But Let's Get Real

On a server with logs

Logstash/Elasticsearch

Demo

Thoughts....

● Easy to try out, but for anything real, you'll want a much

more complicated configuration

● The variety of inputs is great

● Easy to build up a nice stack of filters

More Thoughts....

● Slow to boot monolithic jar file can be frustrating

○ Flatjar?

● Hard to track down why logs aren't flowing

● Elasticsearch node discovery can be difficult

○ If your cluster doesn't have a node added to it when

your client starts, your client isn't connected

More Information

● logstash.net

● grokdebug.herokuapp.com

● www.elasticsearch.org

Questions?

http://smartlogicsolutions.com

http://twitter.com/smartlogic

http://github.com/smartlogic http://fb.me/smartlogic