poznań jug: listening to the sounds of your application

43
Listen to the sounds of your application

Upload: maciej-bilas

Post on 06-May-2015

815 views

Category:

Technology


0 download

TRANSCRIPT

Page 1: Poznań JUG: Listening to the sounds of your application

Listen to the sounds of your application

Page 2: Poznań JUG: Listening to the sounds of your application

!2

Maciej Biłas @maciejb softwaremill.com jbison.com

Page 3: Poznań JUG: Listening to the sounds of your application

Originally authored by: Maciej Biłas

Krzysztof Ciesielski

Presented atJ-Day

Lbn.sc

Page 4: Poznań JUG: Listening to the sounds of your application

Agenda

• Monitoring, huh?

• Introducing Graphite

• Log analysis – the whys

• Logstash architecture & use cases

• Exploring logs with Kibana

Page 5: Poznań JUG: Listening to the sounds of your application

Monitoring, huh?

Page 6: Poznań JUG: Listening to the sounds of your application

source: codeascraft.com

Page 7: Poznań JUG: Listening to the sounds of your application

Types of measurements

Network

Machine

Application

Page 8: Poznań JUG: Listening to the sounds of your application

source: codeascraft.com

Page 9: Poznań JUG: Listening to the sounds of your application

measurement > prediction

Page 10: Poznań JUG: Listening to the sounds of your application
Page 11: Poznań JUG: Listening to the sounds of your application

measurement >> prediction

Page 12: Poznań JUG: Listening to the sounds of your application

Our stack

Server

App

Server

App

Graphite Logstash

Yammer Metrics

Page 13: Poznań JUG: Listening to the sounds of your application
Page 14: Poznań JUG: Listening to the sounds of your application

http://graphite/render?target=server.web1.load&height=800&width=600

Page 15: Poznań JUG: Listening to the sounds of your application

Graphite

Page 16: Poznań JUG: Listening to the sounds of your application

echo "local.random.diceroll 4 `date +%s`" |\ nc graphite-server.your.org 2003

Page 17: Poznań JUG: Listening to the sounds of your application

successful.login.attempt 1 1384471287 successful.login.attempt 1 1384471297

successful.login.attempt = 1

Page 18: Poznań JUG: Listening to the sounds of your application

Yammer metrics

private final Meter successfulLogins = metrics.meter(name(LoginHandler.class, "successful")); !public void login(String user, String password) { if (canLogin(user,password)) { successfulLogins.mark(); // ... } else { // ... } }

Page 19: Poznań JUG: Listening to the sounds of your application

MetricsTypes:

• Gauges

• Counters

• Meters

• Histograms

• Timers

• Health Checks

Reporters:

• STDOUT

• CSV

• SLF4J

• JMX

• Graphite

• Ganglia

Page 20: Poznań JUG: Listening to the sounds of your application

Counters, meters… vs.

Gauges

Page 21: Poznań JUG: Listening to the sounds of your application

Aggregationapp00.webservice.requestTime app01.webservice.requestTime app02.webservice.requestTime

Page 22: Poznań JUG: Listening to the sounds of your application

source: codeascraft.com

Page 23: Poznań JUG: Listening to the sounds of your application

You can also check out• collectd https://collectd.org

• StatsD https://github.com/etsy/statsd/

• Riemman http://riemann.io/

• Twitter’s Ostrich https://github.com/twitter/ostrich

• Ganglia http://ganglia.sourceforge.net/

• Dashboards: http://shopify.github.io/dashing/https://github.com/obfuscurity/descarteshttps://github.com/obfuscurity/dusk

Page 24: Poznań JUG: Listening to the sounds of your application

Log aggregation?

Page 25: Poznań JUG: Listening to the sounds of your application

Event Sourcing

Tracing and storing all the events

Current app state: replaying the stream

Great for data mining and analysis

Page 26: Poznań JUG: Listening to the sounds of your application

Log aggregation

• Less invasive way to build an “event stream”

• Logs are data with plenty of value

Page 27: Poznań JUG: Listening to the sounds of your application

Log aggregation

Log source

Log source

Log source

Central storage

Query engine

Web view

Page 28: Poznań JUG: Listening to the sounds of your application

Logstash

• Open source

• Written in JRuby

• Gathers logs from various inputs

• Parses and extracts metadata

• Writes to various outputs

Page 29: Poznań JUG: Listening to the sounds of your application

Architecture

Page 30: Poznań JUG: Listening to the sounds of your application

Inputscollectd

drupal_dblog

elasticsearch

eventlog

exec

file

ganglia

gelf

gemfire

generator

graphite

heroku

imap

irc

log4j

lumberjack

pipe

rabbitmq

redis

relp

s3

snmptrap

sqlite

sqs

stdin

stomp

syslog

tcp

twitter

udp

unix

varnishlog

websocket

wmi

xmpp

zenoss

zeromq

Page 31: Poznań JUG: Listening to the sounds of your application

Grok

time 2013-11-10 19:41:25.321

thread main

loglevel INFO

source o.a.camel.impl.DefaultCamelContext

message Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds

2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds

Thin layer on top of a regular expression

Page 32: Poznań JUG: Listening to the sounds of your application

Grok

grok { patterns_dir => “./some-dir“ match => ["message", "%{TIMESTAMP_ISO8601:time} \[%{DATA:thread}\] %{LOGLEVEL:loglevel} %{DATA:source} - % {MULTILINE_GREEDYDATA:message}"] }

2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds

Page 33: Poznań JUG: Listening to the sounds of your application

Outputsboundary

circonus

cloudwatch

csv

datadog

datadog_metrics

elasticsearch

elasticsearch_http

elasticsearch_river

email

exec

file

ganglia

gelf

gemfire

google_bigquery

google_cloud_storage

graphite

graphtastic

hipchat

http

irc

jira

juggernaut

librato

loggly

lumberjack

metriccatcher

mongodb

nagios

nagios_nsca

null

opentsdb

pagerduty

pipe

rabbitmq

redis

riak

riemann

s3

sns

solr_http

sqs

statsd

stdout

stomp

syslog

tcp

udp

websocket

xmpp

zabbix

zeromq

Page 34: Poznań JUG: Listening to the sounds of your application

Kibana

Page 35: Poznań JUG: Listening to the sounds of your application

Checking app activity

Page 36: Poznań JUG: Listening to the sounds of your application

Spotting anomalies

Page 37: Poznań JUG: Listening to the sounds of your application

Regular events

Page 38: Poznań JUG: Listening to the sounds of your application

How often is a feature used?

Page 39: Poznań JUG: Listening to the sounds of your application

Plotting multiple event types

Page 40: Poznań JUG: Listening to the sounds of your application

Other tools

• Splunk

• Graylog2

• Fluentd

• log.io

Page 41: Poznań JUG: Listening to the sounds of your application

References• Graphite

http://graphite.readthedocs.org/en/latest/

• Yammer Metricshttp://metrics.codahale.com/

• Logstashhttp://logstash.net/

• Surfing the event stream by Sam Newman at Geeconhttp://www.slideshare.net/spnewman/surfing-the-event-stream

• Lessons from Building and Scaling LinkedIn by Jay Krepshttp://www.infoq.com/presentations/linkedin-architecture-stack

• Code as Craft http://codeascraft.com/

Page 42: Poznań JUG: Listening to the sounds of your application
Page 43: Poznań JUG: Listening to the sounds of your application

Thank you!Questions?