keep an eye on your app with logging and graphs - phpbenelux 2014

So, logging!

!1

http://www.egeniq.com info@egeniq.com

@egeniq

PHPBenelux 2014Felix De Vliegher

Keep an eye on your app with logging and graphs

• Back-end developer

• Sysadmin

• <3 APIs

• Travelling

• @felixdv

About me

!3

About me

!4

About me

!5

About me

!6

Logging shows what exactly happens in your app. !

Metrics show the impact

Terminology

!7

Logging shows what exactly happens in your app. !

Metrics show the impact

Terminology

!8

Both are important (but for different reasons)

!9

What is this talk about?

Knowing what is going on in your application!

!10

Questionnaire

• Who uses logging in their app?

• Aggregation, filtering, manipulation?

• Who collects metrics?

• Who visualises this data in a dashboard?

• Metric correlations?

!11

Logging

!12

Ye olden days

• Logs per server

• Requires SSH for inspection

• Decentralised (for once not an advantage)

• Hard to search

!13

Goals

Make logs searchable

!14

Goals

!15

Goals

Collect and manipulate logs

!16

Goals

!17

Goals

Extract metrics

!18

Enter Mr. logstash (hipster mo’ included)

!19

Logstash

• Inputs

• Filters

• Outputs

!20

Inputs

• File

• Redis

• syslog

• varnishlog

• rabbitmq

• …

!21

Filters

Modify the input data !

• grok • mutate • dns • json • geoip • …

!22

Grok

!23

Grok

!24

Mutate

!25

Json

!26

Json

!27

Outputs

• email • graphite • irc • mongodb • pagerduty • sns • elasticsearch • …

!28

Setup

!29

Example shipper config

!30

Example indexer config

!31

Example indexer config (contd.)

!32

Logstash

!33

Elasticsearch

!34

Elasticsearch

• Distributed search and analytics

• Based on Apache lucene (but better)

• Document-oriented DB

• Easy scaling

• Restful API

• Great for storing logs!

!35

Elasticsearch

!36

!

Getting data in and out:

!

• Logstash defaults to using daily indexes

• Easy retention = delete indexes after x days

• Can be configured in other ways

Elasticsearch

!37

Kibana

!38

Kibana

• Search your logs

• The v3 rewrite is pretty awesome :-)

• HTML + JS only

• Elasticsearch project

• Almost no-config setup

!39

Kibana

!40

Kibana

!41

Kibana

!42

Kibana

!43

Kibana

!44

StatsD

• Metrics aggregation framework

• Made by Etsy

• Node.js

• UDP

• Dead easy to use

!46

Metrics

“Track everything that moves”

!47

Metrics

!48

Running StatsD

!49

StatsD config

!50

StatsD config

!51

Console backend

!52

StatsD Metric types

• Counters

• Timers

• Gauges

!53

Counters

!54

Timers

!55

Gauges

Arbitrary values:

!56

Sampling data

!57

Example: ZF Controller plugin

!58

StatsD naming

!59

But…

!60

StatsD is for collection and aggregation only

We want pretty graphs

!61

Coffee is important too

!62

StatsD config

!63

Graphite

• Store and visualise time-series data

• Carbon, whisper and graphite UI

• Rendering graphs

• Powerful function library

!64

Graphite UI

!65

StatsD => Graphite

!66

production.api.httpstatus.200

StatsD => Graphite

!67

The render API

!68

Json output

!69

Add &format=json to the render API

Functions

• cumulative(‘server*.api.logins’) • asPercent(‘connections.failed’, ‘connections.attempted’) • drawAsInfinite(‘app.deploys’) • summarize(‘app.errors’, ‘1hour’) • stacked() • movingAverage()

!• http://graphite.readthedocs.org/en/latest/functions.html

!70

StatsD

!71

Functions

• cumulative(‘server*.api.logins’) • asPercent(‘connections.failed’, ‘connections.attempted’) • drawAsInfinite(‘app.deploys’) • summarize(‘app.errors’, ‘1hour’) • stacked() • movingAverage()

!• http://graphite.readthedocs.org/en/latest/functions.html

!72

Alternative dashboards

!73

Dashing

!74

Tasseo

!75

Descartes

!76

Conclusion

• No excuse not to log

• Manage your logs with logstash and kibana

• Track everything using statsd

• Visualise with graphite dashboards

• Get insight in your application!

!77

Thank you! Questions?

http://www.egeniq.com info@egeniq.com

@egeniq

Feedback: https://joind.in/10276 Contact: @felixdv