statsd workshop monitorama 2013

StatsD workshop @ Monitorama 2013

Daniel SchauenbergSoftware Engineer, Etsy Infrastructure

@mrtazz

Friday, March 29, 13

• What is this StatsD?

• Metrics types

• Backends

• Graphite and operating StatsD

• Send all the metrics

• Write all the backends


Receiver loop

Metricsprocessing

module

Backends


legacyNamespace: false


Counters

foo.bar:1|c

stats.counters.foo.bar.ratestats.counters.foo.bar.count


Gauges

foo.gar:30|gfoo.gar:-5|gfoo.gar:+5|g

stats.gauges.foo.garFriday, March 29, 13

Sets

foo.sar:30|sfoo.sar:50|s

stats.sets.foo.sar


Timers

foo.tar:30|ms

stats.timers.foo.tar.mean_90stats.timers.foo.tar.std

stats.timers.foo.tar.sum_90Friday, March 29, 13

Histograms

histogram: [{metric: "foo.tar", bins: [10,20,30,50]}]

stats.timers.foo.tar.histogram.bin_10stats.timers.foo.tar.histogram.bin_20


Pluggable backendsfunction MyBackend(startupTime, config, emitter){ var self = this; this.lastFlush = startupTime; this.lastException = startupTime; this.config = config.mybackendconfig || {};

// what to do with metrics emitter.on('flush', function(timestamp, metrics) {}); // update your status emitter.on('status', function(callback) {}); // handle raw packet emitter.on('packet', function(packet, rinfo) {});};


Management Interface

> telnet 127.0.0.1 8126

> stats

> counters

> delcounters

> help


Graphite + StatsD

• default backend

• whisper archives should match flush interval

• sum aggregation for raw counters

• docs/graphite.md


The single-thread dilemma


Tuning UDP

# increase default core memory sizes

net.core.rmem_default = 16777216

net.core.wmem_default = 16777216

net.ipv4.udp_wmem_min = 67108864

net.ipv4.udp_rmem_min = 67108864

net.ipv4.udp_mem = 4648512 6198016 9297024


Sampling

• default sampling of 0.1 on all metrics

• most sent keys (log file) are sampled 0.01


Quis monitoret ipsos monitores?


Let’s do thisFriday, March 29, 13

Basic setup

• git clone https://github.com/etsy/statsd.git

• curl https://gist.github.com/mrtazz/5270376 > test_config.js

• nc -k -l 2003 > output.log

• node stats.js test_config.js


https://github.com/etsy/statsd.git


https://gist.github.com/mrtazz/5270376


Send some stats

> echo -n "foo.bar:1|c" | nc -w0 -u 127.0.0.1 8125

> echo -n "foo.lerl:3|g" | nc -w0 -u 127.0.0.1 8125

> echo -n "foo.lol:33|ms" | nc -w0 -u 127.0.0.1 8125




Some actual statsDisk space:

df -m | grep "/var" | awk '{print "diskspace.var:"$4"|g"}'

Apache HTTPD processes:

echo -n "httpd.procs:$(ps aux | grep [h]ttpd | wc -l)|g"

Logfile size:

du -s /var/log/chef.log | awk '{print "cheflog:"$1"|g"}'


Is this real life?

• Set up a web page

• Have it send stats to StatsD

• page load time

• counters when page was accessed

• look in the examples/ directory for code


Histograms

• You should have timers now

• Configure histograms for them

• { graphite: { histograms: [] }


Level up!


Write a new backend

• Have it HTTP POST to a web server

• write metrics to a file

• send a txt message

• repeat all UDP packets to TCP


Hackathon ideas

• Metrics path validation in backend

• Pluggable receiver frontends

• Forked backends

• Configurable timer metrics


• #statsd on Freenode IRC

• [email protected]

• Ask me


mailto:[email protected]

mailto:[email protected]

Basic setup

• git clone https://github.com/etsy/statsd.git

• curl https://gist.github.com/mrtazz/5270376 > test_config.js

• nc -k -l 2003 > output.log

• node stats.js test_config.js






StatsD workshop @ Monitorama 2013

Daniel SchauenbergSoftware Engineer, Etsy Infrastructure

@mrtazz


statsd workshop monitorama 2013

Technology