grinder to graphite documentation · g2g.run.resume no false read the grinder data ﬁle from the...

Grinder To Graphite DocumentationRelease 0.1.0

Travis Bear

January 07, 2015

Contents

1 Overview 1

2 Who should use g2g? 3

3 Who should use Grinder Analyzer instead of g2g? 5

4 Source Code 7

5 Contents 95.1 Quickstart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95.2 Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105.3 Installation Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105.4 Configuring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115.5 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125.6 Where Is My Stuff? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145.7 Gallery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155.8 Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195.9 Additional Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

i

CHAPTER 1

Overview

Grinder to Graphite (g2g) is a tool that analyzes the logs from your Grinder tests, and sends the data into Graphitewhere it can be visualized in a variety of ways.

Realtime test data may be sent to Graphite while your Grinder run is in progress, or it may be sent to Graphite afteryour test is completed.

Once the data is in Graphite you have a great amount of flexibility in the types of reports and visualizations you wantto generate.

Example Graph

1

Grinder To Graphite Documentation, Release 0.1.0

2 Chapter 1. Overview

CHAPTER 2

Who should use g2g?

g2g may be a good match for you if any of the following are true:

• You have access to a Graphite setup already, or you don’t mind installing it.

• You sometimes do long-duration Grinder runs and you don’t want to wait for the run to complete before youcan see charts of the incoming data.

• You want to integrate data from The Grinder with data from a variety of other sources. (OS metrics like CPUuse, application metrics like DB lookups per second, etc.)

3


4 Chapter 2. Who should use g2g?

CHAPTER 3

Who should use Grinder Analyzer instead of g2g?

If you just want to get some fast, simple graphs from your Grinder run, without a lot of setup hassle, Grinder Analyzeris probably a better bet for you than g2g. See:

http://track.sourceforge.net

5

http://track.sourceforge.net


6 Chapter 3. Who should use Grinder Analyzer instead of g2g?

CHAPTER 4

Source Code

g2g code is hosted on Bitbucket. See

https://bitbucket.org/travis_bear/grinder_to_graphite

7

https://bitbucket.org/travis_bear/grinder_to_graphite


8 Chapter 4. Source Code

CHAPTER 5

Contents

5.1 Quickstart

5.1.1 Prerequisites

Graphite

If you don’t already have a Graphite server running on your network, you will need to set one up. See the Graphiteweb site for details on setting up and configuring Graphite

http://graphite.wikidot.com/

Pip

Pip is the best way to install python packages like Grinder to Graphite. If you don’t have pip, there are many easyways to get it. See

http://www.pip-installer.org/en/latest/installing.html

5.1.2 Grinder to Graphite

Get it!

Install it with pip

pip install grinder_to_graphite

Configure it!

g2g comes with a command-line option to generate a sample config file

g2g -e

This will generate a file named ‘g2g.sample.yaml’ which you can use as the basis for creating your own configuration.

9


http://www.pip-installer.org/en/latest/installing.html


Run it!

Once you have your Grinder logs in place that you want to ingest into Graphite, invoke g2g

g2g <config file>

View it!

Your Grinder test data should now be visible in Graphite

5.2 Compatibility

g2g has been tested on these versions of Python:

• python 2.7

• pypy

g2g is not compatible with Python versions earlier than 2.7.

Python 3 support is a priority, and will be added soon.

5.3 Installation Notes

5.3.1 Install via Pip

g2g is written in Python. The best way to install g2g is via pip.

pip install grinder_to_graphite

5.3.2 Virtualenv

Using virtualenv is not required. You can easily install g2g into your core Python environment. However, you willlikely need to be an admin user to to this, or you will need to preface the pip command with sudo

However, there are many advantages to using virtualenv, and installing g2g into its own environment is the recom-mended way to proceed.

5.3.3 Pypi

g2g runs more than twice as fast on pypy as it does on standard cpython. If you will regularly be ingesting largeamounts of data, consider installing pypy.

http://pypy.org/download.html

5.3.4 Generate Configuration File

Once pip has installed g2g, you will need to generate a g2g config file, and edit it to be suitable for your own environ-ment. g2g comes with a command-line option to generate a sample config file

10 Chapter 5. Contents

http://pypy.org/download.html


g2g -e

This will generate a file named ‘g2g.sample.yaml’ in the current directory which you can use as the basis for creatingyour own configuration.

5.3.5 Graphite

There must be a running installation of Graphite on your network for g2g to forward data to. See the Graphite website for details on setting up and configuring Graphite


5.4 Configuring

5.4.1 Sections

The g2g config file has these sections

Sec-tion

Re-quired?

Notes

g2g.carbonyes This section contains settings relevant to your graphite installation.g2g.data yes This section tells g2g about your local Grinder log files.g2g.analysisno Defines optional additional calculations to generate percentile and response time

group statistics.g2g.run no Controls behavior when g2g is run against the same log files multiple times.log-ging

yes Standard python logging configuration for the g2g output. Seehttp://docs.python.org/2/library/logging.config.html for how to configure thissection. The settings provided should be fine in most cases.

5.4.2 Individual Settings

These are the individual settings

5.4. Configuring 11


http://docs.python.org/2/library/logging.config.html


Setting Re-quired?

De-fault

Notes

g2g.carbon.host yes none Hostname of the Graphite (carbon) server fo g2g to send data to.g2g.carbon.port yes none Port the carbon server listens on.g2g.carbon.prefix yes none Can be empty string. Prefix appears before the host name in the

Graphite namespace.g2g.carbon.suffix yes none Can be empty string. Suffix appears after the host name in the

Graphite namespace.g2g.carbon.interval_secondsyes none This should be set at or below the carbon reporting interval defined

on your graphite server.g2g.data.log_file yes none The path to the Grinder data file on your local filesystem. The data

file normally has a name like <hostname>-0-data.logg2g.data.mapping_fileyes none The path to the Grinder mapping file on your local filesystem. The

mapping file normally has a name like <hostname>-0.log. The finallines of a valid mapping file contain a table summarizing theGrinder run.

g2g.analysis.response_time_percentilesno [ ] If this setting is defined, g2g will calculate the response times forthe specified percentiles. Percentile values must be expressed as alist of numbers between 0.0 and 1.0

g2g.analysis.time_group_millisecondsno [ ] If this setting is defined g2g will calculate the percentage of allrequests that complete within specified ranges of time. Time groupsmust be expressed as a list of millisecond values.

g2g.run.follow no False Enable continually watching the Grinder data file for new entries. Iftrue, g2g will run forever until killed or interrupted with ^C.

g2g.run.follow_interval_secondsno 60 Specifies how often g2g will report new data back to Graphite whenfollow mode has been enabled. Should be a multiple of thecarbon.interval_seconds setting.

g2g.run.resume no False Read the Grinder data file from the last read location instead offrom the beginning. This setting is ignored if follow is enabled.

5.5 Usage

5.5.1 Running

‘g2g’ is the Grinder to Graphite executable. Normal usage (after adjusting the values in your sample config file to beappropriate for your environment) looks like this:

g2g <config_file>

Once you have established a useful g2g configuration file, it is common to use it over and over. You can set a defaultconfig file like this:

g2g -s <g2g config_file>

or

g2g --set-config-file=<g2g config_file>

After you have set a default config file, g2g can then be optionally invoked without specifying any other configuration.

g2g



5.5.2 Modes

There are three primary ways to use g2g:

Grinder Run Completed: Default Mode

The most common use of grinder to graphite is the case where you have completed a Grinder run and you want to sendthe results to Graphite. This is the default usage, which is described above. No special options are required.

Grinder Run In Progress: Follow Mode

If you have a long-duration Grinder run in progress, you might not want to wait for it to finish before you beginforwarding results to Graphite. g2g supports this via a follow (tailing) mode, where new results are continuallyforwarded to graphite.

For follow mode to work, you will first need to do a quick run with your grinder script to generate a valid mappingfile. A mapping file is a grinder log file that most likely has a name like <hostname>-0.log. Once a valid mapping fileis in place, you can do realtime reporting for your Grinder script as many times as you want.

Tailing mode can be enabled in the config file by setting follow to True. It can also be enabled on the command lineusing either of these options:

g2g -f

or

g2g --follow=True

Grinder Run In Progress: Resume Mode

For long-running Grinder tests, there is an alternate to follow mode. It is possible to re-run g2g manually on a Grinderlog that is still being written, and process only at the data that is new since the last time g2g has been run. This avoidsre-processing potentially large amounts of data while giving you control over when g2g runs.

Resuming log processing can be enabled in the config by setting resume to True. It can also be specified on thecommand line using either of these options

g2g -r

or

g2g --resume=True

5.5.3 Options

Many of the settings in the g2g config file can be overridden on the command line. To see a full list of availableoptions:

g2g --help

5.5. Usage 13


5.5.4 Getting Help

Questions about g2g can be asked on the grinder-use mailing list. To subscribe, see

http://sourceforge.net/p/grinder/mailman/

Bugs can be reported to the project site on Bitbucket.

https://bitbucket.org/travis_bear/grinder_to_graphite/issues

5.6 Where Is My Stuff?

When everything is working properly, your metrics will appear in Graphite under the following namespace:

5.6.1 Non-Http Tests

<optional carbon prefix><hostname>

<optional carbon suffix><test name>

per_secondtx_passedtx_failed

latencymeanpercentile

<specific percentiles defined in config>group

<response time groups defined in config>

5.6.2 Http Tests

<optional carbon prefix><hostname>

<optional carbon suffix><test name>

per_secondbytes_downloadedtx_passedtx_failednew_connectionshttp_status

rc_200rc_302rc_404...

latencymean_timepercentile

<specific percentiles defined in config>group

<response time groups defined in config>http

resolve_host


http://sourceforge.net/p/grinder/mailman/

https://bitbucket.org/travis_bear/grinder_to_graphite/issues


mean_timeestablish_connection

mean_timefirst_byte

mean_time

5.7 Gallery

5.7.1 Examples

Once your data is in Graphite, there is a wide variety of ways you can combine it to get interesting charts. Here arejust a few examples of the ways you can present the data collected from g2g.

TPS for All Agents

A stacked graph showing the total TPS achieved by each running agent process, as well as the total for all agents.

Response Times

When the response times for all agents are overlayed, we can get a sense if the agents are getting results that areconsistent with each other.

5.7. Gallery 15


5.7.2 Response Time Groups

Shows the percentage of all requests that happen in user-defined response time thresholds.



Response Time Components

In this graph we take advantage of the fact that Grinder shows us not just the total response time, but also the separatecomponents of response time, including

• host resolution

• connection establishment

• time to first byte

In this test, the “total” and “first byte” times were close enough together that the “total” line is obscures the “first byte”line.

5.7. Gallery 17


New Connections and Total TPS

In this test we discovered that as load increased, a jump in new connections would occur.



5.8 Best Practices

5.8.1 Ease of Use

Use virtualenv

It will make your life easier.

Set a default config file

It is pretty common to want to use the same g2g config file for many different invocations of g2g. If you set a defaultconfig file, your usage changes from

g2g <config file>

becomes simply

g2g

Choose prefix and suffix carefully

These can help you effectively manage your Graphite namespace. Divide up your metrics by application, type, or anyother criteria you can imagine.

Examples:

5.8. Best Practices 19


Carbon Prefix Carbon Suffix<none> grinder<none> grinder.app_namegrinder <none>

5.8.2 Performance

Use pypy

In many cases, g2g runs more than twice as fast on pypy as it does on cpython. If you will regularly be ingesting largeamounts of data, consider installing pypy.

Be judicious when configuring response time metrics

Response time groups and percentiles can add numerous new metrics to Graphite. Be sure you’ve selected the onesyou need. If you change these settings frequently, you could wind up with many stale metrics in Graphite, making ithard to see which ones currently apply.

Possible gotchas

g2g has to keep each grinder transaction in memory until it is flushed to Graphite. It’s possible that very high TPSGrinder tests, coupled with very long graphite reporting intervals, could cause memory issues in g2g. This has not beena problem in any of the testing we’ve done so far. However, if this happens you can lower the carbon_interval_secondssetting to reduce the amount of data g2g has to keep in memory. Because this approach will cause only a subset of theavailable data to be retained by Graphite, it’s best to only use it if you are actually having g2g memory problems.

5.9 Additional Resources

Getting your Grinder data into Graphite is just one part of the bigger picture. Ideally, you want all your stuff – ap-plication data, OS-level performance metrics, Grinder data, deployment timestamps – to feed into a common locationwhere it can all be integrated. These tools can help.

5.9.1 Exposing Application Data

Coda Hale Metrics

This library makes it easy for Java-based servers to expose performance data to JMX:

http://metrics.codahale.com/

It can optionally forward it to Graphite as well.

Statsd

A network daemon that listens for statistics, like counters and timers, sent over UDP and sends aggregates to one ormore pluggable backend services (e.g., Graphite).

https://github.com/etsy/statsd/ http://codeascraft.com/2011/02/15/measure-anything-measure-everything/


http://metrics.codahale.com/

https://github.com/etsy/statsd/

http://codeascraft.com/2011/02/15/measure-anything-measure-everything/


Jmxtrans

Java/JMX counters and application-level metrics can be fed to Graphite using the JMXTrans tool:

https://github.com/jmxtrans/jmxtrans

5.9.2 Exposing OS-level data

quickstatd

OS-level metrics (CPU, mem, etc.) can be fed to Graphite via quickstatd or collectd (with graphite plugin)

https://bitbucket.org/travis_bear/quickstatd

collectd

Another good tool for collecting OS-level performance data and forwardng it to Graphite:

http://collectd.org/

5.9.3 Keeping an eye on your data

If you have a bazillion metrics in your Graphite, it can be hard to keep an eye on all of them, or notice when somethinginteresting happens. The Kale tools can help.

Kale has two parts: Skyline, and Oculus.

• https://github.com/etsy/skyline

• https://github.com/etsy/oculus

5.9. Additional Resources 21

https://github.com/jmxtrans/jmxtrans

https://bitbucket.org/travis_bear/quickstatd

http://collectd.org/

https://github.com/etsy/skyline

https://github.com/etsy/oculus

grinder to graphite documentation · g2g.run.resume no false read the grinder data ﬁle from the...

Documents