Grinder To Graphite DocumentationRelease 0.1.0
Travis Bear
January 07, 2015
Contents
1 Overview 1
2 Who should use g2g? 3
3 Who should use Grinder Analyzer instead of g2g? 5
4 Source Code 7
5 Contents 95.1 Quickstart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95.2 Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105.3 Installation Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105.4 Configuring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115.5 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125.6 Where Is My Stuff? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145.7 Gallery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155.8 Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195.9 Additional Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
i
ii
CHAPTER 1
Overview
Grinder to Graphite (g2g) is a tool that analyzes the logs from your Grinder tests, and sends the data into Graphitewhere it can be visualized in a variety of ways.
Realtime test data may be sent to Graphite while your Grinder run is in progress, or it may be sent to Graphite afteryour test is completed.
Once the data is in Graphite you have a great amount of flexibility in the types of reports and visualizations you wantto generate.
Example Graph
1
Grinder To Graphite Documentation, Release 0.1.0
2 Chapter 1. Overview
CHAPTER 2
Who should use g2g?
g2g may be a good match for you if any of the following are true:
• You have access to a Graphite setup already, or you don’t mind installing it.
• You sometimes do long-duration Grinder runs and you don’t want to wait for the run to complete before youcan see charts of the incoming data.
• You want to integrate data from The Grinder with data from a variety of other sources. (OS metrics like CPUuse, application metrics like DB lookups per second, etc.)
3
Grinder To Graphite Documentation, Release 0.1.0
4 Chapter 2. Who should use g2g?
CHAPTER 3
Who should use Grinder Analyzer instead of g2g?
If you just want to get some fast, simple graphs from your Grinder run, without a lot of setup hassle, Grinder Analyzeris probably a better bet for you than g2g. See:
http://track.sourceforge.net
5
Grinder To Graphite Documentation, Release 0.1.0
6 Chapter 3. Who should use Grinder Analyzer instead of g2g?
CHAPTER 4
Source Code
g2g code is hosted on Bitbucket. See
https://bitbucket.org/travis_bear/grinder_to_graphite
7
Grinder To Graphite Documentation, Release 0.1.0
8 Chapter 4. Source Code
CHAPTER 5
Contents
5.1 Quickstart
5.1.1 Prerequisites
Graphite
If you don’t already have a Graphite server running on your network, you will need to set one up. See the Graphiteweb site for details on setting up and configuring Graphite
http://graphite.wikidot.com/
Pip
Pip is the best way to install python packages like Grinder to Graphite. If you don’t have pip, there are many easyways to get it. See
http://www.pip-installer.org/en/latest/installing.html
5.1.2 Grinder to Graphite
Get it!
Install it with pip
pip install grinder_to_graphite
Configure it!
g2g comes with a command-line option to generate a sample config file
g2g -e
This will generate a file named ‘g2g.sample.yaml’ which you can use as the basis for creating your own configuration.
9
Grinder To Graphite Documentation, Release 0.1.0
Run it!
Once you have your Grinder logs in place that you want to ingest into Graphite, invoke g2g
g2g <config file>
View it!
Your Grinder test data should now be visible in Graphite
5.2 Compatibility
g2g has been tested on these versions of Python:
• python 2.7
• pypy
g2g is not compatible with Python versions earlier than 2.7.
Python 3 support is a priority, and will be added soon.
5.3 Installation Notes
5.3.1 Install via Pip
g2g is written in Python. The best way to install g2g is via pip.
pip install grinder_to_graphite
5.3.2 Virtualenv
Using virtualenv is not required. You can easily install g2g into your core Python environment. However, you willlikely need to be an admin user to to this, or you will need to preface the pip command with sudo
However, there are many advantages to using virtualenv, and installing g2g into its own environment is the recom-mended way to proceed.
5.3.3 Pypi
g2g runs more than twice as fast on pypy as it does on standard cpython. If you will regularly be ingesting largeamounts of data, consider installing pypy.
http://pypy.org/download.html
5.3.4 Generate Configuration File
Once pip has installed g2g, you will need to generate a g2g config file, and edit it to be suitable for your own environ-ment. g2g comes with a command-line option to generate a sample config file
10 Chapter 5. Contents
Grinder To Graphite Documentation, Release 0.1.0
g2g -e
This will generate a file named ‘g2g.sample.yaml’ in the current directory which you can use as the basis for creatingyour own configuration.
5.3.5 Graphite
There must be a running installation of Graphite on your network for g2g to forward data to. See the Graphite website for details on setting up and configuring Graphite
http://graphite.wikidot.com/
5.4 Configuring
5.4.1 Sections
The g2g config file has these sections
Sec-tion
Re-quired?
Notes
g2g.carbonyes This section contains settings relevant to your graphite installation.g2g.data yes This section tells g2g about your local Grinder log files.g2g.analysisno Defines optional additional calculations to generate percentile and response time
group statistics.g2g.run no Controls behavior when g2g is run against the same log files multiple times.log-ging
yes Standard python logging configuration for the g2g output. Seehttp://docs.python.org/2/library/logging.config.html for how to configure thissection. The settings provided should be fine in most cases.
5.4.2 Individual Settings
These are the individual settings
5.4. Configuring 11
Grinder To Graphite Documentation, Release 0.1.0
Setting Re-quired?
De-fault
Notes
g2g.carbon.host yes none Hostname of the Graphite (carbon) server fo g2g to send data to.g2g.carbon.port yes none Port the carbon server listens on.g2g.carbon.prefix yes none Can be empty string. Prefix appears before the host name in the
Graphite namespace.g2g.carbon.suffix yes none Can be empty string. Suffix appears after the host name in the
Graphite namespace.g2g.carbon.interval_secondsyes none This should be set at or below the carbon reporting interval defined
on your graphite server.g2g.data.log_file yes none The path to the Grinder data file on your local filesystem. The data
file normally has a name like <hostname>-0-data.logg2g.data.mapping_fileyes none The path to the Grinder mapping file on your local filesystem. The
mapping file normally has a name like <hostname>-0.log. The finallines of a valid mapping file contain a table summarizing theGrinder run.
g2g.analysis.response_time_percentilesno [ ] If this setting is defined, g2g will calculate the response times forthe specified percentiles. Percentile values must be expressed as alist of numbers between 0.0 and 1.0
g2g.analysis.time_group_millisecondsno [ ] If this setting is defined g2g will calculate the percentage of allrequests that complete within specified ranges of time. Time groupsmust be expressed as a list of millisecond values.
g2g.run.follow no False Enable continually watching the Grinder data file for new entries. Iftrue, g2g will run forever until killed or interrupted with ^C.
g2g.run.follow_interval_secondsno 60 Specifies how often g2g will report new data back to Graphite whenfollow mode has been enabled. Should be a multiple of thecarbon.interval_seconds setting.
g2g.run.resume no False Read the Grinder data file from the last read location instead offrom the beginning. This setting is ignored if follow is enabled.
5.5 Usage
5.5.1 Running
‘g2g’ is the Grinder to Graphite executable. Normal usage (after adjusting the values in your sample config file to beappropriate for your environment) looks like this:
g2g <config_file>
Once you have established a useful g2g configuration file, it is common to use it over and over. You can set a defaultconfig file like this:
g2g -s <g2g config_file>
or
g2g --set-config-file=<g2g config_file>
After you have set a default config file, g2g can then be optionally invoked without specifying any other configuration.
g2g
12 Chapter 5. Contents
Grinder To Graphite Documentation, Release 0.1.0
5.5.2 Modes
There are three primary ways to use g2g:
Grinder Run Completed: Default Mode
The most common use of grinder to graphite is the case where you have completed a Grinder run and you want to sendthe results to Graphite. This is the default usage, which is described above. No special options are required.
Grinder Run In Progress: Follow Mode
If you have a long-duration Grinder run in progress, you might not want to wait for it to finish before you beginforwarding results to Graphite. g2g supports this via a follow (tailing) mode, where new results are continuallyforwarded to graphite.
For follow mode to work, you will first need to do a quick run with your grinder script to generate a valid mappingfile. A mapping file is a grinder log file that most likely has a name like <hostname>-0.log. Once a valid mapping fileis in place, you can do realtime reporting for your Grinder script as many times as you want.
Tailing mode can be enabled in the config file by setting follow to True. It can also be enabled on the command lineusing either of these options:
g2g -f
or
g2g --follow=True
Grinder Run In Progress: Resume Mode
For long-running Grinder tests, there is an alternate to follow mode. It is possible to re-run g2g manually on a Grinderlog that is still being written, and process only at the data that is new since the last time g2g has been run. This avoidsre-processing potentially large amounts of data while giving you control over when g2g runs.
Resuming log processing can be enabled in the config by setting resume to True. It can also be specified on thecommand line using either of these options
g2g -r
or
g2g --resume=True
5.5.3 Options
Many of the settings in the g2g config file can be overridden on the command line. To see a full list of availableoptions:
g2g --help
5.5. Usage 13
Grinder To Graphite Documentation, Release 0.1.0
5.5.4 Getting Help
Questions about g2g can be asked on the grinder-use mailing list. To subscribe, see
http://sourceforge.net/p/grinder/mailman/
Bugs can be reported to the project site on Bitbucket.
https://bitbucket.org/travis_bear/grinder_to_graphite/issues
5.6 Where Is My Stuff?
When everything is working properly, your metrics will appear in Graphite under the following namespace:
5.6.1 Non-Http Tests
<optional carbon prefix><hostname>
<optional carbon suffix><test name>
per_secondtx_passedtx_failed
latencymeanpercentile
<specific percentiles defined in config>group
<response time groups defined in config>
5.6.2 Http Tests
<optional carbon prefix><hostname>
<optional carbon suffix><test name>
per_secondbytes_downloadedtx_passedtx_failednew_connectionshttp_status
rc_200rc_302rc_404...
latencymean_timepercentile
<specific percentiles defined in config>group
<response time groups defined in config>http
resolve_host
14 Chapter 5. Contents
Grinder To Graphite Documentation, Release 0.1.0
mean_timeestablish_connection
mean_timefirst_byte
mean_time
5.7 Gallery
5.7.1 Examples
Once your data is in Graphite, there is a wide variety of ways you can combine it to get interesting charts. Here arejust a few examples of the ways you can present the data collected from g2g.
TPS for All Agents
A stacked graph showing the total TPS achieved by each running agent process, as well as the total for all agents.
Response Times
When the response times for all agents are overlayed, we can get a sense if the agents are getting results that areconsistent with each other.
5.7. Gallery 15
Grinder To Graphite Documentation, Release 0.1.0
5.7.2 Response Time Groups
Shows the percentage of all requests that happen in user-defined response time thresholds.
16 Chapter 5. Contents
Grinder To Graphite Documentation, Release 0.1.0
Response Time Components
In this graph we take advantage of the fact that Grinder shows us not just the total response time, but also the separatecomponents of response time, including
• host resolution
• connection establishment
• time to first byte
In this test, the “total” and “first byte” times were close enough together that the “total” line is obscures the “first byte”line.
5.7. Gallery 17
Grinder To Graphite Documentation, Release 0.1.0
New Connections and Total TPS
In this test we discovered that as load increased, a jump in new connections would occur.
18 Chapter 5. Contents
Grinder To Graphite Documentation, Release 0.1.0
5.8 Best Practices
5.8.1 Ease of Use
Use virtualenv
It will make your life easier.
Set a default config file
It is pretty common to want to use the same g2g config file for many different invocations of g2g. If you set a defaultconfig file, your usage changes from
g2g <config file>
becomes simply
g2g
Choose prefix and suffix carefully
These can help you effectively manage your Graphite namespace. Divide up your metrics by application, type, or anyother criteria you can imagine.
Examples:
5.8. Best Practices 19
Grinder To Graphite Documentation, Release 0.1.0
Carbon Prefix Carbon Suffix<none> grinder<none> grinder.app_namegrinder <none>
5.8.2 Performance
Use pypy
In many cases, g2g runs more than twice as fast on pypy as it does on cpython. If you will regularly be ingesting largeamounts of data, consider installing pypy.
Be judicious when configuring response time metrics
Response time groups and percentiles can add numerous new metrics to Graphite. Be sure you’ve selected the onesyou need. If you change these settings frequently, you could wind up with many stale metrics in Graphite, making ithard to see which ones currently apply.
Possible gotchas
g2g has to keep each grinder transaction in memory until it is flushed to Graphite. It’s possible that very high TPSGrinder tests, coupled with very long graphite reporting intervals, could cause memory issues in g2g. This has not beena problem in any of the testing we’ve done so far. However, if this happens you can lower the carbon_interval_secondssetting to reduce the amount of data g2g has to keep in memory. Because this approach will cause only a subset of theavailable data to be retained by Graphite, it’s best to only use it if you are actually having g2g memory problems.
5.9 Additional Resources
Getting your Grinder data into Graphite is just one part of the bigger picture. Ideally, you want all your stuff – ap-plication data, OS-level performance metrics, Grinder data, deployment timestamps – to feed into a common locationwhere it can all be integrated. These tools can help.
5.9.1 Exposing Application Data
Coda Hale Metrics
This library makes it easy for Java-based servers to expose performance data to JMX:
http://metrics.codahale.com/
It can optionally forward it to Graphite as well.
Statsd
A network daemon that listens for statistics, like counters and timers, sent over UDP and sends aggregates to one ormore pluggable backend services (e.g., Graphite).
https://github.com/etsy/statsd/ http://codeascraft.com/2011/02/15/measure-anything-measure-everything/
20 Chapter 5. Contents
Grinder To Graphite Documentation, Release 0.1.0
Jmxtrans
Java/JMX counters and application-level metrics can be fed to Graphite using the JMXTrans tool:
https://github.com/jmxtrans/jmxtrans
5.9.2 Exposing OS-level data
quickstatd
OS-level metrics (CPU, mem, etc.) can be fed to Graphite via quickstatd or collectd (with graphite plugin)
https://bitbucket.org/travis_bear/quickstatd
collectd
Another good tool for collecting OS-level performance data and forwardng it to Graphite:
http://collectd.org/
5.9.3 Keeping an eye on your data
If you have a bazillion metrics in your Graphite, it can be hard to keep an eye on all of them, or notice when somethinginteresting happens. The Kale tools can help.
Kale has two parts: Skyline, and Oculus.
• https://github.com/etsy/skyline
• https://github.com/etsy/oculus
5.9. Additional Resources 21