Load testing

Who is this dude?

Lindsay Holmwood

BULLETPROOF

I Run this thing

Have tried and failed many

times to implement effective

load testing

A story

New project

Replacing a high-

profile website

(new infrastructure,

new application)

1000

concurrent users

“We need 100%

uptime”

Project starts

Requirements

change

Deadlines

aren’t met

Performance testing?

Fuck it!

We’ll do it live!

Performance testing?

“You have one day”

Problem:

Treated as

WAterfall-esque

QA

It’s a

non-functional

requirement

WHYDo we load test?

Capacity

planning

“Can the site deal

with load?”

Defect

detection

“Is this code

going to cripple

the site?”

Optimisation

“What happens if

we change this

Setting?”

Historical

analysis

“Is the site

getting slower?”

Cache warming

“Prepare the

infrastructure

for Failover”

WHYDo we load test?

A combination of

proactive &

Reactive reasons

A combination of

proactive &

Reactive reasons

It’s a

non-functional

requirement

Whydoes load testing

FAIL?

(from a technical perspective)

How

not the

WHAT

TooL fetishism

Get bogged down

in details

how the testing

infrastructure works

Micro

vs

Macro

Business

Does not care

about your tools

Business

cares About:

“Can the site deal

with load?”

Recommendation:

Use tool with

lowest barrier

TO entry

Blitz

Runs on AWS

Web form

Browser pluginsFirefox, Chrome

REST API

API ClientsRuby, Python, Java

Node.JS, Perl

Common

Command Line

Excellent

documentationdocs.blitz.io

Inexpensive250 concurrent users

over 60 second period == free

SPRINTS

&

RUSHES

Let’s dive in!

Sign up @http://blitz.io/

$ gem install blitz$ blitz helpUsage: blitz <command> <options> help - Display this help account:about - Show information about your account api:init - Validate and login with your API key couch:fuzz - Auto generate blitz tests from CouchDB curl - Run a sprint or a rush curl:help - Show help on sprint and rushing traceroute - Run traceroute remotely traceroute:help - Show help on traceroute version - Show the version of this Ruby gem$ blitz api:init # writes credentials to ~/.blitz/credentials

$ blitz curl:help

Usage: blitz curl <options> <url>

--user-agent -A <string> User-Agent to send to server--cookie -b name=<string> Cookie to send to the server (multiple)--data -d <string> Data to send in a PUT or POST request--dump-header -D <file> Print the request/response headers--referer -e <string> Referer URL--help -h Help on command line options--header -H <string> Custom header to pass to server--pattern -p <s>-<e>:<d> Ramp from s to e concurrent requests in d secs--region -r <string> california|oregon|virginia|singapore|ireland|japan--status -s <number> Assert on the HTTP response status code--timeout -T <ms> Wait time for both connect and responses--user -u <user[:pass]> User and password for authentication--request -X <string> Request method to use (GET, HEAD, PUT, etc.)--variable -v <string> Define a variable to use--verbose -V Print the request/response headers--tlsv1 -1 Use TLSv1 (SSL)--sslv2 -2 Use SSLv2 (SSL)--sslv3 -3 Use SSLv3 (SSL)

$ blitz curl --region singapore --pattern 1-250:20 \ --timeout 10000 --verbose http://example.org/

$ blitz curl --region singapore --pattern 1-250:20 \ --timeout 10000 --verbose http://example.org/

user growth / time

$ blitz curl --region singapore --pattern 1-250:20 \ --timeout 10000 --verbose http://example.org/

user growth / timefor AWS lag

rushing from singapore...

Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27

$ blitz curl --region singapore --pattern 1-250:20 \ --timeout 10000 --verbose http://example.org/

user growth / timefor AWS lag

--pattern 1-250:20

--pattern 1-250:20

blitz calculates the growth automatically

Sawtooth--pattern 1-250:20,1-250:20,1-250:20

TV AD / Twittersplosion--pattern 200-5000:30,5000-400:60

Integrate

with your

development cycle

VCS

CI

Deploy automation

VCS

CI

Deploy automation

git

post-receive hook

GitHub web hooks

Doesn’t take into

account test

failures

Is the code

actually

deployed?

VCS

CI

Deploy automation

CRON JOB

JENKINS JOB

GREAT if you have

a CI system

Runs tests

but

Separate system

What if you’re making

infrastructure changes

during a load test?

VCS

CI

Deploy automation

Fabric

Capistrano

Run tests when

you deploy

Use the same

toolchain

Testing lives IN the app.

What about triggering

load tests on

infrastructure

changes?

MVP

Just get

something

working!

Problem:

No REporting

:-(

rushing from singapore...

Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27

THIS IS IT

SOL ON:

Optimisation

“What happens if

we change this

Setting?”

Historical

analysis

“Is the site

getting slower?”

Showstopper?

...

$ gem install blitz

# ...

rushing from singapore...

Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27

Can we duck

punch our way

to success?

#!/usr/bin/env ruby# load_test.rb

require 'blitz'

arguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"

job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)

# ruby load_test.rb

rushing from singapore...

Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27

#!/usr/bin/env ruby# load_test.rb

require 'blitz'

arguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"

job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)

#!/usr/bin/env ruby# load_test.rb

require 'blitz'

arguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"

job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)

Blitz::Curl.parse(command)

returns

Blitz::Curl::Rush

#!/usr/bin/env ruby# load_test.rb

require 'blitz'

arguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"

job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)

Blitz::Command::Curl#rush

Blitz::Command::Curl#rush

initiates the test + outputs the test results

#!/usr/bin/env ruby# load_test.rb

require 'blitz'

arguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"

job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)

puts job.result.timeline.to_json

Produces muck

duck punch our

way to success

class Blitz::Curl::Rush class Point def to_json(*args) { 'timestamp' => @timestamp, 'duration' => @duration, 'total' => @total, 'hits' => @hits, 'errors' => @errors, 'timeouts' => @timeouts, 'volume' => @volume, 'txbytes' => @txbytes, 'rxbytes' => @rxbytes, 'steps' => @steps, }.to_json end end

# ...

# ...

class Step def to_json(*args) { 'duration' => @duration, 'connect' => @connect, 'errors' => @errors, 'timeouts' => @timeouts, 'asserts' => @asserts, }.to_json end endend

#!/usr/bin/env ruby# load_test.rb

require 'blitz'

arguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"

job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)

puts job.result.timeline.to_json

{"start":1335694346,"finish":1335694367,"results":[{"timestamp":2.502626,"duration":1.13912,"total":1,"hits":1,"errors":0,"timeouts":0,"volume":2,"txbytes":258.0,"rxbytes":17810.0,"steps":[{"duration":2.13912,"connect":0.260004,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":5.016102,"duration":0.640047,"total":5,"hits":5,"errors":0,"timeouts":0,"volume":4,"txbytes":1548.0,"rxbytes":90882.0,"steps":[{"duration":1.640047,"connect":0.158944,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":7.524594,"duration":0.639987,"total":10,"hits":10,"errors":0,"timeouts":0,"volume":5,"txbytes":3354.0,"rxbytes":182222.0,"steps":[{"duration":1.639987,"connect":0.158999,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":10.03388,"duration":0.63961,"total":20,"hits":20,"errors":0,"timeouts":0,"volume":7,"txbytes":5418.0,"rxbytes":364902.0,"steps":[{"duration":1.63961,"connect":0.158847,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":12.543211,"duration":0.640421,"total":31,"hits":31,"errors":0,"timeouts":0,"volume":9,"txbytes":9030.0,"rxbytes":565850.0,"steps":[{"duration":1.640422,"connect":0.158767,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":15.051705,"duration":0.639894,"total":44,"hits":44,"errors":0,"timeouts":0,"volume":10,"txbytes":12384.0,"rxbytes":803334.0,"steps":[{"duration":1.639894,"connect":0.158794,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":17.560136,"duration":-1.0,"total":44,"hits":44,"errors":0,"timeouts":0,"volume":0,"txbytes":12384.0,"rxbytes":803334.0,"steps":[{"duration":0.0,"connect":0.0,"errors":0,"timeouts":0,"asserts":0}]}]}

Do with as

you please

Here’s one I

prepared earlier...

...

You don’t have to

set up a tool

But you have to

write reporting?

At least you

have load

testing?

It’s on the way...

http://support.blitz.io/discussions/questions/114-sharing-reports

Where to

from here?

Pull in data from

other places

new relic

Ganglia/collectd

Git commits

Github:

square/crossfilter

Don’t like Blitz?

SaaS

Load impact

Browser mob

Open Source

Tsung

JMeter

Start Using

something!

Low barrier of

entry is key

Integrate with your

existing dev tools

Questions?