Download - Load testing with Blitz
Load testing
Who is this dude?
Lindsay Holmwood
BULLETPROOF
I Run this thing
Have tried and failed many
times to implement effective
load testing
A story
New project
Replacing a high-
profile website
(new infrastructure,
new application)
1000
concurrent users
“We need 100%
uptime”
Project starts
Requirements
change
Deadlines
aren’t met
Performance testing?
Fuck it!
We’ll do it live!
Performance testing?
“You have one day”
Problem:
Treated as
WAterfall-esque
QA
It’s a
non-functional
requirement
WHYDo we load test?
Capacity
planning
“Can the site deal
with load?”
Defect
detection
“Is this code
going to cripple
the site?”
Optimisation
“What happens if
we change this
Setting?”
Historical
analysis
“Is the site
getting slower?”
Cache warming
“Prepare the
infrastructure
for Failover”
WHYDo we load test?
A combination of
proactive &
Reactive reasons
A combination of
proactive &
Reactive reasons
It’s a
non-functional
requirement
Whydoes load testing
FAIL?
(from a technical perspective)
How
not the
WHAT
TooL fetishism
Get bogged down
in details
how the testing
infrastructure works
Micro
vs
Macro
Business
Does not care
about your tools
Business
cares About:
“Can the site deal
with load?”
Recommendation:
Use tool with
lowest barrier
TO entry
Blitz
Runs on AWS
Web form
Browser pluginsFirefox, Chrome
REST API
API ClientsRuby, Python, Java
Node.JS, Perl
Common
Command Line
Excellent
documentationdocs.blitz.io
Inexpensive250 concurrent users
over 60 second period == free
SPRINTS
&
RUSHES
Let’s dive in!
Sign up @http://blitz.io/
$ gem install blitz$ blitz helpUsage: blitz <command> <options> help - Display this help account:about - Show information about your account api:init - Validate and login with your API key couch:fuzz - Auto generate blitz tests from CouchDB curl - Run a sprint or a rush curl:help - Show help on sprint and rushing traceroute - Run traceroute remotely traceroute:help - Show help on traceroute version - Show the version of this Ruby gem$ blitz api:init # writes credentials to ~/.blitz/credentials
$ blitz curl:help
Usage: blitz curl <options> <url>
--user-agent -A <string> User-Agent to send to server--cookie -b name=<string> Cookie to send to the server (multiple)--data -d <string> Data to send in a PUT or POST request--dump-header -D <file> Print the request/response headers--referer -e <string> Referer URL--help -h Help on command line options--header -H <string> Custom header to pass to server--pattern -p <s>-<e>:<d> Ramp from s to e concurrent requests in d secs--region -r <string> california|oregon|virginia|singapore|ireland|japan--status -s <number> Assert on the HTTP response status code--timeout -T <ms> Wait time for both connect and responses--user -u <user[:pass]> User and password for authentication--request -X <string> Request method to use (GET, HEAD, PUT, etc.)--variable -v <string> Define a variable to use--verbose -V Print the request/response headers--tlsv1 -1 Use TLSv1 (SSL)--sslv2 -2 Use SSLv2 (SSL)--sslv3 -3 Use SSLv3 (SSL)
$ blitz curl --region singapore --pattern 1-250:20 \ --timeout 10000 --verbose http://example.org/
$ blitz curl --region singapore --pattern 1-250:20 \ --timeout 10000 --verbose http://example.org/
user growth / time
$ blitz curl --region singapore --pattern 1-250:20 \ --timeout 10000 --verbose http://example.org/
user growth / timefor AWS lag
rushing from singapore...
Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27
$ blitz curl --region singapore --pattern 1-250:20 \ --timeout 10000 --verbose http://example.org/
user growth / timefor AWS lag
--pattern 1-250:20
--pattern 1-250:20
blitz calculates the growth automatically
Sawtooth--pattern 1-250:20,1-250:20,1-250:20
TV AD / Twittersplosion--pattern 200-5000:30,5000-400:60
Integrate
with your
development cycle
VCS
CI
Deploy automation
VCS
CI
Deploy automation
git
post-receive hook
GitHub web hooks
Doesn’t take into
account test
failures
Is the code
actually
deployed?
VCS
CI
Deploy automation
CRON JOB
JENKINS JOB
GREAT if you have
a CI system
Runs tests
but
Separate system
What if you’re making
infrastructure changes
during a load test?
VCS
CI
Deploy automation
Fabric
Capistrano
Run tests when
you deploy
Use the same
toolchain
Testing lives IN the app.
What about triggering
load tests on
infrastructure
changes?
MVP
Just get
something
working!
Problem:
No REporting
:-(
rushing from singapore...
Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27
THIS IS IT
SOL ON:
Optimisation
“What happens if
we change this
Setting?”
Historical
analysis
“Is the site
getting slower?”
Showstopper?
...
$ gem install blitz
# ...
rushing from singapore...
Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27
Can we duck
punch our way
to success?
#!/usr/bin/env ruby# load_test.rb
require 'blitz'
arguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"
job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)
# ruby load_test.rb
rushing from singapore...
Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27
#!/usr/bin/env ruby# load_test.rb
require 'blitz'
arguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"
job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)
#!/usr/bin/env ruby# load_test.rb
require 'blitz'
arguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"
job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)
Blitz::Curl.parse(command)
returns
Blitz::Curl::Rush
#!/usr/bin/env ruby# load_test.rb
require 'blitz'
arguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"
job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)
Blitz::Command::Curl#rush
Blitz::Command::Curl#rush
initiates the test + outputs the test results
#!/usr/bin/env ruby# load_test.rb
require 'blitz'
arguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"
job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)
puts job.result.timeline.to_json
Produces muck
duck punch our
way to success
class Blitz::Curl::Rush class Point def to_json(*args) { 'timestamp' => @timestamp, 'duration' => @duration, 'total' => @total, 'hits' => @hits, 'errors' => @errors, 'timeouts' => @timeouts, 'volume' => @volume, 'txbytes' => @txbytes, 'rxbytes' => @rxbytes, 'steps' => @steps, }.to_json end end
# ...
# ...
class Step def to_json(*args) { 'duration' => @duration, 'connect' => @connect, 'errors' => @errors, 'timeouts' => @timeouts, 'asserts' => @asserts, }.to_json end endend
#!/usr/bin/env ruby# load_test.rb
require 'blitz'
arguments = "--region singapore --pattern 1:250:60"url = "http://example.org/"command = "#{arguments} #{url}"
job = Blitz::Curl.parse(command)curl = Blitz::Command::Curl.newcurl.rush(job)
puts job.result.timeline.to_json
{"start":1335694346,"finish":1335694367,"results":[{"timestamp":2.502626,"duration":1.13912,"total":1,"hits":1,"errors":0,"timeouts":0,"volume":2,"txbytes":258.0,"rxbytes":17810.0,"steps":[{"duration":2.13912,"connect":0.260004,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":5.016102,"duration":0.640047,"total":5,"hits":5,"errors":0,"timeouts":0,"volume":4,"txbytes":1548.0,"rxbytes":90882.0,"steps":[{"duration":1.640047,"connect":0.158944,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":7.524594,"duration":0.639987,"total":10,"hits":10,"errors":0,"timeouts":0,"volume":5,"txbytes":3354.0,"rxbytes":182222.0,"steps":[{"duration":1.639987,"connect":0.158999,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":10.03388,"duration":0.63961,"total":20,"hits":20,"errors":0,"timeouts":0,"volume":7,"txbytes":5418.0,"rxbytes":364902.0,"steps":[{"duration":1.63961,"connect":0.158847,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":12.543211,"duration":0.640421,"total":31,"hits":31,"errors":0,"timeouts":0,"volume":9,"txbytes":9030.0,"rxbytes":565850.0,"steps":[{"duration":1.640422,"connect":0.158767,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":15.051705,"duration":0.639894,"total":44,"hits":44,"errors":0,"timeouts":0,"volume":10,"txbytes":12384.0,"rxbytes":803334.0,"steps":[{"duration":1.639894,"connect":0.158794,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp":17.560136,"duration":-1.0,"total":44,"hits":44,"errors":0,"timeouts":0,"volume":0,"txbytes":12384.0,"rxbytes":803334.0,"steps":[{"duration":0.0,"connect":0.0,"errors":0,"timeouts":0,"asserts":0}]}]}
Do with as
you please
Here’s one I
prepared earlier...
...
You don’t have to
set up a tool
But you have to
write reporting?
At least you
have load
testing?
It’s on the way...
http://support.blitz.io/discussions/questions/114-sharing-reports
Where to
from here?
Pull in data from
other places
new relic
Ganglia/collectd
Git commits
Github:
square/crossfilter
Don’t like Blitz?
SaaS
Load impact
Browser mob
Open Source
Tsung
JMeter
Start Using
something!
Low barrier of
entry is key
Integrate with your
existing dev tools
Questions?