ruby on rails performance tuning. make it faster, make it better (windycityrails)
DESCRIPTION
(reposting with clearer title) Performance tuning presentation from WindyCityRails 2010. Why performance matters The right way to approach it Front end testing tools Automated testing tools Common problems and the ways to solve them in Rails Rails specific tools bullet slim_scrooge rack bug request log analyzer rails indexesTRANSCRIPT
Rails Performance
John McCaffreyRailsperformance.blogspot.com
John McCaffrey: Its all about me
• Doing Rails since 2007• 3rd time presenting at WindyCityRails
o 2008: Advanced Firebug and JS unit testingo 2009: PDF Generation in Railso 2010: Rails Performance Tuning
• Addicted to Performance Tuning• railsperformance.blogspot.com
• Feedback: spkr8.com/t/4413 Or bit.ly/PerfR8
@J_McCaffrey
(the slides will be available, with tons of extra references)
What are we gonna cover?
• Performance touches a lot of layers• We have a very diverse audience
Measuring Page Load
Fixing page load problems
Analyzing Server Performance
Database issues
Ruby issues Q&A
Starting off on the right foot
It all started...
def organization_branches get_organizations.flatten.uniq.select{|org| org.children.size > 0}end
def organization_leaves get_organizations.flatten.uniq.select{|org| org.children.size == 0}end
def load_organizations branches = current_user.organization_branches leaves = current_user.organization_leaves @branches = branches.to_json(:only => [:id, :name]) @leaves = leaves.to_json(:only => [:id, :name]) @organizations = @criteria.branch ? leaves.collect {|o| [ o.name.titleize, o.id ] }.sort : branches.collect {|o| [ o.name.titleize, o.id ] }.sort end
def organization_branches get_organizations.flatten.uniq.select{|org| org.children.size > 0}end
def organization_leaves get_organizations.flatten.uniq.select{|org| org.children.size == 0}end
def load_organizations branches = current_user.organization_branches leaves = current_user.organization_leaves @branches = branches.to_json(:only => [:id, :name]) @leaves = leaves.to_json(:only => [:id, :name]) @organizations = @criteria.branch ? leaves.collect {|o| [ o.name.titleize, o.id ] }.sort : branches.collect {|o| [ o.name.titleize, o.id ] }.sort end
It all started...
2811 queries!
Down to 17!!
And then...
Invoicing issue: If a customer resubmits an order with 2 years, they should be charged $9 but were only being charged $4.50
Do people ever calculate the ROI on software projects?
53626 * $4.50 = $241,317 (just in the last year)
Our theme of the day: Leverage
Multiplying the power of your efforts, to achieve the greatest value
Performance and Business metrics: Bing.com
Slower site = revenue drop 1 sec = 2.8% 2 sec = 4.3%
Performance and Business metrics: Shopzilla.com
http://www.phpied.com/the-performance-business-pitch/
Went from 6 secdown to 1.2sec
Fred Wilson: Speed is the most important feature
10 Golden Principles of Successful Web Apps Fred Wilson, Venture Capitalist www.avc.com
1.Speed2. Instant Utility3. Software is media4. Less is More5. Make it programmable6. Make it personal7. Make it RESTful8. Discoverability9. Clean10.Playful
OK, so where do we begin?
Give yourself a fighting chance, make a plan
Alois Reitbauer, dynatrace
Anti-patterns1. Thinking Scalability something you just sprinkle on later2. Guessing, not testing3. Ad-hoc, unstructured4. Waiting until it hurts, not proactively planning5. Thinking that any early planning is Premature optimization
Video on wiki.parleys.com
Premature optimization
"Premature optimization is the root of all evil"
-Donald Knuth
Premature optimization
Some people really like Donald Knuth
Premature optimization, with DJ DK
like, really, really like him
Premature optimization
"Premature optimization is the root of all evil'
-Donald Knuth
"Quoting Knuth, as a way to avoid planning for performance issues, is a cop-out"@J_McCaffrey
"Don't waste time on making optimization changes until you KNOW they are necessary"
If you can not measure it, you can not improve it.
— Lord Kelvin
Let's do this the right way
Measure:• Understand the tools to use, at each layer, to measure
and monitor performance
Create repeatable tests:• Be able to invoke the system and easily test
Isolate your changes:• One thing at a time
That's it: Measure, Test, Isolated changes
Terminology
Load:How much work is there?
Utilization:How much of the resources are in use?
Scalability:How well can we handle the load?
Throughput:How many tasks can be done per unit of time?
Concurrency:How many tasks can we do at once?
Capacity:How big can our throughput go before things fall apart?
Response time (avg) == Latency
Requests per second == Throughput
Terminology
Performance:• The resources and Time for a single request• Measured in Response Time
Throughput:• The number requests that can be handled per/time• Measured in Requests per second
Scalability:• Ability to grow and handle more requests, without
significantly degrading the experience
Performance != Scalability
Terminology
Let's say: 1 worker that can compute a job in .5 sec
Throughput = 2 jobs per sec Latency = .5 sec per job
Performance != Scalability
Adding more workers will not improve the latency
But not having enough workers can increase the response time, due to Queuing and delay
There are performance opportunities at each layer
Waterfall: Twitter.com
Waterfall
Total time: 3815 ms Dynamic generation: 250 ms (~7%) Static: 3369 ms (~89%) * Will not add to 100%
Waterfall
When the backend is actually doing work
Walmart: Waterfall
Waterfall
Total time: 5520 ms Dynamic generation: 388 ms (~7%) Static: 4580 ms (~82%)
Firebug Netpanel
Simple and repeatable way to measure page load
Usually, less than 20% of the time is spent in fetching the HTML
So how do we optimize the other 80%?
YSlow
• Created by Steve Souders• Observes the netpanel data, applies rules and gives you a
score.• Deep understanding of how the browser works• Its the best way to have a repeatable, measurement of
your site's load performance
http://developer.yahoo.com/yslow/
YSlow
1. Minimize HTTP Requests• Use a Content Delivery Network• Add an Expires or a Cache-Control Header• Gzip Components• Put StyleSheets at the Top• Put Scripts at the Bottom• Avoid CSS Expressions• Make JavaScript and CSS External• Reduce DNS Lookups• Minify JavaScript and CSS• Avoid Redirects• Remove Duplicate Scripts• Configure ETags• Make AJAX Cacheable• Use GET for AJAX Requests• Reduce the Number of DOM Elements• No 404s• Reduce Cookie Size• Use Cookie-Free Domains for Components• Avoid Filters• Do Not Scale Images in HTML• Make favicon.ico Small and Cacheable
Yslow: Created by Steve Souders
http://stevesouders.com/cuzillion/
Google Page speed
• Similar to YSlow• includes paint events• separates out ads, trackers from your content• better recommendations and optimizations• 1 click to see optimized version of your files
http://code.google.com/speed/page-speed
Google Page speed
Google Page speed
Make it repeatable
http://www.showslow.com/ track your site, or your competitors will check every 24hrs for you
• webpagetest.org• gtmetrix.org• zoompf.com• loadimpact.com• gomez.com
Summary
1. Page load is the place to start2. Read everything in the Yslow and PageSpeed rules3. Encourage other devs to play with YSlow, PageSpeed4. Show Webpagtest.org, and zoompf.com
5. Now that we have a way to measure page performance, we can think about making changes
Measuring Page Load
Fixing page load problems
Analyzing Server Performance
Database issues
Ruby issues Q&A
Starting off on the right foot
Isn't everyone already doing this?
Analysis of the Alexa top 1000 sites found:
1. 42% don't gzip2. 44% have more than 2 css files3. 56% serve css from a cookied domain4. 62% don't minify5. 31% have more than 100k size css
Compress with Gzip
1. Can save you 50% to 80% of your bandwidth2. Simplest performance improvement you can make3. Beware differences in browsers4. File types: html, js, css, xml, rss, txt, font, json
Probably the simplest and smartest thing you can do for your app!
Combine and minify javascript and css
1. Jsmin or YUICompressor, or Sprockets2. Asset_packager3. javascript_include_tag :cache => "cache/all"
Load common js libraries from googlehttp://github.com/rpheath/google_ajax_libraries_api
Sprites and image optimzation1. spriteme.org2. JqueryUI already gives you sprited images
Image optimization1. Smush.it
Expires and Browser Caching
Setting a far future expires tells the browser not to ask for the file
Using the query string lets us break the caching by having the browser ask for a new file
There is a problem with using a URL query string this way
http://blog.eliotsykes.com/2010/05/06/why-rails-asset-caching-is-broken/
Rails Caching
Take your dynamic content, and make it static1. Page2. Action3. Fragment
Full length tutorials1. http://railslab.newrelic.com/2. http://railscasts.com/episodes?search=caching3. Greg Pollack Aloha on Rails http://vimeo.com/10860860
Measuring Page Load
Fixing page load problems
Analyzing Server Performance
Database issues
Ruby issues Q&A
Starting off on the right foot
The Stack
You are here
Request-log-analyzer
• http://github.com/wvanbergen/request-log-analyzer• Request distribution per hour• Most requested• HTTP methods• HTTP statuses returned• Rails action cache hits• Request duration• View rendering time• Database time• Process blockers• Failed requests
Rack::Bug
Rack::Bug
Commercial tools
New Relic• Developer plugin is a good start• Free version doesn’t get you much• Bronze version with heroku
• Scoutapp.com• Similar to new Relic• Large selection of community plugins
Testing tools
Apache bench ab -n 10 -c 2 http://www.somewhere.com/
Httperfhttperf --server localhost --port 3000 --uri / --num-conns 10000
Jmeteryes, its ugly, but its powerful
Measuring Page Load
Fixing page load problems
Analyzing Server Performance
Database issues
Ruby issues Q&A
Starting off on the right foot
Database issues
Common database issues• Bad queries• Not utilizing explain• Inadequate indexes• N+1 queries• Selecting more data than is needed• Inconsistent queries for the same data
Query_reviewer
Common database issues• Bad queries• Not utilizing explain
• http://github.com/dsboulder/query_reviewer • Runs explain on all of your queries, outputs to div in
page
Rails_indexes
http://github.com/eladmeidar/rails_indexes
Foreign key indexs
• Columns that need to be sorted
• Lookup fields
• Columns that are used in a GROUP BY
• Rake tasks to find missing indexes.
New one I haven’t tried yet
http://github.com/samdanavia/ambitious_query_indexer
Bullet plugin
Bullet plugin
• http://github.com/flyerhzm/bullet
Help you reduce the number of queries
with alerts (and growl).
Slim_scrooge
Be more specific in your select• http://github.com/sdsykes/slim_scrooge• Instruments your code• Observes the usage pattern• Suggests/changes your select statement
• When invoked within the exact same context
Ruby Issues
Most are due to memory problems• Slow GC• Not release references• Capturing scope
• Profiling will reveal what’s going on• http://guides.rubyonrails.org/performance_testing.html
If you want to learn everything there is to know about profiling • Aman Gupta• Joe Damato• http://timetobleed.com/ • http://memprof.com/ • They’ve already identified and fixed several memory leaks
Ruby issues: Faster Ruby Libraries
C Extension++
• XML parser http://nokogiri.org/
• JSON parser http://github.com/brianmario/yajl-ruby/
• CSV parser http://www.toastyapps.com/excelsior/
• HTTP client http://github.com/pauldix/typhoeus
Date
http://github.com/jeremyevans/home_run
Take home points
Performance• Performance affects the bottom line• The biggest performance win is usually in improving the load time• Continue to monitor and test your apps performance over time• Gzip, combine and minify to get a big boost• Lack of indexes is likely to be one of your biggest backend issues• Try to stay up to date with Gems, Plugins, and latest Rails• Upgrading to ruby 1.9 will give you a huge performance boost
Links: great info on performance
Performance• Scaling Rails series http://railslab.newrelic.com/ • RailsCasts.com• Velocity Conference http://en.oreilly.com/velocity2010 • google io conference http://code.google.com/events/io/2010/ • dynatrace site http://ajax.dynatrace.com/pages/ • http://www.igvita.com/ • http://www.mysqlperformanceblog.com/
Beyond ySlow: improving page performance outside the initial load
MySql and Postgres Database tuning
Heroku: Building Scalable apps from the start
Tuning performance for Mobile devices
Performance Improvements coming in Rails 3, and 3.1
Speeding up your tests
Wait, I made it through that?
Architectural musings
•Background long running tasks– Email, image processing, reports, feeds
•Leverage Rack, Sinatra or Padrino•Parallel processing, Event Machine•Learn from heroku, scalable from the start•Revaluate your cloudy-ness– Great RailsConf presentation by James Golick
Wait, I made it through that?
Designing for performance• Watch how they really use it
• Multiple tabs?• Back and forth between list and detail view
• Great presentation at WindyCityRails2009 by David Eisinger • Optimizing perceived performance• Make it feel fast
Questions for you
Architectural musings
• Moving to or already on Ruby 1.9?• Serving your content from a separate domain?– Cookieless?