© 2015 Phase2

Performance Profiling Tools & Tricks

19 October 2015

All Things Open 2015

© 2015 Phase2

About Me

● Brad Blake● Software Architect● bblake@phase2technology.com● www.phase2technology.com

© 2015 Phase2

Profiling

© 2015 Phase2

Profiling is not

● Benchmarking● Debugging● A dark art

© 2015 Phase2

Profiling is

● Gathering data on performance of a system○ CPU/Memory usage○ Function calls/times○ Many other metrics

● Find where system is spending its time● Refactoring, not changing functionality

© 2015 Phase2

Profiling is

© 2015 Phase2

Types of Profiling

● Event-based○ Triggered by specific events○ Generally slower○ More data and more accurate

● Statistical ( sampling )○ Record data at specific intervals○ Less accurate and specific○ Less intrusive

© 2015 Phase2

Before you start

● Determine if you have a performance issue● How big of an issue is it?● Benchmark

○ ab, httperf, siege, JMeter● Determine your goal

© 2015 Phase2

Do’s

● Get to a functional system quickly, then measure it○ Needs to be a ‘real’ system○ Alternative is to prematurely optimize

● Be data-driven● tail (-f) (-n) is your friend● Go after items with highest cost

○ Not necessarily the outliers● Test after each change

© 2015 Phase2

© 2015 Phase2

Don’ts

● Don’t make assumptions, be empirical● Don’t assume the data won’t change● Don’t assume caching is working

○ Check HTTP headers○ Check memcache, redis○ Query cache

● Don’t prematurely optimize

© 2015 Phase2

Tips

● Don’t over-optimize● Do data migration early

○ Or use realistic substitute data● Have a good test suite

○ Record metrics, graphs● Implement logging from the beginning

© 2015 Phase2

Tips (cont.)

● Increasing memory is NOT A FIX● Understand bigO concepts● Match production settings ( Vagrant, Chef, OpenStack, etc )

and configs ( production mode )

© 2015 Phase2

Tools

© 2015 Phase2

PHP

● xhprof● xdebug● New Relic

○ SaaS model for real-time monitoring○ Ruby, Python, PHP, Node○ Servers, even Docker containers

© 2015 Phase2

Xhprof

● Created by facebook, on PECL● Gets basic stats● Can do sampling● Can compare runs or aggregate runs

○ Useful in production to run only X runs● Modules/Plugins for Drupal and Wordpress● GUI not great, pretty basic

○ Xhgui

© 2015 Phase2

XDebug

● Debugger and Profiler● Gets a lot of data● Output can be used in kcachegrind ( qcachegrind )● phpStorm integration brew install qcachegrind brew install graphviz qcachegrind path-to-file

© 2015 Phase2

Ruby

● Run with production settings● Built-In Profiler__ module

○ Very basic, limited data

© 2015 Phase2

Ruby-Prof

● Event-based● Lots of reports and options

○ Text/HTML reports○ Flat profiles, call graphs, call stacks

● Can profile whole app or blocks of code

© 2015 Phase2

Ruby-Prof

© 2015 Phase2

Perftools.rb

● perftools.rb○ Sampling profiler○ gperftools for Ruby○ Also many output modes, including Callgrind

● Rack::PerftoolsProfiler○ More limited output modes○ Outputs to browser ( ?profile=true )○ Can change frequency or mode

© 2015 Phase2

Perftools.rb

© 2015 Phase2

Node.js

● Node-Webkit-Agent● Webstorm● StrongLoop Arc● NodeSource

© 2015 Phase2

Flamegraphs

© 2015 Phase2

Pretty

© 2015 Phase2

What is a Flamegraph?

● Visual representation of profiled software○ For our purposes, the call stack

● Each layer is a function● Each column is the amount of time spent

© 2015 Phase2

Why use a Flamegraph?

● Easier for non-technical people● Easy to diagnose the quick problems

○ What was on the CPU at each interval● Low overhead● Pretty

© 2015 Phase2

How to Generate

● https://github.com/brendangregg/FlameGraph● Call stack data converted to right format

● PHP○ https://github.com/msonnabaum/xhprof-flamegraphs○ Xdebug○ Drupal: xhprof_sample + xhprof_flamegraph

© 2015 Phase2

How to Generate

● Ruby○ https://github.com/MiniProfiler/rack-mini-profiler +○ flamegraph gem

● Node○ perf (perf_events)○ http://github.com/davepacheco/node-stackvis

© 2015 Phase2

Healthy

© 2015 Phase2

Unhealthy

© 2015 Phase2

Unhealthy

© 2015 Phase2

Query Tuning

© 2015 Phase2

How to Diagnose

● Slow query log● mysql: slow_query_log, slow_query_log_file

○ Logs all queries - X seconds and examine at least X rows○ Except admin statements ( alter, drop, etc )○ log_queries_not_using_indexes

● mongo: db.setProfilingLevel()○ system.profile collection

© 2015 Phase2

How to Diagnose

● SHOW [FULL] PROCESSLIST○ Sending data, Copying to tmp table, Locked, etc

● mysqldumpslow● Percona: pt-query-digest● Logging in your app

© 2015 Phase2

Why is my query slow?

● Too much data○ Fetching too many columns/rows○ Examining too many rows

● Poorly designed schema○ Shorter rows generally better

● Lack of proper indexes○ Or too many indexes

© 2015 Phase2

Why is my query slow?

● Inefficient SQL○ Subqueries vs. Joins○ COUNT(*)○ High Offsets○ Complex versus split queries

● Disk versus Memory● Sorting● Query Cache

© 2015 Phase2

EXPLAIN

● MySQL○ Prefix SELECT query with EXPLAIN

● Postgres○ EXPLAIN, EXPLAIN ANALYZE

● Mongo○ db.collection.explain()

© 2015 Phase2

The Optimizer

● Chooses execution plan with lowest cost○ Not necessarily the fastest○ Looks at number of rows, indexes, cardinality, length of

keys○ optimizer_trace ( 5.6 )○ EXPLAIN FORMAT=JSON ( 5.7 )

● Can do things like reorder and convert JOINs● Subquery optimization

© 2015 Phase2

EXPLAIN

EXPLAIN SELECT n.title, b.body_value FROM node nINNER JOIN node_revision nr ON n.vid = nr.vidINNER JOIN (SELECT entity_id, body_value FROM

field_data_body) as b ON n.nid = b.entity_idWHERE n.nid > 1000LIMIT 0,10

© 2015 Phase2

EXPLAIN

● id: The SELECT that the row belongs to. ( usually 1 )● select_type: Whether row is SIMPLE or Complex

○ PRIMARY, DERIVED, UNION● table: Generally table name or alias● type: Access Type. How MySQL will access the rows

© 2015 Phase2

EXPLAIN ( Access Type )

● ALL: Generally full table scan● index: Full index scan ( range: limited index scan )● ref: Join two non-unique indexed fields● eq_ref: Join unique non-null index and indexed field● const: Returns 1 value from index with unique key

© 2015 Phase2

EXPLAIN

● possible_keys: Possible indexes● key: Which index actually will be used● key_len: Bytes of the index used● ref: What it used to look up values in the index

© 2015 Phase2

EXPLAIN

● rows: ESTIMATE of rows MySQL will need to read● Extra: Anything else

○ Using filesort○ Using temporary○ Using where○ Using index

© 2015 Phase2

EXPLAIN

● Sometimes it lies or is inaccurate○ Table stats wrong ( ANALYZE TABLE )○ Can’t estimate every possible plan

● If you think optimizer is wrong:○ STRAIGHT_JOIN○ USE/IGNORE/FORCE {INDEXES}

© 2015 Phase2

Thank You

● bblake@phase2technology.com