managing and monitoring application performance
DESCRIPTION
Writing your application is one thing. Making the application to perform well is another. We usually forget there is somebody else on the other side of the screen, that becomes very frustrated and upset when he needs to wait until this one page finally loads. It requires a lot of experience to predict specific behaviour and to know what kind of things to avoid. And even with that there is so many different factors that can affect the end user experience. During this talk I will talk about tools and techniques you can use to measure and monitor your application performance.TRANSCRIPT
Sebastian Marek, Internal Systems Technical Architect
MANAGING AND MONITORING APPLICATION
PERFORMANCEhttp://www.flickr.com/photos/wraith34/9377593973/
a Pole living in Sheffield
over 12 years in software development
big fan of process automation
TDD and CI
occasionally contributes to open-source projects
wants to be a knight @proofek
h"ps://joind.in/9301
HTTPS://WWW.CODECLUB.ORG.UK/
A nationwide network of volunteer-led after school coding clubs for children aged 9-11
Volunteer Code Club Projects Venue Children
+ + +
PERFORMANCE MONITORINGOverview
network level
system level
application level
End-user level
http://www.flickr.com/photos/global-jet/483825828/
PERFORMANCE MONITORINGNetwork level
failed TCP connections and packet losses
bandwidth usage
firewalls and loadbalancers
connectivity to external serviceshttp://www.flickr.com/photos/sbrown6079/2038502257/
PERFORMANCE MONITORINGSystem level
CPU load
memory consumption
disk space
http://www.flickr.com/photos/viagallery/3178699697/
PERFORMANCE MONITORINGApplication level
new deployments
response time
key transactions
memory usage
http://www.flickr.com/photos/qualityfrog/3546657245/
PERFORMANCE MONITORINGEnd-user level
response time
user experience
http://www.flickr.com/photos/alesk/356136498/
remote debugging
code coverage
tracing
profiling
Xdebug
Installation
XDEBUG
# pecl install xdebug
zend_extension=”/usr/lib/php/extensions/xdebug.so”
configure in xdebug.ini
web server restart (optional)
Configuration
XDEBUG
;Profiling settings (always on)xdebug.profiler_enable = 1xdebug.profiler_output_dir = /var/log/xdebug/
;Profiling settings (triggered by GET/POST/COOKIE)xdebug.profiler_enable = 0xdebug.profiler_output_dir = /var/log/xdebug/xdebug.profiler_enable_trigger = 1
Browser integration
XDEBUG
Profiling data
XDEBUG
fl=php:internalfn=php::spl_autoload_register881 3
fl=/Users/Sebastian/git/app/src/include/HTMLPurifier/HTMLPurifier.standalone.phpfn=HTMLPurifier_Bootstrap::registerAutoload878 21cfl=php:internalcfn=php::spl_autoload_functionscalls=1 0 0880 1cfl=php:internalcfn=php::spl_autoload_registercalls=1 0 0881 3
Profiling tools
XDEBUG
KCacheGrind for *nix systems
qcachegrind for MacOsX (and apparently Windows)
MacCallGrind for MacOsX
WebGrind via browser
qCacheGrind
XDEBUG
WebGrind
XDEBUG
profiling
web interface
xhProf
Installation
XHPROF
# pecl install xhprof
extension=”/usr/lib/php/extensions/xhprof.so”
configure in xhprof.ini
web server restart (optional)
Configuration
XHPROF
; XHProfextension = xhprof.soxhprof.output_dir = /var/log/xhprof/
Profiling tools
XHPROF
default web interface
xhGui (old, also known as XH UI)
xhGui (new, with MongoDB backend)
Default web interface - Installation
XHPROF
;Webserver (VirtualHost)<VirtualHost *:80> DocumentRoot “/Users/Sebastian/git/xhprof/xhprof_html” ServerName xhprof.localdomain
<Directory /Users/Sebastian/git/xhprof/xhprof_html> Order allow,deny Allow from all </Directory>
ErrorLog “/private/var/log/apache2/xhprof-error_log” CustomLog “/private/var/log/apache2/xhprof-access_log” common</VirtualHost>
Default web interface - Installation
XHPROF
;auto_prepend file - enable xhprof$xhProfHome = __DIR__ . ‘/..’;$profiler_namespace = ‘myapp’; // namespace for your application
if (extension_loaded(‘xhprof’)) { include_once “$xhProfHome/xhprof_lib/utils/xhprof_lib.php”; include_once “$xhProfHome/xhprof_lib/utils/xhprof_runs.php”; xhprof_enable(XHPROF_FLAGS_CPU | XHPROF_FLAGS_MEMORY);}
Default web interface - Installation
XHPROF
;auto_prepend file - collect datafunction xhprofShutdownFunction(){ global $profiler_namespace; $xhProfUrl = “http://xhprof.localdomain/”;
if (extension_loaded(‘xhprof’)) { $xhprof_data = xhprof_disable(); $xhprof_runs = new XHProfRuns_Default(); $run_id = $xhprof_runs->save_run($xhprof_data, $profiler_namespace); $profiler_url = sprintf(“$xhProfUrl/index.php?run=%s&source=%s”, $run_id, $profiler_namespace); echo “<a href=’$profiler_url’ target=‘_blank’>Profiler output</a>”; }}register_shutdown_function(‘xhprofShutdownFunction’);
Default web interface - Installation
XHPROF
;enable profiling in your application (VirtualHost)<VirtualHost *:80> DocumentRoot “/Users/Sebastian/git/my_app/src” ServerName my_app.localdomain
<Directory /Users/Sebastian/git/my_app/src > Order allow,deny Allow from all </Directory>
php_admin_value auto_prepend_file /Users/Sebastian/git/xhprof/xhprof_html/header.php</VirtualHost>
Default web interface - Home page
XHPROF
Default web interface - Drilldown
XHPROF
Default web interface - Call graph
XHPROF
Default web interface - Full Call graph
XHPROF
XH UI - Installation
XHPROF
;Webserver (VirtualHost)<VirtualHost *:80> DocumentRoot “/Users/Sebastian/git/xhui/xhprof_html” ServerName xhui.localdomain
<Directory /Users/Sebastian/git/xhui/xhprof_html> Order allow,deny Allow from all </Directory>
ErrorLog “/private/var/log/apache2/xhui-error_log” CustomLog “/private/var/log/apache2/xhui-access_log” common</VirtualHost>
XH UI - Configuration
XHPROF
# vim xhui/xhprof_lib/config.php
and reuse already provided xhui/external/header.php
XH UI - Installation
XHPROF
;enable profiling in your application (VirtualHost)<VirtualHost *:80> DocumentRoot “/Users/Sebastian/git/my_app/src” ServerName my_app.localdomain
<Directory /Users/Sebastian/git/my_app/src > Order allow,deny Allow from all </Directory>
php_admin_value auto_prepend_file /Users/Sebastian/git/xhui/external/header.php</VirtualHost>
XH UI - Home page
XHPROF
XH UI - Transaction detail
XHPROF
XH UI - Transaction detail
XHPROF
XHG - Installation
XHPROF
;Webserver (VirtualHost)<VirtualHost *:80> DocumentRoot “/Users/Sebastian/git/xhgui/webroot” ServerName xhgui.localdomain
<Directory /Users/Sebastian/git/xhgui/webroot> Order allow,deny Allow from all AllowOverride All </Directory>
ErrorLog “/private/var/log/apache2/xhgui-error_log” CustomLog “/private/var/log/apache2/xhgui-access_log” common</VirtualHost>
XHG - Requirements
XHPROF
MongoDB server
PHP MongoDB extension
PHP mcrypt extension
XHG - Installation
XHPROF
# cd xhgui
# php install.php
XHG - Installation
XHPROF
;enable profiling in your application (VirtualHost)<VirtualHost *:80> DocumentRoot “/Users/Sebastian/git/my_app/src” ServerName my_app.localdomain
<Directory /Users/Sebastian/git/my_app/src > Order allow,deny Allow from all </Directory>
php_admin_value auto_prepend_file /Users/Sebastian/git/xhgui/external/header.php</VirtualHost>
XHG - Home page
XHPROF
XHG - Transaction trace
XHPROF
XHG - Transaction trace
XHPROF
XHG - Transaction trace
XHPROF
XHG - Transaction history
XHPROF
XHG - Transaction call graph
XHPROF
Glossary
PROFILING
call count - number of times a function/request was called
wt (Wall Time/Wall Clock Time) - time it took to execute a function/request
cpu - CPU time spent executing a function/request
mu - Amount of memory used during function/request execution
pmu - The peak amount of memory used during function/request execution
exclusive - total time spent excluding time to execute other functions
inclusive - total time spent including time to execute other functions
Process
PROFILING
Profile and record the results
Identify functions/requests using the most CPU time/memory (exclusive)
Find out what’s causing the issue
Refactor and optimize the code - look for external resources - database, web services, filesystem
Profile and compare the results
start all over again
server monitoring
application monitoring
real user monitoring
NewRelic
Supported web applications
NEW RELIC
Supported PHP framework
NEW RELIC
Installation
NEW RELIC
# RHEL
$ rpm -Uvh http://yum.newrelic.com/pub/newrelic/el5/x86_64/newrelic-repo-5-3.noarch.rpm
$ yum install newrelic-php5
$ /usr/bin/newrelic-install
# Ubuntu / Debian
$ wget -O - https://download.newrelic.com/548C16BF.gpg | sudo apt-key add -
deb http://apt.newrelic.com/debian/ newrelic non-free
$ apt-get install newrelic-php5
$ /usr/bin/newrelic-install
Server Monitoring
NEW RELIC
Server Monitoring - Main Dashboard
NEW RELIC
Portal01
App01
App02
Portal02
SugarCRM
SugarCRM
Support Centre
Server Monitoring - Overview
NEW RELIC
App02
App02
SugarCRM (App02)
Application Monitoring
NEW RELIC
Server Trace - Summary
NEW RELIC
App01(5.3.19+a2h(2.2.3):SugarCRM)
Server Trace - Details
NEW RELIC
Server Trace - SQL queries
NEW RELIC
Application Monitoring - Error Tracking
NEW RELIC
Real User Monitoring - App Dashboard
NEW RELIC
Understanding RUM
NEW RELIC
Web application - The time spent in the application code
Network - The time it takes for a request to make a round-trip over the internet.
DOM processing - Time spent in the browser parsing and interpreting the HTML.
Page rendering - Time spent in the browser displaying the HTML, running in-line JavaScript and loading images.
Understanding RUM
NEW RELIC
Source: http://blog.newrelic.com
Real User Monitoring - Browser Page Load
NEW RELIC
Real User Monitoring - Browser Page Load
NEW RELIC
Apdex
NEW RELIC
Apdex
NEW RELIC
0 T F
Satisfied zone: < T
Tolerating zone: T - F (4T)
Frustrated zone: > F
Apdex score
NEW RELIC
Satisfied customers + (Tolerating customers/2)
Total number of customers
Apdex by regions
NEW RELIC
Apdex - baseline
NEW RELIC
check the average response time
set your Apdex to average response time
find areas that require performance fixes
when performance improves lower your Apdex
New Relic - Map
NEW RELIC
memcached.local
crm.local
Keeping on top of the changes
NEW RELIC
Server Monitoring - plugins
NEW RELIC
Server Monitoring - APC
NEW RELIC
Server Monitoring - MySQL
NEW RELIC
Server Monitoring - Apache
NEW RELIC
PERFORMANCE MONITORINGSummary
Xdebug - application profiling during development stage
xhProf - application profiling during development stage and on live production platform
New Relic - end-to-end application monitoring on live production platfrom
PERFORMANCE MONITORINGAlternatives
DataDog - http://www.datadoghq.com/product/
GraphDat - http://www.graphdat.com
Scout - https://scoutapp.com/
Nagios - http://www.nagios.org/
Graphite - http://graphite.wikidot.com
ResourcesXdebug - http://www.xdebug.org/docs/profilerkCacheGrind - http://kcachegrind.sourceforge.netWebGrind - https://github.com/jokkedk/webgrindMacCallGrind for MacOsX - http://www.maccallgrind.com
xhProf - https://github.com/facebook/xhprofxhUI - https://github.com/preinheimer/xhprofxhGui - https://github.com/preinheimer/xhgui
New Relic - http://newrelic.comNew Relic Documentation - https://newrelic.com/docsApdex - http://apdex.org
http://techportal.inviqa.com/2009/12/01/profiling-with-xhprof/http://techportal.inviqa.com/2013/10/01/profiling-php-applications-with-xhgui/
Q & A
h"ps://joind.in/9301