drupal performance - serbenfiquista.com case study
DESCRIPTION
Presentation done for Drupal Meetup Porto 27th November 2010.TRANSCRIPT
![Page 1: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/1.jpg)
Drupal Performance - Case Study
Hernâni Borges de Freitas – Drupal PT Meetup – Porto, 27th November
![Page 2: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/2.jpg)
Biggest online fan community about Sport Lisboa e Benfica.
About 10 years old, online since 01/05/2001 Done as hobby by:
Staff (20 people among editors and moderators) Members (~31 000 users, 8000 Active Members)
Articles, blog aggregation, image gallery, forum, press aggregation, matches and players’ profiles.
![Page 3: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/3.jpg)
Average per day (October 2010) 8000 Visits per Day 185 000 PageViews 12 pages per Session (!) 3500 messages per day. 50 blogs aggregated.
![Page 4: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/4.jpg)
According to Alexa.com (Oct
2010) 150th most visited website in
Portugal. On top 50 portuguese
websites. Most popular portuguese
website made in DRUPAL!
![Page 5: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/5.jpg)
Technology was slowing us down Custom designed cms, with 10 years legacy
code and cache control based in smarty. Site must be done by community
Workflow and revision process was weak and unsafe
Developments took to much time Hard to implement or change existing
features. Performance problems on traffic peaks
Around 1700 users at same time.
![Page 6: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/6.jpg)
Dedicated Server Quad Xeon 2,4 Ghz 4 Gb Memory Lighttpd/Apache as App Server Php with eAccelerator.
Most of pageviews are seen by registered users.
Most pageviews are generated by forum (based on Simple Machines Forum).
Cache control was made using smarty and forum cache system.
![Page 7: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/7.jpg)
Development started in August 2010 Done on spare time by 1 Drupalista. We went live in 24th October 2010 (2
months ...) Website redone from scratch Data migration done used custom scripts
60k nodes, 30k users, 2,5k terms, 16 content types
100 modules Portuguese/English , Web/Mobile Site
![Page 8: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/8.jpg)
Optimize using iterative improvements (Progressive Doping) Architecture Profiling Caching Application diet WebServer Change Drupal Tips and Tricks Future Ideas
![Page 9: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/9.jpg)
Optimize queries and function calls Measure non-cached page loading times. Optimize heavier pages. Reduce http calls.
Turn on page caching system (AuthCache)
Replace caching backend
Test, Optmize , Test, Optmize, Test, Optimize !!
![Page 10: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/10.jpg)
Use Pressflow! High speed drupal fork▪ Optimized for PHP5 and Mysql (No
wrapper functions)▪ Designed for Database replication
and Reverse proxies.▪ Squid / Varnish
▪ Optimized in session and path handling▪ Non Anon-Sessions (Lazy session
creation)▪ Fast path alias detection
Avoid tons of calls to drupal_lookup_path.
![Page 11: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/11.jpg)
Use DevelUse xdebug.profiling. Identify heavier pages, functions and
queries. Start by most visited pages Try to identify which functions are taking most
time.You’ll find a non pleasant detail
Bootstrap in a normal site can be slow.Great to understand how drupal core
works!Great to measure cost vs importance
![Page 12: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/12.jpg)
Generic Drupal cache handling
![Page 13: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/13.jpg)
After activate Drupal caching: On Bootstrap until
DRUPAL_BOOTSTRAP_LATE_PAGE_CACHE phase Drupal verify if user is anonymous▪ If so check if we have a valid cached page▪ Delivery it without load all modules, render all
regions... FAST! Blocks and some content is also cached,
and can be served to authenticated users. Cached content is stored in database tables Tables are flushed
Nodes and comments are posted Cron runs Explicits calls to cache_clear_all
![Page 14: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/14.jpg)
However most of our traffic is authenticated We can’t use drupal base cache There’s a module for that ! => authcache
AuthCache Register cookie variables on login like roles,
login name and login profile. On page_early_cache bootstrap verify if there’s
a cached version of that page to the roles the user belongs.
If there isn’t do full_bootstrap , render the page and save a rendered version on cache to future usage.
![Page 15: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/15.jpg)
Include a setting in settings.php:
Configure roles and pages to cache We are not caching anything to
editors/moderators, neither any page in admin section or content edition.
Be Careful with ajax stuff..
$conf['cache_inc'] = './sites/all/modules/authcache/authcache.inc';
![Page 16: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/16.jpg)
Small problem: all page looks the same to everyone.
We want to customize the header with a pleasant message.
Authcache recommendation is to do page replacements using ajax calls => More http calls
To avoid http traffic I tweaked authcache module to do a str_replace of a certain zone, and start to store cached pages not gzipped.
![Page 17: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/17.jpg)
MySql is not designed to be used as a cache backend.
Fortunately Drupal allows pluggable cache backends.
We started by using cache router, using memcached One bin for most cache table We ended up using memcache module
because of several crashs and white screens.
![Page 18: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/18.jpg)
Install MemCache, MemCache Pecl Extensionconf = array(
'memcache_servers' => array('localhost:11211' => 'default', 'localhost:11212' => 'cache_block', 'localhost:11213' => 'cache_page', 'localhost:11214' => 'users', 'localhost:11215' => 'session'),
'memcache_bins' => array('cache' => 'default', 'cache_block' => 'cache_block', 'cache_page' => 'cache_page', 'users'=>'users', 'session'=>'session'), 'memcache_key_prefix' => 'sb10prod',);
$conf['cache_inc'] = './sites/all/modules/authcache/authcache.inc';
![Page 19: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/19.jpg)
On popular pages use custom cache!
We are storing in cache_block to allow block refresh when new content arrives.
if($content=cache_get('artigos:home:cronicas','cache_block')) {$block->content=$content->data;
}else {
$view=views_get_view('artigos');$view->set_display('panel_pane_1');$block->content=$view->render();$block->content.='<span style="clear:both;float:right">'.l(t("View
all"),'cronicas').'</span>’; cache_set('artigos:home:cronicas',$block->content,'cache_block');
}
![Page 20: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/20.jpg)
Sessions Table was heavily used.We replace it with memcache
session module. Serious dropdown on server loadPressflow already does not store
sessions for anon users => non_anon module does the same.
UPDATE sessions SET uid = 1, cache = 0, hostname = '10.1.1.2', session = '', timestamp = 1243567406 WHERE sid = '74db42a7f35d1d54fc6b274ce840736e'
$conf['session_inc'] = './sites/all/modules/memcache/memcache-session.inc';
![Page 21: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/21.jpg)
In forum pages just call what is needed.require_once './includes/bootstrap.inc';drupal_bootstrap(DRUPAL_BOOTSTRAP_PATH);$arg0=arg(0);if($arg0=='forum’) {
require_once './includes/common.inc';drupal_load('module','serbenfiquista');drupal_load('module','filter');drupal_load('module','locale');require_once './includes/theme.inc';
$content=render_forum();/* do some load of modules when content not cached, vars will be
available at theme….
*/require_once
('./sites/default/themes/serbenfiquista/page.tpl.php'); }
![Page 22: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/22.jpg)
Remember we were redirecting only non-static content to apache
What about css/js aggregated files and imagecache files ? => They were going to apache also..
$HTTP["url"] !~ "\.(js|css|png|gif|jpg|ico|txt|swf)$" { proxy.server = ( "" => ( ( "host" => ”localhost ", "port" => 81) ) ) }
![Page 23: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/23.jpg)
07/Nov/2010 – Things went bad
![Page 24: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/24.jpg)
Apache was handling too much connections.
We were runnning out of memory, and no more connections available...
After that nightmare by night we decide to switch to nginx.
.
![Page 25: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/25.jpg)
Using php in php-fpm mode Configuration based on perusio’s examples:
https://github.com/perusio/drupal-with-nginx Using eAccelerator 0.9.5.3 as opcode cache to
use shared keys in memory used by SMF.
extension="eaccelerator.so”eaccelerator.shm_size="300"eaccelerator.cache_dir="/tmp/eaccelerator"eaccelerator.enable="1"eaccelerator.optimizer="1"eaccelerator.check_mtime=“0” #avoids to check if file has changed eaccelerator.shm_max=”0
![Page 26: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/26.jpg)
Don’t forget to use Css and JS aggregation to avoid http connections
Index customization (Use EXPLAIN on your queries)
Run cron twice a day
Do not use cron to import feeds
Use Apache Solr to index your content.
0 */12 * * * cd /var/www/html/ && drush core-cron >> /var/log/crondrupal
*/10 * * * * cd /var/www/html/ && drush feeds-refresh blogs >> /var/log/cronblogs
select title,created,nid from node use index(node_status_type) where status=1 and type='usernews' order by created desc limit 0,4
![Page 27: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/27.jpg)
We have load peaks when some content is changed: most of cached content is erased.
Control in detail what is cached and expire only what is invalid.
Pre-Cache most page details. Use Cache Actions and Rules to clean specific
views/blocks /panes. When page is regenerated its components are
already rendered. I wish we could be more elastic by serving
content from S3 or other smaller webservers.
![Page 28: Drupal Performance - SerBenfiquista.com Case Study](https://reader035.vdocuments.net/reader035/viewer/2022062419/558585ecd8b42aac148b4e56/html5/thumbnails/28.jpg)
SerBenfiquista.com by:Alberto Rodrigues, André Garcia, André Sabino, André Marques, António Alves, António Costa, Bernardo Azevedo, Diogo Sarmento, Élvio da Silva,Filipe Varela, Francisco Castro, Hugo Manita, Hernâni Freitas, João Pessoa Jorge, João Cunha, João Mariz, José Barata, Isabel Cutileiro,, Luis Correia,Miguel Ferreira, Nelson Vidal, Nuno Silva, Paulo Paulos, Pedro Lança, Pedro Neto, Rafael Santos, Rodrigo Marques, Ricardo Martins, Ricardo Solnado, Valter Gouveia
and plenty others !