Tuning Blackboard 9 for high demanded environments

José Manuel López Luján Juan Francisco García García

Contact Information

•  José Manuel López Luján –  Email: josem.lopez@itesm.mx –  Twitter: @jmanuel_ll –  Blog: http://jm.inextgen-it.com – G+: jose.lopez.lujan@gmail.com

•  Juan Francisco García García –  Email: jfgarcia@itesm.mx –  Twitter: @jfgarcia11 – G+: jfgarcia@gmail.com

EXO-ENVIRONMENT

Tec de Monterrey

91,084  students  8,391  teachers  31  campus  22  sites  &  interna8onal  offices  58  undergraduate  programs  42  master  degree  programs  10  doctorate  programs  14,000+  students  served  by  our  Virtual  University  16,811,469  visits  (Jan-­‐Jul)*  

103,403  ac8ve  users  (Sep  ‘10)  2,321,864  unique  visitors  

*GOOGLE  ANALYTICS  &  BBSTATS  

WAN Topology

Alestra  VPN  

260  GB  Blackboard  throughput/day  

Internet  

2Mbps  ea.  

2Mbps  ea.  Videoconferencing  QoS  

2  (most),  4,  6,  34  Mbps    

68  Mbps  >  305  Mbps  due  to  Bb  demand  

2009  status  Not  the  same  figures  Surely  an  approximate  trend  

USA   México  

Welcome to Region 4! (aka Americas’ 4th World)

Facts

•  We can’t afford to waste even 1 KB or 1 Kbps

•  We have to be as efficient as we can •  Some actions seem ridiculous: everything helps

ENDO-ENVIRONMENT

Infrastructure

Infrastructure

Infrastructure

Blackboard in Tec de Monterrey System

BRAINWORK (or braindrain)

Bad Perception

Knowledge & Communication

Compression   Op8miza8on  

Minifying  

Traffic

InMemory  Compression  

gzip  

Client  Side  Cache  Mgmt  

Caching  

Akamai  

Service Outages

Java  Memory  Mgmt  

Accurate  Load  Balancing  

blog  

support  community  

social  media  

©  FUTURAMA:  MATT  GROENIG,  20TH  CENTURY  FOX,  COMEDY  CENTRAL  

reqs    valida8on    

Apache  2.2  

Custom  Error  Pages  

Akamaizing

•  Volume���transferred: ���15.7 TB 2010

•  Office files •  PDFs •  Some scripts •  Media files

–  Audio –  Vídeo –  Flash

Web Compression���Enabling

 bbconfig.webserver.compression=Yes  bbconfig.webserver.compression.level=6  bbconfig.webserver.compression.min.filesize=600  bbconfig.webserver.compression.max.filesize=20000000  bbconfig.webserver.compression.max.filesize.mem=60000  

 

$BB_HOME/config/bb-config.properties

mod_gzip_temp_dir    @@bbconfig.basedir@@/system/temp    CustomLog    @@bbconfig.basedir@@/logs/httpd/mod_gzip.log  common_with_mod_gzip_info2  

$BB_HOME/apps/httpd/conf/gzip.conf.bb

70%  traffic  reduc8on  

32 % CPU AVG

.831% CPU AVG

CPU Usage | Network Traffic | User Perception

As is No compression

Default Bb Compression

Network traffic 100%

30%

Bad  pe

rcep

8on  “ind

ex”  

CPU Usage

BPI  

Web Compression���Enabling

 bbconfig.webserver.compression=Yes  bbconfig.webserver.compression.level=6  bbconfig.webserver.compression.min.filesize=600  bbconfig.webserver.compression.max.filesize=20000000  bbconfig.webserver.compression.max.filesize.mem=60000  

 

$BB_HOME/config/bb-config.properties

mod_gzip_temp_dir    @@bbconfig.basedir@@/system/temp    CustomLog    @@bbconfig.basedir@@/logs/httpd/mod_gzip.log  common_with_mod_gzip_info2  

$BB_HOME/apps/httpd/conf/gzip.conf.bb

70%  traffic  reduc8on  I/O  HDD  

Web Compression

With  Gz  +W  over  HDD  With  Gz  +W  over  RAM   No  compression  

CPU  Usage  

Web Compression Threads  

With  Gz  +W  over  HDD  With  Gz  +W  over  RAM   No  compression  

Web Compression

# Working directory for temporary files and the compression cache if not specified, the following default values are used: [Win32=c:\temp], [UNIX=/tmp]

TEMP  dir  

Mod-gzip_temp_dir /tmp CustomLog "|@@bbconfig.basedir@@/apps/httpd/bin/rotatelogs @@bbconfig.basedir@@/logs/httpd/mod_gzip_log-%Y-%m_%d_%H_%M_%S 3600 -300" common_with_mod_gzip_info2

6.97 % CPU AVG

32 % CPU AVG

.831% CPU AVG

CPU Usage | Network Traffic | User Perception

As is No compression

Default Bb Compression

(HD)

Memory compression

Network traffic 100%

30%

Bad  pe

rcep

8on  “ind

ex”  

CPU Usage

BPI  

BPI  

Default vs Custom Login page Payload  with  No  Compression  

Login

News from various sources thru twitter

Guest login

Links to support social Media and related links

Char size + high contrast

Minifying & compression

648  

184.9  

419  

102.2  

763.4  

506.5  

184.1  

146  

0   100   200   300   400   500   600   700   800   900  

Login  Page      

Login  Page  w/gzip    

Login  Page  w/mini    

Login  Page  w/min+gzip    

Transfer  payload  

5 times less overhead! �Multiply it by 66k logins/hour�

Tec  de  Monterrey  

Bb  Default  

Client-side Cache Mgmt���

Implement mod_headers and mod_expires modules Apache 1.3: Modify apxs scripts in $BBHOME/apps/httpd/bin

Apache 1.3: Modify add config file in $BBHOME/apps/httpd/conf

LoadModule expires_module libexec/mod_expires.so LoadModule headers_module libexec/mod_headers.so # Turn on Expires and set default to 0 ExpiresActive On ExpiresDefault A0 # Set up caching on media files for 1 year (forever?) <FilesMatch "\.(flv|ico|pdf|avi|mov|ppt|doc|mp3|wmv|wav)$"> ExpiresDefault A29030400 Header append Cache-Control "public" </FilesMatch> # Set up caching on media files for 1 week <FilesMatch "\.(gif|jpg|jpeg|png|swf)$"> ExpiresDefault A604800 Header append Cache-Control "public" </FilesMatch> # Set up caching on commonly updated files for 1 week <FilesMatch "\.(xml|txt|html|js|css)$"> ExpiresDefault A604800 # Header append Cache-Control "proxy-revalidate" Header append Vary Accept-Encoding </FilesMatch> # Force no caching for dynamic files <FilesMatch "\.(php|cgi|pl|htm|jsp|jspf|do)$"> ExpiresActive Off Header set Cache-Control "private, no-cache, no-store, proxy-revalidate, no-transform" Header set Pragma "no-cache" </FilesMatch>

Tuning TCP/UDP

tcp_conn_req_max_q 2048

tcp_conn_req_max_q0 10240

tcp_smallest_anon_port 1024

tcp_time_wait_interval 60000

tcp_xmit_hiwat 65536

tcp_recv_hiwat 65536

udp_smallest_anon_port 1024

udp_xmit_hiwat 65536

udp_recv_hiwat 65536

Load Balancing

COOKIE_INSERT_EXPIRATION_DATE 200 minutes Probe HEALTHCHECK_BB

Request to http://APPX.itesm.mx/webapps/portal/healthCheck

Probe TCP8007 Ping TCP 8007

Probe ICMP Probe TCP80

Ping TCP 80

BB9ITESM: cursos.itesm.mx APP2, MaxConn 800, weight 8 APP3, MaxConn 800, weight 8 APP4, MaxConn 800, weight 8 APP5, MaxConn 600, weight 6 APP6, MaxConn 600, weight 6 APP7, MaxConn 600, weight 6

CURSOS-SYNC: cursos-sync.itesm.mx APP5, MaxConn 200, weight 8 APP6, MaxConn 200, weight 8 APP7, MaxConn 200, weight 8

Apache 2.2���why not?

Requests/second   Apache  1.3.41   Apache  2.2.19   Improvement  

Image   745   1413   89%  

Binary  file   149   380   155%  

Dynamic  content   832   1484   78%  

Solaris  10  SPARC,  v490,  16G  RAM  

Using NUMA���on NUMA-Architecture servers

•  30% (32-bit) to 40% (64-bit) performance improvement

•  Turn it on with���-XX:UseNUMA in bbconfig.jvm.options.e

xtra.tomcat parameter in bb-config.properties

SM Support Strategy

Custom error pages

ErrorDocument 500 /common/error500.html ErrorDocument 502 /common/error502.html ErrorDocument 503 /common/error503.html ErrorDocument 403 /common/error403.html ErrorDocument 404 /common/error404.html

ErrorDocument 401 /common/error401.html

Twitter Connect with  Twiqer4j  

Requirements validation

inspired  in  Blackboard  Browser  Tester  b2  by  Dr.  Malcolm  Murray  from  DU  and  implemented  and  modified  a  JavaScript  file  which  Shannon  Meisenheimer  from  UCMO  wrote.  

CA Service Desk Manager Connect

CA Service Desk Manager Connect

Technical information

Facebook Newsfeed module

Please provide feedback for this session by emailing DevConFeedback@blackboard.com.

The title of this session is:

Tuning  Blackboard  9.x  for  high  demanded  environments