debugging ruby systems
TRANSCRIPT
![Page 1: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/1.jpg)
DEBUGGING RUBYAman Gupta
@tmm1
http://tinyurl.com/eyfast
![Page 2: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/2.jpg)
AS RUBYISTS, WE'VE SEEN...
![Page 3: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/3.jpg)
ejpphoto (flickr)
nasty bugs
![Page 4: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/4.jpg)
codefatboyke (flickr)
![Page 5: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/5.jpg)
37prime (flickr)
memory bloat
![Page 6: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/6.jpg)
THIS TALK IS ABOUT...
TOOLS TO FIX THESE ISSUES.
![Page 7: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/7.jpg)
TOOLS FOR LINUX.
pleeker (flickr)
lsofstraceltrace
![Page 8: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/8.jpg)
TOOLS FOR C CODE.
booddin (flickr)
perftoolsgdb
![Page 9: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/9.jpg)
pascal.charest (flickr)
TOOLS FOR NETWORKS.
tcpdumpngrep
![Page 10: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/10.jpg)
marksze (flickr)
TOOLS FOR CPU USAGE.
perftoolsperftools.rb
![Page 11: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/11.jpg)
kgrocki (flickr)
TOOLS FOR MEMORY USAGE.
bleak_housegdb.rb
memprof
![Page 12: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/12.jpg)
mayu (flickr) delgrossodotcom (flickr)
IGNORE THE FINE PRINT
![Page 13: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/13.jpg)
list open filesLSOF
lsof -nPp <pid>
![Page 14: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/14.jpg)
lsof -nPp <pid>-nInhibits the conversion of network numbers to host names.
-PInhibits the conversion of port numbers to names for network files
FD TYPE NAMEcwd DIR /var/www/myapptxt REG /usr/bin/rubymem REG /json-1.1.9/ext/json/ext/generator.somem REG /json-1.1.9/ext/json/ext/parser.somem REG /memcached-0.17.4/lib/rlibmemcached.somem REG /mysql-2.8.1/lib/mysql_api.so 0u CHR /dev/null 1w REG /usr/local/nginx/logs/error.log 2w REG /usr/local/nginx/logs/error.log 3u IPv4 10.8.85.66:33326->10.8.85.68:3306 (ESTABLISHED) 10u IPv4 10.8.85.66:33327->10.8.85.68:3306 (ESTABLISHED) 11u IPv4 127.0.0.1:58273->127.0.0.1:11211 (ESTABLISHED) 12u REG /tmp/RackMultipart.28957.0 33u IPv4 174.36.83.42:37466->69.63.180.21:80 (ESTABLISHED)
jsonmemcached
mysqlhttp
![Page 15: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/15.jpg)
trace system calls and signalsSTRACE
strace -cp <pid>strace -ttTp <pid> -o <file>
![Page 16: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/16.jpg)
strace -cp <pid>-cCount time, calls, and errors for each system call and report a summary on program exit.
-p pidAttach to the process with the process ID pid and begin tracing.
% time seconds usecs/call calls errors syscall------ ----------- ----------- --------- --------- ---------------- 50.39 0.000064 0 1197 592 read 34.65 0.000044 0 609 writev 14.96 0.000019 0 1226 epoll_ctl 0.00 0.000000 0 4 close 0.00 0.000000 0 1 select 0.00 0.000000 0 4 socket 0.00 0.000000 0 4 4 connect 0.00 0.000000 0 1057 epoll_wait------ ----------- ----------- --------- --------- ----------------100.00 0.000127 4134 596 total
![Page 17: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/17.jpg)
strace -ttTp <pid> -o <file>-tPrefix each line of the trace with the time of day.
-ttIf given twice, the time printed will include the microseconds.
-TShow the time spent in system calls.
-o filenameWrite the trace output to the file filename rather than to stderr.
epoll_wait(9, {{EPOLLIN, {u32=68841296, u64=68841296}}}, 4096, 50) = 1 <0.033109>accept(10, {sin_port=38313, sin_addr="127.0.0.1"}, [1226]) = 22 <0.000014>fcntl(22, F_GETFL) = 0x2 (flags O_RDWR) <0.000007>fcntl(22, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000008>setsockopt(22, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000008>accept(10, 0x7fff5d9c07d0, [1226]) = -1 EAGAIN <0.000014>epoll_ctl(9, EPOLL_CTL_ADD, 22, {EPOLLIN, {u32=108750368, u64=108750368}}) = 0 <0.000009>epoll_wait(9, {{EPOLLIN, {u32=108750368, u64=108750368}}}, 4096, 50) = 1 <0.000007>read(22, "GET / HTTP/1.1\r"..., 16384) = 772 <0.000012>rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 <0.000007>poll([{fd=5, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout) <0.000008>write(5, "1\0\0\0\0\0\0-\0\0\0\3SELECT * FROM `table`"..., 56) = 56 <0.000023>read(5, "\25\1\0\1,\2\0x\234m"..., 16384) = 284 <1.300897>
![Page 18: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/18.jpg)
read(22, "GET / HTTP/1.1\r"..., 16384) = 772 <0.0012>
http client connection
read 772 bytes
incoming http request
took 0.0012s
![Page 19: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/19.jpg)
write(5, "SELECT * FROM `table`"..., 56) = 56 <0.0023>read(5, "\25\1\0\1,\2\0x\234m"..., 16384) = 284 <1.30>
mysql connectionwrite sql query to db
read query response
slow query
![Page 20: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/20.jpg)
stracing ruby: SIGVTALRM--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---rt_sigreturn(0x1a) = 2207807 <0.000009>--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---rt_sigreturn(0x1a) = 0 <0.000009>--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---rt_sigreturn(0x1a) = 140734552062624 <0.000009>--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---rt_sigreturn(0x1a) = 140734552066688 <0.000009>--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---rt_sigreturn(0x1a) = 11333952 <0.000008>--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---rt_sigreturn(0x1a) = 0 <0.000009>--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---rt_sigreturn(0x1a) = 1 <0.000010>--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
• ruby 1.8 uses signals to schedule its green threads• process receives a SIGVTALRM signal every 10ms
![Page 21: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/21.jpg)
stracing ruby: sigprocmask
• debian/redhat compile ruby with --enable-pthread• uses a native thread timer for SIGVTALRM• causes excessive calls to sigprocmask: 30% slowdown!
% time seconds usecs/call calls errors syscall------ ----------- ----------- --------- --------- ----------------100.00 0.326334 0 3568567 rt_sigprocmask 0.00 0.000000 0 9 read 0.00 0.000000 0 10 open 0.00 0.000000 0 10 close 0.00 0.000000 0 9 fstat 0.00 0.000000 0 25 mmap------ ----------- ----------- --------- --------- ----------------100.00 0.326334 3568685 0 total
![Page 22: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/22.jpg)
dump traffic on a networkTCPDUMP
tcpdump -i eth0 -s 0 -nqAtcp dst port 3306
![Page 23: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/23.jpg)
tcpdump -i <eth> -s <len> -nqA <expr>
-i <eth>Network interface.
-s <len>Snarf len bytes of data from each packet.
-nDon't convert addresses (host addresses, port numbers) to names.
-qQuiet output. Print less protocol information.
-APrint each packet (minus its link level header) in ASCII.
-w <file>Write the raw packets to file rather than printing them out.
<expr>libpcap expression, for example: tcp src port 80 tcp dst port 3306
tcpdump -i <eth> -w <file> <expr>
![Page 24: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/24.jpg)
tcp dst port 8019:52:20.216294 IP 24.203.197.27.40105 > 174.37.48.236.80: tcp 438E...*[email protected].%&.....%0....POx..%s.oP.......GET /poll_images/cld99erh0/logo.png HTTP/1.1Accept: */*Referer: http://apps.facebook.com/realpolls/?_fb_q=1
![Page 25: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/25.jpg)
tcp dst port 330619:51:06.501632 IP 10.8.85.66.50443 > 10.8.85.68.3306: tcp 98E..."K@[email protected]..[......W....SELECT * FROM `votes` WHERE (`poll_id` = 72621) LIMIT 1
![Page 26: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/26.jpg)
tcpdump -w <file>
![Page 27: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/27.jpg)
google's performance toolsPERFTOOLS
CPUPROFILE=/tmp/myprof ./myapppprof ./myapp /tmp/myprof
![Page 28: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/28.jpg)
wget http://google-perftools.googlecode.com/files/google-perftools-1.6.tar.gztar zxvf google-perftools-1.6.tar.gzcd google-perftools-1.6
./configure --prefix=/optmakesudo make install
# for linuxexport LD_PRELOAD=/opt/lib/libprofiler.so
# for osxexport DYLD_INSERT_LIBRARIES=/opt/lib/libprofiler.dylib
CPUPROFILE=/tmp/ruby.prof ruby -e' 5_000_000.times{ "hello world" }'
pprof `which ruby` --text /tmp/ruby.prof
download
compile
profile
report
setup
![Page 29: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/29.jpg)
Total: 103 samples 95 92.2% rb_yield_0 103 100.0% rb_eval 12 11.7% gc_sweep 52 50.5% rb_str_new3 3 2.9% obj_free 103 100.0% int_dotimes 12 11.7% gc_mark
pprof ruby ruby.prof --text
pprof ruby ruby.prof --gif
![Page 30: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/30.jpg)
Profiling MRI• 10% of production VM time spent in rb_str_sub_bang• String#sub!• called from Time.parse
return unless str.sub!(/\A(\d{1,2})/, '')return unless str.sub!(/\A( \d|\d{1,2})/, '')return unless str.sub!(/\A( \d|\d{1,2})/, '')return unless str.sub!(/\A(\d{1,3})/, '')return unless str.sub!(/\A(\d{1,2})/, '')return unless str.sub!(/\A(\d{1,2})/, '')
![Page 31: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/31.jpg)
Profiling EM + threads
• known issue: EM+threads = slow• memcpy??• thread context switches copy the stack w/ memcpy• EM allocates huge buffer on the stack• solution: move buffer to the heap
Total: 3763 samples 2764 73.5% catch_timer 989 26.3% memcpy 3 0.1% st_lookup 2 0.1% rb_thread_schedule 1 0.0% rb_eval 1 0.0% rb_newobj 1 0.0% rb_gc_force_recycle
![Page 32: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/32.jpg)
perftools for ruby codePERFTOOLS.RB
pprof.rb /tmp/myrbprof
github.com/tmm1/perftools.rb
![Page 33: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/33.jpg)
gem install perftools.rb
RUBYOPT="-r`gem which perftools | tail -1`"CPUPROFILE=/tmp/myrbprofruby myapp.rb
pprof.rb /tmp/myrbprof --textpprof.rb /tmp/myrbprof --gif > /tmp/myrbprof.gif
![Page 34: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/34.jpg)
• Sampling profiler:
• 232 samples total
• 83 samples were in /compute
• 118 samples had /compute on the stack but were in another function
• /compute accounts for 50% of process, but only 35% of time was in /compute itself
require 'sinatra'
get '/sleep' do sleep 0.25 'done'end
get '/compute' do proc{ |n| a,b=0,1 n.times{ a,b = b,a+b } b }.call(10_000) 'done'end
$ ab -c 1 -n 50 http://127.0.0.1:4567/compute$ ab -c 1 -n 50 http://127.0.0.1:4567/sleep
== Sinatra has ended his set (crowd applauds)PROFILE: interrupts/evictions/bytes = 232/0/2152
Total: 232 samples 83 35.8% 35.8% 118 50.9% Sinatra::Application#GET /compute 56 24.1% 59.9% 56 24.1% garbage_collector 35 15.1% 75.0% 113 48.7% Integer#times
![Page 35: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/35.jpg)
CPUPROFILE=app.profCPUPROFILE_REALTIME=1CPUPROFILE=app-rt.prof
![Page 36: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/36.jpg)
redis-rb bottleneck
![Page 37: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/37.jpg)
why is rubygems slow?
![Page 38: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/38.jpg)
faster bundle install
• 23% spent in Gem::Version#<=>
• simple patch to rubygems improved overall install performance by 15%
• http://gist.github.com/458185
![Page 39: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/39.jpg)
CPUPROFILE_OBJECTS=1CPUPROFILE=app-objs.prof
• object allocation profiler mode built-in
• 1 sample = 1 object created
• Time parsing is bothCPU and objectallocation intensive
• using mysql2 movesthis to C
![Page 40: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/40.jpg)
trace library callsLTRACE
ltrace -cp <pid>ltrace -ttTp <pid> -o <file>
![Page 41: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/41.jpg)
% time seconds usecs/call calls function------ ----------- ----------- --------- -------------------- 48.65 11.741295 617 19009 memcpy 30.16 7.279634 831 8751 longjmp 9.78 2.359889 135 17357 _setjmp 8.91 2.150565 285 7540 malloc 1.10 0.265946 20 13021 memset 0.81 0.195272 19 10105 __ctype_b_loc 0.35 0.084575 19 4361 strcmp 0.19 0.046163 19 2377 strlen 0.03 0.006272 23 265 realloc------ ----------- ----------- --------- --------------------100.00 24.134999 82999 total
ltrace -c ruby threaded_em.rb
01:24:48.769408 --- SIGVTALRM (Virtual timer expired) ---01:24:48.769616 memcpy(0x1216000, "", 1086328) = 0x1216000 <0.000578>01:24:48.770555 memcpy(0x6e32670, "\240&\343v", 1086328) = 0x6e32670 <0.000418>
01:24:49.899414 --- SIGVTALRM (Virtual timer expired) ---01:24:49.899490 memcpy(0x1320000, "", 1082584) = 0x1320000 <0.000628>01:24:49.900474 memcpy(0x6e32670, "", 1086328) = 0x6e32670 <0.000479>
ltrace -ttT -e memcpy ruby threaded_em.rb
![Page 42: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/42.jpg)
trace dlopen’d library callsLTRACE/LIBDL
ltrace -F <conf> -bg -x <symbol> -p <pid>
github.com/ice799/ltrace/tree/libdl
![Page 43: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/43.jpg)
ltrace -F <conf> -b -g -x <sym>-bIgnore signals.
-gIgnore libraries linked at compile time.
-F <conf>Read prototypes from config file.
-x <sym>Trace calls to the function sym.
-s <num>Show first num bytes of string args.
-F ltrace.confint mysql_real_query(addr,string,ulong); void garbage_collect(void);int memcached_set(addr,string,ulong,string,ulong);
![Page 44: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/44.jpg)
ltrace -x garbage_collect
19:08:06.436926 garbage_collect() = <void> <0.221679>19:08:15.329311 garbage_collect() = <void> <0.187546>19:08:17.662149 garbage_collect() = <void> <0.199200>19:08:20.486655 garbage_collect() = <void> <0.205864>19:08:25.102302 garbage_collect() = <void> <0.214295>19:08:35.552337 garbage_collect() = <void> <0.189172>
![Page 45: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/45.jpg)
ltrace -x mysql_real_querymysql_real_query(0x1c9e0500, "SET NAMES 'UTF8'", 16) = 0 <0.000324>mysql_real_query(0x1c9e0500, "SET SQL_AUTO_IS_NULL=0", 22) = 0 <0.000322>mysql_real_query(0x19c7a500, "SELECT * FROM `users`", 21) = 0 <1.206506>mysql_real_query(0x1c9e0500, "COMMIT", 6) = 0 <0.000181>
![Page 46: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/46.jpg)
ltrace -x memcached_setmemcached_set(0x15d46b80, "Status:33", 21, "\004\b", 366) = 0 <0.01116>memcached_set(0x15d46b80, "Status:96", 21, "\004\b", 333) = 0 <0.00224>memcached_set(0x15d46b80, "Status:57", 21, "\004\b", 298) = 0 <0.01850>memcached_set(0x15d46b80, "Status:10", 21, "\004\b", 302) = 0 <0.00530>memcached_set(0x15d46b80, "Status:67", 21, "\004\b", 318) = 0 <0.00291>memcached_set(0x15d46b80, "Status:02", 21, "\004\b", 299) = 0 <0.00658>memcached_set(0x15d46b80, "Status:34", 21, "\004\b", 264) = 0 <0.00243>
![Page 47: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/47.jpg)
the GNU debuggerGDB
gdb <executable>gdb attach <pid>
![Page 48: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/48.jpg)
Debugging Ruby Segfaults
#include "ruby.h"
VALUEsegv(){ VALUE array[1]; array[1000000] = NULL; return Qnil;}
voidInit_segv(){ rb_define_method(rb_cObject, "segv", segv, 0);}
test_segv.rb:4: [BUG] Segmentation faultruby 1.8.7 (2008-08-11 patchlevel 72) [i686-darwin9.7.0]
def test require 'segv' 4.times do Dir.chdir '/tmp' do Hash.new{ segv }[0] end endend
sleep 10test()
![Page 49: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/49.jpg)
1. Attach to running process
$ sudo gdb ruby 23611Attaching to program: ruby, process 236110x00007fa5113c0c93 in nanosleep () from /lib/libc.so.6(gdb) cContinuing.
Program received signal SIGBUS, Bus error.segv () at segv.c:77 array[1000000] = NULL;
$ ps aux | grep rubyjoe 23611 0.0 0.1 25424 7540 S Dec01 0:00 ruby test_segv.rb
2. Use a coredump
$ sudo mkdir /cores$ sudo chmod 777 /cores$ sudo sysctl kernel.core_pattern=/cores/%e.core.%s.%p.%t
Process.setrlimit Process::RLIMIT_CORE, 300*1024*1024
$ sudo gdb ruby /cores/ruby.core.6.23611.1259781224
![Page 50: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/50.jpg)
def test require 'segv' 4.times do Dir.chdir '/tmp' do Hash.new{ segv }[0] end endend
test()
(gdb) where#0 segv () at segv.c:7#1 0x000000000041f2be in call_cfunc () at eval.c:5727...#13 0x000000000043ba8c in rb_hash_default () at hash.c:521...#19 0x000000000043b92a in rb_hash_aref () at hash.c:429...#26 0x00000000004bb7bc in chdir_yield () at dir.c:728#27 0x000000000041d8d7 in rb_ensure () at eval.c:5528#28 0x00000000004bb93a in dir_s_chdir () at dir.c:816...#35 0x000000000041c444 in rb_yield () at eval.c:5142#36 0x0000000000450690 in int_dotimes () at numeric.c:2834...#48 0x0000000000412a90 in ruby_run () at eval.c:1678#49 0x000000000041014e in main () at main.c:48
![Page 51: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/51.jpg)
gdb with MRI hooksGDB.RB
gem install gdb.rbgdb.rb <pid>
github.com/tmm1/gdb.rb
![Page 52: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/52.jpg)
(gdb) ruby eval 1+23
(gdb) ruby eval Thread.current#<Thread:0x1d630 run>
(gdb) ruby eval Thread.list.size8
![Page 53: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/53.jpg)
(gdb) ruby threads list0x15890 main thread THREAD_STOPPED WAIT_JOIN(0x19ef4)0x19ef4 thread THREAD_STOPPED WAIT_TIME(57.10s) 0x19e34 thread THREAD_STOPPED WAIT_FD(5) 0x19dc4 thread THREAD_STOPPED WAIT_NONE 0x19dc8 thread THREAD_STOPPED WAIT_NONE 0x19dcc thread THREAD_STOPPED WAIT_NONE 0x22668 thread THREAD_STOPPED WAIT_NONE 0x1d630 curr thread THREAD_RUNNABLE WAIT_NONE
![Page 54: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/54.jpg)
(gdb) ruby objects HEAPS 8 SLOTS 1686252 LIVE 893327 (52.98%) FREE 792925 (47.02%)
scope 1641 (0.18%) regexp 2255 (0.25%) data 3539 (0.40%) class 3680 (0.41%) hash 6196 (0.69%) object 8785 (0.98%) array 13850 (1.55%) string 105350 (11.79%) node 742346 (83.10%)
![Page 55: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/55.jpg)
(gdb) ruby objects strings 140 u'lib' 158 u'0' 294 u'\n' 619 u''
30503 unique strings 3187435 bytes
![Page 56: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/56.jpg)
def test require 'segv' 4.times do Dir.chdir '/tmp' do Hash.new{ segv }[0] end endend
test()(gdb) ruby threads
0xa3e000 main curr thread THREAD_RUNNABLE WAIT_NONE node_vcall segv in test_segv.rb:5 node_call test in test_segv.rb:5 node_call call in test_segv.rb:5 node_call default in test_segv.rb:5 node_call [] in test_segv.rb:5 node_call test in test_segv.rb:4 node_call chdir in test_segv.rb:4 node_call test in test_segv.rb:3 node_call times in test_segv.rb:3 node_vcall test in test_segv.rb:9
![Page 57: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/57.jpg)
rails_warden leak(gdb) ruby objects classes 1197 MIME::Type 2657 NewRelic::MetricSpec 2719 TZInfo::TimezoneTransitionInfo 4124 Warden::Manager 4124 MethodOverrideForAll 4124 AccountMiddleware 4124 Rack::Cookies 4125 ActiveRecord::ConnectionAdapters::ConnectionManagement 4125 ActionController::Session::CookieStore 4125 ActionController::Failsafe 4125 ActionController::ParamsParser 4125 Rack::Lock 4125 ActionController::Dispatcher 4125 ActiveRecord::QueryCache 4125 ActiveSupport::MessageVerifier 4125 Rack::Head
middleware chain leaking per request
![Page 58: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/58.jpg)
mongrel sleeper thread 0x16814c00 thread THREAD_STOPPED WAIT_TIME(0.47) 1522 bytes node_fcall sleep in lib/mongrel/configurator.rb:285 node_fcall run in lib/mongrel/configurator.rb:285 node_fcall loop in lib/mongrel/configurator.rb:285 node_call run in lib/mongrel/configurator.rb:285 node_call initialize in lib/mongrel/configurator.rb:285 node_call new in lib/mongrel/configurator.rb:285 node_call run in bin/mongrel_rails:128 node_call run in lib/mongrel/command.rb:212 node_call run in bin/mongrel_rails:281 node_fcall (unknown) in bin/mongrel_rails:19
def run @listeners.each {|name,s| s.run }
$mongrel_sleeper_thread = Thread.new { loop { sleep 1 } }end
![Page 59: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/59.jpg)
god memory leaks(gdb) ruby objects arrays elements instances 94310 3 94311 3 94314 2 94316 1 5369 arrays 2863364 member elements
many arrays with 90k+ elements!
5 separate god leaks fixed by Eric Lindvall with the help of gdb.rb!
43 God::Process 43 God::Watch 43 God::Driver 43 God::DriverEventQueue 43 God::Conditions::MemoryUsage 43 God::Conditions::ProcessRunning 43 God::Behaviors::CleanPidFile 45 Process::Status 86 God::Metric327 God::System::SlashProcPoller327 God::System::Process406 God::DriverEvent
![Page 60: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/60.jpg)
a heap visualizer for rubyMEMPROF
gem install memprofopen http://memprof.com
github.com/ice799/memprof
![Page 61: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/61.jpg)
•memprof.track
•memprof.dump
•memprof.dump_all
•memprof.com
memprof features
![Page 62: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/62.jpg)
Memprof.track{ 100.times{ "abc" } 100.times{ 1.23 + 1 } 100.times{ Module.new }}
100 file.rb:2:String100 file.rb:3:Float100 file.rb:4:Module
• like bleak_house, but for a given block of code
• use Memprof::Middleware in your webapps to run track per request
![Page 63: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/63.jpg)
•memprof.track
•memprof.dump
•memprof.dump_all
•memprof.com
memprof features
![Page 64: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/64.jpg)
{ "_id": "0x19c610",
"file": "file.rb", "line": 2,
"type": "string", "class": "0x1ba7f0", "class_name": "String",
"length": 10, "data": "helloworld"}
memory address of object
file and line where string was created
length and contentsof this string instance
address of the class “String”
stringsMemprof.dump{ "hello" + "world"}
![Page 65: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/65.jpg)
floats and strings are separate ruby objects
{ "_id": "0x19c5c0",
"class": "0x1b0d18", "class_name": "Array",
"length": 4, "data": [ 1, ":b",
"0x19c750", "0x19c598" ]}
integers and symbols are stored in the array itself
arraysMemprof.dump{ [ 1, :b, 2.2, "d" ]}
![Page 66: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/66.jpg)
hashes{ "_id": "0x19c598",
"type": "hash", "class": "0x1af170", "class_name": "Hash",
"default": null,
"length": 2, "data": [ [ ":a", 1 ], [ "0xc728", "0xc750" ] ]}
hash entries as key/value pairs
no default proc
Memprof.dump{ { :a => 1, "b" => 2.2 }}
![Page 67: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/67.jpg)
classesMemprof.dump{ class Hello @@var=1 Const=2 def world() end end}
{ "_id": "0x19c408",
"type": "class", "name": "Hello", "super": "0x1bfa48", "super_name": "Object",
"ivars": { "@@var": 1, "Const": 2 }, "methods": { "world": "0x19c318" }}
class variables and constants are stored in the instance variable table
superclass object reference
references to method objects
![Page 68: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/68.jpg)
•memprof.track
•memprof.dump
•memprof.dump_all
•memprof.com
memprof features
![Page 69: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/69.jpg)
Memprof.dump_all("myapp_heap.json")• dump out every single live object as json
• one per line to specified file
• analyze via
• jsawk/grep
•mongodb/couchdb
• custom ruby scripts
• libyajl + Boost Graph Library
![Page 70: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/70.jpg)
•memprof.track
•memprof.dump
•memprof.dump_all
•memprof.com
memprof features
![Page 71: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/71.jpg)
a web-based heap visualizer and leak analyzermemprof.com
![Page 72: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/72.jpg)
plugging a leak in rails3• in dev mode, rails3 is leaking 10mb per request
# in environment.rbrequire `gem which memprof/signal`.strip
let’s use memprof to find it!
![Page 73: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/73.jpg)
plugging a leak in rails3
tell memprof to dump out the entire heap to json
$ memprof --pid <pid> --name <dump name> --key <api key>
send the app some requests so it leaks
$ ab -c 1 -n 30 http://localhost:3000/
![Page 74: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/74.jpg)
2519 classes
30 copies of TestController
mongo query for all TestController classes
details for one copy of TestController
![Page 75: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/75.jpg)
find references to object
holding references to all controllers
“leak” is on line 178
![Page 76: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/76.jpg)
• In development mode, Rails reloads all your application code on every request
• ActionView::Partials::PartialRenderer is caching partials used by each controller as an optimization
• But.. it ends up holding a reference to every single reloaded version of those controllers
![Page 77: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/77.jpg)
MORE* MEMPROF FEATURES
•memprof.trace
•memprof::tracer
* currently under development
![Page 78: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/78.jpg)
config.middleware.use(Memprof::Tracer)
{ "time": 4.3442,
"rails": { "controller": "test", "action": "index" },
"request": { "REQUEST_PATH": "/test", "REQUEST_METHOD": "GET" },
total time for request
rails controller/action
request env info
![Page 79: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/79.jpg)
"mysql": { "queries": 3, "time": 0.00109302 },
"gc": { "calls": 8, "time": 2.04925 },
config.middleware.use(Memprof::Tracer)
8 calls to GC2 secs spent in GC
3 mysql queries
config.middleware.use(Memprof::Tracer)
![Page 80: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/80.jpg)
"objects": { "created": 3911103, "types": { "none": 1168831, "object": 1127, "float": 627, "string": 1334637, "array": 609313, "hash": 3676, "match": 70211 } }}
config.middleware.use(Memprof::Tracer)
3 million objs created
lots of stringslots of arrays
regexp matches
object instances1 million method calls
config.middleware.use(Memprof::Tracer)
![Page 81: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/81.jpg)
rack middleware for perftools.rbRACK-PERFTOOLS
gem install rack-perftools_profiler
github.com/bhb/rack-perftools_profiler
![Page 82: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/82.jpg)
require 'rack/perftools_profiler'config.middleware.insert( 0, Rack::PerftoolsProfiler, :default_printer => 'gif')
$ curl http://localhost:3000/__start__$ curl http://localhost:3000/home$ curl http://localhost:3000/about$ curl http://localhost:3000/__stop__
$ curl http://localhost:3000/__data__ -o profile.gif$ curl http://localhost:3000/__data__?printer=text -o profile.txt
$ curl "http://localhost:3000/home?profile=true×=10" -o 10_requests_to_homepage.gif
![Page 83: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/83.jpg)
TOOLS MAKE DEBUGGING EASIER.
![Page 84: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/84.jpg)
LEARN THESE TOOLS.USE THESE TOOLS.
yujie (flickr)
![Page 85: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/85.jpg)
EY CASE STUDY• Example of Professional Services engagement
• Interested? http://tinyurl.com/eyfast
http://tinyurl.com/eyfast
![Page 86: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/86.jpg)
LSOF
15u IPv4 TCP ->10.243.63.80:11211 (ESTABLISHED)20u IPv4 TCP ->10.243.63.80:11211 (ESTABLISHED)23u IPv4 TCP ->10.243.63.80:11211 (ESTABLISHED)
18r DIR /shared/public/javascripts/packaged19r DIR /shared/public/javascripts/packaged22r DIR /shared/public/javascripts/packaged
multiple connections to memcached via different drivers
multiple open handles to javascript assets directories
![Page 87: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/87.jpg)
STRACE% time seconds calls syscall------ ----------- --------- --------- 26.28 0.054178 8731 read 25.81 0.053216 316519 stat 20.37 0.041993 1 clone 15.83 0.032648 11034 getdents 3.54 0.007309 10326 write
40% of kernel time querying filesystem
![Page 88: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/88.jpg)
LTRACE
mysql_query("SELECT * FROM tags WHERE id = 9129")mysql_query("SELECT * FROM tags WHERE id = 9129")mysql_query("SELECT * FROM tag_info WHERE tag_id = 9129")mysql_query("SELECT * FROM tags WHERE id = 9129")mysql_query("SELECT * FROM tags WHERE id = 9129")mysql_query("SELECT * FROM tags WHERE id = 9129")
common queries repeated multiple times during a request
![Page 89: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/89.jpg)
GDB.RB(gdb) ruby objects
HEAPS 10 SLOTS 4450607 LIVE 2006769 (45.09%) FREE 2443838 (54.91%)
hash 12939 (0.64%) array 38007 (1.89%) object 47633 (2.37%) string 234205 (11.67%) node 1637926 (81.62%)
large ruby heap
numerous 'node' objects- required libraries and plugins that are no longer in use
![Page 90: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/90.jpg)
PERFTOOLS.RB
majority of ruby process' time on CPU spent in garbage collection
![Page 91: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/91.jpg)
MEMPROF
wide distribution of response times
expensive controller actions creating many millions of objects per request
![Page 92: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/92.jpg)
MEMPROF
37% of response time attributed to garbage collection
![Page 93: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/93.jpg)
Time taken for tests: 20.156 secondsComplete requests: 100Requests per second: 4.96 [#/sec] (mean)Time per request: 201.555 [ms] (mean)
Time taken for tests: 8.100 secondsComplete requests: 100Requests per second: 12.35 [#/sec] (mean)Time per request: 81.000 [ms] (mean)
BEFORE
AFTER
2.5x improvement in throughput after addressing some of these issues and tuning GC
![Page 94: Debugging Ruby Systems](https://reader034.vdocuments.net/reader034/viewer/2022051400/55a55dfa1a28ab42788b4767/html5/thumbnails/94.jpg)
QUESTIONS?Aman Gupta
@tmm1
http://scr.bi/debuggingruby