troubleshooting node.js
TRANSCRIPT
TROUBLESHOOTINGNODE.JS
what to do when things go wrong
Igor Soarez [email protected] @igorsoarez
Hassy Veldstra [email protected] @hveldstra
WDCNZ, July 2015
HELLO HELLO
• YLD!
• Node.js Consultancy & Training
• Digital transformation for enterprise clients
FIRESSomeone has to put them out
BE PREPARED
LOG EVERYTHINGMake it easy to grep & visualise
TEST TEST TEST
• Functional tests first
• Good coverage for unit tests
• Randomised testing & fuzz testing
MONITORING & ALERTING• Resource monitoring
• CPU
• Memory
• I/O
• Application monitoring
• Errors
• Real-user monitoring
• Latency
• Availability
AUTOMATED DEPLOYMENTS
PAIR PROGRAMMING
TAKE YOUR TIME WHEN BUILDING YOUR APPLICATION
Tight deadlines are the root of all evil
LOAD TESTING
LOAD TESTING
• You should do it
LOAD TESTING
• You should do it
• Find the bottlenecks
LOAD TESTING
• You should do it
• Find the bottlenecks
• Understand your system
LOAD TESTING
• You should do it
• Find the bottlenecks
• Understand your system
LOAD TESTING
LOAD TESTING
YOU SHOULD DO IT IN PRODUCTION
IN PRODUCTION???I must have heard you wrong
WHY NOT?Why — Reason 1
STAGING IS NEVER THE
SAME
Why — Reason 1I
GAIN EXTRA CONFIDENCEWhy — Reason 1II
IF YOU DO THIS• Analytics
IF YOU DO THIS• Analytics
• Custom header
• Cookie
IF YOU DO THIS• Analytics
• Custom header
• Cookie
• Let Marketing know
IF YOU DO THIS• Analytics
• Custom header
• Cookie
• Let Marketing know
• Start outside of peak times
IF YOU DO THIS• Analytics
• Custom header
• Cookie
• Let Marketing know
• Start outside of peak times
• Start slow & ramp up
LOAD TESTING TOOLS • JMeter
LOAD TESTING TOOLS • JMeter
• Tsung
LOAD TESTING TOOLS • JMeter
• Tsung
• Minigun - https://artillery.io/minigun
AFTER THE FIRE STARTS
how to approach it
PERFORMANCE PROBLEMS
ATTACK THE PROBLEM
1. Isolated test case *
2. Measure
3. Change / Tweak / Improve
4. Goto 2
* If possible
TYPES OF ISSUES
• CPU-bound code• Memory leaks• GC pressure• I/O performance• Event loop lag (Node.js)
KNOW THE TOOLS AVAILABLE
• OS-level• perf• sysstat, netstat, netmon, ps, top• sysdig
KNOW THE TOOLS AVAILABLE• Platform-specific (Node.js)
KNOW THE TOOLS AVAILABLE• Platform-specific (Node.js)
• V8 profiler• find hot-path functions that cannot be optimized• trace GC activity
KNOW THE TOOLS AVAILABLE• Platform-specific (Node.js)
• V8 profiler• find hot-path functions that cannot be optimized• trace GC activity
• heapdump• find memory leaks
MEMORY
• ps
• top / htop
• Heapdumps
KNOW THE TOOLS AVAILABLE• Platform-specific (Node.js)
• V8 profiler• find hot-path functions that cannot be optimized• trace GC activity
• heapdump• find memory leaks
• flamegraphs• see what's hot on the CPU
CPU
• ps
• top / htop
• Flamegraphs
KNOW THE TOOLS AVAILABLE• Platform-specific (Node.js)
• V8 profiler• find hot-path functions that cannot be optimized• trace GC activity
• heapdump• find memory leaks
• flamegraphs• see what's hot on the CPU
• node-toobusy-js• track the event loop lag
FLAMEGRAPHS
PERFaka Performance Counters
Collected stack frames
h()
c() h() c() c() z() c()
b() f() b() b() f() b() y() b() b()
a() a() a() m() a() m() x() m() a()
main() main() main() main() main() main() main() main() main()
Collapsed stack frames
h()
c() h() c() z()
b() f() b() y()
a() m() x()
main()
DEMO!
THANK YOU!
@hveldstra@igorsoarez