real-life node.js troubleshooting - damian schenkelman, auth0
TRANSCRIPT
Real-Life Node.js TroubleshootingDamian Schenkelman, Auth0
4 years
Usual suspects
Memory Leaks
–Sebastian Peyrott (a co-worker)
“The main cause for leaks in garbage collected languages are unwanted references”
Memory Modelhttps://developers.google.com/web/tools/chrome-
devtools/memory-problems/memory-101
A
B
C
F
E
D
GGC
H I
b.d=null
A
B
C
F
E
D
GGC
H I
D not referenced
A
B
C
F
E
D
GGC
H I
D is GCed
A
B
C
F
E GGC
H I
Sawtooth
Memory Leak
A
B
C
F
E
D
G
Things we don’t need and are taking up a LOT of space
GC
Memory Leak
A
B
C
F
E
D
G
Things we don’t need and are taking up a LOT of space
GC
More things we don’t need
Crashes
Memory leak busting
Take Control
--max-old-space-sizeis your friend
Drain connections
Restarts
Heap Snapshot
v8-profilerhttps://github.com/node-inspector/v8-profiler
varprofiler=require('v8-profiler');
varsnapshot=profiler.takeSnapshot();
snapshot.export(function(error,result){fs.writeFileSync('snapshot.json',result);snapshot.delete();});
Demo
Mental Picture
{'kinesis.us-west-1.amazonaws.co:443':}
{sockets:}
[…,…,…,…,…,…,…,…,…,TLSSocket
]
Something related to logging…
Not Easyhttps://github.com/auth0/kinesis-writable/pull/6/files
forever-agenthttps://github.com/request/forever-agent/blob/
ece900a6e8dfac734186db3080c00875c0300450/index.js#L70
if(this.freeSockets[name]&&this.freeSockets[name].length>0&&!req.useChunkedEncodingByDefault){varidleSocket=this.freeSockets[name].pop()idleSocket.removeListener('error',idleSocket._onIdleError)deleteidleSocket._onIdleErrorreq._reusedSocket=truereq.onSocket(idleSocket)}else{this.addRequestNoreuse(req,host,port)}
Fixed
CPU bottlenecks
Auth Service
DB
Scenario
Client
Expected response times
Actual response times
Flame graphsa()
b()
c()
d()
h()
i()
e() f()
g()
Flame graphs
Demo
The problem
Scale
Faster Hash
Caching
Scale up
Multiple Auth Services
LB
Auth Service
Auth Service
Auth Service
Auth Service
BaaShttps://github.com/auth0/node-baas
//comparebcryptforreq.passwordtodbhashbaas.compare(req.body.password,user.passwordHash,(err,success)=>{res.send(err||!success?401:200);});
Auto Scaling
Auth Service LB
BaaS
BaaS
BaaS
BaaS
Cost comparison
Price / (1M req) #req per sec / vCPU
t2-micro $0.36 10.00
t2-medium $0.76 9.50
c4-large $1.53 10.00
c3-8xlarge $1.64 8.88
Fail gracefully
Cat picture
Memory Leaks• Memory terminology: https://developers.google.com/web/tools/
chrome-devtools/memory-problems/memory-101
• Memory leaks in JS: https://auth0.com/blog/four-types-of-leaks-in-your-javascript-code-and-how-to-get-rid-of-them/
• Roots: https://stackoverflow.com/questions/9748358/when-does-the-js-engine-create-a-garbage-collection-root
• Increase heap size: https://twitter.com/tjholowaychuk/status/480753206301966336
• v8 GC logs speaking to you: https://github.com/joyeecheung/v8-gc-talk
Memory Leaks (2)• Connection draining: https://cloud.google.com/
compute/docs/load-balancing/enabling-connection-draining
• master-process: https://github.com/jfromaniello/master-process
• Express connection draining: https://github.com/expressjs/express/issues/1366
• v8-profiler: https://github.com/node-inspector/v8-profiler
Memory Leaks (3)• AWS SDK issues: https://github.com/aws/aws-sdk-js/issues/329,
https://github.com/aws/aws-sdk-js/issues/855
• Kinesis event fix: https://github.com/auth0/kinesis-writable/pull/6/files
• Forever agent: https://github.com/request/forever-agent/blob/ece900a6e8dfac734186db3080c00875c0300450/index.js#L70
• Memory profiling for mere mortals: http://thlorenz.com/talks/memory-profiling/book/memory_profiling_for_mere_mortals_/memory_profiling_for_mere_mortals__0.html
• @thlorenz notes on mem profiling: https://github.com/thlorenz/v8-perf/blob/master/memory-profiling.md
Performance• http://security.stackexchange.com/a/83382
• http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html
• https://github.com/davidmarkclements/0x
• https://github.com/thlorenz/v8-perf/issues/4
• https://github.com/auth0/node-baas
• http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/US_SetUpASLBApp.html
• https://gist.github.com/trevnorris/9616784
• https://github.com/dschenkelman/bcrypt-sample
Thankshttps://github.com/dschenkelman/node-troubleshooting
@dschenkelman npmidschenkelman