![Page 1: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/1.jpg)
Shubhra Kar | Products & Educationtwitter:@shubhrakar
Profiling & Memory Leak Diagnosis
nodejs @ hyper-scale
![Page 2: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/2.jpg)
About me
J2EE and SOA architect
Performance architect
Node, mBaaS & APIs
![Page 3: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/3.jpg)
These guys sent me !
Bert Belder
Ben Noordhuis
Node/io Core
RaymondFeng
Ritchie Martori
LoopBack & Express Core
SamRoberts
Miroslav Bajtos
Ryan Graham
![Page 4: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/4.jpg)
Latency demands are uncompromising
Web SaaS Mobile IoT
10
25
50
100100
50
1015
40
100
25
The new curve
Concurrent Users Latency Adoption
![Page 5: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/5.jpg)
That’s not your API, is it ?
![Page 6: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/6.jpg)
So how do we tune performance ?
![Page 7: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/7.jpg)
How does GC work in V8 – Almost the same !
Concept of reachability.Roots: Reachable or in live scope objectsInclude objects referenced from anywhere in the call stack (all local variables and parameters in the functions currently being invoked), and any global variables.Objects are kept in memory while accessible from roots through a reference or a chain of references.Root objects are pointed directly from V8 or the Web browser like DOM elementsGarbage collector identifies dead memory regions/unreachable objects through a chain of pointers from a live object; reallocates or releases them to the OS
![Page 8: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/8.jpg)
Easy right ? Hell No !!!
Pause and then Stop the World
V8 essentially:stops program execution when performing a garbage collection cycle.processes only part of the object heap in most garbage collection cycles. This minimizes the impact of stopping the application.Accurately keeps all objects and pointers in memory. This avoids falsely identifying objects as pointers which can result in memory leaks.In V8, the object heap is segmented into many parts; hence If an object is moved in a garbage collection cycle, V8 updates all pointers to the object.
![Page 9: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/9.jpg)
Short, Full GC and some algorithms to know
V8 divides the heap into two generations:
Short GC/scavengingObjects are allocated in “new-space” (between 1 and 8 MB). Allocation in new space is very cheap; increment an allocation pointer when we want to reserve space for a new object. When the pointer reaches the end of new space, a scavenge (minor garbage collection cycle) is triggered, quickly removing dead objects from new space.large space overhead, since we need physical memory backing both to-space and from-space. Fast by design, hence using for short GC cycles. Acceptable if new space is small – a few Mbs
Full GC/mark-sweep & mark-compactObjects which have survived two minor garbage collections are promoted to “old-space.” Old-space is garbage collected in full GC (major garbage collection cycle), which is much less frequent. A full GC cycle is triggered based on a memory thresholdTo collect old space, which may contain several hundred megabytes of data, we use two closely related algorithms, Mark-sweep and Mark-compact.
![Page 10: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/10.jpg)
New Algorithm implementation
Incremental marking & lazy sweeping
In mid-2012, Google introduced two improvements that reduced garbage collection pauses significantly: incremental marking and lazy sweeping.Incremental marking means being able to do a bit of marking work, then let the mutator (JavaScript program) run a bit, then do another bit of marking work. Short pauses in the order of 5-10 ms each as an example for marking. Threshold based. At every alloc, execution is paused to perform an incremental marking step.
Lazy sweep cleans up set of objects at time eventually cleaning all pages.
![Page 11: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/11.jpg)
OK…so how can I do heap analysis
![Page 12: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/12.jpg)
StrongLoop CLI
$ slc start
$ slc ctl
Service ID: 1Service Name: express-example-appEnvironment variables: No environment variables definedInstances: Version Agent version Cluster size 4.1.0 1.5.1 4Processes: ID PID WID Listening Ports Tracking objects? CPU profiling? 1.1.50320 50320 0 1.1.50321 50321 1 0.0.0.0:3001 1.1.50322 50322 2 0.0.0.0:3001 1.1.50323 50323 3 0.0.0.0:3001 1.1.50324 50324 4 0.0.0.0:3001
$ slc ctl heap-snapshot 1.1.1
![Page 13: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/13.jpg)
StrongLoop API
Programmatic heap snapshots (timer based)
Programmatic heap snapshots (threshold based)
var heapdump = require('heapdump') ... setInterval(function () { heapdump.writeSnapshot() }, 6000 * 30) <strong>(1)</strong>
var heapdump = require('heapdump')var nextMBThreshold = 0 <strong>(1)</strong>
setInterval(function () { var memMB = process.memoryUsage().rss / 1048576 <strong>(2)</strong> if (memMB > nextMBThreshold) { <strong>(3)</strong> heapdump.writeSnapshot() nextMBThreshold += 100 }}, 6000 * 2) <strong>(4)</strong>
![Page 14: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/14.jpg)
StrongLoop Arc
![Page 15: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/15.jpg)
StrongLoop Arc – Memory Analysis
![Page 16: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/16.jpg)
CPU hotspots & event loop blocks ?
![Page 17: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/17.jpg)
Don’t Block the EventLoop
Node.js Server
![Page 18: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/18.jpg)
StrongLoop CLI
$ slc start
$ slc ctl
Service ID: 1Service Name: express-example-appEnvironment variables: No environment variables definedInstances: Version Agent version Cluster size 4.1.0 1.5.1 4Processes: ID PID WID Listening Ports Tracking objects? CPU profiling? 1.1.50320 50320 0 1.1.50321 50321 1 0.0.0.0:3001 1.1.50322 50322 2 0.0.0.0:3001 1.1.50323 50323 3 0.0.0.0:3001 1.1.50324 50324 4 0.0.0.0:3001
$ slc ctl cpu-start 50320 $ slc ctl cpu-stop 50320
![Page 19: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/19.jpg)
CPU profiling in StrongLoop Arc
![Page 20: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/20.jpg)
The Upside down wedding cake
![Page 21: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/21.jpg)
Blocked event loop in Meteor atomosphere
node-fibers implements co-routines. Meteor uses this to hack local thread storage allowing V8 to run multiple execution contexts each mapped to a co-routine.
FindOrAllocatePerThreadDataForThisThread() used in switching context between co-routines
Co-routines are cooperative; the current coroutine has to yield control before another one can run and that is what Meteor does in its process.nextTick() callback; it essentially builds concurrent (but not parallel) green threads on a round-robin scheduler
Too many tiny tasks and not one long running one was blocking the event loop
process.nextTick() has a failsafe mechanism where it will process “x” tick callbacks before deferring the remaining ones to the next event loop tick.
Native MongoDB driver disabled the failsafe to silence a warning message in node v0.10 about maxtickDepth being reached
ticks parent name2274 7.3% v8::internal::Isolate::FindOrAllocatePerThreadDataForThisThread()1325 58.3% LazyCompile: ~<anonymous> packages/meteor.js:6831325 100.0% LazyCompile: _tickCallback node.js:399
![Page 22: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/22.jpg)
The fix
The workaround: switch fromprocess.nextTick() to setImmediate()
![Page 23: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/23.jpg)
StrongLoop Smart Profiling….Arc support coming soon
$ slc ctl cpu-start 1.1.76901 12
$ slc ctl -C http://my.remote.host cpu-start 1.1.76901 12
Local Linux host
Remote Linux host
• Sniff for event loop block• Trigger deep profile when blocking
encountered
![Page 24: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/24.jpg)
And finally the winner is…
![Page 25: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/25.jpg)
Deep Transaction Tracing
![Page 26: Node.js CPU Profiling and Memory Leak Detection with StrongLoop Arc](https://reader036.vdocuments.net/reader036/viewer/2022062406/55cb9bf9bb61eb200d8b4568/html5/thumbnails/26.jpg)
StrongLoop – node.js Development to Production
Build and Deploy
Automate Lifecycle
Performance MetricsReal-time production monitoring
ProfilerRoot cause
CPU & Memory
API ComposerVisual modeling
StrongLoop Arc
Process Manager
Scale applications
Q2201
5Mesh
Deploy containerize
d
ORM, mBaaS, Realtime