big app design for node.js
DESCRIPTION
Designing the code of a large application in production and scaling up to thousands of users while doing it all in Node.js is a challenge that no developer should face alone. I will discuss how we have developed a highly decoupled, plugin-based architecture and a decentralized infrastructure for Cloud9 IDE, along with the technologies we've developed and the difficulties we faced in order to build the largest Node.js application that exists in production.TRANSCRIPT
![Page 1: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/1.jpg)
BIGapplication
designSaturday, November 10, 12
![Page 2: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/2.jpg)
@sergimansilla
Saturday, November 10, 12
![Page 4: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/4.jpg)
Smart code completion
Saturday, November 10, 12
![Page 5: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/5.jpg)
Static analysis
Saturday, November 10, 12
![Page 6: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/6.jpg)
Free linux VM
Saturday, November 10, 12
![Page 7: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/7.jpg)
Bring your own machineSaturday, November 10, 12
![Page 8: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/8.jpg)
Real terminalSaturday, November 10, 12
![Page 9: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/9.jpg)
Hundreds of thousands of lines of JavaScript
Saturday, November 10, 12
![Page 10: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/10.jpg)
dynamic,weakly-typed
language
Saturday, November 10, 12
![Page 11: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/11.jpg)
one single thread
Saturday, November 10, 12
![Page 12: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/12.jpg)
Remote VM
Cloud9 datacenter
IDE serverRemote
VM
Remote VM
Openshift
Saturday, November 10, 12
![Page 13: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/13.jpg)
Saturday, November 10, 12
![Page 14: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/14.jpg)
Saturday, November 10, 12
![Page 15: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/15.jpg)
Modularization
Saturday, November 10, 12
![Page 16: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/16.jpg)
Black box coding
Saturday, November 10, 12
![Page 17: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/17.jpg)
Saturday, November 10, 12
![Page 18: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/18.jpg)
Good at abstracting away
require
Saturday, November 10, 12
![Page 19: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/19.jpg)
Not that good at app-level abstraction
require
Saturday, November 10, 12
![Page 20: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/20.jpg)
duplicated modules
maps to folder names
Relies on FSrequ
ire
hard to configure module
Saturday, November 10, 12
![Page 21: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/21.jpg)
Dependency error handling
coding time compile time run time
KA-BO
OM!
- Server crash- Unhappy customers- Developer gets fired
Saturday, November 10, 12
![Page 22: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/22.jpg)
How to fix it?
Static dependency list
Named services
Easy configuration
Resolve at startup
No FS required
Pass an object
Saturday, November 10, 12
![Page 23: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/23.jpg)
Architect
github.com/c9/architect
Saturday, November 10, 12
![Page 24: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/24.jpg)
Everything is a plugin
Plugins can consume plugins
An application is just a set of plugins
Architect
Saturday, November 10, 12
![Page 25: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/25.jpg)
Dependency model
require
Architect
runtime
compile time
Saturday, November 10, 12
![Page 26: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/26.jpg)
Dependency error handlingArchitect
coding time compile time run time
- Happy customers- Developer keeps job- Fails before release
Saturday, November 10, 12
![Page 27: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/27.jpg)
github.com/sergi/nodejsconfit2012
Saturday, November 10, 12
![Page 28: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/28.jpg)
Saturday, November 10, 12
![Page 29: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/29.jpg)
Declare entity ‘cook’ with behavior
Use ‘cook’ to do some cooking
Saturday, November 10, 12
![Page 30: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/30.jpg)
Cook Cooking action
Dependency model
Saturday, November 10, 12
![Page 31: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/31.jpg)
package.json
Builds dependency tree
executing codewithout
Saturday, November 10, 12
![Page 32: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/32.jpg)
{{
cook/package.json
do-‐cooking/package.json
Saturday, November 10, 12
![Page 33: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/33.jpg)
Next
Extract the code
Wrap in Architect plugin code
Make two plugins
Saturday, November 10, 12
![Page 34: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/34.jpg)
Function signature
Call when done
Saturday, November 10, 12
![Page 35: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/35.jpg)
Architect plugin code
Module.exports
Options - we’ll talk about it later
Imports - everything you ‘consume’
Register - call when done
Saturday, November 10, 12
![Page 36: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/36.jpg)
Saturday, November 10, 12
![Page 37: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/37.jpg)
Saturday, November 10, 12
![Page 38: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/38.jpg)
Easy to testMock dependencies
Saturday, November 10, 12
![Page 39: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/39.jpg)
‘cook’ should be called 8 times
Saturday, November 10, 12
![Page 40: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/40.jpg)
No black magic
Specify dependency model
Feed architect a config file
Call ‘createApp’
Saturday, November 10, 12
![Page 41: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/41.jpg)
Saturday, November 10, 12
![Page 42: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/42.jpg)
Saturday, November 10, 12
![Page 43: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/43.jpg)
Configuration
Per-plugin options
No global options object
Specify in config file
Saturday, November 10, 12
![Page 44: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/44.jpg)
Saturday, November 10, 12
![Page 45: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/45.jpg)
Saturday, November 10, 12
![Page 46: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/46.jpg)
Options
Passed in at startup
Options can be dependencies
Fail if options aren’t present
Saturday, November 10, 12
![Page 47: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/47.jpg)
Saturday, November 10, 12
![Page 48: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/48.jpg)
Saturday, November 10, 12
![Page 49: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/49.jpg)
Your app as chunks of functionality
rather than sets of classes
Saturday, November 10, 12
![Page 50: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/50.jpg)
chunks of functionality
Implicit type constraints
Keep implementation private
Swap features instead of interfaces
Saturday, November 10, 12
![Page 51: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/51.jpg)
How does Cloud9 use it?
Open source version
Local version (OS + sync)
Hosted version
Normal
FTP
SSH
Saturday, November 10, 12
![Page 52: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/52.jpg)
Saturday, November 10, 12
![Page 53: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/53.jpg)
Swap feature per implementation
On OSS: talk local filesystem
On FTP: talk FTP library
On SSH: talk via a SSH bridge
Saturday, November 10, 12
![Page 54: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/54.jpg)
Architect is a DI framework
Saturday, November 10, 12
![Page 55: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/55.jpg)
But a quite special one
Saturday, November 10, 12
![Page 56: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/56.jpg)
IDE instance(FTP)
IDE instance(SSH)
IDE instance(Normal)
IDE instance(Normal)
Single node.js process
Other code (dashboard etc.)
Saturday, November 10, 12
![Page 57: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/57.jpg)
Architect can do
Multiple instances of same pluginRun independently
But in the same process
Saturday, November 10, 12
![Page 58: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/58.jpg)
Centralized event bus
Loose coupling between plugins
No hard dependencies!
Can also do inter-context communication
Saturday, November 10, 12
![Page 59: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/59.jpg)
Eventbus
Plugin
Other plugin
Emit event
React on event
Saturday, November 10, 12
![Page 60: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/60.jpg)
Saturday, November 10, 12
![Page 61: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/61.jpg)
Saturday, November 10, 12
![Page 62: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/62.jpg)
And now scale
Swap it with i.e. Redis PubSub
Plugins will never notice
Modular awesomeness!
Saturday, November 10, 12
![Page 63: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/63.jpg)
Modularize in feature blocks
Don’t over engineer
Don’t create too small blocks
They aren’t interfaces!
Saturday, November 10, 12
![Page 64: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/64.jpg)
Avoid context switching
Less code!
Less errors!
Less boilerplate!
Saturday, November 10, 12
![Page 65: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/65.jpg)
Loose coupling
Use (don’t abuse) an event bus
Smaller dependency graph
Saturday, November 10, 12
![Page 67: Big app design for Node.js](https://reader033.vdocuments.net/reader033/viewer/2022051613/54c855d54a7959b92b8b4573/html5/thumbnails/67.jpg)
Happy coding!
Saturday, November 10, 12