Observable Applications w/ Node.js

Yunong Xiao, Senior Node.js Engineer, UI Platform, Netflix

@yunongx, yunong@netflix.com

March 2015BayNode The Bay Area Node.js Meetup

Node.js @ Netflix

• Website www.netflix.com

• Discovery

• Playback

• Acquisition

• Account Management

• Internal Services

• TV/Mobile/Devices

What’s Wrong?

Increased Errors Increased Latency

Memory Leak

What Now?

“It is a capital mistake to theorize before one has DATA. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.”

Sherlock Holmes

-A Scandal in Bohemia

Data Data Data

Node.js Modules

vasync

• https://github.com/davepacheco/node-vasync

• Similar to caolan/async.

• Added observability.

async f() state

failed f()

successful f()pending f()

finished f()

# of errors

vasync

• Make API requests.

• Persist state to file system.

• Query database.

• Update caches.

Async workflow management is error prone

Example

cb() not invoked

Results

function three pending

How Do I See this in Prod?

• Logs

• Core Dumps

• REPL

• HTTP API

node-bunyan

• https://github.com/trentm/node-bunyan

• Streaming JSON logging library for JS. (node and browser)

Streaming JSON

• One JSON object per line. e.g. Twitter’s streaming API.

• Perfect for machine processing. i.e. works with Unix tools such as grep(1), cut(1), …

• Illegible for Humans

Streaming JSON

bunyan(1) CLI

Features

• Lightweight API

• Log levels: trace, debug, info, warn, error, fatal

• Extensible Streams interface.

• Custom object rendering with serializers.

• DTrace support.

Demo

DTrace

• Dynamic tracing framework.

• Available on Mac OS X (dev env). `man dtrace`

• Node.js DTrace USDT provider: https://github.com/chrisa/node-dtrace-provider

• Pure JS probes.

bunyan -p Demo

Production Tracing• Production environment is Ubuntu Linux on EC2.

• Future work: get probe access. Possibilities:

• SystemTap

• LTTng (Linux Tracing Toolkit Next Gen)

• perf_events markers (currently a proposed patch)

restify• http://restifyjs.com

• Production tested REST framework.

• Observability.

• Metrics.

• First class Bunyan integration.

• DTrace.

Vanilla App

Vanilla App

Logging

Request Capture Stream

• Tradeoffs, Perf vs Verbosity.

• Captures all log statements including trace in memory.

• Dump all logs only on error.

Audit Logs

Audit Logsstatus code request ID (UUID)URL

request headers

response headers

individual handler timers

req latency

Scoped Child Loggers

Logging

• Native Bunyan integration.

• Request capture stream.

• Audit logs with each req.

• Scoped child loggers with each req.

restify + Bunyan• Streaming JSON.

• Processing using Unix tools is easy.

• Helpful tools:

• Unix: cut(1), wc(1), grep(1), awk(1), perl(1), …

• JSON: https://github.com/trentm/json (npm install -g json)

• daggr: https://github.com/joyent/daggr (npm install -g daggr)

Examples

• Find all logs for a specific request.

• Count the # of non-200 responses.

• Show all requests that took longer than 200ms.

Advanced ExampleRequest latency distribution by URL

DTrace

Demo

distributed processing

• elasticsearch • spark • hive

bunyanrestify

audit logs

req id req headers

req latency

handler latenciesURL

res codeErrors

res headers

Processing w/ Unix tools

• grep(1) • awk(1) • cut(1) • wc(1) • sort(1) • json(1) • bunyan(1) • …

req scoped logs application specific

context

req capture stream dumps all logs on error

DTrace*req latencyhandler

latencies

* Where available

restify metrics

Increased Errors Increased Latency

restifyInterested? Call for contributors.

restifyjs.com

The Killer Combo• vasync: https://github.com/davepacheco/node-vasync

• Observable async operations.

• bunyan: https://github.com/trentm/node-bunyan

• Streaming JSON logs.

• restify: restifyjs.com

• Observable REST applications.

• Unix Tools

• Stream of text model. Many tools to manipulate JSON logs.

Data Data Data

Thanks

• Questions?

• yunong@netflix.com

• @yunongx

• http://restifyjs.com

March 2015BayNode The Bay Area Node.js Meetup