Alex Liu

A N E T F L I X E N G I N E E R I N G O R I G I N A L

@stinkydofualiu@netflix.com

codex

conditionaL ModuleS Strike bacK

#netflixeverywhere

contentsI. The Problem

II. Codex: Conditional Bundling

III. Scaling for Netflix

IV. Looking to the Future

i. the Problem

It's all about building JS bundles.

// bundle.js var a = require('a'); var b = require('b');

bundle.js

// bundle.js var a = require('a'); var b = require('b');

bundle.js

var bundles = [ 'bundle.js' ];

bundle.js

home.jsprofile.jsfeed.js album.js signup.js

home.js profile.js feed.js

signup.js account.js setting.js

album.js photo.js login.js

var bundles = [ 'home.js', 'profile.js', 'feed.js', 'signup.js' 'account.js', 'settings.js', 'album.js', 'photo.js', 'login.js' ];

var bundles = [ 'home.js', 'profile.js', 'feed.js', 'signup.js' 'account.js', 'settings.js', 'album.js', 'photo.js', 'login.js' ];

/home /profile /feed

var bundles = [ 'home.js', 'profile.js', 'feed.js', 'signup.js' 'account.js', 'settings.js', 'album.js', 'photo.js', 'login.js' ];

/home /profile /feed

home.js profile.js feed.js

html5shiv es5-shim

var bundles = [ 'home.js', 'homeIE.js', 'profile', 'profileIE', 'feed.js', 'feedIE.js' 'signup.js', 'signupIE.js', ... ];

var bundles = [ 'home.js', 'homeIE.js', 'profile', 'profileIE', 'feed.js', 'feedIE.js' 'signup.js', 'signupIE.js', ... ];

var bundles = [ 'home.js', 'homeIE.js', 'profile', 'profileIE', 'feed.js', 'feedIE.js' 'signup.js', 'signupIE.js', ... ];

/home

var bundles = [ 'home.js', 'homeIE.js', 'profile', 'profileIE', 'feed.js', 'feedIE.js' 'signup.js', 'signupIE.js', ... ];

/home

var bundles = [ 'home.js', 'homeIE.js', 'profile', 'profileIE', 'feed.js', 'feedIE.js' 'signup.js', 'signupIE.js', ... ];

/home

home.js

NO

var bundles = [ 'home.js', 'homeIE.js', 'profile', 'profileIE', 'feed.js', 'feedIE.js' 'signup.js', 'signupIE.js', ... ];

/home

home.js homeIE.js

YESNO

Netflix AB testing!

AB Test w/ multiple cells

Cells Control (Cell 1) Cell 2 Cell 3

Movie Cover Art

AB Test w/ multiple cells

Cells Control (Cell 1) Cell 2 Cell 3

Movie Cover Art

14% 6%

Netflix AB testing!

Netflix AB testing!

home.js

home.js

newSearch.jsoldSearch.js

home.js

oldSearch.js

home.js

oldSearch.js

home.js

newSearch.js

home.js

newSearch.jsoldSearch.js

home.js

newSearch.jsoldSearch.js

jQuery React~80KB ~120KB

larger bundle size: • file sizes • time to download • memory usage • time to interactive (TTI)

old school Lego bricks were generic

new Lego is about specialization

hard to reuse specialized bricks

home.js

newSearch.jsoldSearch.js

home.js

newSearch.jsoldSearch.js

home.js

newSearch.jsoldSearch.js

// starting to look like a // lot of bundles... var bundles = [ 'homeNewSearch.js', 'homeNewSearchIE.js', 'homeOldSearch.js', 'homeOldSearchIE.js', ... ];

// starting to look like a // lot of bundles... var bundles = [ 'homeNewSearch.js', 'homeNewSearchIE.js', 'homeOldSearch.js', 'homeOldSearchIE.js', ... ];

4x variations already!

Netflix runs hundreds of AB tests

Netflix runs hundreds of AB tests

but we personalize on many other dimensions too

|S1| ⋅ |S2| ⋅ ⋅ ⋅ |Sn| = |S1 × S2 × ⋅ ⋅ ⋅ × Sn|

|S1| ⋅ |S2| ⋅ ⋅ ⋅ |Sn| = |S1 × S2 × ⋅ ⋅ ⋅ × Sn|

3100 = 5.1537752e47

40,000,000,000 bricks to reach the

40,000,000,000 bricks to reach the

7,600,000,000,000,000 bricks to reach

40,000,000,000 bricks to reach the

7,600,000,000,000,000 bricks to reach

Enough bricks to reach 6.7812832e32 times.

that’s a #$*%^ ton of bundles!

https://xkcd.com/303/

Website's full bundle is 10MB+

how do we deal with conditional modules?

ii. codex conditional Bundling

:

what if we generate on-demand?

what if we generate on-demand?

1. identify the UI variation 2. generate the bundle

how do we identify the UI variation?

AB Tests

AB Tests

AB Tests

truthsnoun, plural [trooth z, trooths]

a bucket of boolean flags used to build a personalized Netflix experience

{ "webfonts": false, "instantSearch": true, "socialFeatures": false, "motionBanner": true, "html5video": true, "customScrollbar": true }

{ "webfonts": false, "instantSearch": true, "socialFeatures": false, "motionBanner": true, "html5video": true, "customScrollbar": true }

inputs and outputs are NOT 1:1

how do we generate the bundle?

home.js

newSearch.jsoldSearch.js

// home.js if (truths.isNewSearch === true) { require('./newSearch'); } else { require('./oldSearch'); }

home.js

newSearch.jsoldSearch.js

home.js

newSearch.jsoldSearch.js

conditioncondition

home.js

newSearch.jsoldSearch.js

conditioncondition

conditioncondition

isNewSearch!isNewSearch

// home.js if (truths.isNewSearch === true) { require('./newSearch'); } else { require('./oldSearch'); }

home.js

newSearch.jsoldSearch.js

isNewSearch!isNewSearch

home.js

newSearch.js

newEntryPoint.js

oldSearch.js

!isNewSearch isNewSearch

git

git

git

codex(node.js module)

git

codexartifact

(node.js module)

artifact

artifact

artifact

{ "home.js": { "deps": [ "dep1.js", "dep2.js", "dep3.js", ], "conditionalDeps": { "newSearch.js": { "name": "isNewSearch", "value": true },

"dep3.js", ], "conditionalDeps": { "newSearch.js": { "name": "isNewSearch", "value": true }, "oldSearch.js": { "name": "isNewSearch", "value": false } } } }

"dep3.js", ], "conditionalDeps": { "newSearch.js": { "name": "isNewSearch", "value": true }, "oldSearch.js": { "name": "isNewSearch", "value": false } } } }

it's a conditional map!

web/v1 web/v2 web/v3 web/v4

artifacttruths

<html/>

http://codex.nflxext.com/web/v1/83af

http://codex.nflxext.com/web/v1/83af

<script/>

http://codex.nflxext.com/web/v1/83af

<script/>

http://codex.nflxext.com/web/v1/83af

<script/> <script/>

codex

? i got this!

http://codex.nflxext.com/web/v1/83af

codex

http://codex.nflxext.com/web/v1/83af http://codex.nflxext.com/{team}/{version}/{truths}

codexweb/v1

http://codex.nflxext.com/web/v1/83af http://codex.nflxext.com/{team}/{version}/{truths}

codex { 83: newSearchTest, af: isChrome }

web/v1

http://codex.nflxext.com/web/v1/83af http://codex.nflxext.com/{team}/{version}/{truths}

home.js

oldSearch.js

home.js

oldSearch.js

home.js

oldSearch.js

home.js

newSearch.jsoldSearch.js

home.js

home.js

newSearch.jsoldSearch.js

response times <= 80ms

codex

http://codex.nflxext.com/web/v1/83af

<script/>

codex

here you go

http://codex.nflxext.com/web/v1/83af

<script/> <script/>

codex

cached! here you go

http://codex.nflxext.com/web/v1/83af

Recap• Build Time: build conditional graph (artifact)

• Run Time: apply truths to artifact

• Conditional bundling is transparent, universal, configuration free!

iii. Scaling For Netflix

web/v1 web/v2 tv/v5 tv/v7

Storage Metadata

Amazon S3Amazon DynamoDB

Storage Metadata

Build Time: Codex Artifact Management

web/v1

Build Time: Codex Artifact Management

web/v1web/v1

Build Time: Codex Artifact Management

saved!

SAVED!

web/v1web/v1

Build Time: Codex Artifact Management

saved!

Build Time: Codex Artifact Management

activate web/v1 web/v1

activated!

Build Time: Codex Artifact Management

Run Time: Codex Bundler

web/v1

here are the active build ids

Run Time: Codex Bundler

here are the artifacts

web/v1

here are the active build ids

Run Time: Codex Bundler

here are the artifacts

web/v1

here are the active build ids

Run Time: Codex Bundler

prod prod-new canary

16GB ought to be enough for us!__🙂

codex

codex

codex

400+ artifacts!

400+ artifacts!

…and we ran out of memory

32GB ought to be enough for us!__🤔

800+ artifacts!

800+ artifacts!

…and we ran out of memory

64GB ought to be enough for us…?__😭

1600+ artifacts!

1600+ artifacts!

…and we ran out of memory. again.

1600+ artifacts!Our teams will use as

much as we give them.

…and we ran out of memory. again.

Q: What's cheap, plentiful, and fast

enough?

Q: What's cheap, plentiful, and fast

enough?A: Disk.

codex

LevelDB

codex

LevelDB

100% CPU usage.

what's the problem?

~68%

v8::internal::Runtime_ParseJson

codex

LevelDB

codex

LevelDB

JSON.parse

JSON.parse is slow. And blocks the CPU!

codex

LevelDB

JSON.parse

LRU Cache: Saving the CPU

codex

LevelDB

JSON.parse

LRU Cache: Saving the CPU

codex

LevelDB

JSON.parse

LRU Cache: Saving the CPU

codex

LevelDB

JSON.parse

LRU Cache: Saving the CPU

codex

LevelDB

JSON.parse

LRU Cache: Saving the CPU

Breaking change to conditional graph

traversal algorithm!

http://codex.nflxext.com/web/v1/83af

http://codex.nflxext.com/web/v1/83af

old algorithm?

new algorithm?

old algorithm?

new algorithm?

http://codex.nflxext.com/1.0.0/web/v1/83afhttp://codex.nflxext.com/2.0.0/web/v1/83af

1.0.0

2.0.0

zuul

zuul 1.0.0

2.0.0

zuul 1.0.0

2.0.0

Good for now.

Good for now.

Continue to look for engineering wins.

What about operational resiliency?

eu-west-1

us-west-2

us-east-1

eu-west-1

us-west-2

us-east-1

eu-west-1

us-west-2

???

eu-west-1

us-west-2

???

eu-west-1

us-west-2

???

???

???

???

<script/> <script/>

codex

<script/>

codex

???

codex

<script/>

???

Recap• Management plane necessary at scale

• Performance is critical (TTI)

• Redundancy across 3 AWS zones

• Resilient against CDN failure

iv. Looking To The Future

why not {bundler}?

how do we support tree

shaking?

don't be afraid to challenge common

convention.

don't make assumptions about the upper limits.

don't optimize before you understand the system.

use the scientific method:

1. gather data 2. formulate hypothesis 3. test hypothesis 4. repeat

engineer for fault tolerance

Netflix scale is challenging.

https://www.flickr.com/clement127/ https://www.flickr.com/jose_antonio_hidalgo_jimenez/ https://www.flickr.com/reiterlied/

Lego Photo Credits

Image Credits

Image Credits

Image Credits

Artist: alecive (Alessandro Roncone) Iconset Homepage: https://github.com/alecive/FlatWoken

Alex Liu aliu@netflix.com @stinkydofu

fin