scaling patterns for november 2019 netflix’s edge › system › files › presentation-slides ›...

Justin Ryan, Senior Software Engineer @ NetflixTwitter: quidryan, LinkedIn: www.linkedin.com/in/justin-charles-ryan

Scaling Patterns for Netflix’s EdgeNovember 2019

https://twitter.com/quidryan

http://www.linkedin.com/in/justin-charles-ryan

Scaling Patterns for Netflix’s EdgeJustin Ryan

Laundry


Space-Time Compromise


Laundry


ScalingQ

3 20

11

Q3

2019


Scaling

Edge


Scaling Problems

— Debuggability

— Infrastructure

— Managing


Scalability


Trade Offs


Policies


Accessible


Five Use Cases

— Melnitz— Mantis— Passport— Device Types— Sharding


Logged In


Cookies


Melnitz

Trust Cookie Cookie Expiration


Cookies L7 Proxy Account

Service

API

ExpiredYes

No


Boom


Melnitz

Copyright Columbia Pictures


Melnitz

Set<CustomerId> Cookie Expiration


Melnitz Bloom Filter: Compact Set which

can answer the question is an object

“Possibly in Set”?


Melnitz


Melnitz L7 Proxy Account

Service

API

Bloom Filter

ExpiredYes

NoMaybe

No


MelnitzAccount Service Auth

Event

Melnitz A

DB

Melnitz B

1m

L7 Proxy


Melnitz


Melnitz Trade Offs

— Probabilistic Data Structure

— Off the Shelf Components


Five Use Cases

— Melnitz - Distributed Bloom Filter— Mantis— Passport— Device Types— Sharding


Mantis


Mantis 2 million request per second *

3 kilobytes per request *

10 micro-services =

57 GB/s =~ 4.6 PB / day


Naive Approach

Service Elastic SearchKibana


Mantis


Mantis Master

Mantis

Service

MantisAgent

Mantis Source Job

Elastic Search

Kibana

Kafka


Mantis Master

Service

Mantis

Mantis Source Job

Elastic Search

Browser

MantisAgent

Kafka


Avoid Work


Mantis Trade Offs

— Mantis Query Language

— Scale Solution to Zero


Five Use Cases

— Melnitz - Distributed Bloom Filter— Mantis - Event Querying— Passport— Device Types— Sharding


Passport


Service Plan

API

L7 P

roxy

Mid Tier A

Mid Tier B

Mid Tier C

Passport


Passport

Service Plan

API

L7 P

roxy

Mid Tier A

Mid Tier B

Mid Tier C


Cache

Service Plan

API

L7 P

roxy

Mid Tier A

Mid Tier B

Mid Tier C


Cache

API

L7 P

roxy

Mid Tier A

Mid Tier B

Mid Tier C

Service Plan


Passport


Passport Trade Offs

— Data Passing

— Heavy Data Structures


Five Use Cases

— Melnitz - Distributed Bloom Filter— Mantis - Event Querying— Passport - Heavy Data Structure— Device Types— Sharding


Device Types


Device TypesNFAPPL-01-IPAD7=5-09BEC57F64CA5F837C75F924B94D

NFAPPL-01-IPAD3=4-15B2D9B33A994CC373017A9C6586

NFAPPL-02-IOS2=4-FT7WHG8H2UR2MCJ4YKR3P0DFH6

NFANDROID1-PRV-P-GOOGLPIXEL=2-7351-0F441DF98B8D99074

NFANDROID1-PRV-4339-0E50FC7582396F5C4496ED935F2F48ED

13951007

964


Device Types


Device Types

Mid-Tier F

Mid-Tier E Mid-Tier D

Mid-Tier C

Mid-Tier BMid-Tier A

DTS


Device Types

Mid-Tier CMid-Tier BMid-Tier A

DTS PubSub


Device Types

Mid-Tier CMid-Tier BMid-Tier A

DTS PubSub

Device Types Device TypesDevice Types


Device Types


Device Type Trade Offs

— Business Tuned Fallbacks

— Leverage Existing Infrastructure


Five Use Cases

— Melnitz - Distributed Bloom Filter— Mantis - Event Querying— Passport - Heavy Data Structure— Device Types - Device Grouping— Sharding


Sharding


MSL — Message Security Layer == MSL

— Secure Messaging Framework


Sharding MSL

L7 P

roxy APIAPIAPIAPIAPIAPI


Sharding MSL

L7 P

roxy

MSL

APIAPIAPI


CPU Cost Per Request

- CPU to RPS ratio- Load Average

31% decrease


Latency

30% decrease

20% p99 decrease


Sharding MSL- % of Wall Clock in GC- GC Pause Time


Sharding

3 = 1 + 1


Sharding Trade Offs

— Operational Overhead


Five Use Cases

— Melnitz - Distributed Bloom Filter— Mantis - Event Querying— Passport - Heavy Data Structure— Device Types - Device Grouping— Sharding - Runtime Refactoring


Bonus Trade Offs

— Do the laundry

Justin [email protected]

Twitter: @quidryan

Thank You.

mailto:[email protected]

https://twitter.com/quidryan


References — github.com/Netflix◆ eureka◆ zuul◆ mantis◆ msl

— Spinnaker— Apache Pulsar— Google Guava (Bloom Filter)

https://medium.com/netflix-techblog/netflix-shares-cloud-load-balancing-and-failover-tool-eureka-c10647ef95e5

https://medium.com/netflix-techblog/open-sourcing-zuul-2-82ea476cb2b3

http://techblog.netflix.com/2016/03/stream-processing-with-mantis.html

https://medium.com/netflix-techblog/message-security-layer-a-modern-take-on-securing-communication-f16964b79642

https://medium.com/netflix-techblog/moving-from-asgard-to-spinnaker-a000b2f7ed17

scaling patterns for november 2019 netflix’s edge › system › files › presentation-slides ›...

Documents