netflix cdn and open source

Introduction

Netflix CDN and Open Source

Gleb [email protected]

Nginx.conf 2014San Francisco

October 21, 2014

Gleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 1 / 1

Introduction to Netflix CDN

Netflix in numbers

I 50+ millions subscribers in 40+ countries

I Over 1 Petabyte of data


Introduction to Netflix CDN Netflix traffic

2014 1H Sandvine North Amertica Traffic Report


Netflix CDN Netflix Streaming network

Components of Netflix Streaming

I Amazon Web ServicesI Website, Business Logic, Customer AuthenticationI Data Sciences, Encoding/EncryptionI Command and Control

I Content StreamingI Big 3 CDN’s (Akamai, Limelight, Level3)I Own CDN


Netflix CDN Running own CDN

Reasons for running own CDN

I Grow faster

I Reduce costs

I Control the server side of HTTP connection

I Build a CDN specialized in Netflix content delivery

I Put the content closer to a client

Solution: Open Connect1

1https://www.netflix.com/openconnectGleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 5 / 1

OpenConnect Introduction to OpenConnect

The OpenConnect Initiative

Amazon Cloud The Internet ISPs

video

encoding

account

authorization

streaming

control

Origin

Server

ISP

Cache

IX

Cache

Client Client


OpenConnect Introduction to OpenConnect

OpenConnect Appliance


OpenConnect OpenConnect hardware


The big cacheI 4U chassisI Storage

I 36 HDDs, 4 Terabyte eachI 6 SSDs, 0.5 Terabyte each

I Two 10 Gbit/s Chelsio NICs

I 8-core CPU


OpenConnect OpenConnect hardware


The fast cacheI 1U chassisI Storage

I 14 SSDs, 1 Terabyte each

I 40 Gbit/s Chelsio NICs

I 8-core CPU


OpenConnect OpenConnect software

OpenConnect Appliance software

I FreeBSD operating systemI nanobsd(8) based firmwareI some enhancements

I NGINX web serverI custom modules

I BIRD routing daemon



Software choice

I Open source

I BSD license

I FreeBSDI Known to be fast and stableI Strong developer community

I NGINXI Known to be fast and stableI Commercial support from Nginx, Inc.I Flexible framework for custom modules

I FreeBSD + NGINXI kqueue(2) event driven modelI sendfile(2) + aio(4) mode



Software choice

I Open source

I BSD licenseI FreeBSD

I Known to be fast and stableI Strong developer community

I NGINXI Known to be fast and stableI Commercial support from Nginx, Inc.I Flexible framework for custom modules

I FreeBSD + NGINXI kqueue(2) event driven modelI sendfile(2) + aio(4) mode


OpenConnect OpenConnect operation

OpenConnect operation: serve & fill


OpenConnect OpenConnect operation

OpenConnect: streaming numbers

I 400,000 stream files per appliance

I 5000-30,000 client streams per appliance

I 300-1000 clients per disk

I Clients request 80Kb-300Kb ranges


Netflix & open source Dealing with open source

Dealing with open source

The traditional way:

I Take a stable, well tested version,fork it

I Develop your product on top of it


Netflix & open source Dealing with open source

Dealing with open source

The Netflix rules:

I Pull the bleeding-edge version ofsoftware

I Push your changes upstreamaggressively


Netflix & open source Different myths on different versions

Myths about development version

Myth #1:Development version is full of bugs



Myths about development version

Myth #1:Development version is full of bugs

Truth: most bugs are discovered instable versions



Myths about stable version

Myth #2:We will wait for a stable version, and

someone else will fix bugs 2

2http://en.wikipedia.org/wiki/Free rider problemGleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 16 / 1


Myths about stable version

Myth #2:We will wait for a stable version, and

someone else will fix bugs 2

Truth: No one will discover your bugs

2http://en.wikipedia.org/wiki/Free rider problemGleb Smirnoff [email protected] Netflix CDN and Open Source October 21, 2014 16 / 1


Discovering bugs in different versions

development stable

code is “hot” code is unmaintained

no API/ABI constraintsAPI/ABI must be

preserved



More myths

Myth #3:Not following development version saves

us a lot of time



More myths

Myth #3:Not following development version saves

us a lot of time

Truth: some day you will need to goforward


Netflix & open source Opening sources

Myths on open source

Myth #4:Sharing code discloses know-how’s



Myths on open source

Myth #4:Sharing code discloses know-how’s

Truth: know-how’s reside in a tinypercent of code, or even outside



Noble or selfish?

I We want to influence the direction of open sourcedevelopment

I We want to outsource maintainance of our code tocommunity

I We want more eyes to examine our code

I We want more people to discover bugs in it

I And, of course, we want to be considered noblegivers of code to community ,


Netflix & open source Netflix’s commitment to open source

OpenConnect performance

I OpenConnect started in 2011< 10 Gbps per appliance

I Now, in 2014, we achieve> 30 Gbps per appliance

I Next goal is > 80 Gbps per appliance



Areas of focus

I network stack: sockets, TCP, drivers

I storage: drivers, UFS

I VM subsystem: caching



Performance improvements

I Reducing lock/cache line contention inI link aggregation driverI kernel socket buffersI sendfile(2) kernel memory buffers

I Reducing complexity, structuring code inI kernel flowtableI NGINX sendfile(2) code

I Introducing multithreading inI UFS softupdates



Various improvements & bugfixes

I NGINX core and modules

I IPv6 network stack

I UFS journaling



Completely new features

I Per-CPU statistical counters:I Precise: do not lose updatesI Fast: do not use any locking, neither critical sections

I Completely new sendfile(2) implementation:I Doesn’t block on disk I/O!I Allows to specify readaheadI Allows to deny VM caching



Working with community

I unmapped I/O

I VM radix



Future and work in progress

I cc netflix: new TCP congestion control algorithm

I hardware assisted TCP pacing

I kernel-side TLS offload

I SSD I/O performance improvements

I multithreading pagedaemon

I NUMA support



Questions?


netflix cdn and open source

Software

cdn net

open sourcegleb

open connect11https

openconnect introduction

comopenconnectgleb smirno

cdngleb smirno

core cpugleb smirno

petabyte of datagleb