monitorama 2015 monitoring openconnect cdn

Monitoring OpenConnect CDN

Sergey Fedorov, NetflixMonitorama 2015

Sergey Fedorov, Netflix, Monitorama 2015

What is OpenConnect

36.5%

US downstream traffic *

* 2015 Sandvine reportSergey Fedorov, Netflix, Monitorama 2015

OpenConnect Cache Appliance

Space/Power optimized10/40Gbs network interfaceFreeBSD OSNGinx serverBird routing proxy

Gizmodo, “This box can hold an entire Netflix” http://gizmodo.com/this-box-can-hold-an-entire-netflix-1592590450Sergey Fedorov, Netflix, Monitorama 2015

http://gizmodo.com/this-box-can-hold-an-entire-netflix-1592590450

Network

Transit

Internet Exchange

ISP embedded



Intelligent clients

Control Plane

end-user content request router

client locationnetwork conditionsserver utilizationcontent distribution


Who we are

Sergey Fedorov Stefan PraszalowiczSergey Fedorov, Netflix, Monitorama 2015

Monitoring challenge

Testing in prod*

Network changesFirmware deploymentsApp pushesUpdating content...



CachesClients

Control Plane

Microservices

Network

Capacity

Config

Content

Telemetry (Atlas)Logs (ElasticSearch)

Data sources

METRICS

Something breaks all the time

Big problems start small

Context matters


Small SRE team

Elastic

How we do it

Netflix Clients Caches Network ConfigData sources ......

...



...

Orchestration

Data processing

stream processorspollers


FSMState processing


...

Orchestration

Data processing



MAINTENANCE

start fixing end fixing

threshold=75%



action: okfrom: cpu

threshold=75%

MAINTENANCE


MAINTENANCE


action: silencefrom: config

threshold=75%


MAINTENANCE


action: okfrom: cpu

threshold=75%


MAINTENANCE


action: silencefrom: config

threshold=75%


MAINTENANCE


action: breakfrom: cpu

threshold=75%


MAINTENANCE


action: okfrom: cpu

threshold=75%


MAINTENANCE


action: unsilencefrom: config

threshold=75%


MAINTENANCE


action: okfrom: cpu

threshold=75%


MAINTENANCE



threshold=75%


MAINTENANCE


action: start_fixfrom: user

threshold=75%


MAINTENANCE



threshold=75%


MAINTENANCE


action: okfrom: cpu

threshold=75%


MAINTENANCE


action: end_fixfrom: user

threshold=75%


MAINTENANCE


threshold=75%


FSMState processing


...

Orchestration

Data processing



FSMState processing


...

Orchestration

Data processing


Events processingEvent handlers

STATE TRANSITION EVENT● OLD STATE● NEW STATE● Input action● Metric name● Action metadata

○ metric value○ comments○ tags○ timestamp○ ...

Event handlers

Triggers an event

Event handlersRULES



Events priority

Escalation

Do Never

Notice

Warning

Critical

Severity

Info

Do Next

Do Last

Do Now

0 1 2 3

Notice

Warning

Critical

Severity

Info

0 1 2 3Escalation

Notice

Warning

Critical

Severity

Info

0 1 2 3

Notifications


FSMState processing


...

Orchestration

Data processing



Aggregation

C

ClusterCache state = aggregation of states of its metrics

Cluster state = aggregation of states of its caches

OK all OK DEGRADED some BROKEN or DEGRADEDBROKEN most BROKEN

All caches are OK → cluster state is OK


Aggregation

C

Cluster OK all OK DEGRADED some BROKEN or DEGRADEDBROKEN most BROKEN

2/12 caches are BROKEN → cluster state is DEGRADED


Aggregation

C

Cluster OK all OK DEGRADED some BROKEN or DEGRADEDBROKEN most BROKEN

7/12 caches are BROKEN → cluster state is BROKEN


FSMState processing


...

Orchestration

Data processing



Challenges

Setup


Challenges

SetupPredefined groupings


Challenges

SetupPredefined groupingsUI


Challenges

SetupPredefined groupingsUIIssues correlation


Challenges

SetupPredefined groupingsUIIssues correlationFailure forecasting


Challenges

SetupPredefined groupingsUIIssues correlationFailure forecastingOSS


Feedback

jobs.netflix.com/jobs/1693/

jobs.netflix.com/jobs/2240/

Sergey FedorovOpenConnect, [email protected]

monitorama 2015 monitoring openconnect cdn

Technology

netflix monitorama

maintenance sergey fedorov

context matters sergey

entire netflix http

sandvine reportsergey

cpu threshold

maintenance start fixin

config threshold