MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015

DOCKER {AT,WITH} SIGNALFX

MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015

Me?

github.com/mpetazzoni

@mpetazzoni

MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015

What is SignalFx?

The pitch

• Real-time monitoring system for modern applications

• User-defined, streaming and interactive analytics

• Alerting and anomaly detection

• Easy to deploy, to use and to integrate

• Scales; from small deployments to large, complex infrastructures

MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015

Docker at SignalFx

Docker at SignalFx

• Started using Docker in November 2013

• Running Docker in prod ever since we’ve had a “prod”

• Back when Docker’s README said “DO NOT RUN IN PRODUCTION”

Objectives

• Separate infrastructure concerns

• Stronger independence from hosting provider

• Development lifecycle

• Application packaging and delivery

• Higher-level orchestration

Infrastructure vs application

• Separation of concerns

• Infrastructure provides appropriately spec-ed hosts and network

• Abstracts from cloud provider

• Currently on AWS

• Amazon Linux AMIs + Salt to setup a few things, up to Docker

Infrastructure vs application (cont.)

• “Application” stack takes it up from there

• Everything is containerized, including 3rd party components

• No dependence on Amazon specifics

• We can spin up the stack anywhere (laptop, GCE, DO, …)

SDLC

• Run locally, in tests and in prod environments

• Same bits, same tools, same arguments, etc.

• Crucial for software validation

• Maven, Jenkins, Quay.io, MaestroNG

• maestro start

SDLC (cont.)

docker-maven-plugin

SDLC (cont.)

• Continuous build of the release branch

• Builds and pushes :latest tagged Docker images

• Promotion track through :lab, :prod, :rc and :mon

• One-click promote + upgrade, orchestrated by MaestroNG

• Automated integration tests environment

Application packaging and delivery

• All components of the stack are packaged as Docker images

• 3rd-party component packaging was figured out once (Dockerfile)

• SignalFx components all have their Dockerfile too

• Built by docker-maven-plugin

• All components have a run.py init script

Discovery

• ZooKeeper / Curator service discovery

• We built announcing wrappers for Cassandra and ElasticSearch

• Only need to know “realm” name and zkConnectString

• Everything else is discovered from ZooKeeper

• Need to make sure to announce host IP address/port • Maestro gives us that via environment

Orchestration

• We built MaestroNG

• Orchestrator of multi-host Docker environments

• github.com/signalfuse/maestro-ng

• Jinja2/Yaml environment file

• Defines what runs where, and how

• Knows and respects service dependencies

MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015

Monitoring containers

The basics

• Docker’s /stats API provides the bare minimum • CPU

• Memory

• Network I/O

• Disk I/O

• Pitfalls; GET /stats is blocking, up to a second

• Use CollectD and docker-collectd-plugin • github.com/signalfx/docker-collectd-plugin

The basics (cont.)

• Data returned by /stats API needs “math” to yield usable numbers • See Docker client code

• Better: use SignalFx! • Docker Dashboard

• Collect host metrics too if you can (with CollectD)

• System/container-level metrics are very limited

• We need to go deeper

Application monitoring

• Usually needs more specific visibility

• Instrument first, ask questions later

• Metrics are cheap, both in CPU and $

• For 3rd-party apps, use CollectD and its plugins • We’re doing a lot of work to improve/provide CollectD plugins for common FOSS

• And provide curated dashboards for each of them

Application monitoring (cont.)

• For 1st-party software, depends on language platform

• For VM-based services (Java, Scala, Go, …), get key metrics • Heap

• # threads

• Time spent in GC and # collections

• Use metric libraries and instrument everything

• Bring it all together in dashboards and anomaly detectors

Docker with SignalFx

• Objective: time-to-data as short as possible

• Easiest route: use our packaged CollectD • Good default set of host metrics

• Install/enable plugins as needed; docker-collectd-plugin

• SignalFx automatically imports curated dashboards

• That’s it!

More monitoring with SignalFx

• Use our client libraries or API to send your monitoring data • Or tee off your existing Graphite, StatsD, … pipeline towards SignalFx

• Leverage the power of the real-time, streaming analytics

• Correlate system, container and application metrics

• Build better dashboards and alerts from higher-level signals

Thanks!

github.com/signalfuse/maestro-ng

github.com/signalfx/docker-collectd-plugin

signalfx.com/free-trial.html

@signalfx — signalfx.com/careers.html

@mpetazzoni — github.com/mpetazzoni