docker {at,with} signalfx
TRANSCRIPT
MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015
DOCKER {AT,WITH} SIGNALFX
MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015
Me?
github.com/mpetazzoni
@mpetazzoni
MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015
What is SignalFx?
The pitch
• Real-time monitoring system for modern applications
• User-defined, streaming and interactive analytics
• Alerting and anomaly detection
• Easy to deploy, to use and to integrate
• Scales; from small deployments to large, complex infrastructures
MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015
Docker at SignalFx
Docker at SignalFx
• Started using Docker in November 2013
• Running Docker in prod ever since we’ve had a “prod”
• Back when Docker’s README said “DO NOT RUN IN PRODUCTION”
Objectives
• Separate infrastructure concerns
• Stronger independence from hosting provider
• Development lifecycle
• Application packaging and delivery
• Higher-level orchestration
Infrastructure vs application
• Separation of concerns
• Infrastructure provides appropriately spec-ed hosts and network
• Abstracts from cloud provider
• Currently on AWS
• Amazon Linux AMIs + Salt to setup a few things, up to Docker
Infrastructure vs application (cont.)
• “Application” stack takes it up from there
• Everything is containerized, including 3rd party components
• No dependence on Amazon specifics
• We can spin up the stack anywhere (laptop, GCE, DO, …)
SDLC
• Run locally, in tests and in prod environments
• Same bits, same tools, same arguments, etc.
• Crucial for software validation
• Maven, Jenkins, Quay.io, MaestroNG
• maestro start
SDLC (cont.)
docker-maven-plugin
SDLC (cont.)
• Continuous build of the release branch
• Builds and pushes :latest tagged Docker images
• Promotion track through :lab, :prod and :mon
• One-click promote + upgrade, orchestrated by MaestroNG
• Automated integration tests environment
Application packaging and delivery
• All components of the stack are packaged as Docker images
• 3rd-party component packaging was figured out once (Dockerfile)
• SignalFx components all have their Dockerfile too
• Built by docker-maven-plugin
• All components have a run.py init script
Discovery
• ZooKeeper / Curator service discovery
• We built announcing wrappers for Cassandra and ElasticSearch
• Only need to know “realm” name and zkConnectString
• Everything else is discovered from ZooKeeper
• Need to make sure to announce host IP address/port • Maestro gives us that via environment
Orchestration
• We built MaestroNG
• Orchestrator of multi-host Docker environments
• github.com/signalfuse/maestro-ng
• Jinja2/Yaml environment file
• Defines what runs where, and how
• Knows and respects service dependencies
MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015
Monitoring containers
The basics
• Docker’s /stats API provides the bare minimum • CPU
• Memory
• Network I/O
• Disk I/O
• Pitfalls; GET /stats is blocking, up to a second
• Use CollectD and docker-collectd-plugin • github.com/signalfx/docker-collectd-plugin
The basics (cont.)
• Data returned by /stats API needs “math” to yield usable numbers • See Docker client code
• Better: use SignalFx! • Docker Dashboard
• Collect host metrics too if you can (with CollectD)
• System/container-level metrics are very limited
• We need to go deeper
Application monitoring
• Usually needs more specific visibility
• Instrument first, ask questions later
• Metrics are cheap, both in CPU and $
• For 3rd-party apps, use CollectD and its plugins • We’re doing a lot of work to improve/provide CollectD plugins for common FOSS
• And provide curated dashboards for each of them
Application monitoring (cont.)
• For 1st-party software, depends on language platform
• For VM-based services (Java, Scala, Go, …), get key metrics • Heap
• # threads
• Time spent in GC and # collections
• Use metric libraries and instrument everything
• Bring it all together in dashboards and anomaly detectors
Docker with SignalFx
• Objective: time-to-data as short as possible
• Easiest route: use our packaged CollectD • Good default set of host metrics
• Install/enable plugins as needed; docker-collectd-plugin
• SignalFx automatically imports curated dashboards
• That’s it!
More monitoring with SignalFx
• Use our client libraries or API to send your monitoring data • Or tee off your existing Graphite, StatsD, … pipeline towards SignalFx
• Leverage the power of the real-time, streaming analytics
• Correlate system, container and application metrics
• Build better dashboards and alerts from higher-level signals
Thanks!
github.com/signalfuse/maestro-ng
github.com/signalfx/docker-collectd-plugin
signalfx.com/free-trial.html
@signalfx — signalfx.com/careers.html
@mpetazzoni — github.com/mpetazzoni