monitoring docker environments with outlyer

Monitoring, done differently.

Monitoring Docker Environments with Outlyer

Todd RadelDirector, Customer Success

About Me

2

● 20 years’ software development experience

● Java, .NET, Python, Perl, C++

● APM industry veteran4 years at AppDynamics

tradel

http://lnked.in/tradel

todd.radel@outlyer.com

@tradel

Agenda

• How Outlyer works with Docker

• Installing Outlyer agents on Docker

• How to use Host View to visualize your environment

• How plugins and packs work with containers

• Open house for Q&A

3

How Outlyer works with Docker

4

“Docker is hard.”

5

Colin HemmingsCTO, Outlyer

Challenges in the Docker world

• Many versions of Docker, Kubernetes, Swarm, Rancher, etc.

• Breaking API changes between versions

• Evolving and releasing rapidly

• Networking model causes gaps in visibility

6

Comparing Outlyer to other solutions

Basic metrics only (CPU/memory)

Over 500 metrics from base plugin (CPU, memory, network, I/O, load average, etc)

Difficult installation Installation with single command

No service monitoring Most plugins work in Docker

7

A note about labels and tags

• Docker uses “labels” to attach metadata to containers

• Kubernetes/Swarm/Rancher add many of their own labels

• Outlyer agent automatically imports Docker labels as Outlyer tags

(but only label names, not values (yet))

8

Outlyer

TAGS

Docker

LABELS

Installing Outlyer Agent on Docker

9

The “hard” way

DATALOOP_AGENT_KEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

DATALOOP_NAME=docker_container_name

docker run -d --net host \

-e "DATALOOP_AGENT_KEY=${DATALOOP_AGENT_KEY}" \

-e "DATALOOP_NAME=${DATALOOP_NAME}" \

-v /var/run/docker.sock:/var/run/docker.sock:ro \

-v /proc:/rootfs/proc:ro \

-v /sys/fs/cgroup:/rootfs/sys/fs/cgroup:ro \

outlyer/agent:latest

10

The hard way

DATALOOP_AGENT_KEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

DATALOOP_NAME=docker_container_name

docker run -d --net host \

-e "DATALOOP_AGENT_KEY=${DATALOOP_AGENT_KEY}" \

-e "DATALOOP_NAME=${DATALOOP_NAME}" \

-v /var/run/docker.sock:/var/run/docker.sock:ro \

-v /proc:/rootfs/proc:ro \

-v /sys/fs/cgroup:/rootfs/sys/fs/cgroup:ro \

outlyer/agent:latest

11

Needed to allow the agent to

gather host metrics

Automatically uses the latest

version - or check tags at

https://hub.docker.com/r/outlyer

/agent/ for specific versions

Using Docker Compose

wget https://rawgit.com/outlyerapp/docker-alpine/master/agent/docker-compose.yaml

vim docker-compose.yaml

docker-compose -f docker-compose.yaml up

12

Using Docker Compose

version: '2'

services:

dataloop-docker:

image: outlyer/agent:latest

network_mode: host

volumes:

- /var/run/docker.sock:/var/run/docker.sock:rw

- /sys/fs/cgroup:/rootfs/sys/fs/cgroup:ro

- /var/lib/docker/:/var/lib/docker:ro

- /proc:/rootfs/proc:ro

environment:

- DATALOOP_NAME=outlyer-agent

- DATALOOP_AGENT_KEY=<your dataloop agent key>

13

Kubernetes Deployment

wget https://rawgit.com/outlyerapp/docker-alpine/master/agent/kubernetes_daemonset.yaml

kubectl apply -f kubernetes_daemonset.yaml

• Deploys one container on every host in cluster (masters and workers)

• Lights up your entire cluster with a single command

14

Swarm and Rancher Deployment

• Swarm does not currently support host networking

Use manual method for now

• Deploy on Rancher through Rancher UI

Config files will be available soon

15

Using Host View to visualize your cluster

16

What is Host View?

• Displays health of every container

and host

• Sort, group, and filter by tags

or Docker labels

• Visualize health by

CPU/memory/presence

• Show links between containers

and hosts

• Container status at a glance

17

How plugins and packs work in Docker

18

The problem

• Most plugins (and Nagios scripts) want to access “localhost” for ports,

files, etc.

HOST = 'localhost'

PORT = 27017

• But the agent runs in a different container from the app you want to

monitor

19

The magic solution

• Docker compatibility layer

• Transparently intercepts common API calls to localhost

• Replaced with calls to target container

20

The magic solution

Most plugins and packs work the same inside and outside of Docker!

21

Adapting your own plugins

• See if the client library is already covered by plugin-helper.(Requests, MongoClient, Psycopg2, Socket, Subprocess, others)

• If so: no code changes necessary!

• If not: use the plugin-helper to find the target container.

22

Using the plugin helper library

try:

from outlyer.plugin_helper.container import is_container, get_container_ip

if is_container():

container_ip = get_container_ip()

except ImportError:

print(“Plugin helper library is not available!”)

print(“Please upgrade your Outlyer agent to the latest version.”)

23

Coming Soon!

24

New features in Fall 2017 release

• Full support for key/value labels

• Host View becomes the default view

• “Scoped” dashboards with label selectors

25

Questions & Answers

26

Resources

27

Resources

• Monitoring Docker environments with Outlyer

(product documentation)

• Plugin Helper library

(for adapting your plugins to Docker)

• Support Slack

(sign up and ask questions)

28

Thank You!

29