The Dark Art of Container Monitoring

Loris Degioanni

Me

Loris DegioanniCreator and CEO of sysdigPast: WinPcap, Wireshark

@lorisdegio

Containers Are Great…

• less overhead• faster deployments• reproducibility of environments• cost optimizations• Isolation• flexibility

…But Inspecting Containers Is Not Easy

Containers are:• Isolated• Self-Contained• Simple• Lightweight

…But Inspecting Containers Is Not Easy

Containers are:• Isolated• Self-Contained• Simple• Lightweight

Can monitoring and troubleshooting respect these properties?

•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig

• Examples and demos

In This Talk

•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig

• Examples and demos

In This Talk

•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig

• Examples and demos

In This Talk

Hypervisor

Monitoring VMs

VM1 VM3VM2

Hypervisor

Monitoring VMs, Option 1

VM1 VM3VM2

Hypervisor-level instrumentation,Amazon CloudWatch

Hypervisor

Monitoring VMs, Option 2

VM1 VM3VM2

Monitoring Agent

OS

Monitoring Containers

Container1 Container3Container2

OS

Monitoring Containers, Option 1

Container1 Container3Container2

Monitoring Agent

OS

Monitoring Containers, Option 1

Container1 Container3Container2

Monitoring Agent

• Not scalable• Not composable• Adds dependencies/size• Kills the concept of one process per container

OS

Monitoring Containers, Option 2

Container1 Container3Container2

Container runtime – level monitoringKernel-level instrumentation

OS

Monitoring Containers, Option 3

Container1 MonitoringContainer

Container2

•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig

• Examples and demos

In This Talk

•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig

• Examples and demos

In This Talk

•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig

• Examples and demos

In This Talk

In The Rest of this Talk

•Overview of Open Source monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig

• Examples and demos

In The Rest of this Talk

•Overview of Open Source monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig

• Examples and demos

Some Things We Want to Monitor

• Resource usage (CPU/Memory/Disk)• Network activity• File I/O activity• Errors/faults• Application activity/Logs• Topology

Resource Usage (CPU/Memory/Disk)

• ps/top/htop from the host• Pro: always there• Cons: Very limited container context (full cgroup string only), work only from

the main host

• docker ps / docker top / docker stats• Pro: always there, APIs• Cons: Docker only, limited interactivity

• cAdvisor• sysdig

Installing cAdvisorsudo docker run \--volume=/:/rootfs:ro \--volume=/var/run:/var/run:rw \--volume=/sys:/sys:ro \--volume=/var/lib/docker/:/var/lib/docker:ro \--publish=8080:8080 \--detach=true \--name=cadvisor \google/cadvisor:latest

cAdvisor Pros and Cons

Pros• easy to install• works from a container• user friendly (web UI)• API• Integrations (e.g. InfluxDB, Prometheus, Heapster)

Cons: • (very) Limited set of metrics• Containers are black boxes

Docker Stats APIThe most sophisticated Docker stats poller ever:echo -e "GET /containers/mysql/stats HTTP/1.1\r\n" | nc -U /var/run/docker.sock

Info:http://docs.docker.com/reference/api/docker_remote_api_v1.17/#get-container-stats-based-on-resource-usage

Docker Stats API Pros and Cons

Pros• Part of Docker• Richer than cAdvisor• REST API• Integrations

• docker-collectd-plugin github.com/cloudwatt/docker-collectd-plugin• commercial vendors

Cons: • Docker only• Containers are black boxes

sysdig

•Capture system events, filter them, run useful scripts• strace + tcpdump + lsof + htop + Lua•Open Source•Nice curses UI

Design Goals

•Production-ready• Simple • lightweight

•Rich data• Instant value•Natural workflow•Native support for containers

Design Goals

•Production-ready• Simple • lightweight

•Rich data• Instant value•Natural workflow•Native support for containers

Installing sysdig

docker run -i -t --name sysdig --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro sysdig/sysdig

sysdig Architecture

Kernel

Container1

Docker

Container2

Docker

Container3

LXCAppApp

sysdig Architecture

Kernel

Container1

Docker

Container2

Docker

Container3

LXCAppApp

Instrumentation through kernel module

sysdig Architecture

Kernel

Container1

Docker

Container2

Docker

Container3

LXCAppApp

sysdig

Docker

Capture andanalysis

sysdig Architecture

Kernel

Container1

Docker

Container2

Docker

Container3

LXCAppApp

sysdig

Docker

(optionally) Save to a trace file

foo.scap

Viewing Network Activity

•cAdvisor•docker stats <container>• iftop/tcpdump/tshark in the container•sysdig

Viewing Disk Activity

•iotop/lsof in the container•sysdig

Viewing Application Activity

•Code/container instrumentation•Logs•sysdig

Thank You!

www.sysdig.orghttps://github.com/draios/sysdigwww.sysdig.com