sysdig monitorama slides

The Dark Art of Container Monitoring

Loris Degioanni

Me

Loris Degioannisysdig’s creator and CEOPast: WinPcap, Wireshark

@ldegio

Containers Are Great…

• less overhead• faster deployments• reproducibility of environments• cost optimizations• Isolation• flexibility

…But Monitoring Them Is Not Easy

Containers are:• Isolated• Self-Contained• Simple• Lightweight• Disposable

…But Monitoring Them Is Not Easy

Containers are:• Isolated• Self-Contained• Simple• Lightweight• Disposable

Is it possible to respect these properties and also have great monitoring?

•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig

• Examples and demos

In This Talk

•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig

• Examples and demos

In This Talk

In This Talk

•Overview of container monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig

• Examples and demos

In This Talk

Some Things We Want to Monitor

• Resource usage (CPU/Memory/Disk)• Network activity• Application activity• File I/O activity• Errors/faults• Topology

Resource Usage (CPU/Memory/Disk)

• ps/top/htop from the host• Pro: always there• Cons: Very limited container context (full cgroup string only), only work from

the host

• docker ps / docker top• cAdvisor• docker stats• sysdig

Installing cAdvisor

sudo docker run \--volume=/:/rootfs:ro \--volume=/var/run:/var/run:rw \--volume=/sys:/sys:ro \--volume=/var/lib/docker/:/var/lib/docker:ro \--publish=8080:8080 \--detach=true \--name=cadvisor \google/cadvisor:latest

cAdvisor Pros and Cons

Pros• easy to install• works from a container• Nice local UI• API• Integrations (e.g. InfluxDB, Prometheus, Heapster)

Cons: • (very) Limited set of metrics• Containers are black boxes

Docker Stats + API

• docker stats mysql• echo -e "GET /containers/mysql/stats HTTP/1.1\r\n" |

nc -U /var/run/docker.sock

Docker Stats + API

• cpu_stats• network• memory_stats• blkio_stats

http://docs.docker.com/reference/api/docker_remote_api_v1.17/#get-container-stats-based-on-resource-usage

Docker Stats Pros and Cons

Pros• Part of Docker• Richer than cAdvisor• REST API• Integrations

• docker-collectd-plugin github.com/cloudwatt/docker-collectd-plugin• commercial vendors

Cons: • Docker only• Containers are black boxes

sysdig

•Capture system events, filter them, run useful scripts• strace + tcpdump + lsof + Lua•Open Source

csysdig

•Capture system events, filter them, run useful scripts• strace + tcpdump + lsof + htop + iftop + Lua•Open Source•Nice curses UI

Design Goals

•Production-ready• Simple • lightweight

•Rich data•Native support for containers• Runs in a container

•Natural workflow

sysdig Architecture

Kernel

Container1

Docker

Container2

Docker

Container3

LXCAppApp

sysdig Architecture

Kernel

Container1

Docker

Container2

Docker

Container3

LXCAppApp

Instrumentation through kernel module

sysdig Architecture

Kernel

Container1

Docker

Container2

Docker

Container3

LXCAppApp

sysdig

Docker

Capture andanalysis

sysdig Architecture

Kernel

Container1

Docker

Container2

Docker

Container3

LXCAppApp

sysdig

Docker

(optionally) Save to a trace file

foo.scap

Viewing Network/App Activity

•cAdvisor•docker stats• iftop/tcpdump/tshark in the container•sysdig

Viewing File I/O Activity and Errors

•iotop/lsof in the container•sysdig

Topology

•Sysdig cloud

Thank You!

www.sysdig.com

@sysdig