sysdig monitorama slides
TRANSCRIPT
The Dark Art of Container Monitoring
Loris Degioanni
Me
Loris Degioannisysdig’s creator and CEOPast: WinPcap, Wireshark
@ldegio
Containers Are Great…
• less overhead• faster deployments• reproducibility of environments• cost optimizations• Isolation• flexibility
…But Monitoring Them Is Not Easy
Containers are:• Isolated• Self-Contained• Simple• Lightweight• Disposable
…But Monitoring Them Is Not Easy
Containers are:• Isolated• Self-Contained• Simple• Lightweight• Disposable
Is it possible to respect these properties and also have great monitoring?
•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig
• Examples and demos
In This Talk
•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig
• Examples and demos
In This Talk
In This Talk
•Overview of container monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig
• Examples and demos
In This Talk
Some Things We Want to Monitor
• Resource usage (CPU/Memory/Disk)• Network activity• Application activity• File I/O activity• Errors/faults• Topology
Resource Usage (CPU/Memory/Disk)
• ps/top/htop from the host• Pro: always there• Cons: Very limited container context (full cgroup string only), only work from
the host
• docker ps / docker top• cAdvisor• docker stats• sysdig
Installing cAdvisor
sudo docker run \--volume=/:/rootfs:ro \--volume=/var/run:/var/run:rw \--volume=/sys:/sys:ro \--volume=/var/lib/docker/:/var/lib/docker:ro \--publish=8080:8080 \--detach=true \--name=cadvisor \google/cadvisor:latest
cAdvisor Pros and Cons
Pros• easy to install• works from a container• Nice local UI• API• Integrations (e.g. InfluxDB, Prometheus, Heapster)
Cons: • (very) Limited set of metrics• Containers are black boxes
Docker Stats + API
• docker stats mysql• echo -e "GET /containers/mysql/stats HTTP/1.1\r\n" |
nc -U /var/run/docker.sock
Docker Stats + API
• cpu_stats• network• memory_stats• blkio_stats
http://docs.docker.com/reference/api/docker_remote_api_v1.17/#get-container-stats-based-on-resource-usage
Docker Stats Pros and Cons
Pros• Part of Docker• Richer than cAdvisor• REST API• Integrations
• docker-collectd-plugin github.com/cloudwatt/docker-collectd-plugin• commercial vendors
Cons: • Docker only• Containers are black boxes
sysdig
•Capture system events, filter them, run useful scripts• strace + tcpdump + lsof + Lua•Open Source
csysdig
•Capture system events, filter them, run useful scripts• strace + tcpdump + lsof + htop + iftop + Lua•Open Source•Nice curses UI
Design Goals
•Production-ready• Simple • lightweight
•Rich data•Native support for containers• Runs in a container
•Natural workflow
sysdig Architecture
Kernel
Container1
Docker
Container2
Docker
Container3
LXCAppApp
sysdig Architecture
Kernel
Container1
Docker
Container2
Docker
Container3
LXCAppApp
Instrumentation through kernel module
sysdig Architecture
Kernel
Container1
Docker
Container2
Docker
Container3
LXCAppApp
sysdig
Docker
Capture andanalysis
sysdig Architecture
Kernel
Container1
Docker
Container2
Docker
Container3
LXCAppApp
sysdig
Docker
(optionally) Save to a trace file
foo.scap
Viewing Network/App Activity
•cAdvisor•docker stats• iftop/tcpdump/tshark in the container•sysdig
Viewing File I/O Activity and Errors
•iotop/lsof in the container•sysdig
Topology
•Sysdig cloud
