build high-performance, scalable, distributed applications with stacks of containers. jerome...
DESCRIPTION
Build High-Performance, Scalable, Distributed Applications with Stacks of Containers. Jerome petazzoniTRANSCRIPT
Build high-performance, scalable, distributed applications with
stacks of containers
Добрый день
Jérôme Petazzoni(@jpetazzo)
Grumpy French DevOps- Go away or I will replace you
with a very small shell script
Goal in life:run everything in containers- Docker-in-Docker
- VPN-in-Docker
- KVM-in-Docker
- Xorg-in-Docker
- etc
outline
Outline
Why containers?Containers vs Virtual MachinesHigh speed communicationService discovery and load balancingConclusions
why should we use
containers?
Packaging format
It's easier to build a container image than a traditional distro package (.deb, .rpm...)- Dockerfile
Application dependencies are isolated from the host- application can use Python X even if the host has Python Y
- application can use distro X even if the host has distro Y
Downside: increased disk usage- but disks are cheap
Service abstraction*
Service installation “docker pull”→- no more dependency issues
Service start “docker run”→- no more fiddling with language-specific wrappers
Service stop “docker stop/kill”→- no more runaway processes
*If using a standard container format, e.g. Docker
Service abstraction
Service check “docker inspect”→- no more “is this thing really running or not?”
Service reset “docker run”→- discard copy-on-write layer to return to original state
- no more “oops, I broke it, how do I fix it?”
- doesn't work in all situations, e.g. data corruption(but recovery is easier because creating test copies is cheap & fast)
No overhead*
CPU performance= native performance
Memory performance = a few % shaved off for (optional) accounting
Network and disk I/O performance = small overhead; can be reduced to zero
*May require tuning!
Immutable infrastructure
What's an immutable infrastructure?- re-create images each time you change a line of code
- prevent (or track) modifications of running images
Why is it useful?- no more deviant servers after manual upgrades
- no more “oops, how do we rollback?” after catastrophic upgrade
- easier security audit (inspect images at rest)
How containers can help?- container images are easier to create and manage than VM images
Micro-service architecture
What's a micro-service architecture?- decompose big application into many small services
Why is it useful?- it's easier to upgrade/refactor/replace a small service
- encourages to have many small teams*, each owning a service(*small teams are supposedly better, see Jeff Bezos “two-pizza rule”)
How containers can help?- problem: 10 micro-services instead of 1 big application
= 10x more work to deploy everything
- solution: need extremely easy deployment; hello containers!
containersvs
virtual machines
Virtual Machines
Emulate CPU instructions(painfully slow)
Emulate hardware (storage, network...)(painfully slow)
Run as an userland process on top of a kernel(painfully slow)
Virtual Machines
Use native CPU(fast!)
Paravirtualized storage, network...(fast, but higher resource usage)
Run on top of an hypervisor(faster, but still some overhead)
Containers
Processes isolated from each otherVery little extra code path(in many cases, it's comparable to UID checking)
Virtual Machines vs Containers
Native CPUParavirtualized devicesHypervisor
Native CPUNative syscallsNative kernel
Inter-VM communication
Strong isolation, enforced by hypervisor + hardware- no fast-path data tranasfer between virtual machines
- yes, there are PCI pass-throughs and things like xenbus,but that's not easy to use, very specific, not portable
Most convenient method: network protocols (L2/L3)But: huge advantage from a security POV
Inter-container communication
Tunable isolation- each namespace can be isolated or shared
Allows normal Unix communication mechanisms- network protocols on loopback interface
- UNIX sockets
- shared memory
- IPC...
Reuse techniques that we know and love (?)
high speed communication for containers
Shared localhost
Multiple containers can share the same “localhost”(by reusing the same network namespace)
Communication over localhost is very very fastAlso: localhost is a well-known address
Shared filesystem
A directory can be shared by multiple containers(by using a bind-mount)
That directory can contain:- named pipes (FIFOs)
- UNIX sockets
- memory-mapped files
Bind-mount = zero overhead
Shared IPC
Multiple containers can share IPC resources(using the special IPC namespace)
Semaphores, Shared Memory, Message Queues...Is anybody still using this?
Host networking
Containers can share the host's network stack(by reusing its network namespace)
They can use the host's interfaces without penalty(high speed, low latency, no overhead!)
Native performance to talk with external containers
Host filesystems
Containers can share a directory with the hostExample: use fast storage (SAN, SSD...) in container- mount it on the host
- share it with the container
- done!
Native performance to use I/O subsystem
Device nodes
Containers can use the host's devices (if allowed)Access is granted through “devices” control groupPerformance (throughput, latency, CPU usage)...is the same as using the device on the host
Examples:- /dev/sd*: raw block device in container
- /dev/video*: GPU in container
- /dev/kvm: VM in container
- and more!
service discovery,
load balancing
Service discovery is not a new problem
Usually not important for …- small architectures
- low server count
- static deployments
But with containers...- container count can be very high
- deployments are very dynamic
you need service
discovery
Name resolution
Provide custom /etc/hosts in container(or give a custom DNS resolver to the container)
Easy to integrate in code (just connect to e.g. “db”)But:- no way to push changes to the application
- some libraries won't re-resolve
- might need to restart containers when service address changes
- service must run on well-known port
Environment variables (or SRV records)
Solves the “well-known port” requirementHarder to integrate in codeDoesn't solve the other problems
In-app discovery with config DB
Connect to zookeeper/etcd to find service locationWatch zookeeper/etcd for changesIf change occurs, reconnect
In-app discovery with config DB
Works well, but requires deep code changesIf you support multiple languages, it's a lot of workTransitioning from a system to another is hard
ambassadors
container host
database containerI'm frontdb!
web containerI want to talk to frontdb!
loca
lco
nnec
t
database host web host
database containerI'm frontdb!
web containerI want to talk to frontdb!
wiring containerI actually talk to frontdb!
wiring containerI pretend I'm frontdb!
loca
lco
nnec
t local
connect
?
database host web host
database containerI'm frontdb!
web containerI want to talk to frontdb!
wiring containerI actually talk to frontdb!
wiring containerI pretend I'm frontdb!
loca
lco
nnec
t local
connect
?
database host web host
database containerI'm frontdb!
web containerI want to talk to frontdb!
wiring containerI actually talk to frontdb!
wiring containerI pretend I'm frontdb!
loca
lco
nnec
t local
connect
UNICORNS
But! We are adding extra layers!
Yes.But service/ambassador communication can be:- over locahost
- over UNIX socket (even better but not always possible)
- over iptables or IPVS (to bypass the ambassador process)
Ambassador implementations
A few examples- Registrator
https://github.com/progrium/registrator
- Grand Ambassadorhttps://github.com/cpuguy83/docker-grand-ambassador
- AirBNB SmartStackhttps://github.com/airbnb/nervehttps://github.com/airbnb/synapse
Or roll your own- some HA KV store + HAProxy, stunnel, iptables, IPVS...
- serverless, ad-hoc discovery (avahi, multicast...)
conclusions
Thanks to containers, we can...
Build, ship, and run our applications more easilyDecouple application code from “plumbing”(service discovery, load balancing)
Which allows to implement micro-service archtectures efficiently (hopefully without the usual drawbacks!)
Which yields higher agility, shorter dev cycles, etc.Win!
containerize!
thank you!questions?