![Page 1: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/1.jpg)
Choosing the Right Framework for Running
Docker Containers in ProdPresented by: Josh Padnick
Phoenix DevOps
![Page 2: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/2.jpg)
Docker is a game-changer.
![Page 3: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/3.jpg)
Why Devs • Lightweight
Containers are just isolated processes. We can start a new container in seconds.
• Portable My Mac, the linux EC2 instance, and your Windows PC all run the exact same container.
• EcosystemI can easily share images, manage private images, and use “official” images for virtually all open source software.
![Page 4: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/4.jpg)
Why Devs • Squeeze More Resources out of a Single Server
Did you know this dirty secret of the Infrastructure-as-a-Service world?
85%
15%
In Use Free
SOURCE: http://radar.oreilly.com/2014/12/why-the-data-center-needs-an-operating-system.html
Typical Data Center Resource Utilization
![Page 5: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/5.jpg)
So can I run multiple containers in a single VM?
![Page 6: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/6.jpg)
VM-1 VM-2
Service A Service AService B
Service C
Service B
Something like this?
![Page 7: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/7.jpg)
Yes! But, well…
![Page 8: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/8.jpg)
The Gartner Tech HypecycleAny guesses where the “multi-container VM” paradigm is?
SOURCE: https://setandbma.wordpress.com/2012/05/28/technology-adoption-shift/
![Page 9: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/9.jpg)
My Take on ThisIn reality, the exact spot varies by team,
so this is a bit of a generalization.
![Page 10: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/10.jpg)
Today’s talk is about our options for that red dot.
For each option, we’ll cover:
• Pro’s • Con’s • When to use
![Page 11: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/11.jpg)
• Full-stack web-app engineer for 12+ years.
• Since I’ve worked with many different teams, I generally help accelerate the DevOps/AWS learning curve for teams.
• PhxDevOps Clients include: Intel, Infusionsoft, American Bible Society, CÜR Music, plus multiple startups and web design companies.
Josh Padnick
These slides are posted on http://joshpadnick.com
I help software teams scale their app using DevOps and AWS.http://PhoenixDevOps.com
@OhMyGoshJosh
My LinkedIn
My GitHub
Want to know more about building scalable apps on AWS? Check out a 12,000+ word article I wrote on how at http://bit.ly/1EtYRbL.
![Page 12: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/12.jpg)
Disclaimers• I have a bias toward AWS and may leave
out solutions from other IaaS providers such as Azure.
• The solutions we cover today are deep and diverse. This talk reflects my own experiences but your mileage may vary!
![Page 13: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/13.jpg)
Agenda• CoreOS in 60 seconds
• Theory of Multi-Container VM’s
• The Three Paradigms of Multi-Container VM’s
• Cover all the Major Solutions
![Page 14: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/14.jpg)
We only have 60 minutes. So we’ll move fast.
![Page 15: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/15.jpg)
CoreOS in 60 Seconds
![Page 16: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/16.jpg)
What is ? • It’s super stripped-down linux. You don’t even
get a package manager.
• The idea is you run everything as a container.
• CoreOS is based on ChromiumOS, which itself is based on Gentoo Linux.
• Uses systemd for init.
![Page 17: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/17.jpg)
CoreOS and This Presentation• Because CoreOS is “built for Docker”, many
solutions use it as their default OS.
• In reality, you can usually use any OS that runs Docker natively, but CoreOS is often the “recommended” linux distro for Docker.
![Page 18: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/18.jpg)
Theory ofMulti-Container VM’s
![Page 19: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/19.jpg)
What does it take to run this?
VM-1 VM-2
Service A Service AService B
Service C
Service B
![Page 20: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/20.jpg)
Docker BuilderWe need
somewhere to build our image.
![Page 21: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/21.jpg)
• If we build from a fresh environment each time, every Docker image/layer is downloaded from scratch.
• Ideally, our “Docker Builder” has pre-downloaded (“seeded”) all our most popular Docker images.
• In practice, this is managed by your build tool, like Jenkins, CircleCI, Shippable, etc.
Docker BuilderWe need
somewhere to build our image.
![Page 22: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/22.jpg)
Docker RegistryWe need
somewhere to store our built
images.
Docker BuilderWe need
somewhere to build our image.
![Page 23: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/23.jpg)
• Main options here are:
• Cloud
• Docker Hub
• Quay.io
• On-Premise
• Docker Trusted Registry (Paid)
• Docker Distribution (Free)
• Quay.io (Paid)
Docker RegistryWe need
somewhere to store our built
images.
Docker BuilderWe need
somewhere to build our image.
![Page 24: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/24.jpg)
Docker RegistryWe need
somewhere to store our built
images.
Automated DeploymentWe need a way to deploy our Docker image into the cluster.
Docker BuilderWe need
somewhere to build our image.
![Page 25: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/25.jpg)
Docker RegistryWe need
somewhere to store our built
images.
Automated DeploymentWe need a way to deploy our Docker image into the cluster.
Container SchedulingWe need something to decide which host will
launch our new container?
Docker BuilderWe need
somewhere to build our image.
![Page 26: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/26.jpg)
VM-1 VM-2
Service A Service AService B
Service C
Service B
![Page 27: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/27.jpg)
VM-1 VM-2
Service A Service AService B
Service C
Service B
“Increase Service C container count from 1 to 2.”
![Page 28: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/28.jpg)
VM-1 VM-2
Service A Service AService B
Service C
Service B
Service C
![Page 29: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/29.jpg)
Docker BuilderWe need
somewhere to build our image.
Docker RegistryWe need
somewhere to store our built
images.
Automated DeploymentWe need a way to deploy our Docker image into the cluster.
Container SchedulingWe need something to decide which host will
launch our new container?
![Page 30: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/30.jpg)
Docker RegistryWe need
somewhere to store our built
images.
Automated DeploymentWe need a way to deploy our Docker image into the cluster.
Container SchedulingWe need something to decide which host will
launch our new container?• One of the most important
considerations when choosing a host is “who’s got the memory and CPU I need?”
• But we also need to know who’s in a different Availability Zone / Data Center so we can achieve high fault tolerance.
Docker BuilderWe need
somewhere to build our image.
![Page 31: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/31.jpg)
Docker RegistryWe need
somewhere to store our built
images.
Automated DeploymentWe need a way to deploy our Docker image into the cluster.
Routing / Load BalancingWe need a way to route a
request to any of our containers.
Container SchedulingWe need something to decide which host will
launch our new container?
Docker BuilderWe need
somewhere to build our image.
![Page 32: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/32.jpg)
VM-1 VM-2
Service A Service AService B
Service C
Service B
![Page 33: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/33.jpg)
VM-1 VM-2
Service A Service AService B
Service C
Service B
GET ServiceB HTTP/1.1
![Page 34: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/34.jpg)
VM-1 VM-2
Service A Service AService B
Service C
Service B
Routing / Load Balancing Solution
![Page 35: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/35.jpg)
Docker RegistryWe need
somewhere to store our built
images.
Automated DeploymentWe need a way to deploy our Docker image into the cluster.
Routing / Load BalancingWe need a way to route a
request to any of our containers.
Container SchedulingWe need something to decide which host will
launch our new container?
Docker BuilderWe need
somewhere to build our image.
![Page 36: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/36.jpg)
Docker RegistryWe need
somewhere to store our built
images.
Automated DeploymentWe need a way to deploy our Docker image into the cluster.
Service DiscoveryWhen we launch new
containers, we need to tell our router they exist.
Routing / Load BalancingWe need a way to route a
request to any of our containers.
Container SchedulingWe need something to decide which host will
launch our new container?
Docker BuilderWe need
somewhere to build our image.
![Page 37: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/37.jpg)
VM-1 VM-2
Service A Service AService B
Service C
Service B
Routing / Load Balancing Solution
![Page 38: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/38.jpg)
VM-1 VM-2
Service A Service AService B
Service C
Service B
Routing / Load Balancing Solution
“Increase Service C container count from 1 to 2.”
![Page 39: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/39.jpg)
VM-1 VM-2
Service A Service AService B
Service C
Service B
Routing / Load Balancing Solution
Service C
![Page 40: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/40.jpg)
VM-1 VM-2
Service A Service AService B
Service C
Service B
Routing / Load Balancing Solution
Service C
![Page 41: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/41.jpg)
VM-1 VM-2
Service A Service AService B
Service C
Service B
Routing / Load Balancing Solution
Service C
![Page 42: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/42.jpg)
Docker RegistryWe need
somewhere to store our built
images.
Automated DeploymentWe need a way to deploy our Docker image into the cluster.
Service DiscoveryWhen we launch new
containers, we need to tell our router they exist.
Routing / Load BalancingWe need a way to route a
request to any of our containers.
Container SchedulingWe need something to decide which host will
launch our new container?
Docker BuilderWe need
somewhere to build our image.
![Page 43: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/43.jpg)
Docker RegistryWe need
somewhere to store our built
images.
Automated DeploymentWe need a way to deploy our Docker image into the cluster.
Auto-Restart Failed Containers
Something needs to know that a container failed and
auto-restart it.
Service DiscoveryWhen we launch new
containers, we need to tell our router they exist.
Routing / Load BalancingWe need a way to route a
request to any of our containers.
Container SchedulingWe need something to decide which host will
launch our new container?
Docker BuilderWe need
somewhere to build our image.
![Page 44: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/44.jpg)
Docker RegistryWe need
somewhere to store our built
images.
Automated DeploymentWe need a way to deploy our Docker image into the cluster.
Auto-Restart Failed Containers
Something needs to know that a container failed and
auto-restart it.
Extract Container LogsWe need a way to read logs
from all containers.Service Discovery
When we launch new containers, we need to tell
our router they exist.
Routing / Load BalancingWe need a way to route a
request to any of our containers.
Container SchedulingWe need something to decide which host will
launch our new container?
Docker BuilderWe need
somewhere to build our image.
![Page 45: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/45.jpg)
Docker RegistryWe need
somewhere to store our built
images.
Automated DeploymentWe need a way to deploy our Docker image into the cluster.
Extract Container LogsWe need a way to read logs
from all containers.
Monitor EverythingWe need to monitor cluster resources and individual
containers.
Auto-Restart Failed Containers
Something needs to know that a container failed and
auto-restart it.
Service DiscoveryWhen we launch new
containers, we need to tell our router they exist.
Routing / Load BalancingWe need a way to route a
request to any of our containers.
Container SchedulingWe need something to decide which host will
launch our new container?
Docker BuilderWe need
somewhere to build our image.
![Page 46: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/46.jpg)
Docker RegistryWe need
somewhere to store our built
images.
Automated DeploymentWe need a way to deploy our Docker image into the cluster.
Extract Container LogsWe need a way to read logs
from all containers.
Monitor EverythingWe need to monitor cluster resources and individual
containers.
Auto-Restart Failed Containers
Something needs to know that a container failed and
auto-restart it.
Service DiscoveryWhen we launch new
containers, we need to tell our router they exist.
Routing / Load BalancingWe need a way to route a
request to any of our containers.
Container SchedulingWe need something to decide which host will
launch our new container?
Docker BuilderWe need
somewhere to build our image.
![Page 47: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/47.jpg)
Does our cluster have “state”?• Yes!
• Router needs to know which containers are from which services.
• We need to know which hosts are actually in our cluster.
![Page 48: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/48.jpg)
Storing Cluster State• This topic alone warrants full books.
• One option for storing state is to simply use a database like PostgreSQL.
• But the more popular option is for each host in the cluster to store state in an eventually consistent way using a “consensus algorithm.” I call this a cluster datastore.
• The most popular such solutions are: etcd, consul, and zookeeper.
![Page 49: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/49.jpg)
Unit of Container Deployment• We need something that describes what kind of
container to deploy.
• This is typically a declarative file in either YAML or JSON that declares all aspects of our docker run command, whether 2+ containers are run together, etc.
![Page 50: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/50.jpg)
The Theory in Summary• Docker builder
• Docker registry
• Automated deployment
• Container scheduling
• Routing / Load Balancing
• Service discovery
• Auto-restart failed containers
• Logging
• Monitoring
• Cluster datastore
• Unit of Container Deployment
![Page 51: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/51.jpg)
Paradigms of Multi-Container VM’s
![Page 52: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/52.jpg)
Three Paradigms• Cluster Frameworks
• Platform-as-a-Service (PaaS)
• Data Center Operating Systems
![Page 53: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/53.jpg)
Paradigm #1:Cluster Frameworks
![Page 54: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/54.jpg)
The Big Idea• You control the infrastructure (e.g. AWS, Azure)
• You’re given an unopinionated set of primitives on top of which you can build your own solution.
• Primitives include launching containers, but not full deployment.
![Page 55: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/55.jpg)
Major Cluster Frameworks
+ Fleet
Docker Swarm
(We’ll cover each of these today)
![Page 56: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/56.jpg)
Paradigm #2:Platform-as-a-Service (PaaS)
![Page 57: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/57.jpg)
The Big Idea• You control the infrastructure (e.g. AWS, Azure)
• Install the PaaS tool on top of your own infrastructure.
• PaaS tool typically sits on top of a Cluster Framework.
• You’re not 100% sure how it works, but it solves your needs today and you can always deep dive later, or (hopefully) get commercial support.
![Page 58: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/58.jpg)
Major PaaS Solutions
(We’ll cover Deis in depth shortly)
https://github.com/remind101/empire
![Page 59: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/59.jpg)
Paradigm #3:Data Center Operating Systems
![Page 60: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/60.jpg)
The Big Idea• You control the infrastructure (e.g. AWS, Azure)
• You’re given an opinionated framework which has everything you need to deploy.
• You operate at the abstraction level of “cluster” and really don’t care when individual hosts die.
• These tend to be the most powerful, and the most complex.
![Page 61: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/61.jpg)
Major Data Center Operating System Frameworks
(We’ll cover each of these today)
( + ? )
![Page 62: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/62.jpg)
Hybrid Paradigms:Where A runs on B
![Page 63: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/63.jpg)
Hybrid Combo #1:Run Kubernetes on Mesos
![Page 64: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/64.jpg)
Hybrid Combo #2:Run Kubernetes as a Hosted Service
![Page 65: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/65.jpg)
Hybrid Combo #3:Run the Open Source PaaS “Empire” on
top of EC2 Container Service
https://github.com/remind101/empire
![Page 66: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/66.jpg)
Hybrid Combo #4:Use Docker Swarm as the UX to Mesos.
![Page 67: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/67.jpg)
Mentally Managing the Hybrids
• Don’t get too caught up on these exotic combinations.
• Focus first on one of the “non-hybrid” technologies.
• Then evaluate what can be run on top of your choice technology, and whether it will make life easier for you.
![Page 68: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/68.jpg)
The Major Solutions
![Page 69: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/69.jpg)
Disclaimers• We just don’t have enough time to cover
each solution in depth.
• We only get about 3 minutes per solution, so let’s get started!
![Page 70: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/70.jpg)
Cluster Framework Solution:
+ Fleet
![Page 71: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/71.jpg)
How It Works• Launch a CoreOS cluster on the IaaS platform of your
choice (e.g. AWS, Azure, VMWare, etc.)
• CoreOS comes with a CLI tool fleet that enables launching containers, but does not constitute a full deployment system.
• Best thought of as a set of primitives you can work with, not a full-fledged framework.
• Define systemd unit files to describe the Docker container you want to launch.
![Page 72: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/72.jpg)
Docker Builder Roll Your Own
Docker Registry Roll Your Own
Deployment: Scheduling Built into fleet, but no resource-aware scheduling
Deployment: Routing Roll Your Own
Deployment: Service Discovery Roll Your Own
Auto-Restart Failed Containers Built into Fleet
Monitoring Roll Your Own
Logging Roll Your Own
Cluster Data Store etcd
Unit of Deployment Systemd unit file
![Page 73: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/73.jpg)
Pro’s• Relatively mature/stable among Cluster Frameworks.
• Once you’ve setup etcd, everything else “just works”.
• RESTful API into fleet allows for easily building out your own custom solution.
• Fleet will auto-restart failed containers.
• Tagging cluster nodes allows for clever distribution of containers (e.g. across Availability Zones).
• CoreOS gives us a well-defined method for updating individual cluster nodes to the latest CoreOS.
• Commercial support available.
![Page 74: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/74.jpg)
Con’s• Setting up etcd can be painful.
• Fleet does not allow resource-aware scheduling, so containers may run out of resources.
• Fleet does not expose a primitive for “transferring” a container from one cluster node to another.
• No built-in way to monitor cluster-wide resource consumption.
• Not usable for a production cluster without significant setup overhead (e.g. setting up service discovery).
• Learning fleet ultimately requires learning systemd and discovering what fleet commands actually do.
![Page 75: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/75.jpg)
When To Use It• You want to learn the foundations of CoreOS.
• You want high customizability over your setup and can tolerate non-resource-aware scheduling.
• You’re willing to manually handle many operations such as launching additional containers.
![Page 76: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/76.jpg)
Cluster Framework Solution:
![Page 77: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/77.jpg)
How It Works• Launch at least 3 EC2 instances in AWS.
• Install the ECS agent on each node (or launch an AMI with the agent pre-installed).
• Cluster setup “just works”
• Define a “Task Definition” to describe how one or more Docker containers should be launched.
• Define a “Service” that launches one or more instances of the Task Definition, and ECS auto-deploys your Tasks (containers).
![Page 78: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/78.jpg)
![Page 79: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/79.jpg)
![Page 80: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/80.jpg)
![Page 81: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/81.jpg)
![Page 82: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/82.jpg)
![Page 83: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/83.jpg)
![Page 84: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/84.jpg)
Docker Builder Roll Your Own
Docker Registry Roll Your Own
Deployment: Scheduling Resource-aware, pluggable scheduler. Can be swapped w/ custom one.
Deployment: Routing Leverages AWS Elastic Load Balancers
Deployment: Service Discovery Built in to services
Auto-Restart Failed Containers Built in to services
Monitoring Basic monitoring included at cluster level.
Logging Roll Your Own
Cluster Data Store Zookeeper (but this is hidden to us)
Unit of Deployment Task Definition
![Page 85: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/85.jpg)
Pro’s• Very easy to set up.
• Simple UX.
• Low learning curve.
• Covers most of what you need out of the box, including built-in routing and service discovery.
• Presumably AWS will keep improving it.
• Supported via AWS.
![Page 86: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/86.jpg)
Con’s• Doesn’t support dynamic port mapping from container to host.
• Each service requires its own Elastic Load Balancer, which is $18/month. (Unless you’re willing to expose a service on a port other than 80/443)
• Supports rolling deployments provided you have a spare node to launch a new service instance on. Blue/Green deployments are claimed as a feature, but require out-of-band customization.
• Not recommended to leverage the existing Zookeeper cluster datastore in use, so you may have to run two cluster datastores (e.g Zookeeper + Consul).
• Use of “private subnets” requires two separate clusters, one for public and one for private.
![Page 87: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/87.jpg)
When To Use It• You use AWS and…
• You want to get up and running quickly with your Docker-based microservices framework.
• You want to run your monolith using containers today, knowing you can migrate to other cluster tech’s in the future.
• You want an official solution with official support.
• You want to minimize the number of vendors/tech’s you work with.
![Page 88: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/88.jpg)
Cluster Framework Solution:
Docker Swarm
![Page 89: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/89.jpg)
How It Works• You use “docker-machine” to launch multiple EC2
instances (or other VMs). Each EC2 instance is configured with the Docker daemon and the docker-swarm agent (which is just a container).
• You launch one or more “Swarm Masters”, one of which is the “master leader.” You use this to control your cluster.
• You can now run Docker CLI commands to launch containers on your cluster.
![Page 90: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/90.jpg)
![Page 91: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/91.jpg)
![Page 92: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/92.jpg)
Docker Builder Roll Your Own
Docker Registry Roll Your Own
Deployment: Scheduling Resource-aware, pluggable scheduler. Can be swapped w/ custom one.
Deployment: Routing Roll Your Own
Deployment: Service Discovery Roll Your Own
Auto-Restart Failed Containers Open GitHub Issue: https://github.com/docker/swarm/issues/599
Monitoring Roll Your Own
Logging Roll Your Own
Cluster Data Store Pluggable!See https://docs.docker.com/
swarm/discovery/ Unit of Deployment Docker container, or Docker compose manifest
![Page 93: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/93.jpg)
Pro’s• Use the Docker CLI you’ve come to know and love.
• You can run other Docker tools that call the old Docker CLI directly on top of Swarm and they will “just work”. Does this matter?
• Potentially simpler to program against compared to CoreOS fleet.
• Resource-aware scheduling.
• Open source event bus available allows for interesting possibilities in response to cluster events (esp. around service discovery).
• Official Docker solution.
![Page 94: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/94.jpg)
Con’s• Not yet recommended for production.
• My own experience with docker-machine and docker-swarm have been underwhelming in terms of stability.
• Other than “official Docker solution” and “use Docker CLI”, I don’t see any superior features to alternatives.
![Page 95: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/95.jpg)
When To Use It• For experiments and curiosity.
• Docker swarm is intriguing, but under heavy development and doesn’t yet present a clear value proposition compared to alternatives.
• But check back in 6 months, and it may be a solid contender. See Project Orca for an exciting opinionated take on the UX. https://youtu.be/8vSPpPSd00w?t=1h25m16s
![Page 96: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/96.jpg)
PaaS Solution:
![Page 97: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/97.jpg)
How Deis Works
Control Plane Component
Cluster
Control Plane Component
Control Plane Component
Data Plane Component
Data Plane Component
Routerper Host
Routerper Host
![Page 98: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/98.jpg)
How Deis Works
Control Plane Component
Cluster
Control Plane Component
Control Plane Component
Data Plane Component
Data Plane Component
Routerper Host
Routerper Host
Service A Service B Service B Service C
![Page 99: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/99.jpg)
Deis Workflow for Dev’s (basically, Heroku on your own infrastructure)
SOURCE: http://docs.deis.io/en/latest/understanding_deis/concepts/
![Page 100: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/100.jpg)
How Deis Works for Operators
SOURCE: http://docs.deis.io/en/latest/understanding_deis/architecture/
![Page 101: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/101.jpg)
Docker Builder Included!
Docker Registry Included!
Deployment: Scheduling Defaults to resource-unaware fleet. Pluggable schedulers are in tech-preview.
Deployment: Routing Included!
Deployment: Service Discovery Included!
Auto-Restart Failed Containers Included!
Monitoring Roll Your Own
Logging Built-in logspout means you can send your logs anywhere.
Cluster Data Store PostgreSQL + Ceph
Unit of Deployment Heroku buildpack, Dockerfile, or Docker image.
![Page 102: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/102.jpg)
Pro’s• Everything in one package, ready to go. Get up and
running pretty quickly.
• Nice workflow for dev’s
• Open source
• Great community
• Good paradigm of what you would eventually need to build.
• Commercial support available through Engine Yard.
![Page 103: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/103.jpg)
Con’s• Learning curve for operators can feel steep.
• When the PaaS fails, it’s time to start climbing the learning curve. For example, I once terminated a node, broke the ceph cluster and had to dig into the guts to figure out how to fix it.
• Deis’s architectural opinions may differ from your own.
![Page 104: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/104.jpg)
Data Center Operating Solution #1:
![Page 105: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/105.jpg)
How Mesos Works
Cluster
![Page 106: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/106.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
![Page 107: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/107.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
Our job is to store cluster state!
![Page 108: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/108.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
Mesos Master
Mesos Master
Mesos Master
![Page 109: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/109.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
Mesos Master
Mesos Master
Mesos Master
We make “resource offers” to “frameworks”.
![Page 110: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/110.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
Mesos Master
Mesos Master
Mesos Master
![Page 111: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/111.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
Mesos Master
Mesos Master
Mesos Master
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
We run “tasks”, which are Docker containers. We take our orders
from the Master.
![Page 112: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/112.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
Mesos Master
Mesos Master
Mesos Master
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
A Resource Offer:“Master, I have 2 CPU cores, 8 GB RAM,
and 25GB of disk space available!”
![Page 113: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/113.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
Mesos Master
Mesos Master
Mesos Master
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
A Resource Offer:“Master, why go for 2 CPUs when
I’ve got 3 CPUs and 10 GBs of RAM!”
![Page 114: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/114.jpg)
You’ve just seen Tier 1 of the Mesos resource scheduling algorithm.
![Page 115: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/115.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
Mesos Master
Mesos Master
Mesos Master
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
Master Forwards Resource Offers“I have received your offers and they will
be forwarded to whomever I please.”
![Page 116: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/116.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
Mesos Master
Mesos Master
Mesos Master
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
Framework:Chronos
Framework:Marathon
Framework:ApacheSpark
Framework:Cassandra
![Page 117: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/117.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
Mesos Master
Mesos Master
Mesos Master
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
Framework:Chronos
Framework:Marathon
Framework:ApacheSpark
Framework:Cassandra
Master Forwards Resource Offers“Marathon, I choose you first! I can offer
you 3 CPUs and 10 GB of RAM.”
![Page 118: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/118.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
Mesos Master
Mesos Master
Mesos Master
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
Framework:Chronos
Framework:Marathon
Framework:ApacheSpark
Framework:Cassandra
Frameworks Accept/Reject Offers:“Yawn. Pass.”
![Page 119: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/119.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
Mesos Master
Mesos Master
Mesos Master
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
Framework:Chronos
Framework:Marathon
Framework:ApacheSpark
Framework:Cassandra
Master Forwards Resource Offers“Chronos, surely you have need of resources. I offer you the same!”
![Page 120: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/120.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
Mesos Master
Mesos Master
Mesos Master
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
Framework:Chronos
Framework:Marathon
Framework:ApacheSpark
Framework:Cassandra
Frameworks Accept/Reject Offers:“Let’s do this.
I need 1 GB of RAM and 1 CPU core.”
![Page 121: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/121.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
Mesos Master
Mesos Master
Mesos Master
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
Framework:Chronos
Framework:Marathon
Framework:ApacheSpark
Framework:Cassandra
Master Schedules Task on a Slave“It shall be so!
Agent #22, you shall run this task.”
![Page 122: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/122.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
Mesos Master
Mesos Master
Mesos Master
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
Framework:Chronos
Framework:Marathon
Framework:ApacheSpark
Framework:Cassandra
Slave Receives the Task:“Strength and honor, sire!”
![Page 123: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/123.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
Mesos Master
Mesos Master
Mesos Master
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
Framework:Chronos
Framework:Marathon
Framework:ApacheSpark
Framework:Cassandra
Slave starts running the task as a docker container.
![Page 124: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/124.jpg)
How Mesos Works
Zookeeper
Cluster
Zookeeper Zookeeper
Mesos Master
Mesos Master
Mesos Master
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
MesosSlave
(aka Agent)
Framework:Chronos
Framework:Marathon
Framework:ApacheSpark
Framework:Cassandra
![Page 125: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/125.jpg)
Docker Builder Roll Your Own
Docker Registry Roll Your Own
Deployment: Scheduling Resource-aware scheduler. Can run other “frameworks” side by side.
Deployment: Routing Roll Your Own
Deployment: Service Discovery Roll Your Own
Auto-Restart Failed Containers Included!
Monitoring Roll Your Own
Logging Roll Your Own
Cluster Data Store Zookeeper
Unit of Deployment Mesos Task (which will usually be a Docker container, usually submitted through Marathon.
![Page 126: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/126.jpg)
But wait, there’s more!• Setting up Mesos involves coordinating many
different moving pieces.
• Also, there’s no immediate way to gain a cluster-wide view of total memory/CPU/disk space use.
• Also, the learning curve can be steep.
![Page 127: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/127.jpg)
Mesosphere DCOS is meant to solve these problems.
• Offers “turn-key” setup (though the setup itself is not really production-grade).
• Offers a fancy UI for viewing cluster resource usage.
• Offers a special CLI for installing frameworks with 1 command.
• It’s very much in active development and would work best with a Mesosphere support plan.
![Page 128: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/128.jpg)
Pro’s• I find the Mesos abstraction the most intuitive when it
comes to managing cluster resources.
• Scalability is off the charts. Verizon, Siri, Yelp, Twitter and OpenTable all use Mesos.
• Growing community.
• Multiple “frameworks” already supported such as Apache Spark and Cassandra.
• Solomon Hykes called it the “gold standard” for running Docker containers in a cluster.
![Page 129: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/129.jpg)
Con’s• It can take weeks to setup if you need to do it right.
• The learning curve for dev’s is manageable, but for operators there are many moving pieces.
• There are certain edge cases that are rare but that would affect cluster performance over time.
• If you want to run Mesos on CoreOS, either you need to violate the CoreOS way, or run Mesos Master / Slave (Agent) in docker containers which is officially not recommended.
![Page 130: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/130.jpg)
When To Use It• You’re running multiple microservices, and you anticipate
significant scale.
• You want to squeeze as much possible utilization out of your large cluster as possible.
• You’re ready to adopt the cluster as the primary abstraction and expect to co-mingle prod and dev, multiple services, and multiple frameworks.
• Note: Smallest company I met at MesosCon was ~60 employees. That is probably the lower limit of company size before Mesos makes sense (IMO).
![Page 131: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/131.jpg)
Mesos + Docker Swarm• At MesosCon (August 2015), Docker showed
Docker Swarm as the CLI-based way to control Mesos deployments.
![Page 132: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/132.jpg)
Data Center Operating Solution #2:
![Page 133: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/133.jpg)
Disclaimers• Mr. Padnick may or may not have any
actual real-world experience with Kubernetes but felt it necessary to include it here for the sake of completeness.
![Page 134: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/134.jpg)
Kubernetes Pods• A pod is a group of docker containers that
should be run together.
Pod
Web Server
Content Management
Server
SOURCE: Illustrations reproduced from https://www.youtube.com/watch?v=Fcb4aoSAZ98
![Page 135: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/135.jpg)
Kubernetes Labels• A label is a set of key-value pairs that attach to allow Kubernetes
to identify groups of pods.
• Concept of labels is baked into most APIs.
Pod
SOURCE: Illustrations reproduced from https://www.youtube.com/watch?v=Fcb4aoSAZ98
FE
Pod
BI, FE
Pod
v2
![Page 136: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/136.jpg)
Kubernetes Replication Controllers
• A replication controller is a definition: “I want to run this pod 5 times.”
• If one of the pods fails, Kubernetes will auto-restart a new one.
SOURCE: Illustrations reproduced from https://www.youtube.com/watch?v=Fcb4aoSAZ98
Pod
v1
Pod
v1
ReplicationController
#Pods = 2 Label selector: v1
![Page 137: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/137.jpg)
Kubernetes Cluster
Node
Kubernetes Masteretcd
API Server Controller Manager Server
Scheduler Server
Kubernetes Masteretcd
API Server Controller Manager Server
Scheduler Server
kubeletagentproxy
Pod
FE
Pod
v2
Pod
v1
Node
Node
Node
Node
Node
Node
![Page 138: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/138.jpg)
Docker Builder Roll Your Own
Docker Registry Roll Your Own
Deployment: Scheduling Resource-aware scheduler.
Deployment: Routing Included!
Deployment: Service Discovery Included!
Auto-Restart Failed Containers Included!
Monitoring Optimal support with Google Cloud Engine. Limited support for others.
Logging Optimal support with Google Cloud Engine. Limited support for others.
Cluster Data Store etcd
Unit of Deployment Pod
![Page 139: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/139.jpg)
Pro’s• Produced by google.
• Very well-documented.
• Open source.
• The “successor” to CoreOS + Fleet. Commercially supported by CoreOS as tectonic.
• If run in Google Cloud Engine, can potentially be quite powerful.
![Page 140: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/140.jpg)
Con’s• Preferential support for Google Cloud Engine.
• Produced by Google but not necessarily the exact system Google uses to run its own cluster (though based on it).
• I may or may not be aware of additional issues.
![Page 141: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/141.jpg)
When To Use It• You’re running Google Cloud Engine
• You have prior experience from working at Google
• I may or may not be aware of add’l use cases.
![Page 142: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/142.jpg)
Mesos + Kubernetes• You can run kubernetes on top of Mesos as an
alternative to Marathon.
![Page 143: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/143.jpg)
Final Thoughts
![Page 144: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/144.jpg)
Closing Thoughts• To get started quickly, choose EC2 Container
Service.
• To get a feel for the core technologies, choose a PaaS like Deis and slowly learn CoreOS.
• To run multi-container VM’s at (potentially huge) scale, choose Mesos.
• There are many more “satellite” projects I didn’t cover solving unique problems!
![Page 145: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/145.jpg)
Now have fun and docker on!
![Page 146: Choosing the Right Framework for Running Docker Containers in Prod](https://reader037.vdocuments.net/reader037/viewer/2022092623/58758bcb1a28ab901c8b5d09/html5/thumbnails/146.jpg)
Q&A