Docker at scale with Mesos
Phạm Tuấn Anh Lead engineer @ VCCloud Platform Services
$ whoami
• Lead engineer at VCCloud Platform Services
• Build & work with large-scale/production platform services for about 6 years
• Services: Storage, DNS, CDN, DDoS Protection, Logging, Private PaaS
Agenda
1. What is Mesos?
2. How to run docker containers on Mesos
3. Lessons learned
4. Q&A
1. What is Mesos?
2. How to run docker containers on Mesos
3. Lessons learned
4. Q&A
https://blog.twitter.com/2016/overview-of-the-twitter-cloud-platform-compute
https://mesosphere.com/blog/2015/04/23/apple-details-j-a-r-v-i-s-the-mesos-framework-that-runs-siri/
Powered by Mesos:• Apple
• Atlassian
• CERN
• Cisco
• Cloudflare
• Foursquare
• Groupon
• HubSpot
• Netflix
• Opera
• Paypal
• Shopee
• Shopify
• Saleforce
• Uber
• Vimeo
• Verizon
• Yelp
• Xiaomi
http://mesos.apache.org/documentation/latest/powered-by-mesos/
Solomon Hykes, Founder & CTO of Docker, said at DockerCon EU 12/2014:
Mesos is the “gold standard” for large-scale production clusters running containers
Mesos is:
• Open source, top-level Apache project
• Proven at massive scale
• Multi-resource (CPU, RAM, Disk, GPU) scheduling
• Simplified operational model
• Cross platform
• Native Docker support
• PaaS: Marathon, Aurora, Kubernetes, Swarm
• Big Data: Hadoop, Spark, Storm
• Database: Cassandra, ArangoDB,
• ElasticSearch, Chronos, Jenkins, Kafka…
Mesos Frameworks:
1. What is Mesos?
2. How to run docker containers on Mesos
3. Lessons learned
4. Q&A
• Apache Aurora
• Google Kubernetes
• HubSpot Singularity
• Mesosphere Marathon
Features:
• Start, stop, scale, update, rollback apps
• Rolling deploy / restart
• Constraints
• Health checks
• App dependencies
• Highly available, no SPoF
1. What is Mesos?
2. How to run docker containers on mesos
3. Lessons learned
4. Q&A
– Immutable Infrastructure / CodeShip
“Some of the major challenges today when building infrastructure are predictability, scalability and automated recovery.”
• Container style
• Service discovery
• Networking
Lessons Learned
• Managing secrets
• App deployment
• Docker annoyances
Choose a container style:
• "thin" single-process container
• "fat" container
• sshd in container
• Container style
• Service discovery
• Networking
Lessons Learned
• Managing secrets
• App deployment
• Docker annoyances
Services Discovery
• Mesos-DNS:
• <app_id>.<app_group>.marathon.mesos
• record types: A / SRV
• simple & stateless
• Container style
• Service discovery
• Networking
Lessons Learned
• Managing secrets
• App deployment
• Docker annoyances
Networking
• Project Calico:
• IP-per-container
• Pure Layer 3
• Container style
• Service discovery
• Networking
Lessons Learned
• Managing secrets
• App deployment
• Docker annoyances
Managing Secrets
• ansible-vault
• Container style
• Service discovery
• Networking
Lessons Learned
• Managing secrets
• App deployment
• Docker annoyances
App deployment
• ansible-playbook
• marathon.json
• docker tag name = git commit hash
• 1 mesos cluster (3 VMs) per dev
• deploy to servers → deploy to marathon
marathon.json
Monitoring
• Container’s RAM used
Stateful Containers
• Local persistent volumes
• Container style
• Service discovery
• Networking
Lessons Learned
• Managing secrets
• App deployment
• Docker annoyances
Docker 👎
• restart / upgrade docker without stopping the running containers
• old images / tags cleanup
• container’s stdout / stderr log files
• docker pull
• docker hub
"Normal" container vs cloud-native container:
• No manual steps
• Service discovery friendly
• Config files should be optional
• Using env vars for config
1. What is Mesos?
2. How to run docker containers on Mesos
3. Lessons learned
4. Q&A
Links
• Mesos: http://mesos.apache.org/
• Marathon: https://mesosphere.github.io/marathon/
• Project Calico: https://www.projectcalico.org/
• Mesos-DNS: https://github.com/mesosphere/mesos-dns
• A Healthy Platform Checklist: https://gist.github.com/andy-pham/04dedff872b33c07a31277afb900950b
