mongodb world 2016: scaling mongodb with docker and cgroups
TRANSCRIPT
Scaling MongoDB with Docker and cgroups
Marco Bonezzi
Technical Services Engineer, MongoDB
@marcobonezzi
#MDBW16
About the speaker I am Marco Bonezzi, TSE at MongoDB
TSE = helping customers with a variety of issues
Based in Dublin, Ireland Experience in distributed systems, high availability:
#MDBW16
How many of you have ever... 1. … manually deployed a MongoDB replica sets or sharded clusters?
2. ... had issues with resource allocation?
3. ... used Docker?
4. … used MongoDB running on Docker?
#MDBW16
We know how it feels…
Different architectures in
Development and Production
Co-located MongoDB processes
Production != docker run mongodb
#MDBW16
Whatarethepainpoints?Deployment
Complex architectures and deployment patterns Resource Contention
Multiple instances competing for resources MongoDB & Docker
Combining both: network configuration, container location, other
#MDBW16
Howtosolvethis?Deployment
Using predefined cluster patterns Replicating environments
Resource Control Setting limits to key resources
MongoDB & Docker Create once, deploy everywhere Deploy patterns, not processes
Orchestration
Resource Management
Automate for scaling
#MDBW16
AboutthistalkPatterns for successful deployments
Difference between success and failure Orchestrating MongoDB with Docker
MongoDB cluster on AWS with containers Patterns with Swarm and Compose
Managing container resources with cgroups Benefits of cgroups in a MongoDB cluster
PS
S
#MDBW16
Redundancy and fault tolerance Deploy an odd number of voting members Members ó Majority required ó Fault tolerance
High availability and resource colocation Single member of a replica set / server
Shards as Replica Set
Ideally: primary / secondary / secondary
Deployment patterns: Replica Set and Sharded Clusters
Server 3
Server 2
Server 1
mongos
Primary
Primary RS1
Secondary Secondary
Secondary RS2
Secondary RS3
Secondary RS1
Primary RS2
Secondary RS3
Secondary RS1
Secondary RS2
Primary RS3
mongos mongos
cfgsvr1 cfgsvr2 cfgsvr3
#MDBW16
Docker • Noun: a person employed in a port to load and unload ships (from “what
is docker” on Google)
Containers: Isolated process in userspace Application + dependencies Shared kernel and libraries Can run on any infrastructure (or cloud)
www.docker.com
#MDBW16
44% of orgs adopting
microservices
Why use Docker?
41% want application
portability
13x improvement in
release frequency
62% MTTR on software
issues
60% Using Docker to migrate to cloud
Reason to run containers:
SPEED Microservices architectures Efficiency Cloud
(TheDockerSurvey,2016)
#MDBW16
Goodnews:Dockercanhelpwiththis!
Orchestration Coordinate MongoDB containers to deploy a recommended deployment pattern
Resource Control Sizing each instance (and each cluster) to avoid resource contention issues
#MDBW16
Orchestrating MongoDB with Docker How can we use Docker for MongoDB deployments?
How can we deploy these patterns using Docker containers? Why should we use Docker?
Our recipe:
#MDBW16
Dockerecosystem
Provisioning and managing your Dockerized hosts Native clustering: turns a pool of Docker hosts into a single, virtual Docker host.
Define a multi-container application with all of its dependencies in a single file
S
#MDBW16
WhyDockerSwarm?
5x faster than Kubernetes to spin up
a new container
7x faster than Kubernetes to list all
running containers
Evaluating Container Platforms at Scale 1000 EC2 instances in a cluster What is their performance at scale? Can they operate at scale? What does it take to support them at scale?
https://medium.com/on-docker/evaluating-container-platforms-at-scale-5e7b44d93f2c#.k2fxds8c2
hGps://www.docker.com/survey-2016
#MDBW16
swarm-node-2
swarm-node-3
swarm-node-1
Swarm multi-host networking How each mongod process connects to other processes outside of the host? • Swarm overlay container-to-container network • Using the hostname defined in the Compose file
#MDBW16
Swarm filters to build our patterns Constraint filters
Mark each mongod container with a label:
“role=mongod” “replset=rs1”
#MDBW16
Affinity filters Prevent multiple RS members on the same host:
"affinity:replset!=rs1”
swarm-node-1
swarm-node-3
swarm-node-2
Affinity filters for container distribution
#MDBW16
Road to container success Deploying containers to the right node is not enough…
Next step: Resource control on each swarm cluster node using cgroups
Maritime New Zealand
#MDBW16
Resource control with cgroups and Docker Simple parameters to add to docker run or compose:
--cpu-shares
--cpuset-cpus
--memory
--blkio-weight --net
#MDBW16
MongoDB Memory usage in 3.2 with WiredTiger
MongoDB Memory: mongod process: connections, aggregations, mapReduce, etc WiredTiger cache: (0.6 x total memory) – 1 GB
Total = mem(mongod) + mem(WiredTiger cache)
WiredTiger cache
mongod
mongod memory
#MDBW16
cgroup!memory_limit!
Process memory with containers and cgroups
WiredTiger cache
mongod
mongod memory
total memory (seen from mongod process) !
Inside the container • Can see total memory and not
memory limit WiredTiger cache: • memory_limit *0.6
#MDBW16
cgroups: Memory and CPU limits
mongod! mongod! mongod! mongod!
MEM = (TOTAL_MEM - OS_MEM) / NUM_MONGOD WiredTiger cache = (MEM * 0.6) CPUSET= 0, 1, …, MAX_CPU_CORES
mongod!
mongod!
mongod!
mongod!
#MDBW16
MongoDB on Docker + cgroups: Memory
WiredTiger cache: 60% of the container memory limit (for each mongod)
Compose: mem_limit!Docker Engine: --memory !
WiredTiger cache !
!!!!!!
mongod memory !
WiredTiger cache !
!!!!!
mongod memory !
WiredTiger cache !
!!!!!!
mongod memory !
WiredTiger cache !
!!!!!!
mongod memory !
OS Memory !
#MDBW16
MongoDB on Docker + cgroups: CPU
mongod (and mongos) mapped to 1+ core Compose: cpuset!
Docker Engine: --cpuset-cpus!
mongod!rs1a !
mongod!rs1a !
mongod!rs2b !
mongod!rs2b !
core0 ! core1 ! core2 ! core3 !
OS ! OS !mongod cfgsrv
-c !mongos !
core4 ! core5 ! core6 ! core7 !
--cpuset-mems (NUMA) !
#MDBW16
Understanding resource usage:
• docker top rs1a !
• docker stats rs1a !
Container stats available via Docker remote API:
GET /containers/(id)/stats Also available from docker-py:
http://docker-py.readthedocs.org/en/latest/api/#stats
Resource usage with Docker
#MDBW16
Resource usage with Docker
Multiple statistics for each container: Memory limit and usage, CPU (per core level), Network, Disk
Useful to combine with MongoDB metrics (like db.serverStatus())
#MDBW16
Creating a Swarm cluster on AWS to deploy MongoDB
#MDBW16
Creating a Swarm cluster on AWS to deploy MongoDB
DEMO!
Configure docker-machine with ec2 driver (AWS)
Deploy discovery service for Swarm Master
Deploy AWS instances for: • Swarm master • Swarm worker nodes
Connect to the Swarm master
Define compose file for deployment
Define Swarm filters and constraints and cgroup
limits
Deploy the environment with a single command using the compose file
Configure our MongoDB sharded cluster using
Cloud Manager API
#MDBW16
What our dockerized sharded cluster looks like…
rs1a !
rs1c !rs1b ! rs2b !
rs2a !
rs2c ! rs3b !
rs3a !
rs3c !
mongos !
cfgsvr-a !
cfgsvr-b !
cfgsvr-c !
dockerC1 ! dockerC2 ! dockerC3 !
PRIMARY !SECONDARY !MONGOS !
Summary
#MDBW16
Advantages of using MongoDB with Docker
Speed: testing and deploying cluster patterns easily Build once, deploy everywhere
Control: Resource control and utilization
Key to success with containers
Agility: Microservices architectures Making change less expensive
Flexibility: Multi vendor cloud opportunities
AWS, Azure, Google, IBM, CloudFoundry
PS
S
#MDBW16
How successful customers use MongoDB with Docker
• Case Studies @hGps://www.mongodb.com/blog
• Whitepaper:
“Enabling Microservices – Containers & Orchestration Explained” https://www.mongodb.com/collateral/microservices-containers-and-orchestration-explained
#MDBW16
Now it’s YOUR turn
Share with us your use case of MongoDB & Docker: http://bit.do/DockerMongoDB
@marcobonezzi
You can actually try this at home: https://github.com/sisteming/mongo-swarm
