mesos networking with project calico · mesos networking redux per-container ip addresses routable...
TRANSCRIPT
![Page 1: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/1.jpg)
Ed Harrison and Neil JerramChristos Kozyrakis, Spike Curtis, Kapil Arya, Dan Osborne,
Connor Doyle, Niklas Nielsen, Tarak Parekh, Alex Pollitt
Mesos Networking
with Project Calico
![Page 2: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/2.jpg)
The State of Mesos Networking
Containers share the slave agent’s IP address
Containers can use any port on the agent
Service discovery using per-agent proxies
localhost:8888 on any agent redirects to a specific service
![Page 3: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/3.jpg)
This was OK Initially
For clusters where
– a single framework manages all services
– there are only a few, long-running services
– there is a single version of each service
![Page 4: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/4.jpg)
But it’s Problematic Now
For clusters where
– services are launched by tens of frameworks
– there are thousands of services with high churn
– multiple version of each service
prod/test/dev, US/EMEA/Asia, …
![Page 5: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/5.jpg)
Problem #1: Port Conflicts
If two apps want to use same port on an agent one fails to start
Alternative: port isolator enforces non-overlapping port ranges
service discovery problem for the app that does not get standard port
Alternative: bridged networking
service discovery problem for the app behind the bridge
![Page 6: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/6.jpg)
Problem #2: No Isolation
How do we stop a test app from connecting with a prod app?
How we isolate different users, services, or divisions?
How do we stop DoS attacks within the cluster?
![Page 7: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/7.jpg)
Problem #3: Service Discovery
How do multiple frameworks manage proxy settings?
How do clients know which version of a service is at each port?
Do we update the proxies in 10K agents every time a service starts?
![Page 8: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/8.jpg)
This makes no sense…
![Page 9: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/9.jpg)
Mesos Networking Redux
Per-container IP addresses
Routable within and, if needed, outside the cluster
No port conflicts
Network isolationBased on coarse-grain or fine-grain security policies
DNS-based service discoveryDiscovery using hostnames (A & SRV records, HTTP interface)
![Page 10: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/10.jpg)
Implementation
One feature set, many pluggable implementationsDifferent network virtualization technologies (L2 or L3)
Different IP address management schemes
Different DNS servers
First implementation based on Project Calico
L3-based network virtualization & isolation
Simple, scalable, open-source
![Page 11: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/11.jpg)
![Page 12: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/12.jpg)
IP
Service
Router
Router
Router
BGP BGP
IP
Service
IP
Service
IP
Service
IP
Service
IP
Service
IP
Service
IP
Service
Build the DC network like the Internet
![Page 13: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/13.jpg)
IP
Service
Router
Router
Router
BGP BGP
IP
Service
IP
Service
IP
Service
IP
Service
IP
Service
IP
Service
IP
Service
Mesos Agent
Build the DC network like the Internet
Mesos Agent
![Page 14: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/14.jpg)
Mesos Agent
Executor Namespace
Root Namespace
eth0
eth0 cali34
192.168.0.45
10.0.0.1
Executor Namespace
eth0 cali89
10.0.0.2
Linux Kernel Routing(you already have this!)default via 192.168.0.1 dev eth0 192.168.0.0/24 dev eth0 src 10.0.2.15 10.0.0.1/32 dev cali34 scope global10.0.0.2/32 dev cali89 scope global10.0.1.40/32 via 192.168.0.29 dev eth010.0.2.53/32 via 192.168.0.131 dev eth0
veth pair (kernel version 2.6.24+)
Containers on other agents
IP
Calico Data Plane
Containers on this agent
![Page 15: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/15.jpg)
Mesos Agent
Executor Namespace
Root Namespace
eth0
eth0 cali34
192.168.0.45
10.0.0.1
Executor Namespace
eth0 cali89
10.0.0.2
IP
Linux Kernel Filtering (iptables)(you already have this!)
Per-container distributed firewall
Calico Data Plane
![Page 16: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/16.jpg)
Mesos Agent
Executor Namespace
Root Namespace
eth0
eth0 cali34
192.168.0.45
10.0.0.1
Executor Namespace
eth0 cali89
10.0.0.2
IP
Felix
RouteReflectorBGP
Client
Calico Control Plane
![Page 17: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/17.jpg)
Mesos – Calico Integration
NetworkInfo protobuf
Networking isolator
Calico IP address management – IPAM (plug-in)
Calico network virtualizer (plug-in)
Master cleanup module
![Page 18: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/18.jpg)
Update
task state
Networking Workflow
Plug-in (Calico)AgentMasterFramework
IPAM
Networkvirtualizer
Get IP
Isolatormodule
Isolate (IP, policy)
Cleanupmodule
Launch task (NetworkInfo)Launch task (NetworkInfo)
Task update (NetworkInfo)
Task update (NetworkInfo)
Mesos module
Network plug-in
![Page 19: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/19.jpg)
message NetworkInfo {enum Protocol {
IPv4 = 1;IPv6 = 2;
}optional Protocol protocol = 1;
// Requested IP or assigned IP (on task update)optional string ip_address = 2;
// Network isolation group.repeated string groups = 3;
// To tag certain metadata to be used by Isolator/IPAM, e.g., rack, etc.optional Labels labels = 4;
};
NetworkInfo protobuf
![Page 20: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/20.jpg)
Mesos-DNS
MesosMaster
Agent Agent Agent Agent Agent…
MesosDNS
① Watch ZK formaster changes
② Pull task stateGenerate DNS records
③ DNS & HTTPbased discovery
nginx_prod.marathon.mesos 10.13.17.95
_nginx_prod._tcp.marathon.mesos 10.13.17.95:8181
![Page 21: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/21.jpg)
Networking Demo
Mesos cluster with 2 slaves agents
Launching 4 probe tasks
Each probe listens to port 9000
Each probe tries to reach all other probes
We want all 4 to launch successfully (no port conflicts)
We want to isolate them into two groups of 2 probes
![Page 22: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/22.jpg)
Networking Demo
![Page 23: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/23.jpg)
Roadmap
Code release (Mesos 0.25)
Integration with Mesosphere DCOS
Interfaces for coarse-grain and fine-grain isolation policies
Other plug-in implementations
Flexible task naming in Mesos-DNS
Network QoS
![Page 24: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/24.jpg)
Summary
Mesos networking features
Per-container IP addresses
DNS-based service discovery
Network isolation
1st implementation using Project Calico
Try it and contribute!
![Page 25: Mesos Networking with Project Calico · Mesos Networking Redux Per-container IP addresses Routable within and, if needed, outside the cluster No port conflicts Network isolation Based](https://reader033.vdocuments.net/reader033/viewer/2022053018/5f1e97a503024f05150161bf/html5/thumbnails/25.jpg)
References
https://mesosphere.com/
http://www.projectcalico.org/
https://github.com/mesosphere/net-modules
https://github.com/mesosphere/mesos-dns