![Page 1: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/1.jpg)
Ed Harrison and Neil JerramChristos Kozyrakis, Spike Curtis, Kapil Arya, Dan Osborne,
Connor Doyle, Niklas Nielsen, Tarak Parekh, Alex Pollitt
Mesos Networking
with Project Calico
![Page 2: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/2.jpg)
The State of Mesos Networking
Containers share the slave agent’s IP address
Containers can use any port on the agent
Service discovery using per-agent proxies
localhost:8888 on any agent redirects to a specific service
![Page 3: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/3.jpg)
This was OK Initially
For clusters where
– a single framework manages all services
– there are only a few, long-running services
– there is a single version of each service
![Page 4: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/4.jpg)
But it’s Problematic Now
For clusters where
– services are launched by tens of frameworks
– there are thousands of services with high churn
– multiple version of each service
prod/test/dev, US/EMEA/Asia, …
![Page 5: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/5.jpg)
Problem #1: Port Conflicts
If two apps want to use same port on an agent one fails to start
Alternative: port isolator enforces non-overlapping port ranges
service discovery problem for the app that does not get standard port
Alternative: bridged networking
service discovery problem for the app behind the bridge
![Page 6: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/6.jpg)
Problem #2: No Isolation
How do we stop a test app from connecting with a prod app?
How we isolate different users, services, or divisions?
How do we stop DoS attacks within the cluster?
![Page 7: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/7.jpg)
Problem #3: Service Discovery
How do multiple frameworks manage proxy settings?
How do clients know which version of a service is at each port?
Do we update the proxies in 10K agents every time a service starts?
![Page 8: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/8.jpg)
This makes no sense…
![Page 9: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/9.jpg)
Mesos Networking Redux
Per-container IP addresses
Routable within and, if needed, outside the cluster
No port conflicts
Network isolationBased on coarse-grain or fine-grain security policies
DNS-based service discoveryDiscovery using hostnames (A & SRV records, HTTP interface)
![Page 10: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/10.jpg)
Implementation
One feature set, many pluggable implementationsDifferent network virtualization technologies (L2 or L3)
Different IP address management schemes
Different DNS servers
First implementation based on Project Calico
L3-based network virtualization & isolation
Simple, scalable, open-source
![Page 11: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/11.jpg)
![Page 12: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/12.jpg)
IP
Service
Router
Router
Router
BGP BGP
IP
Service
IP
Service
IP
Service
IP
Service
IP
Service
IP
Service
IP
Service
Build the DC network like the Internet
![Page 13: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/13.jpg)
IP
Service
Router
Router
Router
BGP BGP
IP
Service
IP
Service
IP
Service
IP
Service
IP
Service
IP
Service
IP
Service
Mesos Agent
Build the DC network like the Internet
Mesos Agent
![Page 14: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/14.jpg)
Mesos Agent
Executor Namespace
Root Namespace
eth0
eth0 cali34
192.168.0.45
10.0.0.1
Executor Namespace
eth0 cali89
10.0.0.2
Linux Kernel Routing(you already have this!)default via 192.168.0.1 dev eth0 192.168.0.0/24 dev eth0 src 10.0.2.15 10.0.0.1/32 dev cali34 scope global10.0.0.2/32 dev cali89 scope global10.0.1.40/32 via 192.168.0.29 dev eth010.0.2.53/32 via 192.168.0.131 dev eth0
veth pair (kernel version 2.6.24+)
Containers on other agents
IP
Calico Data Plane
Containers on this agent
![Page 15: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/15.jpg)
Mesos Agent
Executor Namespace
Root Namespace
eth0
eth0 cali34
192.168.0.45
10.0.0.1
Executor Namespace
eth0 cali89
10.0.0.2
IP
Linux Kernel Filtering (iptables)(you already have this!)
Per-container distributed firewall
Calico Data Plane
![Page 16: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/16.jpg)
Mesos Agent
Executor Namespace
Root Namespace
eth0
eth0 cali34
192.168.0.45
10.0.0.1
Executor Namespace
eth0 cali89
10.0.0.2
IP
Felix
RouteReflectorBGP
Client
Calico Control Plane
![Page 17: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/17.jpg)
Mesos – Calico Integration
NetworkInfo protobuf
Networking isolator
Calico IP address management – IPAM (plug-in)
Calico network virtualizer (plug-in)
Master cleanup module
![Page 18: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/18.jpg)
Update
task state
Networking Workflow
Plug-in (Calico)AgentMasterFramework
IPAM
Networkvirtualizer
Get IP
Isolatormodule
Isolate (IP, policy)
Cleanupmodule
Launch task (NetworkInfo)Launch task (NetworkInfo)
Task update (NetworkInfo)
Task update (NetworkInfo)
Mesos module
Network plug-in
![Page 19: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/19.jpg)
message NetworkInfo {enum Protocol {
IPv4 = 1;IPv6 = 2;
}optional Protocol protocol = 1;
// Requested IP or assigned IP (on task update)optional string ip_address = 2;
// Network isolation group.repeated string groups = 3;
// To tag certain metadata to be used by Isolator/IPAM, e.g., rack, etc.optional Labels labels = 4;
};
NetworkInfo protobuf
![Page 20: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/20.jpg)
Mesos-DNS
MesosMaster
Agent Agent Agent Agent Agent…
MesosDNS
① Watch ZK formaster changes
② Pull task stateGenerate DNS records
③ DNS & HTTPbased discovery
nginx_prod.marathon.mesos 10.13.17.95
_nginx_prod._tcp.marathon.mesos 10.13.17.95:8181
![Page 21: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/21.jpg)
Networking Demo
Mesos cluster with 2 slaves agents
Launching 4 probe tasks
Each probe listens to port 9000
Each probe tries to reach all other probes
We want all 4 to launch successfully (no port conflicts)
We want to isolate them into two groups of 2 probes
![Page 22: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/22.jpg)
Networking Demo
![Page 23: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/23.jpg)
Roadmap
Code release (Mesos 0.25)
Integration with Mesosphere DCOS
Interfaces for coarse-grain and fine-grain isolation policies
Other plug-in implementations
Flexible task naming in Mesos-DNS
Network QoS
![Page 24: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/24.jpg)
Summary
Mesos networking features
Per-container IP addresses
DNS-based service discovery
Network isolation
1st implementation using Project Calico
Try it and contribute!
![Page 25: Mesos Networking with Project Calicoevents17.linuxfoundation.org/sites/events/files/slides... · 2015-10-07 · Ed Harrison and Neil Jerram Christos Kozyrakis, Spike Curtis, Kapil](https://reader034.vdocuments.net/reader034/viewer/2022042109/5e89de70bc0a4525af290a15/html5/thumbnails/25.jpg)
References
https://mesosphere.com/
http://www.projectcalico.org/
https://github.com/mesosphere/net-modules
https://github.com/mesosphere/mesos-dns