docker multi host networking, rachit arora, ibm
TRANSCRIPT
Docker Multi Host NetworkingRachit Arora , IBM
BigInsights on Cloud
Agenda Docker networking – overview Native Multi Host networking (Experimental) Software defined networks SDNs Multi Host networking options
OpenVSwtich Weave Flannel , SocketPlane Etc..
Goals Make "network" a first class object Provide a Pluggable networking stack Span networks across multiple hosts Support multiple platforms
libnetwork
Updated Networking Stack in Docker
Create Networks using the Docker CLI
Multi-host Networking
Services UI
SDNs value proposal
SDNs proposes network solutions embracing cloud paradigms Massively multi-tenant
Thousands tenants, massively scalable Easy & fast (de)provisioning
Infra as code, API centric Infrastructure agnostic
L3, does not stick with lower levels (physical designs, vlans & co) Decouple infrastructure & tenants lifecycles Cross technology, vendor agnostic
Full-mesh, network agnostic and encapsulated approach Relies on L3 networks and GRE, VXLAN… tunnels for inter-hosts communication (avoid using L2) Network agnostic, til hosts can route trafic SDN Routers must route traffic between an inner virtual net and the ext. world
SDN Properties
Host #3
Host #1
Host #2
SDN network #1
SDN network #1
SDN network #1
SDN network #1
SDN network #1
RouterPhysical net.Flow encapsulation
L3 network
OpenVSwtich
Open vSwitch is a production quality, multilayer virtual switch licensed under the open source Apache 2.0 license.
It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (e.g. NetFlow, sFlow, IPFIX, RSPAN, CLI, LACP, 802.1ag).
In addition, it is designed to support distribution across multiple physical servers similar to VMware's vNetwork distributed vswitch
Host #1 Host #3Host #2
Getting Started
Container ‘app’
Container ‘db’
Brid
ge d
ocke
r0
Container ‘app’
Container ‘db’
Brid
ge d
ocke
r0
Container ‘app’
Container ‘db’
Container ‘app’
Brid
ge d
ocke
r0
Container ‘app’
TCP ports of services running in containers are
mapped
Get rid of actual Docker bridges implementation ! (Turn ICC False) If not, all containers will talk together across a same host
SetUp - OVS Setup Dev Tools
yum -y groupinstall "Development Tools"
yum -y install wget openssl-devel
yum -y install wget openssl-devel gcc make python-devel openssl-devel kernel-devel graphviz kernel-debug-devel autoconf automake rpm-build redhat-rpm-config libtool
Set up Bridge Utils yum -y install bridge-utils
Prepare OVS rpm cd ~
mkdir -p rpmbuild/SOURCES
wget http://openvswitch.org/releases/openvswitch-2.3.1.tar.gz
tar xvfz openvswitch-2.3.1.tar.gz
cd openvswitch-2.3.1/
cp ../openvswitch-2.3.1.tar.gz ~/rpmbuild/SOURCES/
sed 's/openvswitch-kmod, //g' openvswitch-2.3.1/rhel/openvswitch.spec > openvswitch-2.3.1/rhel/openvswitch_no_kmod.spec
rpmbuild -bb --nocheck ~/openvswitch-2.3.1/rhel/openvswitch_no_kmod.spec
You can find the files in ~/rpmbuild/RPMS/x86_64/.
yum -y localinstall openvswitch-2.3.1-1.x86_64.rpm
Use OpenVSwitch to create bridges on each host, for each tenant’s subnet. For instance, on host #1:
» ovs-vsctl add-br tech-br» ovs-vsctl add-port tech-br tep0 -- set interface tep0 type=internal» ifconfig tep0 192.168.1.1 netmask 255.255.255.0» ovs-vsctl add-br sdn-br0» ovs-vsctl set bridge sdn-br0 stp_enable=true
Create SDN compliant bridges
Host #1
sdn-
br0
Host #3
sdn-
br0
sdn-
br1
Host #2sd
n-br
0
sdn-
br1
Simplified view. Detailed insight exposed in later slides
Once per host: common plumbing
For each bridge: create and protect against ethernet loops using
Spanning Tree Protocol (beware, in complex/large deployments, it may
consumes a lot of CPU!)
Link SDN bridges
Host #1
sdn-
br0
Host #3
sdn-
br0
sdn-
br1
Host #2sd
n-br
0
sdn-
br1
Use OpenVSwitch to link corresponding bridges accross hosts In this example, we decided to use the full-mesh approach. On host #1:
ovs-vsctl add-port sdn-br0 gre0 --set interface gre0 type=gre options:remote_ip:1.2.3.2 ovs-vsctl add-port sdn-br0 gre1 --set interface gre1 type=gre options:remote_ip:1.2.3.3
gre0
gre1
1.2.3.1 1.2.3.2 1.2.3.3
Simplified view. Detailed insight exposed in later slides
U
Start containers and map them to the bridges
Host #1
sdn-
br0
Host #3
sdn-
br0
sdn-
br1
Host #2sd
n-br
0
sdn-
br1
Container ‘app’
Container ‘db’
Container ‘app’
Container ‘db’
Container ‘app’
Container ‘app’
Container ‘db’
Container ‘app’
Container ‘db’
Instanciate containers without pre-built interfaces to avoid plugging containers to native docker0 bridge
Use “docker run … ip = ?? -net=none” switch in “docker run” calls
Weave
Weave Weave can traverse firewalls and operate in partially connected
networks. Traffic can be encrypted, allowing hosts to be connected across
an untrusted network. With weave you can easily construct applications consisting of
multiple containers, running anywhere. Weave works alongside Docker's existing (single host)
networking capabilities, so these can continue to be used by containers.
Weave Installation
Ensure you are running Linux (kernel 3.8 or later) and have Docker (version 1.3.1 or later) installed
Weave Setup
Performance Comparison – self test Used qperf to test performance
Physical Host
tcp_bw:
bw = 117 MB/sec OVS with gre tunnel (Default)
tcp_bw:
bw = 81.9 KB/sec OVS with gre tunnel (ethtool -K ovsBridge tx off sg off tso off) ( technique for increasing
outbound throughput of high-bandwidth network connections by reducing CPU overhead , not recommended for production)
tcp_bw:
bw = 104 MB/sec Weave(Latest version) based on VXLAN (Older versions based on pcap had lower
performance)
tcp_bw:
bw = 92.3 MB/sec
Bonus: Alternate design - OpenStack Neutron paradigms Alternate design, based on OpenStack Neutron paradigms
All VMs/containers of a same tenant network are segregated inside a dedicated, local VLAN of a shared unique bridge
Full-mesh of GRE tunnels between all hosts On each host, local mapping between a local tenant network
VLAN and its GRE identifier shared across all hosts