openstack networking juno l3 h-a, dvr
TRANSCRIPT
OpenStack Networking - Juno- - DVR & L3 High Availability
Paul SimTechnical Account [email protected]
● Distributed Virtual Router
○ Packet flow
○ Architecture
■ SNAT,
■ DNAT(Floating IP)
■ East<->West
● L3 High Availability
Index
DVR (Distributed Virtual Router)- Installation
Network node
Neutron server
eth1 eth2
eth0
Neutron ML2 plugin
Neutron metadata-agent
Neutron L3/dhcp-agent
External network
Compute node - 1
Nova compute
eth1 eth2
eth0
Neutron ML2 plugin
Neutron metadata-agent
Neutron L3-agent
Management
Data
Compute node - 2
Nova compute
eth1 eth2
eth0
Neutron ML2 plugin
Neutron metadata-agent
Neutron L3-agent
DVR (Distributed Virtual Router)- Packet flow
Compute node - 1
br-
ex
VM VMGRE Tunnel
VM
br-int
Network node
br-
ex
br-tun
br-int
br-tun
Compute node - 2
VM VM VM
br-int
br-
tun
1.SNAT
External network
3. East-West traffic
2. Floating IP
OVS bridge
DVR (Distributed Virtual Router)- SNAT : Network node
Namespace
OVS bridgeNetwork node
qdhcp-b
r-tun
eth0
br-intpatch-tun
patch-int
gre~
snat- qrouter-
tap taptap
sg~50.50.6.2ns~ qr~
qg~192.168.10.109
SNAT br-ex
tap
DVR (Distributed Virtual Router)- SNAT : Compute node
Compute node
Namespace
OVS bridge
VM
br-int
br-
tun
gre
~
qvo~
qbr~
tap~
qvb~
patch-int
qrouter-
qr~50.50.6.1
patch-tun
tap~
Linux bridge
sg~
on
netw
ork
no
de
packet flow
traffic flow
DVR (Distributed Virtual Router)- SNAT : Compute node
Namespace
OVS bridge
Linux bridge
Compute node
VM
br-int
br-
tun
gre
~
qvo~
qbr~
tap~
qvb~
patch-int
qrouter-
qr~50.50.6.1
patch-tun
tap~sg~
(50.
50.6
.2)
on
netw
ork
no
de
ubuntu@ubuntu-8:~$ sudo ip netns exec qrouter-20838b7d-a7ac-4da9-92aa-adec797d600e ip rule ls
0: from all lookup local 32766: from all lookup main 32767: from all lookup default 842139137: from 50.50.6.1/24 lookup
842139137ubuntu@ubuntu-8:~$ sudo ip netns exec qrouter-20838b7d-a7ac-4da9-92aa-adec797d600e ip route show table 842139137
default via 50.50.6.2 dev qr-9722faba-b7
DVR (Distributed Virtual Router)- Floating IP/DNAT : Compute node
Compute node
Namespace
OVS bridge
VM
br-int
br-
tun
gre
~
qvo~
qbr~
tap~
qvb~
patch-int
qrouter-
qr~50.50.6.1
patch-tun
tap~
Linux bridge
packet flow
br-extap
eth0
fip-
fpr~ rfp~
fg~
RouteRoute
NAT
veth pair
DVR (Distributed Virtual Router)- Floating IP/DNAT : Compute node
Compute node
Namespace
OVS bridge
VM
br-int
br-
tun
gre
~
qvo~
qbr~
tap~
qvb~
patch-int
qrouter-
qr~50.50.6.1
patch-tun
tap~
Linux bridge
packet flow
br-extap
eth0
fip-
fpr~ rfp~
fg~
RouteRoute
NAT
veth pair
ubuntu@ubuntu-6:~$ sudo ip netns exec qrouter-20838b7d-a7ac-4da9-92aa-adec797d600e ip rule ls0: from all lookup local 32766: from all lookup main 32767: from all lookup default 32770: from 50.50.5.5 lookup 16 842138881: from 50.50.5.1/24 lookup 842138881 842138881: from 50.50.5.1/24 lookup 842138881 842139137: from 50.50.6.1/24 lookup 842139137 ubuntu@ubuntu-6:~$ sudo ip netns exec qrouter-20838b7d-a7ac-4da9-92aa-adec797d600e ip route show table 16
default via 169.254.31.29 dev rfp-20838b7d-a
DVR (Distributed Virtual Router)- Floating IP/DNAT : Compute node
Compute node
Namespace
OVS bridge
VM
br-int
br-
tun
gre
~
qvo~
qbr~
tap~
qvb~
patch-int
qrouter-
qr~50.50.6.1
patch-tun
tap~
Linux bridge
packet flow
br-extap
eth0
fip-
fpr~ rfp~
fg~
RouteRoute
NAT
veth pair
ubuntu@ubuntu-6:~$ sudo ip netns exec fip-02f9d340-2caa-4c05-86fb-460c9580f9df ip route show
default via 192.168.10.1 dev fg-f3887d61-2d 192.168.10.114 via 169.254.31.28 dev fpr-20838b7d-a
DVR (Distributed Virtual Router)- East-West traffic flow : Compute node
Compute node-2
Namespace OVS bridge
VM50.50.6.3
br-int
br-
tun
gre
~
qvo~
qbr~tap~
qvb~
patch-int
qrouter-
qr~50.50.6.1
patch-tun
tap~
Linux bridge packet flow
Compute node-1
tap~
qr~50.50.5.1
VM50.50.5.3
br-int
br-
tun
qvo~
qbr~tap~
qvb~
patch-int
qrouter-
qr~50.50.6.1
patch-tun
tap~tap~
qr~50.50.5.1
gre~
ICMP Request
ICMP Replyi.e., ping 50.50.5.3 -> 50.50.6.3
DVR (Distributed Virtual Router)- East-West traffic flow : network topology
DVR (Distributed Virtual Router)- East-West traffic flow : Compute node
Compute node-2
Namespace OVS bridge
br-int
br-
tun
gre
~
qvo~
qbr~tap~
qvb~
patch-int
qrouter-
qr~50.50.6.1
patch-tun
tap~
Linux bridge packet flow
Compute node-1
tap~
qr~50.50.5.1
br-int
br-
tun
qvo~
qbr~tap~
qvb~
patch-int
qrouter-
qr~50.50.6.1
patch-tun
tap~tap~
qr~50.50.5.1
gre~
VM50.50.6.3
ubuntu@ubuntu-6:~$ sudo ip netns exec qrouter-20838b7d-a7ac-4da9-92aa-adec797d600e ip link2: qr-ecffa2a6-dd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether fa:16:3e:15:1e:e0 brd ff:ff:ff:ff:ff:ff5: qr-9722faba-b7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether fa:16:3e:71:3d:5a brd ff:ff:ff:ff:ff:ff
ubuntu@ubuntu-8:~$ sudo ip netns exec qrouter-20838b7d-a7ac-4da9-92aa-adec797d600e ip link2: qr-ecffa2a6-dd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether fa:16:3e:15:1e:e0 brd ff:ff:ff:ff:ff:ff5: qr-9722faba-b7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether fa:16:3e:71:3d:5a brd ff:ff:ff:ff:ff:ff
VM50.50.5.3
DVR (Distributed Virtual Router)- East-West traffic flow : Compute node
Compute node-2
Namespace OVS bridge
VM50.50.6.3
br-int
br-
tun
gre
~
qvo~
qbr~tap~
qvb~
patch-int
qrouter-
qr~50.50.6.1
patch-tun
tap~
Linux bridge packet flow
Compute node-1
tap~
qr~50.50.5.1
VM50.50.5.3
br-int
br-
tun
qvo~
qbr~tap~
qvb~
patch-int
qrouter-
qr~50.50.6.1
patch-tun
tap~tap~
qr~50.50.5.1
gre~
ICMP Request 50.50.5.3 -> 50.50.6.3Segmentation ID : 50.50.5.0/24 : 0x150.50.6.0/24 : 0x3
MAC 50.50.6.3 : fa:16:3e:ff:85:9b50.50.6.1 : fa:16:3e:71:3d:5a50.50.5.1 : fa:16:3e:15:1e:e050.50.5.3 : fa:16:3e:ce:8c:35
DVR Host MAC : Compute Node -1 : fa:16:3f:5e:a0:cfCompute Node -2 : fa:16:3f:72:60:33
SRC MAC : fa:16:3e:71:3d:5aSRC IP : 50.50.5.3 DST MAC : fa:16:3e:ff:85:9bDST IP : 50.50.6.3
SRC MAC : fa:16:3e:71:3d:5aSRC IP : 50.50.5.3 DST MAC : fa:16:3e:ff:85:9bDST IP : 50.50.6.3
SRC MAC : fa:16:3e:ce:8c:35SRC IP : 50.50.5.3 DST MAC : fa:16:3e:15:1e:e0DST IP : 50.50.6.3
GRE tunnel 0x3
SRC MAC : fa:16:3f:5e:a0:cfSRC IP : 50.50.5.3 DST MAC : fa:16:3e:ff:85:9bDST IP : 50.50.6.3
DVR (Distributed Virtual Router)- East-West traffic flow : Compute node
Compute node-2
Namespace OVS bridge
VM50.50.6.3
br-int
br-
tun
gre
~
qvo~
qbr~tap~
qvb~
patch-int
qrouter-
qr~50.50.6.1
patch-tun
tap~
Linux bridge packet flow
Compute node-1
tap~
qr~50.50.5.1
VM50.50.5.3
br-int
br-
tun
qvo~
qbr~tap~
qvb~
patch-int
qrouter-
qr~50.50.6.1
patch-tun
tap~tap~
qr~50.50.5.1
gre~
ICMP Reply 50.50.6.3 -> 50.50.5.3
SRC MAC : fa:16:3e:15:1e:e0SRC IP : 50.50.6.3 DST MAC : fa:16:3e:ff:85:9bDST IP : 50.50.5.3
SRC MAC : fa:16:3e:15:1e:e0SRC IP : 50.50.6.3 DST MAC : fa:16:3e:ce:8c:35DST IP : 50.50.5.3
SRC MAC : fa:16:3e:ff:85:9bSRC IP : 50.50.6.3 DST MAC : fa:16:3e:71:3d:5aDST IP : 50.50.5.3
Segmentation ID : 50.50.5.0/24 : 0x150.50.6.0/24 : 0x3
MAC 50.50.6.3 : fa:16:3e:ff:85:9b50.50.6.1 : fa:16:3e:71:3d:5a50.50.5.1 : fa:16:3e:15:1e:e050.50.5.3 : fa:16:3e:ce:8c:35
DVR Host MAC : Compute Node -1 : fa:16:3f:5e:a0:cfCompute Node -2 : fa:16:3f:72:60:33
GRE tunnel 0x1
SRC MAC : fa:16:3f:72:60:33SRC IP : 50.50.6.3 DST MAC : fa:16:3e:ce:8c:35DST IP : 50.50.5.3
DVR (Distributed Virtual Router)- East-West traffic flow : Compute node
Compute node-2
Namespace OVS bridge
VM50.50.6.3
br-int
br-
tun
gre
~
qvo~
qbr~tap~
qvb~
patch-int
qrouter-
qr~50.50.6.1
patch-tun
tap~
Linux bridge packet flow
Compute node-1
tap~
qr~50.50.5.1
VM50.50.5.3
br-int
br-
tun
qvo~
qbr~tap~
qvb~
patch-int
qrouter-
qr~50.50.6.1
patch-tun
tap~tap~
qr~50.50.5.1
gre~
ICMP Request 50.50.5.3 -> 50.50.6.3Segmentation ID : 50.50.5.0/24 : 0x150.50.6.0/24 : 0x3
MAC 50.50.6.3 : fa:16:3e:ff:85:9b50.50.6.1 : fa:16:3e:71:3d:5a50.50.5.1 : fa:16:3e:15:1e:e050.50.5.3 : fa:16:3e:ce:8c:35
DVR Host MAC : Compute Node -1 : fa:16:3f:5e:a0:cfCompute Node -2 : fa:16:3f:72:60:33
table=0, n_packets=9178, n_bytes=1009035, idle_age=17470, hard_age=65534, priority=1 actions=NORMAL
table=0, n_packets=2066, n_bytes=214544, idle_age=5, hard_age=65534, priority=1,in_port=1 actions=resubmit(,1)table=1, n_packets=1765, n_bytes=172970, idle_age=5, hard_age=65534, priority=1,dl_vlan=2,dl_src=fa:16:3e:71:3d:5a actions=mod_dl_src:fa:16:3f:5e:a0:cf,resubmit(,2)table=2, n_packets=1849, n_bytes=183458, idle_age=5, hard_age=65534, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)table=20, n_packets=1765, n_bytes=172970, idle_age=5, hard_age=65534, priority=2,dl_vlan=2,dl_dst=fa:16:3e:ff:85:9b actions=strip_vlan,set_tunnel:0x3,output:3
DVR (Distributed Virtual Router)- East-West traffic flow : Compute node
Compute node-2
Namespace OVS bridge
VM50.50.6.3
br-int
br-
tun
gre
~
qvo~
qbr~tap~
qvb~
patch-int
qrouter-
qr~50.50.6.1
patch-tun
tap~
Linux bridge packet flow
Compute node-1
tap~
qr~50.50.5.1
VM50.50.5.3
br-int
br-
tun
qvo~
qbr~tap~
qvb~
patch-int
qrouter-
qr~50.50.6.1
patch-tun
tap~tap~
qr~50.50.5.1
gre~
ICMP Request 50.50.5.3 -> 50.50.6.3Segmentation ID : 50.50.5.0/24 : 0x150.50.6.0/24 : 0x3
MAC 50.50.6.3 : fa:16:3e:ff:85:9b50.50.6.1 : fa:16:3e:71:3d:5a50.50.5.1 : fa:16:3e:15:1e:e050.50.5.3 : fa:16:3e:ce:8c:35
DVR Host MAC : Compute Node -1 : fa:16:3f:5e:a0:cfCompute Node -2 : fa:16:3f:72:60:33
table=0, n_packets=1789, n_bytes=175146, idle_age=17, hard_age=65534, priority=2,in_port=3,dl_src=fa:16:3f:5e:a0:cf actions=resubmit(,1)table=1, n_packets=1765, n_bytes=172970, idle_age=17, hard_age=65534, priority=4,dl_vlan=2,dl_dst=fa:16:3e:ff:85:9b actions=strip_vlan,mod_dl_src:fa:16:3e:71:3d:5a,output:8
table=0, n_packets=1857, n_bytes=184993, idle_age=18, hard_age=65534, priority=1,in_port=2 actions=resubmit(,3)table=3, n_packets=1993, n_bytes=195880, idle_age=18, hard_age=65534, priority=1,tun_id=0x3 actions=mod_vlan_vid:2,resubmit(,9)table=9, n_packets=1789, n_bytes=175146, idle_age=18, hard_age=65534, priority=1,dl_src=fa:16:3f:5e:a0:cf actions=output:1
L3 High Availability- Installation
Network node-1
Neutron server
eth1 eth2
eth0
Neutron ML2 plugin
Neutron metadata-agent
Neutron L3/dhcp-agent
External network
Management
Data
KeepAlived
Network node-2
Neutron server
eth1 eth2
eth0
Neutron ML2 plugin
Neutron metadata-agent
Neutron L3/dhcp-agent
KeepAlived Compute node - 1
Nova compute
eth1 eth2
eth0
Neutron ML2 plugin
Compute node - 2
eth1 eth2
eth0
Neutron ML2 plugin
Nova compute
Network node-2
Compute node - 3
Compute node - 2
Network node-1
vRouter A - Master
L3 High Availability
Compute node - 1
Subnet 1
Subnet 3
Subnet 2
Subnet 5
vRouter B - Backup
vRouter C - Backup
vRouter D - Master
vRouter C - Master
vRouter D - Backup
vRouter A - Backup
Subnet 3
Subnet 4
vRouter B - Master
Tenant X
Tenant Y
Tenant Z
VRRP
L3 High Availability
Namespace OVS bridge
Network node-1
qdhcp-
br-tun
br-int
qrouter-
ha~ns~ qr~qg~
br-ex
Network node-2
qdhcp-
br-
tun
br-int
qrouter-
qr~qg~
br-ex
ns~
KeepAlived KeepAlived
ha~
ubuntu@ubuntu-5:~$ sudo ip netns exec qrouter-d8625260-88a1-4312-b788-c04fc9094356 tcpdump -n -i ha-27fe59da-a8tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on ha-27fe59da-a8, link-type EN10MB (Ethernet), capture size 65535 bytes16:16:25.213440 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 2016:16:27.214607 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 2016:16:29.215796 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 2016:16:31.216986 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20
L3 High Availability
Network node-1
qdhcp-
br-tun
eth0
br-int patch-tun
patch-int
qrouter-
tap taptap
ha~ns~ qr~
qg~
br-extap
Network node-2
qdhcp-
br-
tun
eth0
br-intpatch-tunpat
ch-i
nt
gre
~
qrouter-
tap taptap
ha~ns~ qr~
qg~
br-extap
Namespace OVS bridge
gre~
L3 High Availability
Network node-1
qdhcp-
br-tun
eth0
br-int patch-tun
patch-int
qrouter-
tap taptap
ha~ns~ qr~
qg~
br-extap
Network node-2
qdhcp-
br-
tun
eth0
br-intpatch-tunpat
ch-i
nt
gre
~
qrouter-
tap taptap
ha~ns~ qr~
qg~
br-extap
Namespace OVS bridge
gre~
L3 High Availability
Namespace OVS bridge
Network node-1
qdhcp-
br-tun
br-int
qrouter-
ha~ns~ qr~qg~
br-ex
KeepAlived
ubuntu@ubuntu-5:~$ cat /var/lib/neutron/ha_confs/d8625260-88a1-4312-b788-c04fc9094356/keepalived.confvrrp_sync_group VG_1 { group { VR_1 } notify_master "/var/lib/neutron/ha_confs/d8625260-88a1-4312-b788-c04fc9094356/notify_master.sh" notify_backup "/var/lib/neutron/ha_confs/d8625260-88a1-4312-b788-c04fc9094356/notify_backup.sh" notify_fault "/var/lib/neutron/ha_confs/d8625260-88a1-4312-b788-c04fc9094356/notify_fault.sh"}vrrp_instance VR_1 { state BACKUP interface ha-27fe59da-a8 virtual_router_id 1 priority 50 nopreempt advert_int 2 track_interface { ha-27fe59da-a8 } virtual_ipaddress { 192.168.10.118/24 dev qg-8fffbd7e-8a } virtual_ipaddress_excluded { 50.50.1.1/24 dev qr-dee474e1-1e } virtual_routes { 0.0.0.0/0 via 192.168.10.51 dev qg-8fffbd7e-8a }}
Network nodeTenant A
L3 High Availability
Namespace OVS bridge
br-tun
br-int
qrouter-
ha~
br-ex
KeepAlived
qrouter-
ha~
KeepAlived
HA network : 169.254.192.0/18 Segmentation id : 0x6
Tenant B
qrouter-
ha~
KeepAlived
qrouter-
ha~
KeepAlived
HA network : 169.254.192.0/18 Segmentation id : 0x7
● One KeepAlived instance per vRouter● One HA network per tenant
○ Each HA network has separate segmentation id
○ allow_overlapping_ips = True
● Maximum 255 HA routers per tenant.