virtual networking for iaas clouds...midonet sdn solution 40 conclusions • iaas clouds require new...
TRANSCRIPT
Virtual Networking for IaaS Clouds
Dan Mihai Dumitriu, [email protected] CTO, Midokura
SDN Japan - December 7, 2012
Copyright ©2012 Midokura All rights reserved
Requirements
2
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Copyright ©2012 Midokura All rights reserved
Requirements: recap
3
l Multi-tenancy l Scalable, fault-tolerant
devices (or device-agnostic network services).
l L2 isolation l L3 isolation
u VPC, like VRF (virtual routing and fwd-ing)
l L3 Gateway (BGP) l L2 Gateway (STP, LACP) l Scalable control plane
u ARP, DHCP, ICMP l Floating IP
l Stateful NAT
u IP/Port masquerading u DNAT
l ACLs l Stateful (L4) Firewalls
u Security Groups l LB health checks l VPNs at L2 and L3
u L2TP, IPSec l REST API l Integration with CMS
u OpenStack u CloudStack
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
4
Logical Topology
MidoNet Solution
15
Private IP Network
MN
MN
MN
Internet
BGP Multi
Homing
Physical Topology
MN VM
VM
MN VM
VM
MN VM
VM
BGP To ISP3
BGP To ISP2
BGP To ISP1
vPort
Provider Virtual Router
Tenant A Virtual Router
Tenant B Virtual Router
Virtual Switch A1
Virtual Switch A2
Virtual Switch B1
vPort
vPort
vPort
vPort
vPort
Network State Database
MN MN MN
Tunnel
Gateway Nodes Compute Nodes
Copyright ©2012 Midokura All rights reserved
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Requirements
5
Isolated tenant network (virtual
data center)
Copyright ©2012 Midokura All rights reserved
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Requirements
6
L3 isolation (similar to VPC and VRF)
Copyright ©2012 Midokura All rights reserved
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Requirements
7
Isolated L2 networks
Copyright ©2012 Midokura All rights reserved
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Requirements
8
Redundant, optimized and fault-tolerant paths to the
Internet (e.g. via BGP)
Copyright ©2012 Midokura All rights reserved
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Requirements
9
Fault-tolerant devices and links
Copyright ©2012 Midokura All rights reserved
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Requirements
10
NAT, LB, and Filtering
NAT, LB, and Firewalls
Copyright ©2012 Midokura All rights reserved
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Requirements
11
L3 (and L2) VPNs
Copyright ©2012 Midokura All rights reserved
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Requirements
12
Minimize ARP broadcasts by exploiting CMS config
RESTful API for CMS integration and direct
tenant access
Solid integration with leading open CMS:
OpenStack, CloudStack
DHCP, DNS and other services
Copyright ©2012 Midokura All rights reserved
How to build it?
13
1. Virtualized physical devices
2. Centrally controlled OpenFlow-based hop-by-hop switching fabric
3. Edge to edge overlays
Copyright ©2012 Midokura All rights reserved
Virtualized physical devices
14
l 4096 limit on number of unique tags l Large spanning trees terminating on many hosts l High churn in switch control planes due to MAC learning
l Each VM is separate virtual MAC! l Need MLAG for L2 multi-path (vendor specific)
1
VLAN VLAN1
VLAN2
Copyright ©2012 Midokura All rights reserved
Virtualized physical devices
15
l L2 isolation l What about L3 and Internet access? l Use VRF or virtual appliances? Fault-tolerance?
1
VLAN (more) VLAN1
VLAN2
Copyright ©2012 Midokura All rights reserved
Virtualized physical devices
16
1
l Not scalable to cloud scale l Expensive hardware l Not fault tolerant (HSRP?) l L2 and L3 isolation. What about NAT, LB, FW?
出典:http://infrastructureadventures.com/tag/vrf-lite/
VRF
Div.1 VLAN 10 VLAN11 VLAN12
Div.2 VLAN 20 VLAN21 VLAN22
Div.3 VLAN 99
VRF VRF VRF
Copyright ©2012 Midokura All rights reserved
OpenFlow hop-by-hop switch fabric
17
2
OpenFlow Switches
OpenFlow Controller (Cluster)
l Fabric extends to the compute host software switch? • State in each switch is proportional to the virtual network state • Need to update all switches in path when provisioning new virtual
devices or updating them. • Not scalable, slow and non-atomic switch updates.
Copyright ©2012 Midokura All rights reserved
OpenFlow hop-by-hop switch fabric (more)
18
2
OpenFlow Switches
OpenFlow Controller (Cluster)
l Flow rules for VM flows (microflows)? l Flow rules for virtual device simulation?
Copyright ©2012 Midokura All rights reserved
Edge-to-Edge Overlays
19
3
VM
VM Edge
Edge Edge
Edge Edge
Edge
Use scalable IGP (iBGP, OSPF) to build multi-path underlay
Copyright ©2012 Midokura All rights reserved
Edge-to-Edge Overlays
20
3
VM
VM Edge
Edge Edge
Edge Edge
Edge
IP encapsulation provides isolation
Copyright ©2012 Midokura All rights reserved
Edge-to-Edge Overlays
21
3
VM
VM Edge
Edge Edge
Edge Edge
Edge
Virtual network processing at ingress host, decoupled from
physical network
Copyright ©2012 Midokura All rights reserved
Edge-to-Edge Overlays
22
3
VM
VM Edge
Edge Edge
Edge Edge
Edge
Virtual network changes don't affect
underlay state
Copyright ©2012 Midokura All rights reserved
Edge-to-Edge Overlays
23
3
• Packet processing on x86 CPUs (at edge) • Intel DPDK facilitates packet processing • Number of cores in servers increasing fast
• Clos Networks (for underlay) • Spine and Leaf architecture with IP • Economical and high E-W bandwidth
• Merchant silicon (cheap IP switches) • Broadcom, Intel (Fulcrum Micro), Marvell • ODMs (Quanta, Accton) starting to sell directly • Switches are becoming just like Linux servers
• Optical intra-DC Networks
Copyright ©2012 Midokura All rights reserved
Spine and Leaf Network Architecture
24
L3 Switch Spine
Leaf
L3 Switch L3 Switch L3 Switch
L3 Switch L3 Switch L3 Switch x32
x4
48x10G
1536 x 10G
4x40G
e.g Force10 Z9000
IBGP and ECMP
e.g Arista 7050T
e.g Force10 Z9000
Copyright ©2012 Midokura All rights reserved
Overlays are the right approach!
But not sufficient... We still need a scalable control plane.
25
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
26
Distributed State
MidoNet REST API
Dashboard
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
27
Distributed State
Linux Kernel + OVS KMOD
VM1 MidoNet
Ctrl
HW
Linux Kernel + OVS KMOD
VM2 MidoNet
Ctrl
HW
Host A Host B
Lazy state propagation
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
28
Distributed State
Linux Kernel + OVS KMOD
VM1 MidoNet
Ctrl
HW
Linux Kernel + OVS KMOD
VM2 MidoNet
Ctrl
HW
Host A Host B
VM sends first packet; table miss; NetLink
upcall to MidoNet
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
29
Distributed State
Linux Kernel + OVS KMOD
VM1 MidoNet
Ctrl
HW
Linux Kernel + OVS KMOD
VM2 MidoNet
Ctrl
HW
Host A Host B
MidoNet agent locally processes packet (virtual layer simulation); installs
local flow (drop/mod/fwd)
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
30
Distributed State
Linux Kernel + OVS KMOD
VM1 MidoNet
Ctrl
HW
Linux Kernel + OVS KMOD
VM2 MidoNet
Ctrl
HW
Host A Host B
Packet tunneled to peer host; decap; kflow table miss;
Netlink notifies peer MidoNet agent
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
31
Distributed State
Linux Kernel + OVS KMOD
VM1 MidoNet
Ctrl
HW
Linux Kernel + OVS KMOD
VM2 MidoNet
Ctrl
HW
Host A Host B
MN agent maps tun-key to kernel
datapath port#; installs fwd flow rule
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
32
Distributed State
Linux Kernel + OVS KMOD
VM1 MidoNet
Ctrl
HW
Linux Kernel + OVS KMOD
VM2 MidoNet
Ctrl
HW
Host A Host B
Subsequent packets matched by flow rules
at both ingress and egress hosts
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
33
MN
Packet Encapsulated
Tunnel
Drop/Block
Packet from VM, VPN, or external BGP peer enters
kernel datapath
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
34
MN
Packet Encapsulated
Tunnel
Drop/Block One flow rule reflecting the outcome of the virtual layer
simulation AND the mapping of egress vport to peer host
decides to drop or fwd
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
35
• Distributed and scalable control plane Ø Handle all control packets at local MidoNet agent
adjacent to VM • Scalable and fault tolerant central database
Ø Stores virtual network configuration Ø Dynamic network state
² MAC learning, ARP cache, etc Ø Cached at edges on demand
• All packet modifications at ingress Ø One virtual hop
² No travel through middle boxes Ø Drop at ingress
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
36
• Scalable edge gateway interface to external networks • Multihomed BGP to ISP • REST API and GUI • Integration with popular open source cloud stacks • OpenStack • Removes SPOF of network node • Scalable and fault tolerant NAT for floating IP • Implements security groups efficiently
• CloudStack and Eucalyptus
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
37
Deep OpenStack Integration
• Quantum Plugin • L2 isolation, of course • Also… • L3 isolation (without VM / appliance) • Security groups (stateful firewall) • Floating IP (NAT) • Load balancing (L4)
Copyright ©2012 Midokura All rights reserved
HorizonWeb GUI
Quantum Plugin
MidoNet Manager (Web GUI)
Nova API
MidoNet API
RabbitMQ - Passing Queue
Compute Host
NovaCompute
Libvirt driver
Datapath
Inst
an
ce
A1
Inst
an
ce
B1
MidoNetAgent
Quantum
API
MidoNet Plugin
MidoNet Distributed
State
MidoNet Distributed
State
MidoNet Distributed
State
MidoNet EdgeAgent
MidoNet EdgeAgent
MidoNet EdgeAgent
INTERNET
MidoNetVIF Driver
KeystoneAuthentication
OpenStack Integration
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
39
Future Directions
• Scalable L7 virtual appliances • Content aware load balancer • MPLS VPN termination Ø Interconnect with carrier backbones • multiple data center federation Ø Virtual L2 between sites • LISP Ø Global IP mobility between sites
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
40
Conclusions
• IaaS clouds require new networking model • Edge to edge overlays are the right
approach • Servers are good enoug at packet
processing Ø Can use them for edge gateways • Multipath IP network fabric is cheap and
easy to build