2014 openstack summit - neutron ovs to linuxbridge migration
DESCRIPTION
Presentation titled 'Migrating production workloads from OVS to LinuxBridge'. Presented at the Fall 2014 OpenStack summit in Paris, this slide deck introduced the possibility of migrating live workloads from Open vSwitch to LinuxBridge with minimal downtime.TRANSCRIPT
![Page 1: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/1.jpg)
Migrating production workloads from OVS to LinuxBridge
Kevin Stevens ([email protected])
James Denton ([email protected])
![Page 2: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/2.jpg)
Kevin Stevens
•RPC Engineer since 2012 (Essex)
• IRC: k_stev on irc.freenode.net
We’re operators!
James Denton
• RPC Network Engineer since 2012 (Essex)
• IRC: busterswt on irc.freenode.net
![Page 3: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/3.jpg)
What are we doing here?
1
A history of networking in
Rackspace Private Cloud
Our experiences with Open vSwitch
Swapping out OVS
with LinuxBridge
What to expect with
each
2 3 4
![Page 4: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/4.jpg)
2011Started building OpenStack-powered private clouds
2012
Began architecting, building and supporting private clouds in customer DCs
2013
Over 100 customers running RackspacePrivate Clouds
2014Released RPC v9 based on Icehouse. 99.99% API uptime SLA.
![Page 5: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/5.jpg)
RPC
v2.0/3.0
RPC
v4.0/4.1RPC v4.2 RPC v9.0
OpenStack
ReleaseFolsom Grizzly Havana Icehouse
Network Stack nova-network Quantum Neutron Neutron
L2
ConnectivityflatDHCP Open vSwitch Open vSwitch
LinuxBridge
(ML2)
L3 Agent
SupportN/A No Yes Yes
Host OSUbuntu 12.04
LTS
Ubuntu 12.04
LTS
CentOS 6.5
RHEL 6.5
Ubuntu 12.04
LTS
Ubuntu 14.04
LTS
The Evolution of Networking in RPC
![Page 6: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/6.jpg)
Why Neutron?
![Page 7: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/7.jpg)
Why Neutron w/ Open vSwitch?
•Open vSwitch pushed
by community
•Open vSwitch pushed
by packagers
•Wanted overlay
networks
![Page 8: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/8.jpg)
![Page 9: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/9.jpg)
•Kernel panics (1.10)
•ovs-vswitchd segfaults
(1.11)
•Broadcast storms
•Data corruption (2.01)
The problems
![Page 10: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/10.jpg)
Why Linux Bridge?
![Page 11: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/11.jpg)
•Looking for reliability and stability
•Less moving parts
•Easier to troubleshoot
•Supported by the community
Why move to LinuxBridge?
![Page 12: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/12.jpg)
• Flexibility provided by overlay networking
(if not using vxlan)
•Neutron Distributed Virtual Routers (Juno)
•Any customizability provided by OVS not implemented by Neutron itself
www.rackspace.com 12
What do we lose by moving?
![Page 13: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/13.jpg)
Planning
![Page 14: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/14.jpg)
•Snapshot and delete all instances
•Delete all networks
•Change from OVS -> LB
•Recreate all networks
•Boot instances
•…
•It works but…
Plan A: Scorch the earth!
![Page 15: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/15.jpg)
But wait… these are production environments!
![Page 16: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/16.jpg)
•Deploy LinuxBridge environment
•Snapshot all instances
•Import images into new
environment
•Build new instances
•Cutover
•…
•It works, but… $$$
Plan B: Migration Environment
![Page 17: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/17.jpg)
•Stop services
•Update the database
•Change the configuration from OVS -> LB
•Restart services
•…
•Profit!
Plan C: Switch it out!
![Page 18: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/18.jpg)
• Neutron OVS DB schema != Neutron LinuxBridge DB schema
–Migration to OVS ML2 DB schema is required first
• Overlay networks may not supported
– LinuxBridge uses VXLAN rather than GRE
–Requires kernel >= 3.9
• Means GRE networks must be converted to VLAN networks
–Didn’t want to introduce additional complexity
–VLANs easier to troubleshoot if something went wrong
Issues with migrating
![Page 19: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/19.jpg)
The Process
![Page 20: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/20.jpg)
•Determine what’s needed:
–Dependencies
–Some method of converting database to ML2 schema
–Some method of converting data to LB from OVS
–Which configuration files need mangling
–Which services need disabling
–Which services need restarting
–Roll-back plan
Preparation
![Page 21: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/21.jpg)
•Can instances gain a DHCP lease?
•Do instances have internal/external connectivity?
•Are security groups/other functions still operational?
•Were instances placed into the correct bridge?
•Will the changes survive a reboot?
Define a successful outcome
![Page 22: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/22.jpg)
Normal OVS Operation (Network Node)
![Page 23: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/23.jpg)
Normal OVS Operation (Compute Node)
![Page 24: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/24.jpg)
• Backup! Backup! Backup!
•Use migrate_to_ml2.py (modified) to change the DB schema
•Update segments, ports and vlan tables
–Change GRE to VLAN
–Change segmentation id to real VLAN ID
–Set a provider bridge
First steps: Database manipulation
![Page 25: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/25.jpg)
• Install the LinuxBridge plugin
•Update SQL connection strings
• Configure ml2_conf.ini / linuxbridge_conf.ini
• Change driver from OVS to ML2 in Neutron and Nova conf files
Next steps: Install and Configure
![Page 26: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/26.jpg)
• Stop Neutron services on all nodes
• Remove host data-plane port from the OVS bridge(s)
• Pull instance taps out of the OVS-related linux bridges
• Remove router and dhcp interfaces from OVS integration bridge
• Stop Openvswitch
Next steps: Pull ports from bridges
![Page 27: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/27.jpg)
Interfaces removed from bridges
![Page 28: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/28.jpg)
Stop openvswitch services
![Page 29: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/29.jpg)
• Start Neutron services
• Restart compute services
Finally: Restart services
![Page 30: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/30.jpg)
Post Service Restart (Network Node)
![Page 31: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/31.jpg)
Post Service Restart (Compute Node)
![Page 32: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/32.jpg)
![Page 33: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/33.jpg)
•Instances unresponsive?–Check traffic from tap->bridge->physical interface
–Verify VLANs properly trunked through (and VLANs created on the switch)
Failure Scenarios
![Page 34: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/34.jpg)
•IPs disappear or taps placed in QBR bridges–Check Nova instance_info_caches table.
–Cache can be regenerated with a hard reboot of instance, or by adding an interface to the instance
Failure Scenarios (Cont’d)
![Page 35: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/35.jpg)
•Unable to boot new instances?
– Usual troubleshooting techniques should be used
•DHCP Binding_failed error messages?
– Check /etc/default/neutron-server is referencing ML2 configuration file
•BRQ bridges not built?
– Verify New agents checking in?
– Verify the LinuxBridge agent is installed and running
Failure Scenarios (Cont’d Cont’d)
![Page 36: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/36.jpg)
Benchmarks
![Page 37: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/37.jpg)
* Host-to-host testing; no virtualization. Longer is better.
Compare all the things
0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00
1
2
4
8
Aggregate Throughput (Gbps)
# ofThreads
iPerf3 Benchmarks (TCP / 1500 MTU / 10G Data) – Intel X520* (ixgbe driver)
Open vSwitch(VXLAN)
LinuxBridge(VXLAN)
Open vSwitch(GRE)
Open vSwitch(VLAN)
LinuxBridge(VLAN)
![Page 38: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/38.jpg)
* Host-to-host testing; no virtualization. Longer is better.
Compare all the things
59.75 Seconds
61.50 Seconds
110.50 Seconds
104.00 Seconds
115.00 Seconds
0.00 20.00 40.00 60.00 80.00 100.00 120.00 140.00 160.00 180.00 200.00
LB VLAN
OVS VLAN
OVS GRE
LB VXLAN
OVS VXLAN
Transfer Speed (MBps)
SCP File Transfers (10G file)*
![Page 39: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/39.jpg)
* Host-to-host testing; no virtualization. Longer is better.
Compare all the things
59.75 Seconds
61.50 Seconds
110.50 Seconds
104.00 Seconds
115.00 Seconds
0.00 20.00 40.00 60.00 80.00 100.00 120.00 140.00 160.00 180.00 200.00
LB VLAN
OVS VLAN
OVS GRE
LB VXLAN
OVS VXLAN
Transfer Speed (MBps)
SCP File Transfers (10G file)*
![Page 40: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/40.jpg)
* Host-to-host testing; no virtualization. Longer is better.
Compare all the things
59.75 Seconds
61.50 Seconds
110.50 Seconds
104.00 Seconds
115.00 Seconds
0.00 20.00 40.00 60.00 80.00 100.00 120.00 140.00 160.00 180.00 200.00
LB VLAN
OVS VLAN
OVS GRE
LB VXLAN
OVS VXLAN
Transfer Speed (MBps)
SCP File Transfers (10G file)*
![Page 41: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/41.jpg)
• OVS provides a great deal of functionality
• Network stability more important for our customers than being on the cutting edge
• Linux bridge provides almost all of the features we might want to use
• How to migrate existing environments to LinuxBridge
• Improved stability and comparable performance with OVS achieved
www.rackspace.com 41
In Summary
![Page 42: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/42.jpg)
![Page 43: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/43.jpg)
Questions?
![Page 44: 2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration](https://reader034.vdocuments.net/reader034/viewer/2022052311/559441ad1a28ab04258b45f2/html5/thumbnails/44.jpg)
Download @https://github.com/busterswt/openstackparis2014