Rule breaker!Upgrading an OpenStack Cloud while skipping a release

Rick Salevsky Nanuk KrinnerSUSE Cloud Engineer SUSE Cloud Engineerrsalevsky@suse.com nkrinner@suse.com

Introduction

3

Speakers• Rick Salevsky

– SUSE Cloud Engineer– Focus deployment solutions

• Nanuk Krinner– Cloud developer at SUSE– Systems Management Engineer

• SUSE OpenStack Cloud

4

Agenda

• Why and What

• Upgrade strategies

• How we skipped a release

• Where we want to go

Why and What

6

Why upgrading?• Security Fixes

• Stability improvements

• Performance improvements

• Closely follow upstream development

• New features

7

Problems while upgrading?• Downtime

• Preparation

• Testing

• Adapting workflows

• Bugs

• Data loss

8

Customer demands• Reduced downtime

• Live upgrade

• Possible to roll back

• Clear documentation of what is happening

• Upgrading while skipping one or more releases

9

Upgrade marathon

Release Evaluating the release

Planning the upgrade

Testing the upgrade

Fine tuningIntegrating new features Upgrade

10

Upgrade marathon

Release Evaluating the release

Planning the upgrade

Testing the upgrade

Fine tuning

Maybe not this time?

Integrating new features Upgrade

11

• OpenStack User Survey April 2016No upgrade at all

Upgrade strategies

13

Official upgrade process• Always upgrade to next release

• Release cycle of 6 months

• Upgrades are required regularly

• High maintenance cost– Unexpected changes break upgrade– Lot’s of manual effort– Suffer the upgrade pain regularly– Staffing Source:

http://www.openstack.org/brand/openstack-logo/logo-download/

14

Continuous Deployment• Risky

• Needs a lot of development manpower

• Extensive testing required

• Always latest greatest

• No big upgrade, incremental changes

• Not enterprise readySource: https://wiki.jenkins-ci.org/display/JENKINS/Logo

15

Start from scratch• Roll out a fresh deployment• Lots of duplicated work

– Set up projects, users, images…

• Get rid of outdated artifacts• Run a parallel installation

– Move workload from old cloud to new cloud

• Redundant hardware required

16

Many self-made solutions

• Own deployment solutions

• Scenario-specific solutions

How we skip a release

18

High level overview• Upgrading from Juno to Liberty

• Multistep process

• Cloud is not fully functional

• Upgrading OS along with OpenStack

19

The idea• Orchestrated reinstallation

• Ignoring OpenStack Kilo release

• Migration handling

• Config file management

• Still Downtime and Disruptive

• No extra hardware required

20

Requirements• Orchestration mechanism

• Configuration management

• New OpenStack Packages are available

• Enough disk space on Controller– Duplicated database

• Shared disk for nova-compute data

21

Preparation• Stop configuration management

• Update OpenStack configs

• Check which migrations are needed

22

Backup data• Disable (not stop) all OpenStack Services

• Shutdown OpenStack on non DB nodes

• Backup OpenStack database

• Backup other important data if wanted

• Finalize OpenStack Shutdown

23

Setup new OpenStack Cloud• Reinstall Nodes with new OS if required

• Install new OpenStack Packages

• Start configuration management

• Start database service

• Restore backed up data

24

Migrating OpenStack Services• Run all migrations as documented for a upgrade

• Special commands need porting

• Juno to Liberty exceptions– Nova

25

Migrating Nova Service• Migrate to last Kilo migration level

– Last kilo migration = 290– ‘nova-manage db sync --version 290’

• Migrate Flavor data– Porting from Kilo to Liberty was required– ‘nova-manage db migrate_flavor_data’

• Migrate to Liberty migration level– ‘nova-manage db sync’

26

Finalizing Upgrade• Start all OpenStack Services

• Check if everything is running

27

Issues• Configuration File migration

• Migrations

• All or nothing

• Predefined Upgrades

28

Do’s Dont’s• Backups

• Test new configurations• Hope everything runs

29

Do’s and Dont’s

Where we want to go

31

Outlook• Seamless Upgrade

• No downtime of important services

• Reverting upgrades

• Better orchestration

32

Call for Action• Config files (automatically) upgradeable

• Uniform configuration files

• Migrating existing data from every point

• Rollback option

• Non-disruptive upgrades

• Integrating oslo version objects in every project

33

Thank you.

Questions?

35

Real world issues• Skipped or delayed upgrades

• Small operator teams

• Downstream code changes

• Ignoring recommended path

• Services depending on the

cloud

OpenStack User Survey, April 2016