operating openstack - case study in the rackspace cloud

Rainya Mosher, Software Dev ManagerInfrastructure Deployment SystemsTwitter: @rainyamosher | Email: [email protected]

Operating OpenStack at Rackspace for a Hybrid Cloud Experience

20 March 2014

2RACKSPACE® HOSTING | WWW.RACKSPACE.COM

• Name: Rainya Mosher• Twitter: @rainyamosher• IRC Freenode: rainya• Email: [email protected]

Rainya is the software development manager for deployment automation in the Product Infrastructure segment at Rackspace Hosting, Inc. Rackspace is based in San Antonio, Texas, United States.

Your PresenterOperating OpenStack @ Rackspace


“If computers of the kind I have advocated become the computers of the future, then computing may someday be organized as a public utility just as the telephone system is a public utility … The computer utility could become the basis of a new and important industry.” - John McCarthy, American Computer Scientist, MIT Centennial, 1961

Brief History of ComputingOperating OpenStack @ Rackspace

Mainframe1960s

Virtualization1990s

Cloud2000s

Hybrid2010s


Hybrid Cloud ExperienceOperating OpenStack @ Rackspace

The hybrid cloud is the notion that an individual can consume whatever piece of infrastructure they need at the time that they need it with a

consistent experience so they can focus on their core competency, be it coding or research or data analysis.


• Powered by OpenStack

• Open source software with global community of over 1,000 contributors and 80 companies

• OpenStack provides freedom from proprietary software and vendor lock-in

• Rackspace Infrastructure includes the following OpenStack services

• OpenStack Compute• OpenStack Image Service• OpenStack Networks• OpenStack Storage

Public Cloud at RackspaceOperating OpenStack @ Rackspace


OpenStack Cloud Operating SystemOperating OpenStack @ Rackspace


Kickstart• Install hypervisor

onto physical host

Bootstrap• Prepare hypervisor

and create control plane nodes to run OpenStack services

Maintain• Update OpenStack

services in control plane nodes

Building an Open CloudOperating OpenStack @ Rackspace

The hypervisor is where the customer instance / server is

created.

Control plane nodes are virtual instances that run the OpenStack services.


Hypvisors (HVs) & Compute Control Nodes

Hundreds of HVs &Cell Control Plane Nodes

Thousands of HVs & Region Control Plane Nodes

Tens of Thousands of HVs & Control Plane Nodes

Global Cloud

Region Region

Cell Cell Cell

HV HV HV HV HV HV

Cell Cell

Region

Maintain at Cloud ScaleOperating OpenStack @ Rackspace


Code

Package

Deploy

Verify

Updating the Control PlaneOperating OpenStack @ Rackspace

During a control plane update, customer instances on the

hypervisor stay up and running.


Rac

kspa

ce

Itera

tion

Dep

loy

Dep

loy

Dep

loy

Rac

kspa

ce

Itera

tion

Dep

loy

Dep

loy

Rac

kspa

ce

Itera

tion

Dep

loy

Dep

loy

Dep

loy

Code Iterations & ReleasesOperating OpenStack @ Rackspace

6-Month OpenStack Release Cycle

60 Days 60 Days 60 Days


Benefits and Challenges of Trunk DeploysOperating OpenStack @ Rackspace

Why We Do It (Benefits)• Issue Resolution

– Early detection of issues and conflicts

– Shorter feedback loop within the community

– Faster resolution of issues

• Early Feature Delivery– Smaller, incremental periodic releases

– More stable release candidates for the community at end of cycle

Why It’s Hard (Challenges)• Code Management

– Merge conflicts with local patches

– Disruptive DB migrations

– Service restarts

– Temporary version skew

• Testing – Devstack-based testing vs testing at

scale

– Rework when issues found in RAX deploy pipeline

• Process – CI/CD vs 6-month Releases

– Time to merge upstream patches


• per-service venv• .tar of service

venvs + configs

Package

• seed .torrent• distribute

configs• verify

completion

Distribute • switch version• sync databases• apply configs• verify

completion

Execute

Package and DeployOperating OpenStack @ Rackspace

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Deploy and VerifyOperating OpenStack @ Rackspace

• pre-code check-in validation

Dev

• smoke tests• unit tests

Integration • functional tests• integration

tests

QA

• regression tests

• build tests

Pre-Prod • smoke tests• build tests

Production


Jenkins

Ansible

Puppet

• Execution

• Orchestration

• Configuration Management

Tools to Maintain the CloudOperating OpenStack @ Rackspace


Tools to Maintain the CloudOperating OpenStack @ Rackspace

• Puppet – Configuration management for all control plane nodes

– YAML-defined fact files for config variables

– Masterless configuration to prevent bottlenecks at scale

• Ansible – Playbooks for orchestration of OpenStack service bootstrap and updates

– Playbooks for orchestration of out-of-iteration hotpatches and changes

• Jenkins – Scripts called to create virtual environment (venv) packages

– Jobs configured for automating the pull and merge of upstream code

– QE test automation triggered after deployments

– Ansible playbooks invoked from jenkins to reduce human error


• The hybrid cloud concept will enable individuals to access the infrastructure they need when they need it.

• One part of the hybrid cloud portfolio at Rackspace is the public cloud, powered by OpenStack.

• OpenStack is an open source cloud operating system with distinct services that work together to enable the creation of cloud servers.

• Rackspace uses a variety of tools and processes to maintain OpenStack services throughout the OpenStack release cycle.

Final ThoughtsOperating OpenStack @ Rackspace

Twitter: @rainyamosher | Email: [email protected]


Backup SlidesOperating OpenStack at Rackspace

Many of these backup slides were first presented on 4/16/2013 during the OpenStack Summit session “Deploying from OpenStack Trunk” and are

included here for reference.


OpenStack Release CycleOperating OpenStack @ Rackspace

6-Month OpenStack Release Cyclenew features accepted feature freeze

Planning

Community designs, discusses, & targets

release cycleImplementation

Community makes changes to OpenStack

code & creates new functionality Pre-Release

Community focuses on bug fixes, docs,

& testing

Maintenance

Security vulnerability fixes are supported for the last three Releases

Final Release


Rackspace Iteration CycleOperating OpenStack @ Rackspace

Choose Release Branch

Pull upstream code, merge with internal patches, and deploy to CI environment

for validation

Release Branch Integration

Increase test coverage on release branch candidate to include full end-to-end integration testing with Rackspace services

Approve ReleaseOnce testing

passes, release is approved and

production deploy is scheduled

60-Day Rackspace Iteration Cyclecode may change release branch frozen


Merge and Branch StrategyOperating OpenStack @ Rackspace

• Rackspace Development branch is about 40 patches on top of OpenStack trunk for internal service compatability


Scale of Deploy PipelineLearning to Scale OpenStack

Dev

Integration & QA

PreProd

Production

DevStack

10s of Nodes

100s of Nodes

1,000s of Nodes

operating openstack - case study in the rackspace cloud

Technology