Rob Hirschfeld, Dell Matt Ray, Opscode

Automation and DevOps Best Practices

Deploying & Managing a Cloud is not simple.

It does not have to be hard!

•  Deploying to physical gear on layered networks •  Multiple interlocking projects •  Hundreds of configuration options •  An aggressive 6 month delivery cycle •  Constant innovation on core technologies

Problem: Clouds do not operate themselves!

Operations

Expertise

Servers Networks

Storage Etc.

Ops automation (“devops”) jumpstarts learning, reduces complexity and manages scale.

Operations

Expertise

Our Topic

Operations Automation (DevOps)

Servers Networks

Storage Etc.

DevOps is CAMS

‣  Culture

‣  Automation

‣  Measurement

‣  Sharing

Foundation for Success: Open Operations

‣  Start with “Operations as Code”

‣  Leverage the community ‣  Write Operations code so it can be openly shared

‣  Learn from other people’s experience

‣  Make operations a priority ‣  Engage developers in operations challenges

‣  Make deployability a dev & test concern

‣  These are central issues to Opscode & Dell

What is Chef? ‣  An Open Source automation platform to continuously

define, build, and manage infrastructure.

‣  Chef enables people to easily build & manage complex & dynamic applications at massive scale

‣  Built on the idea of Infrastructure as Code

‣  Programmatically provision and configure nodes

‣  Reconstruct business from code repository, data backup, and bare metal resources.

How Can Chef Help?

Build Anything…

Provision compute resources in the Data Center and the Cloud

• Infrastructure

• Application Stacks

• Big Data

• HPC

• IaaS

• PaaS

• SaaS

And Manage It Simply

• Introduce continuous incremental change or total change.

• Automatically reconfigure everything

• Re-provision for disaster recovery

• Fail-over to bare metal

• Monitor for compliance

• Cloud migrations become trivial

Blueprint Your Infrastructure

• Compute

• Application

• Storage

• Security

• Network

• Configuration Standards

Using 1,000’s of man-

days of prior art!

A model for reuse that works!

500+ Cookbooks

© Opscode, 2011 – Confidential – DO NOT DISTRIBUTE 9

Full �Infrastructure Automation

Common �Automation

Tasks: Scripts, OS Compliance, Updates & Patches

Configuration Management Discovery and

Visibility

Application Management

Continuous Deployment

The Path to Full Automation

OpenStack has a lot of interlocking parts!

…and it matters how they are deployed!

Cookbooks

Clouds are a complex: not like an app ‣  Building up OpenStack requires a system view

‣  Breaking projects into discrete cookbooks enables maintainability

‣  Conventions are needed so that projects and cross reference each other

‣  Networking is critical

‣  Each component requires correct VLAN, teaming and virtual bridge configuration.

‣  Each site has unique networking needs (IP, subnet, VLAN)

‣  NIC enumeration can break even the simplest scipts

‣  Hardware configuration is boring but very important

‣  Application choices force configuration changes (late binding)

‣  Small mistakes can be very expensive to find and fix

‣  We needed a way to make deploy simpler, faster & repeatable

OpenStack Installer: Dell Crowbar Project

‣  Field Proven Technologies

‣  Build on locally deployed Chef Server, Raw servers to full cloud in <2 hours, Multi-Node from the ground up, Hardened with over a year of real deployments

‣  Apache 2 Open Source (on github/dellcloudedge)

‣  Multi-Apps (OpenStack & Hadoop), Multi-OS (Ubuntu, RHEL, CentOS, SUSE), Development “in the open” w/ community engagement

‣  OpenStack Releases (dell.com/OpenStack)

‣  Cactus, Diablo and Essex!

‣  Not Dell Hardware Limited

Crowbar Builds Up Clouds in layers D

ell “

Cro

wb

ar”

Op

s M

anag

emen

t

Core Components & Operating Systems

Cloud Infrastructure

Physical Resources

APIs, User Access, & Ecosystem Partners

1) Hardware Configuration D

ell “

Cro

wb

ar”

Op

s M

anag

emen

t

Core Components & Operating Systems

Cloud Infrastructure

Physical Resources

APIs, User Access, & Ecosystem Partners

2) OS and Ops Infrastructure D

ell “

Cro

wb

ar”

Op

s M

anag

emen

t

Core Components & Operating Systems

Cloud Infrastructure

Physical Resources

APIs, User Access, & Ecosystem Partners

* Suse pending, based on open source activity

*

3) Cloud Platform Integration D

ell “

Cro

wb

ar”

Op

s M

anag

emen

t

Core Components & Operating Systems

Cloud Infrastructure

Physical Resources

APIs, User Access, & Ecosystem Partners

* Suse pending, based on open source activity

*

4) Cloud App Management D

ell “

Cro

wb

ar”

Op

s M

anag

emen

t

Core Components & Operating Systems

Cloud Infrastructure

Physical Resources

APIs, User Access, & Ecosystem Partners

OpenStack Orchestration: Chef & Crowbar

Server(s)

Del

l “C

row

bar

” O

ps

Man

agem

ent

Operating System

Chef-Client

Chef-Client

Chef-Client

Chef-Client

Chef Server

Network Configuration

Status (post)

DHCP

Cro

wb

ar M

agic

Hap

pen

s!

Orchestration State Machine

Run List

Config.

PXE Boot

Hardware Config (uses “Sledgehammer”)

OpenStack Components

Crowbar & OpenStack Cookbooks

‣  Having cookbooks is not enough, Crowbar also ‣  Understands how the cookbooks connect together

‣  Helps you plan out the networking & server allocations

‣  Crowbar has a consistent Chef design approach ‣  “Proposal” databags to setup attributes and run-lists

‣  Consistent naming conventions

‣  Roles to link things together – A LOT OF ROLES

‣  Crowbar adds many cookbooks for network, monitoring, provisioning & hardware config.

Chef for OpenStack ‣  Community for the automated deployment and

management of OpenStack ‣  opscode.com/openstack/

‣  Cookbooks

‣  Documentation

‣  Mailing list ‣  groups.google.com/group/opscode-chef-openstack

‣  IRC

‣  #openstack-chef on irc.freenode.net

Chef for OpenStack Goals Already a lot of OpenStack deployments managed with Chef

‣  Provide a stable set of core OpenStack cookbooks

‣  Document how the pieces work together

‣  Evolve to keep up with latest changes

‣  Expand support into new technologies ‣  Operating systems, databases, hypervisors, etc.

‣  Alternate storage types

‣  Alternate networking implementations for Quantum

Chef for OpenStack Roadmap ‣  Get stable Essex cookbooks out

‣  Establish patterns of collaboration

‣  Continuous Integration and gated testing

‣  Folsom milestones

‣  Documentation, documentation, documentation

Upstreaming accelerates strong deploys

‣  Feedback loops enable cookbooks to mature quickly

Community Cookbooks

Dell Crowbar

Dell Customers

Crowbar Community

Chef

Chef Community

Host B

Cloud A Customers

Cloud A

Infrastructure as Code Repositories ‣  Deploy your data center with Crowbar

‣  Store this in version control

‣  Test your changes

‣  Promote to production

‣  Rinse, repeat

‣  OpenStack deployment is managed with Chef ‣  See above

‣  Deploy your applications on OpenStack with Chef

‣  See above

Deploy Your Applications on OpenStack ‣  Nova Dashboard

‣  Knife OpenStack

‣  enStratus

‣  Hybrid Cloud!?

knife-openstack

knife-openstack $ knife openstack

Available openstack subcommands: (for details, knife SUB-COMMAND --help)

** OPENSTACK COMMANDS **

knife openstack flavor list (options)

knife openstack image list (options)

knife openstack server create (options)

knife openstack server delete SERVER [SERVER] (options)

knife openstack server list (options)

knife-openstack knife openstack server create --flavor 1 --image 7 -S keypair -r 'role[hadoop-worker]’

‣  Deploy applications on top of OpenStack

‣  Manage them with Chef

‣  Configuration backed by version control

‣  Transfer your infrastructure anywhere

Challenges

‣  Making Community work sharable

‣  Testing / Validation ‣  Community

‣  Keeping on Upstreams

‣  Automation (Tempest & CI)

‣  Linking Dev and Ops

‣  Upgrade Process

‣  # of configurations

Join the Community & Get Started ‣  Invest in reusing community code

‣  Respect abstractions & patterns from community code

‣  Participate in communities

‣  Submit bugs (and patches!)

‣  Use Upstreams! ‣  Community Cookbooks… Opscode

‣  community.opscode.com

‣  Deploy Infrastructure… Dell Crowbar

‣  github.com/dellcloudedge/crowbar

Thanks! Rob Hirschfeld

‣  rob_hirschfeld@dell.com

‣  Twitter: @zehicle

‣  Blog: robhirschfeld.com

Matt Ray

‣  matt@opscode.com

‣  Twitter/GitHub/IRC: @mattray

‣  Blog: leastresistance.net