webinar - introduction to ceph and openstack

Inktank Delivering the Future of Storage

Introduction to Ceph and OpenStack January 24, 2013

Agenda

•  Software and Companies •  Cloud Storage Considerations •  Ceph architecture and unique features and benefits •  Ceph OpenStack Best practices •  Resources •  Next steps

•  Distributed unified object, block and file storage platform

•  Created by storage experts

•  Open source

•  In the Linux Kernel

•  Integrated into Cloud Platforms

•  Company that provides professional services and support for Ceph

•  Founded in 2011

•  Funded by DreamHost

•  Mark Shuttleworth invested $1M

•  Sage Weil, CTO and creator of Ceph

What is OpenStack? OpenStack is an open standards and open source software with a mission to provide scalable and elastic cloud operating system for both private and public clouds, large and small. It has:

• Largest and most active community • High-profile contributors (ATT, Dell, HP, IBM, Rackspace etc) • Regular release cycles (6 months cadence) • Governed by an independent foundation • Broad suite of services • Sound architecture built from the ground-up in a scale-out manner, loosely-coupled, asynchronous message based, distributed software system • Ideal for massively scalable applications

What is in OpenStack?

OpenStack has a collection of projects that are developed and maintained collaboratively by an active and large community:

– Compute (Nova) – Storage (Swift) – Glance (Imaging) – Horizon (Dashboard) – Keystone (Authentication) – Quantum (Network service)

OpenStack’s basic requirement: “Clouds must be simple to implement and massively scalable.”

OpenStack Value Proposition

•  Limits costly software licenses

•  Limits lock-in by proprietary vendors & by cloud providers (Amazon)

•  Allows for massive scalability

•  Open source hypervisor support – KVM, Xen, LXE – ESX & HyperV support is lagging

•  Offers standard APIs enabling growing cloud ecosystem

Dell Positioned Well in OpenStack

Market Offerings

•  FIRST hardware solutions vendor to back OpenStack

•  FIRST OpenStack integrated Solution in market •  FIRST OpenStack deployment solution: Crowbar •  Industry recognition of Dell leadership •  Regular releases, webinars, white papers, case

studies, press coverage, etc

Community Advocacy

•  Sponsor of each OpenStack Conference to date •  Lead monthly Austin & Boston OpenStack meetups •  Numerous cloud / community events worldwide

OpenStack Governance

•  Gold-level sponsor of new OpenStack Foundation •  2 seats on Foundation Board of Directors •  Active participation in developing community

bylaws, code of conduct, etc

Crowbar

•  Winning, differentiated Dell IP •  Owner / Leader of Crowbar OSS community

•  470 followers; 31,000+ Crowbar site hits in 90 days •  2,000+ downloads in 6 months •  2nd most active Dell Listserv •  Multiple “Hack Days” – 100s of world-wide

participants •  Corp contributors include Intel, Suse, Rackspace,

others

Opensource Software Management Framework •  Innovative approach to bare metal deployment of complex

environments such as Cloud (Openstack and Big Data / Hadoop)

•  Extendable architecture that includes Barclamps –  Independent modules that add functionality such as support for

network Switches or new software components such as Ceph

•  Intel, Secureworks, VMware Cloud Foundry,Dreamhost Ceph database, SUSE Operating System and Rackspace have all contributed to community

•  Enables “operational model “ that deploys bare metal reference architectures to fully operational environments in hours (Openstack and Hadoop)

•  Designed for Hyperscale environments –  Anticipates extensions via “Barclamps” –  Plans for scale using “Proposals”

•  Focuses on shift to DevOps management capabilities and continuous integration

What is Crowbar?

Inktank and Dell Partnership

•  Inktank is a Strategic partner for Dell in Emerging Solutions

•  The Emerging Solutions Ecosystem Partner Program is designed to deliver complementary cloud components

•  As part of this program, Dell and Inktank provide: –  Ceph Storage Software

›  Adds scalable cloud storage to the Dell OpenStack-powered cloud

›  Uses Crowbar to provision and configure a Ceph cluster –  Professional Services, Support, and Training

›  Collaborative Support for Dell hardware customers –  Joint Solution

›  Validated against Dell Reference Architectures via the Technology Partner program

Cloud Storage Considerations

Cloud Storage Requirements

•  Scalable to Many Petabytes •  Flexible Configuration •  Rapid Provisioning •  Delegated Management •  Commodity Hardware •  No Single Point of Failure •  Self-Managing with Fail-in-Place Maintenance •  Dynamic Data Placement •  Proportional Data Movement •  Support for VM Migration •  Unified Storage Capabilities

•  Swift has been a good Object Storage starting point –  Mature and solid foundation for objects and images –  Can be a headache across failures and hardware reconfiguration –  Object only: no Block or File capabilities –  Not for use cases that require

›  High performance storage ›  Database operations ›  Use of existing application that need a file system or block-storage access

•  Nova-Volume using LVM and iSCSI –  Many layers between storage and VM instances

›  LVM, iSCSI Target, iSCSI Initiator, Linux Disk, Hypervisor –  Node a single point of failure –  RAID reconstruction takes a long time and degrades performance

Addressing OpenStack Gap in Block Storage

•  Integrated with Crowbar – Provides a barclamp to provision and configure a Ceph cluster and simplifies growing deployments

•  Tested with OpenStack – Extensive testing of Ceph with OpenStack over several years of both products’ evolution by the Ceph team

•  Lowers cost per Gigabyte – Delivers the capabilities of expensive traditional block storage solutions at utility hardware prices

•  Scalable to many Petabytes – Single cluster can hold hundreds of nodes and many Petabytes of managed data

•  No Single Point of Failure – Infrastructure-aware data placement distributes replicas across fault zones to ensure NoSPoF

•  Self-Managing and Fail-in-Place – Autonomous operation allows deployment in remote datacenters without worry about frequent visits

•  Dynamic data placement – Data distributed evenly across the cluster, and moved only in proportion to how much of the cluster changes

•  Unified Storage – Match access mechanism to application needs by using Ceph’s Block, Object, or File access modes.

Ceph Closes the Block Gap

Key Differentiators CRUSH data placement algorithm Metadata is primarily computed rather than stored CRUSH computation is distributed to the most scalable components Algorithm is infrastructure aware and quickly adjusts to failures Advanced Virtual Block Device Enterprise storage capabilities from utility server hardware Thin Provisioned, Allocate-on-Write Snapshots, LUN cloning In the Linux kernel and integrated with OpenStack components Open Source Solution Maximize value by leveraging free software Control own destiny with access to the source code Customize and modify the code to provide differentiated value Unified storage platform (Object + Block + File) Multiple uses cases satisfied in a single storage cluster Manage single, autonomous system integrated with OpenStack Economies of scale from sharing large set of disks with many apps

Ceph Provides Advanced Block Storage Capabilities for OpenStack Clouds

•  Persistence – volumes are persistent by default; VMs behave more like traditional servers and do not disappear when you reboot them

•  Host independence – enables VM migration; compute and storage resources can be scaled independently; compute hosts can be diskless

•  Easy snapshots – snapshots of instances are easier and act more like backups

•  Easy cloning – 2 step process: •  Use API to create a volume •  Populate it with the contents of an image from Glance, •  You are ready to boot from the new volume

•  Thin provisioning – fast instance creation with copy-on-write cloning if you store both your Glance images and your volumes as Ceph block devices

Easy Spin-up, Back-up, and Cloning of VMs

Ceph Technology Overview

Ceph Object Storage (RADOS) A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes

Ceph Object Library (LIBRADOS) A library allowing applications to directly access Ceph Object Storage

Ceph Block (RBD) A reliable and fully-distributed block device

Ceph Distributed File System (CephFS) A POSIX-compliant distributed file system

Ceph Object Gateway (RADOS Gateway) A RESTful gateway for object storage

APP APP HOST/VM CLIENT

Ceph

Monitors: • Maintain cluster map • Provide consensus for distributed decision-making

• Must have an odd number • These do not serve stored objects to clients

M

RADOS Storage Nodes containing Object Storage Daemons (OSDs): •  One OSD per disk (recommended) •  At least three nodes in a cluster •  Serve stored objects to clients •  Intelligently peer to perform

replication tasks •  Supports object classes

RADOS Components

DISK

FS

DISK DISK

OSD

DISK DISK

OSD OSD OSD OSD

FS FS FS FS btrfs xfs ext4

M M M

RADOS Cluster Makeup

RADOS Cluster

RADOS Node






Distributed Block Access


• Store virtual disks in RADOS • Allows decoupling of VMs and nodes

Ø  Live Migration • Images are striped across the cluster for better throughput • Boot support in QEMU, KVM, and OpenStack Nova • Mount support in the Linux kernel for VM applications

RADOS Block Device

M

M

M

VM

LIBRADOS LIBRBD

VIRTUALIZATION CONTAINER

•  Boot Device •  Application Storage

LIBRADOS

M

M

M

LIBRBD

COMPUTE NODE

LIBRADOS LIBRBD

COMPUTE NODE VM

Live Migration

LIBRADOS

M

M

M

KRBD (KERNEL MODULE)

HOST

Accessible by Host







RADOS Gateway Web Services Access (REST) for Applications to Object Storage

M

M

M

LIBRADOS

RADOSGW

APP

Native Protocol

REST

LIBRADOS

RADOSGW

APP • REST-based interface to RADOS

• Supports buckets, accounting

• Compatible with S3 and Swift applications

REST

RADOS Gateway Web Services Access (REST) for Applications to Object Storage

VOTE

Using the Votes Bottom on the top of the presentation panel please take 30 seconds answer the following questions to help us better understand you. 1.   Are you currently exploring OpenStack for a project? 2.   Are you looking to implement Ceph for OpenStack storage within the next 6 months? 3.   Do you anticipate wanting help to deploy OpenStack and Ceph, for your project?

OpenStack and Ceph Best Practices

Configure Networking within the Rack

•  Plan for low latency and high bandwidth •  Management tasks on low speed access switch •  Leaf switch carries both front (client) and back (OSD) traffic •  2x 40GbE uplink trunks created by aggregating 4x port per trunk

Connect to Nodes in Rack

Connect to Spine Routers

High-Speed Distribution (Leaf) Switch e.g., Dell Force10 S4810

Low-Speed Access Switch

e.g., Dell Force10 S55

Front Side

Back Side

IPMI Management

1GbE link 10GbE link 40GbE link

Configure Networking within the Pod

•  Each Pod (e.g., row of racks) contains two Spine switches •  Each Leaf switch is redundantly uplinked to each Spine switch •  Spine switches are redundantly linked to each other with 2x 40GbE •  Each Spine switch has three uplinks to other pods with 3x 40GbE

10GbE link 40GbE link

High-Speed Top-of-Rack (Leaf) Switch

Nodes in Rack


Nodes in Rack


Nodes in Rack

High-Speed End-of-Row

(Spine) Switch

High-Speed End-of-Row

(Spine) Switch

To Other Rows (Pods) To Other Rows (Pods)

Configure Networking between Pods

•  Each Pod is placed on two counter-rotating rings with cut-through •  Paths maintained via routing tables •  Traffic between Pods can take:

•  Clockwise path •  Counterclockwise path •  Cut-through path

•  Ensures that maximum number of hops is N/4 for N pods

Spine Router

Spine Router

Spine Router

Spine Router

Spine Router Spine Router

Spine Router Spine Router

Object Storage Daemons (OSD)

•  Allocate enough CPU cycles and memory per OSD —  2GB memory and 1GHz of Xeon CPU cycles per OSD — Usually only a fraction needed

›  in error cases all can be consumed —  Trade off over-provisioning with risk tolerance

•  Use SSDs as Journal devices to improve latency —  Some workloads benefit from separate journal on SSD

•  Consider different tiers of storage for mixed application clusters — Can mix OSDs on different types of disks to create tiers —  For each pool, choose the tier and level of replication

Ceph Cluster Monitors

•  Best practice to deploy monitor role on dedicated hardware —  Not resource intensive but critical —  Using separate hardware ensures no contention for resources

•  Make sure monitor processes are never starved for resources —  If running monitor process on shared hardware, fence off resources

•  Deploy an odd number of monitors (3 or 5) —  Need to have an odd number of monitors for quorum voting —  Clusters < 200 nodes work well with 3 monitors —  Larger clusters may benefit from 5 —  Main reason to go to 7 is to have redundancy in fault zones

•  Add redundancy to monitor nodes as appropriate —  Make sure the monitor nodes are distributed across fault zones —  Consider refactoring fault zones if needing more than 7 monitors

Mixed Use Deployments

•  For simplicity, dedicate hardware to specific role —  That may not always be practical (e.g., small clusters) —  If needed, can combine multiple functions on same

hardware •  Multiple Ceph Roles (e.g., OSD+RGW, OSD+MDS, Mon+RGW)

—  Balance IO-intensive with CPU/memory intensive roles —  If both roles are relatively light (e.g., Mon and RGW) can

combine •  Multiple Applications (e.g., OSD+Compute, Mon+Horizon)

—  In OpenStack environment, may need to mix components —  Follow same logic of balancing IO-intensive with CPU

intensive

Deploying Ceph Roles and Objects

•  Ceph depends on replication for reliability — Data (objects) and Roles (hardware/software) exist in multiple

places — RAID can be used but generally just adds to the expense

•  Ceph incorporates notion of multiple nested fault zones — Ceph understands nested fault zones — Each OSD is tagged with location in multiple zones — CRUSH uses info to distribute data as protection from zone

failures •  Roles need to be replicated in the cluster

— Ensure there are multiple nodes in the cluster with needed roles ›  Monitors (3 minimum for production) ›  OSD nodes (2 minimum for production)

•  Ceph needs sufficient nodes per fault zone to replicate objects — Ensure enough of each role exist in each fault zone

Leverage Dell Ceph Expert Support

•  Dell and Inktank are your partners for complex deployments •  Solution design and Proof-of-Concept •  Solution customization •  Capacity planning •  Performance optimization

•  Having access to expert support is a production best practice •  Troubleshooting •  Debugging

Resources for Learning More

Resources

OpenStack-Powered Cloud Solutions: http://content.dell.com/us/en/enterprise/

by-need-it-productivity-data-center-change-response-openstack-cloud

Inktank site

http://www.inktank.com/dell www.OpenStack.org

Community site Mailing lists, design discussions,

blogs, news, events Dell OpenStack Bloggers

BartonGeorge.net RobHirschfeld.com JBGeorge.net Cloudel.com (Kamesh Pemmaraju)

What Next?

Dell and Inktank’s Professional Services

Consulting Services: •  Technical Overview •  Infrastructure Assessment •  Proof of Concept •  Implementation Support •  Performance Tuning

Support Subscriptions: •  Pre-Production Support •  Production Support

A full description of our services can be found at the following: Consulting Services: http://www.inktank.com/consulting-services/ Support Subscriptions: http://www.inktank.com/support-services/

Check out our upcoming and on demand webinars

Upcoming: 1.  DreamHost Case Study: DreamObjects with Ceph

February 7, 2013 10:00AM PT, 12:00PM CT, 1:00PM ET http://www.inktank.com/news-events/webinars/

2.  Advanced Features of Ceph Distributed Storage (delivered by Sage Weil, creator of Ceph) February 12, 2013 10:00AM PT, 12:00PM CT, 1:00PM ET http://www.inktank.com/news-events/webinars/

On Demand:

Getting Started with Ceph http://www.inktank.com/news-events/webinars/

Contact Us

Inktank: [email protected] and 1-855-INKTANK Don’t forget to follow us on: Twitter: https://twitter.com/inktank Facebook: http://www.facebook.com/inktank YouTube: http://www.youtube.com/inktankstorage

Dell: [email protected] http://www.dell.com/OpenStack http://www.dell.com/CrowBar https://github.com/dellcloudedge/

Thank You!

webinar - introduction to ceph and openstack

Technology

cloud openstack

dell openstackpowered

ceph storage software

scalable cloud storage

openstack gap

openstack conference

hours openstack

file storage support