building an open source cloud storage platform for openstack - openattic

www.it-novum.com© it-novum GmbH

Seite 1

Building a flexible and large-scale software-defined storage platform for

OpenStack

OPENSTACK CEE DAYMay 26th 2014, Budapest

Michael Kienle, CEOit-novum GmbH

[email protected]


Seite 2

Source: RED HAT

Open Clouds are picking up right now!


Seite 3

Open Stack: Open cloud operating system supported by the OpenStack Foundation backed up by ### well-known companies

Want to learn more: listen to the other tracks

Major Design Guidelines are OPEN:Open source (apache), Open design (blueprints), Open development (branches), Open community (tech leads)

Although OpenStack is purposely open it partly relies on underlying proprietary storage products.

To build your cloud upon a truly OPEN software-defined storage platform (SDS) you might achieve greater flexibility and $$$$-savings.

openATTIC is a SDS cloud storage platform optimized for OpenStack's Cinder component.

What is this all about?


Seite 4

Introduction & AgendaOpenStack

Focus on Storage within OpenStack, esp. CinderRequirements & Optimization of current approaches

SDS – software-defined storageHow it worksSDS as part of the software-defined datacenter (SDDC)Why now?Advantages

openATTICHow to bring SDS and OpenStack togetherVideo/Examples

Agenda


Seite 5

Leading provider of open source business solutions75 employees currently; approx. 20% growth p.a.Headquarters in Fulda/DE; Branch office in Vienna/ATHundreds of mid-sized and large corporate clientsOperating public listed parent company‘s data center for 20+ yearsand migrated most of this to open source

Mission: Help customers to build and run a 100% open-source software-defined datacenter

Proven Expertise in Business Open Source Infrastructure and ApplicationsSAP Extended Business Partner (for integrating SAP into open source solutions)

Open Business Solutions. Delivered!


Seite 6

Generic Cloud Drivers: Abstraction, Automation, Self-Service, efficient use of resource pools, dynamic & massive scale, common APIs

Manage Virtualization across different Hypervisors (KVM, VMware, Xen, Hyper-V etc.)

Support the whole IaaS stack (Storage, Security, Networking etc.)

Scale far beyond the limitations of an „old virtualized landscape“

Enable the software-defined datacenter (SDDC)

OpenStack: Value Proposition


Seite 7

Generic Cloud Drivers: Abstraction, Automation, Self-Service, efficient use of resource pools, dynamic & massive scale, common APIs

Manage Virtualization across different Hypervisors (KVM, VMware, Xen, Hyper-V etc.)

Support the whole IaaS stack (Storage, Security, Networking etc.)

Scale far beyond the limitations of an „old virtualized landscape“

Enable the software-defined datacenter (SDDC)

OpenStack: Value Proposition


Seite 8

Storage costs optimization ($$$$)Data growth of >>40% p.a. needs to be managed automatically and way cheaper

Flexibility and reduced vendor lock-in (open)Incumbent players are perfect in securing their huge margins Using innovative technology without renewing existing assets

Cloud Computing has characterized the next generation IT delivery model, easy orchestration/automation of (storage) services is essential

need scale-up/scale-out of underlying storage at lesser costsSoftware-Defined DataCenter (SDDC) becomes a major disruptive IT trend (SDS beeing part of this)

Standardisation enables automation & abstraction of software from hardware -> Commodity hardware and SDS gained strength

Customers‘ Perspective on Storage


Seite 9

Cinder / Block Storage Swift / Object StorageObjectives • Storage for running VM

disk volumes on a host• Ideal for performance

sensititve apps

• Ideal for cost effective, scale-out storage

• Backup & Archive• Dropbox-like services

Use Cases • Production Applications• Traditional IT Systems• Database-driven Apps• Dev-/Test Landscape

• ISO Image- and VM-Templates

• Snapshots• Backup & Archive• Repository

Workloads • High chance content• Smaller, Random R/W• Higher / Bursty IO

• More static content• Larger, sequential r/w• Lower IOPS

Storage in OpenStack


Seite 10

Architecture to provide block-level resources to OpenStack („block storage as a service“)Persistent block-level storage volumes for use with OpenStack Nova compute instances (e.g. iSCSI target)Manages creation, attach/detach, clone of these storage volumensIntention of Cinder: keep you from beeing locked-in:

Ability to change the backend (the storage system) easilyMigrate data & expect the same

base level of functionalityMany vendor/storage plug-ins for Cinder available;

the choice between functionality and $$$$

Cinder - Details

LegendPresistent volume controlPresistent volume data

VM instance

Note that iSCSI is just an

example – several

additional protocols are

supported (e.g., FC, NFS)

Nova

/dev/vda

iSCSI initiator

Cinder

KVM

iSCSI target


Seite 11

Limitations of Commodity Storage (like a simple Linux server or comparable) because of lacking features like:

node redundancy & clustering & HA, scalingQoS, Deduplication/Compression, ReplicationExample: Image Cloning (e.g. image from Glance) could be faster than block-based copy&paste…

Limitations of Enterprise Storage (the big names):Vendor lock-in$$$$$

Consider Open Source & SDS !

Choose your underlying storage system!


Seite 12

SDS is the abstraction of the software from the hardware. Benefits:

Independence from big-name commercial developer of proprietary systems (bundle of hardware and storage os) for better pricing and freedom of choiceHardware is treated as a pool, scalable across multiple storage systemsInnovative functionality (e.g. top-down policy driven), easier to administrate, faster to implement (software can be extended and maintained more easily than hardware)Part of the Software-defined DataCenter (SDDC)

By 2016 the avail market for OpenSource-Storage is 2,5$B (Gartner, “Tech Go-to-Market: Storage Providers Should Focus on Three Growth Opportunities Through 2016”, Nov 2013), by 2018 OpenSource Storage will gain 20% of the market , approx. 10$B (Gartner, “2014 Strategic Roadmap for Storage”, March 2014).

Software-defined Storage (SDS)


Seite 13

ConfigurationManagement

System (CMS)

Service Transition Service OperationService Design

Monitoring

Event Correlation

User

Oth

erM

onito

rs

CapacityManagement

RiskManagement

Cloud Self Service

Sys

tem

App

licat

ion

End

-2-E

nd

Pro

cess

ChangeManagement

ProblemManagement

SA

P

Applications Service DeskNetwork Operation Center

ReleaseManagement

Cloud Management

IncidentManagement

Infrastructure

Software-defined Storage Virtualization

CMDB

Discovery

Rep

ortin

gS

ervi

ce L

evel

Man

agem

ent

The Big Picture: SDDC with 100% open source


Seite 14

ITSMi-doit & OTRS

Umbrella Monitoring

openITCOCKPIT

Cloud ManagementOpenStack

ReportingPentaho

Billing, Backup,...

SAP, SEP sesam

Configuration

Deployment

Services & Tools

API

Web

CLI

Apps

StatusChecking

Application Management

AccessSamba, NFS, AFP, iSCSI, FC

AvailabilityDRBD, Pacemaker,

Corosync, ...

EfficiencyZFS, LVM, BTFRS,

XFS

BackupSnapshotting, Tape-

Backup, ...

AnalysisopenITCOCKPIT,

Nagios, ...

Back end

Core

Front end

openATTIC Framework - Architecture

Interfaces


Seite 15

MirroringBlock- & File-based Synchronous & asynchronousObject-based

ClusteringAutomated Failover Multi Node Support

Support & ServicesTraining & OptimizationIntegration & Development Guaranteed Response Time Guaranteed Restore Time (Appliance only)

ProtectionIntegration Tape-SystemsSnapshottingConsistent Snapshotting

EfficiencyDeduplicationCompressionPerformance Config-Set Config-Set for Virtualization

Unified and versatileSAN / Block Storage

(FC, iSCSI, FCoE)NAS / File-based (CIFS, NFS)Object Storage

Central ConfigurationWeb FrontendCommand Line Interface

Dynamic MonitoringAvailabilityPerformanceCapacity

Cloud SupportOpenStack/CinderopenQRM

Umbrella ManagementMaster/Slave Architecture

Current FeaturesOpen API, Scalable, Cinder-ready


Seite 16

Volume create/deleteSnapshot create/deleteVolume attach/detach (called by Nova)Create Volume from SnapshotCopy Image to Volume / Volume to ImageGet Volume Stats, Clone VolumeExtend Volume (mandatory since Icehouse)

openATTIC Cinder plug-in: VideoWe show you some examples how this works…

How it works: Cinder API (Icehouse)


Seite 17

Unit testsCinder driver certification fileRelease hopefully next one…

More Info: www.openattic.org/enIntegration in other cloud platforms has been done already

OpenStack Cinder / openAttic: Status

http://www.openattic.org/en


Seite 18

Open and comprehensive API achivies great flexibility for integration into different cloud platforms Optimization of huge data through intelligent openATTIC functionalities

Consistent snapshots relieve backup, accelerate desaster recovery and reduce downtimesDeduplication, Compression, efficient mirroring and redundancy

Huge cost savings:openATTIC runs on commodity hardwarereducing vendor lock-in Open Source approach saves license costs

Are there limitations of software-defined storage / software-defined datacenter?

Summary: openATTIC within OpenStack


Seite 19

Open and comprehensive API achivies great flexibility for integration into different cloud platforms Optimization of huge data through intelligent openATTIC functionalities

Consistent snapshots relieve backup, accelerate desaster recovery and reduce downtimesDeduplication, Compression, efficient mirroring and redundancy

Huge cost savings:openATTIC runs on commodity hardwarereducing vendor lock-in Open Source approach saves license costs

However, the bad news is: even software-defined datacenters still need some underlying hardware…

Summary: openATTIC within OpenStack


Seite 20

Thank you!Questions? - More than welcome!

Cloud Storage

Intelligent SDS

Unified Storage

Power your cloud Rock-solid storage Scale out data center

Power your cloud & scale out your data center with rock-solid Software-Defined Storage (SDS) truly on open source, join the community & more

about the openATTIC project at http://openattic.org/en

http://openattic.org/en