Red Hat Storage Server: Roadmap & integration with OpenStack Sayan Saha Sr. Manager, Storage Product Management, Red Hat Vijay Bellur Architect, Gluster, Red Hat

How many hours of videos are uploaded to YouTube every

hour?

Use Case: File Store for Big Data Analytics

Profile

  Leading automobile manufacturer

  Store & Analyze sensor data for next generation automobiles

  Expected data growth – 200 TB per week

  5 PB total data store

Pain Points

 Ability to scale rapidly with demand

 Cost-effective scaling

Solution & Outcome

  Selected Red Hat Storage Server to scale cost-effectively with demand

  Leverage SMB & native client client for rapid data ingest

Use Case: Content Store for video production

Profile

  Leading “video production as a service” provider

  Ingest video from various sources, transcode in 9 formats, distribute via web servers

Pain Points  Un-supported storage platform. Using community Gluster  Storage platform key to their principal business process  Could not afford to be on an un-supported platform

Solution & Outcome

  Adopted Red Hat Storage Server to provide a cloud based video platform that enables media processing, distribution & IP streaming

  Simplify delivery of content for TV and content providers.

Use Case: Disaster Recovery Using Replicated Storage

Profile

  Public Transportation Provider

  Filestore for storing data received from various sensors in the subway line that trains monitor

  Replicated copy needed in a nearby data center for continuous availability of the monitoring service

Pain Points

 Stuck with an End of Life proprietary clustered file system

 Unreliable with weak replication capabilities

 Wanted to re-use existing commodity hardware

Solution & Outcome

  Adopted Red Hat Storage Server to provide a reliable monitoring solution for the subway system

  Access from both Windows & Linux boxes

  Secondary usage as a document store

Red Hat Storage Roadmap Strategy

Create the best open software-defined storage for File Serving

Create the best storage provider for

RHELOSP

Continue to do foundational work for Big Data Storage

Use Cases & Workloads Use Case Workload Description Example Workload

Instance(s)

Content Cloud – Storage for unstructured data

Storing & Accessing files with I/O patterns - Write infrequently, Read infrequently/many/never

OwnCloud, pydio, backup target for Commvault, Document & File Store

Storage Provider for OpenStack

Scale-out block & object Storage back-end for Cinder, Nova & Glance

Storage for Big Data Analytics

Log analytics, Big Data batch analytics & Big Sciences Analytics

Splunk, Hadoop Map-reduce, Illumina

Hardware Advances Helping the Cause

GlusterFS Upstream Innovation Pipeline

Red Hat Storage Technology Stack Libgfapi (not public)

GlusterFS Upstream Roadmap GlusterFS 3.5 (GA within a few days)  Distributed geo-replication  Quota Scalability  Readdir ahead translator  File snapshots for virtual machine image files  libgfapi support for NFS Ganesha  Brick failure detection  Encryption at rest (Experimental)  On wire Compression translator (Experimental)

GlusterFS 3.6 (June 2014)  Volume Snapshot   AFRv2   Data Classification   SSL support   Disperse Volumes (erasure coding)   Heterogeneous Brick support   Trash translator   Better Peer Identification   RDMA Improvements

GlusterFS 3.7 Predictions

  Scalability for Big Data   Content cloud enhancements -  Sharding & Multi-protocol compatibility

  Improvements for OpenStack   Data Protection Improvements -  Geo-replicated snapshots, bitrot detection

  Support Btrfs features   SSD/Flash leverage

Best open software-defined storage for file serving

Looking  Back  …  

  Red  Hat  Storage  Server  2.0  launched  June  2012    6  numbered  updates  released    VM  image  store,  performance  &  stability    EOL  June  2014    Red  Hat  Storage  Server  2.1  launched  September  2013    2  numbered  updates  released  so  far    Quota,  Geo-­‐Rep,  management  Console,  SMB  2.0    Current  shipping  release  

Red Had Storage Server Roadmap Summary

3.0 (Denali) mid-year 2014 Theme: data protection & storage management • RHEL 6 & GlusterFS 3.6 • Volume Snapshots with user

serviceability • Monitoring Using Nagios • Hadoop Plug-In • 60 drives per server (up from 36) • Non-disruptive upgrade from previous

major version • Catalog/ID based Logging

3.1 (Everglades) 1H CY 2015

Theme: TCO reduction • RHEL 6 & GlusterFS 3.7 • 3-way replication • SSD support

• SSDs as bricks • SSDs for tiering

• Snapshot Enhancements • Support for RAID less h/w

configurations • NFSv4 full support (tentative) …

Zooming  In  on  RHSS  3.0  (Denali)  

  Official  Version:  Red  Hat  Storage  Server  3    Theme:  Data  protecQon  &  storage  management    Based  on  GlusterFS  3.6    Denali  releases  on  RHEL  6  mainline  not  EUS    Underlying  filesystem:  XFS    Underlying  Volume  Management:  dm-­‐thinp  

Snapshots  

  Point  in  Qme  copy  of  a  GlusterFS  volume    Create,  list,  status,  info,  restore  &  delete    Support  a  maximum  of  256  snapshots  per  volume    A  snapshot  can  be  taken  on  one  volume  at  a  Qme    Snapshot  names  need  to  be  cluster-­‐wide  unique    Management  via  CLI  only    User  serviceable  snapshots  in  scope  but  may  need  some  more  Qme  to  stabilize  

Scope of Monitoring

  Monitor RHSS Logical Entities – cluster, volume, brick, node   Monitor Physical Entities – CPU, disk, network   Alerting when critical components fails – SNMP   Reporting – historical record of outages, events, notifications   Trending – to enable capacity planning

Monitoring  Using  Nagios:  Supported  Use  Cases

  Use-­‐Case  1:  user  has  no  exisQng  monitoring  infrastructure  in  place  or  does  not  use  Nagios  

  Use-­‐Case  2:  user  already  has  Nagios  infrastructure  in  place  –  use  plugins  only  

  Use-­‐Case  3:  usage  in  conjuncQon  with  Red  Hat  Storage  Console  

Red  Hat  Storage  as  an  Add-­‐On  –  RPM  based  delivery    Considering  this  packaging  opQon  to:  

– Comply  with  corporate  governance  &  security  requirements    

– Beber  support  storage  co-­‐resident  applicaQons    – Embedded  use-­‐cases  – Ease  of  usage  for  channels  &  partners  

 

GlusterFS + Other RPMs + XFS

RHEL

GlusterFS + Other RPMs + XFS

RHEL

GlusterFS + Other RPMs + XFS

RHEL

RHSS

RHSS

RHSS

RHSS ISO RHSS as an Add-ON

 3-­‐way  replicaQon   SSD  support  

 Bricks  using  SSDs   Tiering  using  RHEL’s  dm-­‐cache  

 Snapshot  Enhancements   Consistency  Groups   Snapshot  scheduling  

 Support  for  RAID  less  h/w  configuraQons   NFSv4  full  support    

Zooming  In  on  RHSS  3.1  (Everglades)  

 Bit  Rot  DetecQon  &  RestoraQon   Erasure  coding  using  the  disperse  translator     MulQ-­‐protocol  support  for  NFS  &  SMB  

 pNFS  server  side  support      

Everglades  pipeline  …  

The path to best storage for OpenStack

Best storage provider for OpenStack

  Goal  –  Create  the  best  storage  offering  for  RHELOSP  

  What  is  needed?  –  Focus  on  requirements  and  feature  set  that  are  specific  to  OpenStack’s  storage  

substrate  –  Match  product  delivery  model  and  life-­‐cycle  requirements  to  address  

expectaQons  of  RHELOSP  adopters  –  Clear  offering  for  the  market  that  tracks  to  the  RHELOSP  roadmap  

  Plan  –  Create  a  new  product  family  that  is  exclusively  targeted  to  address  OpenStack’s  

storage  use  cases   25

New  Product  Family:  Red  Hat  Storage  Server  for  RHELOSP   Principal Product Line: Red Hat Storage

  Product Family: Red Hat Storage Server for RHELOSP

  Target Use-case: OpenStack Storage Provider

  Delivery Model: RPM-only & CDN

  Delivery Vehicle: Layered product on RHELOSP

  SKU: Provides access to RHSS RPMs only

  Pricing: Align with RHELOSP’s pricing model

  Roadmap: OpenStack Storage & management feature set

Content

  IniQally  derived  from  Red  Hat  Storage  Server  –  Start  with  the  same  core  package  set  

  Package  set  added/removed  as  needed    Provisioning,  configuraQon  management  &  monitoring  will  be  fully  aligned  with  RHELOSP’s  roadmap  &  capabiliQes

April 2014

Deploy & Configure with Foreman & Puppet

Post IceHouse

Cinder enhancements • Backup-Restore • Migrate • Per project user quotas Co-residency of storage & compute Native Swift

Juno

OpenStack-m • Deployment &

configuration for the undercloud

File-As-A-Service (Manila)

Red  Hat  Storage  Server  for  RHELOSP  Roadmap  

RHELOSP  +  RHSS  Co-­‐Residency:  Logical  View

                         RHELOSP  Node  1              Node  1  

                         RHELOSP  Node  2  

                         RHELOSP  Node  N  

                                                                                 RHSS  for  RHELOSP  

Nova  VMs  0..N  

Local  Storage  

RHELOSP  cinder  

Nova  VMs  0..N  

Local  Storage  

Nova  VMs  0..N  

Local  Storage  

RHELOSP  cinder  RHELOSP  cinder        

What’s up with Big Data Storage?

Benefits  of  using  RHSS  for  Hadoop  AnalyVcs  

  NFS & FUSE support for data ingestion   No single point of failure   POSIX compliance   Co-location of compute & data   Multiple Volume Support   Ability to partition data across multiple name spaces   DR capabilities

Hadoop  Plug-­‐In  Roadmap  

Feb 2014

High Touch Beta Released on RHSS 2.1 • Single volume in a cluster with one brick per server

• Supported for HDP 2.0.6 and it’s 8 services: Pig, Hive, HBase, sqoop, Flume, Oozie, Mahout & Zookeeper

RHSS 3.0 (Denali) Promote the Hadoop Plug-In to GA status • Support multiple volumes per cluster running Hadoop

• Run in-place analytics on existing data in RHSS volumes

• Support Hadoop distro HDP 2.1 and it’s new services like Tez & Storm

Summary

  Focus: File Serving on commodity hardware

  New product family for RHELOSP storage

  Foundational work for Big Data Workloads

Check Out Other Red Hat Storage Activities at The Summit   Enter the raffle to win tickets for a $500 gift card or trip to LegoLand! -  Entry cards available in all storage sessions - the more you attend, the more

chances you have to win!   Talk to Storage Experts:  Red Hat Booth (# 211)

-  Infrastructure -  Infrastructure-as-a-Service

  Storage Partner Solutions Booth (# 605)   Upstream Gluster projects -  Developer Lounge

Thank You. Please fill out the feedback forms.