![Page 1: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/1.jpg)
PRESENTATION TITLE GOES HERE Object Storage 101 Understanding the What, How and Why behind Object Storage Technologies
![Page 2: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/2.jpg)
2
Alex McDonald, SNIA –ESF File Protocol SIG Chair - NetApp
Today’s Presenters
Duncan Moore Director, Object Storage
NetApp, Inc.
Paul S. Levy System’s Engineer & Architect
Intel Storage Division
![Page 3: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/3.jpg)
SNIA Legal Notice
! The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted.
! Member companies and individual members may use this material in presentations and literature under the following conditions:
! Any slide or slides used must be reproduced in their entirety without modification ! The SNIA must be acknowledged as the source of any material used in the body of any
document containing material from these presentations. ! This presentation is a project of the SNIA Education Committee. ! Neither the author nor the presenter is an attorney and nothing in this
presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney.
! The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.
3
![Page 4: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/4.jpg)
Topics
! The Emerging Object Storage Market (Market Sizing and Growth)
! Contrasting approaches: Objects, Files & Blocks ! Object Storage Use-Cases ! Components of an Object Storage Solution ! Object Durability Approaches ! Design/Selection Considerations
4
![Page 5: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/5.jpg)
The World Has Gone Digital
24 Petabytes Data processed by
Google* every day in 2011
4 billion Pieces of content
shared on Facebook* every day by July
2011
7 Exabytes Data traffic by mobile users
worldwide in 2011
5.5 million Legitimate emails
sent every second in 2011
Managing petabytes is commonplace
![Page 6: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/6.jpg)
Block File Object
Block, File & Object
6
Specific location on disks / memory
Tracks
Sectors
Specific folder in fixed logical order
File path
File name
Date
Flexible container size
Data and Metadata
Unique ID
![Page 7: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/7.jpg)
Challenges driving the adoption of Object
7
Durability
Scalability Manageability
• Scalability – Accommodate boundless growth • Durability – Tolerate hard drive, system, and datacenter failures • Manageability – Accommodate seamless expansion and migration
![Page 8: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/8.jpg)
Workload will guide the choice…
Occasional
Massive Data Fast Data
File Systems ! Structured datasets ! Lots of readers and writers ! Location/path aware ! Needs volume management
Object Stores • Unstructured data • Embedded metadata • Write-once (immutable) • Location unknown • No volume management
Transactional
![Page 9: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/9.jpg)
A Sample Object Use-Case
9
Business Requirement Technical Requirements
Deliver an “Entertainment as a Service” offering. Manage a billion+ media across a huge range of sizes (MB’s àTB’s).
• Ability to provide SLO based storage and access • Erasure Encoding for cost optimization • Single Copy support • Simplified Install/Expansion/Operations • Half million jobs a day
Media Asset Management use-cases often push the boundaries of traditional storage approaches.
![Page 10: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/10.jpg)
Network Attached File System
! Scalability ! Strict volume and file limits ! File locking ! Fixed attributes (metadata)
! Durability ! Volume and/or RAID
replication ! Snapshot ! Low level solutions (RAID)
! Manageability ! POSIX interface
! Consistency ! Read after write consistent
![Page 11: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/11.jpg)
Object Store
! Scalability ! Flat namespace ! No volume semantics ! No Locking/Attributes ! Contains metadata
! Durability ! Replication or
Erasure code ! Manageability
! REST API ! Low overhead
! Consistency ! Eventually consistent
![Page 12: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/12.jpg)
Data Durability (Granularity)
! At what anchor point do you consider a protection policy?
! Block ! LUN’s – Logical container that is referenced.
! File ! Directory – Smallest reasonable unit to secure. ! Volume/Mount Point – Covenant reference point
! Object ! Object Property – Part of the declaration ! Namespace - Covenant reference point
12
![Page 13: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/13.jpg)
What is Erasure coding
! A method of Forward Error Correction which produces set of fragments by which only a subset is needed to re-hydrate.
! Erasure coding policy defines the number fragments that are created with the number of fragments needed to re-hydrate
! RAID5 & RAID6 are examples of Erasure Code. ! An 8 element RAID5 would be (7 of 8) ! An 8 element RAID6 would be (6 of 8)
13
![Page 14: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/14.jpg)
Erasure Code Encoding
14
![Page 15: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/15.jpg)
Erasure Code Decoding
15
![Page 16: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/16.jpg)
Object
N7 N8
f9 f5
Example: 6 of 9 Erasure Coding
Frag
men
t O
bjec
t
f1 f2 f3
f6
Encode 6 data + 3 parity
fragments
Dis
trib
ute
EC F
ragm
ents
Durability: EC Geo-Spreading
N9
f6
N4 N5
f3 f4
N6
f8
N1 N2
f1 f7
N3
f2
f1 f2 f3
f9
![Page 17: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/17.jpg)
Durability: Object Level Replication Policies
17
N7 N8 N9
! Example of advanced object level replication
N1 N2 N3
N4 N5 N6
Application
D1
D1 D1 D1
1. Upon ingest make a local replica and 2 remote copies
N6 N4
N7
N2
D1 D1
D1
D1
![Page 18: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/18.jpg)
Durability: Object Level Replication Policies
18
N7 N8 N9
! Example of advanced object level replication
N1 N2 N3
N4 N5 N6
Application
D1
D1 D1 D1
1. Upon ingest make a local replica and 2 remote copies
2. After 90 days remove local copies
N6 N4
N7
N2
D1
D1
![Page 19: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/19.jpg)
Durability: Object Level Replication Policies
19
N7 N8 N9
! Example of advanced object level replication
N1 N2 N3
N4 N5 N6
Application
D1
D1 D1 D1
1. Upon ingest make a local replica and 2 remote copies
2. After 90 days remove local copies
3. After 6 years remove all copies
N6 N4
N7
N2
![Page 20: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/20.jpg)
Design Considerations: Performance
! Throughput and Latency ! Object Size and Storage Policy ! PUT/GET Mix ! Scale of System
! Fault Insertion/Failure ! Network & CPU impact ! Application latency
![Page 21: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/21.jpg)
Design Considerations: API’s
! Learn what’s available. RESTful APIs are the norm, but there are many flavors. ! CDMI (Cloud Data Management Interface): SNIA
standard currently in v1.02) ! SWIFT: SWIFT is the object storage system
component of the OpenStack cloud software project.
! Amazon S3: S3 (Simple Storage Service) is Amazon’s cloud storage offering.
! Proprietary RESTful: Many Enterprise vendors include support for their own proprietary RESTful API along with one or more “standard” API’s. (ex. NetApp StorageGRID™ SGAPI)
! Factors influencing API selection ! What API’s do my desired ISV’s use? ! Do I have needs beyond simple CRUD? ! What expertise is available to me?
![Page 22: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/22.jpg)
! Software Only
22
! Appliance
+ Can integrate into existing IT infrastructure
+ May enable adoption of commodity h/w
- IT becomes the integrator - May result finger-pointing
support situations
+ Simple installation and provisioning
+ Single vendor support
- May have limited performance/scale options
Design Considerations: Form Factor
![Page 23: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/23.jpg)
Summary
! The Emerging Object Storage Market (Market Sizing and Growth)
! Contrasting approaches: Objects, Files & Blocks ! Object Storage Use-Cases ! Components of an Object Storage Solution ! Object Durability Approaches ! Design/Selection Considerations
23
![Page 24: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/24.jpg)
After This Webcast
! This webcast will be posted to the SNIA Ethernet Storage Forum (ESF) website and available on-demand ! http://www.snia.org/forums/esf/knowledge/webcasts
! A full Q&A from this webcast, including answers to
questions we couldn't get to today, will be posted to the SNIA-ESF blog ! http://sniaesfblog.org/
! Follow and contribute to the SNIA-ESF blog thread on many storage-over-Ethernet topics, both hardware and protocols ! http://sniaesfblog.org/
24
![Page 25: Object Storage 101 - SNIAObject Storage 101 Understanding the What, How and Why behind Object Storage Technologies. 2 ... Upon ingest make a local replica and 2 remote copies 2. After](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed057eace86386eaa5f066d/html5/thumbnails/25.jpg)
Conclusion
Thank You
25