![Page 1: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/1.jpg)
The Future of Software-Defined Storage – What Does It Look Like in 3 Years Time?
Richard McDougall, VMware, Inc
![Page 2: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/2.jpg)
This Talk: The Future of SDS
2
Hardware Futures & Trends
New Workloads
Emerging Cloud Native Platforms
![Page 3: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/3.jpg)
Storage Workload Map
3
1 Million IOPS
One Thousand
IOPS
10’s of Terabytes
10’s of Petabytes New
Workloads Focus of Most
SDS Today
![Page 4: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/4.jpg)
What Are Cloud-Native Applications?
Developer access via APIs
Microservices, not monolithic stacks
Application
Continuous integration and deployment
Availability architected in the app, not hypervisor
Built for scale
Decoupled from infrastructure
4
![Page 5: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/5.jpg)
What do Linux Containers Need from Storage?
• Ability to copy & clone root images
• Isolated namespaces between containers
• QoS Controls between containers
5
Parent Linux
Container 1
/
/etc /containers
/containers/cont1/rootfs
etc var
/containers/cont2/roofts
etc var
Container 2
Option A: copy whole root tree 5
![Page 6: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/6.jpg)
Containers and Fast Clone Using Shared Read-Only Images
• Actions – Quick update of child to use same boot image
• Downsides: – Container root fs not writable
6
/
/etc /containers
/containers/cont1/rootfs
etc var
6
![Page 7: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/7.jpg)
Docker Solution: Clone via “Another Union File System” (aufs)
CONFIDENTIAL 7
Except:
7
![Page 8: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/8.jpg)
Docker’s Direction: Leverage native copy-on-write filesystem
Goal: – Snap single image into writable clones – Boot clones as isolated linux instances
8
Parent Linux
Golden
/
/etc /containers
/containers/golden/rootfs
etc var
/containers/cont1/roofts
etc var
Container 1
# zfs snapshot cpool/golden@base # zfs clone cpool/root@base cpool/cont1
8
![Page 9: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/9.jpg)
/
/etc /containers
/containers/cont1/rootfs
etc shared var
Shared /containers/cont2/roofts
etc shared var
Shared Data: Containers can Share Filesystem Within Host
Containers Today: Share within the host
Containers Future: Share across hosts
Clustered File Systems
CONFIDENTIAL 9
container1 container2
9
![Page 10: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/10.jpg)
Docker Storage Abstractions for Containers
10
Docker Container
/
/etc /var
/data
Non-persistent boot environment
Local Disks
“VOLUME /var/lib/mysql”
Persistent Data
![Page 11: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/11.jpg)
Docker Storage Abstractions for Containers
11
Docker Container
/
/etc /var
/data
Non-persistent boot environment
“VOLUME /var/lib/mysql”
Persistent Data
Block Volumes VSAN CEPH RBD ScaleIO Flocker Etc,…
![Page 12: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/12.jpg)
Container Storage Use Cases
12
Unshared Volumes Shared Volumes Persist to External Storage
Use Case: Running container-based SQL or noSQL DB
Use Case: Sharing a set of tools or content across app instances
Use Case: Object store for retention/archival, DBaaS for config/transactions
Host
C C
Storage Platform
Host
C C
Host
C C
Host
C C
Storage Platform
Host
C C
Host
C C
Cloud Storage
API API
![Page 13: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/13.jpg)
Hadoop batch analysis
Eliminate the Silos: Converged Big Data Platform
Distributed Resource Manager or Virtualization platform
Host Host Host Host Host Host
HBase real-time queries
NoSQL Cassandra, Mongo, etc Big SQL
Impala, Pivotal HawQ Compute
layer
Distributed Storage Platform
Host
Other Spark, Shark, Solr,
Platfora, Etc,…
File System (HDFS, MAPR, GPFS) Block Storage POSIX
13
![Page 14: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/14.jpg)
Storage Platforms
14
![Page 15: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/15.jpg)
15
One Thousand IOPS
10’s of Terabytes
10’s of Petabytes
Array Technologies 10 Million IOPS
Consolidate
and
Increase Performance
![Page 16: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/16.jpg)
Storage Media Technology: Capacities & Latencies – 2016
16
Magnetic Storage
NVMe SSD
DRAM
~500ns
~10ms
~1s
NVM
~100ns
Capacity Oriented SSD
Object Storage
~1ms
~10us
Trip to Australia in 12 hours
Trip to Australia in 1 hour
Trip to Australia in 36 seconds
Trip to Australia in 1.8 seconds
Trip to Australia in .3 seconds
Trip to Australia in 50 days
1TB
1TB
4TB
16TB
32TB
4TB
4TB
48TB
192 TB
384TB
Per Host
![Page 17: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/17.jpg)
17
Storage Media Technology 1 Million IOPS
One Thousand
IOPS
10’s of Gigabytes
10’s of Terabytes
3D NAND “Capacity Flash”
2D NAND “IOPS Flash”
![Page 18: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/18.jpg)
NVDIMM’s
18
Magnetic Storage
NVMe SSD
DRAM
~500ns
~10ms
~1s
NVM
~100ns
Capacity Oriented SSD
Object Storage
~1ms
~10us
1TB
1TB
4TB
16TB
32TB
Std DRAM Event Xfer NAND DDR3 / DDR4
NVDIMM-N[2] (Type 1)
NAND RAM Buffer Async Block Xfer DDR3 / DDR4
NVDIMM-F[1] (Type 3)
NVM Sync Xfer DDR4
Type 4[1]
DRAM + NAND: Non-volatile Memory Load/Store
DRAM + NAND: Non-volatile Memory Block Mode
Next Generation Load/Store Memory
![Page 19: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/19.jpg)
Intel 3D XPoint™ Technology
19
• Non-Volatile
• 1000x higher write endurance than NAND
• 1000x Faster than NAND
• 10x Denser than DRAM
• Can serve as system memory and storage
• Cross Point Structure: allows individual memory cell to be addressed
• Cacheline load/store should be possible.
• Stackable to boost density
• Memory cell can be read or written w/o requiring a transistor, reducing power & cost 1. “Intel and Micron Produce Breakthrough Memory Technology”, Intel
![Page 20: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/20.jpg)
1 Million IOPS
One Thousand
IOPS
10’s of Terabytes
10’s of Petabytes
Storage Workload Map: SDS Solutions
22 HDFS
VSAN
![Page 21: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/21.jpg)
What’s Really Behind a Storage Array?
23
OS
Storage Stack
Data Services
OS
Storage Stack
Data Services
Dual Failover Processors
Clustered OS
iSCSI, FC, etc
Shared Disk SAS or FC
Fail-over Services
De-dup Volumes Mirroring Caching Etc,…
Industry std X86 Servers
SAS or FC Network
![Page 22: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/22.jpg)
Different Types of SDS (1):
24
Fail-Over Software on Commodity Servers
HP Dell Intel
Super-Micro etc,…
Nextenta Data On-tap Virtual
Appliance EMC VNX VA
etc,…
OS
Storage Stack
Data Services
OS
Storage Stack
Data Services
Your Servers
Software-Defined Storage
![Page 23: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/23.jpg)
(1) Better: Software Replication Using Servers + Local Disks
25
OS
Data Services
OS
Storage Stack
Data Services
Software Replication
Clustered Storage Stack
Simpler Server
configuration
Lower cost
Less to go wrong
Not very Scalable
![Page 24: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/24.jpg)
(2) Caching Hot Core/Cold Edge
26
Scale-out Storage Service
Hot Tier Caching of Reads/Writes
Host Host Host Host Host Host Host
Compute Layer – All Flash
Host
Low Latency Network
iSCSI S3
Scale-out Storage Tier
![Page 25: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/25.jpg)
(3) Scale-Out SDS
27
OS
Storage Stack
OS OS OS
Storage Network Protocol
iSCSI or Proprietary
Compute Scalable
More Fault Tolerance
Rolling
Upgrades
More Management
(Separate compute and storage silos)
Data Services
Data Services
Data Services
Data Services
![Page 26: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/26.jpg)
(4) Hyperconverged SDS
28
Hypervisor
Storage Stack
Hypervisor Hypervisor Hypervisor
Easier Management:
(Top-Down Provisioning)
Scalable
More Fault Tolerance
Rolling
Upgrades
Fixed Compute and Storage Ratio
Data Services
App App App App App
VSAN
![Page 27: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/27.jpg)
Storage Interconnects
29
Compute Tier
HCA Hardware Transports
Fibre Channel Ethernet Infiniband
SAS
Protocols iSCSI
FC FCoE NVMe
NVMe over Fabrics
Device Connectivity SATA SAS
NVMe
![Page 28: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/28.jpg)
30
![Page 29: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/29.jpg)
Network • Lossy
• Dominant Network Protocol
• Enables converged storage network: – iSCSI – iSER – FCoE – RDMA over Ethernet – NVMe Fabrics
31
![Page 30: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/30.jpg)
Device Interconnects
Server
PCI-e Bus
HCA
SATA/SAS HDD/SSDs
Server
PCI-e Bus
NVMe SSD
Intel P3700 @3Gbytes/sec
SATA SAS
Server
PCI-e Bus
Electrical Adapter NVM
Over PCIe
Flash: - Intel P3700 @3GB/s - Samsung XS1715
NVM - Future devices
SAS 6-12G @ 1.2Gbytes/sec
4 lanes gen3 x 4 = 4Gbytes/sec HCA
HCA
PCIe Serial Speed Bytes/sec Gen 1 2gbits per lane 256MB/sec
Gen 2 4gbits per lane 512MB/sec
Gen 3 8gbits per lane 1GB/sec
*There are now PCIe extensions for Storage
32
![Page 31: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/31.jpg)
Storage Fabrics
33
Server PCI-e Bus
NIC iSCSI over
TCP/IP
Server PCI-e Bus
NIC FC over
Ethernet
Server PCI-e Bus
NIC NVMe over
Ethernet
iSCSI
FCoE
NVMe over Fabrics (the future!)
Pros: • Popular: Windows, Linux, Hypervisor friendly • Runs over many networks: routeable. TCP • Optional RDMA acceleration (iSER)
Cons: • Limited protocol (SCSI) • Performance • Setup, naming, endpoint management
Pros: • Runs on ethernet • Can easily gateway between FC and FCoE
Cons: • Not routable • Needs special switches (lossless ethernet) • Not popular • No RDMA option?
Pros: • Runs on ethernet • Same rich protocol as NVMe • In-band administraton & data • Leverages RDMA for low latency
Cons:
![Page 32: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/32.jpg)
Storage Fabrics (PCIe Rack Level)
34
34
NVMe Over PCIe
4Gbytes/sec 4Gbytes/sec 4Gbytes/sec
PCIe Shared Storage RAID
Namespace QoS
Autotiering
PCIe Switch
PCIe Rack Fabric (Shared Drives between Hosts)
![Page 33: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/33.jpg)
Integrated PCIe Fabric
35
![Page 34: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/34.jpg)
PCI-e Rack-Scale Compute+Storage
36
<- Storage-DMA, RDMA over PCIe ->
PCIe Fabric
Storage Box Storage Box
• PCIe Fabric: • Share disk boxes
across hosts • Storage-DMA:
• Host-Host RDMA capability
• Leverages the same PCI-e Fabric
Compute Compute Compute Compute
![Page 35: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/35.jpg)
NVMe – The New Kid on the Block
37
NVMe Core
Namespaces
NVMe Objects
PCIe RDMA
IB Ethernet FC
Rea
d
Writ
e
Cre
ate
Del
ete SCSI-like: Block R/W Transport
NFS-like: Objects/Namespaces vVol-like: Object + Metadata + Policy
NVMe over Fabrics
![Page 36: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/36.jpg)
Now/Future
38
Interconnect Now Next Future (Richard’s Opinion)
SATA 6g - NVMe
SAS 12g 24g NVMe
PCI-e Proprietary 4 x 8g NVMe
SCSI Over FC/Virtual NVMe
FC 8g 32g NVMe over Fabrics
Ethernet 10g 20/40g 100g
![Page 37: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/37.jpg)
SDS Hardware Futures: Conclusions & Challenges • Magnetic Storage is phasing out
– Better IOPS per $$$ with Flash since this year – Better Space per $$$ after 2016 – Devices approaching 1M IOPS and extremely low latencies
• Device Latencies push for new interconnects and networks – Storage (first cache) locality is important again – Durability demands low-latency commits across the wire (think RDMA)
• The network is critical – Flat 40G networks design point for SDS
39
![Page 38: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/38.jpg)
Beyond Block: Software-Defined Data Services Platforms
40
![Page 39: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/39.jpg)
Storage Protocols
41 HDFS
S3/Swift
POSIX
HDFS
SQL Flavors
SCSI
NoSQL K-V
![Page 40: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/40.jpg)
Too Many Data Silos
42
Compute Cloud
Block
Object
K-V
M
anagement
DB
Big D
ata M
anagement
Managem
ent
Managem
ent
Managem
ent
HCL HCL HCL HCL HCL
API
API
API
API
API
Different HCLs
Different SW Stacks
Walled Resources
Lots of Management
Developers Are Happy
![Page 41: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/41.jpg)
All-in-one Software-defined Storage
Option 1: Multi-Protocol Stack
43
Compute Cloud
Block
Object
K-V
DB
Big D
ata
Managem
ent
HCL
API
API
API
API
API
Developers Are Happy
One Stack to
Manage
Shared Resources
Least Capable Services
![Page 42: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/42.jpg)
Software Defined Storage Platform
Option 2: Common Platform + Ecosystem of Services
44
Block
CE
PH
MongoD
B
MyS
QL
Hadoop
Managem
ent
HCL
API
API
API
API
API One Platform
To Manage
Shared Resources
Richest Services
Compute Cloud
Developers Are Happy
![Page 43: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/43.jpg)
Opportunities for SDS Platform • Provide for needs of Cloud Native Data Services
– E.g. MongoDB, Cassandra, Hadoop HDFS
• Advanced Primitives in support of Distributed Data Services – Policies for storage services: replication, raid, error handling – Fault-domain aware placement/control – Fine grained policies for performance: e.g. ultra-low latencies for database logs – Placement control for performance: locality, topology – Group snapshot, replication semantics
45
![Page 44: The Future of Software-Defined Storage – What Does It Look Like in](https://reader037.vdocuments.net/reader037/viewer/2022110221/58a1aa6c1a28ab8e608bb4f3/html5/thumbnails/44.jpg)
46