tech preview: rdma and for publication next-gen …#vmworld tech preview: rdma and next-gen storage...
TRANSCRIPT
-
#vmworld
Tech Preview: RDMA and Next-Gen Storage
Technologies for vSANBiswapati Bhattacharjee, VMware, Inc.Srinath Premachandran, VMware, Inc.
HCI2476BU
#HCI2476BU
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
Disclaimer
2©2018 VMware, Inc.
This presentation may contain product features orfunctionality that are currently under development.
This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
Technical feasibility and market demand will affect final delivery.
Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
Agenda
3©2018 VMware, Inc.
vSAN Adoption and Overview
Server Side Technology Evolution
vSAN Vision on RDMA
vSAN Vision on Next-Gen Storage Technology
Q & A
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
4©2018 VMware, Inc.
vSAN Adoptionand Overview
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
5©2018 VMware, Inc.
HCI is the Fastest Growing Storage SegmentHCI is replacing traditional storage in the enterprise
Source: Wikibon Server SAN Research Project, 2016 Source: IDC Worldwide Quarterly Enterprise Storage Systems Tracker
Traditional Storage Systems Shipped
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
vSAN Is Adding 100 Customers Per Week
vSAN Customer Adoption
>15,000 CustomersMore than top 3 competitors combined
>70% YoY bookings growthVMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
7©2018 VMware, Inc.
Data Criticality
Number of Workloads
Business Critical
Applications
There Is a Clear HCI Adoption Curve
ROBO Management
ClustersVDI
Test/Dev
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
8©2018 VMware, Inc.
Brief Overview of VMware vSAN
Runs on any standardx86 server
Pools SSDs/HDDs intoa shared datastore
Delivers enterprise-grade security, scale and performance
Managed throughper-VM storage policies
Deeply integrated withthe VMware stack
vSAN Storage
Managed by vCenter
vSphere vSAN
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
9©2018 VMware, Inc.
Server SideTechnology Evolution
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
Hardware Segmentation of vSAN ReadyNode Platforms
Emerging MarketExisting Market
StorageDense
Compute Intense
Composable Infrastructure
Edge / IoT
Archiving, Video streaming, Analytics Web apps, VDI
Data warehouse, Log aggregation ROBO
Rack Servers Blade Servers Composable Infrastructure Edge Computing
HPE – ApolloCisco S-series
HPE BL-SeriesCisco M4, M5 – BL series, Dell FX2 ….
HPE Synergy (Independent
Compute &Storage)…
Cisco – E seriesDell – R640 …
General Purpose
Sample Use cases BCA, Database, VDI, DR
HardwareType Rack Servers
Supported HardwareExamples
Dell – R740, 640HPE – DL 360/380Cisco – C220/240
Lenovo ThinkSystems,Supermicro, Fujitsu ……VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
11©2018 VMware, Inc.
How Has Compute Evolved in x86 Servers?
Yesterday Today
Key Trends
• CPU core density is increasing significantly – 12 cores popular today, 24 cores in next 2-3 years
• CPU speeds are becoming faster• FPGA and CPU offloads for higher efficiency• Beefy single socket servers are gaining traction
Dual Socket(12 – 16 Cores)
Dual Socket(8 Cores)
Pro
cess
ing
Po
wer Dual Socket
(24 – 32 Cores)Single Socket
(32 – 64 cores)
FPGA
Intel Cascade Lake(PMEM Enablement)
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
12©2018 VMware, Inc.
How Has Storage Evolved in x86 Servers?
Yesterday Today
Memory&
Storage
Storage
NAND based Intel NVMe
(250K IOPS)
Intel Optane NVMe SSD(500K IOPS)
DDR4 based Persistent Storage Class Memory(Million IOPS,
Nanosecond latency)
Key Trends• Flash adoption is
transitioning from SAS /SATA to NVMe
• 500K-750K IOPS is expected to be the norm for NVMe devices in next 2 yrs
(50K-80K IOPS)
Key Trends• New Persistent
memory (storage class memory) started appearing in near future
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
13©2018 VMware, Inc.
Today’s Flash Device Landscape
Flash Devices Endurance (DWPD)
ExpectedPerformance
(IOPS) Latency (µs)Capacity
Range (TB)
Persistent Memory In Thousands In Millions In Nanoseconds 0.64 – 0.512
High End NVMe(Intel Optane, P48xx) 30 500K+ 10 0.4 – 0.6
Performance NVMe(Intel Cliffdale, P46xx) 3-5 ~350K 40 1.2 – 1.4
Mainstream SAS(SAS SSD) 2-3 ~250K 60 1.6 – 4.0
Mainstream NVMe (Intel Cliffdale, P45xx) 1-2 ~200K 70 1.6 – 2.0
SATA/Essentials NVMe(Intel Cliffdale, P45x1)
-
14©2018 VMware, Inc.
How Has Network Evolved in x86 Servers?
Yesterday Today
10G/25G/40G Network
1G/10G Network
100G/200G Network
Key Trends• 10G/25G/40G is mainstream now
• RDMA enabled (RoCE) NIC is becoming standardized
• 100G projected to be next generation NIC in next 3-5 years
• NVMe-oF technology opens up new storage deployment models choice for HCI
RDMA Enabled
NIC
Net
wo
rk S
pee
d
NVMe-oF
FPGA
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
15©2018 VMware, Inc.
Server Side Technology Evolution & Economics Is Driving HCI
Scale Up Architecture(Traditional Storage Arrays)
Fiber Channel
Scal
e-U
p
SAS SAS SAS SAS
SASExpress
SASExpress
SASExpress
SASExpress
Ethernet
Fibre Channel
Scale Out Architecture(Modern Storage)
Ethernet
Scale-Out
HCI brings in compute, storage and network together with single management interface
Server Side Economics for HCI:1. Falling price of Flash2. Faster & Powerful CPUs &
flash (NVMe) in x86 server3. Unified Management VMw
orld 201
8 Conten
t: Not for
publicat
ion or dis
tribution
-
16©2018 VMware, Inc.
This Evolution Enables New Apps and Capabilities for HCI
In MemoryDatabase Big Data
StreamingApplications
Heavy Sequential Write Application
Real-time Processing
Application Require Faster
Reboot, HA
High Performance Computing
Others…..
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
17©2018 VMware, Inc.
vSAN RDMA Visionand Demo
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
18©2018 VMware, Inc.
What Is RDMA?
In computing, remote direct memory access (RDMA) is a direct memory access from the memory of one computer into that of another without involving either one's operating system. This permits high-throughput, low-latency networking, which is especially useful in massively parallel computer clusters.
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
19©2018 VMware, Inc.
Above bullets points are different scenarios where RDMA potentially could benefit. However experiments need to be done in each area before anything planned. There
is no commitment on any of the above items and timeline yet to be determined
vSAN Vision on RDMA
Potential RDMA Benefits:
Improved application performance
Better VM Consolidation
Speeding up cloning & vMotion operations
Faster metadata updates across vSAN hosts on the network
Further improve resync/rebuild/rebalance time
NVMe-oF technology enablement
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
20©2018 VMware, Inc.
These experiments are not final nor been executed on release build. Early experiments to explore how beneficial RDMA could potentially be in vSAN
Few Early Tech Preview Experiments of vSAN over RDMA
Mellanox ConnectX 5 100 Gbps (MCX16A-CCAT)
Mellanox Switch SN2700
2 Disk Groups in Each Node• All NVMe (Cache + Capacity)• 1 Cache + 1 Capacity
2 * 14 core Intel® Xeon® E5-2697 @ 2.60 GHz
512 GB Host Memory
Default VM Storage Policy (FTT=1)
NO Additional vSAN Data Services Turned ON
HCI Bench Tool
Early Exploratory Work with Partner Mellanox®
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
21©2018 VMware, Inc.
Typical Enterprise Workload IO Profile
Typical Workload Block Size Sample Applications
70/30 RW (100% Random) 4K - 16KDatabase applications running• OLTP - SQL/Oracle• Exchange
100% Write (Sequential) 64K - 256K
• Transactional Logs• Big Data• Analytics• Streaming Media• Backup VDI Admin Operations (Clone,
Compose etc.)
100% Write (Random) 32K - 64K • Web Application
100% Read (Sequential) 64K - 256K • Restore from backup/archive
30/70 RW (100% Random) 4K - 16K • VDI Steady State• VDI Boot Storm
70/30 RW (100% Random) 24K-32K • SharePoint
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
22©2018 VMware, Inc.
Exploratory Testing Throughput Numbers for RDMA and TCP/IP over vSAN
These experiments are not final nor been executed on release build. Early experiments to explore how beneficial RDMA could potentially be in vSAN
No. of Threads
RDMA TCP/IP
6 1545.42 1278.19
6 2995.17 2533.35
6 5287.53 4614.59
6 11331.56 9706.61
6 10813.35 9422.67
6 12716.34 10715.93
1 9099.93 8558.96
70/30 Read/Write ProfileThroughput
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
23©2018 VMware, Inc.
Exploratory Testing IOPS Numbers for RDMA and TCP/IP over vSAN
These experiments are not final nor been executed on release build. Early experiments to explore how beneficial RDMA could potentially be in vSAN
No. of Threads
RDMA TCP/IP
6 395630.9 327213.3
6 383381.5 324271.8
6 338403.7 295334.1
6 181305 155307.3
6 86506.7 75381.2
6 50865.4 42863.8
1 18199.8 17118
70/30 Read/Write Profile
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
24©2018 VMware, Inc.
Exploratory Testing Latency Numbers for RDMA and TCP/IP over vSAN
These experiments are not final nor been executed on release build. Early experiments to explore how beneficial RDMA could potentially be in vSAN
No. of Threads
RDMA TCP/IP
6 2.4 2.9
6 2.4 2.9
6 2.8 3.2
6 5.6 5.6
6 11.5 12.9
6 18.9 22.3
1 8.7 9.2
70/30 Read/Write Profile
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
25©2018 VMware, Inc.
RDMA shows higher throughput, IOPS and lower latency compared to TCP/IP
Across IO profile and sizes, RDMA potentially doing more than TCP/IP
With more IOPS, Throughput; potentially application performance and VM consolidation can be increased
Takeaways of RDMA over vSAN Exploration
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
26©2018 VMware, Inc.
vSAN Vision on Next-Gen Storage Technology
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
27©2018 VMware, Inc.
vSAN Architectural Overview
Source: https://download3.vmware.com/vcat/vmw-vcloud-architecture-toolkit-spv1-webworks/index.html#page/Storage%20and%20Availability/Architecting%20VMware%20vSAN%206.2/Architecting%20Virtual%20SAN%206.2.2.008.html
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
https://download3.vmware.com/vcat/vmw-vcloud-architecture-toolkit-spv1-webworks/index.html#page/Storage%20and%20Availability/Architecting%20VMware%20vSAN%206.2/Architecting%20Virtual%20SAN%206.2.2.008.html
-
28©2018 VMware, Inc.
PMEM/ NVDIMM Overview
Current trend on memory and storage technology convergence creates new generation of devices
Persistent Memory (PMEM/NVDIMM) brings in storage to memory slots
These devices are persistent like storage and at the same time fast like memory. Provides load/store memory semantics
Has memory like performance
Block or byte-addressable. Extremely low latency and high endurance
Projected capacity ranges from 256G to Terabytes
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
29©2018 VMware, Inc.
Next-Gen Storage Devices Vision for vSAN
Overview*• Application performance needs are driving
evolution of fast storage such as NVMe & PMEM at a decreasing price point
• As device capacities are getting exponentially larger, might consider contributing capacity from the caching device
• Even further, a single tier design could be beneficial for efficient use of emerging devices like PMEMs for performance-orientate workloads
Potential Benefits• As PMEM capacity and price points roll out, it
could be a play for just metadata or mix of metadata and capacity or just capacity as well
• To leverage ops/performance of these devices having knobs to control space efficiency at a VM granularity could be offered
CONCEPT ONLY
*This diagram is for conceptual depiction only. There is no commitment or timeline on whether vSAN would evolve in this direction from its current
implementation
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
30©2018 VMware, Inc.
Concept Demo: NVMe-oF for VMware vSAN
Compute Only ESXi NodesJBOF Resource Pool - A
JBOF Resource Pool - BCompute Only ESXi NodesNVMe-oF
NVMe-oF potential benefits for vSAN
• Scale storage and compute independently
• Retain the simplicity of HCI management for provisioning and disaggregation workflows
Storage Node – Intel®Storage System
RAF1000JSPCompute Node – Standard x86 Server Hardware
RDMA Over Converged Ethernet (RoCE)
Please visit VMware Advance Technology booth at VMVillage to know more about this demo VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
31©2018 VMware, Inc.
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
32©2018 VMware, Inc.
Server side technology evolution & economics is driving HCI
Application performance needs are driving evolution of fast storage and network such as NVMe & PMEM, RDMA at a decreasing price point.
vSAN is committed to explore new technologies (PMEM, RDMA, NVMe, NVMe-oF and others) and adopt as they deem fit in HCI
Key Takeaways
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
33©2018 VMware, Inc.
Visit the HCI Zone in the Solutions Exchange
#vSAN #HCIZone#vSANfan
Experience Solutions Powered by vSANEdge
Video Analytics
CloudPKS with containerized MySQL
CoreSAP HANA on vSAN
DataStax
Espresso Bar vSpeaking Podcast Daily Prizes
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
34©2018 VMware, Inc.
Get Ahead of the Curve – vSAN Private Beta
vSAN Data Protection
Native enterprise-grade protection
vSAN File Services
Expanding vSAN beyond block storage
Cloud Native Storage
Persistent storage for containers
Sign up at http://www.vmware.com/go/vsan-beta
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
PLEASE FILL OUTYOUR SURVEY.Take a survey and enter a drawingfor a VMware company store gift card.
#vmworld #HCI2476BU
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
-
THANK YOU!
#vmworld #HCI2476BU
VMworld
2018 Co
ntent: No
t for pub
lication o
r distribu
tion
Tech Preview: RDMA and Next-Gen Storage Technologies for vSANSlide Number 2Slide Number 3vSAN Adoption�and OverviewHCI is the Fastest Growing Storage SegmentvSAN Is Adding 100 Customers Per WeekThere Is a Clear HCI Adoption CurveBrief Overview of VMware vSANServer Side�Technology EvolutionHardware Segmentation of vSAN ReadyNode PlatformsHow Has Compute Evolved in x86 Servers?How Has Storage Evolved in x86 Servers? Today’s Flash Device LandscapeHow Has Network Evolved in x86 Servers?Server Side Technology Evolution & Economics Is Driving HCIThis Evolution Enables New Apps and Capabilities for HCIvSAN RDMA Vision�and DemoWhat Is RDMA?vSAN Vision on RDMAFew Early Tech Preview Experiments of vSAN over RDMATypical Enterprise Workload IO ProfileExploratory Testing Throughput Numbers for RDMA and TCP/IP over vSANExploratory Testing IOPS Numbers for RDMA and TCP/IP over vSANExploratory Testing Latency Numbers for RDMA and TCP/IP over vSANTakeaways of RDMA over vSAN ExplorationvSAN Vision on Next-Gen Storage TechnologyvSAN Architectural OverviewPMEM/ NVDIMM OverviewNext-Gen Storage Devices Vision for vSANConcept Demo: NVMe-oF for VMware vSANSlide Number 31Key TakeawaysVisit the HCI Zone in the Solutions ExchangeGet Ahead of the Curve – vSAN Private BetaSlide Number 35Slide Number 36