g2m research multi-vendor webinar nvme/tcp™- the eventual...

RESEARCH

G2M Research Multi-Vendor Webinar NVMe/TCP™- The Eventual NVMe over Fabric Protocol Winner? February 26, 2019

2 © 2019 G2M Communications, Inc.. All rights reserved.RESEARCH

Webinar Agenda

9:00-9:05 Ground Rules and Webinar Topic Introduction(G2M Research)

9:06-9:25 Sponsoring Vendor presentations on topic (5 minute each) 9:26-9:39 Key Question 1 (2-minute question; 3 minutes response per

vendor) 9:40-9:41 Audience Survey 1 (2 minutes) 9:42-9:55 Key Question 2 (2-minute question; 3 minutes response per

vendor) 9:56-9:57 Audience Survey 2 (2 minutes) 9:58-10:11 Key Question 3 (2-minute question; 3 minutes response per

vendor)10:12-10:13 Audience Survey 3 (2 minutes) 10:14-10:18 Audience Q&A (8 minutes) 10:19-10:20 Wrap-Up

RESEARCH

G2M Research Introductionand Ground RulesMike HeumannManaging Partner, G2M Research


Tom SpencerSr. Director, ProductMarketingSolarflarewww.solarflare.com

Panelists

Host/Emcee: Mike HeumannManaging PartnerG2M Researchwww.g2minc.com

Rob DavisVP, StorageTechnologiesMellanoxwww.mellanox.com

Eliot RosenProduct MarketingDirectorBroadcomwww.broadcom.com

Nishant LodhaSr. ProductMarketing ManagerMarvellwww.marvell.com

http://www.solarflare.com/

http://www.g2minc.com/

http://www.mellanox.com/

http://www.broadcom.com/



What is NVMe® over TCP (NVMe/TCP™)?

NVMe/TCP is a new binding for NVMe over Fabrics (NVMe-oF™)that was recently approved by NVMe.org

NVMe-oF commands are sent over standard TCP/IP sockets Each NVMe queue pair is mapped to a TCP connection TCP provides a reliable transport layer for the

NVMe queueing model NVMe/TCP provides an alternative transport to

NVMe-oRoCE and NVMe-oiWARP for Ethernet,and to NVMe-oFC (Fibre Channel) andNVMe-oIB (InfiniBand)


NVMe/TCP Characteristics

Advantages NVMe/TCP can run on existing TCP/IP Networks NVMe/TCP is compatible with existing Ethernet

management apps

Disadvantages

NVMe/TCP has higher latency than NVMe-oRoCE

Like NVMe-oRoCE and NVMe-IB transports, NVMe/TCP will (typically) run on converged SAN/ LAN networks (not a dedicated SAN)


Questions We Will Explore In This Webinar

What are the use cases for NVMe/TCP, and what strengths does NVMe/TCP bring to these use cases vis-à-vis other NVMe-oF protocols?

How important are NVMe/TCP’s advantages and disadvantages for its use cases in real-world deployment scenarios?

To what extent will NVMe/TCP replace other Ethernet-based NVMe-oF protocols, and how long will it take to do so?

RESEARCH

MellanoxRob DavisVice President,Storage Technologieswww.mellanox.com

http://www.mellanox.com/

9© 2018 Mellanox Technologies | Confidential

Mellanox NVMe over TCP Support

Mellanox supports NVMe over TCP across our Ethernet products including ConnectX NICs and BlueField SOCs

NVMe/TCP and NVMe/RoCE are complimentary solutions, our products can support both simultaneously

NVMe/RoCE and NVMe/IB have lower and more consistent latencyNVMe/RoCE and NVMe/IB use less CPU cycles

TCP is less optimized for performance and efficiency, but we fully support it and see important use cases for it


NVMe/TCP for Stranded Server Connectivity

Enables NVMe-oF connectivity for legacy systems Example given in article titled “Pavilion compares RoCE and TCP

NVMe over Fabrics performance” https://www.theregister.co.uk/2018/08/16/pavilion_fabrics_performance/ Pavilion provided their customer a mixed solution of servers with

non-RDMA NICs using NVMe/TCP and RDMA NICs using NVMe/RoCE The Pavilion AFA supports both RoCE and TCP NVMe-oF on Mellanox

ConnectX NICs Latency test results showed NVMe/TCP (~200µs) at twice that of

RoCE (~100µs) but half that of NVMe-backed iSCSI (~400µs) This includes AFA controller and NVMe SSD latency

https://www.theregister.co.uk/2018/08/16/pavilion_fabrics_performance/


NVMe/TCP for Distance Connectivity

Often networked storage applications need to work over long distancesTCP’s well proven distance

capabilities make NVMe/TCP perfect for these applications We successfully tested this

between the Mellanox Israel and Sunnyvale offices


NVMe/TCP for Control Path

Many data centers have an independent device management network (control path)Usually much lower speedNVMe/TCP enables support for

BMCs with simple IP stacksNVMe-MIAllows for discover controllers on

the control path independent of the data

1Gb100Gb


Performance: NVMe/RoCE vs. NVMe/TCP

Loca

l SSD

Writ

e

RD

MA

Writ

e

TCP

Writ

e

Frac

tion

of IO

s w

ith th

is o

r les

s la

tenc

y

14© 2018 Mellanox Technologies

Importance of Tail Latency in Today’s Datacenters

Most datacenters today need to support interactive real-time requests Online searches generate a small amount of network traffic between the requestor and datacenter, but

the response generates a massive amount of traffic within the datacenter Much of this traffic is related to the core advertising business model of the datacenter’s owner The distributive software architecture that drives these businesses is very susceptible to tail latency

https://www.nextplatform.com/2018/03/27/in-modern-datacenters-the-latency-tail-wags-the-network-dog/

https://www.nextplatform.com/2018/03/27/in-modern-datacenters-the-latency-tail-wags-the-network-dog/

RESEARCH

SolarflareTom SpencerSr. Director, Product Marketingwww.solarflare.com

http://www.solarflare.com/

© 2019 Solarflare Communications, Inc. Confidential & Proprietary 16

OptaneLatency~10us

0

5

10

15

20

25

30

35

40

45

512 4096

Late

ncy

us

Block Size

LinuxKernelNVMe

SPDK

SPDK+

Onload

LinuxKernel

SPDK

SPDK+

Onload

44%Lower

35%Lower

54%Lower

28%Lower

1. Performance

© 2019 Solarflare Communications, Inc. Confidential & Proprietary 17

98% of installed servers connected with TCP

2% of installed servers connected with RDMA

2. Ease of Deployment

© 2019 Solarflare Communications, Inc. Confidential & Proprietary

3. Runs on Legacy Networks


4. Ubiquity


5. TCP Has Addressed Congestion

BeforeAfter


5 Reasons Why to use NVMe-oF TCP

1. Performance

2. Ease of deployment

3. Runs on legacy networks

4. Ubiquity

5. TCP has addressed congestion

RESEARCH

BroadcomEliot RosenDirector of Product Marketingwww.broadcom.com



NVMe/TCP™

Ubiquitous in every network

Use existing TCP SW OS infrastructure

Easy to implement– Runs over existing lossy networks– Scalability without concern of congestion management

Can use existing networking configurations

Benefits vs RoCE Disadvantages vs RoCE

Longer latencies – Can be mitigated by limiting tail latency added by network

• Keeping low and consistent with SW algorithms

Performance is CPU bound in kernel mode– Need more CPU cores to achieve similar performance – User space performance is much better


Broadcom solutions

- Common architecture- SmartNIC and Target- NVMe/TCP™ and RoCE

- Offload CPU intensive tasks- RAID/EC, Dedup, Security

- Low power HHHL adapters - Run TCP and RoCE

simultaneously

https://scalegrid.io/blog/comparing-in-memory-databases-redis-vs-mongodb-percona-memory-engine/

Stingray Benefits


Stingray™ - Driving Next Generation NVMeoF Deployments

Applications

HW Accel

100G PCIe BW

De-Dup + RAID

100G RDMA NIC

8x A72 @ 3GHz

Stingray™

Cloud StorageSmartNIC NVMeJBOFs

• 100G NVMeoF target performance

• SDS enablement - Storage services on SmartNIC

• Open programming model on A72’s

• HW acceleration offload for RAID and Security

• Supports both RoCEv2 and TCP

• 16nm for low power (<35W)

• Production adapters available (100G and 50G)

26 © 2019 G2M Communications, Inc.. All rights reserved.

NetXtreme® -E and -S Series NVMe-oF Solutions

Solves storage scaling problems with end-to-end solutions Largest OEMs and cloud customers standardize with Broadcom

DeployableTODAY

Thor™ Stingray™Stratus

Whitney+

50G100G

200G SmartNIC

RESEARCH

MarvellNishant LodhaSr. Product Marketing Managerwww.marvell.com

http://www.marvell.com/

M a r v e l l C o n f i d e n t i a l28

1 2

3

4

5

Universal RDMA + ToE

Ultimate in choice and investment protection with concurrent support for RoCE, RoCEv2 and iWARPand full offload for NVMe/TCP

FC-NVMe

Industry’s most accepted storage transport, now enhanced for NVMe.

Broad Spectrum of Ethernet and Fibre Channel

Connectivity16G and 32GFC , 10/25/50/100GbE

to host the most demanding Storage workloads and deliver

scalability

Seamless Storage Migration

Concurrent offload for NVMe-oF, iSER, iSCSI and FCoE for

Ethernet and FCP and FC-NVMe for Fibre Channel

enables seamless upgrade paths to next gen storage

End to End SolutionsInitiator and Target mode solutions, including SPDK that leverage years of storage and storage networking experience

NVMe-oF

Marvell NVMe-oF Solution


Challenges remain with NVMe-oF™ RDMA!

RNIC Upgrade Required

RDMA Camps

Creates IslandsBackward Compatibility

Forklift Infrastructure and Skillset change required!

Not Automatic

Not Precise

Not for everyone

Congestion

Keeping the network ‘lossless’

RDMA/OFEDexpertise

Skillset Requirements


Marvell NVMe/TCP at Flash Memory Summit 2018


NVMe/TCP - FastPath

App/FS/MQStart of I/O

NVMe-oF

NVMe Core

NVMe/TCP

TCP/IP Stack

NIC

SQ CQNVMe Layer

In-Capsule DataNVMe-oFCapsule

PDU PayloadPDU Header

TCP PayloadTCP/IP Header TCP PayloadTCP/IP

Header

Eth PayloadETH Header Eth PayloadETH

Header Eth PayloadETH Header

Map each QP to TCP connectionConstruct PDU Header, Digest

Segmentation, Header Processing

Protocol and Header Processing

FastLinQ®NVMe/TCP Offload

Engine

Marvell Offloads and Accelerates NVMe/TCP

M a r v e l l C o n f i d e n t i a l

NVMe-oF Performance Comparisons

32

Read Avg Read Tail Write Avg Write Tail

Latency Performance1 Thread :: Queued :: 8KB

NVMe-oF RoCE NVMe-oF SW TCP NVMe-oF HW TCP RoCEv2 SW TCP HW TCP - Current

Million IOPS PerformanceMulti Thread:: 8KB READ

* Initial metrics based on Marvell internal measurements; performance is system dependent. For HW TCP, only target was “offloaded”, initiator was SW driven.


Making the right “fabric” choice!

Not “just” about “fabrics” performance Use CasesCulture and Install Base

RESEARCH

Panel Discussion


Panel Question #1

What are the use cases for NVMe/TCP, and what strengths does NVMe/TCP bring to these use cases vis-à-vis other NVMe-oF protocols?– Mellanox– Solarflare– Broadcom– Marvell


Audience Survey Question #1

What use cases do you believe that NVMe/TCP is best suited for (versus other NVMe-oF protocols)? (check all that apply; 54 responses):

• Databases/database applications: 44%

• Unstructured data/object storage: 52%

• Real-time analytics: 22%

• High-performance computing and modeling: 20%

• DevOps: 24%

• Others: 20%

• I don’t believe that NVMe/TCP is more suited for these use casesthan other NVMe-oF protocols: 11%


Panel Question #2

How important are NVMe/TCP’s advantages and disadvantages for its use cases in real-world deployment scenarios?– Solarflare– Broadcom– Marvell– Mellanox



Which of NVMe/TCP’s advantages/disadvantages are important for your use cases (check one answer for each question; 48 responses):

Question Very Important

Somewhat Important

Not Important

Not Enough

Information to Answer

No Opinion

Advantage: Ability to run on existing Ethernet networks 79% 15% 2% 4% 0%

Advantage: Ability to support multicast 23% 40% 15% 13% 10%

Disadvantage: Higher latency than other NVMe-oF protocols 31% 67% 2% 0% 0%

Disadvantage: Not a dedicated SAN 17% 31% 44% 4% 4%


Panel Question #3

To what extent will NVMe/TCP replace other Ethernet-based NVMe-oF protocols, and how long will it take to do so?– Broadcom– Marvell– Mellanox– Solarflare



To what extent would your organization consider NVMe/TCP as an alternative to other Ethernet NVMe-oF protocols? (select one; 32 responses):

• We will continue to utilize non-NVMe-oF Ethernet as ourprimary SAN for the next 36 months or greater: 22%

• We are already deploying/planning to deploy NVMe-oFCas our primary SAN: 9%

• We are already deploying/planning to deploy NVMe-oRoCEas our primary SAN: 13%

• We will are considering all NVMe-oF options for our storage deployments in the next 36 months: 47%

• NVMe/TCP is the primary protocol we are considering for ourNVMe-oF deployments during the next 36 months: 9%


Audience Q&A

RESEARCH

Thank You For Attending

RESEARCH

g2m research multi-vendor webinar nvme/tcp™- the eventual...

Documents