who says data center storage has to be inefficient? · 2020-04-10 · who says data center storage...
TRANSCRIPT
PRESENTATION TITLE GOES HERE
Who Says Data Center Storage Has to be Inefficient? Larry Chisvin
PLX Technology, Inc.
2 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Agenda
Who is PLX (and why does it matter)? Current usage of PCIe in storage Improving on data center fabrics
How a PCIe fabric works Why it is better
Usage models for PCIe-based storage New data center architectures Development Tools Call to action
3 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Global Enterprise ~155 Employees, Headquartered in USA
Design Center in India
Public Company NASDAQ: PLXT, $100M+ Rev., ~58% GM, Profitable
PLX at a Glance
Market-Leading Customers Widespread use by 1000’s of key accounts
The Market Leader Over 70% Market Share and Growing
Fabless Model TSMC Partner with Designs and Production at 40nm
4 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
All Market Leaders Using PLX PCIe Products
Customers Serving Cloud/Data Center
Switches & Routers Control Plane Standard
Servers First Market that
Developed for Switches
Storage Largest Market
Growth & Attach Rate
5 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Storage in the Cloud/Data Center
I/O Fan-out, Redundancy & Backplane Large Switches with PLX added features used throughout Enterprise Storage Systems. Represents most of the storage revenue today in the pie chart below.
Total Storage Doubling Every Two Years
Standard Bus for Market Leaders
Storage Systems & HBAs/CNAs
Storage Largest Market
Growth & Attach Rate
Networking
PC/ Consumer
Embedded
Server
Storage
6 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Rest of System
PCIe Switch
PCIe Switch
Storage SAS/SATA Controller
CPU
Storage Subsystem
Basic Fanout of Storage Systems
Storage Subsystem
Storage Subsystem
Storage Subsystem
7 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Flash Memory
PCIe SSD
Controller
Flash Memory
Flash Memory
Flash Memory
PCIe SSD
Controller
PCIe SSD
Controller
SSD-Based Aggregation
PCIe Switch
Rest of System
PCIe SSD
Controller
8 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Making the Data Center Better
9 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
The Rack in the Cloud
Host
Storage
Communications
HBA NIC/LOM
HBA NIC/LOM
HBA NIC/LOM
HBA
NIC/LOM
Host
Host
Host
Host
NIC/LOM
10 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Current Solutions
Host
Storage
Communications
HBA NIC/LOM
HBA NIC/LOM
HBA NIC/LOM
HBA
NIC/LOM
Host
Host
Host
Host
NIC/LOM
Multiple fabrics
“Bridging” Devices on each module
Parts Power Cost
Latency Complexity
More
11 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Problem…or Opportunity?
Host
Storage
Communications
HBA NIC/LOM
HBA NIC/LOM
HBA NIC/LOM
HBA
NIC/LOM
Host
Host
Host
Host
NIC/LOM
Since one major problem here is the need for all of those “bridging” devices…
Then what is the answer?
12 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Is There Something Common?
13 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
PCI Express Fabric
ExpressFabric
Redriver
PCI Express
Host
Host Redriver
Redriver
Redriver
Redriver
Redriver
Single Converged
Fabric
Simple, Inexpensive
Redriver
High performance, Low cost,
Low latency Low power Interconnect
Storage
Storage
Comm
Comm
14 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
PCIe-Based Fabric Solution
Storage
Host
Storage
Storage
Storage
PCI Express Fabric
Host
Host
Host
Standard capabilities of fabric provided by PCIe
mCPU
Shared I/O Using existing
Hardware and Drivers
Host-Host Using standard
DMA & RDMA
Infrastructure
Management CPU To configure system and manage events
15 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Mixing Data Types on PCIe
Retimer Host
Retimer
Retimer
Storage
Comm
Flash Flash Flash Flash
Flash Flash Flash Flash
SSD Controller
SSD Controller
Switch
GPU GPU
GPU
GPU GPU Switch
GPU
ExpressFabric Enterprise SSDs are rapidly
standardizing on PCIe
Hybrid systems can be easily implemented
16 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
PCI Express Initiatives
SR-IOV Allows sharing of end points
SRIS (Separate Refclk Independent SSC) Allows each side of link to have its own clock Enables use of general purpose cables
New PCIe Cable spec (OcuLink) Targeted for low cost copper and optical
Signal integrity Better defined repeater & retimer functions
17 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
DPC/eDPC
Downstream Port Containment (DPC) Enables the prevention of host time-out or “blue screen”
When triggered by error from an endpoint Take down link to endpoint Send error message/interrupt to the host Reply to read request within time-out window
Completion synthesis for root ports Host software to handle switch responses Important for storage systems
PCIe Switch
I/O Device
I/O SSD Device
I/O Device
Host CPU
18 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Quantifying the Advantages
Replacing existing Ethernet-based racks with PCI Express can offer tangible savings
PCI Express inside the rack Ethernet to rest of data center
Top-of-rack switch similar cost to PCIe Greater reduction if converged data
Remove NIC/LOM Each slot costs $200-$300 Each slot dissipates 5-7w Rack has 24-64 slots
PCIe-base solution uses simple redriver Each redriver can be $2-$5 Power is less than 1w
Rest of Data Center/Cloud
19 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Cost and Power Advantages of PCIe
$- $2.0 $4.0 $6.0 $8.0
$10.0 $12.0 $14.0 $16.0 $18.0
ExpressFabric 10GE SFP+ 10GE LOM
Cost/Gbps ($) 10GE
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
ExpressFabric 10GE SFP+ 10GE LOM
Power/Gbps (w) 10GE
PRESENTATION TITLE GOES HERE
Storage Usage Models for a PCIe Fabric
21 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
SSD Adapter Card
Two SSD Controllers NVMe Form-Factor – 2.5” Only ASIC1 (controller) exposed to
Host ASIC1 servicing interrupts from ASIC2
Two or more SSD controllers
aggregated with a switch Adapter Card Form-Factor Host isolation capability One ASIC exposed to Host
Services interrupts from other ASICs
ASIC1
Switch
ASIC2
Switch ASIC ASIC ASIC ASIC ASIC
ASIC ASIC
ASIC
22 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Server Motherboards
Capacity expansion in server enclosures Aggregation through PCIe switch
Server Motherboard
RAM
RAM Switch
NIC
CPU
23 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Hybrid Storage System
RAM
CPU
RAM
Switch
NIC
SAS Drives
SAS HBA
MLC
MLC
SLC
Mixing storage mediums for cost/performance Move data between these mediums
Server Motherboard
24 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
IO
Switch PCIe over Cu/Opt
Server
Creating storage expansion box with PCIe Connect to server through PCIe Cu/Op cable No protocol translation overhead
Storage Expansion Boxes
IO Expansion Box
25 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Share an array of SSD modules in a chassis Associate a segment of SSDs to specific server
Servers
PCIe over Cu/Opt
Flash Appliance
IO
Switch
IO Expansion Box
Switch
26 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
IO
Switch
IO Expansion Box
PCIe over Cu/Opt
Servers
Build Server Redundancy
Switch
Server Redundancy
27 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
IO IO Expansion Box
Share SR-IOV SSD modules in expansion chassis Shared by multiple servers/hosts or blade servers
SR-IOV Sharing of SSDs
PCIe over Cu/Opt Switch
Switch
Servers
28 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Can Form Factors Help?
29 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
From Rack to Blade
Rack servers are self-contained systems Traditional architecture in stackable form CPU, memory, I/O, power, cooling, etc. Standard form factors
Blade servers consolidate common subsystems Common power, cooling, external connection Blades offer processing, storage, I/O Systems are vendor-specific
Improvement on rack Allows some savings But non-standard subsystems And doesn’t really address issue completely
29
30 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Next Step - MicroServers
Traditional (brute force) approach to computing limits compute density due to power Open source OS’s and application software allow portability of CPUs and platforms Large scale applications can be effective with smaller compute engines
Web servers Content delivery Analytics
Bottleneck in such applications are memory and I/O bandwidth rather than raw compute power
31 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
What is a MicroServer?
Array of low power, low cost processors Less than 45w Moving to less than 10w
Each processor has modest performance Aggregate performance can be powerful
Can execute applications where processing can be easily distributed CPU, memory, & I/O on compact module ARM-based systems are most common
X86 platforms are expected to be popular
32 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Problems with Current Solutions
Cannot adequately handle compute-intensive applications easily
Many applications cannot distribute processing Cannot easily trade off processing, storage & I/O, since the subsystems are self-contained Most systems are proprietary
Need to buy into entire system Similar to blade servers vs rack servers
Definite advantages, but… Limited ability to mix best-in-class subsystems
33 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
PCIe-Based MicroServer
µServer
Network of SOCs
µServer
Network of SOCs
PCIe Aggregation
PCIe Aggregation
PCI Express Fabric
34 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Powerful Storage Appliance
µServer
Network of SOCs
µServer
Network of SOCs
PCIe Aggregation
PCIe Aggregation
Flash Flash Flash Flash
Flash Flash Flash Flash
Flash Flash Flash Flash
Flash Flash Flash Flash
SSD Controller
SSD Controller
PCIe Aggregation
Flash Flash Flash Flash
Flash Flash Flash Flash
Flash Flash Flash Flash
Flash Flash Flash Flash
SSD Controller
SSD Controller
PCIe Aggregation
PCI Express Fabric
35 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Consolidate Complete System
µServer
Network of SOCs
µServer
Network of SOCs Traditional Server Blade
High Perf CPU
Storage Storage
Storage Storage
Comm Comm
Comm Comm
High Perf Server Blade
PCIe Aggregation
PCIe Aggregation
PCI Express Fabric
PCIe Aggregation
Flash Flash Flash Flash
Flash Flash Flash Flash
Flash Flash Flash Flash
Flash Flash Flash Flash
SSD Controller
SSD Controller
PCIe Aggregation
36 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Building a Complete Fabric
Development platforms for customer products ExpressNIC
PCIe bus extender Based on PEX 8725 2 QSFP+ ports, x4 PCIe each
Argo Rack-mounted PCIe switch fabric 1U form 32 QSFP+ ports, x4 PCIe
37 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.
Better Storage Through PCIe
PCIe-based solutions offer substantial benefits to storage systems
Direct connection to vast number of devices Low latency, high bandwidth solution Flexible sharing with existing infrastructure Expand as specification improves
Especially attractive to SSD-based systems
PRESENTATION TITLE GOES HERE
Thank You! For More Info, please visit
www.plxtech.com