nand flash solid state storage performance and … slide or slides used must be reproduced in ......
TRANSCRIPT
NAND Flash Solid State Storage Performance and Capability -- an In-depth Look
jonathan thatcher, Fusion-io
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
2
SNIA Legal Notice
The material contained in this tutorial is copyrighted by the SNIA. Member companies and individual members may use this material in presentations and literature under the following conditions:
Any slide or slides used must be reproduced in their entirety without modificationThe SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations.
This presentation is a project of the SNIA Education Committee.Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney.The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information.
NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Abstract
NAND Flash Solid State Storage Performance and Capability"This tutorial provides an in-depth examination of the fundamental theoretical performance, capabilities, and limitations of NAND Flash-based Solid State Storage (SSS). The tutorial will explore the raw performance capabilities of NAND Flash, and limitations to performance imposed by mitigation of reliability issues, interfaces, protocols, and technology types. Best practices for system integration of SSS will be discussed. Performance achievements will be reviewed for various products and applications. Several examples of successful enterprise deployments with comparative performance and cost-performance will be presented."
3
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Data Integrity + Performance
There can be no data integrity trade-off for performance
4
Balance
Reliability & Data Integrity
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Media Reliability / Availability
The GOOD:No moving partsCatastrophic device failures are rare (post infant mortality)
The BAD:Relatively high bit error rate, increasing with wear
MLC wear rate (higher capacity density) worse than SLCHigher density NAND Flash will increase bit error rate
Program and Read Disturbs
The UGLY:Partial Page ProgrammingData retention is poor at high temperatureInfant mortality is high (large number of parts…)
5
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Controller Reliability Management
Wear leveling & Spare Capacity (e.g. Spare Blocks)Read & Program Disturb ControlsData & Index Protection
ECC CorrectionInternal RAIDData Integrity Field (DIF)
Management
Note: Multipage Programming Should Not Be Done
6
Poor Media + Great Controller Great SSS Solution
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Data Integrity .V. Performance
7
BAD GOODP
erfo
rman
ce
Data Integrity99.999%99.99%99.9%99% …
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Lower CapExFewer CPUsLess RAMLess Network GearFewer SW LicensesLess Space
Lower OpExLess HW MaintenanceLess SW MaintenanceGreater UptimeLess Power/CoolingFewer Diverse Skills
8
Higher Productivity
Performance is about ROI
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
SSS ROI: Three Quick Examples
Case StudiesWine.comCloudmark
O&G
9
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Company: Wine.com
On-line internet Retail
Representative Markets: Transaction ProcessingData Mining
ProblemSystems not capable Contract out data miningPerformance hitting 100% Loss of revenue (4Q07)Cost to meet growth: prohibitive
10
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
11
Back EndPayment
Processing
Front End
33% CapEx avoidance50% OpEx reduction33% Footprint reduction
12x improvement on write- Latency down from 4 to <1ms
14x improvement on read- Latency down from 12 to <1ms
Before (3Q2008) After (1Q2009)
“Enough capacity to cover 24 months of growth”Geoffrey Smalling - CTO
Wine.com
InternetInternet
Application level replication for redundancy
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Company: Cloudmark
Email Security – Spam, Phishing, Viruses, etc.
Protecting 850 Million mailboxes globally100+ Service Provider customers (mostly Tier-1)High TPS at master – morning bursts are higher
Problem scaling MySQL & InnoDB worldwideRapid growth – both customers and new threatsI/O limited hit performance ceiling 50% of the timeSlave servers fall behind increasing risk windowHigh transaction rate constant disk failuresCost to meet growth: prohibitive
12
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Cloudmark
Transformed slave clusters to Solid State Storage and retired disk arrays – results:
4 to 1 footprint reductionCapEx avoidance - $2 million10x performance increaseCluster peak utilization now only 15% of IOPS capability12 months growth secured with current hardware
“We never thought our problems would ever change from disk I/O problems to CPU bottlenecks!”
Ryan White – Director of Operations
13
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Company: LLOG
Oil & Gas explorationMarket:
Natural resource discovery using seismic interpretation leading to productionSeismic analysis software used for 3D interpretations
Manipulation and analysis of large datasets (1Tb+)
Quality interpretations result in accurate auction bidsMinimize risk and make informed decisions
Geoscientist productivity challengesIdle time loading data into high-end workstationsLost productivity during basic analysis tasksTime wasted in running jobs in serial
14
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
LLOG
Use of Solid State Storage on workstations Load time reduced from 8 to 2 minutes3D time slice time reduced from 28 to 18 minutes
“Supercharged Virtualization” with Solid State StorageProjects took 30+ minutes causing 100% CPU utilization
Now 10 minutes with projects run in parallelIncreased geoscientist’s productivity 5x (projects in parallel)
“We are more competitive because we make better decisions and did it while reducing our costs”
John Hollins – Geophysicist
15
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
16
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
17
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Media Performance
The GOOD:Performance is great (wrt HDDs)High performance/power (IOPS/Watt)Low pin count: shared command / data bus good balance
The BAD:Not really a random access device
Block orientedSlow effective write (erase/transfer/program) latencyR/W access speed imbalance
Performance changes with wearSome controllers do read/modify/writeOthers use inefficient garbage collection
The UGLY:Some controllers do read/erase/modify/write
18
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Performance Drivers – SSS Design
InterconnectNumber of NAND Flash Chips (Die)Number of Buses (Real / Pipelined)Data Protection (internal/external RAID; DIF; ECC…)SLC / MLCEffective Block (LBA; Sector) SizeWrite AmplificationGC EfficiencyBandwidth ThrottlingBuffer Capacity & Mgmt
19
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Performance Drivers - External Cond
Transfer SizeRead/Write RatiosTemporal Randomness of AccessReserve Capacity Setting (% of used capacity)System Limitations (especially wrt scalability)
External Controller (#, Type, Performance); # ThreadsCPU (#Cores, GHz)System BandwidthSoftware Stack; Interrupt Handler
External RAIDLife of device (change in device affects tErase & tProgram)
20
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Simplified Theoretical Analysis
Bandwidth Only (Not IOPS)Large Transfers (Data length = Integer * # die)Infinite BufferReads/Writes queued for maximum bandwidthNo system latency
Read/Write Ratio %’s fixed100/0, 75/25, 50/50, 25/75, 0/100Steady State, 100% Efficient GC (EB erase / EB written = 1)
Maximum Total BW for SATA-II and PCI-e X4No overhead considered
SLC
21
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Bandwidth Depends on # Die
22
-
200
400
600
800
1,000
1,200
16 32 48 64 80 96 112 128 144 160 176 192
Theoretical BW (MB/s) v Number of Die (SLC, MLC)
SLC BW (0/100, 50/50, 75/25) MB/s
MLC BW (50/50) MB/s
MLC BW (25/75) MB/s
MLC BW (0/100) MB/s
Presumes 8 die per bus& 4 CS per bus
SLC MLC
Transfer Rate (MB/s) tRC & tWC 400 400
Page Program (us) tProgram 200 600
EB Erase (us) tErase 3000 10,000
Load Page (us) tR (tRead) 25 60
Capacity per die 0.5 1.0
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Single Layer Cell v Multi-Layer Cell
23
-
100
200
300
400
500
600
700
800
900
1,000
1 2 3 4 5 6 7 8
Bandwidth (MB/s) v #Chip Select @ R/W Ratio
SLC 100/0
SLC 75/25
SLC 50/50
SLC 25/75
SLC 0/100
MLC 100/0
MLC 75/25
MLC 50/50
MLC 25/75
MLC 0/100
Read / write performance imbalance closed with additional banksGreater R/W imblance in MLC requires more banks
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Performance Data Acquisition
In this presentation, three commercially available SSS storage solutions were tested under a variety of conditions using a common system:Supermicro X7DWE Motherboard
Dual, Quad Core Intel X5460: 3.16 GHz16 GB DRAM
4x4GBKVR800D2D4F5 /4GI, 800MHz FBDIMM DDR2, CL5
PCI-E Gen 2 bus
SATA Controller LSI SAS3081E-R All tests except pathological write, which used on-board SATADriver version used was: 4.00.43.00; firmware rev: FwRev=011b0000h
24
Note: all data collected on devices at beginning of life
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Performance Data Acquisition
Software / OSLinux/CentOS5/RHEL5 2.6.18-92.1.10.el5. “fio” program, version 1.21:
From http://freshmeat.net/projects/fio)
“IOMeter” V 2006.07.27 in Windows 2003/2008 for latency only
SettingsDirect I/O: o_direct used to bypass caching & bufferingI/O Scheduler (elevator algorithm) set to null
25
Note: all data collected on devices at beginning of life
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Measured v Theoretical Bandwidth
26
Note: Theoretical Max BW with 24 channels (4 die per bus, 4 CS per bus) is identical to the PCI-C, 24 channel shown in these charts.
Capacity Multiplier: SATA-B: 1PCI-C: 2
0
500
1000
1500
2000
100/0 (X1)
75/25 (X1)
50/50 (X1)
25/75 (X1)
0/100 (X1)
Measured v Theoretical Max BW
24 channels (8 die per bus; 4 CS per bus)
PCI-C BW Measured
10 channels (SLC; 4 die per bus; 4 CS per bus)
SATA-B BW Measured
PCIe-X4 Max BW
SATA-II Max BW
40%50%60%70%80%90%
100%
100/0 (X1)
75/25 (X1)
50/50 (X1)
25/75 (X1)
0/100 (X1)
Measured BW as % of Theoretical Max
PCI-C BW Actual/Theoretical
SATA-B BW Actual/Theoretical
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Product Specifications
27
SATA (A) SATA (B) PCI (C)
Capacity (GB) 32 32 160
Bus/Link SATA-II (3 Gb/s) SATA-II (3 Gb/s) PCI-E X4 1.1
MemoryType SLC SLC SLC
Adjustable Reserve Capacity No No Yes
SSS Internal RAID-- Running during test
NoN/A
NoN/A
YesYes
K-IOPS (RMS) 8 27 88
K-IOPS (RMS) / WATT 3 ? 7
Bandwidth (RMS, MB/s) 56 208 743
ECC correction 7 bits in 512B ? 11 bits in 240B
Features directly affecting performance measurements
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Access Process (Physics Ignored)
Read AccessAddress Chip / EB / PageLoad Page into RegisterTransfer Data From Register 1-byte per cycle
Write AccessAddress Chip / EBErase EB
…some time later…
Address Chip / EB / PageTransfer Data To Register 1-byte per cycleProgram Register to Page
28
Typical NAND Flash Die:• 2000 Erase Blocks (EB)• 64 Pages per EB• 4000 Bytes per Page• 500 MByte Total Capacity
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Example 1: Read/Erase/Modify/Write
29
Page Erase Block 1
0 b c -- --
1 j -- k l
2 m -- -- --
3 -- -- q r
Page Erase Block 1
0 b c W X
1 j Y k l
2 m
3 q r
Time = t2
Write Buffer & W,X,Y
Time = t1
Starting State
Time = t3
Write Buffer & Z,A,B’,C’,R’
Buffer holds data while EB-1 Erased
Page Erase Block 1
0 B’ C’ w x
1 j y k l
2 m Z A
3 q R’
Page Erase Block 1
0
1
2
3
Buffer holds data while EB-1 Erased
Page Erase Block 1
0
1
2
3
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Explanation of Previous Slide
AssumptionsSimplified to show erase blocks with 4 pages, each page having 4 data blocksInvalid (erased or replaced) data is indicated by “—”Old data is indicated by lower case lettersNew data is indicated by CAPs; Replacement data is indicated by “prime” (e.g. c C’)
Detail T = t1 to T = t2 transitionData is read from EB-1EB-1 is erasedNew data {W,X,Y} modifies previous invalid dataData is written back to EB-1
Detail T = t2 to T = t3 transitionData is read from EB-1 into data bufferEB-1 is erasedNew data {B’, C’, Z, A, R’} modifies previous data in data bufferData is written back to EB-1
30
Note: backup material for those reviewing or looking at
presentation without audio/video
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Example 2: Read/Modify/Write
31
Page Erase Block 1
0 b c -- --
1 j -- k l
2 m -- -- --
3 -- -- q r
Page Erase Block 2
0 b c W X
1 j Y k l
2 m
3 q r
Page Erase Block 3
0 B’ C’ w x
1 j y k l
2 m Z A
3 q R’
Time = t2
Data to Buffer (not shown)
Erase EB-1 (not shown)
Write Buffer & W,X,Y to EB-1
Time = t1
Starting State
Time = t3
Data to Buffer (not shown)
Erase EB-1 (not shown)
Write Z,A & Replace b,c,rwith B’,C’,R’ & Write EB-1
Implicit wear leveling; EB-1 EB-2 EB-3Presumes that destination EB-2 & EB-3 erased prior to transfer of data higher
performance (than previous “Read/Erase/Modify/Write” example)
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
“Write Amplification Impact”
In this example, Data written t1 to t2: 16 blocks
NEW DATA {W, X, Y} 3 blocks; Copied Data {b, c, j, k, l, m, q, r} 8 blocksNull Data: 5 blocks
Data written t2 to t3: 16 blocksNEW DATA {B’, C’, Z, A, R’} 5 blocks; Copied Data {w, x, j, y, k, l, m, q} 8 blocksNull Data: 3 blocks
(2) EB erasures25% (8 of 32) writes are user initiated75% (24 of 32) writes are internal data movement (overhead)
Important:Amount of valid or invalid data in EB-1 is irrelevant to performance impact“Write Amplification” is workload (access pattern) dependent (e.g., what if the write of R’ above was not coincident with B’ & C’)
32
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
SATA-B: IOPS vs Transfer Size
33
0
10000
20000
30000
40000
50000
60000
512 1k 2k 4k 8k 16k 32k 64k 128k 256k 512k 1m
SATA-B: IOPS v Transfer Size (X1)
1x Write
1x 25/75
1x 50/50
1x 75/25
1x Read
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Example 3: Garbage Collection
34
Page Erase Block 1
0 b c -- --
1 j -- k l
2 m -- -- --
3 -- -- q r
Page Erase Block 1
0 b c -- --
1 j -- k l
2 m -- -- --
3 -- -- q r
Page Erase Block 1
0
1
2
3
Page Erase Block 2
0
1
2
3
Page Erase Block 2
0 W b c X
1 Y j k l
2 m q r
3
Page Erase Block 2
0 w -- -- x
1 y j k l
2 m q -- B’
3 C’ Z A R’
Time = t2
EB-1 GC’d to EB-2
W,X,Y added
Time = t3
EB-1 erase
b,c,r replaced by B’,C’,R’
Time = t1
Start Garbage Collect EB-1
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Explanation of Previous Slide
AssumptionsNew data blocks and data blocks being garbage collected are interleaved
DetailsAt Time = t0, erase block 1 (EB-1) is identified for GCAt Time = t1, good data is moved from EB-1 to EB-2 (it is implicit that an index is updated accordingly); New data W, X, and Y are added while the GC is taking place. EB-1 is then ready to be erasedAt Time = t2, EB-2 is erased; Data for b, c, & r have been updated with B’, C’ & R’; b, c & r are indicated as “Invalid.”
35
Note: backup material for those reviewing or looking at
presentation without audio/video
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
GC Performance Impact
In this example, COPIED DATA: {b, c, j, k, l, m, q, r} 8 blocksNEW DATA {W, X, Y, B’, C’, Z, A, R’} 8 blocks50% (8 of 16) writes are user initiated50% (8 of 16) writes are internal movement (overhead)
Important:50% of EB-1 was “invalid data”What if only 10% had been “invalid data?”GC efficiency is dependent upon % of reserve capacity
36
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Tower of Hanoi
37
Want to do this in fewer moves?Add more pegs!
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
GC: Pathological Write Conditions
IF high percentage of total storage capacity utilized
AND
High percentage of data has no correlation-in-time
AND
Continuous writing (no recovery time for GC)
THEN
Efficiency of GC greatly diminished
38
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Pathological Write Condition
39
0
100
200
300
400
500
600
7000
540
1081
1622
2162
2703
3244
3785
4325
4866
5407
5948
6489
7030
7571
8112
8653
9194
MB
/s
Seconds
30GiB of 80G PCI-C
40GiB of 80G PCI-C
60GiB of 80G PCI-C
70GiB of 80G PCI-C
74GiB of 80G PCI-C
28GiB of 30G SATA-B
User Capacity Formatted of Total
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Scalability
Following Slides Show Scalability of {1, 2, 4, 8} unitsOnly 1 SATA controller is used – limiting scalability
Only 1 thread running
Measurements taken at Read/Write Ratios of {100/0, 75/25, 50/50, 25/70, 0/100} RMS value is the “root mean square” of these five values
IOPS measurement taken at 512 Byte TransfersBandwidth taken at 128K Byte Transfers
Unless shown differentlyLinux has a 128K limit
40
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Performance v R/W Ratio
41
0
20000
40000
60000
80000
100000
120000
100/
0 (X
1)
75/2
5 (X
1)
50/5
0 (X
1)
25/7
5 (X
1)
0/10
0 (X
1)
IOPS @ 512 B
0
100
200
300
400
500
600
700
800
900
100/
0 (X
1)
75/2
5 (X
1)
50/5
0 (X
1)
25/7
5 (X
1)
0/10
0 (X
1)
Bandwidth (MB/s) @ 128 KB
SATA-ASATA-BPCI-C
Read/Write Collisions Drop in Mixed Performance
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Scalability v R/W Ratio
42
R/W Ratio (# Units in Parallel)
0
50000
100000
150000
200000
250000
300000
100/
0 (X
1)75
/25
(X1)
50/5
0 (X
1)25
/75
(X1)
0/10
0 (X
1)
100/
0 (X
2)75
/25
(X2)
50/5
0 (X
2)25
/75
(X2)
0/10
0 (X
2)
100/
0 (X
4)75
/25
(X4)
50/5
0 (X
4)25
/75
(X4)
0/10
0 (X
4)
100/
0 (X
8)75
/25
(X8)
50/5
0 (X
8)25
/75
(X8)
0/10
0 (X
8)
IOPS @ 512 B
X1 X2 X4 X8
0
1000
2000
3000
4000
5000
6000
100/
0 (X
1)75
/25
(X1)
50/5
0 (X
1)25
/75
(X1)
0/10
0 (X
1)
100/
0 (X
2)75
/25
(X2)
50/5
0 (X
2)25
/75
(X2)
0/10
0 (X
2)
100/
0 (X
4)75
/25
(X4)
50/5
0 (X
4)25
/75
(X4)
0/10
0 (X
4)
100/
0 (X
8)75
/25
(X8)
50/5
0 (X
8)25
/75
(X8)
0/10
0 (X
8)
Bandwidth (MB/s) @ 128 KB
SATA-A
SATA-B
PCI-C
X1 X2 X4 X8
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
RMS Scalability (# SSS Units)
43
0
1000
2000
3000
4000
5000
6000
1 2 4 8
RMS of Bandwidth v Scale
SATA-A
SATA-B
PCI-C
0
50000
100000
150000
200000
250000
1 2 4 8
RMS of IOPS v Scale
0.6
0.7
0.8
0.9
1
1.1
1 2 4 8
Normalized RMS Bandwidth v Scale
SATA-A
SATA-B
PCI-C
0
0.2
0.4
0.6
0.8
1
1.2
1 2 4 8
Normalized RMS IOPS v Scale
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Performance v Block Size (75/25)
44
0
20000
40000
60000
80000
100000
120000
512 1k 2k 4k 8k 16k
32k
64k
128k
256k
512k 1m
Block Size
75/25 R/W IOPS
0
100
200
300
400
500
600
700
800
900
512 1k 2k 4k 8k 16k
32k
64k
128k
256k
512k 1m
Block Size
75/25 R/W Bandwidth (MB/s)
SATA-ASATA-BPCI-C
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Scalability v RW Ratio v Block Size
45
512
32k0
250
500
750
1000
8x W
rite
8x 2
5/75
8x 5
0/50
8x 7
5/25
8x R
ead
4x W
rite
4x 2
5/75
4x 5
0/50
4x 7
5/25
4x R
ead
2x W
rite
2x 2
5/75
2x 5
0/50
2x 7
5/25
2x R
ead
1x W
rite
1x 2
5/75
1x 5
0/50
1x 7
5/25
1x R
ead
SATA-A Scalability Bandwidth (MB/s)
0
10000
20000
30000
40000
50000
60000
70000
80000
90000
100000
8x W
rite
8x 2
5/75
8x 5
0/50
8x 7
5/25
8x R
ead
4x W
rite
4x 2
5/75
4x 5
0/50
4x 7
5/25
4x R
ead
2x W
rite
2x 2
5/75
2x 5
0/50
2x 7
5/25
2x R
ead
1x W
rite
1x 2
5/75
1x 5
0/50
1x 7
5/25
1x R
ead
SATA-A ScalabilityIOPS
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
5128k
128k0
250
500
750
1000
1250
1500
8x W
rite
8x 2
5/75
8x 5
0/50
8x 7
5/25
8x R
ead
4x W
rite
4x 2
5/75
4x 5
0/50
4x 7
5/25
4x R
ead
2x W
rite
2x 2
5/75
2x 5
0/50
2x 7
5/25
2x R
ead
1x W
rite
1x 2
5/75
1x 5
0/50
1x 7
5/25
1x R
ead
SATA-B ScalabilityBandwidth (MB/s)
Scalability v RW Ratio v Block Size
46
0
10000
20000
30000
40000
50000
60000
70000
80000
90000
100000
110000
120000
8x W
rite
8x 2
5/75
8x 5
0/50
8x 7
5/25
8x R
ead
4x W
rite
4x 2
5/75
4x 5
0/50
4x 7
5/25
4x R
ead
2x W
rite
2x 2
5/75
2x 5
0/50
2x 7
5/25
2x R
ead
1x W
rite
1x 2
5/75
1x 5
0/50
1x 7
5/25
1x R
ead
SATA-B ScalabilityIOPS
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Scalability v RW Ratio v Block Size
47
0100002000030000400005000060000700008000090000100000110000120000130000140000150000160000170000180000190000200000210000220000230000240000250000260000270000
8x W
rite
8x 2
5/75
8x 5
0/50
8x 7
5/25
8x R
ead
4x W
rite
4x 2
5/75
4x 5
0/50
4x 7
5/25
4x R
ead
2x W
rite
2x 2
5/75
2x 5
0/50
2x 7
5/25
2x R
ead
1x W
rite
1x 2
5/75
1x 5
0/50
1x 7
5/25
1x R
ead
PCI-C Scalability IOPS
5128k
128k02505007501000125015001750200022502500275030003250350037504000425045004750500052505500575060006250
8x W
rite
8x 2
5/75
8x 5
0/50
8x 7
5/25
8x R
ead
4x W
rite
4x 2
5/75
4x 5
0/50
4x 7
5/25
4x R
ead
2x W
rite
2x 2
5/75
2x 5
0/50
2x 7
5/25
2x R
ead
1x W
rite
1x 2
5/75
1x 5
0/50
1x 7
5/25
1x R
ead
PCI-C ScalabilityBandwidth (MB/s)
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Media Reliability / Availability
48
Problem SSS Solution System Solution
High Infant Mortality RAID: NAND Flash RAID: SSS
High Error Rate & Wearout
RAID: NAND Flash RAID: SSS
Robust ECC No Defragmentation
DIF DIF
Wear Leveling Access Tuning
Never Multi-page PGM
Thermal Management Temperature & Air Flow
Non-random Address Indirection N/A
R/W not symmetric More Banks (CS); esp for MLC Reduce write thrashing
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
System Level Considerations
Data / Index Protection (DIF ; RAID)
Scalability
Compare system- or data-center-level; not device
Best case: test on real application, not benchmarkPlan to do tuning to reach top performance / objectivesApplications may have contra-indicated optimizations
Example: keeping data in close physical proximity (short stroking)Example: caching algorithms
49
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Questions to Ask : Things to Know
Bandwidth / IOPS atBlock size(s) you needR/W ratio you useSteady State / BurstData’s temporal relationshipScalabilityRAIDingReserve capacity usedBOL / EOL
50
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
Questions to Ask : Things to Know
Design impacts on data integrity; life; failures & performanceECC robustnessWrite amplification / GC efficiencyInternal RAIDBandwidth throttlingPartial Page Programming
Test ConditionsRAID On/Off during testing?Caching On/Off during testing?WorkloadTemporal RelationshipsUser capacity / reserve capacity
51
NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.
52
Q&A / Feedback
Please send any questions or comments on this presentation to SNIA: [email protected]
Many thanks to the following individuals for their contributions to this tutorial.
- SNIA Education Committee
Khaled AmerPhil Mills
Rob PeglarMarius Tudor