ddn update 2019 - npo tsukuba oss technical support...

© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.

Any statements or representations around future events are subject to change.

1

DDN Storage | © 2019 DDN Storage © 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.


DDN Storage | © 2019 DDN Storage

DDN Update 2019Gfarm Workshop 2019

2019/2/1

DataDirect Networks Japan, Inc.

橋爪信明 ([email protected])



2

DDN Storage | © 2019 DDN Storage 2

2018年主な出来事＠DDN

• Declustered RAID(DCR)対応製品へラインナップ刷新

► SFAOS11世代のHWへの切り替え元年

► SFA200/400NV, SFA7990, SFA18K

• 6月 Intel社よりLustre部門を買収

► DDNのWhamcloudディビジョンとして活動

► Open Source Community Edition開発

► Lustre L3サポート

• 9月 Tintri社を買収

► DDN内の独立したTintriディビジョンとして活動

►仮想化・マルチクラウド自動最適化フラッシュストレージ



3


DDN SFA

ALL-FLASH AND HYBRID BLOCK STORAGE PLATFORMS

200NV 400NV 7990 14KX 18K

20GB/s

1M IOP/s

40GB/s

2M IOP/s

20GB/s

1M IOP/s

60GB/s

4M IOP/s

90GB/s

3.2M IOP/s

24 NVME Slots 24 NVME Slots

Up to 450

SSD/HDD

48 NVMe Slots

Up to 1872

SSD/HDD

48 NVMe Slots

Up to 1872

SSD/HDD

EDR IB (4), OPA (2)

FC32 (8), FC (8)

EDR IB (8), OPA (4) EDR IB (4), OPA (2)

FC16 (8)

EDR IB (12|8)

OPA (4), FC16 (24)

EDR IB (16), OPA (8)

COMING 2019COMING 2019 NEW 2018NEW 2018 To be EOS

2019年より全グレードで新ラインナップを展開



4


DDN SFA | 製品名と位置付け

位置付け用途 200NV 400NV 7990 18K

総称全体をまとめて呼ぶ場合 SFA200NV SFA400NV SFA7990 SFA18K

ブロックストレージファイルサーバに接続 SFA200NV SFA400NV SFA7990 SFA18K

エンベデッドアプライアンスコントローラ上にファイルサーバを実装

SFA200NVE SFA400NVE SFA7990E SFA18KE

ExaScalerアプライアンス ExaScaler(Lustre)エンベデッド ES200NV ES400NV ES7990 ES18K

GridScalerアプライアンス GridScaler(GPFS)エンベデッド GS200NV GS400NV GS7990 GS18K

A3Iアプライアンス AI向けエンベデッド AI200 AI400 AI7990



5




SFA18K



6


MAXIMUM SCALABILITY WITH NVMe + SAS

* Cached

SFA18K

FOR CAPACITY & BANDWIDTH WITH DRIVE TYPE FLEXABILITY

Block 90GB/s

Embedded 78GB/s

• Start in as little as 4U and Scale up & Scale out;

Up to 1872 drives per system:

o 72 Slots in “head”, 48 NVMe/SAS + 24 SAS

o Add up to 20x SS9012 Enclosures (90 SAS slots)

• Flexible Access for File: IB or Ethernet (16 ports), OPA

(8 ports)

Integrates high-performance, low-latency, RDMA capable

networks. EDR InfiniBand and 100Gbps (or 40Gbps)

Ethernet and OPA

• Access for Block: IB (16 ports)

Block systems attach via EDR IB to external servers



7


SFA18K Overview

Storage Fusion FabricHighly-Over-Provisioned Backend RAIDed

Fabric Withstands Failures of Drives,

Enclosures, Cables, etc

Data IntegrityReal-Time Data Integrity Verification for every I/O

Flash and NVMe Enabled in

‘head’Up To 48 NVMe devices

Up to 72 SAS SSD

Broad Connectivity Options

EDR IB

40/100 GbE

OmniPath

High Density ArrayUp to 972 drives in 44U rack.

SS9012 enclosure is 90 HDDs / SSDs in

4U

ReACT Cache ManagementReal-Time I/O Data-Aware Cache Optimizer,

Small IOs Go to Cache; Large & Streaming

I/Os Bypass Cache

Real-Time, Multi-CPU RAID EngineFull Implementation of Real-Time Storage OS,

Interrupt-Free; Massively Parallel I/O Delivery

System

SFA Embedded Low-Latency VirtIO data path for DMA IO access

to local VM

0 500 1000

8+2 rebuild vs DCR rebuild in 112 disk pool and 210 disk pool

DCR 210 DCR 112 Legacy 8+2

DeClustered RAID



8


SFA18K Controller



9


ConfigSFA18K

Only+ 1 SS9012 +2 SS9012 +4 SS9012 +5 SS9012 +6 SS9012 +8 SS9012 +10 SS9012 +16 SS9012 +20 SS9012

SFA (BLOCK)

ES

GS

2.5” NVMe or

SAS SSD

Slots

72

(NVMe:48)

72

(NVMe:48)

72

(NVMe:48)

72

(NVMe:48)

72

(NVMe:48)

72

(NVMe:48)

72

(NVMe:48)

72

(NVMe:48)

72

(NVMe:48)

72

(NVMe:48)

3.5” HDD or

SAS SSD

Slots

0 90 180 360 450 540 720 900 1440 1800

Enclosure

ConnectionNA Direct Direct Direct Direct Direct Direct Direct

Direct

Daisy ChainDaisy Chain

最大物理容量 NA 1.26PB 2.52PB 5.04PB 6.30PB 7.56PB 10.08PB 12.6PB 20.16PB 25.20PB

Expansion

Options

SFA18K CONFIGURATIONS

※最大物理容量は14TB NL-SAS HDDをSS9012に実装した場合



10


EXAScaler ES18K Early Performance - IOR 1MB FPP

• EXAScaler IOR Client Benchmarking with an ES18K EDR with 408 NL-SAS

Drives as 8 DeClustered RAID Pools of 51 drives.

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

Ban

dw

idth

(M

B/s

ec)

ES18K IOR(FPP) Write: 408x NL-SAS, 8 x51/MR=1

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

Ba

nd

wid

th (

MB

/se

c)

ES18K IOR(FPP) Read: 408x NL-SAS, 8 x51/MR=1

61GB/s

Write58GB/

s Read

Tuning is

still in

progress



11




SFA7990



12


SFA7990 WORLD’S MOST EFFICIENT HYBRID STORAGE

* Cached

SFA7990

OPTIMIZED FOR SSD, HDD CAPACITY, MULTICORE & MULTI GPU

Block 20GB/s

Embedded 20GB/s

• Start at 4U and Scale Up or Scale Out

Mix Disk and Flash SSD for Hybrid Performance

• Flexible Access for File: (IB, Ethernet, OPA)


networks. Both EDR InfiniBand and 100Gbps (or 40Gbps)

Ethernet plus OPA

• Flexible Access for Block: (IB, FC)

Block systems attach via FC or IB to external servers

DDN’s ES7990 and GS7990, EXAScaler® and GRIDScaler® appliances,

offer scalable, high performance parallel file systems in an integrated

package. Designed, deployed and supported by the experts in data intensive

workloads, these appliances are the gold standard for eliminating bottlenecks and

maximizing application performance.



13


SFA7990 OVERVIEW

Redundantly Connected

EnclosuresAll configurations feature directly

attached, redundant links to

SFA7990

Flexible ConnectivityBlock Target: 4x EDR IB OR 8x FC16

Embedded File System: 4 x EDR IB

/40/100 GbE

OR 2 x OPA

Full SAS Performance for all

configurationsBalanced SAS Architecture provides

saturation level performance for all

expansion options

High Density HDD and SAS SSDSS9012 Enclosures and SFA7990 house

up to 90 3.5” Devices.

High Performance SASBackend Bandwidth supported by

12Gbps SAS Links from all

enclosures to SFA7990

SFAOS

Mirrored, battery-backed write-back cache

ReACT Cache Management

Fully Declustered Data Placement for rapid rebuilds

EMBEDDED FILESYSTEM APPLIANCES

EXAScaler®

GRIDScaler®



14


Configuration SFA7990 Only + 1 SS9012 +2 SS9012 +4 SS9012

Supported

Solutions

SFA (BLOCK)

ES7990

GS7990

3.5” HDD or SAS

SSD Slots90 180 270 450

最大物理容量 1.26PB 2.52PB 3.78PB 6.30PB

Expansion

Options

SFA7990 CONFIGURATIONS

※最大物理容量は14TB NL-SAS HDDをSS9012に実装した場合



15


ES7990 WORLD’S MOST EFFICIENT HYBRID STORAGE

ES7990


20GB/s

0

5000

10000

15000

20000

25000

80xNL-SAS 160xNL-SAS

Thro

ugh

pu

t (M

B/s

ec)

ES7990 Sequential IO Performance

Write Read

0

2

4

6

8

10

12

14

16

18

20

0

100

200

300

400

500

600

700

800

900

4KB 8KB 16KB 32KB 64KB 128KB

Thro

ugh

pu

t (M

B/s

ec)

IOP

S (o

ps/

sec)

IO Size (Byte)

ES7990 Random IO Performance (10xSSD)

IOPS B/W

780K IOPS@4KB

13.7GB/s Write

205MB/s Per Disk



16


GS7990 WORLD’S MOST EFFICIENT HYBRID STORAGE

* Cached

GS7990


20GB/s



17




SFA200/400NV



18


WORLD’S MOST EFFICIENT HPC FLASH

* Cached

SFA200NV

OPTIMIZED FOR NVMe IOPS, MULTICORE & MULTI GPU

Block 20GB/s

Embedded 20GB/s

• Start at 2U and Scale-out

All NVMe Flash, high density, extreme

performance

• Flexible Access for File: (IB, Ethernet,

OPA)

Integrates high-performance, low-latency,

RDMA capable networks. EDR InfiniBand and

100Gbps (or 40Gbps) Ethernet and OPA

• Flexible Access for Block: (IB, FC)

Block systems attach via FC or IB to external

servers



19


WORLD’S MOST EFFICIENT HPC FLASH

* Cached

SFA400NVBlock 40GB/s

Embedded 40GB/s

• Start at 2U and Scale-out

All NVMe Flash, high density, extreme performance

• Flexible Access for File: (IB, Ethernet, OPA)


networks. EDR InfiniBand and 100Gbps (or 40Gbps)

Ethernet and OPA

OPTIMIZED FOR NVMe IOPS, MULTICORE & MULTI GPU



20


GS200NV : SCALABLE FILE PLATFORMS

• Saturation level random read

performance with small I/O

transfers

• Ideal for Analytics platforms

requiring Enterprise Features

OVER 23GB/s

THROUGHPUT EVEN

FOR RANDOM READ

WORKLOADS



21


0

200

400

600

800

1000

1200

1400

Raw

Capacity (

TB

)

SFA200NV and SFA400NV Capacities

SFA200NV AND SFA400NV | SCALABLE FILE PLATFORMS

• Choose your starting option

from as small as 38TB or up to

152TB in just 2 RU

• Scale-out with NVMe to

Petabytes of Flash and TB/s of

performance

Start with a complete All

NVMe Flash, Parallel File

System from DDN in just

2RU

S M L 2L 4L 8L38TB 76TB 152TB 304TB 608TB 1.2PB

Coming soon

15.36TB NVMe



22




A3IAccelerated, Any-scale AI Solutions From DDN



23


AI7990 AI200

All NVME parallel file storage appliance

Optimized for the most intensive workloads

30TB, 60TB, 120TB in 2RU

23GB/s, 395K IOPS • 100Gb Ethernet or EDR InfiniBand • Start with 1, Scale out and mix

Hybrid parallel file storage appliance

Optimized for capacity, intermix with flash

1PB in 4RU

DDN PARALLEL FILE STORAGE APPLIANCES FOR AI AND DL



24


®



25


®



26


A3I

wit

h s

ing

le D

GX

-1

A3I

wit

h q

ua

d D

GX

-1

A3I

wit

h D

GX

-1 P

OD

DDN A3I WITH DGX-1 SCALABLE REFERENCE ARCHITECTURES



27


Over 100K IOPs and

10GB/s to 1 Container

EXAScaler client demonstrates over 10GB

per second to a single container and over

100K IOPs.

Typical Deep Learning Codes will perform

IOs around 128K IOs – and see around

6GB/s with this pattern.

DDN A3I Solutions: Performance to a single container



28


60% Faster -

Consistent, Linear AI

Performance

AI benchmarking of AI Frameworks with

popular test sets demonstrates images per

second rates 60% higher than competing

systems

DDN A3I Solutions: Fully-optimized with widely-used DL frameworks



29


AI Benchmarking: DDN enables and accelerates

1796

4344

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000

Imag

es P

er

Seco

nd

CAFFE GOOGLENET TRAINING THROUGHPUT2.4X MORE IMAGES WITH DDN STORAGE

DDN Storage Other Storage Products

379

197

0 50 100 150 200 250 300 350 400

Tim

e (i

nm

inu

tes)

CAFFE GOOGLENET TRAINING TIME2X FASTER WITH DDN STORAGE




30


Full GPU Saturation,

Maximum Productivity

DDN AI200 delivers a steady high-

throughput, low-latency stream of data to

the DL training application

Achieving full GPU saturation ensures

every compute cycle is put to productive

use

DDN true end-to-end parallelism enables

acceleration at scale for distributed

computing with multiple multi-GPU nodes

AI Benchmarking: IO during TensorFlow training



31


AI Benchmarking: Optimized TensorFlow Data Sets

Optimized Data Sets,

Accelerated Training

Convert discrete data and metadata asset

collections into series of streamlined binary files for

TensorFlow

Maximize DL training and validation efficiency while

minimizing data management and preparation

overhead

Achieve more accurate results by eliminating

manual data shuffle bias

67

23

0 10 20 30 40 50 60 70 80

Tim

e (i

n m

inu

tes)

TFRECORD CONVERSION BENCHMARK3X FASTER WITH DDN STORAGE


1614

671

0 200 400 600 800 1000 1200 1400 1600 1800

Tim

e (i

nse

con

ds)

TENSORFLOW RESNET-50 TRAINING PERFORMANCE

2.5X FASTER WITH TFRECORDS

TFRecord Data Set Discrete Data Set



32


DDN PARALLEL ARCHITECTURE AND PROTOCOL vs NFS

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

GPU GPU

FULL GPU SATURATION

MAXIMUM AI PRODUCTIVITY

GPU STARVATION

CRIPPLED AI PERFORMANCE



33


DDN FULL GPU

SATURATION

MAX GPU

UTILIZATION

DGX-1 GPU UTILIZATION (%) AI200 READ (MB/S)

DGX-1 GPU UTILIZATION (%) NFS STORAGE READ

(MB/S)

2X FASTER

Consistent DL performance

Achieves full GPU utilization

Efficient loading of data set

NFS GPU

STARVATION

MAX GPU

UTILIZATION

DDN PARALLEL ARCHITECTURE AND PROTOCOL vs NFS



34


DDN A3I SOLUTIONS PERFORMANCE ENHANCEMENTS

AI200 and AI400

Optimizations for AI

Performance improvements for

Random Read up delivers up to to

around 750K IOPs per RU for AI400

and over 22GB/s per RU

0

10

20

30

40

50

60

0

200

400

600

800

1000

1200

1400

1600

1800

4KB 8KB 16KB 32KB 64KB 128KB

Thro

uh

gpu

t (G

B/s

ec)

IOP

S (K

op

s/se

c)

IO Size(Byte)

AI200 and AI400 (Random Read IOPS and Throughput)

AI200-Throughput

AI400-Throughput

AI200-IOPS

AI400-IOPS



35


Tokyu Bancho Bldg. 8F

6-2 Yonbancho Chiyoda-ku,

Tokyo 102-0081

+81-3-3261-9101

+81-3-3261-9140

company/datadirect-networks

@ddn_limitless

[email protected]

Thank You!Keep in touch with us

ddn update 2019 - npo tsukuba oss technical support...

Documents