© 2014 ibm corporation experience ibm flashsystem™ as an application accelerator what’s new,...

© 2014 IBM Corporation

Experience IBM FlashSystem™ as an Application Acceleratorwhat’s new, model 840

Detlef Helmbrecht

European Storage Competence Center – Mainz – Germany


Agenda

Introduction to Flash Technology

New features of Model 840

New GUI and CLI

Encryption

GA Restrictions

Q&A


Performance Gap

CPU performance up 10x this last decade

Storage has grown capacity but unable to keep up in performance

Systems are now Latency & IO bound resulting in significant performance gap

From 1980 to 2010, CPU performance has grown 60% per year*

…and yet, disk performance has grown ~5% per year during that same period**

Why FlashSystem


MicroLatency: How it Affects Your Business

Time Recovered

Disk-Based

FlashSystem

CPU, Network, Memory and Bus are getting faster… so all of your applications WAIT on disk

CPU TimeI/O Time Network

Time

CPU Time

I/O Time

Network Time

Applications see time waiting, not IOPSApplications see time waiting, not IOPS


IBM Disk and FlashSystem Portfolio

Storwize V3700

Storwize V5000

Storwize V7000

EntryEntry

MidrangeMidrange

EnterpriseEnterprise

FlashSystem

DS8870

XIV

Storwize Family

FlashSystem Family

FlashSystem Solutions

Winning Means:

Knowing when, where and how to deploy Flash

Focus on Storage Solutions for Improved Data Center Economics

IBM Flash Drawers,Adapters, & SSDs


IBM Flash Offerings Strategy

• Standalone all-flash array building block• Lowest latency• Eliminate IO bottleneck or cap BW• Use application-level or SDS services

• Software Defined Storage• FlashSystem Enterprise Performance Solution• Storwize V7000 FlashSystem Edition

• FlashSystem is a smart storage shelf• Use traditional SAN-based services

IBM FlashSystem addresses…

IBM FlashSystem Solutions

IBM FlashSystem

2. Enterprise features that still maximize flash performance

1. Absolute performance for throughput, IOPS, and latency


IBM FlashSystem 840 Data Center Optimized

• 1.1M IOPS• 8 GB/s bandwidth• Multiple connectivity interfaces

- 16Gb/8Gb Fibre Channel- 40Gb QDR InfiniBand- 10Gb FCoE

• Fully redundant and hot swappable architecture:flash modules, power supplies, batteries, interfaces, fans, RAID controllers, etc

• Concurrent code load• AES 256 HW base encryption

• 2U• 625 watts• Field upgradeable, granular capacity:

4, 8, 12, 16, 20, 24, 32, 40, 48 TB options• Intuitive standardized GUI

• Low latency: 135/90 µs R/W• Purpose-built, FPGA-based design• Boosts host CPU efficiency and productivity

Macro Efficiency

MicroLatency™

Enterprise Reliability

Extreme Performance


Minimum latency

Write 90 µs

Read 135 µs

Maximum IOPS 4 KB

Read (100%, random) 1,100,000Read/write (70%/30%, random)

775,000

Write (100%, random) 600,000

Maximum bandwidth 256 KB

Read (100%, sequential) 8 GB/s

Write (100%, sequential) 4 GB/s

FlashSystem 840

Capacity Options

Flash module configuration

2 x 2 TB 4 x 2 TB 4 x 4 TB 8 x 2 TB12 x 2

TB8 x 4 TB

12 x 4 TB

Raw capacity 5 TB 11 TB 21 TB 21 TB 32 TB 42 TB 65 TB

RAID 0 usable capacity 4 TB 8 TB N/A 16 TB 24 TB 32 TB 48 TB

RAID 5 usable capacity N/A 4TB 8 TB 12 TB 20 TB 24 TB 40 TB

Performance at-a-glance

Data center optimized to deliver extreme performance, flexible capacity and total system protection


Solving a Problem

No application runs better with storage than it does without

storage

Storage Performance means returning I/O to App/User as quick as possible

0 100,000 200,000 300,000 400,000 500,000 IOPS

FlashSystem 840, 70/30 4K Random Reads

FlashSystem 840

Ideal

Other

ll

ll

Latency difference means immediate, higher application efficiency and user/app productivity

Sustainable latency means application scalability, cost-controlled growth, investment protection & value generation

Premise: The best storage is the one that sucks the

least


IBM FlashSystem 840

• IMPROVED RAS features: ALL hot swappable components: - Flash modules, power supplies, batteries, interfaces, fans

• Concurrent code load• Encryption

• 2U - Purpose-built/FPGA – eMLC 24nm flash chips• Granular Capacities: 2TB and 4TB modules

- 4, 8, 12, 16, 20, 24, 32, 40, 48• NEW GUI

• Low latency - 135/90 µs R/W parallel design• CPU efficiency and productivity: 625 watts• 1.1M IOPS• 8 GB/s bandwidth• Multiple connectivity interfaces


1 or 3 year warranty 24x7 – 4hrs – WW support

9843 Model AE1 – 3 year9840 Model AE1 – 1 year

Flash Modules (not SSDs)


IBM FlashSystem 840 (Detail)

2TB Module 4TB Module

Chip Type eMLC – 24nm – 64GiB# of Chips per Module

40 80

Total Raw Bytes 2,748,779,069,440 5,497,558,138,880 Total Presented Bytes

2,061,581,156,352 4,123,162,312,704

Raw Maximum Capacity

32.99TB 30TiB 65.97TB 60TiB

Usable RAID0 Capacity

24.74TB 22.5TiB 49.48TB 45TiB

Usable RAID5 Capacity

20.62TB 18.75TiB 41.23TB 37.50Ti

B

Capacity DetailsCapacity Details


IBM FlashSystem 840

Capacity & Performance Configuration Guidelines

•Cannot Mix 2TB with 4TB modules•4TB modules provide HIGHER capacity and scalability options•Module type has little effect on Read performance•Write performance continues to be based primarily on number of chips & flash controllers

• More chips and flash controllers = more Write Bandwidth• Only significant in “write-heavy” environments• Only significant in “sustained” write environments

•Capacity upgrades are not yet available online, unless virtualized


IBM FlashSystem 840 (Compared to 820)

• General Observations• 4x more front-end interfaces (FC) – 128Gb/s vs 32 Gb/s interface bandwidth• 2x flash chip density (64GiB 24nm) = double capacity per box • Similar FPGA design to FlashSystem 820: 12 modules, 48 flash controllers, 960

chips, etc.• Improved RAS features – concurrent code load, concurrent maintenance on

everything

• Performance Baseline Premises• 2x read performance – 1.1M IOPS• More linear & consistent write performance – 4GB/s• Latency continues to be WAY AHEAD OF THE COMPETITION

• Consistent low read latency – 135µs• Low, scalable write performance – 90µs – 4GB/s



• More about latency: fixing the Blurry Vision… • 100/25µs or 135/90µs?

These are SINGLE I/O latency measurements In a SAN environment, it means queue_depth=1 While this value represents the overall benefit of response time Applications with this behavior are very uncommon

0 100,000 200,000 300,000 400,000 500,000 600,000IOPS

50/50 4K Random Workload

FlashSystem 840 Ideal

“Blurry Vision” point

ll

Consistent low latency & scalability to heavy workloads



820 840Capacity Points 2 12

Field upgradeable Capacity ✗ ✓20TB per RU ✓ ✓

RAID-5 or RAID-0 ✓ ✓Encryption ✗ ✓

<400 watts per RU ✓ ✓Modern, Intuitive GUI ✗ ✓

Concurrent Code Load (CCL) ✗ ✓Non-disruptive maintenance,

hot swappable ✗ ✓Plug-in Serviceability & tool-

less access ✗ ✓

16 Gbps Fibre Chanel8 Gbps Fibre Chanel

40 Gbps QDR InfiniBand10 Gbps FCoE

✗✓✓✗

✓✓✓✓

Successful Design Factors•FPGA, Flash-Optimized design•MicroLatency™•VSRTM and Two-Dimensional (2D) Flash RAID •eMLC NAND technology•RAID-5 or RAID-0

Improved . . . Performance ServiceabilityManagementConnectivity



IBM FlashSystem 820 IBM FlashSystem 84010, 12, 20, 24 Capacities (TB) 4, 8, 12, 16, 20, 24, 32, 40, 48

No Field upgradeable Capacity Yes1U Form Factor 2U

33TB Maximum Raw Capacity 66TB33TB Raw Capacity per 1U Rack 33TB525K Performance (IOPS) 1.1M

5 GB/s Max Read Bandwidth 8 GB/s110us / 25us Latency (read/write) 135us / 90us

8/4Gb FC40Gb QDR IB

Available Interfaces16/8Gb FC10Gb FCoE

40Gb QDR IB2 # of Interfaces 4

eMLC Chip Type eMLCYes Chip RAID YesYes System RAID YesNo Encryption Yes

300W Power Consumption 625W (estimate)

HTML 3.0 c.2001 GUI interface ManagementHTML 5.0 c.2011 GUI Modern, Intuitive,

easy-to-use, reduced learning curve

Hot-swap flash modules and power supplies

Serviceability

Front/Back accessible Hot-swap Flash Modules, Power Supplies, Batteries,

Fans, Controllers w/ interface cards and non-disruptive maintenance and

firmware updates (concurrent code load)


IBM FlashSystem 840

The The GUIGUI that revolutionized storage, now available in that revolutionized storage, now available in FlashSystem 840FlashSystem 840


IBM FlashSystem 840

Encryption• Optional, licensable feature• AES-XTS 256-bit data-at-

rest• Local management• Module-level

implementation (Self-Encrypted Flash Controller)

• Zero impact on performance

• IMPROVED RAS features: ALL hot swappable components: - Flash modules, power supplies, batteries, interfaces, fans

• Concurrent code load• Encryption

• 2U - Purpose-built/FPGA – eMLC 24nm flash chips• Granular Capacities: 2TB and 4TB modules

- 4, 8, 12, 16, 20, 24, 32, 40, 48• NEW GUI

• Low latency - 135/90 µs R/W parallel design• CPU efficiency and productivity: 625 watts• 1.1M IOPS• 8 GB/s bandwidth• Multiple connectivity interfaces



Flash Modules (12)

RAID Controllers (2)

Battery Packs (2)

Power Supplies (2)

Fan Packs (4)

Interface Modules (4)

Management Modules (2)

Canisters (2)

IBM FlashSystem 840: Hardware View

Improved RAS featuresFront/back accessible hot-swap flash modules, power supplies, batteries, fans,controller canisters w/ interface cardsNon-disruptive maintenance and firmware updates (concurrent code load)


Redundant Power Supplies

Redundant Fans

12 Flash Modules(10+1+1)

1U Chassis

N+1 batteries

Redundant Management Control Processors

Redundant RAID controllers

Two Dual-Ported 8 Gb/s FC or40 Gb/s QDR IB Interfaces

IBM FlashSystem 720/820: Hardware View


IBM FlashSystem 840: Concurrent Code Load (CCL) and Maintenance

Fully redundant

no single point of failure

Data path is entirely independent of the control path

Canister-1 Canister-1 Canister-2 Canister-2


IBM FlashSystem 840: Logical View

• Hardware-only data path with extremely LOW latency– Custom FPGA-based data movement decreases latency vs. software

• Distributed out-of-data-path CPU processing• High Performance (IOPS), High Bandwidth

CPUs (18)

Interface Controller

Management Module

RAID Controller

Flash Modules (12)

FPGA


IBM FlashSystem 840: Flash Module, Logical View

Gateway Interface FPGAI/O interface and Direct Memory Access path

Control PPC and DRAM Out of Data path operationsGarbage collection, Error Handling, System HealthWear Leveling, Statistics, etc.

Flash Chips20 per Flash Controller40 or 80 per Module

XOR parityData

Primary Board (2TB) Secondary Board (2TB) - Optional

FPGA

FPGA

FPGA

FPGA

FPGA

FPGAFPGA Flash Controller - FPGA

2 or 4 per ModuleData path, Hardware I/O logic Look up Tables and Write BufferEach controls 20 flash Chips

Scalability and Parallelism: 7,680 parallel flash operations and 1,600 DMA parallel operations

for a second:

In most hybrid or SSD-based solutions, most of these actions are done in SW-heavy layers that ultimately impact LATENCY.

for a second:

In most hybrid or SSD-based solutions, most of these actions are done in SW-heavy layers that ultimately impact LATENCY.


IBM FlashSystem 840: Flash Controller Design

Flash Controller - FPGAData path, hardware I/O logic Look up tables and write bufferControls 20 flash chips

Lookup TablesDRAM

Control PPC and DRAM Out of data path operationsGarbage collection, error handling, system healthWear leveling, statistics, etc.

DRAM Write Buffer

Gateway Interface FPGAI/O and direct memory access

NAND Flash Memory

FPGA

FPGA


Superior Durability:Using the Best Flash

10X

3X

Superior Protection: Beyond Disk RAID Chip/Plane/Die level protection

Self-Recovering Flash Modules

Avoid system rebuilds

Protection Within And Across Flash Modules

Variable Stripe SizesRead Disturb Mitigation

Automatic Read SweeperHigh-Speed Clock

Recovery

Advanced Engineering = Less Maintenance

IBM FlashSystem 840: Reliability Ingredients

SLC Market demand decreasing. eMLC data protection techniques

delivering more wear life than what market demands

eMLC delivers best Price/Performance


IBM FlashSystem 840 Built-in Encryption

Protecting business’s most valuable asset

Industry standard AES-XTS required by most leading compliance regulations such as HIPAA and FIPS

Local key management

All performance measurements published at GA and later include the affect of encryption, even if it is not enabled – we encrypt with a default key


Basic encryption concepts:

Symmetric vs. Asymmetric cryptography

Symmetric (e.g. Triple-DES, AES): uses a single secret key for both encryption and decryption

– Allows for very fast and efficient encryption & decryption– Ideal for high speed bulk encryption– Having a single “secret” key creates a key distribution problem

Asymmetric (e.g. D-H, RSA, ECC): uses a public/private key pair– uses one key to wrap, or sign– uses the other key to unwrap, or check signature– the concepts that underpin the Public Key Infrastructure (PKI)

Advanced Encryption Standard (AES) – Symmetric block encryption specification established by US Government’s NIST in 2001,

encrypts 16-byte blocks of data.– XTS-AES-256

• XTS effectively encrypts every block of data differently• 256 is the length, in bits


The Keys

FlashController

FlashController

FlashController

FlashController

FlashMemoryFlash

MemoryFlash

MemoryFlash

Memory

FlashMemoryFlash

MemoryFlash

MemoryFlash

Memory……

……

Flash ModuleFlash ModuleFlashVPDFlashVPD

FEKFEK

FWKFWK

Flash Wrapped Key (FWK), the wrapped key

Flash Encryption Key (FEK), the key to be wrapped

Flash Access Key (FAK), the wrapping key

Flash Access KeyFlash Access Key


1. Local Key Management

2. External (& possibly remote) Key Management (e.g. TKLM)

Key Management

Local Remote

Key Serving Human Application

Key Replication Human Application

Susceptible to human error

On-going basis At initial configuration

FlashSystem 840 Yes No

Key Management – Two of the Types:


FlashSystem 840 Streamline Management

Modern design

Icon-based navigation, informative graphics, and visualizations

Reduce complexity

Improve efficiency and productivity

Unified, cross-platform design


FlashSystem 840 CLI


Easily integrated into any environment

• 8Gb or 16Gb Fibre Channel

• 10Gb Fibre Channel over Ethernet (FCoE)

• 40Gb InfiniBand


IBM FlashSystem SolutionsFlashSystem Enterprise Performance Solution & Storwize V7000 FlashSystem Edition

FlashSystem Enterprise

Performance Solution

Storwize V7000 FlashSystem

Edition


FlashSystem Enterprise

Performance Solution

Storwize V7000 FlashSystem

Edition

IBM FlashSystem SolutionsCombine Function and Extreme Performance

• Extreme performance of IBM FlashSystem with IBM MicroLatency™

• Advanced storage functionality with• Thin Provisioning – allocate storage “just in time”

• Easy Tier – storage efficiency

• FlashCopy – point in time copies

• Mirroring/Copy Services – data replication and protection

• Real-Time Compression – up to 5X more data in the same physical space

• External Virtualization

• Easy to cost effectively deploy quickly and realize immediate results


FlashSystem V840

Smarter storage for the optimized data center

• Thin provisioning• Local and remote replication• Data migration• Virtualizing 3rd party arrays • Automated tiering• Real-time Compression

• Fully redundant controllers • Hot swappable architecture:

- Flash modules, power supplies, batteries, interfaces, fans

• Concurrent code load• AES 256 HW base encryption• Technical Advisor included

• 6U form factor• Two controller enclosures• One storage enclosure• Up to 40 TB usable (RAID protected)

• VMware certified• Application aware snap shots

Form factor

Software capabilities

RAS

Application integration


IBM FlashSystem Solutions are Efficient

Thinprovisioning

Value: Purchase only the storage you need when you need it

Dynamicgrowth

Real-timecompression

Value: Store more data with less disk

Value: Optimized performance at lower overall cost

Thin Provisioning

More productive use of available storage

Across all supported host platformsWithout thin provisioning, pre-allocated

space is reserved whether the application uses it or not.

With thin provisioning, applications can grow dynamically, but only consume space

they are actually using.

Dynamicgrowth

Busiest data extents are identified and automatically relocated to highest performing Solid-state Disks

Remaining data extents can take advantage of higher capacity, price optimized disksHot-spots due to poor data layout. Optimized performance and throughput.

AutomaticRelocation

SSDs HDDs SSDs HDDs

IBM Easy Tier SSD Management

Mirror data off-site

Synchronously over Metro distances.

Asynchronously over Global distances.

Application-level consistency groups.Network

IBM Easy Tier® flash storage management


IBM FlashSystem Solutions are Resilient

Practice disaster recovery techniques

Value: Train staff and validate business continuity plans without disrupting applications

Application-awarecopies

Value: Virtually eliminate application downtime for backup

High availabilityconfigurations

Thin Provisioning

More productive use of available storage

Across all supported host platformsWithout thin provisioning, pre-allocated

space is reserved whether the application uses it or not.

With thin provisioning, applications can grow dynamically, but only consume space

they are actually using.

Dynamicgrowth

Busiest data extents are identified and automatically relocated to highest performing Solid-state Disks

Remaining data extents can take advantage of higher capacity, price optimized disksHot-spots due to poor data layout. Optimized performance and throughput.

AutomaticRelocation

SSDs HDDs SSDs HDDs

IBM Easy Tier SSD Management

Mirror data off-site

Synchronously over Metro distances.

Asynchronously over Global distances.

Application-level consistency groups.Network

Value: Enable near-continuous data availability


IBM FlashSystem 840 Solutions FlexibilityFlashSystem Enterprise Performance Solution & Storwize V7000 FlashSystem Edition

Base Configuration Scale-up Scale-up and Scale-out


Help clients understand how to deploy: Flexibility

IBM FlashSystemAbsolute Performance & Enterprise Features

Implementation choices:• Absolute Performance• Enterprise Features • Mix of Performance and Features

Flexible services and product integrations: Oracle RAC, Atlantis ILIO, VMware, etc.


IBM FlashSystem Solution Deployment Architectures

Easy Tier• Completely transparent• Accelerates all workloads

Preferred Read• V7000 vdisk copy• Application-based• OS-based

Manual Data Placement• OS-based/LVM/virtualized access• Partial apps: hot tables, indexes, temp, undo, logs• Complete apps: whole databases• The first step on the path toward “all flash” data centers

© 2014 IBM Corporation46 Global Offering Development46 Global Offering Development

The STG Technical Advisor (TA) enhances end-to-end support for client’s complex IT solutions. TA is an integrated approach for proactive, coordinated support to allow customers to maximize IT availability at an effective cost

The TA offering is built around three value propositions:– Proactive approach to ensure high availability for vital IT services – Client Advocate that manages problem resolution through the entire support process– Integrated Support for both hardware and software

History:– Global Program established successfully in 2008 for XIV storage array– Technical Advisors are located worldwide in close proximity to clients– TA’s are an integral part of the leading edge IBM Support Structure– Technical Advisor support is now being extended to include XIV, ProtecTIER, SoNAS

and FlashSystem 840 & V840

Technical Advisor Overview (FlashSystem 840 & V840)Technical Advisor Overview (FlashSystem 840 & V840)


New Features (One Page)

Up to twelve 4 TB (or 2 TB) flash modules in a small 2U form factor.

Up to 1,100,000 IOPS, up to 8GB/s bandwidth and latency as low as 90µs.

Hot swappable components replaced via front and rear of the unit– no need to disrupt the system in the rack.

New IBM SAN Volume Controller CLI and GUI for common management look and feel.

Connectivity– eight 16Gb/s Fibre-channel ports– sixteen 8Gb/s Fibre-channel ports– sixteen 10 Gb/s Fibre-channel over Ethernet ports– eight 40Gb/s InfiniBand ports.

Encryption for data at rest.

Concurrent code loads providing no client application downtime.


Open points solved with SP1

Components not yet concurrently maintainable

– batteries, – RAID canisters,– interface cards.

– FCoE– Infiniband

Revised support for IBM FlashSystem 840 concurrent code load– http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&infotype=an&appname=iSource&supplier=897&letternum=ENUS114-024


Q & A


Resources

IBM Flash Storage:http://www-03.ibm.com/systems/storage/flash/index.html

Support Matrix (SSIC):http://www-03.ibm.com/systems/support/storage/ssic/interoperability.wss

Redbooks FlashSystem:http://www.redbooks.ibm.com/redbooks.nsf/searchsite?SearchView&query=FlashSystem

FlashSystem 840 Tour:http://www-03.ibm.com/systems/data/flash/storage/flash/tour/840/index.html


Disclaimers

Copyright © 2013 by International Business Machines Corporation. This publication is provided “AS IS.” IBM product information is subject to change without notice. No part of this document may be reproduced or transmitted in any form without written permission from IBM Corporation. Product data has been

reviewed for accuracy as of the date of initial publication. Product data is subject to change without notice. This document could include technical inaccuracies or typographical errors. IBM may make changes, improvements or alterations to the products,

programs and services described in this document, including termination of such products, programs and services, at any time and without notice. Any statements regarding IBM’s future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. The information contained in this document is current as of the initial date of publication only and is subject to change without notice. IBM shall have no responsibility to update such information.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products.

IBM makes no representations or warranties, expressed or implied, regarding non-IBM products and services, including those designated as ServerProven.

IBM is not responsible for the performance or interoperability of any non-IBM products discussed herein. Performance data for IBM and non-IBM products and services contained in this document was derived under specific operating and environmental conditions. The actual results obtained by any party implementing such products or services will depend on a large number of factors specific to such party’s operating environment and may vary significantly. IBM makes no representation that these results can be expected or obtained in any implementation of any such products or services.


Disclaimers (continued)

MB, GB and TB equal 1,000,000, 1,000,000,000 and 1,000,000,000,000 bytes, respectively, where referring to storage capacity. Actual storage capacity will vary based upon many factors and may be less than stated. Some numbers given for storage capacities give capacity in native mode followed by capacity using data compression technology.

THE INFORMATION IN THIS DOCUMENT IS PROVIDED “AS-IS” WITHOUT ANY WARRANTY, EITHER EXPRESSED OR IMPLIED. IBM EXPRESSLY DISCLAIMS ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NONINFRINGEMENT. IBM products are warranted according to the terms and conditions of the agreements (e.g., IBM Customer Agreement, Statement of Limited Warranty, International Program License Agreement, etc.) under which they are provided.

References in this document to IBM products, programs or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Consult your local IBM business contact for information on the product or services available in your area. Any reference to an IBM program or product in this document is not intended to state or imply that only that program may be used. Any functionally equivalent program or product that does not infringe IBM’s intellectual property rights may be used instead. It is the user’s responsibility to evaluate and verify the operation of any non-IBM product, program or service.

IBM's customer is responsible for ensuring its own compliance with legal requirements. It is the customer's sole responsibility to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer's business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.

The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents or copyrights. Inquiries regarding patent or copyright licenses should be made, in writing, to:

IBM Director of LicensingIBM CorporationNorth Castle DriveArmonk, NY 10504-1785U.S.A.

© 2014 ibm corporation experience ibm flashsystem™ as an application accelerator what’s new,...

Documents