intel® solutions for lustre* software sc16 presentation · tests document performance of...

19

Upload: others

Post on 04-Jul-2020

6 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,
Page 2: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,

Craig Hendren Global OEM Alliances Manager for Lustre and HPC OrchestratorHigh Performance Data Division, Intel Corporation

Page 3: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,

Legal informationTests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance.

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at http://www.intel.com/content/www/us/en/software/intel-solutions-for-lustre-software.html.

You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.

The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.

Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.

3D Xpoint, Intel, the Intel logo, Intel Core, Intel Xeon Phi, Optane and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries.

* Other names and brands may be claimed as the property of others.

© 2016 Intel Corporation

3

Page 4: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,

December 2015: Intel’s Analysis of Top 100 Systems (top100.org)

9 of Top10 Sites

73% of Top100

Most Adopted PFS

Most Scalable PFS

Open Source GPL v2

Commercial Packaging

Vibrant Community

4

73%

16%4% 7%

Lustre GPFS other unknown

Page 5: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,

* Other names and brands may be claimed as the property of others.

1999 2011 2013

20152001

2003

2007

2009

2010 2012

2016

1.0

1.6

1.8

2.0

2.1

2.3

2.5

2.7

2.8

5

Page 6: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,

1 Source: Chris Morrone, Lead of OpenSFS Lustre Working Group, April 2016

Intel65%

8%

6%

6%

3%3%

2%2%

1%2%

Intel ORNL* Seagate* Cray* DDN*

Atos* LLNL* CEA* IU Other

Intel65%

18%

4%2%

2%2%1%1%1%

Intel ORNL Cray Atos Seagate

DDN IU CEA Other

Commit per Organization Lines of codes per organization

6

Page 7: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,

Intel® Scalable System FrameworkA Holistic Solution for All HPC Needs

Small Clusters Through Supercomputers

Compute and Data-Centric Computing

Standards-Based Programmability

On-Premise and Cloud-Based

Intel® Xeon® Processors

Intel® Xeon Phi™ Processors

Intel® FPGAs and Server Solutions

Intel® Solutions for Lustre*

Intel® Optane™ Technology

3D XPoint™ Technology

Intel® SSDs

Intel® Omni-Path Architecture

Intel® Silicon Photonics

Intel® Ethernet

Intel® HPC Orchestrator

Intel® Software Tools

Intel® Cluster Ready Program

Intel Supported SDVis

ComputeFabric

Memory / Storage

Software

7

Page 8: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,

* Other names and brands may be claimed as the property of others.

Bioscience Government research and defense Large-scale manufacturingMechanical, computer-aided design & computer-aided engineering systems

Genomic data analysis, modeling and simulations

Weather and climate Energy FinanceFraud detection, Monte Carlo simulations,

risk management analysisHighly complex CGI rendering Seismic processing, reservoir modeling /

characterization, sensor data analysis

8

Government funded research. Surveillance, Signal Processing, encryption etc.

Page 9: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,

Intel® Scalable System Framework for HPC

* Other names and brands may be claimed as the property of others.

Intel® FOUNDATION Edition for Lustre* software

Delivers the latest functions and features, fully supported by Intel

Ideal for organizations that prefer to design and deploy their own open

source configurations

Intel® ENTERPRISE Edition for Lustre* software

Maximum performance with minimal complexity and cost for multi-

petabyte file system. Management with Intel® Manager for Lustre*

software

Intel® CLOUD Edition for Lustre* software

Cost-effective access to parallel storage on Amazon Web Services*

(AWS) and Microsoft Azure* to boost cloud-computing

9

Page 10: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,

10

Page 11: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,

Proven Experience of #1 Lustre Provider

Industry’s Fastest Storage – SFA

DDN + Intel – Unique Features, Support

Versatile – Embedded / External / Custom

End2End Data Management

11

DDN

Page 12: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,

Select DDN EXAScaler & Lustre Customers

12

Page 13: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,

13

Page 14: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,

* Other names and brands may be claimed as the property of others.

14

Page 15: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,

* Other names and brands may be claimed as the property of others.

Software Core for Intel® EE for Lustre* software. Bulk IO

Performance, Subdirectory Mounts

Management OpenZFSstorage. Jobstats Monitoring. LFSCK, Dynamic Lnet; Intel®

Manager for Lustre*

Latest updates and security patches to customers.

Kerberos* Network Authentication and

Encryption. SELinux* Lustre Client Support

Hadoop* Adapter for Lustre and HPC Adapter for MapReduce.

Lustre 2.7

Intel® Omni-Path Architecture support.

Metadata Performance Improvements. Snapshots

15

New in v3.1

Page 16: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,

Product placement not representative of final launch date within the specified quarter* Other names and brands may be claimed as the property of others

Q4 ’16 Q1 ’17 Q2’17 Q3 ’17 Q4 ’17

EnterpriseHPC, commercial technical computing, and analytics Client m’data perf.

OpenZFS perf.SnapshotsSELinux, KerberosOPA supportRHEL 7OpenZFS 0.6.5.3

EE 3.0

Lustre* 2.7.x

DNE 2Client m’data perf.LFSCK perf.SELinux, Kerberos

FE 2.8

Lustre 2.8.x

CloudHPC, commercial technical computing and analytics delivered on cloud services

FoundationCommunity Lustresoftware coupled with Intel support

Expanded OpenZFS support in IMLIML graphical UI enhancementsBulk IO Performance (16MB IO)OpenZFS metadata performance Subdirectory mounts

Enterprise Edition 3.1

Lustre 2.7.x

DNE 2Multi-rail LNETIML asset mgmt. ZED updates

EE 4.0

Lustre <TBA>

RHEL 7 Server SupportLustre release update

Cloud Edition 1.3

Lustre 2.8.x

SW updatesNew instances

CE 1.5

Lustre <TBA>

Updated Lustre and base OS buildsNew instance support

Cloud Edition 1.4

Lustre 2.9.x

UID/GID MappingShared key cryptoLarge Block IOSubdirectory mounts

Foundation Edition 2.9

Lustre 2.9.x

OpenZFS SnapshotsMulti-rail LNETProject quotas

Foundation Edition 2.10

Lustre 2.10.x

Shipping

Development

Planning

16

Page 17: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,

Intel Manager for Lustre

* Other names and brands may be claimed as the property of others.

ManagementNetwork

High Performance Data Network(Infiniband*, 10GbE)

MetadataServers(1-10s)

Object Storage Servers

(10s-1000s)

Lustre Clients (1 – 100,000+)

Object StorageTargets (OSTs)

Object StorageTargets (OSTs)MetadataTarget (MDT)

ManagementTarget (MGT)

Native Lustre* Client for Intel® Xeon Phi™ processorIntel® Omni-Path Support

Robin HoodOpenZFS, RAIDz

Hadoop* Adapters

17

HSM

Page 18: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,

* Other names and brands may be claimed as the property of others.

Community LustreSystem integrators selling systems

based on community Lustre* do not receive maintenance releases from Intel

Reliant on open source community for support

Patches, when available, are supplied as source code only

Integration and testing left to SI or end user to verify

Deploying solutions based on community edition carries risk for end user

18

Intel support globally—24/7/365

Regular cadence of maintenance and patch releases

Scheduled proven features and value-add major releases

Enterprise Edition is bundled with features (IML, HSM, DSS, Hadoop*, ZFS, etc.)

Optimizations for Intel® architecture –Intel® Scalable System Framework

Online training and testing for end users, channel partners and OEM partners

www.intel.com/lustre-training & www.intel.com/lustre-reseller

Page 19: Intel® Solutions for Lustre* Software SC16 Presentation · Tests document performance of components on a particular test, in specific systems. Differences in hardware, software,