hpc on aws - mdx2.plm. · pdf fileaws. ut those did not satisfy hgst’s requirement....

1

Hiroshi Kobayashi, Dev./Lab. IT SystemHGST Japan, Ltd.

Jun 3, 2015

HPC on AWS

2

HPC on AWS

HPC = High Performance Computing

AWS = Amazon Web Service

3

Agenda

• HGST

• Why choose Cloud?

• Performance

• Flexibility

• What’s Next…

• Summary

4

HGST Company Profile

Founded in 2003 through the combination of the hard drive businesses of IBM, the inventor of the hard drive, and Hitachi, Ltd (“Hitachi”)

Acquired by Western Digital in 2012

Headquartered in San Jose, California

Approximately 38,000 employees worldwide

More than 4,700 active worldwide patents

Develops innovative, advanced hard disk drives, enterprise-class solid state drives, external storage solutions and services

Delivers intelligent storage devices that tightly integrate hardware and software to maximize solution performance

5

Broadening Lineup of Storage SolutionsRECENT INNOVATIONS

Solid State Storage Solutions HGST Storage Software

Ultrastar®

SN100 Series NVMe PCIe

Ultrastar®

SSD800MH.B,SSD1600MM &SSD1600MRSAS SSD

FlashMAX® III PCIe

HGST ViridentSolutions

HGSTViridentSpace

HGST10TB SMR HDD

HGST Ultrastar® He8

HDD Storage Solutionswith HelioSeal™ Technology

Active ArchivePlatform

Petabyte-scale Data CenterStorage Solutions

6

HGST Active Archive System

Our first fully integrated system with 4.7PB raw capacity per rack!

Complete scale-out object storage system for cloud data centers

4.7PBraw capacity per rack

Optimized for active archive workloads

BreakthroughTCO

BeatsWhite BoxEconomics

Highest DensityImproves Data

Center Efficiency

Lowest Power per TB with Fast Data

Access

Scales to Exabytes of Capacity

7

Market Leadership

8

Agenda

• HGST

• Why choose Cloud?

• Performance

• Flexibility

• What’s Next…

• Summary

9

Why choose Cloud?

• Background₋ A few years ago, HPC implementation project was started.

Project team investigated several cloud HPC services except for AWS. But those did not satisfy HGST’s requirement.

₋ CIO Steve Phillpott recommended AWS for HPC. He had much experience of HPC on AWS at life-science industry.

₋ Through several Proof of Concept projects, began to understand Pros/Cons of On-premise and Cloud HPC.

• Key factors are…₋ Scalability, Data transfer, Remote Visualization

₋ Commercial Application, Cost…

10

Agenda

• HGST

• Why choose Cloud?

• Performance

• Flexibility

• What’s Next…

• Summary

11

Scalability

• CD-adapco provided the benchmark data on their cluster.• C3 provide significant improvement to the scalability• C3 is 1.81x faster than CR1• Still behind to physical cluster with InfiniBand

1.81x faster

1.70x slower

※1 EN = Enhanced Networking

※2 placement group enable

※3 evaluated by elapse time

※4 only 200steps

12

Remote Visualization

• Result data is too huge to download• Transferring huge data is NOT a option• Require Remote Visualization for huge result data

Server – Client Mode

Remote Desktop Console

AWS graphic

server

Client

Users

Consume server side

GPU resource and license

Remote access

via RDC/VNC

AWS file

server

Client

Users

Consume server

side license

Consume client side

GPU resource

Not good performance…Slower responseSlower rendering

Great performance!!!Almost same performance as

local workstation with high-end graphic card

G2

13

Data Collaboration

• Transferring huge data is NOT a option• Even 48TB of d2.8xlarge may not be sufficient

for long term / huge data repository• High cost for re-computing of large scale model

ClientUsers

ClusterMaster

Computing

Nodes

Shared storage

S3 bucket

job submission

small data back to client

AWS Simple

Storage Service

(S3)

14

Performance

• Scalability₋ C3.8xlarge improved the scalability dramatically

₋ Higher scalability is better

• Remote Visualization₋ Star-CCM+ is ready

₋ Other applications are NOT ready

• Data Collaboration₋ No need to struggle with the storage capacity and durability

• AWS can support whole process of simulation works!!!

15

Agenda

• HGST

• Why choose AWS for HPC?

• Performance

• Flexibility

• What’s Next…

• Summary

16

Hybrid HPC Architecture

• Local + Cloud = Hybrid HPC environment

• AWS + Cycle Computing http://www.cyclecomputing.com/

HGST

Virtual Private Cloud

AWS

Cluster

MasterComputing

Nodes

ClientUsers

Shared

Storage

data I/Oattached

Local Cluster

S3 bucket

Auto ScaleOut / In

Fixed Capability

17

Shape Compute To Match Work To Be Done

• All Jobs Run In Parallel on AWS 1.67x Throughput Improvement

Time

Before:

Shared Cluster Computer

512 core 512core 512core

64 core

64 core

64 core

64 core

64 core

64 core

64 core

64 core

Today:

AWS EC2 CC2 Cluster

(Max Total 512 core)

512corewaiting

256 core 256 core

128 core 128 core

waiting

waiting

64 core

64 core

64 core

64 core

64 core

64 core

64 core

64 core

18

Shape Compute To Match Work To Be Done(Cont.)

19

Shape Storage To Match Work To Be Done

• No need to struggle with the storage capacity and durability!!!

ClientUsers

ClusterMaster

Computing

Nodes

Shared storage

S3 bucket

job submission

small data back to client

∞

20

Shape Cost To Match Work To Be Done

• Workload is NOT constant

• Server Reservation Discount = Reserved Instances (RI)

• Analyzing workload Utilizing RI Optimizing cost

21

Agenda

• HGST

• Why choose Cloud?

• Performance

• Flexibility

• What’s Next…

• Summary

22

What’s next for Cloud HPC…

• Computing Performance₋ More scalability, like InfiniBand

• Remote Visualization₋ Higher performance than RDC-TCP/IP

₋ PC over IP®? NICE DCV®? Star-CCM+ is ready!!!

• Commercial Application License₋ End User License Agreement (EULA)

₋ Hybrid License Server

₋ Consumption Based License Power On Demand!!!

Local

License

Server

23

Agenda

• HGST

• Why choose Cloud?

• Performance

• Flexibility

• What’s Next…

• Summary

24

Summary

• At this moment, HPC on AWS is NOT perfect₋ Scalability, Remote Visualization except for Star-CCM+

• HPC on AWS has extremely high flexibility₋ Hybrid HPC, Shape Compute/Storage/Cost To Match Work To

Be Done

• Flexibility will drive to responding to the changing business model

• Benefit of HPC on AWS should be verified with each applications based on its characteristic

• Required collaboration with application venders

25

Helping the World Harnessthe Power of Data withSmarter Storage Solutions