high performance computing: state of the industry

IMEX©2000-2005 IMEX Research

IMEXRESEARCH.COM

High Performance Computing

Anil Vasudeva Principal Analyst & President [email protected]

State of the Industry


•• HPC HPC -- Markets Drivers & Industry DynamicsMarkets Drivers & Industry Dynamics

•• State of HPC TechnologiesState of HPC TechnologiesClustering, Interconnects, Storage, Blade ServersClustering, Interconnects, Storage, Blade Servers

•• Industry Issues/Technology HurdlesIndustry Issues/Technology HurdlesStandards, Management, Fabrics, Thermals, IP AccelerationStandards, Management, Fabrics, Thermals, IP Acceleration

•• Futures Trends Futures Trends -- Computing & GridsComputing & Grids

•• Holy GrailsHoly GrailsVolume Driven Economics, IP Evolution, TF/DesktopVolume Driven Economics, IP Evolution, TF/DesktopHPC in our Daily Lives (Academia > Hollywood > Wall Street)HPC in our Daily Lives (Academia > Hollywood > Wall Street)


Server Architectures: Competitive Technologies

Scalable Perf. vs. Availability

Scal

able

Per

form

ance

AvailabilityUniP

MPP

FTSMP

Clustering

N-way SMP

i

46

83216

Perf

orm

ance

-tp

c

Number of Processors

w/8 way

Multinode Clusters

w/4 way

2

Performance: SMP vs. Clustering

Clustering

SerialMF

SMP

MPP

Performance

Pric

e

Price/PerformanceSerial vs.Parallel Computing


Market Segments by Applications

*IOs per sesond for a required response time ( ms)

(RAID - 0, 3)

500100MB/sec101 505

1K

10 K

100

10

1

eCommerceeCommerceTransaction Transaction ProcessingProcessing

OLTPOLTPOLTP

Data Warehousing

Visual DB

DSS(RAID - 1, 5, 6)

IOPs

(L

aten

cy)

StreamingStreamingStreamingAudioAudio

VideoVideo

Scientific ComputingScientific Computing

ImagingImaging

HPCHPCHPC

TPC

HPCHPC


HPC – 2 decades of rapid progress

1991 1998 2005 2010System Cray Y-MP C916 Sun HPC10000 Shuttle@newegg Blade

Architecture 16x Vector, 4 GB, Bus

24x 333 MHz, Ultra Sparc II, 24 GB,S-

Bus

4x 2.6 GHz, x64 Bit, 4

GB, GbE4x 10GHz, x64 bit,

16GB, 10GbE

OS UNICOS Solaris 2.5.1Win Server 2003

SP2 LongHorn/LinuxGflops ~10 ~10 ~10 ~100Price $40 million $1 Million <$4000 ~$1000Price Reduction 1 1/40 1/10,000 1/400,000

Target Audience Government National Labs Large Enterprises

Small>Large Businesses Every Professional

Applications Classified, Climate, Physics

Manufacturing, Energy,Finance

Bioinformatics, Materials Sciences

Architects,Wall Street, Hollywood


The rapid rise of Clusters in HPC

Rise of Cluster Computing

0

50

100

150

200

250

300

350

1999

e

2000

2001

2002

2003

2004

e

# C

lust

ered

Com

pute

rs in

Top

500

.org


HPC Market Drivers

• Availability of industry standard HW making HPC mainstream

•• HSV Servers @$200, Storage @25c/GB, AddHSV Servers @$200, Storage @25c/GB, Add--On GbE NICS at $10, GbE Switches On GbE NICS at $10, GbE Switches at $2at $2--5/port5/port

•• Migration to Embedded ASICs on Blades Migration to Embedded ASICs on Blades

• Rapid procurement, installation & System Integration simplifying acquisition

• Cluster ready apps accelerating market growth•• Engineering (LSTC, Engineering (LSTC, AnsysAnsys..)..)

•• Bioinformatics (Blast, Gaussian..)Bioinformatics (Blast, Gaussian..)

•• Finance (Finance (MatlabMatlab, Excel, Excel……))

•• Oil & Gas (Eclipse, Oil & Gas (Eclipse, PromagicPromagic..)..)

•• Government (Government (UexploreUexplore, NOAA , NOAA HysplitHysplit..)..)


HPC Servers Rev - 15% of Total Server Market

$-

$1,000

$2,000

$3,000

$4,000

$5,000

$6,000

$7,000

$8,000

$9,000

2003 2004 2005 2006 2007 2008

Rev

$M

Departmental

Divisional

Enterprise

Capability

WW HPC Rev ($M)


Market Leaders

HPC Market Leaders 2004


Rise of Industry Standard Architecture in HPC

X86 Architecture (Intel & AMD) using Linux and Microsoft based clusters dominate in

High Perrformance Computing Architecture.

TeraFlop HPC Using Off-the-Shelf ModulesIntel servers with PCI Express, Infiniband HBAs & Switches, SATA/SAS HDD orAMD Opteron Servers, GbE HBA/Switches, SATA


Factors Affecting Performance

• Applications Configurations affect Linpack significantly(3 metrics have the most affect)•• Problem SizeProblem Size

•• Size of BlocksSize of Blocks

•• TopologyTopology

• Tightly Coupled MPI Applications•• Very sensitive to network performance characteristics Very sensitive to network performance characteristics

-- internodalinternodal communications delaycommunications delay

-- OS Network Stack is a significant factorOS Network Stack is a significant factor

•• Very sensitive to mismatched node performanceVery sensitive to mismatched node performance

-- Random OS activities can add Random OS activities can add msecmsec delays to delays to

usecusec type communication line delaystype communication line delays


HPC Architecture & Challenges

HPC Communications Stack

Architecture Goals & Criteria

Application

Memory

OS

Upper Layers

I/O interconnect

Data

Data

Network Interface(w or w/o TOE)

Chipset

CPU

TCP/IP (or Offload I/F)

Entire Stack Needs Attention

Challenge is to move data from application to the network and back

with minimum latency

Goals• Maximize compute time

vs. messaging time

• High constant cross- sectional throughput

• Reduce or hide latency

Metrics• Bandwidth

• Latency

• System Interface

• SW Stack


Control

RDMA Adapter

Application Application

Application Memory

Application Memory

CPU

Kernel Memory

CPU

Kernel Memory

RDMA Adapter

Network

Data Data

Con

trol

Traditional Adapter

Application

Host

Socket

TCP

IP

Ethernet

Host

RDMA

TCP

IP

Ethernet

Host CPU RDMA

Adapter


Interconnect Price/Performance Hierarchy

Microsoft Chimney Software

1 GbE

10 GbE w/ TOE

10 GbE w/ RDMA, TOE

InfiniBand

Myrinet

Quadrics

Market Size 2004-07

High MB/s, Low Latency, Price/Port


State of Interconnect Fabrics

Bandwidth/link

Late

ncy/

hop

EHERNET - LAN/SAN/MAN/WAN

IBSGIMyrinet

Servernet

ATM

GbEFE

IBM/SP

T3E

MPP

Dolphin

QuadricsMellanox

FJ/Synfinity

10GbEw/RDMA

GbEw/TOE

GbEw/RDMA

SMP/NUMA


Interconnects: Industry Progress


InfiniBand Clustering•• InfiniBand and Industry Standard ServersInfiniBand and Industry Standard Servers

–– Displacing Mainframe Computers in the Data Center Displacing Mainframe Computers in the Data Center with Three Standard Hardware Componentswith Three Standard Hardware Components

–– Powerful Low Cost Combo Out Performs SMPPowerful Low Cost Combo Out Performs SMP12-Nodes (24-CPUs) Up to 72 GFlops of

CPU Power; 24+ GB Memory

Industry Standard Server

InfiniBand Switch

InfiniBand HCA

=>

Enterprise Availability:With Fabric Failover


InfiniBand rears up its head

• Low Cost IB 10G Silicon & Adapters

–– Very Low Cost 10G Host AdaptersVery Low Cost 10G Host Adapters•• Host Channel Adapter ~ $69 in volumeHost Channel Adapter ~ $69 in volume•• Switch Si at less than $30 a port in volumeSwitch Si at less than $30 a port in volume

• Compelling price/performance • InfiniBand vs. 10GE Advantages in

the Short Term–– 15X 15X –– 20X price advantage 20X price advantage vsvs 10GE10GE–– 3X 3X –– 10X latency advantage10X latency advantage–– 10X 10X –– 20X CPU utilization advantage20X CPU utilization advantage–– 3X3X--5X throughput advantage5X throughput advantage

• Adoption by most majors• Products

–– InfiniBand to IPInfiniBand to IP–– InfiniBand to FCInfiniBand to FC–– Advanced ManagementAdvanced Management

*


Ethernet - TCP + iSCSI Offload Engines

EthernetIP

TCPiSCSI

SCSI Device DriverVolume Manager

File System

Typical 1GbE NIC

EthernetIP

TCPiSCSI

SCSI Device Driver

Volume ManagerFile System

EthernetIP

TCPiSCSI

SCSI Device DriverVolume Manager

File System

Hos

tSW

Ada

pter

Host CPU

HBA Hardware

HBA Firmware


10 GigEthernet

40 GigEthernet

1 GigEthernet

Grids

Data Center LAN

SAN Storage

NAS Storage

Clusters

Blade Chassis Network

SMP Backplanes

<1 Gigabit 1 Gigabit 10 Gigabit >10 Gigabit

SunFire Link, IBM Federation

InfiniBand

Interconnects: Applicability

100 MbitEthernet

Myrinet

Quadrics

PCI Express & Advanced Switching

Fibre ChannelRDMA NICs with iSCSI

RDMA NICs


InfiniBand Adopters

• Major providers of InfiniBand Solutions

• Enterprise Class Products–– InfiniBand to IPInfiniBand to IP–– InfiniBand to FCInfiniBand to FC–– Advanced Mgmt Advanced Mgmt

• Platform support includes–– Intel (IA32 + 64EMT, Itanium), Intel (IA32 + 64EMT, Itanium), AMD Opteron, Power PC, SparcAMD Opteron, Power PC, Sparc

• Operating Systems–– Linux, Windows, AIX, HPUX, Linux, Windows, AIX, HPUX,

Solaris, MAC OS X, VxWorksSolaris, MAC OS X, VxWorks


HPC Interconnect – Leaders

Myrinet for the highlylatency sensitive applications and GbE for the majority applicationsDominate the Interconnect for HPC.Infiniband is rearing to dominate at the Midrange latency applications


HPC Tools Availability


Data Center Environmentals Prep

Speed fpm

Temp F

Sources: Fluent Airpak,

Ansys CFD,


HPC embraces a broad spectrum of markets ..

High Performance High Performance ComputingComputing

WorldWorld


3 Tier Computing Infrastructure

Uniform and ubiquitous physical Connectivity – “Wire Once”

Any user to any server via Internet

Any server to any server via the LAN

Any sever to any storage via the SAN

Dynamic Logical Binding of…

server network identification

server OS version

server application assignment

application data volumes

Resulting in

Application Mobility across servers

Data mobility across storage

ApplicationEnd-Users

Tier –1 Network Services:

- Web Servers, - Firewalls,

- Load balancers

Tier-2 Application Services:

- Application Servers

Tier-3 Data Base Services

- Database Servers- Storage

Storage

Internet

LAN

SAN


E2E Internet & 3 Tier Computing

Web

Web

Web

Web

Web

Web

Web

Web

Data CenterApplicationsEdge

Directory Security Policy Management

Software OS Platform

AppApp

AppApp

AppApp

AppApp

DBDB

DBDB

DBDB

IP Storage Network

FC SAN IP SAN NAS NAS

InternetCoreOptical

Edge

HA, Secure Data Center

http://images.google.com/imgres?imgurl=www.thomas-pr.com/136/photos/TP-200%2520kiosk.jpg&imgrefurl=http://www.thomas-pr.com/136/photos/tp200.html&h=2450&w=2011&prev=/images%3Fq%3Dkiosk%26start%3D20%26svnum%3D10%26hl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26sa%3DN


SNMP,IETF/ CIM,

SMI-S,SMASH.

Quadrics,Myrinet,SCIInfiniBand,Ethernet / IP,Ethernet IP w/TOE,Ethernet IP w/TOE and RDMA.

SCSI,Fiber Channel,ISCSI.

Ethernet,Wi-Fi.Network Fabric

System FabricM

anag

emen

tFa

bric

Fabric based Clusters Architecture


Stor

age

Syst

ems

Fabric based Grid Applications


Server Platforms - Migrations

Scale Out (Clustered Blade Servers)

Large SMPParallel Sysplex- Scalable Nodes, - Dynamic LPAR- SW Provisioning/VMWareIBM: i890/p690/z990SUN: SunFire 4800, 12K

Clusters High Availability/Hi Perform.Clusters- Linux/AIX/Solaris Clusters- HPC ClustersDELL: PowerEdge 6650ORACLE: 9iRAC/PE6650

Blade Servers High Density Rack Mount- Highly Integrated Dense Form Factor - Rapid Deployment, Flexible Architect. RLX: ServerBlade 300i, 800i, HP: 40b, pFUJITSU: BX300, DELL:1655 IBM: HS20, Sun:

Scal

e U

p (C

onso

lidat

ed S

MP

Serv

ers)


Blades Servers - InfrastructureMidplane

W/ConnectorsTo Blades & Back

Modules

Storage: IP NAS or FC SAN Switch

Management Modules Remote Mgmt+KVM over IP

Cooling N+1Fans/Cooling ModulesPower: N+1Power Supplies

Networking: Gbit Ethernet Switches

BladeControlPanel

Processor Blades(6-24 typically)

MemoryDDR wECC

GbitEthernet

I/FSystemsMonitorModule

Micro-Processors

USB Ports, CD/Floppy I/F

© 2003 Source: IMEX Research

~ GbE Switch supportsTrunking/Port AggregationFlow ControlQoS Packet PrioritizationSNMP/RMONIGMP/BOOTP/TFTP……


Management – Blade Servers

80%

2020%%

Ultra-dense HW• Density• Power Consumption• Cable Management

Management• Platforms integration:

Hardware Software Network Management

• Scalability• Security• Manageability• Lowest TCO

80 %

20 %

IMEX©2000-2005 IMEX ResearchData: IMEX Research 2005

Blade Servers Market Opportunity

$-

$2

$4

$6

$8

$10

$12

2003 2004 2005 2006 2007 2008

Fact

ory

Rev

$B

WW Blade Servers Market Opportunity (cum 2004-2008)

Revenues ~ $24 BillionUnits ~ 9 Million


Blade Servers: Vendor Positioning Index(As of 31 Mar 2004 - See IMEX Blade Servers Industry Report 2005 for new data)

Stra

tegy

(Pot

entia

l)

Delivery (Execution)

IBM

HP

DellTatungNexcom

Intel

Penguin

Fujitsu Siemens

NEC

Sun Egenera

©20

03-0

5 IM

EX

Res

earc

h

Rackable SysVerari

Linux Networx

Appro

HPC Syst

Hitachi


iSCSI on roadmap of storage vendorsC

OS

T

SERVICE LEVEL

High EndHigh End FC & iSCSIFC & iSCSI

MidMid--TierTier NAS & iSCSINAS & iSCSI

MidMid--TierTier FC & iSCSIFC & iSCSI

EntryEntry FC & iSCSIFC & iSCSI

SAN(Fibre

Channel)NAS

(IP)iSCSI

(IP)

•iSCSI delivers low-cost Networked Storage–Higher availability,Increased usage

–Centralized management,Storage functionality


System Cost vs. High Availability

S/390MVSIBM Propl

S/390MVSIBM Propl

Bull Dell Fujitsu HP IBM NEC Siemens Stratus

Bull Dell Fujitsu HP IBM NEC Siemens Stratus

IBMIBM

Sun-SolarisHP-UXIBM-AIXNCR-SVR4

Sun-SolarisHP-UXIBM-AIXNCR-SVR4

Prop.

AS400

WinUNIX

Cluster Win

DowntimeHrs./Syst/Yr

99.999% 99.9% 99.0%

$10K

System Availability99.99%

0.1 101 100

Avg.Syst

Price

$100 K

$1 M

$10 M

z/390

z/390Sysplex

ClusteredUNIX

Clustered LinuxLinux


The iSCSI SAN

To Other LANs,Servers or IP SANs

Ethernet Header CRCIP TCP DATA

Client requests data from App Server

Ethernet CRCHeader IP

TCP iSCSI SCSI DATA

iSCSI-equipped App Server (“initiator”) requests data from an iSCSI storage (“target”)

Storage Arrays(iSCSI Target)

Application & DB Servers

(iSCSI Initiator)

Client Workstations

SAN GigE

Switches

LANEthernet Switches


Tiered Storage by Price/Performance

ServersMidline Storage

Mainstream NAS/SAN

Large Enterprise SAN & NAS

Desktop PCEntry NAS

Workstations

Source:Maxtor

Drive Interface

Drive Types

FC Drive

FC RAID

SATA Drive

PC Chipset

5400 RPM7200 RPM

SATA

SAS HBA/RAID

SASSAS

SAS Drive

10K RPM15K RPM

SAS HBA


Cost Driven Data Storage Markets

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2003 2004 2005 2006 2007 2008

SATA

P_SCSI

FC

SAS


Grid Computing

• Grid is a Catch-all Marketing Term•• Means different things to different constituenciesMeans different things to different constituencies

•• Desktop CycleDesktop Cycle--StealingStealing

•• ManahedManahed HPC ClustersHPC Clusters

•• Virtualization of Data Center ResourcesVirtualization of Data Center Resources

•• Outsourcing to Outsourcing to ““Utility Data CentersUtility Data Centers””

• Multiple, opposing requirements•• ComputeCompute--intensive applications want to ship data intensive applications want to ship data tptp idle processor resourcesidle processor resources

•• DataData--Intensive applications want to ship computations to appropriate Intensive applications want to ship computations to appropriate large data large data repositoriesrepositories

• An evolving entity with many pulls & pushes•• Economics, Feasibilities, management of hetero Economics, Feasibilities, management of hetero vsvs homogeneous HW homogeneous HW


Grid Challenges: Hype vs. Reality

• Cost of Local Grids (LAGs) – ala Blades cheaper than Wide Area Grids (WAGs)

•• Computing Cost $1000 > 1 Computing Cost $1000 > 1 cpucpu day (10 day (10 TeropsTerops) =$1) =$1

•• 10 TB network transfer Cost = $110 TB network transfer Cost = $1

•• Internet BW cost $100/mbps/mo > 1GB network Internet BW cost $100/mbps/mo > 1GB network xfrxfr cost ~$1cost ~$1

•• Result: Local HPC Cluster is 10,000 cheaper than WAN CommunicatiResult: Local HPC Cluster is 10,000 cheaper than WAN Communicationon

• MPI Style Apps work well in LAN Clusters but uneconomical in WAN

•• Data analysis best done by moving programs to data not data to pData analysis best done by moving programs to data not data to programsrograms

•• Small data, high compute applications Small data, high compute applications woekwoek well across internetwell across internet

•• Internet is not the CPU back planeInternet is not the CPU back plane


Grids – Homogeneous vs Heterogeneous


Grid Services Architecture & Standards


Grid Computing – Across 3 Tiers

Grid ControlStorage

DatabasesApplication Servers


Utility Computing/Web Services

Network Virtualization

Server Virtualization

Storage Virtualization

Internet Internet

Switching Pool

Load Balancer Pool

Server Pool

Switching Pool

Storage Pool

Util

ity C

ontr

olle

r

NAS Pool

Firewall Pool

• HP - Utility Data Center• Sun - N1/SunONE/Orion• Microsoft - .Net• IBM - OnDemand/Autonomics

Autonomics• Self Healing• Self Optimizing• Universal Identity• Single System Image• ….


Future Computing: Processes Driven

HBAInitiator

Target

Old model New model

ApplicationApplicationApplication

Initiator Initiator Initiator

Processes are the initiatorsHardware is the initiator

Allows applications to move storage

wherever they are - storage no longer

cares where they are

ID


Application-Driven Storage Virtualization

Moving from storage associated with

servers...

ApplicationApplicationApplication

...to storage associated with application

processes

New Model Allows applications to move storage from the

virtualized pool, wherever they may be

Old Model Server Hardware is the

initiator. Storage Drives are the Target


Future: Embedded10Gb,TOE, iSCSI

TOE chipset2x1Gb TOE NIC 10 Gb TOE NIC Embedded

20042003 2005 2006 onwards

MCMC

CPUCPUCPU CPUCPUCPU

MCMC

CPUCPUCPU CPUCPUCPU

PCI-X

TOE with 2GbE

PCI Exp PCI Exp

1GbE

1GbETOE

DMA

CPUCPUCPU

MCMC

CPUCPUCPU

PCI Exp

DMA

Uses Syst Mmry

TOE 10GbE

Ext.Mmry

PCI PCI-XExt Mmry

I/OBridge

I/OBridge

GbE

GbETOE

PCI-XPCI

I/OBridge

Uses Syst Mmry

PCI ExpPCI Exp

10 GbE1GbE1GbE

CPUCPUCPU CPUCPUCPU

MCMC

TOE


Future: IP Storage Management on a chip

Host Services Integration StorageProvisioning Layer

IP SAN Management Layer

Appliance Service Layer

File system monitoring

IP SAN ManagementManagement

ConsoleManagement of

iSCSI HBAsMultiPath IO Supp

and FailoverSecurity

(iSNS, CHAP, SRP)

Virtualization Mirroring Snapshot Fail-Over

iSCSI Target ManagementLVM, Error Handling, SCSI Daemon, API Interoperability

HW Acceleration: TOE, iSCSI Offload, IPsec

Storage provisioning Win, LINUX, Solaris


Mgmt WS

Intelligent Uniform IP-based

Infrastructure

Storage Sub Systems

Application & DB Servers

Clients

SAN

LAN Ntwk Mgmt WS

Storage Mgmt WS

Present Data Center Infrastructue Future Infrastructue

Future Intelligent IP Infrastructure


Future - IP Based Infrastructure

Networks

Storage

VirtualizationProvisioningAutomation

Servers

TelecomIP

Infrastructure


Data rate & capacity

Throughput : : DSL/Cable

100+ Teraflops

Throughput = 100 GB/s

Rendering (Texture & Polygons)

Throughput = 1.2 GB/s

EntertainmentEntertainmentAudio/Video On DemandAudio/Video On Demand


Commercial Commercial VisualizationVisualization

BioInformaticsBioInformatics DecisionDecisionSupport SystSupport Syst

Dat

a: IM

EX R

esea

rch

& Pa

nasa

s



100+ Teraflops








Dat

a: IM

EX R

esea

rch

& Pa

nasa

s



100+ Teraflops






100+ Teraflops












Dat

a: IM

EX R

esea

rch

& Pa

nasa

s


Entertainment Audio/Video OnDemand

Decision-Support Systems

BioinformaticsCommercial Visualization

©2004--05 IMEX Research.com

HPC – From Academia to Wall St to Hollywood


HPC: From Homeland Security to Bioinformatics


HPC From Academia to Hollywood


IMEXRESEARCH.COM


Anil Vasudeva Principal Analyst & President [email protected]

State of the Industry


State of Interconnect Fabrics

Bandwidth/link

100 Mb/sec 1Gb/sec 10 Gb/sec

EHERNET - LAN/SAN/MAN/WAN

IB - CLUSTER

SMP/NUMA

SGI

Synfinity-1

MyrinetServernet

ATM

GbEFE

IBM/SP

T3E

10 us

100 us

1 ms

1us

MPP

Dolphin

Scali

Quadrics Mellanox

Latency/hop

FJ/Synfinity10GbEw/RDMA

GbEw/TOE

GbEw/RDMA