hpc market update from idc

IDC HPC Update at ISC’15

Earl Joseph – [email protected] Sorensen– [email protected]

Lloyd Cohen – [email protected] Conway – [email protected]

mailto:[email protected]




IDC Has Over 1,000 Analysts In 62 Countries

IDC’s HPC Team

Earl Joseph

IDC HPC research studies, HPC

User Forum

Steve Conway

Strategic consulting, HPC User

Forum, market trends, Big Data

Bob Sorensen

Strategic research projects,

government studies and

international analysis

Lloyd Cohen

HPC data and workstations

Mike Thorp and Kurt Gantrish

Government account support

and special projects

Mary Rolph

HPC User Forum conference

planning and logistics

About IDC: IDC HPC Activities

• Track all HPC servers sold each quarter

• 4 HPC User Forum meetings each year

• Publish 85 plus research reports each year

• Visit all major supercomputer sites & write reports

• Assist in collaborations between buyers/users and vendors

• Assist governments in HPC plans, strategies and direction

• Assist buyers/users in planning and procurements

• Maintain 5 year forecasts in many areas/topics

• New: Developing a worldwide ROI measurement system

• New: HPDA program

• New: Quarterly tracking of GPUs/accelerators

Agenda

1. HPC Market Update

2. The HPC Market in Europe

3. HPDA Update: Where Big Data Meets HPC

4. IDC ROI Research Update

5. HPC User Forum Update

6. IDC HPC Innovation Program Update

7. Conclusions

CHECK OUT OUR WEB SITE:www.hpcuserforum.com

Important Dates For Your Calendar

2015 HPC USER FORUM MEETINGS:

• Broomfield, ColoradoLocation: Omni Interlocken

September 8-10, 2015

• Paris -- Institute of Earth Physics of Paris Host: GENCI

October 12-13, 2015

• Munich -- Leibniz RechenzentrumHost: LRZ

October 15-16, 2015

HPC

Market Update

Top Trends in HPC

2014 ended soft, after a number of very strong years

• Top5 system purchases has slowed for ~2 years

• The IBM/Lenovo deal delayed many purchases

• 2014 was basically flat with 2013 ($10.2 billion in 2014

vs. $10.3 billion in 2013)

• 2015 to 2018 are predicted to be healthy growth years

Big data combined with HPC is creating new solutions

• Adding many new users/buyers to the HPC space

Software Issues continue to grow

GPUs & accelerators extend their impact

Top Trends in HPC

Top Trends in HPC:

Growing Use of Clouds

Top Trends in HPC:

Evolving Issues On Our Minds

Non-x86 processors could alter the landscape

• ARM, Power, others

• Coprocessors, GPUs, FPGAs

China looms large(r)

• Lenovo, growing domestic market, export intentions

• Other Chinese vendors are planning to extend to Europe

Growing influence of the data center in IT food chain

• Impact on HPC technology options

• Perhaps providing new approaches?

HPC in the Cloud Gaining Traction

• How much, how soon?

Top Trends in HPC:

HPC Storage

Storage is the fastest-growing part of the HPC market

HPC storage revenue will grow to record levels

The HPC storage market remains fragmented

The big players are turning their attention to this market

The HPC interconnect market is in transition

Data movement/management is a major pain point

• Multi-year shift away from today’s extreme compute-centrism

• Many buyers still not savvy enough about purchasing storage

HPDA will boost storage budgets

© IDC Visit us at IDC.com and follow us on Twitter: @IDC

IDC HPC Competitive Segments: 2014

Departmental ($250K - $100K)

$3.83B

Divisional ($250K - $500K)

$1.52B

Supercomputers(Over $500K)

$3.15B

Workgroup(under $100K)

$1.72B

HPC

Servers

$10.2B

2014 HPC Revenue Results

HPC Revenues: By Vendor

2014 Revenue Share by Vendor:

Supercomputer Segment ($500K+)

© IDC Visit us at IDC.com and follow us on Twitter: @IDC

2014 HPC Market: By Industry/Applications

HPC

Market Forecasts

HPC Forecasts

• Forecasting a 8.2% yearly growth from 2014 to 2019

• 2019 should reach $15.2 billion

The Broader HPC Market

European

HPC Directions

The HPC Market In Europe: Revenues

The HPC Market In Europe: Processors

The HPC Market In Europe: By Workloads

The HPC Market In Europe: The Broader

Market

The HPC Market In Europe: By Country

IDC Is Completing a New Study for

the European Commission

The Study Evaluates Progress on the

EU HPC Action Plan Published in 2012

HPDA Update

High Performance Data Analysis

Needs HPC resources• High complexity (algorithms)

• High time-criticality

• High variability

• (On premise or in cloud)

Data of all kinds• The 4 V’s: volume, variety, velocity, value

• Structured, unstructured

• Partitionable, non-partitionable

• Regular, irregular patterns

Simulation & analytics

• Search, pattern discovery

• Iterative methods

• Established HPC users + new

commercial users

HPC Adoption Timeline (Examples)

1960 1970 1980 1990 2000 2012

IDC’s HPDA Program

IDC’s HPDA Program:

Example Research Documents • Supercomputer-Based Analytics Are Starting to Transform Healthcare

• United States Postal Service Turns to HPC for Big Data and Analytics

• High-Performance Data Analytics: Market Status and User Exemplars

• PayPal Says More Fortune 2000 Firms Could Benefit from HPC for Big Data Analytics

• Market Analysis Perspective: Worldwide High-Performance Data Analysis, 2014

• NEC's HPC Vision: Bringing Vector Supercomputing to a Broader Data-Intensive User Base

• Bull Launches an Exascale Program for 2020, But the Vision Extends to Big Data and Beyond

• Total S.A. Retakes the Lead in Energy Sector Supercomputer Race

• Los Alamos National Laboratory: Seven Decades of Computing Leadership

• Summary of HPC and HPDA Technology, Development, and Applications: HPC User Forum, September 15–17, 2014, Seattle, Washington

• HPC Interconnects: New Options Emerging from a Sector in Transition

IDC’s HPDA Program: Server Forecasts

IDC’s HPDA Program: Four New

Industry/Application Workload Segments Fraud and anomaly detection

• This "horizontal" workload segment centers around identifying harmful or potentially harmful patterns and causes using graph analysis, semantic analysis, or other high performance analytics techniques.

Marketing• This segment covers the use of HPDA to promote products or services, typically

using complex algorithms to discern potential customers' demographics, buying preferences and habits.

Business intelligence• The workload segment uses HPDA to identify opportunities to advance the market

position and competitiveness of businesses, by better understanding themselves, their competitors, and the evolving dynamics of the markets they participate in.

Other Commercial HPDA• This catchall segment includes all commercial HPDA workloads other than the three

just described. • Over time, IDC expects some of these workloads to become significant enough to

split out, i.e. the use of HPDA to manage large IT infrastructures, and Internet-of-Things (IoT) infrastructures.

Note Financial, classified buyers, etc. will continue to be listed under the existing IDC segments

IDC’s HPDA Program:

New Research Findings

Machine Learning/Deep Learning

Like a game: a goal and rules for reaching it.

Stage 1: Search

• Computer is given the goal and trained in the rules (e.g.,

Jeopardy Watson).

Stage 2: Unknown pattern discovery

• Computer is given the goal and has to discover the rules

(e.g., medical Watson, some x86 clusters)

Stage 3: Cognitive computing/AI

• Computer has to discover the goal and the rules (future).

Machine Learning: Typical Attributes

Mainstream IT Market Serial algorithms

Data scientists acquire

algorithms

Major data reduction before

analytics (sometimes >99%)

Not very time-critical

Public cloud often fine

HPC (HPDA) Market Parallel algorithms

Data scientists create

algorithms

No big data reduction: high-

resolution results needed

Very time-critical

Public cloud often inadequate

Summary: HPDA Market Opportunity

HPDA: simulation + newer high-

performance analytics

• IDC predicts fast growth from a small

starting point

HPC and high-end commercial

analytics are converging

• Algorithmic complexity is the common

denominator

Economically important use cases are

emerging

No single HPC solution is best for all

problems

IDC ROI Research

Our 3 Year Grant From DOE

Three-Year Research Plan

2013 pilot study:

Tested 3 approaches and set the models

Populated the models with 208 cases of scientific innovation

and industrial ROI (it now exceeds 400)

Created the innovation index

Three-year study: sponsored by DOE Science/NNSA

Refine the models to move from association towards causation

Collect thousands of cases:

• Dense collection in the U.S.

• Sparser collection elsewhere*

* DOE has asked IDC to invite other countries to participate

The Financial ROI Models

ROI models that have been developed include:

1. ROI based on revenues generated (similar to GDP)

divided by HPC investment

2. ROI based on profits generated (or costs saved)

divided by HPC investment

3. ROI based on jobs created

Two Innovation Index Scales

How would you rate this innovation compared to all other innovations in this field over the last ten years?

Scale One:

5 = One of the top 2 to 3 innovations in the last decade

4 = One of the top 5 innovations in the last decade




Scale Two:

5 = It had a major impact and is useful to many organizations

4 = A minor innovation that is useful to many organizations

3 = A minor innovation useful only to 2 -3 organizations

2 = A minor innovation useful only to 1 organization

1 = An innovation that is recognized ONLY by experts in the field

Example Findings: The Financial ROI Model

Example of The ROI Model

www.hpcuserforum.com/ROI

HPC User Forum

Update

HPC User Forum Mission

To Improve the Health of the High

Performance Computing Industry through

Open Discussions, Information-sharing

and Initiatives Involving HPC Users In

Industry, Government and Academia Along

with HPC Vendors and Other Interested

Parties

Steering Committee Members Paul Muzio, City University of New York, Chair Rupak Biswas, NASA Ames, Vice Chair Earl Joseph, Executive Director Swamy Akasapu, General Motors Vijay Agarwala, Penn State University Alex Akkerman, Ford Motor Company Doug Ball, HPC expert (formerly Boeing) Jeff Broughton. NERSC/Lawrence Berkeley National Lab Paul Buerger, Avetec Chris Catherasoo, Caltech Jack Collins, National Cancer Institute Steve Conway, IDC Research Vice President Steve Finn, Cherokee Information Services Merle Giles, NSCA/University of Illinois Keith Gray, British Petroleum James Kasdorf, Pittsburgh Supercomputing Center, Chairman Doug Kothe, Oak Ridge National Laboratory Jysoo Lee, NICN Michael Resch , HLRS, University of Stuttgart Vince Scarafino, Industry Expert Suzy Tichenor, Oak Ridge National Laboratory

50+ Meetings Worldwide Since 2000

Amsterdam, Netherlands (SARA)

Annecy, France

Bangalore, India (Indian Institute

of Technology)

Beijing, China (Chinese Academy

of Sciences)

Bologna, Italy (CINECA)

Bristol, UK

Bruyères-le-Châtel, France

(Teratec)

Canberra, Australia

Geneva, Switzerland (CERN)

Kobe, Japan (RIKEN)

Lausanne, Switzerland (EPFL)

London, UK (Imperial College)

Manchester, UK (Manchester

University)

Melbourne, Australia

New Delhi, India (Indian Institute

of Science)

Seoul, Korea (National Institute of

Supercomputing & Networking)

Stuttgart, Germany (HLRS)

Warsaw, Poland (University of

Warsaw)

Yokohama, Japan (Earth

Simulator Center)

Zurich, Switzerland (ETH Zurich)

United States (many locations)

Recent HPC User Forum Meetings 2014:

• April: Santa Fe, New Mexico

• July: Kobe, Japan (RIKEN)

• September: Seattle, Washington

• October: Canberra & Melbourne, Australia

• October: Stuttgart, Germany (HLRS)

Future HPC User Forum Meetings

2015:• September 8 to 10, Denver, Colorado

• October 12-13. Paris (host: GENCI)

• October 15-16. Munich (host: LRZ)

CHECK OUT OUR NEW WEB SITE:

www.hpcuserforum.com

Updates On The IDC

HPC Innovation Award

HPC Innovation Award Program:

www.hpcuserforum.com/innovationaward

The Trophy For Winners

HPC Innovation Awards for ISC 15

After careful deliberation, IDC’s HPC User Forum Steering Committee, which serves as the judging panel for the IDC HPC Innovation Excellence awards, has decided to make no awards for this round (July 2015).

• Some of the submissions described work that seemed both important and interesting

• However, none of the submission included information that was complete enough to permit fair judging.

• IDC has invited them to apply for the fall award.

• To avoid this issue going forward, IDC will be updating the application process and making the rules/objectives more explicit and easier to follow.

HPC Award Program Process

The Original Innovation Awards Goals and Objectives:

Help to expand the use of HPC by showing real ROI examples:

Expand the “Missing Middle” – SMBs, SMEs, SMSs -- by

providing examples of what can be done with HPC

Show mainstream and leading edge HPC success stories

Create a large database of success stories across many

industries/verticals/disciplines

To help justify investments and show non-users ideas on how

to adopt HPC in their environment

Creating many examples for funding bodies and politicians to

use and better understand the value of HPC to help grow

public interest in expanding HPC investments

For OEMs to demonstrate success stories using their products

HPC Award Program Process

Look for changes to the application process soon:

A new application form better designed to:

More clearly explain the rules and key data required

More clearly explain the type of accomplishments that are

desired

Better accomplish the goals:

• Recognize users and their vendors for major HPC-supported

achievements in industry, government and academia.

• Build a large portfolio of quantified ROI success stories to

strengthen the case for boosting investments in and funding for

HPC.

• Increase HPC adoption by educating the market on its benefits –

using clear and specific ROI examples.

HPC Award Program New Judging Process

A new more timely and interactive multi-step review process designed to extract the best insights from the base of HPC submitters:

• Stage 1: Before preliminary deadline, application is submitted using

new form;

• Stage 2: Application is reviewed by a Steering Committee member

and marked ‘Accept, Reject, or Return for Additional Information’;

• Stage 3: When needed, a Steering Committee member will work

with a submitter to assemble needed information;

• Stage 4: Re-worked application is submitted before final deadline.

• Stage 5: Full Steering Committee membership judges applications

Please standby as IDC will soon be issuing new guidance for future

submissions, starting with SC15 in November.

HPC Award Program Submission Form:

www.hpcuserforum.com/innovationaward

In

Summary

Why HPC Is Projected To Grow

1. The low half of the market is finally back to a recovery

mode

2. It has become a competitive weapon

3. Governments view HPC leadership as critical

For national pride, but more importantly for economic

prosperity

4. There are very critical HPC issues that need to be

solved

Global warming, alternative energy, safe NE, financial

disaster modeling, healthcare, homeland security, …

And 3D movies and large scale games are fun

But There are Still Major Customer

Pain Points

Software is the #1 roadblock• Better management software is needed

• Parallel software is lacking for most users

Many applications will need a major redesign

Clusters are still hard to use and manage • System management & growing cluster complexity

• Power, cooling and floor space are major issues

• Third party software costs

• Storage and data management are becoming new bottle necks

• Lack of support for heterogeneous environment and accelerators

ROI is becoming a requirement, especially as system costs escalate

… Some good news in that there are new technologies in Big data, accelerators, clouds, etc.

HPC is still expect to be a strong growth market

• Growing recognition of HPC’s strategic value is helping to drive

high-end sales

• Low-end buyers are back into a growth mode

Vendor share positions will shift greatly in 2015

Recognition of HPC’s strategic/economic value will

drive the exascale race

The formative HPDA market will expand

opportunities for vendors

The high ROI will drive major segments (finance,

health care)

China will be a wild card – with the US-China

supercomputer battles

Conclusions

Important Dates For Your Calendar

2015 HPC USER FORUM MEETINGS:

• Broomfield, ColoradoLocation: Omni Interlocken

September 8-10, 2015

• Paris -- Institute of Earth Physics of Paris Host: GENCI

October 12-13, 2015

• Munich -- Leibniz RechenzentrumHost: LRZ

October 15-16, 2015

Questions?

[email protected]

[email protected]

[email protected]

Back-up Slides

An Example: U.S. Postal Service

An Example: U.S. Postal Service

v

MCDB = memory-centric database