Research & Development of ParallelComputing – a software perspective

Bob C. Liang, Ph.D. 梁兆柱 博士

Chief Research Officer 首席研发官

Intel Software & Solutions Group PRC

英特尔中国软件与解决方案事业群

Intel Asia Pacific R & D

英特尔亚太研发中心

Agenda

Perspective

Examples of Academic Research in US

- UCB & UIUC

Application – Driven

Parallel Programming

What’s Drives HW & SW Advances

SW Development Spiral

Moore’s Law

Application - with Open Source

69% MSS

Est. 10M Installations

PBX1000 Downloads/day

ERP1.2M Downloads

Office Tools800K Downloads/month

Integration Backup/Recovery SolutionBusiness Intelligence

Development Tools

Web Servers

Application Servers

Databases

Applications

Operating Systems

Serv

er

So

luti

on

Sta

ck

Linux, ~30% Server MSS

550K Developers, 23M+ Websites

9M Downloads, 31% MSS

Growing Virtualization EcosystemS

olu

tio

n S

tack

Applications

Infrastructure

OEMs

Advanced in Silicon Technology in last Decade

Intel’s ASCI Option Red

9000 CPUs

one megawatt of electricity.

1600 square feet of floor space.

One CPU

67 watt

275 mm2

First Terascale computer: 1997 First Terascale chip: 2007

Intel’s 80 core research chip

10 years

Source: IntelSource: Intel

Computing Beyond Teraflops

One CPU

67 watt

275 mm2

Terascale chip: 2007

Intel’s 80 core research chip

10 years

Your “Cell phone” will have more

Processing power than ASCII Red

2017

What?

“This (Multi-core system) is a unprecedented challenge –Patterson (UC Berkeley)

The Powerful Hardware needs parallel programming

Handhelds Powered by Many Core

9

Parallel Web Browser By Ras Bodik et al.

UC Berkeley

Perspective

Perspective– We are racing into the “Age of Many Cores”– Parallelism is the basis for performance scaling –> overall advance of the

IT industry and engine of economic growth– Industry has a grip on small-scale parallelism (4-32 threads) in the next

1-2 years

What is needed 2+ years out– Catalyze breakthrough research to enable pervasive parallel computing

with >100 T– Key focus on breakthrough applications and programming tools/systems– Secondary focus on architecture and system software

Catalyze breakthrough research to enable pervasive use of parallel computing

Parallel Programming

Languages, Compilers,Runtime, Tools

Parallel Applications

For desktop, gaming, and mobile systems

Parallel Architecture

Support new generation of programming models

and languages

Parallel Sys. S/W

Performance scaling, memory Utilization,

& power consumption

Program Goals

Agenda

Perspective

Examples of Academic Research in US

- UCB & UIUC

Application – Driven

Parallel Programming

UCB Center – Research Plan Overview

Applications– Personalized medicine – CHD - Keavney– Music/Hearing Augmentation – Signal processing, real time

constraints – Wessel– Speech Recognition – Noisy environments, Morgan– Context-based Image Retrieval – Keutzer – PIRO code from CTG– Parallel Web Browser – parsing, layout, rendering, implicitly

parallel scripting lang’

Composition of Computing– 13 different areas/kernels/apps that describe much of computing– Compose these to form real applications– Layered system stack – applications, productivity,

efficiency/library, OS/VMM, HW– Correctness interweaves entire stack

UCB Center – Research Plan Overview

UIUC Research Plan Make Client Parallel Programming EasyApplications

– Natural Language – Roth– Vision – Forsyth– Graphics – Hart – Tele-immersive Env – Nahrstedt

Programming models– Disciplined sharing – Adve – Shared-nothing parallelism – Agha– Domain-specific – Johnson

Tools– Compilers – Hwu – Auto-tuning Libraries – Padua – Correctness – Zhou

Execution components– Runtime – Kale– Architecture – Torrellas – Managed execution – Zilles

Tele-immersive Env

DFT-1M (out-of-cache)

runtime (sec)

ener

gy (J

)

Auto-tuning Libraries

Pair each core:

LeadersOverclockedMay suffer errors

CheckersChecks – no errors

Paceline Architecture

UIUC Research Plan Overview

Agenda

Perspective

Examples of Academic Research in US

- UCB & UIUC

Application – Driven and Client Focus

Parallel Programming

Synthesis – Ray Tracing

Recognition, Physics, Ray-tracingIdentifying the data with models

Motion Captured by 4 cameras driving a 3D Avatar

Mining – “What is, Where is, etc.”Personal Data Mining” – My Semantic Media

Agenda

Perspective

Examples of Academic Research in US

- UCB & UIUC

Application – Driven

Parallel Programming

22 Copyright © 2008, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners

Intel® Parallel Studio Suite of products to add parallelism to Microsoft* Visual Studio*Intel® Parallel Studio address four phases of development:– Intel® Parallel Advisor design– Intel® Parallel Composer code– Intel® Parallel Inspector debug– Intel® Parallel Amplifier tune

Maximize for multicore processors now,forward-scale to manycore processors of the future (including Larrabee architecture)Inter-operate with and extend Microsoft Visual Studio,ready to support common Concurrency Runtime (Microsoft future)

www.intel.com/go/parallel– to be in on the beta trials from the very start– to learn more about Intel® Parallel Studio for Microsoft Visual Studio developers– Linux* & Mac OS* X developers can sign-up to use parallel tools too: beta for 11.0

compilers now

Focused solutions based on current products, new technology, research results, innovations and user studies.

Parallelism Innovation ThenParallelism Innovation Then………………....

ABCPLACE ACT++ Active messages AdlAdsmithADDAPAFAPIALWANAMAMDCAppLeSAmoeba ARTSAthapascan-0bAuroraAutomapbb_threads BlazeBSPBlockCommC*. "C* in C C** CarlOSCashmereC4CC++ ChuCharlotteCharmCharm++CidCilkCM-Fortran ConverseCodeCOOL

CORRELATE CPS CRLCSPCthreadsCUMULVSDAGGERDAPPLE Data Parallel C DC++ DCE++ DDDDICE.DIPC DOLIBDOME DOSMOS.DRLDSM-ThreadsEase .ECOEiffel EileanEmerald EPL ExcaliburExpressFalconFilamentsFMFLASHThe FORCE ForkFortran-MFXGA GAMMA Glenda

GLUGUARDHAsL.Haskell HPC++JAVAR.HORUSHPCIMPACTISIS.JAVARJADE Java RMIjavaPGJavaSpaceJIDLJoyceKhorosKarma KOAN/Fortran-SLAMLilac LindaJADA WWWindaISETL-Linda ParLinEileanP4-LindaGlenda POSYBLObjective-LindaLiPSLocustLparxLucidMaisieManifold

MentatLegionMeta Chaos MidwayMillipedeCparParMirageMpCMOSIXModula-PModula-2*MultipolMPIMPC++MuninNano-ThreadsNESLNetClasses++ NexusNimrodNOWObjective LindaOccamOmegaOpenMPOrcaOOF90P++P3Lp4-LindaPabloPADEPADRE Panda Papers AFAPI.Para++

Paradigm

Parafrase2 ParalationParallel-C++ ParallaxisParCParLib++ParLinParmacsPartipCpC++PCNPCP: PHPEACEPCUPETPETScPENNYPhosphorus POET.Polaris POOMAPOOL-TPRESTOP-RIO ProsperoProteus QPC++ PVMPSIPSDMQuakeQuarkQuick ThreadsSage++SCANDALSAM

pC++ SCHEDULESciTLPOET SDDA.SHMEM SIMPLESinaSISAL.distributed smalltalkSMI.SONiCSplit-C.SRSthreadsStrand.SUIF.SynergyTelegrphosSuperPascalTCGMSG.Threads.h++.TreadMarksTRAPPERuC++ UNITY UC V ViC* Visifold V-NUS VPEWin32 threads WinParWWWindaXENOOPS

XPCZoundsZPL

Parallel programming environments in the 90’s (a partial list)

Architecting Parallel Software

Kurt Keutzer(UCB), Tim Mattson (Intel)and the UCB PALLAS team:

Bryan Catanzaro, Jike Chong, Matt Moskewicz, Michael Murphy, NR Satish, Bor-Yiing Su, Naryanan Sundaram, Youngmin Yi

UCB = University of California, BerkeleyUCB = University of California, Berkeley

Our general approachSoftware must be architected to achieve productivity,

efficiency, and correctness

Key to architecture (software or otherwise) is design patterns and a pattern language

Resulting software design then uses a hierarchy of software frameworks for implementation

Application frameworks for application (e.g. CAD) developersProgramming frameworks for those who build the application frameworks

Alexander’s Pattern LanguageChristopher Alexander’s approach to (civil)

architecture:"Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice.“ Page x, A Pattern Language, Christopher Alexander

Alexander’s 253 (civil) architectural patterns range from the creation of cities (2. distribution of towns) to particular building problems (232. roof cap)

A pattern language is an organized way of tackling an architectural problem using patterns

Main limitation:It’s about civil not software architecture!!!

Patterns for Parallel Programming• PLPP is the first attempt to develop a complete patternlanguage for parallel software development.

• PLPP is a great model for a pattern language for parallel software

• PLPP mined scientific applications that utilize amonolithic application style

•PLPP doesn’t help us much with horizontal composition

•Much more useful to us than: Design Patterns: Elements of Reusable Object-Oriented Software, Gamma, Helm, Johnson & Vlissides, Addison-Wesley, 1995.

Computational Patterns

Remember the Dwarfs from Remember the Dwarfs from ““The Berkeley ViewThe Berkeley View””??Dwarfs form our key computational patternsDwarfs form our key computational patterns

Structural programming patternsIn order to create more complex software it is necessary to compose programming patterns For this purpose, it has been useful to induct a set of patterns known as “architectural styles”Examples:

pipe and filterevent based/event drivenlayeredAgent and repository/blackboardprocess controlModel-view-controller

Put it all together in a semester

Graph Algorithms

Dynamic Programming

Dense Linear Algebra

Sparse Linear Algebra

Unstructured Grids

Structured Grids

Model-view controller

Iterator

Map reduce

Layered systems

Arbitrary Static Task Graph

Pipe-and-filter

Agent and Repository

Process Control

Event based, implicit invocation

Graphical models

Finite state machines

Backtrack Branch and Bound

N-Body methods

Circuits

Spectral Methods

Task Decomposition ↔ Data DecompositionGroup Tasks Order groups data sharing data access

Applications

Pipeline

Discrete Event

Event Based

Divide and Conquer

Data Parallelism

Geometric Decomposition

Task Parallelism

Graph algorithms

Fork/Join

CSP

Master/worker

Loop Parallelism

BSP

Distributed Array Shared-Data

Shared Queue

Shared Hash Table

Barriers

Mutex

Thread Creation/destruction

Process/Creation/destruction

Message passing

Collective communication

Speculation

Transactional memory

Choose your high level structure – what is the structure of my application? Guided expansion

Identify the key computational patterns –what are my key computations?Guided instantiation

Implementation methods – what are the building blocks of parallel programming? Guided implementation

Choose your high level architecture - Guided decomposition

Refine the structure - what concurrent approach do I use? Guided re-organization

Utilize Supporting Structures – how do I implement my concurrency? Guided mapping

Prod

uctiv

ity L

ayer

Eff

icie

ncy

Lay

er

Digital Circuits

Semaphores

Programming Pattern Language 1.0 Programming Pattern Language 1.0 Keutzer& MattsonKeutzer& Mattson

StatusDesign Pattern Language:

We have rough drafts of 80% of the patterns.We have started working with the UIUC patterns team led by Ralph Johnson

Applications to drive our research:Content Based Image Recognition (CBIR)Continuous Speech RecognitionGraph models in CAD

Summary

We are racing from Multicore to Many Cores era

We had made great advance in software, but

Need to do more on Application –Driven Parallel Programming Research