parallel & concurrent programming: zpl - umass amherst · zpl parallel array language...

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science

Parallel & Concurrent Programming:

ZPLEmery BergerCMPSCI 691WSpring 2006

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 2

Outline

Previously:MPI – point-to-point & collective

Complicated, far from problem abstraction

OpenMP - parallel directivesLanguage extensions to Fortran/C/C++Questionable semantics, error-prone

Today:Something way better: ZPL

lecture material from ZPL project, UW

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 3

ZPL

Parallel array languageImplicitly parallel

No parallel constructs per se

Very high levelAssignments work at array level, as inA := B + C

Machine independentCompiles to ANSI C +communication library calls (e.g., MPI)

Efficient

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 4

Comparison

Matrix-multiplication:C

triply-nested loop

ZPLdot-product of rows & columnsefficiently implemented on parallel machines

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 5

ZPL Outline

Language overviewRegionsDirectionsParallel array operationsHandling boundary conditions

ZPL programs & performance

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 6

Regions

Key abstraction in ZPL: regionsIndex sets (rows,cols) partition matricesOperate on regions, not indexed items!

rows

columns

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 7

Region Examples

Interior of matrix

Left-most column

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 8

Directions

Directions:Offset vectors used to manipulate regions & array data

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 9

Creating New Regions

Prepositions create new regions:in

Applies direction to select part of region

ofCreates new region outside existing region

atShifts a region by a direction

byCreates new region strided by direction

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 10

Applying Directions

Use “in” to apply direction to region

+ =

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 11

Create Region Outside

Use “of” to create regionoutside existing region

Extends region

+ ==

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 12

Shifting Regions

Use “at” to create new regionshifted by a direction

+ =

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 13

Striding Regions

Use “by” to create new regionstrided by a direction

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 14

Parallel Arrays

Parallel arrays declared over regions

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 15

Computing Over Arrays

Can use regions as modifiers that define computations over arrays:

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 16

Arrays & Communication

Most computations in ZPLdo not involve communication

Exceptions include:ShiftingReductionBroadcastAll-to-all

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 17

Shifting Arrays

@ operator shifts data in directionThis translation induces point-to-pointcommunication

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 18

Reduction

Op<< computes reductionsReduction (tree-style) communication

+<< (sum), *<< (times), min<<, max<<…

For prefix (scan), use op||

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 19

Broadcast (Flooding)

>> (flood) replicates data across dimensions of array

Triggers broadcast operation

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 20

Mapping

Remap (#) moves data between arraysSpecified by “map” arraysBuilt-in Index1, Index2

Index1 = row indices, Index2 = col indices

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 21

Boundary Conditions

Boundary conditions (“corner cases”)Usually tedious, error-proneVery simple in ZPL

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 22

Boundary Conditions

Periodic boundary conditions with wrap

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 23

ZPL Example

Jacobi iteration – replace elements in array with average of four nearest neighbors, until largest change < δ

Consider difficulty of parallelizing with MPI/OpenMP

boundary conditions, etc.

3

1

2

4

2.5 3

1

2

4

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 24

ZPL Example

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 25

ZPL Performance

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 26

ZPL Example: Life

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 27

ZPL Example: Life

UUNIVERSITY OF NIVERSITY OF MMASSACHUSETTSASSACHUSETTS AAMHERST MHERST •• Department of Computer ScienceDepartment of Computer Science 28

The End

Next time:Your turn!Occam & Multilisp