1

John Mellor-Crummey

Cristian Coarfa, Yuri Dotsenko

Department of Computer ScienceRice University

Experiences Building a Multi-platform Compiler for

Co-array Fortran

AHPCRC PGAS Workshop September, 2005

2

Goals for HPC Languages

• Expressiveness

• Ease of programming

• Portable performance

• Ubiquitous availability

3

PGAS Languages

• Global address space programming model

– one-sided communication (GET/PUT)

• Programmer has control over performance-critical factors

– data distribution and locality control

– computation partitioning

– communication placement

• Data movement and synchronization as language primitives

– amenable to compiler-based communication optimization

HPF & OpenMP compilers must get this right

simpler than msg passing

lacking in OpenMP

4

Co-array Fortran Programming Model

• SPMD process images– fixed number of images during execution– images operate asynchronously

• Both private and shared data– real x(20, 20) a private 20x20 array in each image– real y(20, 20)[*] a shared 20x20 array in each image

• Simple one-sided shared-memory communication – x(:,j:j+2) = y(:,p:p+2)[r] copy columns from image r into local columns

• Synchronization intrinsic functions– sync_all – a barrier and a memory fence– sync_mem – a memory fence– sync_team([team members to notify], [team members to wait for])

• Pointers and (perhaps asymmetric) dynamic allocation• Parallel I/O

5

integer a(10,20)[*]

if (this_image() > 1)

a(1:10,1:2) = a(1:10,19:20)[this_image()-1]

a(10,20) a(10,20) a(10,20)

image 1 image 2 image N

image 1 image 2 image N

One-sided Communication with Co-Arrays

6

CAF Compilers

• Cray compilers for X1 & T3E architectures

• Rice Co-Array Fortran Compiler (cafc)

7

Performance comparable to that of hand-tuned MPI codes

Rice cafc Compiler

• Source-to-source compiler

– source-to-source yields multi-platform portability

• Implements core language features

– core sufficient for non-trivial codes

– preliminary support for derived types

• soon support for allocatable components

• Open source

8

Implementation Strategy

• Goals

– portability

– high performance on a wide range of platforms

• Approach

– source-to-source compilation of CAF codes

• use Open64/SL Fortran 90 infrastructure

• CAF Fortran 90 + communication operations

– communication

• ARMCI and GASNet one-sided comm libraries for portability

• load/store communication on shared-memory platforms

9

Key Implementation Concerns

• Fast access to local co-array data

• Fast communication

• Overlap of communication and computation

10

Accessing Co-Array Data

Two Representations

• SAVE and COMMON co-arrays as Fortran 90 pointers– F90 pointers to memory allocated outside Fortran run-time system

– original references accessing local co-array data

• rhs(1,i,j,k,c) = … + u(1,i-1,j,k,c) - …

– transformed references

• rhs%ptr(1,i,j,k,c) = … + u%ptr(1,i-1,j,k,c) - …

• Procedure co-array arguments as F90 explicit-shape arrays– CAF language requires explicit shape for co-array arguments

real :: a(10,10,10)[*]

type CAFDesc_real_3 real, pointer:: ptr(:,:,:) ! F90 pointer to local co-array dataend Type CAFDesc_real_3type(CAFDesc_real_3):: a

11

Performance Challenges

• Problem

– Fortran 90 pointer-based representation does not convey

• the lack of co-array aliasing

• contiguity of co-array data

• co-array bounds information

– lack of knowledge inhibits important code optimizations

• Approach: procedure splitting

12

Procedure Splitting

subroutine f(…)real, save :: c(100)[*]interface subroutine f_inner(…, c_arg) real :: c_arg[*] end subroutine f_innerend interface

call f_inner(…,c(1))

end subroutine f

subroutine f_inner(…, c_arg)real :: c_arg(100)[*]

... = c_arg(50) ...

end subroutine f_inner

subroutine f(…)real, save :: c(100)[*]

... = c(50) ...

end subroutine f

CAF to CAF optimization

Benefits• better alias analysis• contiguity of co-array data• co-array bounds information• better dependence analysis

result: back-end compiler can generate better code

13

Implementing Communication

• x(1:n) = a(1:n)[p] + …

• General approach: use buffer to hold off processor data

– allocate buffer

– perform GET to fill buffer

– perform computation: x(1:n) = buffer(1:n) + …

– deallocate buffer

• Optimizations

– no buffer for co-array to co-array copies

– unbuffered load/store on shared-memory systems

14

Strided vs. Contiguous Transfers

• Problem

– CAF remote reference might induce many small data transfers

• a(i,1:n)[p] = b(j,1:n)

• Solution

– pack strided data on source and unpack it on destination

• Constraints

– can’t express both source-level packing and unpacking for a one-sided transfer

– two-sided packing/unpacking is awkward for users

• Preferred approach

– have communication layer perform packing/unpacking

15

Pragmatics of Packing

Who should implement packing?

• CAF programmer

– difficult to program

• CAF compiler

– must convert PUTs into two-sided communication to unpack

• difficult whole-program transformation

• Communication library

– most natural place

– ARMCI currently performs packing on Myrinet (at least)

16

Synchronization

• Original CAF specification: team synchronization only

– sync_all, sync_team

• Limits performance on loosely-coupled architectures

• Point-to-point extensions

– sync_notify(q)

– sync_wait(p)

Point to point synchronization semantics

Delivery of a notify to q from p

all communication from p to q issued before the notify has been delivered to q

17

Hiding Communication Latency

Goal: enable communication/computation overlap

• Impediments to generating non-blocking communication

– use of indexed subscripts in co-dimensions

– lack of whole program analysis

• Approach: support hints for non-blocking communication

– overcome conservative compiler analysis

– enable sophisticated programmers to achieve good performance today

18

Questions about PGAS Languages

• Performance

– can performance match hand-tuned msg passing programs?

– what are the obstacles to top performance?

– what should be done to overcome them?

• language modifications or extensions?

• program implementation strategies?

• compiler technology?

• run-time system enhancements?

• Programmability

– how easy is it to develop high performance programs?

19

Investigating these Issues

Evaluate CAF, UPC, and MPI versions of NAS benchmarks

• Performance

– compare CAF and UPC performance to that of MPI versions

• use hardware performance counters to pinpoint differences

– determine optimization techniques common for both languages as well as language specific optimizations

• language features

• program implementation strategies

• compiler optimizations

• runtime optimizations

• Programmability

– assess programmability of the CAF and UPC variants

20

Platforms and Benchmarks

• Platforms

– Itanium2+Myrinet 2000 (900 MHz Itanium2)

– Alpha+Quadrics QSNetI (1 GHz Alpha EV6.8CB)

– SGI Altix 3000 (1.5 GHz Itanium2)

– SGI Origin 2000 (R10000)

• Codes

– NAS Parallel Benchmarks (NPB 2.3) from NASA Ames

– MG, CG, SP, BT

– CAF and UPC versions were derived from Fortran77+MPI versions

21

MG class A (2563) on Itanium2+Myrinet2000

Intel compiler: restrict yields factor of 2.3 performance

improvement

UPCstrided comm

28% faster thanmultiple transfers

UPCpoint to point

49% faster than barriers

CAFpoint to point

35% faster than barriers

Higher is better

22

MG class C (5123) on SGI Altix 3000

Intel C compiler: scalar performance

Fortran compiler: linearized array subscripts 30% slowdown compared

to multidimensional subscripts

Higher is better

64

23

MG class B (2563) on SGI Origin 2000

Higher is better

24

CG class C (150000) on SGI Altix 3000

Intel compiler: sum reductions in C 2.6 times slower than Fortran!

point to point

19% faster than

barriers

Higher is better

25

CG class B (75000) on SGI Origin 2000

Intrepid compiler (gcc): sum reductions in C is up to 54% slower than SGI C/Fortran!

Higher is better

26

SP class C (1623) on Itanium2+Myrinet2000

restrict yields 18%

performance improvement

Higher is better

27

SP class C (1623) on Alpha+Quadrics

Higher is better

28

BT class C (1623) on Itanium2+Myrinet2000

UPC: use of restrict boosts the performance 43%

CAF: procedure splitting improves performance 42-60%

UPC: comm. packing 32%

faster

CAF: comm. packing 7%

faster

Higher is better

29

BT class B (1023) on SGI Altix 3000

use of restrict improves

performance 30%

Higher is better

30

Performance Observations

• Achieving highest performance can be difficult

– need effective optimizing compilers for PGAS languages

• Communication layer is not the problem

– CAF with ARMCI or GASNet yields equivalent performance

• Scalar code optimization of scientific code is the key!

– SP+BT: SGI Fortran: unroll+jam, SWP

– MG: SGI Fortran: loop alignment, fusion

– CG: Intel Fortran: optimized sum reduction

• Linearized subscripts for multidimensional arrays hurt!

– measured 30% performance gap with Intel Fortran

31

Performance Prescriptions

For portable high performance, we need …

• Better language support for CAF synchronization

– point-to-point synchronization is an important common case!

– currently only a Rice extension outside the CAF standard

• Better CAF & UPC compiler support

– communication vectorization

– synchronization strength reduction: important for programmability

• Compiler optimization of loops with complex dependences

• Better run-time library support

– efficient communication support for strided array sections

32

Programmability Observations

• Matching MPI performance required using bulk communication

– communicating multi-dimensional array sections is natural in CAF

– library-based primitives are cumbersome in UPC

• Strided communication is problematic for performance

– tedious programming of packing/unpacking at src level

• Wavefront computations

– MPI buffered communication easily decouples sender/receiver

– PGAS models: buffering explicitly managed by programmer