coarrays in gnu fortran - sea · introduction coarray fortran (also known as caf) is a syntactic...

50
Coarrays in GNU Fortran Alessandro Fanfarillo [email protected] June 24th, 2014 Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 1 / 50

Upload: leliem

Post on 16-Apr-2018

230 views

Category:

Documents


3 download

TRANSCRIPT

Coarrays in GNU Fortran

Alessandro Fanfarillo

[email protected]

June 24th, 2014

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 1 / 50

Introduction

Coarray Fortran (also known as CAF) is a syntactic extension of Fortran95/2003 which has been included in the Fortran 2008 standard.

The main goal is to allow Fortran users to realize parallel programswithout the burden of explicitly invoke communication functions ordirectives (MPI, OpenMP).

The secondary goal is to express parallelism in a “platform-agnostic” way(no explicit shared or distributed paradigm).

Coarrays are based on the Partitioned Global Address Space model(PGAS).

Compilers which support Coarrays:

Cray Compiler (Gold standard - Commercial)

Intel Compiler (Commercial)

Rice Compiler (Free - Rice University)

OpenUH (Free - University of Houston)

G95 (Coarray support not totally free - Not up to date)

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 2 / 50

PGAS Languages

The PGAS model assumes a global memory address space that is logicallypartitioned and a portion of it is local to each process or thread.

It means that a process can directly access a memory portion owned byanother process.

The model attempts to combine the advantages of a SPMD programmingstyle for distributed memory systems (as employed by MPI) with the datareferencing semantics of shared memory systems.

Coarray Fortran.

UPC (upc.gwu.edu).

Titanium (titanium.cs.berkeley.edu).

Chapel (Cray).

X10 (IBM).

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 3 / 50

Coarray concepts

A program is treated as if it were replicated at the start of execution,each replication is called an image.

Each image executes asynchronously.

An image has an image index, that is a number between one and thenumber of images, inclusive.

A Coarray is indicated by trailing [ ].

A Coarray could be a scalar or array, static or dynamic, and ofintrinsic or derived type.

A data object without trailing [ ] is local.

Explicit synchronization statements are used to maintain programcorrectness.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 4 / 50

Coarray example

real , dimension (10), codimension [*] :: x, y

integer :: num_img , me

num_img = num_images ()

me = this_image ()

! Some code here

x(2) = x(3)[7] ! get value from image 7

x(6)[4] = x(1) ! put value on image 4

x(:)[2] = y(:) ! put array on image 2

sync all

x(1:10:2) = y(1:10:2)[4] ! strided get from image 4

sync images (*)

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 5 / 50

Coarray support

The most mature, efficient and complete implementation is providedby Cray.

Cray runs on proprietary architecture.

Intel provides a CAF implementation but it works only on Linux andWindows.

Intel CAF has two modes: shared and distributed. For the distributedmode the Intel Cluster Toolkit is required.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 6 / 50

GNU Fortran Libcaf

GFortran uses an external library to support Coarrays (libcaf).

Currently there are three libcaf implementations:

MPI Based

GASNet Based

ARMCI Based (not updated)

The idea is to provide the MPI version as default and the GASNet/ARMCIas “expert version”.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 7 / 50

GASNet

GASNet (Global Address Space Networking) provided by UC Berkeley.

Efficient Remote Memory Access operations.

Native network communication interfaces.

Useful features like Active Messages and Strided Transfers.

Main issue: GASNet requires an explicit declaration of the total amountof remote memory to use (allocatable coarrays?).

Secondary issue: GASNet requires a non-trivial configuration during theinstallation and before the usage.

The GASNet version of Libcaf has not been deeply studied yet.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 8 / 50

MPI Version - Current status

Supported Features:

Coarray scalar and array transfers (efficient strided transfers)

Synchronization

Collectives

Vector subscripts

Unsupported Features:

Derived type coarrays with non-coarray allocatable components

Atomics

Critical

Error handling

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 9 / 50

Hardware

Eurora: Linux Cluster, 16 cores per node, Infiniband QDR QLogic(CINECA).

PLX: IBM Dataplex, 12 cores per node, Infiniband QDR QLogic(CINECA).

Yellowstone/Caldera: IBM Dataplex, 16 cores per node, InfinibandMellanox (NCAR).

Janus: Dell, 12 cores per node, Infiniband Mellanox (CU-Boulder).

Hopper: Cray XE6, 24 cores per node, 3-D Torus Cray Gemini(NERSC).

Edison: Cray XC30, 24 cores per node, Cray Aries (NERSC).

Only Yellowstone and Hopper used for this presentation.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 10 / 50

Software

On Hopper:

Cray: Cray (CCE) 8.2.1

GFortran: GCC-4.10 experimental (gcc branch) + Mpich/6.0.1

On Yellowstone:

Intel: Intel 14.0.2 + IntelMPI 4.0.3

GFortran: GCC-4.10 experimental (gcc branch) + MPICH IBM opt.Mellanox IB

For Intel CAF, every coarray program runs in CAF Distributed Mode.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 11 / 50

Test Suite Description

CAF Psnap (Dan Nagle)

It is a network noise analyzer. It can be useful for a statistical study ofthe communication between processes.

EPCC CAF Micro-benchmark suite (Edinburgh University)

It measures a set of basic parallel operations (get, put, strided get,strided put, sync, halo exchange).

Burgers Solver (Damian Rouson)

It has nice scaling properties: it has a 87% weak scaling efficiency on16384 cores and linear scaling (sometimes super-linear).

CAF Himeno (Prof. Ryutaro Himeno - Bill Long - Dan Nagle)

It is a 3-D Poisson relaxation.

Distributed Transpose (Bob Rogallo).

It is a 3-D Distributed Transpose.

Dense matrix-vector multiplication (MxV).

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 12 / 50

Tests execution

Every test has been run on Yellowstone/Caldera and Hopper.

On Yellowstone we are able to run only Intel and GFortran.

On Hopper we are able to run only GFortran and Cray.

GFortran runs with whatever MPI implementation is available.

On GFortran we used only the -Ofast flag.

On Intel we used the following flags: -Ofast -coarray -switchno launch

On Cray we used the -O3 flag. We also loaded craype-hugepages2Mand set XT SYMMETRIC HEAP SIZE properly.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 13 / 50

CAF Psnap - Yellowstone 32 cores

This test shows the performance of the communication layer underthe Coarrays interface.

Psnap sends several times a single integer between “peer” processes(0-16, 1-17, ..).

Using Intel CAF (and thus IntelMPI) transferring a single integertakes on average 1.018 msecs.

Using GFortran/MPI (and thus IBM MPI) transferring a singleinteger takes on average 1.442 msecs.

The time distribution is different (Gamma for Intel, Normal for IBMMPI).

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 14 / 50

CAF Psnap - Hopper 48 cores

Using Cray transferring a single integer takes on average 1.021 msecs.

Using GFortran/MPI transferring a single integer takes on average2.996 msecs.

Both distributions look like heavy-tail.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 15 / 50

EPCC CAF Micro-benchmark suite

Single point-to-point: image 1 interacts with image n.

Put, Get, Strided Put, Strided Get.

Multiple point-to-point: image i interacts with image i+n/2.

Put, Get, Strided Put, Strided Get.

Synchronization: several pattern of synchronization. (Not shown)

Sync all, sync images.

Halo swap: exchange of the six external faces of a 3D array. (Notshown)

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 16 / 50

Latency/Bw Single Put 16 cores Yellowstone (Small)

0

2e-05

4e-05

6e-05

8e-05

0.0001

0.00012

1 2 4 8 16 32 64

Tim

e (

sec)

Block size (byte)

Latency put 16 cores (no network)

GFor/MPIIntel

0

5

10

15

20

25

30

35

40

1 2 4 8 16 32 64

MB

/sec

Block size (byte)

Bandwidth put 16 cores (no network)

GFor/MPIIntel

Latency: lower is better - Bandwidth: higher is better

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 17 / 50

Latency/Bw Single Put 16 cores Yellowstone (Big)

1e-05

0.0001

0.001

0.01

0.1

1

10

128

256

512

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

2097152

4194304

Tim

e (

sec)

Block size (byte)

Latency put 16 cores (no network)

GFor/MPIIntel

1

10

100

1000

10000

128

256

512

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

2097152

4194304

MB

/sec

Block size (byte)

Bandwidth put 16 cores (no network)

GFor/MPIIntel

Latency: lower is better - Bandwidth: higher is better

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 18 / 50

Bw Multi Put 16 cores Yellowstone

0.1

1

10

100

1000

10000

1 2 4 8 16

32

64

128

256

512

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

2097152

4194304

MB

/sec

Block size (byte)

Bandwidth multi put Yellowstone 16 cores (no network)

GFor/MPIIntel

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 19 / 50

Bw Difference Single/Multi Put 16 cores Yellowstone

0.1

1

10

100

1000

10000

1 2 4 8 16

32

64

128

256

512

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

2097152

4194304

MB

/sec

Block size (byte)

Bw difference put Yellowstone 16 cores single/multi (no network)

Diff_GFor/MPIDiff_Intel

A constant trend means that the network contention does not impact (toomuch) the performance.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 20 / 50

Bw Single Strided Put 16 cores Yellowstone

1

10

100

1000

10000

1 2 4 8 16 32 64 128

MB

/se

c

Stride size (byte)

Bandwidth strided put 16 cores (no network)

GFor/MPIIntel

GFor/MPI_NS

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 21 / 50

Bandwidth Single Put 24 cores Hopper (Small)

0

20

40

60

80

100

120

140

160

180

200

1 2 4 8 16

32

64

MB

/sec

Block size (byte)

Bandwidth put Hopper 24 cores (no network)

GFor/MPICray

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 22 / 50

Bandwidth Single Put 24 cores Hopper (Big)

0

500

1000

1500

2000

2500

3000

3500

128

256

512

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

2097152

4194304

MB

/sec

Block size (byte)

Bandwidth put Hopper 24 cores (no network)

GFor/MPICray

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 23 / 50

Bw Multi Put 24 cores Hopper

0.1

1

10

100

1000

10000

1 2 4 8 16

32

64

128

256

512

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

2097152

4194304

MB

/sec

Block size (byte)

Bandwidth multi put 24 cores Hopper (no network)

GFor/MPICray

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 24 / 50

Bw Difference Single/Multi Put 24 cores Hopper

0.1

1

10

100

1000

10000

1 2 4 8 16

32

64

128

256

512

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

2097152

4194304

MB

/sec

Block size (byte)

Bw difference single/multi put 24 cores Hopper (no network)

Diff_GFor/MPIDiff_Cray

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 25 / 50

Bw Single Strided Put 24 cores Hopper

10

100

1000

10000

1 2 4 8 16 32 64 128

MB

/se

c

Stride size (byte)

Bandwidth strided put 24 cores Hopper (no network)

GFor/MPICray

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 26 / 50

Latency/Bw Single Put 32 cores Yellowstone

1e-05

0.0001

0.001

0.01

0.1

1

10

100

1000

1 2 4 8 16 32 64 128 256 512

Tim

e (

sec)

Block size (byte)

Latency put 32 cores

GFor/MPIIntel

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

10

100

1000

1 2 4 8 16 32 64 128 256 512

MB

/sec

Block size (byte)

Bandwidth put 32 cores

GFor/MPIIntel

Latency: lower is better - Bandwidth: higher is better

Note: Latency is expressed in seconds and Bw in MB/Sec.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 27 / 50

Latency/Bw Intel 32 cores Yellowstone

With 2 nodes we have a weird behavior from the Intel compiler.The benchmark gets an internal error during the subput test for everyconfiguration of nodes.

blksize (Bytes) latency (sec) bwidth (MB/s)

1 0.837 0.9133E-05

2 1.67 0.9178E-05

4 3.35 0.9157E-05

8 6.66 0.9159E-05

16 13.4 0.9156E-05

32 26.8 0.9155E-05

64 53.4 0.9148E-05

128 107. 0.9150E-05

256 214. 0.9149E-05

512 428. 0.9141E-05

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 28 / 50

Bw Single Put 48 cores Hopper (small)

0

5

10

15

20

25

30

35

40

45

50

1 2 4 8 16 32 64

MB

/sec

Block size (byte)

Bandwidth put Hopper 48 cores

GFor/MPICray

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 29 / 50

Bw Single Put 48 cores Hopper (big)

0

500

1000

1500

2000

2500

3000

128

256

512

1024

2048

4096

8192

16384

32768

65536

13107

2

26214

4

52428

8

10485

76

20971

52

41943

04

MB

/sec

Block size (byte)

Bandwidth put Hopper 48 cores

GFor/MPICray

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 30 / 50

Bw Single Strided Get 48 cores Hopper

10

100

1000

10000

1 2 4 8 16 32 64 128

MB

/se

c

Stride size (byte)

Bandwidth strided get 48 cores Hopper

GFor/MPICray

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 31 / 50

Bw Single Strided Put 48 cores Hopper (2)

During a strided transfer Cray sends a predefined amount of data(perhaps elem-by-elem).

GFortran uses the MPI Derived Data Types.

Only the bandwidth is not enough for a fair comparison.

Derived data types require a lot of memory (a Yellowstone’s computenode does not have enough memory for running the entirebenchmark).

A good way to compare the two CAF implementation is through aTime/Memory trade-off.

We plan to improve the strided transfer very soon.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 32 / 50

BurgersCAF

It uses Coarrays mainly for scalar transfers.

It communicates with neighbour images which are usually placed onthe same node.

The nature of the algorithm influences the performance a lot.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 33 / 50

BurgersCAF - Yellowstone

00:00

00:15

00:30

00:45

01:00

01:15

01:30

01:45

02:00

02:15

02:30

02:45

03:00

16 32 64 128 256

Tim

e (

min

:sec)

# Cores

BurgersCAF on Yellowstone

GFor/MPIIntelCAF

(Lower is better)Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 34 / 50

BurgersCAF - Hopper

00:00

00:30

01:00

01:30

02:00

02:30

03:00

03:30

04:00

04:30

24 48 192 384

Tim

e (

min

:sec)

# Cores

BurgersCAF on Hopper

GFor/MPICray

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 35 / 50

CAF Himeno

It uses the Jacobi method for a 3D Poisson relaxation.

Intel requires more than 30 minutes to complete the test with 64cores.

Cray requires several tuning arrangements in order to run (the 32cores test has been run with 8 images on each node).

GFortran was the easiest to run.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 36 / 50

CAF Himeno - Yellowstone

0

5000

10000

15000

20000

25000

30000

16 32

MF

LO

PS

Cores

Performance CAFHimeno Yellowstone

GFor/MPI

Intel

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 37 / 50

CAF Himeno - Hopper

0

5000

10000

15000

20000

25000

30000

16 32

MF

LO

PS

Cores

Performance CAFHimeno Hopper

GFor/MPICray

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 38 / 50

Distributed Transpose

3-D Distributed Transpose.

It is performed sending element-by-element.

The problem size has been fixed at 1024,1024,256.

It is a good communication test case.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 39 / 50

Distributed Transpose - Yellowstone

0

10

20

30

40

50

60

70

16 32 64

Tim

e (

sec)

Cores

Performance Distributed Transpose Yellowstone

GFor/MPIIntel

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 40 / 50

Distributed Transpose - Hopper

0.5

1

1.5

2

2.5

3

3.5

16 32 64

Tim

e (

sec)

Cores

Performance Distributed Transpose Hopper

GFor/MPICray

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 41 / 50

Distributed MxV

A group of rows is sent to each image.

The usual array syntax has been used (it implies strided arraytransfer).

The bottleneck is communication (broadcast and gather).

With a single MxV the computational part is too small for gettingbenefits from a parallel approach.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 42 / 50

Distributed MxV - Executions

Yellowstone 16 cores (single node)

GFor/MPI Intel

0:40 (m:s) 5:09 (m:s)

Hopper 16 cores (single node)

GFor/MPI Cray

1:54 (m:s) 0:08 (m:s)

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 43 / 50

Conclusions - GFortran vs. Intel

Intel shows better performance than GFortran only during scalartransfers within the same node.

Intel pays a huge penalty when it uses the network (multi node).

GFortran shows better performance than Intel on array transferswithin the same node.

GFortran shows better performance than Intel in every configurationwhich involves the network.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 44 / 50

Conclusions - GFortran vs. Cray

Cray has better performance than GFortran within the same node.

Cray has better performance than GFortran during contiguous arraytransfer on multi node (using the network).

GFortran has better performance than Cray for strided transfers onmulti node. (!!!)

Cray requires some tuning (env vars and modules).

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 45 / 50

Conclusions

GFortran provides a stable, easy to use and efficient implementationof Coarrays.

GFortran provides a valid and free alternative to commercialcompilers.

Coarrays in GFortran can be used on any architecture able to compileGCC and a standard MPI implementation.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 46 / 50

Future developments

On the compiler side the multi image Coarray support is already onGCC 4.10 trunk (It also supports OpenMPv4).

On the library side we plan to release the MPI version very soon.

Improve the strided transfer support and add all the missing featuresexpected by the standard.

Improve the experimental GASNet version.

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 47 / 50

The Team

Development Team:

Tobias Burnus

Alessandro Fanfarillo

Support Team:

Valeria Cardellini

Salvatore Filippone

Dan Nagle

Damian Rouson

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 48 / 50

Acknowledgment

CINECA (Grant HyPSBLAS)

Google (Project GSoC 2014)

UCAR/NCAR

NERSC (Grant OpenCoarrays)

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 49 / 50

Thanks

Suggestions/Feedback/Questions?

Alessandro Fanfarillo Coarrays in GFortran June 24th, 2014 50 / 50