Introduction to Athena++

Kengo TOMIDA (富田賢吾, Osaka University)

James M. Stone (Princeton University)

The Athena++ Development Team

2018/01/29

Winter School @ SHAO

The Athena++ Project

• Static / Adaptive Mesh Refinement (AMR)

• Flexible coordinates: non-uniform spacing, Spherical…

• Support various physical processes (not limited to star formation)

- MHD(CT), self-gravity, radiation, general relativity…

• Highly parallelized： Z-ordering & dynamic scheduling

• Hybrid parallelization: MPI + OpenMP

• High performance： vectorization, memory hierarchy, etc.

• Parallel IO with MPI/HDF5, support standard software (VisIt)

• Easy-to-use, learn, maintain: documents, tests, schools

• Support various architectures; Intel, IBM BG/Q, Xeon Phi etc.

⇒ a common, efficient, easy-to-use, and flexible framework

English：http://princetonuniversity.github.io/athena/

http://princetonuniversity.github.io/athena/

http://princetonuniversity.github.io/athena/

Current Status of Athena++Already available Not public yet Being implemented Planned

HD/MHD

Curvilinear coordinate

SMR/AMR

Special Relativity

General Relativity

(Fixed metric)

MPI + OpenMP

Parallel IO (MPI/HDF)

User-defined functions

Support for Intel, GCC,

IBM, Cray, incl. KNL

Website / tutorial

Non-ideal MHD

(Ohmic, Hall, AD)

Radiation transfer

(Direct ray tracing)

Self-gravity (FFT on

uniform grid)

Chemical reactions

Shearing Box

4th-order scheme

Self-gravity (Multigrid

on uniform grid)

Particles (star / dust)

Self-gravity (Multigrid

on SMR/AMR)

Heterogeneous

parallelization

Developer’s guide

Code paper

General EOS

Post-processing

radiation transfer

(ALMA Science Proj.)

Hybrid PIC Plasma

Radiation transfer

(VTEF+implicit etc.)

Full General Relativity

(dynamic metric)

Red: topics that will be covered in this school

The MHD Basic Equations

𝜕𝜌

𝜕𝑡+ 𝛻 ∙ 𝜌𝒗 = 0 mass conservation

𝜕𝜌𝒗

𝜕𝑡+ 𝛻 ∙ 𝜌𝒗𝒗 − 𝑩𝑩+ 𝑃tot = 𝟎 equation of motion

𝜕𝐸

𝜕𝑡+ 𝛻 ∙ (𝐸 + 𝑃tot)𝒗 − 𝑩(𝑩 ∙ 𝒗) = 0 energy equation

𝑃tot = 𝑃 +𝐵2

2total pressure

𝐸 =𝑃

𝛾−1+

1

2𝜌𝑣2 +

𝐵2

2total energy

𝜕𝑩

𝜕𝑡− 𝛻 × 𝒗 × 𝑩 = 𝟎 induction equation

Note that while we usually use the CGS-Gauss system in

astrophysics, we even simplify it by “the computational unit” for

MHD, in which we renormalize 𝐵2

4𝜋𝜇to 𝐵2.

MHD Scheme

The standard MHD scheme in astrophysics:

• Approximate Riemann solvers (HLLD/Roe/HLLE)

• Second-order piecewise linear reconstruction

(Fourth-order: arXiv astro-ph:1711.07439)

• Constrained Transport (Stone & Gardiner 2009)

• Integrators: 2nd-order van-Leer, 2nd/3rd/4th-order Runge-Kutta

Other features in the current version:

• Spherical and cylindrical coordinates

• Oct-Tree-Block based AMR in any coordinate system

• Special and general relativity (fixed metric)

• Non-ideal MHD effects (Ohmic dissipation / ambipolar diffusion)

• User-defined boundary, source, output functions

• Parallel IO (MPI-IO / HDF5)

• etc…

(Figure from Matsumoto

-san’s presentation)

Flexible CoordinatesAthena++ allows more flexible coordinates including

• Curvilinear coordinates (cylindrical, spherical)

• Non-uniform mesh spacing (e.g. logarithmic)

• Mesh Refinement

Example: star-disk interaction

• large volume (far boundary)

• cells not too elongated

→ logarithmic spacing in r

• low-resolution near the pole

• high-resolution in the disk

→ compressed mesh

and static mesh refinement

(SMR/AMR in curvilinear

coordinates are still rare)

low resolution

high resolution

Typical implementation with fixed ordering:

Dynamic Scheduling: Computation / Communication overlapping

We split the program into small tasks and put them in a queue

→dynamic control with task & dependency lists and status flags.

⇒ Makes the code more flexible, modular and scalable

It is also possible to assign different TaskLists to processes

→ Future extension to heterogeneous parallelization

Dynamic Scheduling with TaskList

Block1

Fluid

Update

Block1

Fluid

ISend

Block2

Field

Update

Block1

Field

Update

Block1

Field

ISend

Block2

Fluid

Update

Block2

Fluid

ISend

Block2

Field

ISend

Block1

Fluid

Wait

Block1

Field

Wait

Block2

Fluid

WaitBlock2 Field Wait

Block1

Fluid

Update

Block1

Fluid

ISend

Update

Block1

Fluid

Update

Block1

Fluid

ISend

Block2

Field

Update

Block1

Field

Update

Block1

Field

ISend

Block2

Fluid

Update

Block2

Fluid

ISend

Block2

Field

ISend

Block1

Fluid

Test

Block1

Field

Test

Block2

Fluid

Test

→ Block2 Field Test

Block1

Fluid

Update

Block1

Fluid

ISend

Block2

Fluid

Update

Athena++ Logo Demonstration

Performance on NAOJ Cray XC30

No significant loss

with SMR (or faster!)

97% scalability btw.

24 → 6144 cores

>x2 faster than Athena

MHD SMR costs

more due to level

boundaries

Performance on IBM BG/Q (Mira)

HD uniform HD SMR 10 levels

as fast as uniform

MHD, 32proc/node (2/core)

MHD, 64proc/node (4/core)

BG/Q is ~ x20 / process or ~ x6 / node slower

than Cray, but is scalable, more nodes available.

*without any machine-specific optimization*

Xeon Phi (preliminary)KNL Xeon Phi 7250 : 1.4GHz x 68 cores (single node, quadrant)

MCDRAM 16GB + DDR4 48GB (Cache mode)

NO code modification + Flat MPI (OpenMP tuning in progress)

MHD (HLLD＋CT, 2nd order), uniform grid, double precision

5.7x105 cells / sec / core

@ 643 / physical core

~71% of 2.6GHz Haswell

1 KNL ~ 46 Haswell cores

cf. peak performance

KNL ~ 3Tflops

XC30 24 cores ~ 1Tflops

→can be more optimized,

but looks promising.

323 / physical core

TACC Stampede-2

Xeon Phi 7250

OmniPath

Molecular Cloud Formation

(Iwasaki, Tomida, Inoue and Inutsuka in prep.)Molecular clouds are formed by compression in converging flows(e.g. galactic spirals, supernovae, galaxy mergers etc.)ISM consists of different thermally stable phases: WNM, CNM and MCMHD simulations including chemical reaction and radiation coolingImpact of magnetic field (the angle between v and B) and metallicity onmolecular cloud formation efficiency, turbulence and its anisotropy, etc.

θ=11°θ=0°

Magnetic Field Dependency

In the aligned cases, the turbulence is highly anisotropic and compression is less efficient.In the misaligned cases, the larger angle results in less efficient compression.The post shock density can be well estimated from balance between ram pressure & magnetic pressure

Note: the aligned regime occupies very small solid angle (<~2%), so probably it is not very important.

Post-K Project: Global DiskProtoplanetary disks should

have rich structures

depending on B-fields and

non-ideal MHD effects

Scientific Goals:

• Turbulence and accretion

• Global structures (e.g. wind,

Dead-zone boundary)

What we need:

• High resolution (MRI)

• Non-ideal MHD effects

(Ohmic , Ambipolar, Hall)

• Dynamic range

• (Radiation, chemistry)

(Simon et al. 2015)

ideal → OD → Hall → AD

Disk Simulations with Athena++

←

Density

→

Plasma Beta

long-term

stability

↓

steady state

(Takasao et al. submitted, arXiv:1801.07245)

Global Magnetic Fields

←α=stress/pressure ↑ vr + field lines

Turbulent transport is rather slow, and

global wind/accretion are important.

Non-ideal MHD simulations in progress.

Rossby Wave Instability Vortexes

Collaboration with T. OnoNon-linear development of shear at pressure bumps

2D cylindrical coordinates with static mesh refinement

Fukagawa+13

HD142527But note that the simulation is gas, while the observation is dust.

Ono et al. submitted to ApJ.

GRMHD Accretion Torus

GRMHD (fixed-metric) simulations (White, Stone & Gammie 2016)any Riemann solver can be used → more accurate (and faster) than other codes using LLF. Costs ～x2.7 (HD) - x4.6 (MHD).

Before Starting The Tutorial

I expect that you have some experience with…

• Linux (or similar UNIX-like OS), especially terminal CUI

• Programming, preferably C/C++

But I do not expect you are familiar with…

• Numerical simulations

• Parallel computing

We will use these software:

• C++ compiler on your laptop (if you run simulations on it)

• Python

• SSH and SFTP client

• gnuplot or any graph software of your favorite

• VisIt (will be used from Day 2)

The Goals of This Tutorial

• Learn the basic of hydrodynamics and MHD

• Learn how to run some simple simulations using Athena++

• Learn how to visualize the simulation results

• Learn how to run parallel simulations

• Learn how to customize the code for your application

Tutorial Website

Now let’s move onto the tutorial.

http://vega.ess.sci.osaka-u.ac.jp/~tomida/athena_eng/

If you notice any problem, either in the tutorial or in the code,

please let us know.

AMR in Athena++

Pros High efficiency

Uniform within block

Use of existing scheme

Simple relations btw levels

Uniform within block

Use of existing scheme

Parallelization by space-

filling curve

Highest efficiency

Logically beautiful

Parallelization by space-

filling curve

Cons Grids are not unique

Non-trivial grid generation

Complex parallelization

Lower efficiency

(depending on patch size)

Performance Issue

Complicated grids

(non-trivial neighbor cell)

Hard to write,read,analyze

Examples Original: Berger & Colella 1989

Orion, PLUTO(Chombo)

CASTRO(Boxlib), Enzo,…

FLASH(PARAMESH)

Peano, Nirvana, SFUMATO,…

RAMSES, ART