gyrokinetic simulations of turbulence in magnetic fusion ...€¦ · gyrokinetic simulations of...

1

Gyrokinetic simulations of turbulence in magnetic fusion

plasmas

L. Villard, P. Angelino (*) , A. Bottino(1), S. Brunner, S. Jolliet, B.F. McMillan(2), B. Teaca (*) , T.M. Tran, T. Vernay

Centre de Recherches en Physique des Plasmas, EPFL, Switzerland(1) Max Planck IPP, Garching, Germany

(2) Center for Fusion, Space and Astrophysics, U Warwick, UK(*) Supported by the HP2C project

Speedup Meeting, ETH, Zurich, September 7, 2012

2

3

2D sliceSnapshot

Contours of density perturbations

1G grid (3D)(*)

2G particles (5D)

HELIOS 1.5 PFlops

Turbulence in an ITER plasma

(*) more grid points on this 2D slice than pixels

4

Outline

1. IntroductionFusion, tokamaks and turbulence Scalability issues: size matters

2. The ways to performance improvements (HP2C)Profiling, Kernel extraction, Kernel optimization, Code implementation, Performance testsMatch the numerical scheme to the physics: geometry, anisotropy, choice of coordinates

3. First global gyrokinetic turbulence simulations of ITERMain finding: ITER plasma shaping favours turbulence suppression by increasing coupling to zonal flows

4. Conclusion and outlook

5

Outline

1. IntroductionFusion, tokamaks and turbulenceScalability issues: size matters

2. The ways to performance improvementsProfiling, Kernel extraction, Kernel optimization, Code implementation, Performance testsMatch the numerical scheme to the physics: geometry, anisotropy, choice of coordinates

3. First global gyrokinetic turbulence simulations of ITERMain finding: ITER plasma shaping favours turbulence suppression by increasing coupling to zonal flows

4. Conclusion and outlook

6

Magnetic fusionIN: D, Li OUT: He + energy

Abundant fuel, He is neither radioactive nor greenhouse gasLow activation, short period structure materials

D-T mix in the core of the reactorT recycled on site0.3g of T in the core Total site T inventory: 1 – 2 kg

No nuclear chain reactionEven in the case of a maximum conceivable accident no population evacuation is requiredExtremely weak nuclear proliferation risk

No U, no Pu, etc. -- no material in Non Proliferation TreatyResearch was declassified in 1958 -- middle of cold war

7

ITER: the way to fusion

EU+CH, Japan, USA, China, India, South Korea, RussiaO(10G€) construction cost (0.30€ / person / year)

Source:ITER

The plasma will be there (see next p)

You are here

8

Magnetic confinement: tokamak

Larmorradius ρL

linefieldBr

surfacemagnetic drifts curvature and B∇

Trapped particle

Passing particleparticle

trajectory

particles confine toessentialis lines field of transform)

l(rotationatorsionhelicalBv

9

Collisions, turbulenceMagnetic fusion plasmas are very weakly collisional

Temperature ~300’000 x standard atmosphereDensity ~ 1/100’000 x standard atmospherePressure ~ 3 bar

Mean free path (>1km) >> system size (10m)Particles have the time to span a large fraction of the magnetic field configuration before the effect of collisions are felt

Geometry of the magnetic field affects not only single particle motion, but also collective effects like turbulenceKinetic effects are important (i.e. fluid models insufficient)

The main losses (heat flux) are due to turbulencei.e. much larger than collision-induced

Degradation of the quality of confinement large enough system sizes are required for achieving economical fusion

10

Timescales in the ITER plasma

]s[t

machine lifetime 1 shot energy

confinement

turbulence

ion cyclotron

electron cyclotron

810−510−010310810 1210−

Physics spans several orders of magnitudeDirect Numerical Simulation (DNS) of “everything” is unthinkable

Need to separate timescales using approximations

11

Net energy transfer from the wave to the particles if

Collisionless Landau damping

0/ <∂∂ vf

Kinetic effects: wave-particle interaction

Surfers with velocity just below the phase velocity of the wave will be accelerated

-> momentum and energy transfer

Surfers with velocity too different from the phase velocity of the wave will not ride the wave

General: distribution function in 6D phase spaceTo be solved with consistent electromagnetic fields

);,( tvxf rr

f

vk/ω

),(),,( txBtxE rrrr

12

Turbulence and transportFinite system size gradients of T, n, BWhen these gradients exceed threshold values, instabilities develop and saturate to a turbulent state heat, particle, momentum transport

Low frequencies (ω << ωci)Small scales perpendicular to B (~ρs)Large scales parallel to B (~ system size)

Strong anisotropy(~1000 for ITER)

Average out the fast motion of the particle around the guiding centerFast parallel motion, slow perpendicular motion (drifts) of the gyro-centre

phase space dimension reduction 6D ---> 5D

13

Gyrokinetic equations

),,( // μvRfs

rdistribution function of species s in 5D phase space

),( '//

sssss ffC

Rf

dtdv

Rf

dtRd

tf

=∂∂

+∂∂⋅+

∂∂

rr

r

),(fct...),,(fct... // Adt

dvAdtRd rrr

φφ ==

),( Ar

φ solution of Maxwell’s equations, with ρ, j obtained as moments of fs

PDE, 3D

advection-diffusionPDE, 5D

equations of motion(orbits)

ODE, 5D

14

Solving GK equations: ORB5 code

5D phase space: Particle-In-Cell (PIC)3D field solver: cubic B-spline finite elementsTime stepping: Runge-Kutta 4th orderSampling noise reduction measures

Control variates, δf schemeField-aligned Fourier filter: eliminate unphysical modes

Sampling noise control measuresMomentum- and zonal flow-conserving Krook-like operatorCoarse graining scheme

Parallelization schemeDomain decompositionDomain cloning

Developed at CRPPContributions from Max-Planck IPP and U Warwick

15

8/10ITERsize

4/10

Turbulence: system size matters

2/10

1/10ITERsize

16

Size matters

effective system size

heat

tran

spor

t

ORB5: Lagrange-PIC [TM Tran et al] – GENE: Euler [F Jenko et al]

now ITER

17

Scalability with system size

With a PIC code, toroidal coordinates:3D field grid ~ (size)3

Nparticles ~ (size)3

Ntimesteps ~(size)1

Computational cost ~(size)4

With field-aligned Fourier filter:3D field grid ~ (size)3

Nparticles ~ (size)2

Ntimesteps ~(size)0

Computational cost ~(size)2 for local (particle) operations~(size)3 for (grid) communications

Remove that bottleneck limiting // scalability for large nr of procs and large system size

Identify, remove or reduce non-scalable communications

18

Outline

1. IntroductionFusion, tokamaks and turbulenceScalability issues: size matters

2. The ways to performance improvements (HP2C)Profiling, Kernel extraction, Kernel optimization, Code implementation, Performance testsMatch the numerical scheme to the physics: geometry, anisotropy, choice of coordinates

3. First global gyrokinetic turbulence simulations of ITERMain finding: ITER plasma shaping favours turbulence suppression by increasing coupling to zonal flows

4. Conclusion and outlook

19

Parallelization scheme – ORB5

Domain decomposition (toroidal direction)

Domain Cloning:field quantities (density, potential) are replicated

shift

globalsum

Npes = Nd*Nc

particles

field

s

Nd = 100 ~ 1000

Nc

= 1

~ 10

0

// Data transpose (Fourier)

20

ORB5 code profiling

In-depth performance analysis by A. Tineo (CSCS) on Cray XE6 (Palu). Scalasca, Tau.Main findings: data locality & communications bottlenecks

CSCS

CRPP

--localField solverFourier+grid

grid

grid

domains

clones

global

global

// data transposeradial reassembly

grid to/from Fourier

-grid

-clones

localglobal

assignsum

particles to/from grid

-particles

-domains

localglobal

pushshift

particles

21

Load imbalance for global sums over the clones

ORB5 code profiling

3d system topology of processors (Palu 2048 procs)

Training of CRPP staff to use profiling toolsCSCS

CRPP

4 clones of the same grid domain

22

Kernel extraction

Bottlenecks identified in some global communications Essentially non-scalable with number of clonesExtract subset of routines from ORB5 that deal with these operations: essentially 3D grid data communications

CSCS

CRPP

At CSCS: Adrian Tineo, Gilles Fourestey, Neil Stringfellow

23

Kernel performance improvement

Transposition of cartesian MPI communicator put clones on nodeReplace all_reduce with all_gather in // data transpose operations (radial domain decomposition)Kernel performance improved by a factor of 5 (for 24 clones)Kernel scalability with clones drastically improvred

Cray XE6 Palu160 domains3-24 clones(960x1980x160) grid

old

new

24

Improvements implemented in ORB5 (1)

Strong scalability with clones is improved from 72% to 90%(32768 procs) for a grid (512x1024x1024), #particles=6G

Cray XE6 (Monte Rosa)Clones 8-32

25

For a 4 x larger grid (1024x2048x1024) the improvement is a factor of 2 at 32768 procs. Case relevant for ITER-size simulation of ITG turbulence

Improvements implemented in ORB5 (2)

Cray XE6 (Monte Rosa)Clones 324x larger grid

26

Petascale Computing

Improvements brought in the frame of the HP2C project helped us to get access to a 1.5 PetaFlop platform (HELIOS, IFERC-CSC, Rokkasho, Japan) dedicated to magnetic fusion research (EU+CH & Japan). [#12 on June 2012 top500]ORB5 was one among 4 codes designated as high-level benchmark used for the acceptance tests of HELIOS (capability testing)Demonstrated strong scalability from 16k to 64k cores: speedup 3.36 (84% parallel efficiency)Nominated as one of 4 “LightHouse Projects” with several million cpu-hours allocation grantedAllowed us to perform the first global gyrokinetic turbulence simulation of ITER

27

Outline

1. IntroductionFusion, tokamaks and turbulence Scalability issues: size matters

2. The ways to performance improvements (HP2C)Profiling, Kernel extraction, Kernel optimization, Code implementation, Performance testsMatch the numerical scheme to the physics: geometry, anisotropy, choice of coordinates

3. First global gyrokinetic turbulence simulations of ITERMain finding: ITER plasma shaping favours turbulence suppression by increasing coupling to zonal flows

4. Conclusion and outlook

28

ITG turbulence in ITER

29

30

31

Zonal Flows and turbulence

Turbulence zonal flows (ZF) shearing turbulent eddies turbulence suppression

Self-organizationRadial structure of alternating bands of ZFs, related to regions of suppressed turbulence

ITER plasma shapingmore effective turbulence suppression by ZFs than circular shaped plasmas(see next slide)

32

Geometry, Zonal Flows and Turbulence

ITERcircular

33

Zonal flow dynamics - ITER

Contours of ZF shearing rate vs radius (s) and time

edge

core

34

Zonal flow dynamics - circular

Contours of ZF shearing rate vs radius (s) and time

core

edge

35

Turbulent heat transport

Drastic reduction of turbulent ion heat transport with the ITER shape

36

Outline

1. IntroductionFusion, tokamaks and turbulence Scalability issues: size matters

2. The ways to performance improvements (HP2C)Profiling, Kernel extraction, Kernel optimization, Code implementation, Performance testsMatch the numerical scheme to the physics: geometry, anisotropy, choice of coordinates

3. First global gyrokinetic turbulence simulations of ITERMain finding: ITER plasma shaping favours turbulence suppression by increasing coupling to zonal flows

4. Conclusion and outlook

37

Conclusion and outlook

First principles based “direct” numerical simulations of turbulence in magnetized plasmas remain a challenge – and impact fusion researchOur codes are ready for PetaFlop range platforms. The HP2C project was instrumental in bringing us to this important stepPresent and future and works: I/O, data analysis & visualization (field and particle data), …First global gyrokinetic turbulence simulations of ITER have predict a drastic reduction of turbulent heat transport due to plasma shaping – to be confirmed with more complete physics