High-performance computationalmodeling of space plasmasIlja Honkonenilja.honkonen@fmi.fi

Finnish Meteorological Institute31.7.2019

Outline

Motivation

Introduction to modeling

Introduction to plasma

Approximations of plasma

I. Honkonen: Modeling space plasma 2 / 24

Supercomputer statistics from top500.org

• Unimaginableincrease in CPU,memory and storagefor 30+ years• Similar increase

in scope andaccuracy ofsimulations

• Gaming phones nowabout as fast as #1in 1993• Special software

solutions for takingadvantage of fastestcomputers

Massively parallel simulation development

• Can get results in 1 week instead of 1000 years• Vectorization gives < 30x speedup (in 2020)

• Takes advantage of vector/multimedia CPU instructions• Same operation executed simultaneously for several data items

• Threading gives < 100x speedup (in 2020)• Not available 20 years ago• Several ”copies” of same program executed simultaneously• Memory and part of CPU caches shared between copies

• Message Passing Interface gave ∼ 1000x speedup in 1993• Compiler cannot use MPI for you• Can replace threading to some extent

• Above factors almost multiplicative when used together properly

I. Honkonen: Modeling space plasma 4 / 24

Massively parallel simulation development

• MPI is de facto standard for high-performance computing• Networked machines execute copies of same program• Originally very simple application programming interface, e.g.:

• Process 0: send N doubles from this memory address to process 1• Process 1: receive N doubles from process 0 to this memory address

• MPI (and admin) take care of nasty details• Setting up network connections• Preparing shared storage for file I/O• Connecting/logging into all machines• Starting the program on all machines

• Various helper functions for writing parallel programs• Point to point communication: see above• Collective communication & computation: all/one to one/all transform/reduce• Parallel reading & writing of files• Newest versions: one sided communication, fault tolerance, etc

I. Honkonen: Modeling space plasma 5 / 24

Hardware latencies

• Effective use ofcaches and memorycan provide orders ofmagnitude speedup• On top of MPI,

threading,vectorization

• Apparent even wheniterating over i or jfirst in a[i][j]

Task approx. time (ns)execute one instruction 0.3fetch from L1 cache 0.5fetch from L2 cache 7fetch from L3 cache 20mutex (un)lock 25fetch from main memory 100compress 1 kB with Snappy 3 000send 2 kB over 1 Gb/s network 20 000read 1 MB from memory 250 000

Adapted from norvig.com/21-days.html andgist.github.com/jboner/2841832

I. Honkonen: Modeling space plasma 6 / 24

Basic computational modeling of continuum systems

• Analytic solution not available• Impossible or not known

• Equations must be discretized• For processing with IEEE floating point numbers

• Numerical ”features” (e.g. sin(x) != sin(y) even if x == y)• Due to finite resolution & precision, concurrency,

register allocation, bugs• Artificial diffusion, resistivity, viscosity, heating,

acceleration, ∇ · B, etc.

• Same basic principle as Conway’s Game of Life

en.wikipedia.org/wiki/File:CFD Shuttle.jpg

I. Honkonen: Modeling space plasma 7 / 24

Game of Life

Zero player game and cellular automaton / latticemodel invented by John Conway in 1970• Played on rectangular grid• Each cell is either alive or dead• At every step, for each cell, number of live

neighbors is counted• 8 nearest neighbors

• Rules for cell state in next step:• if 3 neighbors alive cell is alive• if 2 neighbors alive cell state doesn’t change• otherwise cell is dead

Glider from Wikipedia

I. Honkonen: Modeling space plasma 8 / 24

Game of Life

All basic features of plasma/gas/fluid simulations• Each cell stores: state and its change• Change calculated from data within some distance

• State update after change in neighbors calculated• MPI programs must exchange states near process

boundaries• Details vary by problem & algorithm

• Fraction of computation to communication• Larger neighborhood ∼ more accurate solution in space

• Can use fewer simulation cells: longer timesteps, fastertime to solution, problem fits into memory

• More states ∼ more accurate solution in time• Can use longer timesteps: faster time to solution, less

diffusion

Glider from Wikipedia

I. Honkonen: Modeling space plasma 9 / 24

Simple model: advection

• Transport of matter in velocity field• Fluid, gas, smoke, ash, sand, dust, salt• Transported by wind, ocean current, pipe, ice

• Velocity may change due to advection• Shallow water or hydrodynamic eqs.

• Various levels of complexity, accuracy, speed,resource use• Cell, edge, vertex centered velocity• V interpolated from N ”nearest” values

I. Honkonen: Modeling space plasma 10 / 24

Simulation frames of reference

Laboratory frame: Eulerian• Simulation grid ”stationary”• Matter moves between grid cells• Numerical diffusion

• Unless time step and velocity exactlymatch cell size

• Constant resolution (by default)• Easier to implement than Lagrangian

Fluid frame: Lagrangian• Parcels of matter move in simulated

volume• ”Dynamic” resolution• Value interpolated from N nearest

parcels• Bad statistics in regions with few

parcels

Combinations of above possible in space, time, species, etc.

I. Honkonen: Modeling space plasma 11 / 24

Constraints of time to solution

• Spatial resolution ∼ smallest interesting/required feature• Worse than e gyroradius will neglect most electron physics• Ditto for ions (gyroradius ∼1000x e rg)• Neglecting both leads to ∼ magnetohydrodynamics

• Temporal resolution ∼ fastest interesting/required feature• As above but for gyrofrequencies of particles around B• Laser matter/plasma interactions need at most ∼ 10−15s timestep

• Performance ∼ solved cells and/or particles per second• Optimization critical, algorithm also important• ∼ 106 cells or ∼ 107 particles / core / s

I. Honkonen: Modeling space plasma 12 / 24

Constraints of time to solution

• Time to solution ∼ (lenght of simulated time / temporal) · (spatial / performance)• Or number of timesteps · time to solve one step

• Constraints in Earth’s magnetosphere• Each parameter can easily vary by 10x• Simulation length 10 h• e: rg ∼ 1 km fg ∼ 10 kHz, i: rg ∼ 1000 km fg ∼ 10 Hz• Volume ∼ 107 · 106 · 106 km3

• Number of grid cells with e physics ∼ 1019, i physics ∼ 1010, MHD ∼ 107

• Time to solution using 106 core system: e ∼ 108 years, i ∼ 10 h, MHD 1 min

• Perfectly scalable system/program impossible though

I. Honkonen: Modeling space plasma 13 / 24

Plasma versus other types of matter

• 99.999 % of visible matter in universe• Gyration of charged particles around

magnetic field direction• Collective electromagnetic behavior• Usually collisionless (e.g. between

planets, stars, galaxies)• Several temperatures and bulk

velocities simultaneously!

• Several wave types and speedssimultaneously!• Study is art of approximation

en.wikipedia.org/wiki/File:Charged-particle-drifts.svgI. Honkonen: Modeling space plasma 14 / 24

Examples

• Above: 2d MHD simulation of Jupiter+Saturnshowing log density• Right: 1r+3v-dimensional hybrid particle-in-cell

simulation of ion two-stream instability showingparticle count

Accurate equations for non-relativistic collisionless plasma

Maxwell’s equations for E and B fields• ∇ · B = 0

• ∇ · E = ρ/ε0

• ∂B/∂t = −∇× E• ∂E/∂t = (∇× B/µ0 − J)/ε0

where ρ is charge density, J current density.

Lorentz force for particle acceleration• a = (q/m)(E+ v× B)

where q is particle charge, m particlemass, v particle velocity

Too detailed even today for modeling most systems...

I. Honkonen: Modeling space plasma 16 / 24

Approximations to plasma 1

• Cannot model every particle so use collisionless kinetictheory (Vlasov)...• ∂f(t, r, v)/∂t = −v · ∂f/∂r− (q/m)(E+ v× B) · ∂f/∂v• Implementable as e.g. 6-dimensional advection

equation for particles• Grid resolution directly constrains available physics

• Simulation state: #/m3/(m/s)3 = #s3/m6, where # isnumber of particles

• ...or represent many real particles as one simulationparticle• Particle/cloud/etc in cell approaches

• Another way to implement Vlasov equation• In magnetospheric modeling ”ion” mass ∼ 1 mg

• Charge to mass ratio still correct

I. Honkonen: Modeling space plasma 17 / 24

Approximations to plasma 2

• Cannot model all electromagnetic radiation so usually don’t• Almost always much larger constraint than e on time to solution• Few systems where EM radiation essential

• Laser ignition, supernova, large accretion disks• Several systems where EM interesting but not modeled directly

• Radio propagation in/through ionosphere• UV, light, IR in/through atmosphere

• Use ∂E/∂t = 0 and solve E e.g. from kinetic eq. of e & i• Assumes: ne = ni, me << mi, thermal equilibrium, etc.• Technically only transverse part ∂ET/∂t = 0

• E = −Vi × B+ J/σ+ (ne)−1(J× B−∇ ·Pe +me/e · ∂J/∂t)• Electron pressure is a matrix

I. Honkonen: Modeling space plasma 18 / 24

Generalized Ohm’s law

• E = −Vi × B+ J/σ+ (ne)−1(J× B−∇ ·Pe +me/e · ∂J/∂t)• With most important terms for space plasma

• ne = ni: removes most electrostatic instabilities, ∇ · J = 0

• ∂ET/∂t = 0: ignores light, removes relativity• me = 0: removes most electron physics• ∇ ·P = ∇P: collisional plasma or wave-particle interactions• ∂J/∂t = 0 (and = J× B): leaves only largest scale and slowest phenomena• ...• all of above + very high conductivity: ∼ ideal magnetohydrodynamics

Mostly from doi:10.1007/s11214-008-9384-6

I. Honkonen: Modeling space plasma 19 / 24

(ideal) Magnetohydrodynamics

Simplest fluid (3d) description of plasma (∼ hydrodynamics + magnetic field)

• Advection of plasma: ∂ρ/∂t = −∇ · (ρV)• where ρ is mass density

• Evolution of momentum:ρ(∂/∂t+ V · ∇)V = J× B+∇P• where V is bulk velocity of plasma, P is

(scalar) bulk pressure of plasma

• Total energy: ∂U/∂t =−∇ · [V(U+ P + B2/(2µ0)) − B(V · B)/µ0]• ∂B/∂t = −∇× E = ∇× (V × B)• J = ∇× B/µ0• ∇ · B = 0, otherwise unphysical forces acting

on plasma• Only numerical heat conduction, viscosity,

reconnection, etc.

sohowww.nascom.nasa.gov/bestofsoho/Helioseismology/mdi026.html

Bounce-averaged kinetic plasma

• 2r+2v-dimensional kinetic plasma• r: pos. of B, v: speed, angle from B• Much looser requirements on time to

solution than full trajectory modeling• Used for ions/electrons near planets

with dipole B• Or fusion plasma with bounce-avg e,

gyro-avg i

• Averages out particle gyration andbounce motion near dipole• For modeling particle drifts around

planet• ring current(s), radiation belt(s)

en.wikipedia.org/wiki/File:Van Allen radiation belt.svg

I. Honkonen: Modeling space plasma 21 / 24

Gyro-averaged kinetic plasma

• 3r+2v-dimensional kinetic plasma• r: pos. of e, i; v: speed, angle from B• Significantly shorter time to solution than full modeling

• Used for magnetospheric and fusion plasmas, turbulence• E.g. fully kinetic i, gyro-averaged e

• Averages out particle gyration and bounce motion near dipole• For modeling particle drifts around planet

• ring current(s), radiation belt(s)

I. Honkonen: Modeling space plasma 22 / 24

Things not addressed

Some environments require modeling plasma & fields and/or:• Special/general relativity

• Formation of GeV, TeV, PeV, EeV, ZeV cosmic rays• High energy shock formation, evolution, particle acceleration• Supernova, neutron star, magnetar, black hole environments

• Quantum physics• e−e+ pair production,• Supernova, neutron star, magnetar, black hole environments

• Gravity• Star, galaxy formation

I. Honkonen: Modeling space plasma 23 / 24

Ilmatieteen laitosErik Palmenin aukio 100560 HelsinkiPL 503, 00101 Helsinkipuh. 029 539 1000

Meteorologiska institutetErik Palmens plats 100560 HelsingforsPB 503, 00101 Helsingforstel. 029 539 1000

Finnish Meteorological InstituteErik Palmenin aukio 100560 HelsinkiP.O.Box 503, 00101 Helsinkitel. +358 29 539 1000

www.fmi.fiTwitter: @FMIspace, @IlmaTiede