Slide 1

The Pencil Code -- a high order MPI code for MHD turbulence Anders Johansen (Sterrewacht Leiden) Axel Brandenburg (NORDITA, Stockholm) Wolfgang Dobler Tobias Heinemann (DAMTP) Tony Mee (Newcastle) Wladimir Lyra (Uppsala) etc. (...just google for Pencil Code)

Slide 2

2 Pencil Code Started in Sept. 2001 by Axel Brandenburg and Wolfgang Dobler High order (6 th order in space, 3 rd order in time) Cache & memory efficient MPI, can run PacxMPI (across countries!) Maintained/developed by many people (CVS!) Automatic validation (over night or any time) Max resolution so far 1024 3, 256 procs

Slide 3

3 Pencil formulation In CRAY days: worked with full chunks f(nx,ny,nz,nvar) Now, on SGI, nearly 100% cache misses Instead work with f(nx,nvar), i.e. one nx-pencil No cache misses, negligible work space, just 2N Can keep all components of derivative tensors Communication before sub-timestep Then evaluate all derivatives, e.g. call curl(f,iA,B) Vector potential A=f(:,:,:,iAx:iAz), B=B(nx,3)

Slide 4

4 Switch modules magnetic or nomagnetic (e.g. just hydro) hydro or nohydro (e.g. kinematic dynamo) density or nodensity (burgulence) entropy or noentropy (e.g. isothermal) radiation or noradiation (solar convection, discs) dustvelocity or nodustvelocity (planetesimals) Coagulation, reaction equations Homochirality (reaction-diffusion-advection equations) Other physics modules: MHD, radiation, partial ionization, chemical reactions, self-gravity

Slide 5

5 Pencil Code check-ins

Slide 6

6 High-order schemes Alternative to spectral or compact schemes Efficiently parallelized, no transpose necessary No restriction on boundary conditions Curvilinear coordinates possible (except for singularities) 6th order central differences in space Non-conservative scheme Allows use of logarithmic density and entropy Copes well with strong stratification and temperature contrasts

Slide 7

7 (i) High-order spatial schemes Main advantage: low phase errors

Slide 8

8 Wavenumber characteristics

Slide 9

9 Higher order less viscosity

Slide 10

10 Less viscosity also in shocks

Slide 11

11 (ii) High-order temporal schemes Main advantage: low amplitude errors 3 rd order 2 nd order 1 st order 2N-RK3 scheme (Williamson 1980)

Slide 12

12 Shock tube test

Slide 13

13 Hyperviscous, Smagorinsky, normal Inertial range unaffected by artificial diffusion Haugen & Brandenburg (PRE, astro-ph/0402301) height of bottleneck increased onset of bottleneck at same position

Slide 14

14 256 processor run at 1024 3

Slide 15

15 MHD equations Induction Equation: Magn. Vector potential Momentum and Continuity eqns

Slide 16

16 Vector potential B=curlA, advantage: divB=0 J=curlB=curl(curlA) =curl2A Not a disadvantage: consider Alfven waves B-formulation A-formulation 2 nd der once is better than 1 st der twice!

Slide 17

17 Comparison of A and B methods

Slide 18

18 Wallclock time versus processor # nearly linear Scaling 100 Mb/s shows limitations 1 - 10 Gb/s no limitation

Slide 19

19 Pre-processed data for animations

Slide 20

20 Ma=10 supersonic turbulence

Slide 21

21 Animation of B vectors

Slide 22

22 Animation of energy spectra Very long run at 512 3 resolution

Slide 23

23 MRI turbulence MRI = magnetorotational instability 256 3 w/o hypervisc. t = 600 = 20 orbits 512 3 w/o hypervisc. t = 60 = 2 orbits

Slide 24

24 Fully convective star

Slide 25

25 Geodynamo simulation

Slide 26

26 Homochirality: competition of left/right Reaction-diffusion equation

Slide 27

27 Transfer equation & parallelization Analytic Solution: Ray direction Intrinsic Calculation Processors

Slide 28

28 The Transfer Equation & Parallelization Analytic Solution: Ray direction Communication Processors

Slide 29

29 The Transfer Equation & Parallelization Analytic Solution: Ray direction Processors Intrinsic Calculation

Slide 30

30 Current implementation Plasma composed of H and He Only hydrogen ionization Only H - opacity, calculated analytically No need for look-up tables Ray directions determined by grid geometry No interpolation is needed

Slide 31

31 Sunspot with radiative transfer Heinemann et al. 2007

Slide 32

32Conclusions High order schemes Low phase and amplitude errors Need less viscosity Subgrid scale modeling can be unsafe (some problems) shallower spectra, longer time scales, different saturation amplitudes (in helical dynamos) The Pencil Code is versatile Created for MHD problems, but recent modules include Radiative transfer and ionization Coagulation equation Homochirality Global accretion disks Solid particles Self-gravity...