Pencil Code: multi-purpose and Pencil Code: multi-purpose and multi-user maintainedmulti-user maintained

Axel Brandenburg (Nordita, Stockholm)

Wolfgang Dobler (Univ. Calgary)

and now many more….

(...just google for Pencil Code)

PencilPencilCodeCode

• Started in Sept. 2001 with Wolfgang Dobler• High order (6th order in space, 3rd order in time)• Cache & memory efficient• MPI, can run PacxMPI (across countries!)• Maintained/developed by ~40 people (SVN)• Automatic validation (over night or any time)• Max resolution so far 10243 , 4096 procs

• Isotropic turbulence– MHD, passive scl, CR

• Stratified layers– Convection, radiation

• Shearing box– MRI, dust, interstellar– Self-gravity

• Sphere embedded in box– Fully convective stars– geodynamo

• Other applications– Homochirality– Spherical coordinates

3

Pencil formulationPencil formulation

• In CRAY days: worked with full chunks f(nx,ny,nz,nvar)

– Now, on SGI, nearly 100% cache misses

• Instead work with f(nx,nvar), i.e. one nx-pencil• No cache misses, negligible work space, just 2N

– Can keep all components of derivative tensors

• Communication before sub-timestep• Then evaluate all derivatives, e.g. call curl(f,iA,B)

– Vector potential A=f(:,:,:,iAx:iAz), B=B(nx,3)

4

Switch modulesSwitch modules• magnetic or nomagnetic (e.g. just hydro)• hydro or nohydro (e.g. kinematic dynamo)• density or nodensity (burgulence)• entropy or noentropy (e.g. isothermal)• radiation or noradiation (solar convection, discs)• dustvelocity or nodustvelocity (planetesimals)• Coagulation, reaction equations• Chemistry (reaction-diffusion-advection equations)

Other physics modules: MHD, radiation, partial ionization, chemical reactions, selfgravity

5

High-order schemesHigh-order schemes

• Alternative to spectral or compact schemes– Efficiently parallelized, no transpose necessary

– No restriction on boundary conditions

– Curvilinear coordinates possible (except for singularities)

• 6th order central differences in space• Non-conservative scheme

– Allows use of logarithmic density and entropy

– Copes well with strong stratification and temperature contrasts

6

(i) High-order spatial schemes(i) High-order spatial schemes

x

fffffff iiiiii

i 60

945459 321123'

2321123''

180

227270490270272

x

ffffffff iiiiiii

i

Main advantage: low phase errors

x x x x x x x x x

Near boundaries:

ghost zones interior points

7

Wavenumber characteristicsWavenumber characteristics

kx

dxkxdkeff sin

/cos

xk

kx

dxkxdk Nyeff / ,

cos

/cos 222

8

Higher order – less viscosityHigher order – less viscosity

9

Less viscosity – also in Less viscosity – also in shocksshocks

10

(ii) High-order temporal schemes(ii) High-order temporal schemes

),( 111 iiiii utFtww

Main advantage: low amplitude errors

iiii wuu 1

3)1()(

0 , uuuu nn

2/1 ,1 ,3/1

1 ,3/2 ,0

321

321

1 ,2/1

2/1 ,0

21

21

1

0

1

1

3rd order

2nd order

1st order

2N-RK3 scheme (Williamson 1980)

11

Shock tube testShock tube test

12

Increase in # of auto testsIncrease in # of auto tests

13

Evolution of code sizeEvolution of code size

User meetings:User meetings:2005 Copenhagen2006 Copenhagen2007 Stockholm2008 Leiden2009 Heidelberg

14

Pencil Code check-insPencil Code check-ins

15

Vector potentialVector potential

• B=curlA, advantage: divB=0• J=curlB=curl(curlA) =curl2A• Not a disadvantage: consider Alfven waves

z

uB

t

b

z

bB

t

u

00 and ,

uBt

a

z

aB

t

u02

2

0 and ,

B-formulation

A-formulation 2nd der onceis better than1st der twice!

16

Comparison of A and B methodsComparison of A and B methods

2

2

02

2

2

2

0 and ,z

auB

t

a

z

u

z

aB

t

u

2

2

02

2

0 and ,z

b

z

uB

t

b

z

u

z

bB

t

u

17

Faster and bigger machinesFaster and bigger machines

18

256 processor run at 1024256 processor run at 102433

19

Hyperviscous, Smagorinsky, normalHyperviscous, Smagorinsky, normal

Inertial range unaffected by artificial diffusionHau

gen

& B

rand

enbu

rg (

PR

E, a

stro

-ph/

0402

301)

height of bottleneck increased

onset of bottleneck at same position

20

Online data reduction and visualizationOnline data reduction and visualization

non-helically forced turbulence

21

Scalars on periphery of the boxScalars on periphery of the box

22

MRI turbulenceMRI turbulenceMRI = magnetorotational instabilityMRI = magnetorotational instability

2563

w/o hypervisc.t = 600 = 20 orbits

5123

w/o hypervisc.t = 60 = 2 orbits

23

Vorticity and DensityVorticity and Density

See poster by Tobi Heinemann on density wave excitation!