example application: 3d scalar wave equation the cactus team albert einstein institute...
TRANSCRIPT
Scalar waves in 3D are solutions of the hyperbolic wave equation
-,tt + ,xx + ,yy + ,zz = 0
This is an initial value problem, if we give data for and its first time derivative at some initial time, the wave equation tells us how it evolves in the future
r
time
Scalar Wave Model ProblemScalar Wave Model Problem
Numerical solve by discretising on a grid, using explicit finite differencing (centered, second order)
n+1i,j,k = 2n
i,j,k - n-1i,j,k
+ t2/x2(ni+1,j,k -2 n
i,j,k + ni-1,j,k )
+ t2/y2(ni,j+1,k -2 n
i,j,k + ni,j-1,k )
+ t2/z2(ni,j,k+1 -2 n
i,j,k + ni,j,k-1 )
time
r
Numerical MethodNumerical Method
Finite grid, so need to apply outer boundary conditions
Main parameters: grid spacings: t, x, y, z, which coords?, which initial data?
Simple problem, analytic solutions, but contains many features needed for modelling more complex problems
Numerical MethodNumerical Method
c ==================================== program WaveToyc ====================================c Fortran 77 program for 3D wave equation.c Explicit finite difference method.c ====================================
c Global variables in include file include "WaveToy.h" integer i,j,k
c SET UP PARAMETERS nx = 30 [MORE PARAMETERS]
c SET UP COORDINATE SYSTEM AND GRID x_origin = (0.5 - nx/2)*dx y_origin = (0.5 - ny/2)*dy z_origin = (0.5 - nz/2)*dz
do i=1,nx do j=1,ny do k=1,nz
x(i,j,k) = dx*(i-1) + x_origin y(i,j,k) = dy*(j-1) + y_origin z(i,j,k) = dz*(k-1) + z_origin r(i,j,k) = sqrt(x(i,j,k)**2+y(i,j,k)**2+z(i,j,k)**2) end do end do end do
c OPEN OUTPUT FILES open(unit=11,file=“out.xl”) open(unit=12,file=“out.yl”) open(unit=13,file=“out.zl”)
c SET UP INITIAL DATA call InitialData call Output
c EVOLVING do iteration = 1, nt call Evolve if (mod(iteration,10).eq.0) call Output end do
stop end
Stand Alone Code: Main.fStand Alone Code: Main.f
Standalone ProgramStandalone Program
Setting up parameters
Setting up grid and coordinate system Opening output files Setting up initial data Performing iteration 10 Performing iteration 20 Performing iteration 30 Performing iteration 40 Performing iteration 50 Performing iteration 60 Performing iteration 70 Performing iteration 80 Performing iteration 90 Performing iteration 100 Done
Developed as single processor 3D program (~1hr)
Should you use Cactus? Not needed for such a simple program but if you need:
parallelism for more resolution?
to share development with colleagues?
to run on a variety of platforms?
2D/3D output and checkpointing?
to solve an elliptic equation for initial data?
adaptive mesh refinement?, remote monitoring? ….
All can be provided by Cactus
Wave Equation ProgramWave Equation Program
Parameter parser with types, ranges, checking
Scheduling of routines
Make system on many architectures
Dynamic argument lists
Utilities such as program checking test suites
Cactus Flesh ProvidesCactus Flesh Provides
Coordinates
(Cartesian, various domains with symmetries)
Boundary conditions
(Fixed, flat, radiation,...)
IO
(screen, 0D-3D, IsoSurfaces, Jpeg, streaming, configurable)
Driver
(create grid variables, handle storage, communications)
Cactus Thorns ProvideCactus Thorns Provide
c ==================================== program WaveToyc ====================================c Fortran 77 program for 3D wave equation.c Explicit finite difference method.c ====================================
c Global variables in include file include "WaveToy.h" integer i,j,k
c SET UP PARAMETERS nx = 30 [MORE PARAMETERS]
c SET UP COORDINATE SYSTEM AND GRID x_origin = (0.5 - nx/2)*dx y_origin = (0.5 - ny/2)*dy z_origin = (0.5 - nz/2)*dz
do i=1,nx do j=1,ny do k=1,nz
x(i,j,k) = dx*(i-1) + x_origin y(i,j,k) = dy*(j-1) + y_origin z(i,j,k) = dz*(k-1) + z_origin r(i,j,k) = sqrt(x(i,j,k)**2+y(i,j,k)**2+z(i,j,k)**2) end do end do end do
c OPEN OUTPUT FILES open(unit=11,file=“out.xl”) open(unit=12,file=“out.yl”) open(unit=13,file=“out.zl”)
c SET UP INITIAL DATA call InitialData call Output
c EVOLVING do iteration = 1, nt call Evolve if (mod(iteration,10).eq.0) call Output end do
stop end
Stand Alone Code: Main.fStand Alone Code: Main.f
Output from
parameters
CartGrid3D
Configuation files
You write these!!
Decide on arrangement/thorn structure Write thorn configuration files
param.ccl, interface.ccl, schedule.ccl
Add Cactus “stuff” to source code Add source code filenames to make.code.defn Compile and debug ... Write a parameter file Run and debug … Create a test suite to be sure it keeps working If you want, make your thorns available to
friends/colleagues/the world
Converting Your Code ...Converting Your Code ...
Could just write one thorn containing all the routines
Better to split the code in different thorns
IDScalarWave (Initial data)
WaveToyF77 (Evolving routines)
Make use of Toolkit thorns for Coordinates, Boundaries, IO.
Why separate thorns?
Easier to develop and share
Better code planning - modularity
Splitting Into ThornsSplitting Into Thorns
Checkout Cactus, and in the Cactus home directory type
gmake newthorn
Follow instructions to create a new thorns “WaveToyF77” and “IDScalarWave” in an arrangement e.g. “Tutorial”
This gives you the thorn structure and the basic files needed
Create New ThornsCreate New Thorns
schedule.cclparam.cclinterface.cclREADM E m ake.code.defn
W aveToy.F77InitSym Bound.F77
src doc par test
W aveToyF77
Thorn StructureThorn Structure
Thorn Configuration FilesThorn Configuration Files
Need to decide:
What are the grid variables:
– interface.ccl
What are the parameters:
– param.ccl
In which order should the routines be scheduled:
– schedule.ccl
Do I use grid variables, parameters or scheduling from other thorns?
These configuration files are parsed (Perl) during compilation, generating code for argument lists, parameters, program flow, etc.
Parameters: set in a parameter file read and verified at run time.
Example: How many grid points? What initial data? What to output?
Specify:
data type (real, integer, keyword, boolean, string)
range (nx>0, initial_data is “gaussian wave” or “plane wave”)
description
visability to other thorns (private, restricted, global)
default value
parameters used from other thorns
Thorn Configuration: param.cclThorn Configuration: param.ccl
# Parameter definitions for thorn WaveToyF77
private:
KEYWORD bound "Type of boundary condition to use"
{
"none” :: "No boundary condition"
"flat" :: "Flat boundary condition"
"radiation" :: "Radiation boundary condition"
"zero" :: "Zero boundary condition"
} "none"
WaveToyF77: param.cclWaveToyF77: param.ccl
Only the evolver needs to know about the boundary condition
# Parameter definitions for thorn IDScalarWave
shares: gridUSES KEYWORD type
restricted:KEYWORD initial_data "Type of initial data"{ "plane" :: "Plane wave" "gaussian" :: "Gaussian wave" "box" :: "Box wave"} "gaussian"
private:REAL radius "The radius of the gaussian wave"{ 0:* :: “Radius must be positive”} 0.0
IDScalarWave: param.cclIDScalarWave: param.ccl
IDScalarWave uses the parameter type from CartGrid3D
Other thorns can use or extend initial_data
Only for IDScalarWave to use
Object orientated concepts:
Implementation, Inherits, Friends
Grid variables
group name (many flesh functions act on groups)
group type (grid array, grid function, grid scalar)
variable type (real, int, complex)
dimension
description
visability to other thorns (private, protected, public)
Thorn configuration: interface.cclThorn configuration: interface.ccl
WaveToyF77: interface.cclWaveToyF77: interface.ccl
# Interface definition for WaveToyF77
implements: wavetoy
public:
cctk_real scalarevolve type = GF timelevels=3
{ phi} "The evolved scalar field"
Implements: describes what this thorn “does”, WaveToyF77 can be replaced by any other thorn which “does” the same thing and has the same public interface.
Timelevels: finite difference method is a 3 time level scheme, phi_n, phi, phi_p. Time levels are rotated at each iteration.
Scope: grid variables can be public, protected or private.
IDScalarWave: interface.cclIDScalarWave: interface.ccl
# Interface definition for IDScalarWave
implements: idscalarwave
inherits: wavetoy grid
Inherits: what IDScalar wave takes from other thorns (implementations)
Needs phi from WaveToyF77 (wavetoy) to fill in initial data
Needs coordinate grid functions and parameters from CartGrid3D (grid)
Schedules when the thorn routines are run, and when storage (and communications) for variables are activated
WaveToyF77 has one base routine, WaveToyF77_Evol when should it be run?
should it be BEFORE or AFTER any other routine?
what language (Fortran or C) is it written in?
is variable storage or communication needed?
is the variable storage needed just for the routine, or for the whole run?
Thorn Configuration: schedule.cclThorn Configuration: schedule.ccl
Example Scheduling TreeExample Scheduling Tree Startup routines Cactus: Register banner for Cactus CartGrid3D: Register GH Extension for GridSymmetry CartGrid3D: Register coordinates for the Cartesian grid IOASCII: Startup routine IOBasic: Startup routine IOUtil: IOUtil startup routine PUGH: Startup routine. WaveToyC: Register banner
Parameter checking routines CartGrid3D: Check coordinates for CartGrid3D IDScalarWave: Check parameters
Initialisation CartGrid3D: Set up spatial 3D Cartesian coordinates on the GH PUGH: Report on PUGH set up Time: Set timestep based on speed one Courant condition WaveToyC: Schedule symmetries IDScalarWave: Initial data for 3D wave equation
do loop over timesteps WaveToyC: Evolution of 3D wave equation t = t+dt if (analysis) endif enddo
CCTK_STARTUP
CCTK_EVOL
CCTK_INITIAL
CCTK_PARAMCHECK
# Schedule definitions for thorn WaveToy77
STORAGE: scalarevolve
schedule WaveToyF77_Evolution as WaveToy_Evolution at EVOL
{
LANG: Fortran
SYNC: scalarevolve
} "Evolution of 3D wave equation”
schedule WaveToyF77_Boundaries as WaveToy_Boundaries at EVOL
AFTER WaveToy_Evolution
{
LANG: Fortran
} "Boundaries of 3D wave equation"
WaveToyF77: schedule.cclWaveToyF77: schedule.ccl
Function alias
Synchronize group on exit
Storage always assigned
# Schedule definitions for thorn IDScalarWave
schedule IDScalarWave_CheckParameters at CCTK_PARAMCHECK
{
LANG: Fortran
} "Check parameters"
schedule IDScalarWave_InitialData at CCTK_INITIAL
{
STORAGE: wavetoy::scalarevolve
LANG: Fortran
} "Initial data for 3D wave equation"
IDScalarWave: schedule.cclIDScalarWave: schedule.ccl
Should already be on, but make sure
WaveToy.F77WaveToy.F77
#include "cctk.h" #include "cctk_Parameters.h”#include "cctk_Arguments.h"
subroutine WaveToyF77_Evolution(CCTK_ARGUMENTS)
implicit none
c Declare variables in argument list DECLARE_CCTK_ARGUMENTS
INTEGER i,j,k,ierr CCTK_REAL dx,dy,dz,dt CCTK_REAL dx2,dy2,dz2,dt2 CCTK_REAL dx2i,dy2i,dz2i CCTK_REAL factor
c Set up shorthandsc ----------------- dx = CCTK_DELTA_SPACE(1) dy = CCTK_DELTA_SPACE(2) dz = CCTK_DELTA_SPACE(3) dt = CCTK_DELTA_TIME
dx2 = dx*dx dy2 = dy*dy dz2 = dz*dz dt2 = dt*dt
dx2i = 1.0/dx2 dy2i = 1.0/dy2 dz2i = 1.0/dz2 factor = 2*(1-(dt2)*(dx2i+dy2i+dz2i))
c Do the evolutionc ---------------- do k = 2, cctk_lsh(3)-1 do j = 2, cctk_lsh(2)-1 do i = 2, cctk_lsh(1)-1 phi_n(i,j,k) = factor*phi(i,j,k) -& phi_p(i,j,k) + (dt2) * & ((phi(i+1,j,k)+phi(i-1,j,k))*dx2i& +(phi(i,j+1,k)+phi(i,j-1,k))*dy2i& +(phi(i,j,k+1)+phi(i,j,k-1))*dz2i)
end do end do end do
return end
WaveToy.F77 (2)WaveToy.F77 (2)
c Apply the outer boundary conditions if (CCTK_EQUALS(bound,"flat")) then call BndFlatVN(ir,cctkGH,sw,"wavetoy::phi") else if (CCTK_EQUALS(bound,"zero")) then call BndScalarVN(ir,cctkGH,zero,sw, & "wavetoy::phi") else if (CCTK_Equals(bound,"radiation").eq.1) then call BndRadiativeVN(ir,cctkGH,sw,zero,one, & "wavetoy::phi”,"wavetoy::phi") end if
if (ierr < 0) then call CCTK_WARN(0,”BCs not applied!!"); end if
return end
subroutine & WaveToyF77_Boundaries(CCTK_ARGUMENTS)
implicit none
DECLARE_CCTK_ARGUMENTS DECLARE_CCTK_PARAMETERS DECLARE_CCTK_FUNCTIONS
c Local declarations CCTK_REAL zero,one integer ir integer sw(3)
ierr = -1 zero = 0.0 one = 1.0
c Set the stencil width sw(1)=1 sw(2)=1 sw(3)=1
c Apply the symmetry boundary conditions call & CartSymGN(ir,cctkGH,"wavetoy::scalarevolve")
Get hold of the additional Cactus thorns needed
CactusBase/CartGrid3D (sets up the Cartesian grid) CactusBase/Boundary (provides standard boundary conditions) CactusBase/Time (set the timestep) CactusPUGH/PUGHSlab (extract hyperslabs of data from grid arrays) CactusPUGH/PUGHReduce (reduction operations) CactusPUGH/PUGH (the driver, always needed) CactusBase/IOASCII (output in 1D ASCII format) CactusBase/IOBasic (output scalars (inc. reductions) and to screen) CactusBase/IOUtil (all the Cactus IO thorns need this
thorn)
Compiling WaveToy ApplicationCompiling WaveToy Application
Set up a configuration by typing
gmake wave (or gmake wave-config)
Compile the code using
gmake wave
If successful you’ll create exe/cactus_wave
Compiling WaveToy ApplicationCompiling WaveToy Application
Cactus runs from a parameter file
> exe/cactus_wave wavetoy.par
All thorns you need must be activated using the ActiveThorns parameter
Parameters must be qualified with the implementation or thorn name, depending on their type
ActiveThorns = "idscalarwave time wavetoyf77 pugh pughreduce cartgrid3d pughslab ioutil ioascii"
time::dtfac = 0.5
idscalarwave::initial_data = "gaussian"idscalarwave::sigma = 2.8idscalarwave::radius = 0
wavetoyf77::bound = "zero"
grid::type = "BySpacing"grid::dxyz = 0.6
driver::global_nx = 30driver::global_ny = 30driver::global_nz = 30
cctk_itlast = 100
IOASCII::out1D_every = 10IOASCII::out1D_vars = "wavetoy::phi "IOASCII::outinfo_every = 10IOASCII::outinfo_vars = "wavetoy::phi"IO::outdir = "StandAlone"
Running ExecutableRunning Executable
Run Time OutputRun Time Output> mpirun -np 2 exe/cactus_wave wavetoy.par----------------------------------------------------------------------------- 10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
-----------------------------------------------------------------------------Activating thorn Cactus...Success -> active implementation CactusActivating thorn iobasic...Success -> active implementation IOBasicActivating thorn idscalarwave...Success -> active implementation idscalarwaveActivating thorn time...Success -> active implementation timeActivating thorn wavetoyf77...Success -> active implementation wavetoyActivating thorn pugh...Success -> active implementation driverActivating thorn pughreduce …Success -> active implementation pughreduceActivating thorn cartgrid3d...Success -> active implementation gridActivating thorn ioutil...Success -> active implementation IOActivating thorn ioascii...Success -> active implementation IOASCII-----------------------------------------------------------------------------
Run Time Output (2)Run Time Output (2)Startup routines CartGrid3D: Register GH Extension for GridSymmetry CartGrid3D: Register coordinates for the Cartesian grid PUGH: Startup routine PUGHReduce: Startup routine IOUtil: Startup routine IOASCII: Startup routine IOBasic: Startup routine WaveToyF77: Register banner
Parameter checking routines CartGrid3D: Check coordinates for CartGrid3D IDScalarWave: Check parameters
Initialisation CartGrid3D: Set up spatial 3D Cartesian coordinates on the GH IOASCII: Choose 1D output lines IOASCII: Choose 2D output planes PUGH: Report on PUGH set up Time: Set timestep based on Courant condition WaveToyF77: Schedule symmetries IDScalarWave: Initial data for 3D wave equation
Run Time Output (3)Run Time Output (3)do loop over timesteps WaveToyF77: Evolution of 3D wave equation WaveToyF77: Boundaries of 3D wave equation t = t+dt if (analysis) endif enddo Termination routines PUGH: Termination routine Shutdown routines----------------------------------------------------------------------------------------------------------------------------------------------------------Driver provided by PUGH-----------------------------------------------------------------------------WaveToyF77: Evolutions of a Scalar Field-----------------------------------------------------------------------------INFO (CartGrid3D): dx=>3.0000000e-01 dy=>3.0000000e-01 dz=>3.0000000e-01 INFO (CartGrid3D): x=>[-0.150, 8.550] y=>[-0.150, 8.550] z=>[-0.150, 8.550] INFO (PUGH): MPI Evolution on 4 processorsINFO (PUGH): 3-dimensional grid functionsINFO (PUGH): Size: 30 30 30INFO (PUGH): Processor topology: 1 x 2 x 2INFO (PUGH): Local load: 7680 [30 x 16 x 16]INFO (PUGH): Maximum load skew: 0.000000
Run Time Output (4)Run Time Output (4)-------------------------------------------- it | | phi | | t | Min Max |-------------------------------------------- 0 | 0.000| 0.00000000 | 0.99142726 | 10 | 1.500| -0.14944313 | 0.00005206 | 20 | 3.000| -1.16992203 | 0.00000000 | 30 | 4.500| -0.72347868 | -0.00000002 | 40 | 6.000| -0.41127922 | -0.00000094 | 50 | 7.500| -0.29834817 | -0.00002427 | 60 | 9.000| -0.23577373 | -0.00035044 | 70 | 10.500| -0.19532474 | 0.00000311 | 80 | 12.000| -0.16688911 | 0.00010812 | 90 | 13.500| -0.14582895 | 0.00022987 | 100 | 15.000| -0.12957000 | 0.00033802 | 110 | 16.500| -0.11631732 | 0.00025752 | 120 | 18.000| -0.06945640 | 0.00019346 |-----------------------------------------------------------------------------Done.
If it Doesn’t Work ...If it Doesn’t Work ... Read the Thorn Writers chapter in the documentation Remember that the Fortran code is preprocessed, to see the
postprocessed code that is actually compiled, look in <config>/build/<thorn> (wave/build/WaveToyF77)
Run with a high warning level and read all the warning messages
Review the parameters for the thorns you are using, they often have some debugging features (PUGH, enable_all_storage)
Submit bug reports via www.CactusCode.org Join the relevant mailing lists via www.CactusCode.org Contact [email protected]