4/30/04 LSU 2004 Cactus Retreat 1
Toward Relativistic Hydrodynamics on Adaptive Meshes
Joel E. Tohline
Louisiana State University
http://www.phys.lsu.edu/~tohline
4/30/04 LSU 2004 Cactus Retreat 2
Principal Collaborators• Simulations to be shown today:
– Shangli Ou – (LSU)– Mario D’Souza – (LSU)– Michele Vallisneri – (Caltech/JPL)
• Code development over the years:– John Woodward – (Valtech; Dallas, Texas)– John Cazes – (Stennis Space Center; Stennis, Mississippi)– Patrick Motl – (Colorado)
• Science:– Juhan Frank (LSU)– Lee Lindblom (Caltech)– Luis Lehner (LSU)– Jorge Pullin (LSU)
4/30/04 LSU 2004 Cactus Retreat 3
Show 3 Movies
• Nonlinear development of the r-mode in young neutron stars [w/ Lindblom & Vallisneri] http://www.cacr.caltech.edu/projects/hydrligo/rmode.html
• Nonlinear development of the secular bar-mode instability in rapidly rotating neutron stars [w/ Ou & Lindblom] http://paris.phys.lsu.edu/~ou/movie/fmode/new/fmode.b181.om4.2e5.mov
• Mass-transferring binary star systems [w/ D’Souza, Motl, & Frank] http://paris.phys.lsu.edu/~mario/models/q_0.409_no_drag_3.8orbs/movies/q_0.409_no_drag_3.8orbs_top.mov
4/30/04 LSU 2004 Cactus Retreat 4
Storyline• Present Algorithm – has been producing publishable astrophysical
results for over 20 years: – Entirely home-grown code outside of Cactus environment– Manual domain decomposition– Explicit message-passing using mpi– Visualizations on serial machines (generally, post-processing)
• Plans for this calendar year:– Move present algorithm into Cactus environment
• Over the next few years, modify algorithm to:– Follow relativistic hydrodynamical flows on adaptive mesh– Accept evolving space-time metric– Visualize results “in parallel” with dynamical evolution
4/30/04 LSU 2004 Cactus Retreat 5
Storyline• Present Algorithm – has been producing publishable astrophysical
results for over 20 years: – Entirely home-grown code outside of Cactus environment– Manual domain decomposition– Explicit message-passing using mpi– Visualizations on serial machines (generally, post-processing)
• Plans for this calendar year:– Move present algorithm into Cactus environment
• Over the next few years, modify algorithm to:– Follow relativistic hydrodynamical flows on adaptive mesh– Accept evolving space-time metric– Visualize results “in parallel” with dynamical evolution
4/30/04 LSU 2004 Cactus Retreat 6
Storyline• Present Algorithm – has been producing publishable astrophysical
results for over 20 years: – Entirely home-grown code outside of Cactus environment– Manual domain decomposition– Explicit message-passing using mpi– Visualizations on serial machines (generally, post-processing)
• Plans for this calendar year:– Move present algorithm into Cactus environment
• Over the next few years, modify algorithm to:– Follow relativistic hydrodynamical flows on adaptive mesh– Accept evolving space-time metric– Visualize results “in parallel” with dynamical evolution
4/30/04 LSU 2004 Cactus Retreat 7
Present Algorithm
• Select grid structure and resolution• Construct initial configuration• Perform domain decomposition• While t < tstop
– Determine Newtonian gravitational accelerations– Push fluid around on the grid using Newtonian dynamics– If mod[ t , (orbital period/80) ] = 0
• Dump 3-D dataset for later visualization
– EndIf
• EndWhile• Visualize results
4/30/04 LSU 2004 Cactus Retreat 8
Principal Governing Equations
4/30/04 LSU 2004 Cactus Retreat 9
Principal Governing Equations
4/30/04 LSU 2004 Cactus Retreat 10
Present Algorithm
• Select grid structure and resolution• Construct initial configuration• Perform domain decomposition• While t < tstop
– Determine Newtonian gravitational accelerations– Push fluid around on the grid using Newtonian dynamics– If mod[ t , (orbital period/80) ] = 0
• Dump 3-D dataset for later visualization
– EndIf
• EndWhile• Visualize results
4/30/04 LSU 2004 Cactus Retreat 11
Present Algorithm
• Select grid structure and resolution• Construct initial configuration• Perform domain decomposition• While t < tstop
– Determine Newtonian gravitational accelerations– Push fluid around on the grid using Newtonian dynamics– If mod[ t , (orbital period/80) ] = 0
• Dump 3-D dataset for later visualization
– EndIf
• EndWhile• Visualize results
4/30/04 LSU 2004 Cactus Retreat 12
Present Algorithm
• Select grid structure and resolution• Construct initial configuration• Perform domain decomposition• While t < tstop
– Determine Newtonian gravitational accelerations– Push fluid around on the grid using Newtonian dynamics– If mod[ t , (orbital period/80) ] = 0
• Dump 3-D dataset for later visualization
– EndIf
• EndWhile• Visualize results
Serial
Serial
4/30/04 LSU 2004 Cactus Retreat 13
Present Algorithm
• Select grid structure and resolution• Construct initial configuration• Perform domain decomposition• While t < tstop
– Determine Newtonian gravitational accelerations– Push fluid around on the grid using Newtonian dynamics– If mod[ t , (orbital period/80) ] = 0
• Dump 3-D dataset for later visualization
– EndIf
• EndWhile• Visualize results
Parallel
4/30/04 LSU 2004 Cactus Retreat 14
Parallel Code’s Chronological Evolution
• John Woodward:– 8,192-processor MasPar @ LSU
• John Cazes:– CM5 @ NCSA; T3D/E @ SDSC
• Patrick Motl:– mpi on T3E @ SDSC; SP2/3 @ SDSC & LSU
• Michele Vallisneri:– HP Exemplar @ CACR
• Mario D’Souza & Shangli Ou:– SuperMike (1,024-proc Linux cluster) @ LSU
• Shangli Ou:– Tungsten (2,560-proc Linux cluster) @ NCSA
Early 90’s
Mid-90’s
Late 90’s
2000
2002-03
2004
4/30/04 LSU 2004 Cactus Retreat 15
Select Grid Structure and Resolution
• Unigrid, cylindrical mesh
• Fixed in time
• Typical resolution– Single star: 66 x 128 x 130 (as
shown on the left)
– Binary system: 192 x 256 x 98
4/30/04 LSU 2004 Cactus Retreat 16
Select Grid Structure and Resolution
4/30/04 LSU 2004 Cactus Retreat 17
Need for Non-unigrid and Adaptive Meshes
4/30/04 LSU 2004 Cactus Retreat 18
Perform Domain Decomposition
• Grid resolution 192 x 256 x 96• 64 processors• Distribute 192 x 96 (R,Z) grid
across 8 x 8 processor array• Leave angular zones “stacked”
in memory• Result: Each processor has
data arrays of size 24 x 256 x 12
• I/O: Scramble and unscramble handled manually
Z
R
4/30/04 LSU 2004 Cactus Retreat 19
Determine Newtonian Gravitational Accelerations(Three-dimensional Elliptic PDE on cylindrical mesh)
4/30/04 LSU 2004 Cactus Retreat 20
Principal Governing Equations
4/30/04 LSU 2004 Cactus Retreat 21
Determine Newtonian Gravitational Accelerations(Three-dimensional Elliptic PDE on cylindrical mesh)
• Perform FFT (in memory) in azimuthal coordinate direction reduce to decoupled set of (256) two-dimensional Helmholtz equations.
• Use ADI (alternating direction implicit) to solve each 2-D equation:– Data transpose– 1-D, in-memory ADI sweep– Data transpose– 1-D, in-memory ADI sweep– Data transpose– Etc.
• Inverse FFT
Z
R
4/30/04 LSU 2004 Cactus Retreat 22
Determine Newtonian Gravitational Accelerations(Three-dimensional Elliptic PDE on cylindrical mesh)
• Perform FFT (in memory) in azimuthal coordinate direction reduce to decoupled set of (256) two-dimensional Helmholtz equations.
• Use ADI (alternating direction implicit) to solve each 2-D equation:– Data transpose– 1-D, in-memory ADI sweep– Data transpose– 1-D, in-memory ADI sweep– Data transpose– Etc.
• Inverse FFT
Z
m
4/30/04 LSU 2004 Cactus Retreat 23
Determine Newtonian Gravitational Accelerations(Three-dimensional Elliptic PDE on cylindrical mesh)
• Perform FFT (in memory) in azimuthal coordinate direction reduce to decoupled set of (256) two-dimensional Helmholtz equations.
• Use ADI (alternating direction implicit) to solve each 2-D equation:– Data transpose– 1-D, in-memory ADI sweep– Data transpose– 1-D, in-memory ADI sweep– Data transpose– Etc.
• Inverse FFT
R
m
4/30/04 LSU 2004 Cactus Retreat 24
Visualize Results
• Specify isodensity surface(s)
• Find vertices and polygons on each surface (using marching cubes algorithm)
• Write out vertices & polygons in “OBJ” format
• Delete 3-D dataset
• Utilize “Maya” to render nested surfaces (from pre-specified viewer orientation)
• Write out TIFF image (typically 640 x 480)
• Generate .mov
4/30/04 LSU 2004 Cactus Retreat 25
Future Algorithm
• Select grid structure and resolution and [preferred AMR thorn]• [We] Construct initial configuration• [Let Cactus] Perform domain decomposition• While t < tstop
– [Call GR Group’s thorn] Determine structure of space-time metric– [We (or Whisky thorn)] Push fluid around on the grid using Relativistic dynamics– If mod[ t , (orbital period/80) ] = 0
• Generate vertices and polygons in parallel• Spawn “Maya” rendering task on additional processor(s)
– EndIf– [Call AMR thorn] Modify mesh, as necessary
• EndWhile• [Use Cactus thorn] Write article and Publish results
4/30/04 LSU 2004 Cactus Retreat 26
Future Algorithm
• Select grid structure and resolution and [preferred AMR thorn]• [We] Construct initial configuration• [Let Cactus] Perform domain decomposition• While t < tstop
– [Call GR Group’s thorn] Determine structure of space-time metric– [We (or Whisky thorn)] Push fluid around on the grid using Relativistic dynamics– If mod[ t , (orbital period/80) ] = 0
• Generate vertices and polygons in parallel• Spawn “Maya” rendering task on additional processor(s)
– EndIf– [Call AMR thorn] Modify mesh, as necessary
• EndWhile• [Use Cactus thorn] Write article and Publish results
4/30/04 LSU 2004 Cactus Retreat 27
THE END