Download - Production Version of Flash : 1.61
An Accelerated Strategic Computing Initiative (ASCI) Academic Strategic Alliances Program (ASAP) Center
at The University of Chicago
The Center for Astrophysical Thermonuclear Flashes
Production Version of Flash: 1.61Paul Ricker
University of Chicago
The Flash Code GroupA. Caceres, A. Calder, T. Dupont, J. Dursi, B. Fryxell, T. Linde, A. Mignone,
K. Olson, P. Ricker, R. Rosner, K. Riley, A. Siegel, F. Timmes, H. Tufo,N. Vladimirova, G. Weirs, K. Young, M. Zingale
ASCI Flash Center University of Chicago
Overview
Flash 1.61 in context
Algorithmic architecture
Automated verification
Summary
ASCI Flash Center University of Chicago
Flash 1.61 in context
ASCI Flash Center University of Chicago
Flash framework evolution
FLASH 0.0Initial
FLASH 1.0Organization
FLASH 1.6Architecture,Optimization
FLASH 2.0Data
ASCI Flash Center University of Chicago
Physics modules (* - new this year) Compressible hydrodynamics
Piecewise-parabolic method (PPM) Weighted essentially non-oscillatory (wENO) * Tadmor central-difference scheme *
Compressible magnetohydrodynamics * Total variation diminishing (TVD) *
Equations of state (EOS) Partially degenerate stellar EOS (Coulomb corrections) * Mixture of perfect gases *
Source terms Nuclear burning - variety of reaction networks * Single-mode turbulent energy injection *
Flash capabilities
ASCI Flash Center University of Chicago
Flash capabilities
Physics modules (continued) Gravitational field
Externally imposed (constant, plane parallel *, point source *) Self-gravity *
Multipole summation * Multigrid * Couchman FFT method *
Diffusion * Thermal diffusion (explicit) * Radiation (implicit) *
I/O formats Hierarchical data format (HDF) version 4 and 5 *
ASCI Flash Center University of Chicago
Implementation 83,000 lines of FORTRAN 90 + C
~ 24,000 in driver + mesh package ~ 59,000 in physics modules
Block-structured adaptive mesh using PARAMESH Parallelization using MPI I/O using HDF4/5
Ports ASCI platforms (Red, Blue, Nirvana) Cray T3E Linux clusters (Intel & Alpha)
Flash 1.61 implementation
ASCI Flash Center University of Chicago
Algorithmic architecture
ASCI Flash Center University of Chicago
Framework design requirements Performance
Must be able to do the largest problem possible Modularity
Must be easy to switch initial/boundary conditions Must be easy to add/change solvers
Ease of adoption Should work with existing F77 routines Should not require many extra packages Must be portable Should use standard data formats
Ease of experimentation Must be easy to vary problem parameters Should permit "quick-look" visualization
ASCI Flash Center University of Chicago
Flash interfacesDriver
runtime_parameterslog_file
performance_monitorphysical_constants
time_dependentinit_block()
evolve()output()
timestep()
steadyinit_block()
solve()output()
tree data structure
init_from_scratch()init_from_checkpoint()
init_flash()
Mesh MaterialsHydro Source_termsGravity
Diffusion
Other
init()fill_guardcells()
test_refinement()refine_derefine()
init()tstep()
set_ext_guardcell()hydro_3d()
init()tstep()
set_ext_guardcell()gravity_3d()
init()tstep()
src_terms()
eos_3d()eos()
abund_renorm()query_iso_db()
Paramesh Uniform Explicit Implicit
Split
PPM
Unsplit
WENO PPM
PPM
constant
point_mass
Poisson
Multigrid
Gamma
Helmholtz
...
burn
coolheat
iso13...
Multipole
ASCI Flash Center University of Chicago
F90 Implementation: compile-time "polymorphism"
Config filesPlaintext files describingmodule interrelationships
PARAMETERREQUIRESDEFAULTEXCLUSIVE
Modules filePlaintext files describingmodules to include in code
INCLUDE
Hydroinit()
tstep()set_ext_guardcell()
hydro_3d()Makefile
Explicit
Split
PPMhydro_1d()
etc.Makefile
hydro_3d()hydro_sweep()update_soln()
hydro_1d()
Implicit
Unsplit
WENOhydro_3d()
etc.Makefile
PPMhydro_3d()
etc.Makefile
hydro_3d()
tstep() tstep()
PPMhydro_3d()
etc.Makefile
Setup scriptConfigures build directoryaccording to Modules fileinstructions for a specifiedmachine and problem type
make
ASCI Flash Center University of Chicago
Changes from version 1.0
Architectural consistency "Object-oriented" structure
Mimic virtual functions Cumulative inherited Makefiles
Only internal and mesh common blocks Exclusive and non-exclusive modules Test driver capability
Simulation services Runtime parameter contexts Physical constants database (plus unit conversions) Materials properties database Logging and timing libraries
New modules
ASCI Flash Center University of Chicago
Automated verification
ASCI Flash Center University of Chicago
Flash calculations
Wide range of compressibility Wide range of length and time scales Many interacting physical processes Only indirect validation possible Rapidly evolving computing environment Many people in collaboration
Cellular detonations
Compressible turbulence
Helium burning on neutron starsRayleigh-Taylor instability
Richtmyer-Meshkov instability
Flame-vortex interactions
Laser-driven shock instabilitiesNova outbursts on white dwarfs
ASCI Flash Center University of Chicago
Automated verification of Flash
Want verification to be easy and routine Flash includes several test problems
Sod, Sedov, Woodward-Colella blast wave, advection, wind tunnel,
Shu-Osher, isothermal collapse, Jeans instability, … More are added as needed
Automated testing using flash_test Compiles and runs Flash Compares against benchmark results using focu
focu (Flash Output Comparison Utility) Local or global error criteria Generates a short report for flash_test
ASCI Flash Center University of Chicago
flash_test
Automated verification architecture
Test databaseRuntime parameterfiles and configurationinformation for alltest problems includedwith code
Site databaseQueue and OSinformation for eachconfigured site; joband job submissiontemplates
OutputLog file indicating success ofeach compile/run/compare;output files
Jobs tracked vialockfile mechanism
Stored benchmark dataIndexed by date; output filescreated by previous invocationof flash_test
Comparisons viafocu
ASCI Flash Center University of Chicago
Summary
Flash... Is a small, focused, but flexible framework Offers state-of-the-art simulation capabilities Is extensively verified Achieves high performance (Fryxell talk)
0.24 TFlop/s on ASCI Red (2000 Gordon Bell finalist) Scales to > 6,000 processors
Is producing exciting science (Truran, Dupont talks) Continues to evolve (Siegel talk)