Martin Berzins (Steve Parker)

What are the hard apps problems? How do the solutions get shared? What non-apps work is needed?

Thanks to DOE for funding since 1997, NSF since 2008

Uintah solver for multiphase-fluid-structure interaction problems- explosive filled container in fire

Martin Berzins (Steve Parker)

What are the hard apps problems? ADAPTIVE DYNAMIC GLOBALHow do the solutions get shared? ENCAPSULATION ABSTRACTIONWhat non-apps work is needed? APPLICATION DRIVEN TOOLS

Thanks to DOE for funding since 1997, NSF since 2008

Uintah solver for multiphase-fluid-structure interaction problems- explosive filled container in fire

Hard Apps Problems are Multi-Physics/Multiscale with Adaptive

Methods and/or Global Comms.

1. Lack of predictability forces use of dynamic load balancing methods.

2. AMR data structures – migration.

3. Radiation problems –global communications needed.

4. Particles move across grid with our methods

5. Any attempt to compute to a particular solution accuracy will need adaptive methods

UPREDICTABLE!!

Particle Variables

Cell Centered Variables

Cell –Vertex Variables

Fundamental Uintahdata Structure is a patch – multiple variable types

SFC Load balancing uses patches

Uintah Domain Decomposition

User writes code for a patch and its communicationsonly - Uintah uses this information to construct communications pattern via a task graph

Combining MPM and Mesh refinement

How Does Uintah Work?

SimulationController

SimulationController

ProblemSpecification

ProblemSpecification

XML

Simulation(One of Arches, ICE,

MPM, MPMICE, MPMArches, …)

Simulation(One of Arches, ICE,

MPM, MPMICE, MPMArches, …)

SchedulerScheduler

Tasks

DataArchiver

DataArchiver

Tasks

Callbacks

CallbacksMPI

Assignments

LoadBalancer

LoadBalancer

Configuration

→

Burgers Equation code void Burger::timeAdvance(const ProcessorGroup*, const PatchSubset* patches, const MaterialSubset* matls, DataWarehouse* old_dw, DataWarehouse* new_dw){ //Loop for all patches on this processor for(int p=0;p<patches->size();p++){//Get data from data warehouse including 1 layer of "ghost" nodes from surrounding patches old_dw->get(u, lb_->u, matl, patch, Ghost::AroundNodes, 1);

// dt, dx Time and space increments Vector dx = patch->getLevel()->dCell(); old_dw->get(dt, sharedState_->get_delt_label()); // allocate memory for results new_dw->allocateAndPut(new_u, lb_->u, matl, patch); // define iterator range l and h …… lots missing here and Iterate through all the nodes for(NodeIterator iter(l, h);!iter.done(); iter++){ IntVector n = *iter; double dudx = (u[n+IntVector(1,0,0)] - u[n-IntVector(1,0,0)]) /(2.0 * dx.x()); double du = - u[n] * dt * (dudx); new_u[n]= u[n] + du; }

0t xU UU+ =

8

Task graph• Each algorithm defines a

description of the computation– Required inputs and outputs

(names and spatial relationships)

– Callbacks to perform each task on a single subregion of space

• Communication is performed at the edges in the graph

• Uintah uses this information to create a graph of computation and communication

AMR Scalability

Challenging dynamicallychanging workload

8x8x8 patches: timings over about 30 timesteps 8K to 20K patchesOriginal remeshes at every step Dilated only every 4 steps or so

Atlas LLNL 1152 nodes of 4 Opteron dual-core Infiniband Thunder LLNL 1024 nodes of 4 Intel Itainum2 Quadrics switch Redstorm SNL 13284 nodes of Opteron dual-core XT3 Ranger UT Austin 3,936 nodes of 4 Barcelona quad-core Infiniband

SmallProblem only 2-3patches per proc at 4096procs

Summary Uintah is adaptive multi-physics AMR code

Clear separation between application user and system components

Very general CS approach to load balancing and scalability

Expensive multidisciplinary effort.

Existing AMR scalability work

• Brian Van Stralen: weak scaling easy strong scaling hard

• Strong scaling problems include message overload, malloc inconsistencies and load imbalance.

• Not clear that these problems cannot be solved

• AMR does scale to 12K processors Flash Code BG/L and beyond (?)

Strong scalability: fixed problem size doubling processors should halve execution time

Weak scalability: problem size grows with processors Doubling processors give constant execution time