Recent Advances in Parallel Implicit Solution of Fluid Plasma Systems

(Wed. March 4th, 2009)

• 2:00-2:25 Fully-Implicit Finite Element Formulations for Resistive Magneto-Hydrodynamic Systems

– Roger Pawlowski and John Shadid, Sandia National Laboratories; Luis Chacon, Oak Ridge National Laboratory; Jeffrey Banks, Lawrence Livermore National Laboratory

• 2:30-2:55 Towards Full Braginskii Implicit Extended MHD

– Luis Chacon, Oak Ridge National Laboratory • 3:00-3:25 The Magnetic Reconnection

Code: Using Code Generation Techniques in an Implicit Extended MHD Solver

– Kai Germaschewski, University of New Hampshire

• 3:30-3:55 Development and Applications of HiFi -- Adaptive, Implicit, High Order Finite Element Code for General Multi-fluid Applications

– Vyacheslav S. Lukin and Alan H. Glasser, University of Washington

• 4:30-4:55 Nonlinear Multigrid Methods for Fully Implicit Resistive MHD Simulations

– Ravi Samtaney, Princeton Plasma Physics Laboratory; Mark F. Adams, Columbia University; Achi Brandt, Weizmann Institute of Science, Israel

• 5:00-5:25 A Preconditioned JFNK Method for Resistive MHD in a Mapped-grid Tokamak Geometry

– Dan Reynolds, Southern Methodist University; Ravi Samtaney, Princeton Plasma Physics Laboratory; Carol S. Woodward, Lawrence Livermore National Laboratory

• 5:30-5:55 Progress in Parallel Implicit Methods for Tokamak Edge Plasma Modeling

– Lois Curfman McInnes, Argonne National Laboratory; Sean Farley, Louisiana State University; Tom Rognlien and Maxim Umansky, Lawrence Livermore National Laboratory; Hong Zhang, Argonne National Laboratory

• 6:00-6:25 Implicit Adaptive Mesh Refinement for 2D Resistive Magnetohydrodynamics

– Bobby Philip, Los Alamos National Laboratory; Luis Chacon, Oak Ridge National Laboratory; Michael Pernice, Idaho National Laboratory

Part 1: MS81 Part 2: MS91

Fully-Implicit Finite Element Formulations for Resistive Magneto-Hydrodynamic Systems

R. P. Pawlowski, J. N. Shadid, and E. T. Phipps,Sandia National Laboratories

L. Chacon, Los Alamos National Laboratory

J. W. Banks, Lawrence Livermore National Laboratory

SIAM Conference on Computational Science and EngineeringWednesday, March 4th, 2009

Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy

under Contract DE-AC04-94AL85000

MotivationMagnetohydrodynamics (MHD)

describes a variety of important physics:

• Geophysics: Earth’s magnetosphere

• Astrophysics: Solar flares, sunspots, stars, interplanetary medium, nebulae etc...

• Fusion: Tokamak, Stellerator

• Engineering: plasma confinement, liquid metal transfer, nuclear reactors, etc...

• Inertial confinement fusion (Rayleigh-Taylor instabilities)

ITERN

IMR

OD

FSP Report

SandiaZ-Machine

Magnetosphere Credit: Steele Hill/NASAThese systems are characterized by a myriad of complex, interacting,

nonlinear multiple time- and length-scale physical mechanisms.

Multiple-time-scale systems: E.g. Driven Magnetic Reconnection with a Magnetic Island Coalescence Problem (Incompressible)

Approx. Computational Time Scales: • Ion Momentum Diffusion: 10-7 to 10-3

• Magnetic Flux Diffusion: 10-7 to 10-3

• Ion Momentum Advection: 10-4 to 10-2

• Alfven Wave : 10-4 to 10-2

• Whistler Wave : 10-7 to 10-1

• Magnetic Island Sloshing: 100 • Magnetic Island Merging: 101

The Goal: Stable, Accurate, Scalable, and Efficient xMHD Unstructured FE Solution Methods

• Why fully Implicit?– Stability (stiff systems)– Accuracy (high order, variable order, local and global error control, ...) – Use time steps on the size of the dynamics of interest (no CFL constraint)– Avoid instabilities from operator splitting (Ropp and Shadid, JCP 2005)– Allows direct stability and bifurcation analysis (Salinger et al. IJBC 2005)– Allows embedded (fast) optimization (Bartlett et al. SAND 2003)

• Develop scalable solvers: physics-based/multi-level preconditioners

• Develop stabilized and compatible xMHD formulations using unstructured FE

• Produce large-scale computational demonstrations of MHD Systems• Magnetic Reconnection Studies• Hydro-Magnetic Rayleigh-Taylor (e.g. Z-pinch [HEDP])• Hydromagnetic Rayleigh-Bernard (towards geo-dynamo effects)• Fusion Energy (Tokamak etc…)

Divergence Conservation Form

Involution:

Extended MHD Model in Residual Form

General Case a Strongly Coupled, Multiple Time- and Length-Scale, Nonlinear, Nonsymmetric System with Parabolic and Hyperbolic Character

Involution:

Resistive, Extended MHD Equations

Formulations

• Multiple formulations are used to enforce the solenoidal involution: , and to address conservation.

1. Vector Potential (2D)

2. Projection (3D)

3. Lagrange Multiplier w/ VMS (3D)

4. Compatible discretizations (mixed Node/Edge/Face Elements)

r ¢B = 0

• For 2D Linear Lagrange Elements: Divergence free condition is satisfied to machine precision point-wise on element interiors and in an L2 sense over any sub-region of the domain (but not necessarily on the element boundaries since C0 elements are used).

• Convection/Diffusion/Reaction equation can use SUPG Stabilization.

Solenoidal involution is automatically satisfied provided that the discrete differential operator enforces to machine accuracy.

Select a Gauge and in 2D

Magnetic Vector Potential Formulation (2D)

Summary of Initial Stabilized FE Weak form of Equations for Low Mach Number MHD System

Governing

EquationStabilized FE Residual (following Hughes et. al.,

Shakib - Navier-Stokes; extension of Codina et. al. -Magnetics)

Momentum

Total Mass

Thermal Energy

Magnetics

(Vector Potential)

Fm,i

Rm,i

d

mu R

m,id

e

e

m,i Gcu

id

e

e

FP R

Pd

m R

md

e

e

ˆe e

cT T P T T T

e e

F R d C R d Td

u G

k k k k k

e e

cY Y Y Y Y k

e e

F R d R d Y d

u G

3D Lagrange Multiplier Formulation(Munz 2000, Dedner 2002, Codina 2006)

Remarks:• Elliptic constraint used to enforce divergence free condition.• Only weakly divergence free in FE implementation• VMS formulation for convection & coupling effects under development

Stabilization to circumvent inf-sup (LBB) condition(s):

Consistent, residual based stabilization (Hughes et al.):

Regularization (Dohrman-Bochev-Gunzburger):

Similar algorithms used for Magnetics equation and solenoidal constraint:

• Low Mach Number Resistive MHD

• Initial MHD Formulations:

– 2D Vector Potential

– 2D & 3D B field Projection

– Lagrange Multiplier Method (VMS)

• Massively Parallel: MPI

• 2D & 3D Unstructured Stabilized FE

• Fully Coupled Globalized Newton-Krylov solver

– Sensitivities: Templated C++, Automatic Differentiation (Saccado)

– GMRES (AztecOO, Belos)

– Additive Schwarz DD w/ Var. Overlap (Ifpack, AztecOO)

– Aggressive Coarsening Graph Based Block Multi-level [AMG] for Systems (ML)

• Fully-implicit: 1st-5th variable order BDF (Rythmos) & native integrator

• Direct-to-Steady-State (NOX), Continuation, Linear Stability and Bifurcation (LOCA / Anasazi), PDE Constrained Optimization (Moocho)

trilinos.sandia.gov

Nonlinear equations:

Newton System:

Sensitivities:

Results and Analysis

• Formulation Verification (selected examples)– Flux Expulsion (Unstructured Mesh)– Alfven Wave (spatial and temporal order of

accuracy)

• Scalability and Multicore• Stability and Bifurcation Analysis• Magnetic Reconnection

Flux Expulsion(Unstructured Mesh)

Analytic Solution:

MHD Rayleigh Flow and Alfven Wave(Transient w/ Analytic Solution)

FluidU

Analytic Solution: B

MHD Rayleigh Flow and Alfven Wave

Scalability(MHD Pump, Cray XT3)

Preconditioners• 1-level ILU(2,1)• 1-level ILU(2,3)• 1-level ILU(2,7)• 3-level ML(NSA,Gal)• 3-level ML(EMIN, PG)

ML: Tuminaro, HuIfpack: Heroux

By

Velocity

MHDPump

Scalability(MHD Pump, Cray XT3)

~20x

Multicore(Inter-core comm. with MPI)

Nodes Cores per

node

Compute Jac+Prec Linear Solve Total

Time (sec) Eff. Time (sec) Eff. Time (sec) Eff.

4096 1 16.9 ------- 4.3 ------- 21.2 ------

2048 2 18.2 93% 4.5 95% 22.6 94%

1024 4 17.7 95% 4.9 88% 22.6 94%

Multi-core Efficiency StudyNew 2.2 GHz Quad Cores Cray XT3/4 (09/29/08)Total of 4096 cores

12800x1280 mesh: ~65M unknowns; Agg = 33;Coarse Operator: ~60K unknownsML: V(1,1) with ILU(1,2)/ILU(1,2)/KLU and Petrov-Galerkin Projection

Our Largest Steady-state Simulation to Date:1+ Billion unknowns

250 Million Quad elements24,000 cores Cray XT3/4

Newton-GMRES / ML: PG-AMG 4 level18 Newton steps

86 Avg. No. Linear Its. / Newton step33 min. for solution

Hydromagnetic Rayleigh-Bernard

Parameters: • Q ~ B0

2 (Chandresekhar number)• Ra (Rayleigh number)

• Buoyancy driven instability initiates flow at high Ra numbers.• Increased values of Q delay the onset of flow. • Domain: 1x20

Ra (fixed Q)

No flow Recirculations

B0 g

Hydro-Magnetic Rayleigh-Bernard Stability: Direct Determination of Nonlinear Equilibrium Solutions

(Steady State Solves, Ra=2500, Q=4)

Temp

Vx

Vy

Bx

By

Jz

Hydro-Magnetic Rayleigh-Bernard Stability: Direct Determination of Nonlinear Equilibrium Solutions

(Steady State Solves, Ra=2500, Q=4)

Robustness and Efficiency of DD and Multilevel Preconditioners

* Failed to converge

DD failed to converge while ML quickly converged

Hydro-Magnetic Rayleigh-Bernard Stability: Direct Determination of Linear Stability and Nonlinear

Equilibrium Solutions (Steady State Solves)

• 2 Direct-to-steady-state solves at a given Q• Arnoldi method using Cayley transform to determine

approximation to 2 eigenvalues with largest real part• Simple linear interpolation to estimate Critical Ra*

Temp.

Vx

Vy

By

Bx

Leading Eigenvector at Bifurcation Point, Ra = 1945.78, Q=10

HRB Shutdown

Ra = 4000

• Q=81: Flow recirculations present!

• Q=144: Zero flow solution!

Vx,B

Jz,B

Arc-length Continuation: Identification ofPitchfork Bifurcation, Q=10

Nonlinear system:

Newton System:

Bordered Solver: Ra

Q=10

Q=0

Design (Two-Parameter) Diagram

Vx

Ra

Q

Ra

Q

No Flow

Buoyancy Driven Flow

• “No flow” does not equal “no-structure” – pressure and magnetic fields must adjust/balance to maintain equilibrium.

• LOCA can perform multi-parameter continuation

• Turning point formulation:

• Newton’s method (2N+1):

• 4 linear solves per Newton iteration:

Bifurcation Tracking(Govaerts 2000)

• Widely used algorithm for small systems:

• J is singular if and only if s = 0

• Turning point formulation (N+1):

• Newton’s method:

• 3 linear solves per Newton iteration

Moore-Spence Minimally Augmented

Extension to large-scale iterative solvers

Leading Mode is different for various Q values

• Analytic solution is on an infinite domain with two bounding surfaces (top and bottom)

• Multiple modes exist, mostly differentiated by number of cells/wavelength.

• Therefore tracking the same eigenmode does not give the stability curve!!!

• Periodic BCs will not fix this issue.

Mode: 20 Cells: Q=100, Ra=4017

Mode: 26 Cells: Q=100, Ra=3757

Q

Ra

Leading mode is 20 cells

Leading mode is 26 cells

2000

3000

4000

• Much work has been done to understand the Island Coalescence problem [1-9].

• GOAL: Understand limits in driven magnetic reconnection

• Poor numerical tools (dissipative, inefficient) led scientists to “find” an asymptotic regime with independent reconnection rates (fast) [2,3,4].

• Novel JFNK algorithms with strict control of dissipation set the record straight

• There is no true fast reconnection asymptotic region [8,9].

Impact of Numerical Algorithms on Scientific Discovery

1. J.M. Finn and P.K. Kaw, Phys. Fluids 20, 72, 19772. P.L. Pritchett and C.C. Wu, Phys. Fluids 22, 2140, 19793. D. Biskamp and H. Welter, Phys. Rev. Letters 44, 1069, 19804. D. Biskamp, Phys. Rev. Lett 87A, 357, 19825. A. Battacharjee, F. Brunel, and T. Tajima, Phys. Fluids 26, 3332, 19836. G.J. Rickard and I.J.D. Craig, Phys. Fluids B 5, 956, 19937. J.C. Dorelli, and J. Birn, J. Geophys. Res. 108, 1133, 20038. D. Knoll and L. Chacon, Phys. Plasmas, 13 (3), 032307, 20069. D. Knoll and L. Chacon, Phys. Rev. Letters., 96, 135001, 2006

Unstructured Mesh and Solution

t=0.0 t=9.0

t=10.0 t=12.0

Sloshing in Resistive MHD: Island Coalescence problem (FE MHD)

Sloshing and Reconnection Rate in Resistive MHD

Sloshing in Resistive MHD: Island Coalescence problem (FE MHD)

• Red Square: FV (Knoll and Chacon 2006)

• Black dots: FE, same mesh as FV (130K)

• Red diamond and triangle: Unstructured mesh (40K)

Preliminary Weak Scaling Results on Island Coalescence Problem (@resistivity =1.0e-3)

Grid Newton GMRES/dt CPU(s)64x64 3.3 3.3 222

128x128 4 4.5 1087256x256 4.5 6.2 5834512x512 4.7 8.3 27380

L. Chacon, FV

Surprising comparison: Only ~4 times slower, considering...• Research code – no investment in efficiency (coming soon)• No physics based preconditioning (AMG)• Unstructured FE mesh vs Structured Fv solver: no leveraging of mesh

structure.

Charon, FE Procs Mesh # Unk Newton /

DtGmres / Newton

Time / Newton

Gmres / Dt

Time / Dt

Est. Serial Time

Ratio

1 64x64 16K 3.9 4.4 2.1 17.2 8.1 810 3.64864 128x128 64K 4.6 5.8 2.6 26.7 11.9 4760 4.37916 256x256 .25M 4.9 6.3 2.9 30.9 14.2 22720 3.894464 512x512 1M 6.2 8.8 4 54.6 24.6 157440 5.7502

Physics based preconditioning will be needed to scale to lower and for faster transients.

Summary

• New unstructured grid, fully implicit, Stabilized FEM developed for single-fluid resistive MHD equations.

• Demonstrating important capabilities:

– Fundamental discretization algorithms (unstructured implicit FE)

– Analysis Tools: Stability/Bifurcation Tracking

• Able to reproduce difficult magnetic reconnection results.

• Largest problem to date: 1 Billion unknowns on 24,000 cores (Cray XT3/4).