Implementing ALE Motion in a Discontinuous Finite Element Hydro Code*

Manoj K. Prasad, Jose L. Milovich, Aleksei I. Shestakov, David S. Kershaw, and Michael J. Shaw

Lawrence Livermore National Laboratory, Livermore, CA 94550, USA

*Work performed under the auspices of the U.S. Department of Energy by the Lawrence Livermore National Laboratory under contract number W-7405-ENG-48

Motivation

ALE Hydro code that combines the accuracy of a higher order Godunov scheme with the

unstructured mesh capabilities of finite elements which can be explicitly evolved in time.

Brief History of Discontinuous Galerkin (DG) Finite Element Methods

• P. LeSaint & P. A. Raviart: rigorous error analysis & convergence rates for DG finite element solution of steady state linear neutron transport equations, 1974.

D. S. Kershaw & J. A. Harte, implemented a fully implicit 2D time dependent linear neutron transport on triangular mesh – solving (3 x 3) block lower triangular linear system at each time step – using the partial ordering scheme of LeSaint & Raviart with no cycles, LLNL 1993.

• G. Chavent & G. Salazano: applied DG methods to time dependent nonlinearporous media equations with explicit Euler time stepping, 1982.

• G. Chavent & B. Cockburn: linear error analysis and slope limiters for approximating shocks for scalar conservation laws, INRIA 1989.

• B. Cockburn & C.-W. Shu: combined Runge-Kutta explicit time discretization with DG space discretization for scalar conservation laws, 1991.

Model

• Godunov scheme: 2nd order.• Piecewise Linear Finite Element discretization.• Roe upwind surface flux & Harten-Hyman Entropy fix• Explicit time stepping : 2nd order Runge-Kutta.• 3D shock stabilization: VanLeer “minmod” by Quadratic

Programming + Lapidus artificial viscosity.• Algorithm appears in: Computer Methods in Applied Mechanics and

Engineering 158, 81-116 (1998).

Implementation

• 3D Unstructured Mesh: tets, pyramids, prisms & hexes.• 1-Step Arbitrary Lagrangian-Eulerian (ALE) moving mesh.• 3D Geometries: Cartesian, Cylindrical & Spherical.• Object oriented C++ design untangles mesh from physics.• Parallelized using Domain Decomposition & MPI message passing. • Portability: code runs on uniprocessor workstations and massively

parallel platforms with distributed and shared memory.• Integrated with electron, radiation diffusion transport & laser ray

tracing for 3D ICF simulations: Computer Methods in Applied Mechanics and Engineering 187, 181-200 (2000).

Allowable Cell Types

Mesh connectivity requires that cells share like-kind faces. No slide lines allowed.

Tetrahedron

PyramidPrism

Hexahedron

3D ALE Moving Mesh Hydro Equations

( )( )( )( )

( )

Mass / ForceBody External(EOS) MassEnergy / Internal=) , (

MassEnergy / Total=) , ( 21=

Pressure=P Velocity,= Density,

0

3

2

1

33

22

11

3

2

1

0

=

+

=

=

−+−+−+−+

−

=

=

=∂∂

+∂∂

i

ii

iigjji

gjjj

gjjj

gjjj

gjj

jii

i

i

GPI

PIvvE

v

GvGGG

JS

vvEvPvvvPvvvPvvvP

vv

F

Evvv

JA

SxF

tA

ρ

ρ

ρ

ρρρρ

ρδρδρδ

ρ

η

ρρρρρ

ααα

ααα

r

igi

gi

ijjii

j

i

ii

jii

i

vv

v

JJJJxx

J

txv

txx

txxx

x

≈

=

==≡

∂∂

=

==

−

: grid Lagrangian

,0 : grid Eulerian

|| , ,

, : velocityGrid

) 0 (

, ) , ( = :mesh moving ALE

t,:st variableIndependen

1ji0ij

gi

0

0

0

η∂∂

Conservative form: Grid Motion:

Finite Element Discretization

faceselement across continuous be torequired is faceselement across ousdiscontinu generalin are P , ,

elementin nodes ofnumber = others allat 0 and nodeat 1 fns basisLinear =

, nodeat ),(=

elementeach in tion representa )( ericIsoparamet

P, , , : Variables Dependent Linear Piecewise

1=

xv

n

QQQQ

xvQ

n

r

r

l

lr

r

rr

l

lll

l

ρ

φ

ξφ

ξ

ρ

=

≡∑

Roe Upwind Surface Flux

+− ≠

Γ

Γ∇

∇

Γ

∫∫∫∫

AA

N

FN

FA

i

x hereupwind fluuse Roe's

iii

iKt

rr

44 344 21

r

rr

ll

ll

) elements finite uous(discontin generalIn side. (+) theto

side (-) thefrom surface tonormal pointing outward is where

)(d - )x(d F=

)x(d -=)x(d

: surfaceK with element each for EquationsGalerkin

K

KK

K i

iK

K

α∂

α

αα

φφ

φφ∂

[ ]{ }

),,0max( ,|)|,0max(||||:fix Entropy Hyman-Harten

),(

:Average Roeby the definedmatrix ationdiagonaliz the speed, sound theis

)( ),( ,matrix eigenvalue diagonala is where

)( 21

*****

1****

***

ii

**

**i

*i

**

ii1***

iii

λλλλελελλ

∂

λ

β

ααβββαβαα

ααααα

−−=−+→

Λ≡∂

≡−≡−

±−−=Λ

−Λ++≡

+−

−+−+−

+−−+−

RRAFNWAAWFNFN

Rc

cvvNvvN

FNFNRsignRFNFNFN

iiii

gii

gii

iiiiRoei

Roe Averages:Roe Flux:

+_

N

=

*

Boundary Conditions

• Lagrange BC: specified pressure or normal velocity with no mass flux

• We use “ghost” states on the outside of boundary faces

• Roe flux from the “ghost” and interior states is the required boundary flux

gii

bndry

giiii

giii

gjjj

bndry

bndrygii

bndry

bndry

bndry

Roeii

giiii

gjjjii

ghosti

ghostghost

vNP

vNvN

vvNcvvNPP

PvN

PN

PN

PN

FN

vNvN

vvNNvvPP

for solve given, If

for it use bndry,on given If

)]()][([

,

0:boundaryon flux Roe

:faceboundary on velocity normal Average Roe

)]([2 ,

:facesboundary ofoutsideon statesGhost

int*intintint

3

2

1

*

intint

intint

−+−+=

=

=

−−=

==

ρ

ρρ

α

Almost Lagrangian Grid Velocity

• Grid Position and Velocity must be continuous while fluid velocity is generally discontinuous across elements.

• We use a “least squares” estimate to determine the grid velocity from the fluid velocity.

• We can impose 1,2, or 3 linear constraints on the grid velocity.

• For 1D this gives an exact Lagrangian code. In general we get an “almost” Lagrange code. estimate. squaresleast

thein on sconstraintlinear 3,2,1 impose also can We

from determinedor

BC velocity normalby given is : facesboundary For

for 0] [ solve : facesinterior For

:or BC acrossflux mass no requiringby determined is

node share that faces all :where

, ][ Minimize

:for nodeat velocity fluid of estimate SquaresLeast

1

2

}{

ginc

bndry

fn

ffRoeiin

nf

n

gififff

f

ginif

gin

vn

P

Yf

YYFNf

fY

nf

vNYYvN

vn

n

nn

n

nnnn

n

n

=

=

≡

≡−∑

Roe Mass Flux

Normal Grid Velocity giving Zero Mass Flux on Face f

Least squares estimate of velocity at vertex p from all faces f around it

Explicit 2nd order Runge-Kutta Time Advancement

|))(||,)(||,)(max(|||

||

Min

3. ,

*****max

*

max*

3

K

cvvNcvvNvvN

d

xdt

CFLtCFLt

giii

giii

giii

K

KCourant

Courant

+−−−−=

Γ=

≈≤∆

∫∫

∂

λ

λ

• The DGM equations, for each element K, reduce to a system of n ODE s for the moments:

• We use 2nd order Runge-Kutta (RK) time stepping.

• Courant time step control with CFL number of about 1/3.

• Cockburn & Shu, Math. Comput. 52 (1989) 411.

,

~

, )~~(

, ~

~ :veConservati , )P , , ( :Primitive

),(= VariablesLinear Piecewise

3

3

1

31

n

1=

∫

∫

∫

∑

>=<∂∂

=

><−><≈><−

=ΦΦ=

=

−

−

K

K

mK

llmmlml

xd

xdQQ

BA

T

AATBB

xdMA

AvB

QQ

β

ααβ

ββαβαα

ϕϕ

ρ

ξφ

r

rl

ll

xdAMK

ll3 ∫= ϕ

3D VanLeer Minmod Slope Limiting via Quadratic Programming

limiter slope MINMOD sVanLeer' toreduces this2= 1D,For

and :subject to

)( 21 Minimizing

by Find :problem gProgrammin Quadratic

eit thru thmodify wesconstraint sVanLeers' violates

1

VL max

VLmin

1

2VL

VL

n

QQwQQQ

QQw

Q

QIf

n

n

⟩⟨=⟩⟨≤≤⟩⟨

−

∑

∑

=

=

llll

llll

l

l

boundaries on nodesfor 0boundaries generalon nodesfor 1

)1( :Nodes For

node sharing elements allover max)min,( : limitsion stabilizatVanLeer

nodeat of aroundn fluctuatio )21=( let K,element each In

0 , , :t requiremen Physical

maxmin,

maxmin

symmetry

QQQBoundary

lQQQQQQ

QQQ,...n,QQQ

EP

facebndry

l

l

l

==

⟩⟨+⟩⟨−→⟩⟨

⟩⟨=⟩⟨⟩⟨≤≤⟩⟨

⟩⟨≡+⟩⟨=

≥

ωω

ωω

δ

δδ

ρ

l

l

l

l• Positivity of density, pressure must be enforced.

• VanLeer’s shock stabilization limits the fluctuations of variables around their average values within an element.

• Our 3D unstructured mesh generalization of VanLeer’s stabilization has a unique and simple solution.

Lapidus Artificial Viscosity

• 3D strong shocks need “extra” diffusive artificial viscosity.

• We use Lapidus type artificial viscosity.

• Used only in vicinity of strong shocks to keep 2nd order accuracy in smooth regions.

• Our DGM implementation: Correction to the Roe Flux forinterior faces.

∫

∫

∫

∫∫

∂

∂

∂

+−

∂∂

+

Γ

Γ±−−=

≈=

±

Γ><−><−

Γ→Γ

→∆∆→

∇⋅∇+=⋅∇+∂

K

gii

gii

Ki

L

L

Roei

Roei

L

ityVisLapidus

Lt

cvvvvNl

D

AAl

D

FNFN

txD

ADSFA

K

K***

*

*

KK

KK

KK

cos

d

d ) |)(||,)(| ( max

.3 ,

face theof sideeither on elements 2 the torefers

)(d ][

)(d )(d

:shocks strongnear facesinterior For:tion implementa DGMOur

0),( as 0

)(

λ

κλκ

φ

φφ

αα

αα

αααα

l

ll

rr

r

43421

rrrr

Need for Lapidus Viscosity

2. at time viscosity LapidusWith13. at time viscositypidusWithout La

:)0at vs.( mesh ofPlot

)26,0(at 6)(0,-70.746 velocity Normal :BC10 , 619.0 , 260

hexes of mesh 1x 6x 208 Limit Godunov

mode LagrangianGeometry ),,( Cartesian 3D

)]1(10000 ,7466.70 ,4[] , ,[ )1 ,0 ,1(] , ,[

2613 : , 130 :0 , 5/3 withgas Law Ideal

:State Initial

:contact no andshock strongVery : ProblemTubeShock

==

=

==≤≤≤≤≤≤

−−=

=≤≤≤≤

===

tt

zxy

xzyx

zyx

PvPv

xRightxLeftvv

Rightx

Leftx

zy

γρ

ρ

γγ

Shock Stabilized Solution

To avoid tampering with the solution in situations where accuracy is required, we implemented a hybridization scheme that smoothly interpolates from adiabatic (Rayleigh-Taylor) flows to strong shock regions

flowsadiabaticssshocksstrongrr

srBB

sBBrrBsBHybrid

Godunov

RKGodunovVLhybrid

near 1 withrate productionentropy measures near 0 withstrengthshock measures

1,0 , :where

])1([ )1(: variablesprimitive of blend a Construct

→→

≤≤><≡

+−+−=

αα

αααα

C++ Object Oriented Code

Cell Classtets, pyramids, prisms, hexes

Face Classtriangles, quads

Node ClassC++ Class Structure:

EOS ClassIdeal Gas, Real Gas, Sesame Table Lookup

• Mesh Generator creates class objects and appropriate pointers across the various classes

• Physics modules process objects through class virtual functions using pointers to access data across classes

Parallelization Strategy• Domain Decomposition of Physical Space

- Each PE only knows about a piece of the whole domain plus a layer of ghost cells.

• Use of the SPMD programming model• Communication between processors via MPI• Input Files: Modification of AVS UCD file to allow for

ghost cells and cell ownership- Contain PE ownership of cells- Contain local and global sequence number of nodes and

cells• Output Files: Every processor writes its own file (no ghost

cell information is written)

Parallelization of the Hydro Package

Three kind of objects are required:• Calculation of fluxes:

– need fields across faces • Calculation of Van Leer limiting:

– need average fields of cells around nodes.• Calculation of Lagrangian motion:

– needs normal velocity that zeros mass flux

Mesh Partitioning for Hydro

PE2

PE1PE0

PE3

Needed for fluxes

Needed for fluxes

Needed for VL, vgrid

Also VL, vgrid

Also VL, vgrid

Examples

Hydro Test Problems

• Noh Implosion Problem, 1D Spherical Geometry• Sedov Point Explosion, 2D Cylindrical Geometry• Rayleigh-Taylor Instability, 2D & 3D Cartesian Geometries• Imploding Sphere with Tetrahedral Mesh, 3D Cartesian Geometry

Noh Implosion Problem

geomtries )(for )2,1,0( where)/1( , 0 , 1

:hregion wit unshockedOuter

, , 0:hregion witshock -post Central

:shock expandingan by separated regions 2:solution analyticalSimilar -Self

boundary.symmetry a into )1(ity unit velocwith )0 1,( gas coldinitially ofImpact

00

0

00

phericalindrical,splanar,cylgrtvPv

PPvv

vP

g

sss

=−==−=

====

−===

ρρ

ρρ

ρ

0at vs. and of Plots3/64 , 64

3/4

sphereouter on -1Velocity Normal :BC4/0 , 2/0 , 10

hexes of mesh 1x 1x 1280 , 1 , 1 , 5/3

: withgas Law Ideal :Condition Initial

gas implodingy sphericalla of stagnation simulates),,(Geometry Spherical 1D

0

====

==

=≤≤≤≤≤≤

=−===

φθρρ

πφπθ

ργγ

φθ

rPP

Ud Shock Spee

r

Pv

r

ss

s

r

Sedov Point Explosion Problem

−

−

−

→+=→

→+=→

→−+=→

===

=

====

1/ , )1/(2

1/ , )1/(2

1/ , )1/()1(151667.1 : 3/5For

)/()5/2( :

)/( : :Solution AnalyticalSimilar -Self Shock. Strength Infinite

)( , 0 , 0 , :Gas Cold Law Idealensity Constant D :Condition Initial

20

0

5/1300

5/10

20

300

sss

sss

ss

s

s

rrUPP

rrUuu

rrk

tEkUVelocityShock

tEkrPositionShock

xEEvP

γρ

γ

γγρρρ

γρ

ρ

δρρ

γrr

1 at time0for vs./ , /

, ) vs.(mesh ofPlot 3/5 , 4935889. , 1

0Velocity Normal :BC20 , 125.1,0

Hexes ofMesh 45 x 1 x 45),,(Geometry lCylindrica 3D

Mode Lagrangian

00

==

====≤≤≤≤

zrPPzr

E

zr

zr

ssρρ

γρ

πθ

θ

3D Rayleigh-Taylor Instability

10 Interfaceat RatioDensity R1)1)/(R-(R ,) / g (2 RateGrowth Linear

z .00045 .001, , / 2)(

)cos()cos(k .5z :onPerturbati :Interface

.33671717g ,1.1 ,d g PP

2for ,)/(100for ,)/(100

10 :Gas Ideal

1/2

2/122

x

0

9/1

9/1

==+==

∆===+=

==

==∫+=

≤≤=

≤≤=

=

αλαπ

χλλπ

χδλ

λρ

λλρ

λρ

γ

yx

y

kkk

ykxz

lz

zlzzlz

modesr rectangula & square 3D 2D,:for time vs.ude)Log(amplit ofPlot .1 amplitude when interface ofPlot

0 :2Dfor

3 :moder Rectangula

:mode Square0Velocity Normal :BC

Mode Lagrange2 0 , 5. , 0

hexes 40 x 20 x 20 :GridGeometry )(Cartesian 3D

λ

λλ

≈

=

=

==

≤≤≤≤

y

yx

yx

k

kk

kk

zyx

x,y,z

Unstructured Mesh Implosion

• Unstructured Tetrahedral Mesh• 3D Cartesian (x,y,z) Geometry• Icosahedral wedge domain bounded by:

• 28,208 Tetrahedra (50 radial cells)• 5791 nodes & 58,455 faces• Grid generated by: LaGriT

(www.t12.lanl.gov/~lagrit)• Domain decomposed by: METIS

(www-users.cs.umn.edu/~karypis/metis)• 64 Processing Elements (PE)

)5/ ,5

1(cos) ,(

5/ : 21 :

1 πφθ

πϕ

±=

±==

−andoriginthroughplane

planesazimuthalrsphere

Unstructured Mesh Implosion

27.46 )max( run 1D 26, )max( run 3D

.6 .58, ,56.at s. Density vof Plots4/0 /2,0 ,10 hexes, 1x 1x 200

run comparison ),,(Geometry D1

563.)max( with6.at Density of view on-Side :ofPlot

3/4 :BCmotion Grid ALE

mode Lagrange0 1, :IC

Gas Law 5/3 Ideal

=≈

=≤≤≤≤≤≤

==

=

====

ρρ

πϕπθϕθ

ργ

trr

rSpherical

zt

PdConstraine

vP

bndry

r

Integrated Hydro Test Problems

• Laser Driven Capsule. • Radiation Driven Capsule. • Electron Thermal Conduction.• Details in: Journal of Computational Physics 170, 81-111 (2001).

Solution Method

Use Operator Splitting. Processes are solved in the following order:

– Hydrodynamics – Material Properties – Laser Energy Deposition – Heat conduction– Radiation Transport– Synchronization

),,,,( epEρρρ v

,...),,,( v RPcT κκ

)( eS)(T

),( TEr

),,( TEe

Equations of Interest

fieldradiation theoft coefficiendiffusion derivative Lagrangian

couplingmatter toradiation laser) a todue (e.g. source external

flux,heat of divergence Force,gravity

)(

)(

)(0(

====

==

−∇⋅∇=

+++⋅=⋅∇+∂

=⋅∇+∂=⋅∇+∂

∫

ν

νε

ε

ε

ννν

νεεερ

ρ

ρ

νρρ

ρρρρ

Ddtd

KS

H

KuDudtd

dKSHE

νε

Et

t

t

g

vgF

gFvv)

v

Laser Ray Tracer

density critical density,electron

11

plasma edunmagnetiz cold aIn 2c

dtd

motionofEquation Ray

2

22

22

2

2

==

−=−=

−∇=

ce

c

epe

nnnn

ωω

η

ηx

Approximations• Constant density in a cell

– Advantages:• Easy to implement• Good for highly underdense plasmas• Ray equation of motion is linear in time within a cell

– Disadvantages:• Trajectories may be inaccurate

• Constant density gradient in a cell– Advantages:

• Rays can refract inside cell, and cell boundaries• Energy deposited more evenly• Ray equation is quadratic in time within a cell

– Disadvantages:• Care needed to solve face crossing equation (slow wandering rays)

Radiation Package

• The equations are:

• Note that the transport term in the T equation is missing, since it has been split off, in the numerical scheme.

• In the diffusion approximation there is no guarantee that the radiation will travel at c. Most codes use a flux limiting, ie.

[ ][ ] 4)/4()( , )(

)(

TcTBETBcTc

ETBcEDE

rPtv

rPrrrt

σρκρ

ρκ

=−−=∂

−+∇⋅∇=∂

[ ]|)|/|)(|/2(1/3

rrrrR

r EEcDDlcD ∇+→=

Laser Driven Capsule• 12 Beams with circular cross-section and 101 rays each

• Beams are distributed on the vertices of an Icosahedron

• Physical Domain: (11,580 tets, 2053 nodes, 23,320 faces)

Laser Driven Capsule

• Choose parameters to create different regions of dominance of physics packages (laser, heat conduction, hydro)

• Absorbed laser energy is a source of heat which is quickly diffused over the surface and drives a supersonic thermal wave inward.

• When heat wave slows down, hydro takes over and an imploding shock wave arises

• Ideal gas EOS:

• Heat Flux:

• Laser critical density• Initial radius: • Initial density:

• Initial temperature: 1 eV• Boundary pressure

• Laser: 12 beams, circular cross section. • Intensity:• Laser pulse: Flat top, on t=0, off t=2 ns• At t=12ns, max(r) = 50 x initial density

Laser Driven Capsule

KeV) ergs/(g 10c ,4.1 15v ==γ

190

2/50 10 , , ==∇− χχχχ TT

3g/cm 001668.0=cρ

crrrr ρρρρρρ 01.0)( ,2)1615( ,10)

87( 0c0c0 ===≤

370 erg/cm 10 67.6),10/( ⋅== Tpp cb ρ

213 W/cm10875.2 ⋅

cm 2.00 =r

Radiatively Driven Capsule

• 10 cells in the gas, 12 in fuel and 11 in ablator (5104 tets, 1246 nodes, 10,915 faces)

• Use LANL SESAME tables• IC: T=.001 keV, • BC:

– Hydro: Pbdry=58. GPa (corresponds to Be at ρ=1.85 g/cc and T)

– Heat Conduction: F=0– Radiation: Energy source (correspond to

Tr = 0.16 keV)• Mesh generated with LaGrit and partitioned

with METIS

Radiatevely Driven Capsule

• Thin imploding ablator shell.• Electron & radiation T coupled in central region.• From central region to ablation front, electron & radiation T is cold.• Beyond ablation front, electron & radiation T coupled.

Profiles just after shock bouncing time

Profiles just after shock bouncing time

Conclusions on DG ALE Hydro scheme

• DGM is a viable ALE hydro scheme on 3D unstructured meshes

• Robustness requires further work on: artificial viscosities, velocity filtering, & mesh relaxation

• Symmetry & local postprocessing on unstructured meshes………?????????

Summary• Described the multi-physics unstructured 3D parallel code: ICF3D• Tested the different physics packages independently and in

combination.• Show parallel efficiency and scalability to large number of

processors.• Demonstrate the ability of computing on arbitrarily unstructured

meshes• Show accuracy of results.• Needs more work on hydro scheme:

• Artificial Viscosity • Velocity filtering• Mesh relaxation• Unstructured mesh refinement