characteristicbasedslowwavefastwave …...llnl#pres#731129...

LLNL-PRES-731129This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC. This material is based upon work supported by the U.S. Department of Energy, Office of Science, Advanced Scientific Computing Research, under contract DE-AC02-06CH11357.

Characteristic-‐Based Slow-‐Wave-‐Fast-‐Wave Partitioning for Semi-‐Implicit Time Integration of Atmospheric FlowsIMAGe 2017 Theme of the Year: Workshop on Multiscale Geoscience Numerics, National Center for Atmospheric Research, Boulder, CO

May 16 — 19, 2017

Debojyoti Ghosh1 Emil M. Constantinescu21 Center for Applied Scientific Computing, Lawrence Livermore National Laboratory

2 Mathematics & Computer Science Division, Argonne National Laboratory

LLNL-PRES-7311292

Challenges in Atmospheric Flow Simulations

3D Rising Thermal Bubble: Solution obtained with NUMA (http://faculty.nps.edu/fxgirald/projects/NUMA/)

o Compressibilityo Large spatial gradientsLength Scale Issues

o Sound waves much faster than flow velocitieso Insignificant effect on atmospheric phenomenaTime Scale Issues

Nonoscillatory spatial discretization scheme

Multiscale time integration

2D Rising Thermal Bubble: Solution obtained with HyPar (http://hypar.github.io/)

LLNL-PRES-7311293

Governing Equations Formulations

o COAMP5 (US Navy), MM5 (NCAR/PSU), NMM (NCEP)o Does not conserve mass/momentum/energy

Exner Pressure and Potential Temperature

o WRF (NCAR), NUMA (NPS)o Conserved mass and momentum, not energyo Does not allow inclusion of true diffusion terms

Mass, Momentum, and Potential Temperature

o Examples?o Conserves mass, momentum, and energyo Allows inclusion of viscosity and thermal conduction

Mass, Momentum, and Energy

Main Advantage? Allows the application of the vast number of CFD codes with minimal modifications

Atmospheric flows: small perturbations around hydrostatic balance

Perturbation form of governing equations

Balanced formulation with full quantities

LLNL-PRES-7311294

Acoustic Time Scale (Nonhydrostatic Models)

Explicit Time Integrationo Time step size restricted by acoustic waveso Acoustic waves do not significantly impact any atmospheric phenomenono Split-explicit methods

Implicit Time Integration

o Unconditionally stableo Requires solutions to non-‐linear system or linearized approximation

Implicit-Explicit (IMEX) Time Integration

“Fast” waves implicitly “Slow” waves explicitly

Horizontal-Explicit, Vertical-Implicit

Flux-Partitioned Methods

LLNL-PRES-7311295

IMEX Time Integration

Spatial discretization yieldssemi-‐discrete ODE in time

IMEX: time step constrained by eigenvalues (time scales) of nonstiff component of RHS

Explicit time integration:Runge-‐Kutta methods

Time step constrained by eigenvalues (time scales) of entire RHS

Implicit-‐Explicit (IMEX) time integration:Additive Runge-‐Kutta (ARK) methods

Implicit Explicit

LLNL-PRES-7311296

Objectives

6

@

@t

2

664

⇢

⇢u

⇢v

e

3

775+@

@x

2

664

⇢u

⇢u

2 + p

⇢uv

(e+ p)u

3

775+@

@y

2

664

⇢v

⇢uv

⇢v

2 + p

(e+ p)v

3

775 =

2

664

0⇢g · i⇢g · j

⇢ug · i+ ⇢vg · j

3

775

Develop a conservative, high-order finite-difference based on the compressible Euler equations (conservation of mass, momentum, energy)

Balanced formulation for full quantities:Hydrostatic balance preserved to machine precision without writing equations in terms of perturbations

Flux-partitioning for IMEX time-integration:Isolate acoustic and gravity waves from convective mode

Selective preconditioning of acoustic modes

• Implicit Continuous Eulerian (ICE) technique (Harlow, Amsden, 1971)

• Preconditioning applied to stiff modes (Reynolds, Samtaney, Woodward, 2010)

LLNL-PRES-7311297

Conservative Finite-‐Difference Schemes

Conservative finite-‐difference discretization of a 1D hyperbolic conservation law:

7

j j+10 Nj-1j-1/2 j+1/2

Cell centersCell interfaces

Δx

Spatially-‐discretized ODE in time

5th order WENO(Jiang & Shu, J. Comput. Phys., 1996)

5th order CRWENO(Ghosh & Baeder, SIAM J. Sci. Comput., 2012)

LLNL-PRES-7311298

Balanced, Conservative Finite-‐Difference Formulation (1)

Governing Equations for 2D flows(gravity acting along –y axis)

Hydrostatically balanced equilibrium Pressure gradient balanced by gravitational source

Flow variables at reference altitude

LLNL-PRES-73112999

Extension of Xing & Shu’s method (J. Sci. Comput., 2013)

Isothermal equilibrium Constant potential temperature(Rising thermal bubble)

Stratified atmosphere with a specified Brunt-‐Vaïsälä frequency(Inertia-‐gravity wave)


LLNL-PRES-731129101

0

Differential Form Discretized Form Well-‐Balanced Formulation:Discretized equilibrium holds not just for but for any grid resolution.

Finite-‐Difference DiscretizationDG [p] = ⇢RT0 {% (y)}�1 Ds⇤ [' (y)]

If flux and source discretized by same operator

) Dhp� ⇢RT0 {% (y)}�1 ' (y)

i= D

hp0' (y)� ⇢0% (y)RT0 {% (y)}�1 ' (y)

i= 0

ModifiedGoverningEquations


LLNL-PRES-731129111

1

Represents the WENO/CRWENO finite-‐difference operator with thenon-‐linear weights computed based on G(u)

Diffusion term in upwinding must vanish for equilibrium solution

Constant at steady state

Interpolation

Upwinding(Rusanov)

Differencing

Flux Source


LLNL-PRES-73112912

Verification of Balanced Formulation

12

Case 1: Isothermal equilibriumCase 2: Stratified atmosphere with constant potential temperatureCase 3: Stratified atmosphere with specified Brunt-‐Vaïsälä frequency

Difference of the final solution with the initial solution(Verification that hydrostatic balance is preserved to machine precision)

-0.002

0

0.002

0.004

0.006

0.008

0.01

0 0.2 0.4 0.6 0.8 1

Pressure perturbation

x

InitialReference

WENO5CRWENO5

-2e-05

0

2e-05

4e-05

6e-05

8e-05

0.0001

0 0.2 0.4 0.6 0.8 1

Pressure perturbation

x

InitialReference

WENO5CRWENO5

Small perturbation to isothermal hydrostatic balance

Δp = 0.01 Δp = 0.0001§ Algorithm is able to preserve the

hydrostatic balance at any grid resolution and for any duration to machine precision.

§ Small perturbations to the hydrostatic balance are accurately resolved.

LLNL-PRES-73112913

Characteristic-‐based Flux Partitioning (1)

13

1D Euler equations Semi-‐discrete ODE in timeSpatial discretization

Discretization operator (e.g.:WENO5, CRWENO5)

Flux Jacobian

-3

-2

-1

0

1

2

3

-3 -2.5 -2 -1.5 -1 -0.5 0

Imaginary

Real

CRWENO5

-200

-150

-100

-50

0

50

100

150

200

-250 -200 -150 -100 -50 0 50

Imaginary

Real

u+au-‐a

u

Eigenvalues of the CRWENO5 discretization

Eigenvalues of the right-‐hand-‐side operator (u=0.1, a=1.0, dx=0.0125)

Example: Periodic density sine wave on aunit domain discretized by N=80 points.

Time step size limit forlinear stability

Eigenvalues of the right-‐hand-‐side of the ODE are the eigenvalues of the discretization operator times the characteristic speeds of the physical system

LLNL-PRES-731129141

4

Splitting of the flux Jacobian based on its eigenvalues

“Slow” flux “Fast” Flux

⇤F =

2

40

u+ au� a

3

5


Acoustic flux (fast)

Convective flux (slow)

LLNL-PRES-731129151

5

-200

-150

-100

-50

0

50

100

150

200

-250 -200 -150 -100 -50 0 50

Imaginary

Real

F(u) FF(u) FS(u)

Example: Periodic density sine wave on a unit domain discretized by N=80 points (CRWENO5).

Small difference between the eigenvalues of the complete operator and the split operator.(Not an error)


LLNL-PRES-73112916

IMEX Time Integration with Characteristic-‐based Flux Partitioning (1)

16

Apply Additive Runge-‐Kutta (ARK) time-‐integrators to the split form

Stage values(s stages)

Step completion

Non-‐linear system of equations

Solution-‐dependent weights forthe WENO5/CRWENO5 scheme

Nonlinear flux

LLNL-PRES-731129171

7

Redefine the splitting as

Note: Introduces no error in the governing equation. -200

-150

-100

-50

0

50

100

150

200

-250 -200 -150 -100 -50 0 50

Imaginary

Real

F(u) FF(u) FS(u)

At the beginning of a time step:-‐

Is FF a good approximation at each stage?

Linearization of Flux Partitioning

Linearization of the WENO/CRWENO discretization:

Within a stage, the non-‐linear weights are kept fixed.Example: 2-‐stage ARK method

LLNL-PRES-731129181

8

Linear system of equations for implicit stages:

ARKIMEX 2c• 2nd order accurate• 3 stage (1 explicit, 2 implicit)• L-‐Stable implicit part• Large real stability of explicit part

ARKIMEX 2e• 2nd order accurate• 3 stage (1 explicit, 2 implicit)• L-‐Stable implicit part

ARKIMEX 3• 3rd order accurate• 4 stage (1 explicit, 3 implicit)• L-‐Stable implicit part

ARKIMEX 4• 4th order accurate• 5 stage (1 explicit, 4 implicit)• L-‐Stable implicit part

ARK Methods (PETSc)

Preconditioning (Preliminary attempts)

First order upwind discretizationPeriodic boundaries ignored

o Jacobian-‐free approach à Linear Jacobian definedas a function describing its action on a vector

o Preconditioning matrixà Stored as a sparse matrix

Block n-‐diagonal matrices• Block tri-‐diagonal (1D)• Block penta-‐diagonal (2D)• Block septa-‐diagonal (3D)

IMEX Time Integration with Characteristic-‐based Flux Partitioning (2)

LLNL-PRES-73112919

Example: 1D Density Wave Advection (M∞ = 0.1)

19

Initial solution

CRWENO5, 320 grid points

10−4 10−3 10−2 10−110−13

10−12

10−11

10−10

10−9

10−8

10−7

10−6

10−5

dt

Erro

r

arkimex(2c )arkimex(2e )arkimex(3 )rk (2a )rk (3 )

≈10xExplicit limit

Semi-‐implicit limit

Semi-‐implicit time step size limit 1/ M∞ than explicit time step size limit

Eigenvalues

-800

-600

-400

-200

0

200

400

600

800

-1000-900-800-700-600-500-400-300-200-100 0 100

Imaginary

Real

F(u) FF(u) FS(u)

-100

-50

0

50

100

-100 -80 -60 -40 -20 0

Imaginary

Real

F(u) FF(u) FS(u)

Explicit

Implicit

⇢ = ⇢1 + ⇢ sin (2⇡x) , u = u1, p = p1; 0 x 1⇢ = ⇢1 + ⇢ sin (2⇡x) , u = u1, p = p1; 0 x 1⇢ = ⇢1 + ⇢ sin (2⇡x) , u = u1, p = p1; 0 x 1

LLNL-PRES-731129202

0

Number of function calls Wall time

1e-11

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

1e-04

1e+03 1e+04 1e+05

Erro

r (L

2)

Number of RHS function calls

ARK2c - No preconditionerARK2c - Block Jacobi

ARK2c - ILU(0)RK2a

≈ 5x

1e-11

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

1e-04

1e-01 1e+00 1e+01

Erro

r (L

2)

Wall time (seconds)


ARK2c - ILU(0)RK2a

≈ 4x

Example: 1D Density Wave Advection (M∞ = 0.1)Computational Cost

Number of function calls = (Number of time steps × number of stages) + Number of GMRES iterations(does not reflect cost of constructing preconditioning matrix and inverting it)

LLNL-PRES-731129212

1

-800

-600

-400

-200

0

200

400

600

800

-1000 -800 -600 -400 -200 0

Imaginary

Real

F(u) FF(u) FS(u)

-100

-50

0

50

100

-100 -80 -60 -40 -20 0

Imaginary

Real

F(u) FF(u) FS(u)

Explicit

Implicit

10−4 10−3 10−2 10−1 10010−13

10−12

10−11

10−10

10−9

10−8

10−7

10−6

10−5

dt

Erro

r

arkimex(2c )arkimex(2e )arkimex(3 )rk (2a )rk (3 )

≈100x

Explicit limit


Eigenvalues Initial solution ⇢ = ⇢1 + ⇢ sin (2⇡x) , u = u1, p = p1; 0 x 1⇢ = ⇢1 + ⇢ sin (2⇡x) , u = u1, p = p1; 0 x 1⇢ = ⇢1 + ⇢ sin (2⇡x) , u = u1, p = p1; 0 x 1

CRWENO5, 320 grid points

Semi-‐implicit time step size limit 1/ M∞ than explicit time step size limit

Example: 1D Density Wave Advection (M∞ = 0.01)

LLNL-PRES-731129222

2

1e-13

1e-12

1e-11

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

1e-04

1e+03 1e+04 1e+05 1e+06

Error (L2)



ARK2c - ILU(0)RK2a

≈ 60x

1e-13

1e-12

1e-11

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

1e-04

1e-01 1e+00 1e+01 1e+02

Error (L2)

Wall time (seconds)


ARK2c - ILU(0)RK2a

≈ 45x

Example: 1D Density Wave Advection (M∞ = 0.01)Computational Cost

Number of function calls Wall time

Number of function calls = (Number of time steps × number of stages) + Number of GMRES iterations(does not reflect cost of constructing preconditioning matrix and inverting it)

LLNL-PRES-73112923

Example: 2D Low Mach Isentropic Vortex Convection

23

Freestream flow

Vortex (Strength b = 0.5)

Eigenvalues of the right-‐hand-‐side operators

-15

-10

-5

0

5

10

15

-20-18-16-14-12-10 -8 -6 -4 -2 0 2

Imaginary

Real

F(u) FF(u) FS(u)

-1.5

-1

-0.5

0

0.5

1

1.5

-1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0

Imaginary

Real

F(u) FF(u) FS(u)

Grid: 322 points, CRWENO5

LLNL-PRES-731129242

4

10−3 10−2 10−1 100 10110−14

10−12

10−10

10−8

10−6

10−4

10−2

dt

Erro

r

arkimex(2e )arkimex(2c )arkimex(3 )arkimex(4 )rk (2a )rk (3 )rk (4 )

≈10x


Explicit limit

−3 −2.5 −2 −1.5 −1 −0.5 0 0.5−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5

Re(λ) ∆ t

Im(λ

) ∆ t

ARK2e (IMEX)ARK2e (Expl)

−4 −3.5 −3 −2.5 −2 −1.5 −1 −0.5 0 0.5−3

−2

−1

0

1

2

3

Re(λ) ∆ t

Im(λ

) ∆ t

ARK3 (IMEX)ARK3 (Expl)

o Optimal orders of convergence observed for all methodso Time step size limited by the “slow” eigenvalues.

Example: 2D Low Mach Isentropic Vortex Convection

LLNL-PRES-73112925

Example: Vortex Convection (Computational Cost)

25

1e-09

1e-08

1e-07

1e-06

1e-05

1e-04

1e-03

1e-02

1e+03 1e+04 1e+05

Error (L2)



ARK2c - ILU(0)ARK2c - LU

RK2a

1e-09

1e-08

1e-07

1e-06

1e-05

1e-04

1e-03

1e-02

1e+01 1e+02 1e+03 1e+04

Error (L2)

Wall time (seconds)


ARK2c - ILU(0)ARK2c - LU

RK2a

1e-11

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

1e-04

1e-03

1e-02

1e+03 1e+04 1e+05 1e+06

Error (L2)


ARK3 - No preconditionerARK3 - Block Jacobi

ARK3 - ILU(0)ARK3 - LU

RK3

1e-11

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

1e-04

1e-03

1e-02

1e+01 1e+02 1e+03 1e+04

Error (L2)

Wall time (seconds)

ARK3 - No preconditionerARK3 - Block Jacobi

ARK3 - ILU(0)ARK3 - LU

RK3

ARK 2c ARK 3Number of function calls

Wall time

LLNL-PRES-73112926

Example: Inertia – Gravity Wave

26

o Periodic channel – 300 km x 10 kmo No-‐flux boundary conditions at top and

bottom boundarieso Mean horizontal velocity of 20 m/s in a

uniformly stratified atmosphere (M∞≈ 0.06)o Initial solution – Potential temperature

perturbation

x

y

0 0.5 1 1.5 2 2.5 3x 105

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5x 10−3

Potential temperature perturbations at 3000 seconds (Solutionobtained withWENO5 and ARKIMEX 2e, 1200x50 grid points)

Eigenvalues of the right-‐hand-‐side operators

Grid: 300x10 points, CRWENO5

LLNL-PRES-73112927

Example: Inertia – Gravity Wave

27

Cross-‐sectional potential temperature perturbations at 3000 seconds (y = 5 km) at CFL numbers 0.2 – 13.6

Fastest RK4CFL ~ 1.0, Wall time: 5400 sFunction counts: 24000

CFL Wall time Function countsAbsolute (s) Normalized (/RK4) Absolute Normalized (/RK4)

8.5 6,149 1.14 24,800 1.0313.6 4,118 0.76 17,457 0.7317.0 3,492 0.65 14,820 0.6220.4 2,934 0.54 12,895 0.54

LLNL-PRES-73112928

Example: Rising Thermal Bubble

28

Fastest RK4CFL ~ 0.7, Wall time: 30,154 sFunction counts: 160,000

CFL Wall time Function counts

Absolute (s) Normalized (/RK4) Absolute Normalized

(/RK4)6.9 73,111 2.42 360,016 2.25

34.7 22,104 0.73 111,824 0.70

138.9 8,569 0.28 45,969 0.29

LLNL-PRES-73112929

Conclusions

29

Characteristic-‐based flux splitting:o Partitioning of flux separates the acoustic and entropy modes à Allows larger

time step sizes (determined by flow velocity, not speed of sound).

o Comparison to alternatives

• Vs. explicit time integration: Larger time stepsàMore efficient algorithm

• Vs. implicit time integration: Semi-‐implicit solves a linear system without anyapproximations to the overall governing equations (as opposed to: solve non-‐linear system of equations or linearize governing equations in a time step).

Future work:o Improve efficiency of the linear solve

• Better preconditioning of the linear system

o Extend to 3D flow problems

Thank you. Questions?

Papers:

o Ghosh, D., Constantinescu, E. M., Well-Balanced, Conservative Finite-Difference Algorithm for Atmospheric Flows, AIAA Journal, 54 (4), 2016

o Ghosh, D., Constantinescu, E. M., Semi-Implicit Time Integration of Atmospheric Flows with Characteristic-Based Flux Partitioning, SIAM J. Sci. Comput., 38 (3), 2016

Code: http://hypar.github.io/

characteristic*basedslow*wave*fast*wave …...llnl#pres#731129...

Documents

characteristicbasedslowwavefastwave …...llnl#pres#731129...