Recent Developments in the NRL Spectral Element Atmospheric Model

(NSEAM)*

Francis X. Giraldo

www.nrlmry.navy.mil/~giraldo/projects/nseam.html

*Funded by the Office of Naval Research

Motivation

Mission of NRL-MRY:

• Develop NWP models for the Navy

Computing Resources:

• High Performance Computing has shifted towards distributed-memory architectures comprised of tens of thousands of processors (e.g., clustered systems). Current fastest system has 32,000 processors (BlueGene by IBM).

Exploiting New Architectures:

• Need numerical discretization methods that decompose the domain naturally into many smaller subdomains

• Each subdomain can then be solved independently on-processor

Motivation (continued)

State-of-the-Art Global Forecast Models:• Based on the spectral transformspectral transform method (i.e. spherical

harmonics)• Spectral transform method uses global approximation

functions and Nproc scales like O(T) and costs O(T^3)

New Model:• NSEAM is based on the spectral elementspectral element method which

combines the high-order accuracy of spectral transforms with domain decomposition property of finite elements

• Uses local approximation functions and Nproc scales like O(T^2) and costs O(N T^2)

Domain Decomposition

ST Models: Nproc = O(T) NSEAM: Nproc=O(T^2)

Spectral Transform Method(Legendre Modal Functions)

-1 -0.5 0 0.5 1-1

0

1

x

y(x

)

y1

y2

y3

y4

y5

Spectral Element Method(Legendre Nodal Functions)

-1 -0.5 0 0.5 1-1

0

1

x

y(x

)

y1

y2

y3

y4

y5

What We Need for a New Model

To construct a model the following things are required:

1. Governing set of Partial Differential Equations along with initial and boundary conditions (PDE)

2. A Coordinate System (CS)

3. A Grid Generator in order to place solution values within the domain (GG)

4. A Spatial Discretization method in order to approximate derivatives (SD)

5. A Time-Integrator in order to advance the solution in time (TI)

6. A Very Large Computer (VLC)

Description of NSEAM

• Hydrostatic primitive equations (PDE)

• Equations formulated in Cartesian and sigma coordinates (CS)

• Spectral Element method in horizontal (SD)

• Finite Volume method in vertical (SD)

• 2nd order semi-implicit in time (TI)

• GMRES with block Jacobi Preconditioning (TI)

• Message-Passing Interface (VLC)

• Can use all types of grids including quadrilateral and triangular adaptive grids (GG)

pcP

Hydrostatic Primitive Equations

0

u

t

)()(2

2uS

Pcuk

a

zuuu

t

up

)(

Sut

(Mass)

(Momentum)

(Energy)

(State)

)(qSq

qut

q

(Tracers such as

Moisture/Salinity)

• Primitive Spherical Form:

• Pole problem can be avoided by:– Using Gauss quadrature (as in spherical harmonics)– Creating a hole in the domain (as in gridpoint models)– Using Cartesian coordinates (NSEAM)

0cos

cos

1

vu

at

Why Cartesian Coordinates?

• Primitive Equations:

• Write Primitive Equations as:

• Weak Problem Statement: Find

– such that

– where

0)(

qSt

qqL

)(1 Hq

0 dqLy

e

N

e

e

1

y

SE Method

)(qSt

q

• Polynomial space for quads and tris

• Polynomial Approximation on the Quadrilateral:

– With polynomial space

• Polynomial Approximation on the Triangle:

– With polynomial space

i

N

iii

N

ii qqzyxzyxq

P

2)1(

11

),(ˆ),,(),,( yy

SE Method: Basis Functions

i

NN

iii

N

ii qqzyxzyxq

P

)2)(1(

2

1

11

),(ˆ),,(),,( yy

},0|{ NjiP jiN

}0|{ NjiP jiN

NN PP

Accuracy of SE Method (Shallow Water on Sphere)

Icosahedral TelescopingHexahedral

Icosahedral Lat-Lon Adaptive

Grids

Lat-Lon

Thin/Reduced

• For the system:

• There are 2 classes of possible time-integrators:– Eulerian - fixed reference frame– Lagrangian - moving reference frame

• There are 2 subclasses of these methods:– Explicit – strict time-step restriction– Semi-Implicit – lenient time-step restriction– Fully Implicit – no time-step restriction

)(qSt

q

Time-Integrators

• Start with:

• Discretize by 2nd order TI

– Everything on the RHS is known. Very fast to solve because only requires the evaluation of a vector of length 3*Np*Nlev (detail of this approach can be found in MWR 2004 paper)

)(qSt

q

Explicit Time-Integrator

mn

mm

mn

mm

n qStqq

)(2

0

2

0

1

• Start with:

• Discretize by 2nd order TI

– A global sparse NONLINEAR system must be solved. This can be quite expensive! The size of the system is (3*Np*Nlev) x (3*Np*Nlev)

)(qSt

q

Fully-Implicit Time-Integrator

mn

mm

nn qqtSq

2

0

11 )(

• Start with:

• Rewrite as:

• Discretize by 2nd order TI

– A global sparse LINEAR system must be solved. Via a vertical mode decomposition we can do this by solving: Nlev systems of Np x Np sparse linear systems. This is very fast! (details of this approach can be found in QJRMS 2005 paper).

)(qSt

q

Semi-Implicit Time-Integrator

)()()( qLqLqSt

q

mn

mm

mn

mm

mn

mm

n qtLqStqq2

1

2

0

2

0

1 )(

Performance for T239 L30 (DT=300 seconds on IBM SP4)

(Spectral Model)

(SE Model)

Performance of NSEAM (216 Processors on IBM SP4)

T159 L24

T239 L30

T580 L60

NOGAPS T80 L24

Rossby-Haurwitz Wave 4(T=5 days)

NSEAM T80 L24

Rossby-Haurwitz Wave 4(T=5 days)

ECMWF T106 L30

Held-Suarez Test Case(T=1200 days)

NSEAM T64 L20

Held-Suarez Test Case(T=1200 days)

Zonal Wind (m/s) Temperature (K)

Comparison of the explicit (dashed line) and semi-implicit (solid line) NSEAM. The semi-implicit uses a time-step 8 times larger than the explicit. The difference between the two are virtually indistinguishable which is what one would hope to see.

Jablonowski-Williamson Balanced Initial State(Surface Pressure Error for NSEAM T185 L26)

-90 Meridian 0 Meridian

The terrain field for NSEAM T239 from two different views of the globe. In this test an initially-balanced still atmosphere is integrated for 30 days. The atmosphere remained perfectly balanced for the length of the integration which is exactly what should happen.

Initially-Balanced Still Atmosphere(NSEAM T239 L30 with terrain after 30 days)

No Terrain Terrain

The viewpoint of these plots is from the –120 meridian. The temperature contours look similar for both but the effects of the Rockies, Andes, and Antarctica are clearly visible on the right panel.

Held-Suarez Test Case(Surface Temp for NSEAM T185 L24 after 30 days)

Pressure Temperature

U Velocity V Velocity

Jablonowski-Williamson Baroclinic Instability(Surface Values at Day 9 for NSEAM T185 L26)

Jablonowski-Williamson Baroclinic Instability

Jablonowski-Williamson Baroclinic Instability(Surface Values for NSEAM T185 L26 during 0-30 days)

Pressure Temperature

Future Work

• Further testing of NSEAM with terrain

• Physical parameterization (sub-grid scale processes)

• Non-hydrostatic equations (to better handle dx < 10 km)

• Non-hydrostatic Coastal Ocean Model (underway)