a community t errain-following o cean m odeling s ystem

A Community Terrain-following Ocean Modeling

System

2003 Terrain-Following Ocean Models Users WorkshopPMEL, Seattle, WA, August 5, 2003

Developers and CollaboratorsHernan G. Arango

Alexander F. Shchepetkin

W. Paul Budgell

Bruce D. Cornuelle

Emanuele DiLorenzo

Tal Ezer

Mark Hadfield

Kate Hedstrom

Robert Hetland

John Klinck

Arthur J. Miller

Andrew M. Moore

Christopher Sherwood

Rich Signell

John C. Warner

John Wilkin

Rutgers University

UCLA

IMR, Norway

SIO

SIO

Princeton University

NIWA, New Zealand

University of Alaska, ARSC

TAMU

Old Dominion

SIO

University of Colorado

USGS/WHOI

SACLANT

USGS/WHOI

Rutgers University

Executive Committee

Dale B. Haidvogel

James C. McWilliams

Robert Street

Rutgers University

UCLA

Stanford University

ONR Support

Manuel Fiadeiro

Terri Paluszkiewicz

Charles Linwood Vincent

• To design, develop and test an expert ocean modeling system for scientific and operational applications over a wide range of scales from coastal to global

• To provide a platform for coupling with operational atmospheric models, sediment models, and ecosystem models

• To support multiple levels of nesting and composed grids

• To provide tangent linear and adjoint models for variational data assimilation, ensemble forecasting, and stability analysis

• To provide a framework for massive parallel computations

Objectives

• Use state-of-the-art advances in numerical techniques, subgrid-scale parameterizations, data assimilation, nesting, computational performance and parallelization

• Modular design with ROMS as a prototype

• Test and evaluate the computational kernel and various algorithms and parameterizations

• Build a suite of test cases and application databases

• Provide a web-based support to the user community and a linkage to primary developers

Approach

ROMS/TOMS 2.0 released to beta testers onJanuary 16, 2003 and full user community onJune 30, 2003.

Accomplishments

Built tangent linear and adjoint models and tested on realistic applications of the US West and East Coasts: eigenmodes and adjoint eigenmodes, singular vectors, pseudospectra, forcing singular vectors, stochastic optimals, and ensemble forecasting.

(Relief Image from NOAA Animation by Rutgers)

The model is used in oceanographic studies in over 30 countries by:

• Universities

• Government Agencies

• Research Organizations

Ocean Modeling Web Site

http://www.ocean-modeling.org/

• Free-surface, hydrostatic, primitive equation model

• Generalized, terrain-following vertical coordinates

• Boundary-fitted, orthogonal curvilinear, horizontal coordinates on an Arakawa C-grid

• Non-homogeneous predictor/corrector time-stepping algorithm

• Accurate discretization of the baroclinic pressure gradient term

• High-order advection schemes

• Continuous, monotonic reconstruction of vertical gradients to maintain high-order accuracy

Kernel Attributes

Vertical Terrain-following Coordinates

Dubrovnik(Croatia)

Vieste(Italy)

Longitude

Depth(m)

Curvilinear Transformation

CartesianSphericalPolar

• Modular, efficient, and portable F90/F95 Fortran code with dynamical allocation of memory via de-referenced pointer structures.

• C-preprocessing managing

• Multiple levels of nesting and composed grids

• Lateral boundary conditions options for closed, periodic, and radiation

• Arbitrary number of tracers (active and passive)

• Input and output NetCDF data structure

• Support for parallel execution on both shared- and distributed -memory architectures

Code Design

Model Grid Configuration

Nested Composed

• Coarse-grained parallelization

Parallel Framework

}

} Nx

Ny

Parallel Tile Partitions

8 x 8

• Coarse-grained parallelization

Parallel Framework

• Shared-memory, compiler depend directives MAIN (OpenMP 2.0 standard)

• Distributed-memory (MPI)

• Optimized for cache-bound computers

• ZIG-ZAG cycling sequence of tile partitions

• Few synchronization points

• Serial and Parallel I/O (via NetCDF)

• Efficiency 4-64 threads

(Ezer)

The cost of saving output and global averaging is much

higher for the MPI code

(for the shared-memory SGI machine)

(Ezer)

• Horizontal mixing of tracers along level, geopotential, isopycnic surfaces

• Transverse, isotropic stress tensor for momentum

• Local, Mellor-Yamada, level 2.5, closure scheme

• Non-local, K-profile, surface and bottom closure scheme

• Local, Mellor-Yamada, level 2.5, closure scheme

• General Length-Scale turbulence closure (GOTM)

Subgrid-Scale Parameterizations

• Air-Sea interaction boundary layer from COARE (Fairall et al., 1996)

• Oceanic surface boundary layer (KPP; Large et al., 1994)

• Oceanic bottom boundary layer (inverted KPP; Durski et al., 2001)

Boundary Layers

• Wave / Current / Sediment bed boundary layer (Styles and Glenn, 2000; Blaas; Sherwood)

• Lagrangian Drifters (Klinck, Hadfield, Capet)

• Tidal Forcing (Hetland, Signell)

• River Runoff (Hetland, Signell, Geyer)

• Sea-Ice (Budgell, Hedstrom)

• Biology Fasham-type Model (Moisan, Di Lorenzo, Shchepetkin, Frenzel, Fennel, Wilkin)

• EcoSim Bio-Optical Model (Bissett, Wilkin)

• Sediment erosion, transport and deposition (Warner, Sherwood, Blaas)

Modules

Ongoing and Future Work

• One- and two-way nesting• Wetting and drying capabilities• Sediment model• Bottom boundary layer models• Ice model• Parallelization of adjoint model• Variational data assimilation • Parallel IO• Framework (ESMF)• Web-based dynamic documentation• Test cases• WRF coupling

One-Way Nesting

North Atlantic Basin

• 1/10 degree resolution (1002x1026x30)• Levitus Climatology• NCEP daily winds: 1994-2000• COADS monthly heat fluxes• Requirements:

• Memory: 11 Gb• Input data disk space: 16 Gb• Ouput data disk space: 280 Gb• 32 Processors Origin 3800, 4x16• CPU: 46 hours per day of simulation• Wall clock: 153 days for 7-year simulation

Bathymetry

1/10 degree

(m)

Resolution

ETOPO5

r-factor = 3.2

Free-Surface(m)

Temperature

at 100 m

US East Coast

• 30 km resolution (192x64x30)• Initialized for North Atlantic Basin Simulation• NCEP daily winds• COADS monthly heat fluxes with imposed daily

shortwave radiation cycle.• One-way nesting

• Boundary conditions from 3-day averages• Flather/Chapman OBC for 2D momentum• Clamped OBC for 3D momentum and tracers

• Rivers• Fasham-type biology model

Temperature

One-Way

at 100 m

Coupling

(Wilkin)

Temperature

Potential

at 50 m

(Celsius)

30 km

Resolution

(Wilkin)

SurfaceTemperature

Surface Chlorophyll

PublicationsEzer, T., H.G. Arango and A.F. Shchepetkin, 2002: Developments in Terrain-Following Ocean Models: Intercomparisons of Numerical

Aspects, Ocean Modelling, 4, 249-267.

Haidvogel, D.B., H.G. Arango, K. Hedstrom, A. Beckmann, P. Malanotte-Rizzoli, and A.F. Shchepetkin, 2000: Model Evaluation Experiments in the North Atlantic: Simulations in Nonlinear Terrain-Following Coordinates, Dyn. Atmos. Oceans, 32, 239-281.

MacCready, P. and W.R. Geyer, 2001: Estuarine Salt Flux through an Isoline Surface, J. Geoph. Res., 106, 11629-11639.

Malanotte-Rizzoli, P., K. Hedstrom, H.G. Arango, and D.B. Haidvogel, 2000: Water Mass Pathways Between the Subtropical and Tropical Ocean in a Climatological Simulation of the North Atlantic Ocean Circulation, Dyn. Atmos. Oceans, 32, 331-371.

Marchesiello, P., J.C. McWilliams and A.F. Shchepetkin, 2003: Equilibrium Structure and Dynamics of the California Current System, J. Phys. Oceanogr., 34, 1-37.

Marchesiello, P., J.C. McWilliams, and A.F. Shchepetkin, 2001: Open Boundary Conditions for Long-Term Integration of Regional Ocean Models, Ocean Modelling, 3, 1-20.

Moore, A.M., H.G. Arango, A.J. Miller, B.D. Cornuelle, E. Di Lorenzo, and D.J. Neilson, 2003: A Comprehensive Ocean Prediction and Analysis System Based on the Tangent Linear and Adjoint Components of a Regional Ocean Model, Ocean Modelling, Submitted.

Peven, P., C. Roy, A. Colin de Verdiere and J. Largier, 2000: Simulation and Quantification of a Coastal Jet Retention Process Using a Barotropic Model, Oceanol. Acta, 23, 615-634.

Peven, P., J.R.E. Lutjeharms, P. Marchesiello, C. Roy and S.J. Weeks, 2001: Generation of Cyclonic Eddies by the Agulhas Current in the Lee of the Agulhas Bank, Geophys. Res. Let., 27, 1055-1058.

Shchepetkin, A.F. and J.C. McWilliams, 2003: The Regional Ocean Modeling System: A Split-Explicit, Free-Surface Topography-Following Coordinates Ocean Model, J. Comp. Phys., Submitted.

Shchepetkin, A.F. and J.C. McWilliams, 2003: A Method for Computing Horizontal Pressure-Gradient Force in an Oceanic Model with a Non-Aligned Vertical Coordinate, J. Geophys. Res., 108, 1-34.

She, J. and J.M. Klinck, 2000: Flow Near Submarine Canyons Driven by Constant Winds, J. Geophys. Res., 105, 28671-28694.

Warner, J.C., H.G. Arango, C. Sherwood, B. Butman, and Richard P. Signell, 2003: Performance of four turbulence closure methods Implemented using a Generic Length Scale Method, Ocean Modelling, Revised and Resubmitted.

Modular Design

Code Design

#include "cppdefs.h“

MODULE mod_ocean USE mod_kinds

implicit none

TYPE T_OCEAN real(r8), pointer :: rubar(:,:,:) real(r8), pointer :: rvbar(:,:,:) real(r8), pointer :: rzeta(:,:,:) real(r8), pointer :: ubar(:,:,:) real(r8), pointer :: vbar(:,:,:) real(r8), pointer :: zeta(:,:,:)#ifdef SOLVE3D real(r8), pointer :: pden(:,:,:) real(r8), pointer :: rho(:,:,:) real(r8), pointer :: ru(:,:,:,:) real(r8), pointer :: rv(:,:,:,:) real(r8), pointer :: t(:,:,:,:,:) real(r8), pointer :: u(:,:,:,:) real(r8), pointer :: v(:,:,:,:) real(r8), pointer :: W(:,:,:) real(r8), pointer :: wvel(:,:,:)# ifdef SEDIMENT real(r8), pointer :: bed(:,:,:,:) real(r8), pointer :: bed_frac(:,:,:,:) real(r8), pointer :: bottom(:,:,:)# endif#endif

END TYPE T_OCEAN

TYPE (T_OCEAN), allocatable :: ALL_OCEAN(:)

END MODULE mod_ocean

CONTAINS

SUBROUTINE allocate_ocean (ng, LBi, UBi, LBj, UBj)

USE mod_param#ifdef SEDIMENT USE mod_sediment#endif

integer, intent(in) :: ng, LBi, UBi, LBj, UBj

IF (ng.eq.1) allocate ( OCEAN(Ngrids) )

allocate ( OCEAN(ng) % rubar(LBi:UBi,LBj:UBj,2) ) allocate ( OCEAN(ng) % rvbar(LBi:UBi,LBj:UBj,2) ) allocate ( OCEAN(ng) % rzeta(LBi:UBi,LBj:UBj,2) ) allocate ( OCEAN(ng) % ubar(LBi:UBi,LBj:UBj,3) ) allocate ( OCEAN(ng) % vbar(LBi:UBi,LBj:UBj,3) ) allocate ( OCEAN(ng) % zeta(LBi:UBi,LBj:UBj,3) )

#ifdef SOLVE3D allocate ( OCEAN(ng) % pden(LBi:UBi,LBj:UBj,N(ng)) ) allocate ( OCEAN(ng) % rho(LBi:UBi,LBj:UBj,N(ng)) ) allocate ( OCEAN(ng) % ru(LBi:UBi,LBj:UBj,0:N(ng),2) ) allocate ( OCEAN(ng) % rv(LBi:UBi,LBj:UBj,0:N(ng),2) ) allocate ( OCEAN(ng) % t(LBi:UBi,LBj:UBj,N(ng),3,NT(ng)) ) allocate ( OCEAN(ng) % u(LBi:UBi,LBj:UBj,N(ng),2) ) allocate ( OCEAN(ng) % v(LBi:UBi,LBj:UBj,N(ng),2) ) allocate ( OCEAN(ng) % W(LBi:UBi,LBj:UBj,0:N(ng)) )

# ifdef SEDIMENT allocate ( OCEAN(ng) % bed(LBi:UBi,LBj:UBj,Nbed,MBEDP) ) allocate ( OCEAN(ng) % bed_frac(LBi:UBi,LBj:UBj,Nbed,NST) ) allocate ( OCEAN(ng) % bottom(LBi:UBi,LBj:UBj,MBOTP) )# endif#endif

RETURN END SUBROUTINE allocate_ocean

SUBROUTINE initialize_ocean (ng, tile)

USE mod_param#ifdef SEDIMENT USE mod_sediment#endif

integer, intent(in) :: ng, tile integer :: IstrR, IendR, JstrR, JendR, IstrU, JstrV

real(r8), parameter :: IniVal = 0.0_r8

#include "tile.h"#ifdef DISTRIBUTE IstrR=LBi IendR=UBi JstrR=LBj JendR=UBj#else# include "set_bounds.h"#endif

OCEAN(ng) % rubar(IstrR:IendR,JstrR:JendR,1:2) = IniVal OCEAN(ng) % rvbar(IstrR:IendR,JstrR:JendR,1:2) = IniVal OCEAN(ng) % rzeta(IstrR:IendR,JstrR:JendR,1:2) = IniVal

OCEAN(ng) % ubar(IstrR:IendR,JstrR:JendR,1:3) = IniVal OCEAN(ng) % vbar(IstrR:IendR,JstrR:JendR,1:3) = IniVal OCEAN(ng) % zeta(IstrR:IendR,JstrR:JendR,1:3) = IniVal

...

RETURN END SUBROUTINE initialize_ocean

END MODULE mod_ocean

#include "cppdefs.h" MODULE omega_mod

implicit none

PRIVATE PUBLIC omega

CONTAINS

SUBROUTINE omega (ng, tile)

USE mod_param USE mod_grid USE mod_ocean

integer, intent(in) :: ng, tile

# include "tile.h“# ifdef PROFILE CALL wclock_on (ng, 13)# endif CALL omega_tile (ng, Istr, Iend, Jstr, Jend, & & LBi, UBi, LBj, UBj, & & GRID(ng) % Huon, & & GRID(ng) % Hvom, & & GRID(ng) % z_w, & & OCEAN(ng) % W)# ifdef PROFILE CALL wclock_off (ng, 13)# endif

RETURN END SUBROUTINE omega

SUBROUTINE omega_tile (ng, Istr, Iend, Jstr, Jend, & & LBi, UBi, LBj, UBj, & & Huon, Hvom, z_w, W)

USE mod_param USE mod_scalars USE bc_3d_mod, ONLY : bc_w3d_tile

integer, intent(in) :: ng, Iend, Istr, Jend, Jstr integer, intent(in) :: LBi, UBi, LBj, UBj

real(r8), intent(in) :: Huon(LBi:,LBj:,:) real(r8), intent(in) :: Hvom(LBi:,LBj:,:) real(r8), intent(in) :: z_w(LBi:,LBj:,0:) real(r8), intent(out) :: W(LBi:,LBj:,0:)

integer :: IstrR, IendR, JstrR, JendR, IstrU, JstrV integer :: i, j, k real(r8), dimension(PRIVATE_1D_SCRATCH_ARRAY) :: wrk

# include "set_bounds.h" DO j=Jstr,Jend DO i=Istr,Iend W(i,j,0)=0.0_r8 END DO DO k=1,N(ng) DO i=Istr,Iend W(i,j,k)=W(i,j,k-1)- & & (Huon(i+1,j,k)-Huon(i,j,k)+ & & Hvom(i,j+1,k)-Hvom(i,j,k)) END DO END DO

...

END DO

RETURN END SUBROUTINE omega_tile