regcm-roms tutorial: coupling...

25
RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu ICTP (International Center for Theoretical Physics) Earth System Physics Section

Upload: dangnhu

Post on 17-Aug-2018

281 views

Category:

Documents


12 download

TRANSCRIPT

Page 1: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu ICTP (International Center for Theoretical Physics) Earth System Physics Section

Page 2: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

Outline •  Information about coupling and ESMF

•  Installation and setup

• Coupled model (RegCM+ROMS) •  Design, Limitations and Pros/Cons •  Installation •  Configuration •  Running and Restarting •  Debugging

•  Future Plans

- Outline

Page 3: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Information about ESMF (1/2)

• Why ESMF?

Different ways for online coupling •  Creating subroutinized sub-components (like sub-grid

parameterizations or current version of the CLM coupling) •  Most of the existing code use this approach. It is not generic

solution and not flexible. Think about changing version of CLM in current version of RegCM model.

•  Using existing communication protocols such as (MPI, InterComm) •  Implementation is not easy. It is not generic and efficient solution.

•  Using coupling libraries or frameworks (i.e. ESMF, MCT) •  Flexible, efficient, scalable and generic solution. It needs time to

convert existing code into components. •  Using a coupler (OASIS, MPCCI etc.)

•  Flexible, efficient, scalable and generic solution. It also needs time to implement into existing codes.

Increasing needs to such tools like ESMF, MCT and OASIS !!! The current models evolve to a multi-component Earth System Models (ESM).

Page 4: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Information about ESMF (2/2)

• Basics

ESMF •  The complexity of the regular tasks to create a coupled modeling

system (synchronization of the model components, exchanging coupling fields among modeling components, interpolation between deferent grids etc.) can be simplified.

•  ESMF can perform different interpolation types (bilinear, conservative etc.) over exchange fields (i.e. sea surface temperature, heat and momentum fluxes) in a parallel way without using external grid weight file generated by SCRIP.

•  Supports both C/C++ or Fortran programming language. •  Highly optimized and portable (the library is tested regularly across

different computing platforms, compilers (and also versions), mpi implementations etc.) library

•  Scalable beyond few thousand processor (if you can feed the CPUs) •  Supports different technologies such as parallel IO, XML, web services

(like OpenMI).

Page 5: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Installation of ESMF (1/3)

• Steps to install ESMF library …

Get ESMF library source code •  Latest public release 5.2.0rp2

http://www.earthsystemmodeling.org/esmf_releases/public/ESMF_5_2_0rp2/reg/ESMF_Framework_Reg.html

•  Fill simple form and download the source code •  Installation information for different platforms can be found in

http://www.earthsystemmodeling.org/download/platforms/ Install third-party libraries •  NetCDF •  Parallel NetCDF (required for coupled model debugging)

http://trac.mcs.anl.gov/projects/parallel-netcdf/wiki/Download •  Xerces XML Parser (required for exporting component, field level

metadata in XML format) http://xerces.apache.org/xerces-c/ Install ESMF and Setup Environment Variables •  Third party libraries must be installed first !!! •  Same compiler and MPI implementation must be used !!!

Page 6: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Installation of ESMF (2/3)

• Example installation environment on ARGO

third-party libraries

ESMF library related definitions

Install both optimized and debug versions

Used for RegCM+ROMS

installation

Page 7: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Installation of ESMF (3/3)

• Commands

To install optimized version •  mkdir ~/progs; cd progs •  tar -zxvf esmf_5_2_0rp2_src.tar.gz; mv esmf esmf_5_2_0rp2 •  cd esmf_5_2_0rp2 •  make info (it shows the configuration. important for solving installation

problems) •  make >& make_O.log •  make install To install debug version •  make clean •  export ESMF_BOPT=g (for tcsh setenv ESMF_BOPT g) •  make >&make_g.log •  make install Notes To test the installation, the ESMF test suite can be installed

Page 8: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Design (1/5)

• General structure (1)

Ocean / Lake

Psurf, T2m, Q2m, Rain Swrad, Lwrad, U10m, V10m SST Hice

•  ATM àSurface atmospheric conditions, heat fluxes and rain. The atmospheric variables is used to calculate fluxes in the ROMS.

•  OCN à SST and Ice Thickness (if the ice model exist). The SST is used to calculate sensible heat flux, evaporation, ground temperature etc. Ice Thickness is used to change land type and updates the evaporation flux over ice

•  The interaction or coupling time step (in second) can be defined using RegCM namelist file

Page 9: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Design (2/5)

• General structure (2)

Init

Run

0 1 Number of PETs: N

Number of PETs: N

Total Numberof PETs: 2N

N-1

N N+1 2N-1

Init

Run

Finalize

Finalize

Export States

Export States

12

11

15

16 17

18112

111

Import States

Import States

Backward

Forward

Initialize

Run

Finalize

Initialize

Run

Finalize13

14

RegCMAtmosphere Model

ROMSOcean Model

110

19

COUPLER: Online Interpolation,Vector Rotation,

Unit Conversion

MPI Comm. World

COUPLER

ATM to OCN

OCN to ATM

Page 10: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Design (3/5)

• General structure (2)

MTX ZENG LAKE UPDCPLFWD

T = 0

T = Tcpl

BATS

Time Axis

send oceanvariables

(SST, ICE etc.) to RegCM

OCNCPLBCK

Time Axis

ATM

updateforcing

variablesin ROMS

updateforcing

variablesin ROMS

update SST and ICE

in RegCM

Page 11: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Design (4/5)

• Design details (1)

Supports •  It can be used together with Zeng Ocean and one-dimensional Lake

Model. •  ESMF online re-gridding capability and conservative type interpolation

is used for flux variables (bilinear type interpolation for others) •  Works with ROMS 3.5 tagged version and also ROMS-Ice branch. •  Different number of processor can be assigned to gridded components

(coupler component uses all processors) Limitations •  It only interacts with BATS (no support CLM or BAND) •  The ocean model still needs forcing files (RegCM model updates them

when coupled mode is activated). •  Current version only supports Gregorian calendar •  Ocean model must be configured to write daily restart file and this

adds extra overhead into coupled model run. •  There is a bug in ROMS (PERFECT_RESTART+LcycleRST == F)

Page 12: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Design (5/5)

• Design details (2) Pros •  The version of ocean component (ROMS) can be changed easily by

porting patch (it may also requires minor change in the RegCM component code). This is tested before is tested before, 3.4 à 3.5 à 3.5_ice.

•  The coupled model provides better representation of oceans, lakes and inland waters (If the ROMS model is tuned well).

•  New components (like CLM) can be added by modifying coupler interface.

Cons •  It needs experience also in ocean modeling, which the configuration of

realistic application is not straightforward (create ocean model grid, initial, boundary and forcing files).

•  Adds additional computational and messaging overhead into the model •  User needs to tune the ocean model like atmospheric model. This

must be done in standalone mode first !!!

Page 13: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Installation (1/3)

• Applying patch to ROMS and install Patch includes following modifications •  Updates forcing variables with RegCM provided ones •  Enables to use two namelist (*.in for RegCM and ROMS) file in the job

submission step •  Fixes caldate subroutine in (ROMS/Utils/get_date.F90). •  The changes are activated by new pre-processor flag (#define

REGCM_COUPLING) which must be set in *.h header file. Without this option ./build.sh compiles original version of ROMS.

Applying patch Creating and installing new ROMS case (before compiling RegCM !!!)

Page 14: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Installation (2/3)

• Configure and install RegCM

options for coupling

./configure --help

Example command to configure RegCM in coupled model mode

Page 15: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Installation (3/3)

• Additional information about installation

•  User may need to install newer version of autoconf (>2.68) and automake (>1.11.1) tools to run bootstrap.sh script.

•  Script --with-esmf option looks for ESMF_LIB environment variable. •  The ROMS Build path must be given as an absolute path into

configure script using --with-roms option !!! •  After ./configure step, the RegCM code can be compiled with

conventional way using make install command. •  The ROMS ice branch is automatically identified by configure script

and required code is added into RegCM (it just check the seaice.o file in the ROMS Build directory).

•  The patched version of ROMS (with REGCM_COUPLING option) and also ESMF library must be installed before RegCM installation !!!

•  The ROMS does not need ESMF library. So, do not install it with existing ESMF_COUPLING and/or ESMF_LIB header definitions. It is also gives error with these options because it is not fully implemented yet.

Page 16: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Configuration (1/2)

• New parameters (*.in)

•  physicsparam •  iocncpl

•  0 – no coupling (default) •  1 – activates ROMS coupling

•  cplparam •  cpldt – controls the coupling time step (in second, 21600) •  cpldbglevel – specify the debug level for coupled model. (1)

•  0 – no debugging •  1 – only informative print statement (array and pointer

dimensions etc.). •  2 – previous option + write grid information in VTK (The

Visualization Toolkit) format which can be visualized by using such tools like Visit.

•  3 – previous option + write exchange fields in NetCDF format •  4 – previous option + write exchange fields in ASCII format

Page 17: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Configuration (2/2)

• New parameters (regcm.rc)

•  The new namelist file (regcm.rc) is created to pass parameters to the coupled model (sample file is in the Testing/CPL_FILES)

•  In the current implementation, the new parameter file is used just to set number of processor for each gridded (M, N) component.

•  If the file does not exist, then the total number of processor (M+N) divided to two and assigned to each gridded component (M, N). In this case N+M can be divided evenly or not.

•  We plan to move all the parameters (coupling time step, debug level etc.) to the new parameter file (regcm.rc).

•  The example for the PETs parameter: •  PETs: 32 28 •  In this case atmospheric model (RegCM) will use 32 (M)

processor and ocean model (ROMS) will use 28 (N) and the coupler component uses all the available CPUs (32+28 = 60) to perform interpolation between grids.

•  The M must be evenly divided to the jx and placed under case directory like regcm.in

Page 18: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Running (1/4)

•  Job submission under PBS regcm.job

path of ESMF shared libraries

definition of namelist files …

•  The same ESMF path must be defined with ./configuration step. •  The ESMF must be compiled with same version of the compiler and

ESMF implementation •  The total number of processor must be consistent. In the example script

•  regcm.rc (M+N = 32+28 = 60 = 12*5 in regcm.job) •  cas.in (NtileI*NtileJ = 4*7 = N = 28)

Page 19: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Running (2/4)

• Restarting (1)

Requirements •  User must configure ROMS to write restart data each day without

cycling. To do that set NRST to 86400/DT and LcycleRST to F. •  The date of the records in the restart file might be checked by CDO.

cdo sinfo ocean_rst.nc. Steps in ROMS side •  In namelist file, set NRREC to -1 (indicates that the run is restart) •  Split desired date from ocean_rst.nc file (this date must be same with

the date of latest RegCM restart file - SAV). •  Point created file as a restart file in namelist file. To do that set

ININAME as output/ocean_rst_splited.nc (or point the correct directory).

Steps in RegCM side •  Set ifrest as .true. •  Set mdate1 as a date of the latest SAV file •  Run RegCM model

Page 20: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Running (3/4)

• Restarting (2) Notes •  Restating of the coupled model can be little bit confusing •  Both model must have restart information in same date •  RegCM normally creates restart file at the beginning of the each month

but you can not define monthly time interval for ROMS restart files. As a result we choose to write ROMS restart each day. By this way, ROMS will have restart information at the beginning of the month.

•  This can be potential performance bottleneck and waste of disk usage •  In theory, it is possible to set date for RegCM model to write

intermediate weekly restart file (TMPSAV) by using savfrq parameter. In this case, ROMS model can be configured to write restart file with the same interval (i.e. weekly). It is not tested !!! There could be time lag between restart files created by RegCM and ROMS.

•  To simplify the current restart mechanism, we created a restart script (Tools/CPL_FILES/restart.sh) that automatize the steps in the previous slide.

Page 21: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Running (4/4)

• Restarting (3) restart.sh

definition of RegCM and ROMS namelist files …

Required modules (CDO and NCO).

•  User may need to modify this script. •  It uses Module (user environment manager) to setup paths for NCO

and CDO tools. It is written in bash shell and uses awk and sed. •  User need to check the RegCM and ROMS namelist files after using

this script. The backup is created for namelist and restart file.

Page 22: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Debugging (1/2)

• VTK (The Visualization Toolkit) and Visit

•  Used to see the grid representation of each gridded component

•  Created for each grid stencil •  Files with *.vtk extension for each

processor can be visualize in Visit •  https://wci.llnl.gov/codes/visit/

Page 23: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Debugging (2/2)

• Notes

If you have problem with the coupled model, check following before doing anything … •  Check grid definitions using Visit (see previous page …) •  Check import and export fields (can be written in NetCDF and ASCII

format). The ATM/OCN export files must match with OCN/ATM import files. The debug level of the coupled model must be >2.

•  Use debug version of ESMF library (compiled with ESMF_BOPT=g). The ESMF can give some information in the PET* files.

•  Use debug version of the model components. The ROMS can be compiled in debug mode by modifying ./build.sh script (set USE_DEBUG as on). The RegCM can be compiled in debug mode by running configure script with --enable-debug option. In the stdout file you can see the actual line number of the source file that triggers the error.

•  Check input files and the configuration of the model components (RegCM and ROMS).

Page 24: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- Future Plans

• More work … •  Add support for identical mesh/grid.

•  No need to interpolation between grid. •  The possible candidate to test this configuration is the BAND

version of the RegCM. •  The tool to create ROMS grid from RegCM domain file is almost

ready. It is written in NCL. •  Move all coupled model configuration options to regcm.rc •  Add support for boundary smoothing. It is important for the ocean or

sea centric applications (i.e. Mediterranean Sea, Indian Ocean). •  Add support for selective interpolation types for each variable. User

can change from conservative type re-gridding to bilinear or others. •  Improve the air-sea interaction interface. In the current implementation,

ocean send SST and ICE thickness to atmospheric model. The aerosol related variables, surface roughness etc. can be added into list.

•  Benchmarking is needed to see the actual performance of the model. It also helps to find possible performance bottleneck sources of the current implementation.

Page 25: RegCM-ROMS Tutorial: Coupling RegCM-ROMSgforge.ictp.it/gf/download/user/292/764/regcm_tutorial_02_coupled.pdf · RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu

- News

• RegCM is in the component list of the ESMF J

http://www.earthsystemmodeling.org/components/