29 march 2007dkrz / m&d benutzerschulung, stephanie legutke1 session 1: agenda integrating model...
TRANSCRIPT
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 1
Session 1: Agenda
Integrating Model and Data Infrastructure
Overview
History High level design
Source Code Management (SCM) Standard Compile Environment (SCE) Standard Running Environment (SRE)
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 2
IMDI: History
Develop for the European Earth System (ES) modelling community a software infrastructure to
compile run set up analyse coupled ES model experiments
and thereby establish a European ES research network
Since 2005: cooperation with PRISM Support Initiative (PSI)
see http://prism.enes.org continued development of SCE/SRE integration of data aspects (WDCC, data proc./vis.)
EU FP5 Project for Integrated Earth System Modeling (PRISM) Dec 2001-Nov 2004:
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 3
IMDI: High Level Design
Create a common infrastructure for ES modeling and data Modularity:
all components of IMDI independent Interoperability:
all components of IMDI work together Modularity in terms of experiment tasks Common look&feel for all (coupled) models and platforms Integrate best practices Modularity in terms of ES components
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 4
IMDI: Benefits
Facilitates exchange of component models
between modellersbetween institutes
communication between model users exchange of results intercomparison of simulations
Common look&feel: learn only once Choose what you like/need
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 5
SCM/SCE: Work Flow
Source & ScriptRepository
(SVN)
UpdateMakefiles
Modify source code
Retrieve source & script
code
Adapt source to SCE
CreateCompile Scripts
Compile
mo
del
.x
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 6
Use tools for SC analysis with[out] SCE list header file suffixes list directories containing header files list VPATH directoriesdetect multiple file namesmultiple MODULEs per file detectionMOULE name != file name detection
Use prerequisite files with[out] SCE Makefiles Use Makefile with[out] SCE compile script
SCM/SCE: Modular but Interoperable
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 7
SCE/SRE: Independent but Interoperable
Adapt I/O to SRE
Retrieve SRE code
Script codeRepository
(SVN)
CreateExp. Scripts
Configuresetup file
Launch Exp.
model.x
SCE
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 8
Post-processing
DB-Filling
LE-Visualisation
Preprocessing
Start Rerun Stop
www
FileArchive
FileArchive
HE-Visualisation
Processing
SRE-tasks: Modularity by Tasks
Archiving
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 9
Create Compilescripts
CompileCreate
Run Scripts
Run Experiment
user
SCE/SRE
model_tmpl.xsd model.xml tmpl.xsd
tmpl.xsd
model_tmpl.xsd
model_tmpl.xsd
model.xml model.xml
model.xml
SCE/SRE/DataManagement: Modular but Interoperable
model.xml
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 10
SCE / SRE: Modularity by ES Components
Applications:
PRISM (coupled) models (~12 components in 10 combinations on 10 platforms)
COSMOS model family of MPI-Met (cosmos-a, cosmos-as, cosmos-ao, cosmos-aob, cosmos-asob, cosmos-aso, cosmos-ob, cosmos-o)
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 11
ECHAM5ECHAM5
JSBACHJSBACH
MPIOM
Surface fluxes
Surface conditions
CO2
Grid
tra
nsfo
rmat
ion
Syn
chro
nisa
tion
OASIS3
HAMOCC
CO2
cosmos-acosmos-as cosmos-ocosmos-obcosmos-asob
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 12cosmos-aocosmos-ao
ECHAM5ECHAM5 MPIOM
Surface fluxes
Surface conditions
Grid
tra
nsfo
rmat
ion
Syn
chro
nisa
tion
OASIS3
HAMOCC
CO2
JSBACHJSBACH
CO2
cosmos-aobcosmos-asob
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 13
Session 1: Agenda
Integrating Model and Data Infrastructure
Overview Source Code Management (SCM)
SVN repositorySource/script code directory structureBrowsing facility (being revised)Scripts to analyse and transform source code
Standard Compile Environment (SCE) Standard Running Environment (SRE)
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 14
SCM: Modularity by ES components & libraries
Entities: the OASIS3/4 coupler
interpolation and synchronisation component models
source code packages describing sub systems of the ES (e.g. sea ice, ocean, clouds, ...) main models:
components which create an executable (e.g. MPIOM) submodels:
component (optionally) linked to a main model(e.g. HAMOCC in MPIOM)
librariessource code package not representing subsystems of the ES
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 15
SCE: Libraries and Models
Components: Configurable => Can not be precompiled
OASIS3/4 Not configurable => Can be precompiled Not representing a subsystem of ES
Libraries: Not configurable => Can be precompiled Potentially usable by all components
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 16
s rc
su pp o rt
s rc
p sm ile
s rc
m o d -lib
lib
s rc
m od2
in c lu de
s rc s rc2
in c lu de
m od1
m od
s rc
SRCROO T
USE, CALL, INCLUDE
SVN/SCM: Source Code Directory src
ok
ok for call of SBR if mod2 is submodel of mod1
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 17
SCM: Restructure_directories.ksh
s rc1
in c lu de
s rc2
o c ean
s rc1
tra c e rs
s rc1 in c lu de
se a ice in c lu de file .F 90file .cf ile .h
m y m o d e l h is m o d e l h e r m o d e l
m od
s rc
ro o t
tra ce r_ s rc1 d irle ss file s
se a _ ice _ sc r1 se a _ ice_ in clu de
o cea n _s rc1 in c lu de
in c lu de
o cea n _s rc2
m y m o d e l h e r m o d e l h is m o d e l
m od
s rc
ro o t
tools/Restructure_directories.ksh
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 18
Source code conventions: no two files with same base name (file.f90 / file.c) one MODULE per file only MODULE name = file name
Use Coding_conventions.ksh for renaming / splitting in last two cases enables use of Create_prerequisites...
No change of source code proper required !
SCM: Coding_conventions.ksh
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 19
SCM/SCE/SRE: Implementation
Stand-alone models Unix-ksh (g)make m4, perl
Coupled models OASIS 3/4 (http://prism.enes.org) OASIS model interface library PSMILe; NetCDF library;
message passing library MPI-1/2; I/O library mpp_io=> use help function: script.ksh --help
=> get help from [email protected]
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 20
Session 1: Agenda
Integrating Model and Data Infrastructure
Overview Source Code Management (SCM) Standard Compile Environment (SCE)
SCE toolkit: Support for Makefile writing (templates) Creating compile scripts Maintaining Makefiles (prerequisites)
Standard Running Environment (SRE)
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 21
SCE/SRE: Directories
com pile
running
util
lib
m od
include
src tools
cplm od
data
SR C R O O T
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 22
SCE/SRE: Toolkit Directory util
header_xyz.h...
include
S itespecific_node.hO Sspecific_node.h...
include_node
C ppflags_m odel.h...
incl_m od_m odel
C om pile_m ode_m odel_node.h...
include_node_m odel
headers
C reate_C O M P_cpl_m odels.ksh
C reate_C O M P_libs.frm
C reate_C O M P_m odels.frm
C reate_prerequisites
C reate_prerequisites_cpp
tools
SC E_handbook
doc
com pile
C reate_TASK S.frm
tools
include
include_node
incl_m od_m odel
incl_cplm od_cplm od
include_node_cplm od
headers
SR E_handbook
doc adjunct_files
setup_cplm od_ID
setup functions
running
util
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 23
SCE: Toolkit Directory util/compile
header_xyz.h...
include
S itespecific_node.hO Sspecific_node.h...
include_node
C ppflags_m odel.h...
incl_m od_m odel
C om pile_m ode_m odel_node.h...
include_node_m odel
headers
C reate_C O M P_cpl_m odels.ksh
C reate_C O M P_libs.frm
C reate_C O M P_m odels.frm
C reate_prerequisites
C reate_prerequisites_cpp
tools
SC E_handbook
doc
com pile
util
SR C R O O T
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 24
No compile scripts are provided …
… but tools that create them …
… for the specific platform … for the specific component model or library … configured for the coupled combination of models
common look&feel
SCE: Compile Toolkit
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 25
Create_COMP_cpl_models.ksh:create all compile scripts for a coupled model (coupler, components, libraries)
Create_COMP_models.frm:create compile scripts for a component model or the coupler
Create_COMP_libs.frm:create compile scripts for the libraries
Create_prerequisitescreate a file containing USE or include based prerequisites
Create_prerequisites_cppcreate a prerequisite file controlling recompilation if cpp flags have changed
SCE: Compile Toolkit
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 26
SCE: Creating compile scripts
Create_COMP_cpl_models.ksh cplmod \ [--id=ID [-n=node [-s=MPI1 [-c=sun [--stdout=“-”]]]]
Create_COMP_models.frm mod1 -s=MPI1 -nnode –id=ID “mod1 mod2 mod3”Create_COMP_models.frm mod2 -s=MPI1 -nnode –id=ID “mod1 mod2 mod3”Create_COMP_models.frm mod3 -s=MPI1 -nnode –id=ID “mod1 mod2 mod3”Create_COMP_models.frm oasis3 -s=MPI1 -nnodeCreate_COMP_libs.frm -s=MPI1 -nnode
COMP_mod1_ID.kshCOMP_mod2_ID.kshCOMP_mod3_ID.kshCOMP_oasis3_MPI1.kshCOMP_libs.ksh
cplmod = mod1/mod2 & mod3 & oasis3:
mod1_mod2_ID.MPI[1,2].xmod2_ID.MPI[1,2].amod3_ID.MPI[1,2].x oasis3.MPI[1,2].x
…. , psmile.MPI[1,2].a
mod1_mod2_ID.MPI[1,2].xmod2_ID.MPI[1,2].amod3_ID.MPI[1,2].x oasis3.MPI[1,2].x
…. , psmile.MPI[1,2].a
mod1_mod2_ID.MPI[1,2].xmod2_ID.MPI[1,2].amod3_ID.MPI[1,2].xoasis3_MPI[1,2].x…. , psmile.MPI[1,2].a
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 27
SCE: Creating Compile Scripts
util/compile/headers/incl*/*.h
Compile scripts for - libraries (1 for all) - component mod
- coupler
m4
...include incl_mod_$model/GUIspecif_$model.hinclude include_$node/Sitespecific_$node.hinclude incl_mod_$model/Cppflags_$model.hinclude include_$node_$model/Compile_mode_$model_$node.hinclude Make_model.h
…
Create_COMP_models.frm model –nnode
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 28
cosmos-aob = echam5 & mpiom+hamocc & oasis3
Create_COMP_cpl_models.ksh cosmos-aob COMP_echam5_ID.ksh COMP_mpiom_ID.ksh default IDCOMP_hamocc_ID.ksh COMP_oasis3_MPI1.kshCOMP_libs.ksh
cosmos-a = echam5cosmos-as = echam5+jsbach cosmos-ao = echam5 & mpiom & oasis3cosmos-aob = echam5 & mpiom+hamocc & oasis3cosmos-asob = echam5+jsbach & mpiom+hamocc & oasis3cosmos-aso = echam5+jsbach & mpiom & oasis3cosmos-ob = mpiom+hamocccosmos-o = mpiom
SCE: Creating compile scripts (examples)
[--id=ds]
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 29
Creating compile scripts: examples
clm = clm Create_COMP_cpl_models.ksh clm
COMP_clm_ID.ksh
toyclim = toyatm & toyoce & toyche & oasis3Create_COMP_cpl_models.ksh toyclim
COMP_toyatm_ID.ksh COMP_toyoce_ID.ksh COMP_toyche_ID.ksh COMP_oasis3.kshCOMP_libs.ksh
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 30
Libraries
lib*.a
Libraries
lib*.aCompile libs of main
start
Compile sub model
Compilation: Main+Sub Model
Compile libs of sub
executable
Compilemain
model
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 31
mpiom_hamocc_ID[.MPIn].x
n = 1 or 2
COMP_hamocc_ID.ksh
COMP_libs.ksh support hamocc / libhamocc_ID[.MPIn].ampiom / libmpiom_hamocc _ID[.MPIn].a
hamocc / libhamocc_ID[.MPIn].ampiom / libmpiom_hamocc _ID[.MPIn].a
Local
libsupport.alibsupport.a
Central
start
CO
MP
_mp
iom
_ID
.ksh
Compilation: Main+Sub Model with central libraries
COMP_libs.ksh support
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 32
bin
m odn_ID
m od
libn
lib
build
lib
a rch
m od1
...
m odn
m od
lib1
...
libn
lib
src
SR C R O O T
*.o*.mod
*.a
modn*_ID.a
*_ID.x
*.o*.mod
SCE: Architecture Dependent Directories
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 33
Model compilation
*.o / *.mod*.o / *.mod
mod1_ID.xlib*.alib*.a
make
src directories:
make dir/Makefile
Top_Makefile
set / export par.create (in-doc.)
COMP_mod1_ID.ksh all / lib / clean / tar
library list:
make src / Makefile
*.o / *.mod*.o / *.mod
Top_Makefilemake
set / export par.create (in-doc.)
COMP_libs.ksh “library list”
with ‚all‘ only
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 34
All compilation is based on the (g)make software:Make ‘targets’ having ‘prerequisites’ with well defined ‘rules’ while avoiding redundant actions
Targets, rules & prerequisites are defined in file Makefile:
SCE: (GNU) Make
…target: prerequisite1, prerequisite2 ….
rules …
…‘what to make’: ‘what has to be made before’
‘how to make’ …
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 35
SCE Portable Makefiles
make [–f Makefile]
All non portable code is exported from the compile scripts
1 Makefile for each source directory with compiler input(*.F90, *.f90, *.F, *.f, *.c)
All Makefiles have the same look and feel
Prerequisites Tool for generation of *.o, *.h prerequisitesTool for generation of cpp flag prerequisites
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 36
SCE Make Targets
Executable: OASIS3_MPI1/2.x, ECHAM5_ID[_MPI1/2].x, *.x
Libraries: libblas.a, libsupport.a, libpsmile.MPI1/2.a, lib*.a
Model libraries: hamocc_ID[_MPI1/2].a, *_ID.a
Binary object code: *.o
Empty build directory
tar file of model code
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 37
SCE make prerequisites
Executable: Libraries and model libraries
echam5_ID.x: libsupport.a
mpiom_ID.x: libmpiom_ID.a
Library: object binarieslibsupport.a: *.o
Binary objects: F90 MODULEs and include files
file.o: module.mod (aliased to module.o)file.o: include.hfile.o: file.F90
Note:
with multiple target lines for the same target:
the join of all prerequisite sets is valid
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 38
SCE make rules $(PROG): $(LIBRARY)
$(F90) <flags> master.o echam5_ID.a support.a –o echam5_ID.x
$(LIBRARY): *.o$(AR) libsupport.a *.o
%.o: %.F90$(F90) <flags> -c $<
file.o: file.F90 $(F90) <special flags> -c $<
Note:
with multiple target lines for the same target:
the last rule is valid
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 39
Example Makefile: Targets & Rules
clean: rm -f $(LIBRARY) ls | xargs rm –f
lib: $(LIBRARY)all: $(PROG)
$(PROG): $(DEPLIBS) $(LIBRARY)$(F90) $(LDFLAGS) -o $@ $(MAINPRG).o $(LIBS)
$(LIBRARY): $(OBJS1) $(OBJS2) $(OBJS3) $(OBJS4) $(OBJS5)$(AR) $(ARFLAGS) $(LIBRARY) *.o
.SUFFIXES:
.SUFFIXES: .o .c .f .F .f90 .F90
%.o: %.F90$(F90) $(F90FLAGS) $(INCLS) -c $<
...
%.o: %.c $(CC) $(CCFLAGS) $(INCLSC) -c $<include ../../../../src/mod/$(MODEL_DIR)$(strip $(SRC))/Make.inc.deps-include ../../../../src/mod/$(MODEL_DIR)$(strip $(SRC))/Make.inc.cpp
-include ../../../../src/mod/$(MODEL_DIR)$(strip $(SRC))/Make.inc.special-include ../../../../src/mod/$(MODEL_DIR)$(strip $(SRC))/Make.inc.$(ARCH)
Create_prerequisites[_cpp]
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 40
Example Makefile: Definition of variables
SRCS1 = $(shell \find ../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC)) –name '*.F90‘ print)
OBJS1 = $(patsubst../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC))/%.F90, %.o, $(SRCS1))
...SRCS5 = $(shell \
find ../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC)) -name '*.c' -print)OBJS5 = $(patsubst \
../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC))/%.c, %.o, $(SRCS5))VPATH = .:\
../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC)):\
../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC))/include:\
../../../../src/mod/$(MODEL_DIR)/include:\
../../lib/psmile.${CHAN}:\$(NETCDF_INCLUDE):$(MPI_INCLUDE):$(SYS_INCLUDE):\../../../lib/$(MODEL_DIR):\../../../lib/$(SUBMOD1):\../../../lib:\../../../../src/mod/include/make_dummies
LIBRARY = ../../../lib/$(MODEL_DIR)/lib$(MODLIB).aPROG = ../../../bin/$(EXEC).xINCLS = -I../../../../src/mod/$(MODEL_DIR)/include \
-${I4mods}../../lib/clim.${CHAN}INCLSC = -I../../../../src/mod/$(MODEL_DIR)/include
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 41
Make.inc.deps
mod_unitncdf.o: mod_kinds_oasis.oalloc_src.o: mod_anais.o mod_analysis.o mod_coast.o mod_experiment.o \
mod_extrapol.o mod_kinds_oasis.o mod_memory.o mod_nproc.o \mod_parallel.o mod_parameter.o mod_pipe.o mod_rainbow.o mod_sipc.o \mod_string.o mod_timestep.o mod_unitncdf.o
dealloc_src.o: mod_anais.o mod_analysis.o mod_coast.o mod_experiment.o \mod_extrapol.o mod_kinds_oasis.o mod_memory.o mod_nproc.o \mod_parallel.o mod_parameter.o mod_pipe.o mod_rainbow.o mod_sipc.o \mod_string.o mod_timestep.o mod_unitncdf.o
extrap.o: mod_extrapol.o mod_kinds_oasis.o mod_parameter.o mod_printing.o \mod_unit.o
getfld.o: mod_analysis.o mod_clim.o mod_experiment.o mod_hardware.o \mod_kinds_oasis.o mod_label.o mod_memory.o mod_parameter.o \mod_printing.o mod_sipc.o mod_string.o mod_timestep.o mod_unit.o \mod_unitncdf.o netcdf.inc
Create_prerequisites [model,library] directory
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 42
... mod_unitncdf.o: use_netCDF.cpp
alloc_src.o: use_comm_MPI1.cpp use_comm_MPI2.cpp use_comm_GSIP.cpp
dealloc_src.o: use_realtype_single.cpp ...
Make.inc.cpp
Create_prerequisites_cpp [model,library]
29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 43
SCE Handbook: SVN:
http://svn-mad.zmaw.de/svn/mad/ \Model/IMDI/tags/’tag’/util/compile/doc
Web:http://mad.zmaw.de/imdi (bottom of page):
- pdf-version - html-version
Handbooks