Collabora've  Development  

Ricardo  Todling  NASA  GMAO  

 2013  Joint  DTC-­‐EMC-­‐JCSDA  GSI  Workshop  

 

1  

In  the  Beginning  …  

•  NCEP:  SSI  (late  1980’s)  –  Spectral  formula'on  of  background  error  cov  – Direct  assimila'on  of  radiances  

•  DAO:  PSAS  (early  1990’s)  –  Physical-­‐space  formula'on  of  background  error  cov  – Assimila'on  of  temperature  retrievals  (actually,  geopoten'al  height  retrievals)  

•  NCEP:  GSI  (early  2000’s)  •  GMAO:  Abandoned  PSAS  (c.  2003),  joined  GSI  dev  

2  

Non-­‐intrusive  collabora'on  From  the  ini'al  stages  in  the  collabora'on  we  were  faced  with  having  to  accommodate  differences  without  being  disrup've  to  each  other’s  development.    •  NCEP’s      global  model:  spectral  •  GMAO’s  global  model:  grid-­‐point  +  ESMF-­‐based  

GMAO  adopted  the  concept  of  non-­‐intrusiveness:    •  We  did  not  want  NCEP  to  have  to  carry  components  it  did  not  use              (neither  did  NCEP  want  to  carry  stuff  it  didn’t  need!)  

•  GMAO  interfaced  GSI  with  its  GEOS-­‐AGCM  without  ever  having  NCEP  to  compile  a  single  code  related  to  establishing  that  interface.  

•  For  a  while  GSI  carried  a  file  single  that  provided  the  GMAO  hook.  •  We  have  since  evolved  to  having  absolutely  NO  visible  GMAO-­‐specific  

code  at  NCEP.  •  This  is  how  we  advocate  anyone  to  hook-­‐up  to  GSI.  

3  

A  Unified  Analysis  System  •  In  the  mid-­‐90’s  NCEP  supported  (as  it  s'll  does)  a  mul'tude  

of  applica'ons  from  global  to  various  regional  components.  

•  Global  and  Regional  analyses  were  detached.  

•  The  GSI  grid-­‐space  formula'on  of  B  allowed  for  unifica'on  of  the  various  regional  analyses.  

•  Since  all  development  for  hooking  up  various  regional  applica'ons  took  place  at  NCEP,  all  happened  inside  the  GSI  code.  

•  Our  view  today  prefers  that  these  hooks  be  handled  as  “couplers”,  and  thus  not  be  in  GSI.  

4  

Non-­‐intrusiveness  vs  contribu'on  •  Non-­‐intrusiveness  should  not  be  taken  as  need  to  hide  

development  of  general  features  and  contribu'ons.    

•  If  a  feature  is  general,  we  want  to  see  it  made  available  to  all  users  of  GSI,  regardless  of  their  applica'on.  

•  Non-­‐intrusiveness  simply  means  that  project-­‐specific  knobs  should  be  avoided  as  much  as  possible.  

Example:  4DVAR  Ø  code  should  have  general  interface  to  allow  users  to  hook-­‐up  their  

own  TL  and  AD  models.  Example:  Hybrid  Ø  code  should  have  general  interface  to  allow  users  to  provide  

members  at  will  (more  on  this  later).  

5  

Diversity  and  Flexibility  •  Suppor'ng  mul'ple  model  interfaces  

–  GSI  provides  hooks  to  a  mul'tude  of  models,  but  …  –  It  should  provide  a  single  interface  to  the  background  and  placing  support  to  mul'ple  models  in  external  libraries  

•  Suppor'ng  mul'ple  observing  systems  –  GSI  is  rather  flexible  in  this  respect,  but  needs  …  –  Acceptance  of  modern  observa'on  format  (NetCDF)  –  Avoidance  of  wired-­‐in  user-­‐specific  QC  and  obs  choices  

•  Suppor'ng  mul'ple  assimila'on  strategies  –  The  flexibility  for  this  is  in  place,  but  needs  …  –  Agen'on  when  it  comes  to  choices  of  control  vector    

6  

Rela'vely  Flat  GSI  (to  most)  

7  

bufr   crtm   I/O   sp   w3  

math  

system  

GSI  

gsi.x  

GFS  NMMB  RTMA  WRF  

•  Presently,  a  single  executable  handles  mul'ple  background  choices.  •  May  seem  flexible  but  it’s  rather  cumbersome  for  maintenance:  GSI  shouldn’t  need              to  change  because  changes  had  been  made  to  how  the  background  is  read  in.  

 GSI  Split  into  Mul'ple  Libraries  

8  

bufr   crtm   I/O   sp   w3  

math  

system  

GFS_gsi.x  NMMB_gsi.x  RTMA_gsi.x  WRF_gsi.x  GEOS_gsi.x  

•  To  a  large  extent  the  split  displayed  above  already  exists  at  GMAO.  •  This  split  gives  greater  flexibility  and  reduces  burden  to  maintain  GSI  CORE  components.  

GSI_U'l  

GSI_Obsvr  

GSI_Solver   User_Suff  

GSI_Coupler  

GSI_Appl  

ESMF/WRF  

GSI    Core  

 GSI  Split:  Added  Flexibility  

9  

bufr   crtm   I/O   sp   w3  

math  

system  

           GEOSgcm.x  This  selng  already  Allows  GMAO  to  Hook-­‐up  its  AGCM  with  the  GSI  observer.  

•  It  already  possible  to  build  a  real  observer  capable  of  calcula'ng  observa'on  residuals            by  having  the  GSI-­‐observer  called  from  within  the  atmospheric  model.  •  If  desired,  it  is  possible  to  have  a  single  executable  run  perform  both  the  model  integra'on            and  GSI  analysis.    

GSI_U'l  

GSI_Obsvr  

AGCM  

GSI_Coupler  

AGCM_Appl  

ESMF/WRF  

Enhancing  Flexibility  •  Improved  model/background  interface  

–  The  library  split  will  amount  to  having  a  single  entry  point  handling  the  background  in  the  Core  GSI;  

–  This  will  allow  for  flexible  choice  of  background  fields;  for  example,  permilng  GSI  to  be  used  to  assimilate  chemical  cons'tuents  without  need  of  meteorological  fields;  

–  Ensemble  hybrid  and  4dvar  fields  should  become  equally  flexible;    –  In  the  future,  the  GRID  the  analysis  operates  on  should  be  made  

flexible  and  be  defined  at  entry  point  (beyond  simply  regular).  •  Improved  observa'on  handling  

–  There  is  need  for  generaliza'on  of  the  observa'ons  operators,  to  handle  user-­‐GRID;  interpola'on  to  intermediate  grid  can  be  avoided    

–  Residual  output  should  be  made  more  flexible  (NetCDF-­‐based)  •  Improved  code  robustness  

–  Improved  consistency  checks  and  op'ons  handling  –  Improved  memory  management    

10  Ques4on  some  have  asked:  Should  GSI  apply  the  ESMF-­‐paradigm?  

NOAA/EMC  Partners  &  The  GSI  Commigee  

Members  of  CommiBee  •  NASA  GMAO    •  NOAA  ERSL/GSD  

•  NCAR/DTC  •  NOAA  OAR  •  NOAA  ESSL/MMM  •  AFWA  •  JCSDA  

CommiBee  DuDes  •  Manage  dev/implementa'on  •  Forum  for  discussion  •  Meets  twice  a  year  •  Ac've  year-­‐round  to  eval  mods  •  Review  each  modifica'on  •  Enforce  code  standards  •  Evaluate  large  proposed  mods  •  Users  not  represented  are  

encouraged  to  contributed  via  DTC  

11  

The  GSI  Commigee  •  A  Commigee  made  up  of  the  current  principal  contributors  to  GSI  has  

been  created  to  manage  development  and  implementa'on.  •  The  Commigee’s  primary  responsibility  is  to  serve  as  a  forum  for  

discussion  of  present  and  future  development,  coordina'ng  mul'ple  interests  and  avoiding  redundant  efforts.  

•  The  Commigee  meets  at  least  twice  a  year.  •  The  Commigee  is  ac've  year-­‐round,  evalua'ng  &  approving  changes.  •  Each  change  must  be  presented  (submiged)  to  the  Commigee  and  be  

approved  before  being  officially  accepted.  •  Desire  to  make  large  changes  to  the  code  must  be  submiged  to  the  

Commigee  in  the  form  of  a  proposal  for  change.  A  prototype  of  the  idea  might  help  the  Commigee  come  up  with  a  decision.  

•  Those  users  not  represented  in  the  Commigee  are  s'll  encouraged  to  submit  proposal  and  changes  to  the  Commigee,  using  the  DTC  connec'on.  

•  The  Commigee  tries  to  enforce  the  agreed-­‐upon  code  standards,  so  remember  those  before  submilng  your  changes.  

12  

8

• Community GSI repository Since 2009, EMC have put all its operational systems under subversion code manage system. The operational GSI repository in EMC has been actively used by NCEP and GMAO to track and share the GSI code development since then. At the same time, the DTC built a community GSI repository to support the community GSI developers using the latest GSI code in the research. In figure 4, we illustrated the purpose and relation of these two GSI repositories: the EMC GSI repository is mainly used by GSI developers in EMC and GMAO for the operational development, while the community GSI repository is mainly used by community GSI developers such as NOAA/GSD and NCAR/MMM for the research or operation development in a computer environment other than EMC computers. The community GSI repository also provides the code manage system to hold the official released GSI packages.

Fig. 4. The function of the EMC GSI repository and community GSI repository The figure 5 illustrates the structure and contents of the two GSI repositories. Here, we can see another major difference between the two repositories: the community repository includes several components that are not in the EMC repository. As we introduced before in section 2 on the release of community GSI package, these components such as external libraries, compile system, and utilities are part of the released community GSI package. DTC created and maintain these components to make the community GSI system can be easily installed on multiple platforms other than the EMC computer system. Although the two GSI repositories are maintained by the EMC and DTC separately to serve the operational and research community, the core part of the repository, the GSI source code, is always identical in the two repositories. As indicated in the figure 3 and figure 5, the DTC keeps this identical GSI source code by syncing the GSI source code from the EMC repository (./src directory) to the community repository (./src/main directory) weekly. When bug fixes or new codes from the DTC and research community are ready to commit to the GSI trunk, the DTC always work with the EMC GSI group to

From  Huang  et  al.  (2011)  

Contribu'ng  via  DTC  

13  

Though  some  of  us  in  the  community  have  access  to  NCEP’s  repository,  the  reality  is  that  many  don’t  and  should  have  no  need  to.    The  DTC  maintains  an  up-­‐to-­‐date  copy  of  GSI  for  public  consump'on.  The  DTC  has  the  capability  to  test  user-­‐specific  changes  to  verify  code  integrity.  The  DTC  has  the  mechanisms  in  place  to  allow,  not  only  O2R,  but  also  R2O  transi'on.  

A  biased  example:  main  GMAO  contribu'ons  

14  

•  GSI  Infrastructure:  –  GSI_Bundle:  generaliza'on  of  control  vector  –  Stub-­‐framework  to  allow  for  user-­‐plug-­‐ins  

•  New  Observa'on  Types:  –  MOPITT  CO  –  SSMI/S  (joint  w/  NCEP)  –  MLS  temperature  retrievals  –  MLS  moisture  retrievals  –  MLS  radiances  –  Doppler  Wind  Lidar  (now  on  hold)  

•  Methodologies:  –  4DVAR  framework  (in  collabora'on  with  Yannick  Tremolet  from  ECMWF)  –  GSI  Adjoint  (in  collabora'on  with  Yannick  Tremolet  from  ECMWF)  –  Various  minimiza'on  op'ons,  including  Lanczos  and  BiCG  procedures  –  (GOCART)  Aerosol  influence  on  (IR)  radiance  assimila'on  –  Capability  to  apply  GSI  adjoint  for  B-­‐precond  (via  bicg  single  loop  only)  –  Capability  to  use  sqrt(B)-­‐precond  within  hybrid  framework  (joint  w/  NCEP)  –  Assimila'on  of  cloudy  IR  radiances  –  Aircrat  bias  correc'on  (joint  w/  NCEP)  

In  works  

A  Truly  Global  Analysis  System  

From  hgp://www.dtcenter.org/com-­‐GSI/users/overview/index.php  

Though  GSI  has  literally  been  explored  by  people    everywhere  in  the  world,  it  has  been  largely  a  one-­‐  way  road.    Those  par'cipa'ng  in  this  Tutorial  &  Workshop  should  Keep  in  mind  the  great  opportunity  to  contribute,  via    the  DTC  to  a    truly  global  and  open  analysis  system.    

15  

Closing  Remarks  •  GSI  is  truly  the  only  publically  available,  free-­‐of-­‐charge,  mature  

analysis  system  in  the  world.  

•  Collabora've  work  in  GSI  has  proven  rather  successful.  

•  There  are  challenges:  –  code  serves  research  as  well  as  opera'ons;  –  keeping  up  with  fast  changing  code  is  hard;  –  the  lager  is  required  for  new  changes  to  be  accepted;  –  code  might  not  meet  your  standards  and  wishes:  which  should  be  a  

good  reason  to  become  involved!    

•  It  is  exci'ng  to  be  able  to  contribute  to  the  U.S.  Na'onal  Data  Assimila'on  effort.  

•  Thanks  are  due  to  NOAA/NCEP/EMC  for  this.  16