the jcsda spearheading an effort to bridge research and
TRANSCRIPT
The JCSDA Spearheading an Effort to Bridge Research and Operations
Tom Auligné, Director, Joint Center for Satellite Data Assimilation (JCSDA) Acknowledgment: Yannick Trémolet, JCSDA
NASA GSFC
NOAA NWS
NOAA OAR
U.S. Navy
Mission: to accelerate and improve the quantitative use of research and operational satellite data in weather, ocean, climate and environmental analysis and prediction models.
Vision: An interagency partnership working to become a world leader in applying satellite data and research to operational goals in environmental analysis and prediction
Science priorities: Radiative Transfer Modeling (CRTM), new instruments, clouds and precipitation, land surface, ocean, atmospheric composition.
NOAA NESDIS
U.S. Air Force
Joint Center for Satellite Data Assimilation
• Observations provide information about “reality” but are disparate and irregular in space and time • Models provide regular, physically consistent information about the system, but are prone to systematic errors
Source: ECMWF
Introduction
Data assimilation systems usually combine together information from a set of observations, a short term forecast, and possibly other information to estimate the most probable state of a physical system.
Introduction to Data Assimilation
Contributions to NWP forecast: Initial Conditions = Model (Magnusson and Källen, 2013)
Socio-economic benefit of NWP forecast: estim. $100B-$1T per year (Riishojgaard, 2014)
Initial Conditions: Satellites dominate global NWP impact of observations
Source: Jung (2012)
Introduction to Data Assimilation
Forecast lead time
Anom
aly
Corr
elat
ion
(500
hPa
geo
pote
ntia
l)
NOAA-14 recorded warm-target temperature changes, due to orbital drift (Grody et al. 2004)
Source: Dick Dee (ECMWF)
Jan 1989: Transition between two production streams
Introduction to Data Assimilation
6
Big Data: Volume, Variety, Velocity
Observations • Big Data paradigm: most of the total error reduction comes from a large number of
observations with small or moderate individual impacts Models • Better value for society: forecast model for more components of Earth system • Each model is becoming increasingly complex as science progresses • Models are getting coupled to better account for interactions
Data Assimilation Algorithms • Today’s best data assimilation algorithms are hybrid
• Ensemble DA for computing background error covariances, • Variational DA to provide the high resolution (or best) analysis.
• DA systems have become very complex: comparing all options almost impossible
Earth System Modeling
Scalability is the ability of a system, network, or process to handle a growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth (Wikipedia). Challenges for next-generation Data Assimilation to scale for: • More observations and higher model resolution
(Code performance on current/future HPC architecture )
• More domains, grids, and applications • More concurrent developments and distributed partners
“Software is like entropy. It is difficult to grasp, weighs nothing, and obeys the second law of thermodynamics; i.e. it always increases“. Norman Ralph Augustine
8
Scalability Challenges
We want a very flexible, reliable, efficient, generic, readable and modular code. This is not specific to NWP: the software industry has moved to generic and object-oriented programming 20 years ago. The key idea is separation of concerns - All aspects exist but scientists focus on one aspect at a time. - Different concepts should be treated in different parts of the code. - Nobody can know it all!
Code efficiency is an important aspect of the project - Experience shows that refactoring generally improves code performance - New structure will make scalability investigations easier
Modern Software Engineering
STRATEGY 1. Collective path toward Nation Unified Next-Generation Data Assimilation 2. Modular, Object-Oriented code for flexibility, robustness and optimization 3. Mutualize model-agnostic components across
• Applications (atmosphere, ocean, land, aerosols, etc.) • Models & Grids (regional/global, FV3, MPAS) • Observations (past, current and future)
OBJECTIVES 1. Facilitate innovation to address next scientific grand challenges 2. Increase R2O transition rate 3. Increase science productivity and code performance
Joint Effort for Data assimilation Integration (JEDI)
Abstract Layer Generic Applications
Abstract building blocks
Systems
Forecast DA …
State Observations Covariance Model
NGGPS Lorenz MOM6
Uses
…
Implements
Abstract interfaces are the most important aspect of the design
JEDI Abstract Design
Data Assimilation Components
Interface for Observation Data Access (IODA)
Obs. Pre-processor Read, select, basic QC
Solver Variational EnKF, Hybrid
Background & Obs Error
Observations
Model
• Verification • Model post-proc. • Cal/Val, Monitoring • Retrievals • Simulated Obs.
obs + model equivalent
Unified
Forward Operator
(UFO)
• Model Initial Conditions • Observation Impact • Situational awareness • Reanalysis
for Atmosphere, Ocean, Waves, Sea-ice, Land, Aerosols, Chemistry, Hydrology, Ionosphere
Analysis increments
observations
Modern project management methods recognize this Start with an overall plan and a detailed plan for initial stage At the end of each stage:
- Revise the overall plan - Define the detailed plan for the next stage
We are at Stage 0: General plan, detailed plan for Stage 1 Stage 1: Unified Forward Operator (UFO), Interface for Observations Data Access (IODA) Stage 2: Covariance matrices, linearized UFO, 3D solvers, bias correction Stage 3: Optimized components, 4D solvers Stage 4: Multi-scale DA, coupled DA
JEDI Project Roadmap
Interfaces, interfaces, interfaces!!! IODA: Isolate science code from data storage. Two levels to the API: - Standardized format for large shared historic database - In memory handling of observations UFO: Interpolation from variety of model grids, observation operators (e.g. interface to CRTM), refactoring of operational Quality Control.
JEDI: Components in Stage 1
Governance is about management keeping in control and deciding what features should be in the system Code reviews are about quality of the code Two different levels of control with different people and paces - Good code might stay outside of central repository (stability of interfaces is important) - A desired feature that does not satisfy quality requirements cannot be accepted as is Testing, documentation, training, support - Automated testing framework - Pre-requirement for code reviewing
Governance and Code Reviews
Bridging R & O
DART GSI
D
D
D
E
E E
C
A
A
B
B
C
A
B
A
B C A
B
C
Navy
A
Research
Community
Oper
C
…
C
C
C
B
Operations
Scientific efforts in academia
Scientific efforts in OAR
Scientific efforts in research community
Scientific efforts in satellite DA in Navy
JCSDA’s own DA Activities
Ope
ratio
nal
Res
earc
h
1
Com
mun
ity
2
3
(TRL 1-4)
(TRL 7-9)
(TRL 4-7)
Cod
e S
tand
ards
& C
onst
rain
ts
Central repo.
JEDI core
ESRL
.edu
GMAO
EMC
NRL
NCAR Lab. controlled
Forks
Push + Pull request
Community controlled
Univ
Community Ecosystem for R & O
NOAA/EMC
JEDI core
ESRL
.edu GMAO
NRL
NCAR
EMC controlled
Central Prod.
Community Ecosystem for R & O
Oper.
Good software engineering does not solve scientific problems: it is a tool to express and manage computations more efficiently. All the technologies are proven. Nobody in the software industry would try to develop software without using objects or better (functional programming…) “Our problem is too complex...” – Object Oriented Programming was invented to manage complexity in code more efficiently! – What about the software you (or your kids) use everyday? The difficulties are not technical, they are human – Most scientists do not have advanced training in software development – Can we afford to be so inefficient in the future? JEDI is for operational forecasting and scientific exploration!
Final Comments
Source: Will McCarty (NASA/GMAO)
Eye Candy
21
Discussion…
Unified Data Assimilation Planning Meeting – Apr. 4-5, 2017 - College Park, MD.