Warn-on-Forecast and High Impact Weather Workshop,Norman, OK, 8-9 Feb 2012

Challenges of assimilating all-sky satellite radiances

Milija Zupanski, Man Zhang, and Karina Apodaca

Cooperative Institute for Research in the AtmosphereColorado State University

Fort Collins, Colorado, U. S. A.[ http://www.cira.colostate.edu/projects/ensemble/ ]

Acknowledgements:

- NOAA NESDIS/GOES-R

- NOAA NESDIS/JCSDA

- NSF Collaboration in Mathematical Geosciences

Relevance of all-sky radiance assimilation

Current status

Challenges

Future

Overview

Satellite observations

Remote sensing is the major source of observations - radars- satellites

AMSU-A GOES-11 SNDR

Need to maximize the utility of cloud observations

- hurricanes

- severe weather

Satellite data are available everywhere- open ocean- polar regions- other isolated areas on the globe

Current status of all-sky radiance data assimilation

Most operational centers assimilate only clear-sky radiances

- The wealth of cloud-related measurements is discarded

- However, most high impact weather events are characterized by the presence of clouds and precipitation

- The consequence is a sub-optimal use of satellite observations

Limited research and operational efforts

- Mostly related to the use of variational methods, only recent use of ensemble and hybrid variational-ensemble methods

- All-sky microwave data assimilation operational at ECMWF since 2009 (Bauer et al. 2010; Geer et al. 2010 - SSM/I, TMI. AMSR-E)

- Pre-operational testing at NCEP, also pursued at other operational weather centers

Potential benefit of all-sky radiance assimilation is generally accepted, but it is difficult to extract the maximum information from these observations

- modeling of clouds (e.g., microphysics)

- data assimilation methodology

- computing resources, high resolution

Re-development of the TS Erin (2007): Distribution of AMSU-B radiance data in the NCEP operational data stream: (a) all observations, (b) accepted observations after cloud clearing. Data are collected during the period 15-18Z, August 18, 2007. Note that almost all observations in the area of the storm got rejected by cloud clearing. (from Zupanski et al. 2011, J. Hydrometeorology)

Impact of cloud clearing (radiance assimilation)

Need assimilation of all-sky radiances to improve the observation information value

Motivation - CIRA DA research

Develop a robust and efficient data assimilation for high impact weather events

- tropical cyclones

- severe weather

Focus on assimilation of cloud and precipitation affected satellite measurements, such as all-sky radiance assimilation

Utilize operational codes as much as possible, focus on realistic issues

- WRF NMM, HWRF

- GSI

- CRTM

Assimilate cloudy radiance from various sources:

- microwave, infrared, lightning

- combine information from different sources to find most beneficial combinations

List of instruments

- New: GOES-R (ABI, GLM), JPSS (ATMS, CrIS),

- Existing: AMSU-A,B, MHS, AMSR-E, TMI, MSG SEVIRI, WWLLN

Challenges of all-sky radiance data assimilation

Data assimilation: Methodological and computational issues

Microphysical control variables

- allow cloud observations to impact hydrometeors

Forecast error covariance

- Forecast error covariance needs to be state-dependent, and also to represent dynamical and microphysical correlations

Nonlinearity and non-differentiability of Radiative Transfer (RT) operator

Correlated observation errors

Non-Gaussian errors

Quantifying all-sky radiance information:

- How to provide a maximum utility of these data, and how to measure success?

Other relevant issues: verification, code maintenance, bias correction, …

Everything is connected, need to take into account all components

Relevance of microphysical control variables

Adjustment of microphysical control variables:- provides a more complete control of initial conditions- allows most direct impact of cloud observations on the analysis- critical for high impact weather (e.g., TC and severe weather)

Microphysics control variables: impact on DA

Physically unrealistic analysis adjustment without hydrometeor control variable (cloud ice in this example)

Temperature analysis increment at 850 hPa

No cloud ice adjustment

> 25 K

With cloud ice adjustment5-10 K

Forecast error covariance

Analysis correction from variational and ensemble DA can be represented as a linear combination of the forecast error covariance singular vectors ui

Singular value decomposition (SVD) of the forecast error covariance

Fundamentally important to have adequate forecast error covariance.

For clouds and precipitation, this implies flow-dependent and dynamically meaningful representation of model uncertainties.

Structure of forecast error covariance defines the analysis correction!

The quality of data assimilation can be assessed by examining the structure of forecast error covariance!

Use single observation experiment to assess the structure:

Forecast error covariance: Algebra

Only Pdd is well known:

- Correlations among microphysical variables not well known

- Even less known correlations between dynamical and microphysical variables

Complex inter-variable correlations (e.g., standard dynamical variables and microphysical variables)

Correlations between dynamical variables Correlations between microphysical variables

Cross-correlations between dynamical and microphysical variables

Forecast error covariance: Algebra

Both methods have limitations in representing cloud-related correlations

Variational: modeling of cross-covariances, time-dependence

Ensemble: reduced rank

Ensemble methods: PRR = reduced rank error covariance

Hybrid variational-ensemble DA methods are likely the optimal choice for assimilation of cloud-related observations (i.e. all-sky radiances)

Variational methods: PM = modeled error covariance

Single observation of cloud snow at 650 hPa:ensemble DA horizontal response

- Corresponds to high-frequency MW radiance observation

- WRF model (nest at 3km)

- 09 Sep 2012 at 1800 UTC

Horizontal analysis increments for (a) snow, and (b) north-south wind component

(b) V-wind at 650 hPa (Pv,snow)(a) Snow at 650 hPa (Psnow,snow)

A B

Single observation of cloud snow at 650 hPa:ensemble DA vertical response

Vertical analysis increments for (a) snow, and (b) rain.

(a) Snow at 34N (Psnow,snow) (b) Rain at 34N (Prain,snow)

Difficult to model rain-snow correlation:

non-centered response and time-dependence

X

The same cost function can be defined for variational and for Kalman Filter (e.g., EnKF) methods (Jazwinski 1970):

- KF: an explicit minimizing solution of quadratic cost function using Newton method

- VAR: an iterative solution of an arbitrary nonlinear cost function

Nonlinear observation operators: (Forgotten) Role of Hessian preconditioning

Nonlinearities increase for precipitation affected radiances- scattering- clouds, aerosol

Cost function in (a) physical, and (b) preconditioned space

• Hessian preconditioning has to be “balanced” (i.e. similar adjustment in all variables). Otherwise, minimization will create imbalances.

Non-differentiable RT observation operators

All-sky radiative transfer calculation has two computational branches:- clear-sky- cloudy and precipitation-affected

Decision about required calculation depends on model variables, thus creates a discontinuity in gradient and/or cost function

Since commonly used iterative minimization is gradient-based, non-differentiability could have a large impact on the analysis

Assimilation of all-sky radiances may benefit from non-differentiable minimization, or other means of addressing discontinuities

all-sky radiance observation information content (Degrees of Freedom for Signal – DFS)

MW: AMSR-E all-sky radiance data assimilation (Erin, 2007)

(from Zupanski et al. 2011, J. Hydrometeorology) OBS 89v GHz Tb Wind analysis uncertainty (500

hPa)Degrees of Freedom for Signal

(DFS)

IR: Assimilation of synthetic GOES-R ABI (10.35 mm) all-sky radiances (Kyrill, 2007) (from Zupanski et al. 2011, Int. J. Remote Sensing)

Cloud ice analysis uncertainy Degrees of Freedom for Signal (DFS)

METEOSAT Imagery valid at19:12 UTC 18 Jan 2007

Analysis uncertainty and DFS are flow-dependent, largest DFS in cloudy areas of the storm.

Quantification of Shannon information for all-sky radiances

Use mutual information (I) and entropy (H)

Joint entropy H(Y1,Y2) can be used to quantify the loss of information due to correlations between all-sky radiance observations Y1 andY2

Mutual information of dependent variables is smaller than mutual information of independent variables

By definition

Use

To obtain

• All-sky radiance observations are correlated

• How to measure information from correlated observations?

Future

Important to develop capability to extract maximum information from cloudy and precipitation-affected radiances

Critical for improved analysis and prediction of TC and severe weather outbreaks

Take into account all relevant components (e.g., current challenges)

- partial solutions will not bring true progress

- if Gaussian error statistics is not correct, but used, the cost function is inadequate, implying incorrect minimizing solution

- unbalanced Hessian preconditioning will adversely change the adjustment of variables by creating dynamical imbalances of the analysis

Computation

- RT computation increases 2-3 times with scattering

- number of observations increases by an order of magnitude due to cloudy information

- 20-30 times more expensive to compute

Improved information measures for all-sky radiances (e.g., assessment)

Combine information from various sources: GOES-R, JPSS (MW, IR, Lightning)

References:

Non-differentiable minimization

Steward, J. L., I. M. Navon, M. Zupanski, and N. Karmitsa, 2011: Impact of Non-Smooth Observation Operators on Variational and Sequential Data Assimilation for a Limited-Area Shallow-Water Equation Model. Quart. J. Roy. Meteorol. Soc., DOI: 10.1002/qj.935.

All-sky IR

Zupanski D., M. Zupanski, L. D. Grasso, R. Brummer, I. Jankov, D. Lindsey and M. Sengupta, and M. DeMaria, 2011: Assimilating synthetic GOES-R radiances in cloudy conditions using an ensemble-based method. Int. J. Remote Sensing, 32, 9637-9659.

All-sky MW

Zupanski, D., S. Q. Zhang, M. Zupanski, A. Y. Hou, and S. H. Cheung, 2011: A prototype WRF-based ensemble data assimilation system for downscaling satellite precipitation observations. J. Hydromet., 12, 118-134.