how 4dvar can benefit from or contribute to enkf (a 4dvar...

How 4DVAR can benefit from or contributeto EnKF (a 4DVAR perspective)

Dale Barker

WWRP/THORPEX Workshop on 4D-Var and EnsembleKalman Filter Intercomparisons

Sociedad Cientifica Argentina, Buenos Aires, Argentina,November 10-13th 2008

Alternative Formulations of Advanced DA

Information/analysis space analysis step (ECMWF, Met. Office,etc…):

Error/observation space form (PSAS, NAVDAS, EnKF):

Both can be written in terms of a “Kalman Gain” K

Equivalence between 4D-Var/EKF for Gaussian errors/linear model.

xa ! x

b= H

TR

!1H + P

f

!1"# $%!1H

TR

!1y ! H (xb

)"# $%

xa ! x

b= P

fH

THP

fH

T+ R

!1"# $%!1

y ! H (xb)"# $%

xa ! x

b= K y ! H (xb

)"# $%

4D-Var vs. EnKF (Kalnay et al 2007)

“The data assimilation community is at a transition point, with achoice between variational methods…., and ensemble methods.”Kalnay et al. (2007a).

“The idea of gradual development should..be applied to the ongoingdiscussions on 4D-Var and EnKF…More appropriate to ask“How can ideas from EnKF and 3/4D-Var best be combined”.”Gustaffson (2007).

“Completely agree….Ideas developed in 4D-Var…can be easilyadapted and included within EnKF”. Kalnay et al. (2007b)

Kalnay vs. Gustaffson (2007)

Fundamental Issues (Covered this week)

• Balance (Errico, Polavarapu, Kepert, Fillion, …):• 4D-Var: Dynamical/statistical balance, Weak-constraint DF.• EnKF: Use balance constraint to correct localization effects.• Joint: Spin-up/initialization (especially at convective-scale).

• Nonlinearity/non-Gaussianity (Fisher, Yang, …):• 4D-Var: Outer-loop, variable transforms, VarQC.• EnKF: Adoption of outer-loop. MLEF approach.

• Model Error (Tremolet, Kalnay, …):• 4D-Var: Weak constraint 4D-Var. Equivalance with EKF.• EnKF: ‘Particularly vulnerable’. Inflation + bias removal (e.g.

Dee & DaSilva)

Ensemble/NMC-method based Climatological Statistical Balance

�

!b

•! ! •!

�

Tb

•T T •T

NM

C-M

etho

dEN

S M

etho

d

• T+48-T+24 fcst. Differences for NMC-method.• T+12 KMA EPS (16 bred-mode) ensemble Data.• June 2005 data.

�

!b

'= c" '

�

Tb

'(k) = G(k,k1)! '

(k1)k1

"

ETKF vs NMC-method ClimatologicalCovariances (regional WRF)

Horizontal Correlation Scales Eigenvalues (Variances)

Positive Impact of ETKF-based covariances(Wang, Barker,Hamill, and Snyder, 2008a)

EnKF Convective-Scale Multivariate Forecast ErrorCovariances (Z=reflectivity) (Ming Xue)

Shading : True Fields Line Contours : Error Correlations

Study offline to train statistical balance in VAR?


• Balance (Errico, Polavarapu, Kepert, Fillion, …):• 4D-Var: Dynamical/statistical balance, Weak-constraint DF.• EnKF: Use balance constraint to correct localization effects.• Joint: Spin-up/initialization (especially at convective-scale).

• Nonlinearity/non-Gaussianity (Fisher, Yang, …):• 4D-Var: Outer-loop, variable transforms, VarQC.• EnKF: Adoption of outer-loop. MLEF approach.

• Model Error (Tremolet, Kalnay, …):• 4D-Var: Weak constraint 4D-Var. Equivalance with EKF.• EnKF: ‘Particularly vulnerable’. Inflation + bias removal (e.g.

Dee & DaSilva)

Maximum Likelihood Ensemble Filter(Zupanski 2005)

• A variational or an ensemble data assimilation scheme?

• Minimizes nonlinear 4D-Var cost function:

• Hessian preconditioning via change of variable to minimize inensemble space (using ensemble covariance Pf):

• Like the ETKF, cannot localize directly in ensemble space.

J =1

2x ! x

b( )TPf

!1 x ! xb( ) +

1

2yo ! H (x)"# $%

T

R!1 yo ! H (x)"# $%

x ! xb= P

f

1/2I +C( )

!T /2"

Ensemble-Based 4D-Var ‘En4DVAR’(Liu et al. 2008a,b)

• A variational or an ensemble data assimilation scheme?

• Define ‘control variable transform’ using ensemble perturbations

• Solves incremental, preconditioned 4D-Var cost function:

• Gradient calculation uses ensembles rather than adjoint model:

J(w) =1

2wTw +

1

2HM!X

fw + d

i[ ]T

i=0

I

" Ri

#1HM!X

fw + d

i[ ]

!x = !X fw

!wJ = w + U

TM

THT

i=0

I

" Ri

#1HMUw + d

i[ ]

!wJ = w + HM"X

f[ ]T

i=0

I

# Ri

$1HM"X

fw + d

i[ ]


• Background Error Estimation (Berre, Hamill, Bishop, …):• 4D-Var: Full-rank Pf. Flow-dependence modelled/via linear

model.• EnKF: Sampling errors. Flow-dependent, based on nonlinear

model. Adaptive localization.

Flow-Dependent Forecast Errors in Var: Early AttemptsDesroziers (1997) Barker (1999)

Purser et al (2003) Barker et al (2004) Buehner (2005)

(Daley and Barker 2000)

3D-Var (NoFGAT) 4D-Var

500mb θ increments from 3D-Var at 00h and from 4D-Var at 06h due to a 500mb T observation at 06h

(Hans Huang)

Hybrid Variational/Ensemble DABenefits for Var:

Introduces flow-dependent initial PDF in 3/4D-Var.

Explicit coupling between moisture/temp/wind fields (tropics, high-res).

Easily incorporated in Var framework.

Relatively cheap (if properly preconditioned and compressed).

Can’t do worse – can switch off ensemble covariances if detrimental.

Flow-dependent QC.

Benefits over ensemble filters:

Hybrids more robust for small ensemble sizes and large model error.

Cost does not scale with observations.

Can couple with nonlinear QC (serial filter can’t do that by itself).

Ensemble used for covariance modeling only - can still run high-res mean.

Hybrid Var/Ensemble DA References Barker (1999): Demonstration with 1 bred-mode + UKMO 3D-Var.

Hamill and Snyder (2000): Hybrid in Ensemble Framework.

Etherton and Bishop (2004): QG model.

Buehner (2005): Additional control variable/3D-Var: Small impact.

Wang, Hamill, Whitaker and Bishop (2007): Compare EnSRF/Hybrid/OI.

Wang, Barker, Hamill and Snyder (2008a, b): WRF AlphaCV+3D-Var.

Hybrid Testing With Lorenz 1996 model(K. Y. Chung: KMA Visitor to NCAR)

• 40-variable model (i=1, 40). F=8.• Periodic boundary conditions• OSSE: dt=6h. Simulate obs every 12h. 400d run, verify last 200d.• Hybrid 3D-Var/EnKF (Hamill and Snyder 2000):• Flow-dependence via a) Lagged forecast diff., b) EnKF perturbations.

�

dxi

dt= x

i+1 ! x i!2( )x i!1 ! x i + F

�

x1

= xI

�

Pf

= 1! "( )Pef

+ "B0

�

Pef

Lorenz Model OSSE Analysis Error, N=10

5.205.004.864.374.324.464.194.334.21Hybrid4.59EnSRF

0.90.80.70.60.50.40.30.20.1β5.703D-Var

Root Mean Square Analysis Error (x100)

3D-Var

β=0.3

EnKF

β=0.7

Example Hybrid Result(Etherton and Bishop 2004)

• Barotropic vorticitymodel.

• ETKF not localized.

• 3D (not 4D)-Var.

• Optimal mix of Var/Enscovariance ~70/30%.

Cycling WRF/WRF-Var/ETKF System (Hybrid DA)

�

xf 3/4D-Var

�

!xVar

a

�

!x1

f

�

!x2

f

�

!xNf

�

x1

a

�

xNf

�

x2

f

�

x1

f

�

H(x1

f),! o

�

yo

�

H(x2

f),! o

�

H(xNf),! o

E

T

K

F

�

!x1

a

�

!x2

a

�

!xN

a

.

.

.

.

.

.

.

.

.

.

.

.

�

x1

a

�

x2

a

�

xN

a

.

.

....

Forecast Assimilation

�

x2

a

�

xN

a

�

xna

= xf

+ !xVara

+ !xETKFna

Hybrid DA Via Additional Control Variables• Define the matrix of ensemble perturbations as

• Hybrid 3/4D-Var analysis increments give by

• Note flow-dependence constrained by a new set of control variables

• Could alternatively define the hybrid in control variable space, e.g.

•(4) better than (2 ) when balance well known (ref. Kepert).

!x0= !x

0d + !X f • a = Uv + !X f • a�

!X f = !x f1,!x f 2 ,...,!x fN( )

�

aT

= !1,!

2,...,!

N( )

�

!"0

= !"0d + !" f • a

�

!"u0 = !"u0d + !"uf • a

(1)

(2)

(4)�

!X f

(3)

Cost Functions and Variance Conservation

• Flow-dependence is constrained by an additional cost-function Jα , i.e.

• Define empirical alpha covariance matrix

• Wb and Wα are weights defined to conserve forecast error.

• Lorenc (2003)-type hybrid conservation:

• Forecast error variance conservation:

�

J =W

b

2!x

0

TBo

"1!x

0+W#

2aTA

"1a +

1

2H

i!x t

i( ) " di

[ ]i=0

n

$T

Ri

"1H

i!x t

i( ) " di

[ ]

�

A = !"

2A

c

�

1

Wb

+1

W!/"

!

2( )= 1

�

1

Wb

+1

W!/"

!

2= 1

Example Application of ACV in Global WRF-Var• Alpha correlation Ac is empirical function (e.g. Gaussian) with prescribed scale Lα.

• Lorenc (2003) suggest equivalence between Ac and covariance localization.

• Test ACV approach in global WRF-Var (spectral localization)Correlation Function Ac Corresponding Power Spectra

Lα=1500km

Practical vertical localization possible via similar data compression

Single Observation Test - Alpha CV• Specify single T observation (O-B, σο=1K) at 50N, 150E, 500hPa.•Example: Flow-Dependence given by 1 member of KMA’s EPS.

T’ in

crem

ent

u’ in

crem

ent

No Alpha Alpha, No Localization Alpha, Localization(1500km Gaussian)

WRF Test with single observation (X. Wang)

Flow-dependent ETKF ensemble covariance is successfully incorporated in WRF-Var

Static covariance Ensemble covariance with localization

Analysis increment

Hybrid In SPEEDY Model(Incomplete, but I promised Eugenia!)

3DVAR Hybrid (Wa=2) LETKF

• Single u observation• Hybrid equal weight on Var/ENS covariances.• 1000km localization applied in hybrid.

Regional WRF Application(E. Asia, US Air Force, 10 members)

U Mean U Spread

Ensemble Perturbations

WRF-Var Full Assimilation

Control (no hybrid)

U250 Analysis Increment U250 Ensemble Spread



We=2 , Wb=2



We=1.25 , Wb=5



We=1.1, Wb=11

Hybrid 1-Month Real-Data (Sonde) Trial(Wang, Barker, Hamill and Snyder 2008b)

RMS 12hr Fcst Fit to Sonde

RMS Analysis Fit to Sonde

3D-Var analysis fitsobservations more closely.

Hybrid forecast fitsobservations more closely.

Met. Office 4D-Var/LETKF Hybrid System

fx VAR x

a

�

!x1

f

�

!xNf

�

x1

a

�

xNf

�

x1

f

y1

�

yo

yn = yo,H (xn

f),! o,...

L

E

T

K

F

�

!x1

a

�

!xN

a

.

.

.

.

.

.

.

.

.�

x1

a

�

xN

a

.

.

.

�

xN

a

xn

a= x

a+ !x

n

a

xa OPS

OPS

OPS

yN

y

MOGREPS

VAR

UM1

UMN

UM

B

Conclusions• 4D-Var/EnKF solving same problem. Devil is in the detail.

• Distinction between variational/ensemble DA is blurred.

• Ensemble forecasting is here to stay. DA should make use of it.

• Hybrid in VAR shown. Hybrid in EF also possible.

• Flow-dependent covariances only one (now minor?)consideration. Bigger issues: non-Gaussianity, model error…..

• ‘Seamless assimilation/prediction’ implies flexible DA system.

Questions• Do we agree that the issue of flow-dependence in forecast error

covariances is now a relatively ‘solved’ problem, and we shouldfocus more on larger concerns e.g. non-Gaussianity, modelerror?

• Is the 4D-Var/EnKF computational cost comparison only anissue for those (fewer and fewer) centers running purelydeterministic forecast systems?

• Will full operational implementations of EnKF really be‘simpler’ than variational ones?

how 4dvar can benefit from or contribute to enkf (a 4dvar...

Documents