Using Pseudo RAOB Observations to Study GFS Skill Score Dropouts

NOAA/NWS/NCEP/ Environmental Modeling Center

Jordan C. Alpert, Brad Ballish, Da Na Carlis, V. Krishna Kumar

5A.6 AMS 23WAF_19NWP Conf, 1-5 June 2009, Omaha, NE

Using Pseudo RAOB Observations to Study GFS Skill Score Dropouts

NOAA/NWS/NCEP/ Environmental Modeling Center

Jordan C. Alpert, Brad Ballish, Da Na Carlis, V. Krishna Kumar

5A.6 AMS 23WAF_19NWP Conf, 1-5 June 2009, Omaha, NE

The “Dropout” Team

3

Need to find a Remedy for the Dropouts

Determine if “dropouts” are from QC, GSI analysis, GFS forecast model

• Define Dropout (verification web page stats)• Quantify Dropout Events• If QC is responsible then construct (implement) a

real time monitoring (detection/correction) system.• Determine Common Threads in Dropout Cases

4

What is a “dropout”?The criteria that a 5-day 500 mb Anomaly Correlation (AC) height score must meet in order to be considered a dropout (Alpert et al, 2009) :

• At least one of the following criteria must be met:– a) ECMWF minus GFS ≥ 15 AC points– b) GFS AC ≤ 0.70– c) ECMWF AC ≤ 0.75– d) Monthly avg GFS AC score minus GFS forecast ≥ 15– e) Monthly avg ECMWF AC score minus ECMWF forecast ≥ 15

• Criteria is for NH and SH dropouts.

On approximately a monthly basis, poor forecasts or “Skill Score Dropouts” plague GFS performance. GFS production in Black, ECMWF in Red.

5

Tools to compare ECMWF and NCEP Dropouts

• Use ECMWF analysis to generate “Pseudo Obs” for input to the Gridded Statistical Interpolation (GSI) and GFS forecasts.

• Generate a Climatology of NH and SH dropouts – what are the systematic differences

• Interpolate ECM and GSI analyses to observations to determine comparative strengths and weaknesses.– Statistically analyze observation type fits– stratify by pressure, type, and difference magnitude

Goal: Diagnose Quality Control problems to implement real-time QC detection/correction and improvements to analysis system algorithms

6

Poor Forecasts or Skill Score “Dropouts” Lower GFS Performance.

For this Dropoutsee IC on next slide...

7

Trough in central Pacific shows differences between ECMWF(no dropout) and GFS (had dropout)

Ovrly “patch” boxECM in this area

but GSI elsewhere

8

Oct. 22,2007 Pacific OVRLY

GFS: NCEP OpsECMWF: ECMWF OpsECM: ECMWFGSIOVRLY: Patch

F00

9

GFS: NCEP OpsECMWF: ECMWF OpsECM: ECMWFGSIOVRLY: Patch

F120

Lat/Lon Box IC Date GFS ECMWF ECM OVRLY

20N→ 80N150E→ 230E

2007102212 0.61 0.87 0.89 0.90

10

INPUT-----------OUTPUT( )

ECM(WF) Analysis1x1 deg, 14 levels------------------------- PSEUDO ECM RAOBS

ECMWF INITIAL CONDITIONS FOR GFS FORECASTS

“ECM” Runs”

11

INPUT-----------OUTPUT( )

ECM(WF) Analysis1x1 deg, 15 levels------------------------- PSEUDO ECM RAOBS

PSEUDO ECM RAOBSGFS GUESS--------------

ECM ANLYSIS“PRE-COND” GUESS

ECMWF INITIAL CONDITIONS FOR GFS FORECASTS

“ECM” Runs”

12

INPUT-----------OUTPUT( )

ECM(WF) Analysis1x1 deg, 15 levels------------------------- PSEUDO ECM RAOBS

PSEUDO ECM RAOBSGFS GUESS--------------

ECM ANLYSIS“PRE-COND” GUESS

Run GSI -----------Analysis

ECMWF INITIAL CONDITIONS FOR GFS FORECASTS

“ECM” Runs”

PRE-CONDECM GUESS

PSEUDO ECM RAOBS

---------------ECM ANL

13

INPUT-----------OUTPUT( )

ECM(WF) Analysis1x1 deg, 15 levels------------------------- PSEUDO ECM RAOBS

PSEUDO ECM RAOBSGFS GUESS--------------

ECM ANLYSIS“PRE-COND” GUESS

Run GSI -----------Analysis

Run GSIagain

ECMWF INITIAL CONDITIONS FOR GFS FORECASTS

“ECM” Runs”

PRE-CONDECM GUESS

PSEUDO ECM RAOBS

---------------ECM ANL

14

INPUT-----------OUTPUT( )

ECM(WF) Analysis1x1 deg, 15 levels------------------------- PSEUDO ECM RAOBS

PSEUDO ECM RAOBSGFS GUESS--------------

ECM ANLYSIS“PRE-COND” GUESS

Run GSI ------------Analysis

Run GSIagain

ECMWF INITIAL CONDITIONS FOR GFS FORECASTS

“ECM” Runs”

PRE-CONDECM GUESS

PSEUDO ECM “OBS”

---------------ECM ANL

ECM ANL ------------GFS 5-d

FORECAST

15

06Z Pseudo OBS3,6,9-h GUESS----RUN GSI----

ECM ANALYSIS

12Z Pseudo OBS3,6,9-h GUESS----RUN GSI----

ECM ANALYSIS

Make 3,6,9-h GFS forecast

Make 3,6,9-h GFS forecast

Make 3,6,9-h GFS forecast

Make 3,6,9-h GFS forecast

INPUT-----------OUTPUT( )

First Time

ECMWF INITIAL CONDITIONS FOR GFS FORECASTS “ECMCYC” Runs”

Figure 2. Schematic representation of an ECM run using the GSI/GFS system and ECMWF 14 level pressure and 1x1 degree analysis files.

PSEUDO ECM “OBS”Production GFS GUESS

------RUN GSI------ECM ANLYSIS

00Z

18Z Pseudo OBS3,6,9-h GUESS----RUN GSI----

ECM ANALYSIS

00Z Pseudo OBS3,6,9-h GUESS----RUN GSI----

ECM ANALYSIS

16

ECMCYC

ECMCYC

17

• ECM runs (blue) are a good representation for ECMWF analysis• OVRLY runs (green) with ECM psuedo-obs over the Central Pacific drastically improve two October 2007 dropout cases (102200 & 102212).

5 Day Anomaly Correlation Scores at 500 hPa for Dropout CasesECM Performs Better than GFS (NH)

2007-2008

0.4

0.5

0.6

0.7

0.8

0.9

1

2007102112 2007102200 2007102212 2007102312 2008021712 2008030112 2008030400 2008030412

Initialization Date

500

mb

Ano

mal

y C

orre

latio

n

GFSECMWFECMOVRLY

18

SH 5-day 500 mb Anomaly Correlation Scores

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

2008011100 2008011212 2008020300 2008030312 2008031800 2008031812 2008042512 2008042600 2008052200 2008062512

Initial Date

500

mb

Ano

mal

y C

orre

latio

nGFS

ECMW F

ECM

CNTRL

ECM runs (blue) in the SH do almost as well as ECMWF

CNTRL runs (green) improve upon GFS scores 9 of 10 times but only alleviates about half of the dropouts, but large skill gap remains.

19

SOUT HERN HEM ISPHERE 5-DAY 500 M B ANOM ALY CORRELAT ION SCORES

0.4

0.45

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

INITIAL CONDITION DATE (YYYYMMDDHH)

AN

OM

ALY

CO

RR

ELA

TIO

N

MINDATA

PREPB

AMSUA

AMSUB

MHS

GPSRO

AIRS

HIRS

CNTRL

Figure 10. Impact of satellite radiance data on Southern Hemisphere dropouts. 5-day anomaly correlation scores shown for 10 cases. MINDATA is the GFS/GSI run with only TRMM and SSMI data present, PREPB is the GFS/GSI run with only conventional observations (i.e. RAOB, satellite winds, profiler, etc..), AMSUA is the GFS/GSI run with conventional observations and amsua satellite radiance data, AMSUB is the GFS/GSI run with conventional observations and amsub satellite radiance data, MHS is the GFS/GSI run with conventional observations and mhs satellite radiance data, GPSRO is the GFS/GSI run with conventional observations and gpsro satellite radiance data, AIRS is the GFS/GSI run with conventional observations and airs satellite radiance data, HIRS is the GFS/GSI run with conventional observations and hirs satellite radiance data, and CNTRL is the GFS/GSI run with conventional observations and all satellite radiance data present with a 6-hr data ingest window.

20

SOUTHERN HEMISPHERE 5-DAY 500 MB RMS SCORES

0

20

40

60

80

100

120

2008011100 2008011212 2008020300 2008030312 2008031800 2008031812 2008042512 2008042600 2008052200 2008062512

INITIAL CONDITION DATE (YYYYMMDDHH)

RMS

MINDATA

PREPB

AMSUA

AMSUB

MHS

GPSRO

AIRS

HIRS

CNTRL

Figure 11. Impact of satellite radiance data on Southern Hemisphere dropouts. 5-day root-mean-error scores shown for 10 cases. The experiments are named the same as Fig. 10.

21

Impact of Conventional and Satellite Radiance Obs on 3-d and 5-d GFS Forecasts

10 Case Composite Satellite Radiance Impact on NH and SH 500 mb AC

0.5

0.6

0.7

0.8

0.9

1

3-D NH MEAN 3-D SH MEAN 5-D NH MEAN 5-D SH MEAN

Ano

mal

y C

orre

lati

on NODATA

PREPB

AMSUA

AMSUB

MHS

GPSRO

AIRS

HIRS

CNTRL

• Satellite radiance data shows positive impact on NH 3-D and 5-D forecasts• In the SH, conventional (PREPB alone) show a large negative impact (8 points) in 5-D forecasts! -- another puzzle• Addition of satellite radiance data to conventional and satellite observations have a positive impact on 5-D forecasts• AMSUA along with PREPB conventional observations (yellowyellow) show the largest positive impact in the NH and SH experiments and typically are correlated with the results of the CNTRL•This is confirmed by other studies (ECMWF and Joint Center) and shows how each case has individual characteristics requiring incisive diagnostics.

22

Summary • ECM analyses show dropouts can be alleviated in

GFS forecasts• Running the operational GSI with an ECM

derived background guess results in better forecast skill than the operational GFS but not as good as ECM runs

• Running the GSI after removing selected observation types offers a systematic approach for assessing the impact of different observation types

• Quality control problem detection and correction algorithms per observation type maybe be possible using analysis differences and areas of potential activity like baroclinic instability index

23

Future Work• ECM analyses show dropouts can be alleviated in GFS

forecasts• Expand Dropout Team• Additional effort focused on improving GFS QC and

Model• Augmented and corrected observation data base information (Ballish, et al 2009)• Develop and test bias correction for conventional observations • Diagnose potentially important impact of humidity through satellite data and improve use of current satellite data• Negative moisture in background model field - Continue to develop global model• Test Sat wind QC e.g., Develop and test thinning (a Super-Ob strategy)• Develop use of ODB for diagnostics and verification

• Construct a detection system of Dropouts (Cycle ECM runs to compare with production)• Battery of Diagnostic displays for examination based on an alarm or event• Detection and event recognition to trigger alarms to start offline procedures for post

mortem • Explore additional diagnostic tools

– Adjoint techniques– Obs. Sensitivity

• Bishop & Toth (THORPEX WSR Targeting Technique)• Kalnay (EnKF Technique)

24

Background Slides

25

SH Dropout Composite Cntrl vs InterpGES

Zonal average (above) and mean composite difference between GFS Cntrl and InterpGES (Increments). GFS has low level SH warm “bias” if we consider ECMWF as ground truth.

SH CNTRL 5-day Composite AC = 0.71 SH InterpGES 5-day Composite AC = 0.73

26

Figure 3. The total Eady Baroclinic Index (EBI) (day -1) at 500 hPa for the F00 forecast from the August 16, 2008, 00 UTC (2008081600) initial conditions for the a) GFS model, and b) ECM model with two iterations of the GSI and from April 13, 2009 12 UTC for the ECMCYC run. Note the difference in the dates.

c)

27

Satellite Data Counts on March 4, 2009 12Z  Received Selected Assimilated

amsua 14,971,259 3.42% 672,912 7.28% 487,741 22.69%

amsub 10,939,849 2.50% 69,468 0.75% 48,559 2.26%

airs 194,014,483 44.38% 2,068,232 22.37% 420,579 19.56%

sndr 7,911,144 1.81% 69,282 0.75% 27,099 1.26%

gpsro 137,548 0.03% 127,458 1.38% 34,649 1.61%

hirs3 6,822,395 1.56% 313,766 3.39% 34,576 1.61%

hirs4 6,711,471 1.54% 311,906 3.37% 36,841 1.71%

mhs 6,936,840 1.59% 47,200 0.51% 38,693 1.80%

ssmi 21,584 0.00% 21,584 0.23% 3,784 0.18%

trmm 10,628 0.00% 10,628 0.11% 1,526 0.07%

sbuv 22,220 0.01% 21,670 0.23% 6,996 0.33%

iasi 188,385,736 43.10% 4,750,523 51.38% 727,367 33.83%

satwnd 244,210 0.06% 244,210 2.64% 244,210 11.36%

quikscat 516,763 0.12% 516,763 5.59% 37,439 1.74%

TOTAL 437,129,367 100.00% 9,245,602 100.00% 2,150,059 100.00%

28

18% loss 36% loss

33% loss

100% loss 100% loss

32% loss

AMSUA

HIRS4

GPSRO

AIRS

MHS

QSCAT

Figure 13. Satellite data counts for the period 20081230-20090107 covering the 5-day forecast dropout initializing on 2009010312.