on the value of reforecasts for the tigge database
DESCRIPTION
On the value of reforecasts for the TIGGE database. Tom Hamill NOAA/ESRL/PSD. Renate Hagedorn European Centre for Medium-Range Weather Forecasts. Motivation. One goal of TIGGE is to investigate whether multi-model predictions are an improvement to single model forecasts - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/1.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 1/27
On the value of reforecasts for the
TIGGE database
Renate Hagedorn European Centre for Medium-Range Weather Forecasts
Tom HamillNOAA/ESRL/PSD
![Page 2: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/2.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 2/27
Motivation
• One goal of TIGGE is to investigate whether multi-model predictions are an improvement to single model forecasts
• The goal of using reforecasts to calibrate single model forecasts is to provide improved predictions
• Questions:
What are the relative benefits (costs) of both approaches?
What is the mechanism behind the improvements?
Which is the “better” approach?
![Page 3: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/3.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 3/27
Possible verification datasets
• If we don’t verify against model independent observations we need to agree on a ‘fair’ but also ‘most useful’ verification dataset
• Use each model’s own analysis as verification Multi-model has no “own analysis”
Intercomparison of skill scores “difficult” because reference forecast scores differently for different analysis
• Use a multi-model analysis as verification Incorporating less accurate analyses might not necessarily lead to an
analysis which is closest to reality
Calibration needs a consistent verification dataset used in both training and application phase, MM-analysis not available for reforecast training period
• Use “semi-independent” analysis: ERA-interim Assumed to be as close as possible to reality
Available for long period in the past and near real-time
For upper air fields in Extra-Tropics close to analyses of best models / MM-analysis
For Tropics and near-surface fields use bias-corrected forecasts for ‘fair’ assessment
![Page 4: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/4.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 4/27
Choice of analysis: upper air, extra-tropics
dashed: ERA-interim as verification
T-850hPa, DJF 2008/09Northern Hemisphere (20°N - 90°N)
NCEPMet OfficeECMWFTIGGE
solid: multi-model analysis as verification
Using ERA-interim leads to onlyminor differences, except for short lead times when scores getworse (applies for all models)
![Page 5: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/5.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 5/27
Choice of analysis: upper air, tropics
dashed: ERA-interim as verification
NCEPMet OfficeECMWFTIGGE
solid: multi-model analysis as verification
Using ERA-interim worsens scores considerably / less / least for MO / ECMWF / NCEP
T-850hPa, DJF 2008/09Tropics (20°S - 20°N)
![Page 6: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/6.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 6/27
Choice of analysis: surface
dashed: ERA-interim as verification
T2m, DJF 2008/09Northern Hemisphere (20°N - 90°N)
NCEPMet OfficeECMWFTIGGE
solid: multi-model analysis as verification
Using ERA-interim worsens scores, in particular at early lead times,more for MO and NCEP, less for ECMWF
![Page 7: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/7.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 7/27
Choice of analysis: surface, bias-corrected
dashed: DMO with ERA-interim as verification
T2m, DJF 2008/09Northern Hemisphere (20°N - 90°N)
NCEPMet OfficeECMWFTIGGE
solid: Bias-Corr. with ERA-interim as verification
Bias-correction improves scores, in particular at early lead times,more for MO and NCEP, less for ECMWF
![Page 8: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/8.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 8/27
Comparing 9 TIGGE models & the MM
T-850hPa, DJF 2008/09NH (20°N - 90°N)DMO vs. ERA-interim
Symbols used forsignificance levelvs. MM (1%)
![Page 9: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/9.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 9/27
Comparing 9 TIGGE models & the MM
T-2m, DJF 2008/09NH (20°N - 90°N)BC vs. ERA-interim
![Page 10: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/10.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 10/27
Comparing 4 TIGGE models & the MM
T-850hPa, DJF 2008/09NH (20°N - 90°N)DMO vs. ERA-interim
![Page 11: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/11.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 11/27
Comparing 4 TIGGE models & the MM
T2m, DJF 2008/09NH (20°N - 90°N)BC vs. ERA-interim
![Page 12: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/12.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 12/27
2
( )( ) ens
ens
q a bxP v q
c ds
⎡ ⎤− +⎢ ⎥≤ =Φ⎢ ⎥+⎣ ⎦
with: Φ = CDF of standard Gaussian distribution
Calibration using reforecasts
• All calibration methods need a training dataset, containing a number of forecast-observation pairs from the past
• Non-homogeneous Gaussian Regression (NGR) provides a Gaussian PDF based on the ensemble mean and variance of the raw forecast distribution
• Calibration process:
Determine optimal calibration coefficients by minimizing CRPS for training dataset
Apply calibration coefficients to determine calibrated PDF from ensemble mean and variance of actual forecast to be calibrated
Create calibrated NGR-ensemble with 51 synthetic members
Combine NGR-ensemble with ‘30-day bias corrected’ forecast ensemble
![Page 13: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/13.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 13/27
The reforecast dataset2008 2009Nov Dec Jan
29 30 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 01 02
1990
1991
1992
1993
1994
.
.
.
.
2003
2004
2005
2006
2007
![Page 14: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/14.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 14/27
The reforecast dataset2008 2009Nov Dec Jan
29 30 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 01 02
1990
1991
1992
1993
1994
.
.
.
.
2003
2004
2005
2006
2007
![Page 15: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/15.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 15/27
Comparing 4 TIGGE models, MM, EC-CAL
2m Temperature, DJF 2008/09NH (20°N - 90°N)BC & refc-cali vs. ERA-interim
![Page 16: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/16.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 16/27
Comparing 4 TIGGE models, MM, EC-CAL
2m Temperature, DJF 2008/09EU (35°N-75°N, 12.5°E-42.5°W)BC & refc-cali vs. ERA-interim
![Page 17: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/17.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 17/27
Comparing 4 TIGGE models, MM, EC-CAL
MSLP, DJF 2008/09NH (20°N - 90°N)BC & refc-cali vs. ERA-interim
![Page 18: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/18.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 18/27
Comparing 4 TIGGE models, MM, EC-CAL
T-850hPa, DJF 2008/09NH (20°N - 90°N)DMO & refc-cali vs. ERA-interim
![Page 19: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/19.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 19/27
Mechanism behind improvements
SPREAD (dash)
RMSE (solid)2m Temperature, DJF 2008/09Northern Hemisphere (20°N - 90°N)Verification: ERA-interim
![Page 20: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/20.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 20/27
Mechanism behind improvements
SPREAD (dash)
RMSE (solid)2m Temperature, DJF 2008/09Northern Hemisphere (20°N - 90°N)Verification: ERA-interim
![Page 21: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/21.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 21/27
Mechanism behind improvements
SPREAD (dash)
RMSE (solid)2m Temperature, DJF 2008/09Northern Hemisphere (20°N - 90°N)Verification: ERA-interim
![Page 22: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/22.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 22/27
Reduced TIGGE multi-model
2m Temperature, DJF 2008/09Northern Hemisphere (20°N - 90°N)Verification: ERA-interimCRPS_ref = CRPS (full TIGGE)
![Page 23: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/23.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 23/27
TIGGE vs. ECMWF vs. EC-CAL
2m Temperature, DJF 2008/09Northern Hemisphere (20°N - 90°N)Verification: ERA-interim
![Page 24: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/24.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 24/27
Impact of calibration & MM in EPSgrams
31/12 02/01 04/01 06/01 08/01 10/01 12/01 14/01Forecast Day
-15
-10
-5
0
5
10
15
T2m [C]
2m TemperatureFC: 30/12/2008
ECMWFECMWF-NGRTIGGEAnalysis
Monterey
31/12 02/01 04/01 06/01 08/01 10/01 12/01 14/01Forecast Day
-15
-10
-5
0
5
10
15
London
![Page 25: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/25.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 25/27
What about station data?
(No significance test applied)
![Page 26: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/26.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 26/27
Relative benefits and costs
TIGGE multi-modelNGR Calibration
using reforecastsBenefits: upper air fields
limited limited
Benefits:surface fields
Improved scores through reduced systematic error and increased spread
Improved scores through reduced systematic error and more appropriate spread
Costs:Computational aspects
No extra computer time but data transfer costs
Moderate increase in computing time (~10%),“for free” if reforecasts are produced for other purposes
Costs:Logistic aspects
Significantly increased complexity could make system more prone to failures, and timing issues could arise
Slight increase in complexity, e.g. when changing model cycles
![Page 27: On the value of reforecasts for the TIGGE database](https://reader036.vdocuments.net/reader036/viewer/2022062518/568143d5550346895db062e0/html5/thumbnails/27.jpg)
18 September 2009: On the value of reforecasts for the TIGGE database 27/27
Summary
• What are the relative benefits (costs) of both approaches?
Both multi-model and reforecast calibration approach can improve predictions, in particular for (biased and under-dispersive) near-surface parameters
• What is the mechanism behind the improvements?
Both approaches correct similar deficiencies to a similar extent
• Which is the “better” approach?
On balance, reforecast calibration seems to be the easier option for a reliable provision of forecasts in an operational environment
Both approaches can be useful in achieving the ultimate goal of an optimized, well tuned forecast system