federal department of home affairs fdha federal office of meteorology and climatology meteoswiss...
TRANSCRIPT
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Extended range forecasts at MeteoSwiss: User experience and
probabilistic verification
ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel,
Mark Liniger, Paul Della-Marta,
Christof Appenzeller
2 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Overview
Monthly forecasts
WWW: Seasonal forecasts
Verification: The RPSSD
Comparison:
ECMWF vs. other prediction strategies
3 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Overview
Monthly forecasts
WWW: Seasonal forecasts
Verification: The RPSSD
Comparison:
ECMWF vs. other prediction strategies
4 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Monthly forecasts
100 %0
Probability of T2m to be in lowest tercile
Forecast of week 1Start: 20-04-2006
5 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Monthly forecasts
100 %0
Probability of T2m to be in lowest tercile
Forecast of week 1Start: 04-05-2006
6 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Monthly forecasts
Probability of T2m to be in lowest tercile
100 %0
Forecast of week 1Start: 11-05-2006
7 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Monthly forecasts
Observed anomalies for May
What is wrong?Problems to deal with enhanced snow cover?...
8 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Overview
Monthly forecasts
WWW: Seasonal forecasts
Verification: The RPSSD
Comparison:
ECMWF vs. other prediction strategies
9 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
WWW: Seasonal forecasts
Since winter 2005/06 MeteoSwiss issues an internet bulletin
(»Climate Outlook«) on the upcoming season for Switzerland.
Designed to... provide seasonal forecast give background information on methodology point out uncertainties provide climatologic background information
Provide common reference for public and media, and
avoid dissemination of semi-true information
Use seasonal forecasts to promote public interest in other aspects of climate analysis
10 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Facts and figures on the summer in Switzerland What do the records show?
WWW: Seasonal forecasts
11 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Maximum temperaturein °C
Precipitationin mm
Average sunshine durationin %
WWW: Seasonal forecasts
12 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Seasonal forecast current model run.
WWW: Seasonal forecasts
13 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
WWW: Seasonal forecasts
Terciles from station data
14 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
What is a seasonal forecast? Methodology
WWW: Seasonal forecasts
15 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Past seasonal forecasts for Switzerland Verification
WWW: Seasonal forecasts
16 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
WWW: Seasonal forecasts
observation
17 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Overview
Monthly forecasts
WWW: Seasonal forecasts
Verification: The RPSSD
Comparison:
ECMWF vs. other prediction strategies
18 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Verification of probabilistic forecasts
Real-valued observations
Probabilisticforecasts
Common approach: Convert observation into probability distribution
Ranked Probability Score (RPS)
19 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Verification of probabilistic forecasts
Real-valued observations
Probabilisticforecasts
Ensemble predictions
But: Ensemble predictions are not truly probabilistic !!
Ranked Probability Score (RPS)
20 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
0
100%CDF
C N W
Example:
Three equiprobable categories (e.g. “cold”, “normal”, “warm”)
Let the verifying observation fall into the second category
Convert real-valued observation into CDF
The Ranked Probability Score (RPS)
21 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
C N Wd1,EPS
d2,EPS Ensemble prediction system:RPS = d2
1,EPS + d 22,EPS
Example:
Three equiprobable categories (e.g. “cold”, “normal”, “warm”)
Let the verifying observation fall into the second category
Compare with CDF of ensemble forecast
The Ranked Probability Score (RPS)
22 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
C N W
1/32/3
1
d1,Cl
d2,Cl Ensemble prediction system:RPS = d2
1,EPS + d 22,EPS
Climatologic forecast:RPSCl = d2
1,Cl + d 22,Cl
Example:
Three equiprobable categories (e.g. “cold”, “normal”, “warm”)
Let the verifying observation fall into the second category
... or with CDF of climatologic forecast
The Ranked Probability Score (RPS)
23 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
The Ranked Probability Skill Score (RPSS) is defined by relating the RPS of a forecast system with the corresponding RPS of the climatologic reference:
The RPSS is negatively biased for small ensemble size !
The RPSS
24 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Synthetic random white noise forecasts, verified against random white noise observations.
Skill of this forecast system should be zero by definition !
Three equiprobable categories
The RPSS
25 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Negative bias consequence of inconsistent definition of climatologic reference forecast.
Müller et al. 2005, J.Clim.Weigel et al. 2006,
Mon. Wea. Rev.
The RPSS
1/3
26 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Solution
K: Number of forecast categoriespi: Prob. of i-th forecast categoryM: Ensemble size
General case
Weigel et al. 2006,Mon. Wea. Rev.
The RPSSD
27 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Special case 1:
K equiprobableforecast categories
M: Ensemble size
Solution
The RPSSD
Weigel et al. 2006,Mon. Wea. Rev.
28 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Special case 2:
Brier score, i.e. twocategories with probp and (1-p)
M: Ensemble size
Solution Weigel et al. 2006,Mon. Wea. Rev.
The RPSSD
29 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
The RPSSD
Synthetic random white noise forecasts, verified against random white noise observations.
Skill of this forecast system should be zero by definition !
Three equiprobable categories
30 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
ECMWF System 2 forecasts (1988-2002), verified against ERA40 T2m predictions for March, lead time 4 months 2 equiprobable forecast categories (i.e. Brier Score situation)
Southern Africa
The RPSSD
31 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Large ensembles still useful!
RPSSD determines the “true skill” of the EPS
It measures model quality, not forecast quality
Particularly useful for model assessment studies:
multi-model studies, when models of different ensemble size are to be compared
comparison of deterministic and probabilistic forecasts
The RPSSD
32 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Overview
Monthly forecasts
WWW: Seasonal forecasts
Verification: The RPSSD
Comparison:
ECMWF vs. other prediction strategies
33 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Example 1: Statistical model
Model: CCA statistical model
Training: 1880-1960
Verification: 1960-2001
Predictors: • DJF North Atlantic SST • JFMA total precipitation (north. Mediterranian)
Predictand: JJA daily homogenized Tmax station series
Reference: Della-Marta et al. (2006), Clim. Dyn.
34 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
ECMWF (DEMETER) CCA model
Example 1: Statistical model
• Three equiprobable forecast categories• JJA forecasts of T2m, initialized in May• Verification period: 1960-2001
RPSSD
35 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Example 2: The Böögg
Böögg: RPSSD = -0.15 ECMWF: RPSSD = 0.19
36 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Böögg’s Prognosis forsummer 2006:
Time until head exploded:10 minutes 28 seconds
warm summer
37 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
The Ranked Probability Score (RPS)
Example:
Three equiprobable categories (e.g. “cold”, “normal”, “warm”)
Let the verifying observation fall into the second category
C N W
Observation
38 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
The Ranked Probability Score (RPS)
Example:
Three equiprobable categories (e.g. “cold”, “normal”, “warm”)
Let the verifying observation fall into the second category
0
100%PDF
C N W
Convert real-valued observation into PDF
39 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Böögg
Time until head explodes (min)
Mea
n JJ
A t
empe
ratu
re
R2 = 0.0071p = 0.599
RPSSD = -0.15
ECMWF (DEMETER)
Ensemble mean for JJA T2m
R2 = 0.2497p = 0.0016
RPSSD = 0.19
heat summer 2003
Example 2: The Böögg
40 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification
ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006
Andreas Weigel
Central Europe
The RPSSD
ECMWF System 2 forecasts (1988-2002), verified against ERA40 T2m predictions for March, lead time 4 months 2 equiprobable forecast categories (i.e. Brier Score situation)