bayesian processor of ensemble members · ensemble forecasts observation dataset and veri cation...
TRANSCRIPT
Bayesian Processor of Ensemble Members:combining the Bayesian Processor of Output with Bayesian ModelAveraging for reliable ensemble forecasting
R. Marty1,2 V. Fortin2 H. Kuswanto3
A.-C. Favre4 E. Parent5
1Universite Laval, Quebec, Canada2Environnement Canada, Dorval, Canada3Institute Technology of Sepuluh Nopember, Indonesia4Grenoble INP/LTHE, Grenoble, France5AgroParisTech/INRA, Paris, France
12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Outline
1. IntroductionEnsemble ForecastsObservation Dataset and VerificationWhy Calibrate Ensemble Predictions ?
2. Bayesian Processor of Ensemble MembersNotations and HypothesesBMA ComponentBPO ComponentCombining BPO with BMA
3. ResultsIdentification of the Optimum Training LengthVerification of Calibrated Ensemble Forecasts
4. Conclusions
2 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Ensemble Forecasts
. . . a mean to assess the uncertainty in meteorological forecasts
North American Ensemble Forecasting System
• 21 members from GEM(CMC)
• 21 members from GFS(NCEP)
• Site-located air temperaturedownscaled from 1-degreegrid
• Runs 00Z
• Lead times from +24h to+384h by 24h
• Summer 2008observations vs ensemble forecast (Quebec City)
3 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Observation Dataset and Verification
Canadian Daily Climate Data
• Temperature observation from meteorological station in JeanLesage Intl. Airport (YQB) in Quebec City over the period1978-2007
• Temperature observed at 00Z
• http://www.climat.meteo.gc.ca
Continuous Ranked Probability Score
Reliability
4 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Observation Dataset and Verification
Canadian Daily Climate Data
Continuous Ranked Probability Score
• Comparison of cumulativedistribution functions fromensemble forecasts F, withobservation y through theHeaviside function
• Negatively oriented
å smaller is better
CRPS =
∫ ∞−∞{F (u)− H(u − y)}2 du
Reliability
4 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Observation Dataset and Verification
Canadian Daily Climate Data
Continuous Ranked Probability Score
Reliability
• Statistical consistency between a priori predicted probabilitiesand a posteriori observed frequencies of the occurrence
• Reliability measured by the Reliability Component of theCRPS decomposition (Hersbach, 2000)
4 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Why Calibrate Ensemble Predictions ?
Verification of raw ensemble forecasts on Summer 2008
• GFS: lowest skill for all leadtimes
• GEM+GFS: skill closed toGEM
å No additional benefit inforecasting by using 42members
• Ensemble predictions lessskillful than climatology
• Ensemble predictions are notreliable
GEM vs GFS vs GEM+GFS vs Clim
Skill (CRPS)
Reliability (CRPS Rel)
+ (y) log scale
5 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Why Calibrate Ensemble Predictions ?
Verification of raw ensemble forecasts on Summer 2008
• Unfortunately uncertainty underestimated by current ensembleprediction systems (EPS)
• Unfortunately ensemble often provided at unsuitablespatial/temporal scales (e.g. for hydrological predictions)
5 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Why Calibrate Ensemble Predictions ?
Verification of raw ensemble forecasts on Summer 2008
• Unfortunately uncertainty underestimated by current ensembleprediction systems (EPS)
• Unfortunately ensemble often provided at unsuitablespatial/temporal scales (e.g. for hydrological predictions)
å Statistical post-processing required to obtain reliable ensembleforecasts at appropriate scales
å Here: calibration of ensemble predictions from GEM
5 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Notations and Hypotheses
Notationst valid dateS ensemble sizeh forecast’s lead timeyt predictand (quantity to predict)
X(h)t ensemble forecasts
X(h)t = {X (h)
t,s , s = 1, 2, ...,S}temporal independance: h is omitted
Assumptions
6 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Notations and Hypotheses
Notations
Assumptions
• Ensemble members generated in the same way
å exchangeability
• Numerical model well suited for predicting an unobserved(latent) variable ξt (e.g. gridded temperature)
• Latent variable ξt exchangeable with all ensemble membersXt,s
• ξt contains all the information required to predict ytå Xt and yt are conditionally independant
6 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
BMA Component (Raftery et al., 2005)
• Law of total probability
p(yt |Xt) =
∫p(yt |ξt ,Xt)p(ξt |Xt)dξt
• As Xt and yt are conditionally independant
p(yt |Xt) =
∫p(yt |ξt)p(ξt |Xt)dξt
• Since the latent variable is exchangeable with ensemblemembers
p(ξt |Xt) ≈1
S
S∑s=1
δ(ξt − Xt,s)
å BMA framework: Non-parametric approximation of thepredictive distribution
p(yt |Xt) ≈1
S
S∑s=1
p(yt |ξt = Xt,s)
7 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
BMA Component (Raftery et al., 2005)
• Law of total probability
p(yt |Xt) =
∫p(yt |ξt ,Xt)p(ξt |Xt)dξt
• As Xt and yt are conditionally independant
p(yt |Xt) =
∫p(yt |ξt)p(ξt |Xt)dξt
• Since the latent variable is exchangeable with ensemblemembers
p(ξt |Xt) ≈1
S
S∑s=1
δ(ξt − Xt,s)
å BMA framework: Non-parametric approximation of thepredictive distribution
p(yt |Xt) ≈1
S
S∑s=1
p(yt |ξt = Xt,s)
7 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
BPO Component (Krzysztofowicz, 2004)
Bayes’ rulep(yt |ξt)︸ ︷︷ ︸posterior
∝ p(ξt |yt)︸ ︷︷ ︸likelihood
p(yt)︸ ︷︷ ︸prior
Prior distribution of the predictand = climatology
• Temperature: Gaussian distribution
• Estimated from the past 30 years (1978–2007) with a movingwindow of 5 days around the valid date
p(yt) = N(yt ;µ, σ
2)
Likelihood
8 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
BPO Component (Krzysztofowicz, 2004)
Likelihood function
• Assuming linear model with Gaussian residuals between thepredictand and the latent variable
ξt = α + βyt + εt εt ∼ N (0, σ2ε|y l1)
• But ξt not observable
Xt = α + βyt + ηt ηt ∼ N (0, σ2X |y l1)
• Bayesian specification of the linear regression with informativeconjugate priors
• Less information in Xt than in ξt : σ2ε|y ∈
[0;σ2
X |y
]å Optimal variance σ2
ε|y estimated by minimizing the CRPS
8 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
BPO Component (Krzysztofowicz, 2004)
Bayes’ rule p(yt |ξt)︸ ︷︷ ︸posterior
∝ p(ξt |yt)︸ ︷︷ ︸likelihood
p(yt)︸ ︷︷ ︸prior
Prior p(yt) = N(yt ;µ, σ
2)
Likelihood p(ξt |yt) = N(ξt ;α + βyt , σ
2ε|y
)
8 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
BPO Component (Krzysztofowicz, 2004)
Bayes’ rule p(yt |ξt)︸ ︷︷ ︸posterior
∝ p(ξt |yt)︸ ︷︷ ︸likelihood
p(yt)︸ ︷︷ ︸prior
Prior p(yt) = N(yt ;µ, σ
2)
Likelihood p(ξt |yt) = N(ξt ;α + βyt , σ
2ε|y
)å BPO framework:
p(yt |ξt) = N (yt ;µy |ξ, σ2y |ξ)
µy |ξ =σ2β2( ξt−αβ ) + σ2
ε|yµ
σ2β2 + σ2ε|y
σ2y |ξ =
σ2σ2ε|y
σ2β2 + σ2ε|y
8 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Bayesian Processor of Ensemble Members
BMA
p(yt |Xt) ≈1
S
S∑s=1
p(yt |ξt = Xt,s)
BPOp(yt |ξt) = N (yt ;µy |ξ, σ
2y |ξ)
BPEM
p(yt |Xt) ≈1
S
S∑s=1
N (yt ;µy |Xt,s, σ2
y |Xt,s)
µy |Xt,s=σ2β2(
Xt,s−αβ ) + σ2
ε|yµ
σ2β2 + σ2ε|y
σ2y |Xt,s
=σ2σ2
ε|y
σ2β2 + σ2ε|y
9 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Bayesian Processor of Ensemble Members
BPEM
p(yt |Xt) ≈1
S
S∑s=1
N (yt ;µy |Xt,s, σ2
y |Xt,s)
µy |Xt,s=σ2β2(
Xt,s−αβ ) + σ2
ε|yµ
σ2β2 + σ2ε|y
σ2y |Xt,s
=σ2σ2
ε|y
σ2β2 + σ2ε|y
• µy |Xt,s: weighted average of bias-corrected member
Xt,s−αβ
and of the prior mean from climatology µ
• σ2y |Xt,s
: weighted mixture of residuals variance of the linear
model σ2ε|y and of the climatological variance σ2
9 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Bayesian Processor of Ensemble Members
• The predictive distributions’s shape depends on the empiricaldistribution of the ensemble members
å the predictive distribution is not necessary Gaussian
◦ • GEM / calibrated members Predictive dist.Observation Constituent dist.
9 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Identification of the Optimum Training Length
Summer 2008
• training period:10-days to 50-daysjoint samples
• No significant gain inforecasts’s skill beyond15 days
+ (y) log scale
• Calibrated forecasts less skillful than climatology-basedforecasts for longer-range lead times
• Optimal training length: 15 days
• Short training length: limit effects of seasonality and frequentchanges to operational forecasting systems
10 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Verification of Calibrated Ensemble Forecasts
Summer 2008 (Quebec City only)
• Optimal training length foreach calibration method
• Forecasts’ skill improved bycalibration (up to +192h)with similar pattern for eachcalibration method
• Significant improvment offorecasts’ reliability
+ (y) log scale
11 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Verification of Calibrated Ensemble Forecasts
Summer 2008 (8 sites in Quebec)+24h +96h
GEM Clim. BPEM BPO ensBMA
11 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Conclusions
Bayesian Processor of Ensemble Members. . .
. . . a new approach to calibrated ensemble forecasts
• Based on BMA and BPO frameworks
• Capable to generate reliable forecasts
• Outperforms slightly both the BMA and BPO approaches aswell as a climatology
• Short optimal training length: avoid negative impacts ofseasonality and of frequent changes to operational forecastingsystems
• Successfully applied to 7 other stations accross the Quebec
12 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France
Introduction Bayesian Processor of Ensemble Members Results Conclusions
Thanks for your attention
13 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France