probabilistic forecast verification
DESCRIPTION
Probabilistic Forecast Verification. Allen Bradley IIHR Hydroscience & Engineering The University of Iowa. RFC Verification Workshop 16 August 2007 Salt Lake City. Advanced Hydrologic Prediction Service. Ensemble streamflow forecasts. Advanced Hydrologic Prediction Service. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/1.jpg)
1
Probabilistic Forecast Verification
Allen BradleyIIHR Hydroscience &
Engineering The University of Iowa
RFC Verification Workshop16 August 2007Salt Lake City
![Page 2: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/2.jpg)
2
Advanced Hydrologic Prediction Service
Ensemble streamflow forecasts
![Page 3: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/3.jpg)
3
Advanced Hydrologic Prediction Service
Ensemble streamflow forecastsMultiple forecast locations
![Page 4: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/4.jpg)
4
Advanced Hydrologic Prediction Service
Ensemble streamflow forecastsMultiple forecast locationsThroughout the United States
![Page 5: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/5.jpg)
5
Forecast Location
Forecast Date
Forecast Variable
How good are the
ensemble forecasts
produced by AHPS?
AHPS Verification
![Page 6: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/6.jpg)
6
Outline
Illustrate a consistent diagnostic framework for verification of AHPS ensemble forecastsDescribe a prototype interactive web-based system for implementing this verification framework within an RFCPresent a future vision for the role of verification archives in AHPS forecasting
6
![Page 7: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/7.jpg)
7
Forecast Verification Framework
![Page 8: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/8.jpg)
8
Perspective: Forecast Users
Evaluate the quality of forecasts at a specific location for a particular forecast variable and date
Examine one “element” in the data cube
8
![Page 9: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/9.jpg)
9
Elemental Problem
Use a distributions-oriented approach (DO) to evaluate probability forecasts for “events” defined by a flow thresholdForecast quality attributes quantified over a range of flow thresholds
9
![Page 10: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/10.jpg)
10
0
2000
4000
6000
8000
10000
Jun/14 Jul/7 Jul/31 Aug /23 Se p /15
Da
ily F
low
Vo
lum
es
(cfs
-da
ys)
D a te
D es M oines R iver
E nsem ble Stream flow P red ictions
Ensemble Forecast
![Page 11: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/11.jpg)
11
Ensemble Forecast
Probability forecast of a discrete event
C onditional D istribu tion Forecast
1 0 4
1 0 5
1 0 6
1 0 7
.01 .1 1 5 10 2030 50 7 08 0 9 0 9 5 99 9 9 .9
Pe rc e nt
D es M oines R iver
Sea
son
al F
low
Vo
lum
e (
cfs-
da
ys)
yy
ff
![Page 12: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/12.jpg)
12
Ensemble Forecast
Probability forecast of a discrete eventProbability forecasts of multicategory events
C onditional D istribu tion Forecast
1 0 4
1 0 5
1 0 6
1 0 7
.01 .1 1 5 10 2030 50 7 08 0 9 0 9 5 99 9 9 .9
Pe rc e nt
D es M oines R iver
Sea
son
al F
low
Vo
lum
e (
cfs-
da
ys)
WetWet
Near Near AvgAvgDryDry
fdry favg fwet
12
![Page 13: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/13.jpg)
13
Ensemble Forecast
Generalize by defining event forecasts as a continuous function of threshold
C onditional D istribu tion Forecast
1 0 4
1 0 5
1 0 6
1 0 7
.01 .1 1 5 10 2030 50 7 08 0 9 0 9 5 99 9 9 .9
Pe rc e nt
D es M oines R iver
Sea
son
al F
low
Vo
lum
e (
cfs-
da
ys)
f0.50
yy0.500.50
yy0.750.75
f0.75
yy0.250.25
f0.25
Index function by the threshold’s climatological probability
13
![Page 14: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/14.jpg)
14
Ensemble Forecast Verification
Forecast y<y p?
Date f x
1949/09 0.805 1
1950/09 0.952 1
1951/09 0.128 0: : :
1964/09 0.804 0
1965/09 0.732 01966/09 0.962 1
: : :1999/09 0.365 02000/09 0.130 1
Compute forecast-observations pairs for specific thresholds yp
Evaluate forecast quality for a range of thresholds yp
![Page 15: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/15.jpg)
15
Des Moines River near Stratford
Standard Errors
Skill Skill depends on the thresholdUncertainty is greater for extremes
April 1st Forecasts
15
![Page 16: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/16.jpg)
16
Distributions-Oriented Measures
Skill Score Decomposition:
(SS)Skill
22
22
x
xf
x
ffxfxMSESS
(RES)Resolution
(CB)Conditional
Bias
(UB)Unconditional
Bias
SlopeReliability
StandardizedMean Error
PotentialSkill
![Page 17: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/17.jpg)
17
-0 . 6
-0 . 4
-0 . 2
0
0 . 2
0 . 4
0 . 6
0 . 8
1
0 0 . 2 0 . 4 0 . 6 0 . 8 1
N o n e x c e e d a n c e p r o b a b i li ty ( )p
D e s M o i n e s R i v e r a t J a c k s o n ( J C K M 5 )
S S
P S
S R E L
S M E
April 1st
Implications for Verification
IncreaseProbability
forecast skill
Eliminatewith bias-correction
Minimum 7-Day Flow
SSRESCBUB
17
![Page 18: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/18.jpg)
18
Perspective: NWS RFC Forecaster
Assess the overall performance of the forecasting system Diagnose attributes limiting forecast skill (e.g., biases)
Examine “slices” and “blocks” of the data cube
18
![Page 19: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/19.jpg)
19
Multidimensional Problem
The forecaster needs summary verification measures suitable for comparing forecasts at different locations and/or forecasts issued on different datesSummary measures describe attributes of the skill functions derived from the elemental verification problem
19
![Page 20: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/20.jpg)
20
Summary Verification Measures
Ranked Probability Skill Score (RPSS):
MSEiMSEi
i SSpSSwRPSS )(
Weighted-average skill over probability thresholds
iii
iii pp
ppw
)1(
)1(
![Page 21: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/21.jpg)
21
Summary Verification Measures
Skill RPSS shows average skillCenter of mass shows asymmetries in the skill function
RPSS
Center ofMass
21
![Page 22: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/22.jpg)
22
Hypothetical Skill Functions
All skill functions have same average skill
22
![Page 23: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/23.jpg)
23
Hypothetical Skill Functions
All skill functions have same average skillSecond central moment shows shape
23
![Page 24: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/24.jpg)
24
Hypothetical Skill Functions
All skill functions have same average skillSecond central moment shows shape
24
![Page 25: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/25.jpg)
25
NCRFC Forecasts
7-day minimum flow forecasts for mainstem locations for three rivers
MinnesotaRiver (MIN)
Des MoinesRiver (DES)
Rock River (RCK)
25
![Page 26: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/26.jpg)
26
Forecast Skill Attributes
Forecasts made at the 1st of the month
26
![Page 27: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/27.jpg)
27
Forecast Skill Attributes
Average skill is highest for DESThe skill function is peaked in the middle
27
![Page 28: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/28.jpg)
28
Summary Measure Decomposition
Skill Score Decomposition:
Skill
)()()()( iiii pUBpCBpRESpSS
Resolution ConditionalBias
UnconditionalBias
UBCBRESSSRPSS
Weighted-average measures of resolution and biases
![Page 29: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/29.jpg)
29
AHPS Minimum 7-Day Flow
A single MIN site has large biases for low flowsThe largest biases for other sites centered on higher flows
29
![Page 30: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/30.jpg)
30
Forecast Bias Attributes
Unconditionalbias is dominateSimple bias-correction can significantly improve forecasts
Simple bias-correction
Post-hoccalibration
30
![Page 31: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/31.jpg)
31
Verification Framework
Forecast quality for ensemble forecasts (e.g., skill) is a continuous function of the forecast outcome (or its climatological probability)Summary measures can be interpreted as measures of the “geometric shape” of the forecast quality functionThis interpretation provides a framework for concisely summarizing the attributes of ensemble forecasts
31
![Page 32: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/32.jpg)
32
AHPSVerification
System
![Page 33: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/33.jpg)
33
AHPS Verification System
Web-based tools for online
access, analysis, and comparison of retrospective
AHPS forecasts for River Forecast
Centers (RFCs)http://www.iihr.uiowa.edu/ahps_ver
![Page 34: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/34.jpg)
34
Map-Based Navigation
2
1
3
![Page 35: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/35.jpg)
35
Verification Data Archive
35
![Page 36: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/36.jpg)
36
0
2000
4000
6000
8000
10000
Jun/14 Jul/7 Jul/31 Aug /23 Se p /15
Da
ily F
low
Vo
lum
es
(cfs
-da
ys)
D a te
D es M oines R iver
E nsem ble Stream flow P red ictions
Verification Data Archive
Retrospective forecasts for a 50-year period
36
![Page 37: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/37.jpg)
37
Verification Data Archive
Retrospective forecasts for a 50-year periodProcessed ensemble forecasts & observations
C onditional D istribu tion Forecast
1 0 4
1 0 5
1 0 6
1 0 7
.01 .1 1 5 10 2030 50 7 08 0 9 0 9 5 99 9 9 .9
Pe rc e nt
D es M oines R iver
Sea
son
al F
low
Vo
lum
e (
cfs-
da
ys)
37
![Page 38: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/38.jpg)
38
Verification Data Archive
Retrospective forecasts for a 50-year periodProcessed ensemble forecasts & observationsVerification results
38
![Page 39: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/39.jpg)
39
Verification System ConceptsRetrospective ensemble traces available in their native format (*.VS files)Processed ensemble forecasts & observations for a suite of variables
Uses *.qme files from the calb systemForecast quality measures based on the ensemble forecasts
39
![Page 40: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/40.jpg)
40
Disk Requirements
• 6 Forecast periods per month (72 per year)• All segments have 50 years observed record
1 600Segment Segments
Elements (MB) (GB)Ensemble Traces (*.VS) 96.8 58.1Ensemble forecasts/obs 91.3 54.8Verification measures 89.3 53.6
Total Disk Usage 277.4 166.5
![Page 41: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/41.jpg)
41
Advantages
Interactive exploration of verification results
Provides a diagnostic “report card” for sites within an RFC
Instant access to forecasts and quality measures for verification sitesSeamless integration with other components of the NWSRFS system
41
![Page 42: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/42.jpg)
42
A Vision for theFuture
![Page 43: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/43.jpg)
43
Vision
Generation and archival of retrospective forecasts will be a routine component of forecasting systems
Verification methods can assess qualityVerification results would form the basis for accepting (or rejecting) proposed improvements to the forecasting systemArchival information will form the basis for generating improved forecast products
![Page 44: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/44.jpg)
44
Product Generation with ArchiveRaw ESP forecastArchive verification indicates biases and skillOptimal merging and bias correctionEnsemble Forecast
VerificationArchive
OptimizedCS
![Page 45: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/45.jpg)
45
Conclusions
![Page 46: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/46.jpg)
46
ConclusionsA consistent verification framework provides both users and forecasters with the means evaluating forecast products (exploring the “data cube”)AHPS-VS integrates retrospective forecast generation and forecast verification within the operational setting of an RFCRetrospective forecast archives will become a routine component of a hydrologic forecasting system, enhancing forecast evaluation and product generation
![Page 47: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/47.jpg)
47
![Page 48: Probabilistic Forecast Verification](https://reader036.vdocuments.net/reader036/viewer/2022062315/5681584e550346895dc5ab5d/html5/thumbnails/48.jpg)
48
Des Moines Forecast Skill
Skill is higher (lower) downstream (upstream)Skill decline from April to June
JCKM5
DESI4
STRI4
TotalBias