tom.pagano@porda 503 414 3010
DESCRIPTION
Introduction to Forecasts and Verification. [email protected] 503 414 3010. Forecasts and decision-making What is a forecast? What kinds of forecasts exist? What makes a forecast good?. Predictions are everywhere!. Weather forecasts. Predictions are everywhere!. Weather forecasts. - PowerPoint PPT PresentationTRANSCRIPT
![Page 2: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/2.jpg)
2 of 30
Forecasts and decision-making
What is a forecast?
What kinds of forecasts exist?
What makes a forecast good?
![Page 3: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/3.jpg)
3 of 30
Predictions are everywhere!
Weather forecasts
![Page 4: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/4.jpg)
4 of 30
Predictions are everywhere!
Weather forecasts
Climate forecasts
![Page 5: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/5.jpg)
5 of 30
Predictions are everywhere!
Weather forecasts
Climate forecasts
River Forecasts
![Page 6: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/6.jpg)
6 of 30
Whether you use a “forecast” product or not, almost all resource decisions assume something
about the future (and not just for climate…)
![Page 7: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/7.jpg)
7 of 30
Whether you use a “forecast” product or not, almost all resource decisions assume something
about the future (and not just for climate…)
Population
Budgets
![Page 8: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/8.jpg)
8 of 30
The focus of today: Seasonal water supply volume forecasts(available in a variety of formats) NRCS formats:
![Page 9: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/9.jpg)
9 of 30
The focus of today: Seasonal water supply volume forecasts(available in a variety of formats) NRCS formats:
![Page 10: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/10.jpg)
10 of 30
NWS formats:
![Page 11: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/11.jpg)
11 of 30
NWS formats:
![Page 12: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/12.jpg)
12 of 30
Location
Elements of a forecast:Location, Time, Magnitude, Probability
![Page 13: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/13.jpg)
13 of 30
Location
Time Period
Historical Average
Elements of a forecast:Location, Time, Magnitude, Probability
![Page 14: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/14.jpg)
14 of 30
Location
Time Period
“Most Probable”*Water Volume
Historical Average
Error Bounds
* Term retired in 2005
Elements of a forecast:Location, Time, Magnitude, Probability
![Page 15: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/15.jpg)
15 of 30
Ways of expressing a forecast:
Categorical – “It will be warm”
Deterministic – “It will be 103% of normal”
Probabilistic – “There is a 50% chance of being more than 23 kaf”
![Page 16: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/16.jpg)
16 of 30
Ways of expressing a forecast:
Categorical – “It will be warm”
Deterministic – “It will be 103% of normal”
Probabilistic – “There is a 50% chance of being more than 23 kaf”
Ways of verifying forecasts:
Categorical – Did you predict the right category?
Deterministic – Was the observed amount very far from the forecast?
Probabilistic – Was the outcome unlikely?
![Page 17: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/17.jpg)
17 of 30
Water supply forecasts are probabilistic at their core, but most products are deterministic
![Page 18: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/18.jpg)
18 of 30
Climate forecasts are probabilistic too, but mostly they’re discussed categorically
Tilts in the odds
![Page 19: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/19.jpg)
19 of 30
What is the “Climatology” or “Equal Chances” forecast?
The safest forecast in the absence of skill or a signal…“The outcome will be between zero and infinity”
![Page 20: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/20.jpg)
20 of 30
What is the “Climatology” or “Equal Chances” forecast?
The safest forecast in the absence of skill or a signal…“The outcome will be between zero and infinity”
Try a little harder…“The range of temperatures observed on earth”
Keep trying…“The typical range of temperatures in Durango in January”
![Page 21: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/21.jpg)
21 of 30
What is the “Climatology” or “Equal Chances” forecast?
The safest forecast in the absence of skill or a signal…“The outcome will be between zero and infinity”
Try a little harder…“The range of temperatures observed on earth”
Keep trying…“The typical range of temperatures in Durango in January”
But what is “typical”?Standard definition is the 30-year normal, 1971-2000
Long enough to be stable, recent enough to be relevant
“Naïve baselines” are useful for determininghow much “better” a forecasting system is, comparatively
Interestingly, there is no “Equal Chances” forecast in hydrology…
![Page 22: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/22.jpg)
22 of 30
Resolution versus Reliability
Resolution/Sharpness: How confident are your forecasts? How much do they differ from climatology? How much do you go out on a limb?
![Page 23: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/23.jpg)
23 of 30
Resolution versus Reliability
Resolution/Sharpness: How confident are your forecasts? How much do they differ from climatology? How much do you go out on a limb?
2002 2003 2004
2005 2006 2007
2008
Feb-Apr precipitation
forecasts Issued Jan 15
Never says anything interesting
Forecast often relatively strong
![Page 24: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/24.jpg)
24 of 30
Resolution versus Reliability
Reliability: Are you going out on the right limb?When you make a statement, is it generally correct?
e.g. When you say there’s a 30% chance of rain, does it rain 30% of the time?
![Page 25: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/25.jpg)
25 of 30
Resolution versus Reliability
Reliability: Are you going out on the right limb?When you make a statement, is it generally correct?
e.g. When you say there’s a 30% chance of rain, does it rain 30% of the time?
Percent of time the observed should fall in different categories
10%20% 20% 20% 20%
10%
![Page 26: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/26.jpg)
26 of 30
Forecast “Appropriateness”
Is this the most appropriate forecast given the information at hand?
![Page 27: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/27.jpg)
27 of 30
Forecast “Appropriateness”
Is this the most appropriate forecast given the information at hand?
Year20062007
Apr 1 snow Apr-Jul obs
Animas at Durango
Apr-Jul fcst68%68%
68%89%
Fcst - Obs0%
-18%57%54%
07M31S
Was 2007 a “bad” forecast?
![Page 28: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/28.jpg)
28 of 30
Forecast “Appropriateness”
Is this the most appropriate forecast given the information at hand?
Year20062007
Apr 1 snow Apr-Jul obs
Animas at Durango
Apr-Jul fcst68%68%
68%89%
Fcst - Obs0%
-18%57%54%
07M31S
Was 2007 a “bad” forecast?
In 2007, Apr-Jun precipitation was >400% what it was in 2006.
2006 Apr-Jul precip 3rd lowest on record.
Maybe 2006 was the “bad” forecast with a lucky outcome?
![Page 29: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/29.jpg)
29 of 30
Skill and Value are different
Consider also:Timeliness
Cost to produceUnderstandability
RelevanceSpecificity
TransparencyCredibility
Ability to use
Skill is probably one of the hardest things on the list to improve
![Page 30: Tom.Pagano@porda 503 414 3010](https://reader035.vdocuments.net/reader035/viewer/2022062517/56813bee550346895da51c8a/html5/thumbnails/30.jpg)
30 of 30
Summary
3 Classes of Forecasts:Categorical, Deterministic, Probabilistic
At their core, all forecasts are probabilistic,but derived products are not.
Resolution: How often do you show up for “exams”?Reliability: What is your score on exams you do take?
Appropriateness: Are you using the best information on hand?
Skill is only one part of value and is one of the hardest things to improve