verification of rare extreme events 1.definitions and questions 2.eskdalemuir precipitation example...
TRANSCRIPT
![Page 1: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/1.jpg)
Verification of Rare Extreme Events
1. Definitions and questions
2. Eskdalemuir precipitation example
3. Results for various scores
Dr. David B. Stephenson1, Dr Barbara Casati, Dr Clive Wilson
1Department of MeteorologyUniversity of Reading
www.met.rdg.ac.uk/cag
WMO verification workshop, Montreal, 13-17 Sep 2004
![Page 2: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/2.jpg)
What is an extreme event?Different definitions:• Maxima/minima• Magnitude• Rarity• Severity
Train crash here …
“Man can believe the impossible,
but man can never believe the
improbable.” - Oscar Wilde
Gare Montparnasse, 22 October 1895
![Page 3: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/3.jpg)
What is a severe event?
Severe events (extreme loss events) caused by:
• Rare weather events
• Extreme weather events
• Clustered weather events (e.g. climate event)
Natural hazarde.g. windstorm
Damagee.g. building
Losse.g. claims ($)
Risk=p(loss)=p(hazard) X vulnerability X exposure
“Rare and Severe Events” (RSE) – Murphy, W&F, 6, 302-307 (1991)
![Page 4: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/4.jpg)
Sergeant John Finley’s tornado forecasts 1884Oldest known photograph of a tornado 28 August 188422 miles southwest of Howard, South Dakota
O=Y O=N
F=Y 28 72 100
F=N 23 2680 2703
51 2752 2803
PercentageCorrect=96.6%!!
Gilbert (1884)F=No 98.2%!!
Peirce (1884)PSS=H-F
NOAA Historic NWS Collection www.photolib.noaa.gov
![Page 5: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/5.jpg)
How to issue forecasts of rare events• Let {X=0/1} when the event/non-event occurs:0 0 0 1 1 0 0 0 0 …[probability of event p=Pr(X=1) (base rate) is small ]
• Ideally one should issue probability forecasts {f}:0.1 0.2 0.3 0.6 0.5 0.1 0.3 0.4 0.6 …
• Generally forecaster or decision-maker invokes athreshold to produce deterministic forecasts {Y=0/1}:0 0 0 1 1 0 0 0 1 …
A. Murphy, “Probabilities, Odds, and Forecasts of Rare Events”,Weather and Forecasting, Vol. 6, 302-307 (1991)
![Page 6: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/6.jpg)
Some important questions …• Which scores are the best for rare event forecasts?
PC, PSS, TS, ETS, HSS, OR, EDS
• Can rare event scores be improved by hedging?
• How much true skill is there in forecasts of extreme events?
• Are extreme events easier to forecast than small magnitude events? Does skill0 as base rate0?
• Others? Please let’s discuss them!
![Page 7: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/7.jpg)
Time series of the 6 hourly rainfall totals
1998 1999 2000 2001 2002 2003
05
15
25
pre
cip
ita
tio
n (
mm
)
time (year)
Eskdalemuir observations
1998 1999 2000 2001 2002 2003
05
15
25
pre
cip
ita
tio
n (
mm
)
time (year)
Eskdalemuir T+6 forecasts
Met Office mesoscalemodel forecasts of 6hahead 6h precipitationamounts (4x times daily)
Total sample size n=6226
![Page 8: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/8.jpg)
Scatter plot of forecasts vs. observations
0 5 10 15 20 25
05
10
15
20
25
observation (mm)
fore
ca
st (m
m)
some positive association between forecasts and observations
![Page 9: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/9.jpg)
Empirical Cumulative Distribution F(x)=1-p
0 5 10 15 20 25
0.0
0.2
0.4
0.6
0.8
1.0
Precipitation amount (mm)
1 -
ba
se r
ate
x
rate" base"
1
)Pr()(
p
p
xXxF
can use E.D.F. to map values onto probabilities (unit margins)
![Page 10: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/10.jpg)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
cumulative observation probability
cum
ula
tive
fore
cast
pro
ba
bili
ty
Scatter plot of empirical probabilities
aa
bb
cc
dd
note dependency for extreme events in top right hand cornerxp1
fp1
![Page 11: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/11.jpg)
Joint probabilities versus base rate
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
base rate
join
t p
rob
ab
ilitie
s
As base rate tends to 0, counts b=c>a0 and d1
------ a------ b=c------ d
p
![Page 12: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/12.jpg)
2x2 binary event asymptotic model
p = prob. of event being observed (base rate)B = forecast bias (B=1 for unbiased forecasts)
H = hit rate 0 as p0 (regularity of ROC curve)
so H~hpk as p0 (largest hit rates when k>0 is small)(random forecasts: H=Bp so h=B and k=1)
Obs=Yes Obs=No Marginal
Fcst=Yes a=pH b=p(B-H) a+b=pB
Fcst=No c=p(1-H) d=1-p(1+B-H) c+d=1-pB
Marginal a+c=p b+d=1-p 1
![Page 13: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/13.jpg)
Joint probabilities vs. base rate (log scale)
0.002 0.005 0.020 0.050 0.200 0.500
5
e-0
45
e
-03
5
e-0
25
e
-01
base rate
join
t pro
ba
bili
ties
note power law behaviour of a and b=c as function of base rate
------ a------ b=c------ d
![Page 14: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/14.jpg)
Hit rate as function of threshold
0 5 10 15 20
0.0
0.2
0.4
0.6
0.8
1.0
threshold (mm)
Hit r
ate
0 0.83 0.91 0.95 0.982 0.998
1 - base rate
Both Met Office and persistence have more hits than random
------ Met Office------ Persistence T+6h------ H=p random
![Page 15: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/15.jpg)
False Alarm Rate as a function of threshold
0 5 10 15 20
0.0
0.1
0.2
0.3
0.4
0.5
threshold (mm)
Fa
lse
ala
rm r
ate
0 0.83 0.91 0.95 0.982 0.998
1 - base rate
Both forecast false alarm rates converge to F=pB as p0
------ Met Office------ Persistence------ F=p random
![Page 16: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/16.jpg)
ROC curve (Hit rate vs. False Alarm rate)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
F
H
ROC curves above H=F no-skill line and converge to (0,0)
------ Met Office------ Persistence------ H=F random
Asymptotic limitAs (F,H)(0,0)
![Page 17: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/17.jpg)
Proportion correct
• perfect skill for rare events!!
• only depends on B – not on H!
pretty useless for rare event forecasts!
0 as 1)1(1~
pBp
daPC
![Page 18: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/18.jpg)
Proportion correct versus threshold
0 5 10 15 20
0.5
0.6
0.7
0.8
0.9
1.0
threshold (mm)
Pro
po
rtio
n C
orr
ect
0 0.83 0.91 0.95 0.982 0.998
1 - base rate
PC goes to 1 (perfect skill) as base rate p0
------ Met Office------ Persistence------ PC=1-2p random
![Page 19: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/19.jpg)
Peirce Skill Score (True Skill Statistic)
• tends to zero for vanishingly rare events
• equals zero for random forecasts (h=B k=1)
• when k<1, PSSH and so can be increased by overforecasting (Doswell et al. 1990, W&F, 5, 576-585.)
1for
1for )(
1for
~
kpB
kpBh
khp
Bphp
FHPSSk
k
![Page 20: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/20.jpg)
Peirce Skill Score versus threshold
0 5 10 15 20
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
threshold (mm)
Pe
irce
Skill S
co
re
0 0.83 0.91 0.95 0.982 0.998
1 - base rate
PSS tends to zero (no-skill) as base rate p0
------ Met Office------ Persistence------ PSS=p
![Page 21: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/21.jpg)
Threat Score (Gilbert Score)
• tends to zero for vanishingly rare events• depends explicitly on the bias B
(Gilbert 1884; Mason 1989; Schaefer 1990)
0 as 01
~
phpB
hp
cba
aTS
k
k
![Page 22: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/22.jpg)
0 5 10 15 20
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
threshold (mm)
Th
rea
t Sco
re
0 0.814 0.91 0.95 0.974 0.99 0.998
1 - base rate
Threat Score versus threshold
TS tends to zero (no-skill) as base rate p0
------ Met Office------ Persistence------ TS=p/2 random
![Page 23: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/23.jpg)
Brief history of threat scores• Gilbert (1884) - “ratio of verification”(=TS)
“ratio of success in forecasting”(=ETS)• Palmer and Allen (1949) - “threat score” TS• Donaldson et al. (1975) - “critical success index”(=TS)• Mason (1989) – base rate dependence of CSI(=TS)• Doswell et al. (1990) – HSS2TS/(1+TS)• Schaefer (1990) – GSS(ETS)=HSS/(2-HSS)• Stensrud and Wandishin (2000) – “correspondence ratio”
Threat score ignores counts of d and so is strongly dependent on the base rate. ETS tries to remedy this problem.
![Page 24: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/24.jpg)
Equitable threat Score (Gilbert Skill Score)
• tends to zero for vanishingly rare events• related to Peirce Skill Score and bias B
0 as 11
~
pB
PSS
pBhpB
pBhp
acba
aaETS
k
k
r
r
![Page 25: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/25.jpg)
Equitable Threat Score vs. threshold
0 5 10 15 20
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
threshold (mm)
Eq
uita
ble
Th
rea
t S
co
re
0 0.83 0.91 0.95 0.982 0.998
1 - base rate
ETS tends to zero as base rate p0 but not as fast as TS
------ Met Office------ Persistence------ ETS=p
![Page 26: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/26.jpg)
Heidke Skill Score
• tends to zero for vanishingly rare events
• advocated by Doswell et al. 1990, W&F, 5, 576-585
• ETS is a simple function of HSS and both these are related to the PSS and the bias B.
B
PSS
ETS
ETS
pBB
pBH
da
dadaHSS
rr
rr
1
2
1
2
21
)(2
1
![Page 27: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/27.jpg)
0 5 10 15 20
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
threshold (mm)
He
idke
Ski
ll S
core
0 0.814 0.91 0.95 0.974 0.99 0.998
1 - base rate
Heidke Skill Score versus threshold
HSS tends to zero (no-skill) as base rate p0
------ Met Office------ Persistence------ HSS=p
![Page 28: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/28.jpg)
Odds ratio
• tends to different values for different k
(not just 0 or 1!)• explicitly depends on bias B
1for 0
1for /
1for
~
1
1
1
k
kBh
k
pB
h
F
F
H
HOR
k
![Page 29: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/29.jpg)
Log odds ratio versus threshold
0 5 10 15 20
01
23
45
6
threshold (mm)
log
od
ds r
atio
0 0.83 0.91 0.95 0.982 0.998
1 - base rate
Odds ratio for these forecasts increases as base rate p0
------ Met Office------ Persistence------ odds=1 random
![Page 30: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/30.jpg)
Logistic ROC plot
-6 -4 -2 0 2
-6-4
-20
2
log F/(1-F)
log
H/(
1-H
)
Linear behaviour on logistic axes – power law behaviour
------ Met Office------ Persistence------ H=F random
![Page 31: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/31.jpg)
Extreme Dependency Score
• does not tend to zero for vanishingly rare events• not explicitly dependent on bias B• measure of the dependency exponent:
k=(1-EDS)/(1+EDS)
0 as 1
1~
1)log(
)log(2
pk
k
a
caEDS
S. Coles et al. (1999) Dependence measures for Extreme Value Analyses,Extremes, 2:4, 339-365.
![Page 32: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/32.jpg)
Extreme Dependency Score vs. threshold
0 5 10 15 20
0.0
0.2
0.4
0.6
0.8
1.0
threshold (mm)
Ch
i b
ar
0 0.83 0.91 0.95 0.982 0.998
1 - base rate
EDS=0.6 k=1/4
EDS=0.4 k=3/7
strikingly constant non-zero dependency as p0
------ Met Office------ Persistence------ EDS=0 random
![Page 33: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/33.jpg)
Hedging by random underforecasting
Underforecasting by random reassignment causes scores to:• Increase – proportion correct (see Gilbert 1884)• No change – odds ratio, extreme dependency score• Decrease – all other scores that have been shown
Obs=Yes Obs=No Marginal
Fcst=Yes a(1-f) b(1-f) (a+b)(1-f)
Fcst=No c+af d+bf c+d+(a+b)f
Marginal a+c=p b+d=1-p 1
![Page 34: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/34.jpg)
Hedging by random overforecasting
Overforecasting by random reassignment causes scores to:• Increase – Hit Rate, False Alarm Rate• No change – odds ratio, extreme dependency score• Decreased magnitude – PC, PSS, HSS, ETS• Other: TS?
Obs=Yes Obs=No Marginal
Fcst=Yes a+cf b+df (a+b)+f(c+d)
Fcst=No c(1-f) d(1-f) (c+d)(1-f)
Marginal a+c=p b+d=1-p 1
Compare with C. Marzban (1998), W&F, 13, 753-763.
![Page 35: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/35.jpg)
Conclusions• Which scores are the best for rare event forecasts?
EDS, Odds ratio, … (PSS,HSS,ETS0!)
• Can rare event scores be improved by hedging?Yes (so be very careful when using them!)
• How much true skill is there in forecasts of extreme events? Quite a bit!
• Are extreme events easier to forecast than small magnitude events? skill0?
Perhaps yes – there is extreme dependency
![Page 36: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/36.jpg)
Some future directions• Methods to infer rare event probability
forecasts from ensemble forecasts
• Methods to verify probabilistic rare event forecasts (not just Brier score!)
• Methods for pooling rare events to improve verification statistics
• Other?
![Page 37: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/37.jpg)
www.met.rdg.ac.uk/cag/forecasting
![Page 38: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/38.jpg)
The End
![Page 39: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/39.jpg)
2x2 table for random binary forecasts
• p = prob. of event being observed (base rate)• B = forecast bias (B=1 for unbiased forecasts)• H=Bp=F (h=B and k=1)
Obs=Yes Obs=No Marginal sum
Fcst=Yes a=p2B b= p(1-p)B a+b=pB
Fcst=No c=p-p2B d=1-p(1+B-pB) c+d=1-pB
Marginal sum a+c=p b+d=1-p 1
![Page 40: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/40.jpg)
Summary• Proportion Correct and Heidke Skill Score tend to 1 for
vanishingly rare events
• Peirce Skill Score, Threat Score and Equitable Threat Score all tend to 0 for vanishingly rare events
• All these scores can be improved by underforecasting the event (reducing B)
• There is redundancy in the scores: HSS~PC and ETS~PSS/(1+B)
• The odds ratio and Extreme Dependency Score give useful information on extreme dependency of forecasts and observations for vanishingly rare events
![Page 41: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/41.jpg)
Chi measure as function of threshold
0 5 10 15 20
0.0
0.2
0.4
0.6
0.8
1.0
threshold (mm)
Ch
i
0 0.83 0.91 0.95 0.982 0.998
1 - base rate
![Page 42: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/42.jpg)
Plan
1. Definition of an extreme event forecastBinary rare deterministic (o,p) obtainable from (x,y)Or (x,F(x)) by thresholding rx,ry or rx.
2. The Finley example and some rare event scores
3. The Eskdalemuir example – problem with scores
Some suggestions for future scores?Extremes=low skill noise OR causal events?
![Page 43: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/43.jpg)
Verification methods for rare event literature
• Gilbert (1884)• Murphy (19??)• Schaeffer (19??)• Doswell et al. (19??)• Marzban (19??)• … a few others (but not many!)
![Page 44: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/44.jpg)
Types of forecastO=observed value (predictand)F=predicted value (predictor)
Types of predictand:• Binary events (e.g. wet/dry, yes/no)• Multi-categorical events (>2 categories)• Continuous real numbers• Spatial fields etc.
Types of predictor:• F is a single value for O (deterministic/point forecast)• F is a range of values for O (interval forecast)• F is a probability distribution for O (probabilistic
forecast)
![Page 45: Verification of Rare Extreme Events 1.Definitions and questions 2.Eskdalemuir precipitation example 3.Results for various scores Dr. David B. Stephenson](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649f515503460f94c73b77/html5/thumbnails/45.jpg)
Peirce Skill Score versus threshold
0 5 10 15 20
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
threshold (mm)
Pe
irce
Skill S
co
re
0 0.83 0.91 0.95 0.982 0.998
1 - base rate
PSS tends to zero (no-skill) as base rate p0
------ Met Office------ Persistence------ PC=1-p random