statistics - definitions and issues deriving fiunbiased ...introduction o what are problems with...
Post on 15-Jul-2020
2 Views
Preview:
TRANSCRIPT
Shaocai YuShaocai Yu*,*, Brian Eder*Brian Eder*++++, Robin Dennis*, Robin Dennis*++++,,Shao-Hang Chu**, StephenShao-Hang Chu**, Stephen Schwartz Schwartz******
**Atmospheric Sciences Modeling Division, NERLAtmospheric Sciences Modeling Division, NERL** ** Office of Air Quality Planning and StandardsOffice of Air Quality Planning and Standards
U.S. EPA, RTP, NC 27711.U.S. EPA, RTP, NC 27711.
******Brookhaven National Laboratory, Upton, NY 11973Brookhaven National Laboratory, Upton, NY 11973++++ On assignment from Air Resources Laboratory, NOAA On assignment from Air Resources Laboratory, NOAA
Statistics - Definitions and IssuesStatistics - Definitions and IssuesDeriving �Unbiased Symmetric� MetricsDeriving �Unbiased Symmetric� Metrics
IntroductionIntroductionIntroduction
What are What are problems problems with commonly used metrics for model evaluationwith commonly used metrics for model evaluation??
Operational evaluation (EPA, 1991; Russell and Dennis, 2000 )Determine a model’s degree of acceptability andUsefulness for specific task
Commonly used metrics (EPA, 1991 )Difference between model and obs
Mean Bias (BMB)Mean Absolute Gross Error (EMAGE), RMSE
Relative difference (normalized by Obs) Mean Normalized Bias (BMNB)Mean Normalized Gross Error (EMNGE)
See Table 1 for other metrics
Introduction
O
IntroductionWhat are What are problems problems with commonly used metrics ? (Continued)with commonly used metrics ? (Continued)
Two problems with metrics in Table 1Asymmetry for underpredictioin and overprediction
Mean Bias: - to +∞Mean Normalized Bias, NMB: -100% to +∞%
Biased because of small numbers in the denominatorMean Normalized Bias:
Problem with Fractional BiasAgainst both Obs and ModelSeriously compressed beyond ±1 to ±2Unclear meaning: 0.60 ?
%100)1(1%100)(11
×−=×−
= ∑∑= i
iN
i i
iiMNB O
MNO
OMN
B
∑= +
−=
N
i ii
iiFB OM
OMN
B1
2)()(1
Objective
Propose new unbiased symmetric metricson the basis of concept of factor
Test new metrics and other metrics, and apply the new metrics in the CMAQ evaluation
New Metrics DescriptionNew Metrics DescriptionNormalized mean Bias Factor (Normalized mean Bias Factor (BBNMBFNMBF), Normalized mean error factor (), Normalized mean error factor (EENMEFNMEF))
Concept of Factor (symmetry)For Model>Obs (overprediction):
For Model<Obs (underprediction):Obs
ModelFactor =
ModelObsFactor =
Symmetry: overprediction and underprediction are treated proportionately
0
5
10
15
20
25
30
0 5 10 15 20 25 30
Mod
el (
NO
3- , g
m-3
)
Observation (NO3
-, g m-3)µ
µ
(1)
(2)
(3)(4)
1:1
1:2
2:1
New Metrics Description (Continued)New Metrics Description (Continued)BBNMBFNMBF and and EENMEFNMEF
Normalized Mean Bias Factor (Normalized Mean Bias Factor (BBNMBFNMBF))For (overprediction):
For (underprediction):
OM ≥
OM <
)1()1(
1
1 −=−=
∑
∑
=
=
OM
O
MB N
ii
N
ii
NMBF
)1()1(
1
1
MO
M
OB N
ii
N
ii
NMBF −=−=
∑
∑
=
=
BNMBF: symmetry , (Range) -∞ to +∞,
+ is overprediction– is underprediction
]1)ln[exp(
1
1
1 1
1 1 −−
−=
∑
∑
∑ ∑
∑ ∑
=
=
= =
= =N
ii
N
ii
N
i
N
iii
N
i
N
iii
NMBF
O
M
OM
OMB
New Metrics Description (Continued)New Metrics Description (Continued)BBNMBFNMBF and and EENMEFNMEF
Normalized Mean Error Factor (Normalized Mean Error Factor (EENMEFNMEF))For (overprediction):
For (underprediction):
OM ≥
OM <
OE
O
OME MAGE
N
ii
N
iii
NMEF =−
=
∑
∑
=
=
1
1
ME
M
OME MAGE
N
ii
N
iii
NMEF =−
=
∑
∑
=
=
1
1
ENMEF: 0 to +∞
2/]1[
1
2/]1[
1
1
1 1
11
1 1
11
)()(∑ ∑
∑∑
∑ ∑
∑∑
−=
= =
==
= =
==
−
−
−
=
+
−
−
=
=
∑∑
∑
N
i
N
iii
N
ii
N
ii
N
i
N
iii
N
ii
N
ii
OM
OM
N
ii
OM
OM
N
ii
N
iii
NMEF
MO
OME
New Metrics Description (Continued)New Metrics Description (Continued)Normalized Mean Bias Factor : Normalized Mean Bias Factor :
For (overprediction):
For (underprediction):
OM ≥
OM <
])([)(
11
11
1
1
1
i
iiN
iN
ii
iN
ii
N
iii
N
ii
N
ii
NMBF OOM
O
O
O
OM
O
MB −
=−
=−= ∑∑∑
∑
∑
∑=
==
=
=
=
])([)(
11
11
1
1
1
i
iiN
iN
ii
iN
ii
N
iii
N
ii
N
ii
NMBF MOM
M
M
M
OM
M
OB −
=−
=−= ∑∑∑
∑
∑
∑=
==
=
=
=
BNMBF: result of sum of indiv. factor bias with obs (or model) conc. as a weighting function
Unbiased: avoid undue influence of
small numbers in denominator
Test of Metrics
0
5
10
15
20
25
30
0 5 10 15 20 25 30
Mod
el (
NO
3- , g
m-3
)
Observation (NO3
-, g m-3)µ
µ
(1)
(2)
(3)(4)
1:1
1:2
2:1
Test of Metrics (Continued) :11 models from IPCC (2001) (nss-SO4
2-)3
2
1
0
A
C
3210
K 3210
H
3
2
1
03210
I
B
D
F 3
2
1
0
E
3210
J
G
Mm
odel
/(µg
m-3
)
Mobserved/(µg m-3)
Test of Metrics (Continued) :11 models for nss-SO42-
3
2
1
0
A
C
3210
K 3210
H
3
2
1
03210
I
B
D
F 3
2
1
0
E
3210
J
G
Mm
odel
/(µg
m-3
)
Mobserved/(µg m-3)
•Model H: best; Model A: worst
•Models E, G, H: acceptable
If criteria: ±25% (BNMBF), 35% (ENMEF)
Application of new Metrics for CMAQ evaluation
0
5
10
0 5 10
SO4-East (CASTNet)SO4-W est (CASTNet)SO4-east (IMPROVE)SO4-west (IMPROVE)SO4-east (STN)SO4-west (STN)
Mod
el
Observation
SO4
2- ( g m-3)µ
Jan. 8 to Feb. 18, 2002
Application of new Metrics (Continued)
Jan. 8 to Feb. 18, 2002
0
10
20
30
0 10 20 30
NO3-East (CASTNet)NO3-W est (CASTNet)NO3-east (IMPROVE)NO3-west (IMPROVE)NO3-east (STN)NO3-west (STN)
Mod
el
Observation
NO3
- ( g m-3)µ
ConclusionsNormalized mean bias factor and normalized mean error factor are proposed to quantify the relative departure between model and obs .
The newly proposed metrics are:Symmetric: overprediction and underprediction are treated proportionatelyUnbiased: avoid undue influence of small numbers in the denominator
Tests show that the newly proposed metrics are useful, their meanings are clear and easy to explain.
To represent the whole performance of the model:Mean (model, obs), r, Number, difference (BMB, EMAGE), relative difference (BNMBF, ENMEF)
Values of relative differences depend on the units of model prediction and obs !!!!
top related