pete riley1 2and leila mays - cpaess › sites › default › files › ... · currently (as of...
TRANSCRIPT
Pete Riley1 and Leila Mays2
1Predictive Science Inc
2NASA CCMC
With input from, and thanks to the CCMC CME Scoreboard teams
What is the “CME Scoreboard”
What can we say about CME-shock arrival-time forecasts? Which models/types do better?
Do we see improvements over time?
What’s the current state of accuracy and precision in the forecasts?
How does lead-time compare amongst the models?
How can we improve the forecasts in the future?
Currently (as of April 20, 2018): • 724 forecasts of CME shock arrival times
• 139 events
• 2013 - 2018
• 30+ models
WSA-ENLIL + Cone (GSFC SWRC)
WSA-ENLIL + Cone (NOAA/SWPC)
ips.gov.au
H3DMHD (HAFv.3+3DMHD)
Anemomilos
Average of all Methods
ESA
DBM
BHV
SIDC
STOA
HAFv2w
Ensemble WSA-ENLIL + Cone (GSFC SWRC)
WSA-ENLIL + Cone
Expansion Speed Prediction Model
COMESEP
SARM
SAO Crowdsource
WSA-ENLIL + Cone (Met Office)
Rice-ENLIL Dst
WSA-ENLIL + Cone (KSWC)
ElEvo
SPM2
WSA-ENLIL + Cone (BoM)
SPM
DBM + ESWF
EAM (Effective Acceleration Model)
BGS
Ooty IPS
NSSC SEPC
Other
CAT-PUMA
2014 2015 2016 2017 2018
-60
-20
02
040
60
Time (Years)D
t (h
rs)
Average of all Methods
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 60
05
10
15 Median =-3.07
St.Dev.=17.11
WEC (GSFC SWRC)
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 60
05
10
15 Median =-5.9
St.Dev.=18
SIDC
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 60
05
10
15 Median =-0.58
St.Dev.=18.51
WEC (NOAA/SWPC)
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 60
05
10
15 Median =-1.07
St.Dev.=15.52
WEC (Met Office)
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 60
05
10
15 Median =1.5
St.Dev.=23.84
Ensemble WEC (GSFC SWRC)
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 60
05
10
15 Median =-5.37
St.Dev.=17
Average of all Methods
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 60
01
23
45
6 Median =0.32
St.Dev.=12.66
WEC (GSFC SWRC)
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 60
01
23
45
6 Median =0.79
St.Dev.=13.58
SIDC
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 60
01
23
45
6 Median =-1.61
St.Dev.=14.97
WEC (NOAA/SWPC)
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 60
01
23
45
6 Median =0.975
St.Dev.=17.85
WEC (Met Office)
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 60
01
23
45
6 Median =-1.13
St.Dev.=15.22
Ensemble WEC (GSFC SWRC)
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 60
01
23
45
6 Median =-0.59
St.Dev.=14.52
Average of all Methods
Dt (hrs)
Fre
qu
en
cy
0
20
40
60
80
100
0
5
10
15 Median =NASt.Dev.=NA
WEC (GSFC SWRC)
Dt (hrs)
Fre
qu
en
cy
0
20
40
60
80
100
0
5
10
15 Median =55.9St.Dev.=33.7
SIDC
Dt (hrs)
Fre
qu
en
cy
0
20
40
60
80
100
0
5
10
15 Median =54.4St.Dev.=67.3
WEC (NOAA/SWPC)
Dt (hrs)
Fre
qu
en
cy
0
20
40
60
80
10
0
0
5
10
15 Median =52.1St.Dev.=19.5
WEC (Met Office)
Dt (hrs)
Fre
qu
en
cy
0
20
40
60
80
10
0
0
5
10
15 Median =46.4St.Dev.=22.2
Ensemble WEC (GSFC SWRC)
Dt (hrs)
Fre
qu
en
cy
0
20
40
60
80
10
0
0
5
10
15 Median =40.1St.Dev.=23.9
Average of all Methods
Dt (hrs)
Fre
qu
en
cy
0
20
40
60
80
100
0
1
2
3
4
5
6 Median =NASt.Dev.=NA
WEC (GSFC SWRC)
Dt (hrs)
Fre
qu
en
cy
0
20
40
60
80
100
0
1
2
3
4
5
6 Median =44.8St.Dev.=19.6
SIDC
Dt (hrs)
Fre
qu
en
cy
0
20
40
60
80
100
0
1
2
3
4
5
6 Median =40.5St.Dev.=14.3
WEC (NOAA/SWPC)
Dt (hrs)
Fre
qu
en
cy
0
20
40
60
80
10
0
0
1
2
3
4
5
6 Median =43.2St.Dev.=19.3
WEC (Met Office)
Dt (hrs)
Fre
qu
en
cy
0
20
40
60
80
10
0
0
1
2
3
4
5
6 Median =36St.Dev.=17.3
Ensemble WEC (GSFC SWRC)
Dt (hrs)
Fre
qu
en
cy
0
20
40
60
80
10
0
0
1
2
3
4
5
6 Median =35.7St.Dev.=22.6
2013 2014 2015 2016 2017 2018
-60
-40
-20
02
04
06
0
Time (Years)
Dt (h
rs)
2013
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 40 60
02
46
8
2014
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 40 60
02
46
8
2015
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 40 60
02
46
8
2016
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 40 60
02
46
8
2017
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 40 60
02
46
8
2018
Dt (hrs)
Fre
qu
en
cy
-60 -20 0 20 40 60
02
46
8
SWPC forecasts
0 20 40 60 80 100 120
-60
-40
-20
020
40
Lead Time (hrs)
Dt (h
rs)
0 10 20 30 40 50 60 70
-30
-20
-10
010
20
30
Lead Time (hrs)
Dt (h
rs)
WEC (GSFC SWRC)
WEC (NOAA/SWPC)
ips.gov.au
HAFv.3
Anemomilos
ESA
DBM BHV
SIDC
STOA
HAFv2w
Ensemble WEC (GSFC SWRC)
WEC
ESPM
COMESEP
SARM
SAO Crowdsource
WEC (Met Office)
Rice-ENLIL Dst
WEC (KSWC)
ElEvo
SPM2
WEC (BoM)
SPM DBM + ESWF
EAM
BGS
Ooty IPS
NSSC SEPC
Other
CAT-PUMA
Generally, the accuracy (mean error) is:
±10 hours (considering all models)
~-1 hr (for the better models)
Generally, precision (MAE) in forecast time is ±13 hours
“Average of all methods” illustrates the value of super-ensembles
SIDC, NOAA/SWPC, and the UK Met Office’s model:
Provided most accurate forecasts
Maintained the longest-running forecasts in database
Resulted in a systematic bias of ~1 hour earlier than observed
Associated with long lead times
No obvious improvement with time
Forecasts relatively consistent across wide range of methodologies
Lead times are a crucial aspect for forecast
How do we convolve them with arrival-time errors into a single metric ?
Most current forecasts driven by solar inputs (so L.T. generally high for all submissions)
Time-of-arrival is of limited intrinsic value, but as a metric it will drive performance improvements across other parameters
Uncertainties and areas for improvement in global model forecasts: Specification of ambient solar wind
Solar magnetograms used to drive global models
Specification of ejecta used to drive CME pulse
Forecaster error (bias, noise)
Limitations of this study Not the “official” forecasts from some teams (NOAA/SWPC)
No consideration of “false alarms” and “missed hits”
More sophisticated statistical analysis could be undertaken (e.g., Mann-Whitney or Wilcoxon signed rank tests to compare pairs of models)
Future Actions Recommend that official forecast centers publish forecasts in agreed-upon format
Allow new models the ability to run backtests/hindcasts against the CCMC CME Scoreboard to test novel ideas.