forecast verification research laurie wilson, environment canada beth ebert, bureau of meteorology...
TRANSCRIPT
Forecast Verification Research
Laurie Wilson, Environment Canada
Beth Ebert, Bureau of Meteorology
WWRP-JSC, Geneva, 17-19 July, 2013
2
Verification working group members
Beth Ebert (BOM, Australia)Laurie Wilson (CMC, Canada)• Barb Brown (NCAR, USA)• Barbara Casati (Ouranos, Canada)• Caio Coelho (CPTEC, Brazil)• Anna Ghelli (ECMWF, UK)• Martin Göber (DWD, Germany)• Simon Mason (IRI, USA)• Marion Mittermaier (Met Office, UK)• Pertti Nurmi (FMI, Finland)• Joel Stein (Météo-France)• Yuejian Zhu (NCEP, USA)
3
Aims
Verification component of WWRP, in collaboration with WGNE, WCRP, CBS
• Develop and promote new verification methods
• Training on verification methodologies
• Ensure forecast verification is relevant to users
• Encourage sharing of observational data
• Promote importance of verification as a vital part of experiments
• Promote collaboration among verification scientists, model developers and forecast providers
Relationships / collaboration
4
WGNE
SDS-WAS
S2S
Polar Prediction
WGCM
TIGGE
SWFDP
CBS operational verification
SRNWP COST-731
HIW
CIMO - SPICE
Front Page HEADLINES
• Final draft of “Verification of Tropical Cyclone Forecasts” has been released for comment and feedback.– Comments to be received until the end of August, then
document to be published by WMO.
• The science of verification methods has advanced with the publication of a special issue of Meteorological Applications, June, 2013, containing 12 papers based on presentations at the Melbourne workshop 2011.
5
6
Promotion of best practice
Verification of tropical cyclone forecasts
1. Introduction
2. Observations and analyses
3. Forecasts
4. Current practice in TC verification – deterministic forecasts
5. Current verification practice – Probabilistic forecasts and ensembles
6. Verification of monthly and seasonal tropical cyclone forecasts
7. Experimental verification methods
8. Comparing forecasts
9. Presentation of verification results
Verification of deterministic TC forecasts
7
Beyond track and intensity…
8
Model 2Model 1
Precipitation (MODE spatial method)
Track error distribution
TCgenesis
Wind speed
Verification of probabilistic TC forecasts
9
TIGGE ensembleintensity error before bias correction
After bias correction
Courtesy Yu Hui(STI)
Verification of TC seasonal frequencies
10
Met Apps special issue
11
1. Progress and challenges in forecast verification E. Ebert, L. Wilson, A. Weigel, M. Mittermaier, P. Nurmi, P. Gill, M. Göber, S. Joslyn, B. Brown, T. Fowler and A. Watkins2. A unified verification system for operational models from Regional Meteorological Centres of China Meteorological Administration Jing Chen, Yu Wang, Li Li, Bin Zhao, Fajing Chen, Yinglin Li and Yingjie Cui3. Factors affecting the quality of QPF: a multi-method verification of multi-configuration BOLAM reforecasts against MAP D-PHASE observations Marco Casaioli, Stefano Mariani, Piero Malguzzi and Antonio Speranza4. An assessment of the SEEPS and SEDI metrics for the verification of 6 h forecast precipitation accumulations Rachel North, Matthew Trueman, Marion Mittermaier and Mark J. Rodwell5. A long-term assessment of precipitation forecast skill using the Fractions Skill Score Marion Mittermaier, Nigel Roberts and Simon A. Thompson6. Using MODE to explore the spatial and temporal characteristics of cloud cover forecasts from high-resolution NWP models M. P. Mittermaier and R. Bullock7. Exploratory use of a satellite cloud mask to verify NWP models Ric Crocker and Marion Mittermaier8. A new index for the verification of accuracy and timeliness of weather warnings Laurence J. Wilson and Andrew Giles9. Expected impacts and value of improvements in weather forecasting on the road transport sector Pertti Nurmi, Adriaan Perrels and Väinö Nurmi10. Verification of marine forecasts using an objective area forecast verification system Michael A. Sharpe11. Comparative skill assessment of consensus and physically based tercile probability seasonal precipitation forecasts for Brazil Caio A. S. Coelho12. Three recommendations for evaluating climate predictions Thomas E. Fricker, Christopher A. T. Ferro and David B. Stephenson
SEDI for ECMWF vs UKMet 6 h precip forecasts
12FROM: North et al 2013
1-SEEPS for UKMet and ECMWF 6h precip fcsts
13
FROM: North et al, 2013
Comparison of physical and statistical tercile precip probability forecast accuracy for Brazil
14From: Coehlo, 2013
Uses “GeneralizedDiscrimination” score(Mason and Wiegel, 2009)
In this case, springAnd summer, theConsensus forecasts“win”
15
FDPs and RDPs
Sydney 2000 FDP
Beijing 2008 FDP/RDP
FROST-14 FDP/RDP-Participants to do own verification-JWGFVR to assist with special data and road verification-JWGFVR to establish compulsory
measures for verification
SCMREX and INCE-CE-Requests for advice, projects
starting
MAP D-PHASE
Severe Weather FDP
Typhoon Landfall FDP-Mainly training sessions at workshops so far
SNOW-V10 RDP
16
Verification of model precipitation forecasts for E. Africa
All GTS data received by ECMWF and NCEP for 2010-11 rainy season.
The full set of results is beingwritten up and will be published by WMO later this year as part of an SWFDPverification training document
RSMC chart and Hydro-est
18
Spatial Verification Method Intercomparison Project
• Falls under “promotion of best practice in verification”
• International comparison of many new spatial verification methods
• Phase 2 in planning stage
– Complex terrain
– MAP D-PHASE / COPS dataset
– Wind and precipitation, timing errors
– Case selection underway
– More information at EMS/ECAM Reading September
– Led by Eric Gilleland (NCAR)
19
Outreach and training• Verification workshops and
tutorials
– On-site, travelling
– Ensemble verification methods
(EMS/ECAM) Sept 8, Reading
– East Africa SWFDP
• EUMETCAL training modules
• SWFDP verification document
• Verification web page
• Sharing of tools
• Proposal for 6th International
Verification Methods Workshop
http://www.cawcr.gov.au/projects/verification/
Proposal for 6th International Verification Methods Workshop
• Invited by Indian Meteorological Department (IMD) and National Center for Medium Range Weather Forecasting (NCMRWF)
• March 13-19, 2014• Similar format to previous workshops (Three day tutorial
– one day off – three day science workshop)• Special emphasis on Monsoon verification and Tropical
Storm verification in both tutorial and workshop.
20
Topics for 6IVMW
• Verification of high impact weather forecasts and warnings, especially tropical cyclones and monsoon events.
• Verification of ensembles and probability forecasts • Spatial forecast verification• Seasonal forecast verification• Climate projection evaluation• Propagation of uncertainty• User issues including communicating verification to
decision makers• Verification tools
21
Seamless verification
22
Seamless forecasts - consistent across space/time scalessingle modelling system or blendedlikely to be probabilistic / ensemble
climatechange
local
point
regional
global
Sp
atia
l sca
le
forecast aggregation timeminutes hours days weeks months years decades
NWP
nowcasts
decadalprediction
seasonalprediction
sub-seasonalprediction
veryshortrange
More thoughts on seamless verification• Principles of all verification:
– Why is it being done? What does the user want to know about the forecast?
• “Attributes of forecast” – reliability, discrimination, accuracy, skill etc.
• Verification doesn’t care about the source of the forecast, or its presentation – that is decided by the forecast user (hopefully)
• It does care about the nature of the predictand (deterministic/continuous, categorical, probabilistic)
• Attributes can be measured for any forecast projection, or averaging period; the meaning is similar.
• Example: Generalized discrimination score23
Near Future Events
• WMO publication of TC document following comments• EMS one-day ensemble verification training workshop• Advice on precipitation verification metrics for WGNE (by
Sept 2013)• JWGFVR-WGTMR joint meeting (Oct 2013)• 6IVMW (March 2014)
24
Final thoughts
• “Good will” participation (beyond advice) in WWRP and THORPEX projects getting harder to provide– Videoconferencing
– Capacity building of “local” scientists
– Include verification component in funded projects
• Tendency towards “Verification within”– May be fine for research users of verification only– Not consistent with “best verification practice” when other users
are considered, e.g. SERA– SOCHI– HyMEX– A change from the original intent of JWGFVR – the
INDEPENDENT verification of products from an RDP/FDP
25
26
Thank you