Forecast Verification Research

Laurie Wilson, Environment Canada

Beth Ebert, Bureau of Meteorology

WWRP-JSC, Geneva, 17-19 July, 2013

2

Verification working group members

Beth Ebert (BOM, Australia)Laurie Wilson (CMC, Canada)• Barb Brown (NCAR, USA)• Barbara Casati (Ouranos, Canada)• Caio Coelho (CPTEC, Brazil)• Anna Ghelli (ECMWF, UK)• Martin Göber (DWD, Germany)• Simon Mason (IRI, USA)• Marion Mittermaier (Met Office, UK)• Pertti Nurmi (FMI, Finland)• Joel Stein (Météo-France)• Yuejian Zhu (NCEP, USA)

3

Aims

Verification component of WWRP, in collaboration with WGNE, WCRP, CBS

• Develop and promote new verification methods

• Training on verification methodologies

• Ensure forecast verification is relevant to users

• Encourage sharing of observational data

• Promote importance of verification as a vital part of experiments

• Promote collaboration among verification scientists, model developers and forecast providers

Relationships / collaboration

4

WGNE

SDS-WAS

S2S

Polar Prediction

WGCM

TIGGE

SWFDP

CBS operational verification

SRNWP COST-731

HIW

CIMO - SPICE

Front Page HEADLINES

• Final draft of “Verification of Tropical Cyclone Forecasts” has been released for comment and feedback.– Comments to be received until the end of August, then

document to be published by WMO.

• The science of verification methods has advanced with the publication of a special issue of Meteorological Applications, June, 2013, containing 12 papers based on presentations at the Melbourne workshop 2011.

5

6

Promotion of best practice

Verification of tropical cyclone forecasts

1. Introduction

2. Observations and analyses

3. Forecasts

4. Current practice in TC verification – deterministic forecasts

5. Current verification practice – Probabilistic forecasts and ensembles

6. Verification of monthly and seasonal tropical cyclone forecasts

7. Experimental verification methods

8. Comparing forecasts

9. Presentation of verification results

Verification of deterministic TC forecasts

7

Beyond track and intensity…

8

Model 2Model 1

Precipitation (MODE spatial method)

Track error distribution

TCgenesis

Wind speed

Verification of probabilistic TC forecasts

9

TIGGE ensembleintensity error before bias correction

After bias correction

Courtesy Yu Hui(STI)

Verification of TC seasonal frequencies

10

Met Apps special issue

11

1. Progress and challenges in forecast verification E. Ebert, L. Wilson, A. Weigel, M. Mittermaier, P. Nurmi, P. Gill, M. Göber, S. Joslyn, B. Brown, T. Fowler and A. Watkins2. A unified verification system for operational models from Regional Meteorological Centres of China Meteorological Administration Jing Chen, Yu Wang, Li Li, Bin Zhao, Fajing Chen, Yinglin Li and Yingjie Cui3. Factors affecting the quality of QPF: a multi-method verification of multi-configuration BOLAM reforecasts against MAP D-PHASE observations Marco Casaioli, Stefano Mariani, Piero Malguzzi and Antonio Speranza4. An assessment of the SEEPS and SEDI metrics for the verification of 6 h forecast precipitation accumulations Rachel North, Matthew Trueman, Marion Mittermaier and Mark J. Rodwell5. A long-term assessment of precipitation forecast skill using the Fractions Skill Score Marion Mittermaier, Nigel Roberts and Simon A. Thompson6. Using MODE to explore the spatial and temporal characteristics of cloud cover forecasts from high-resolution NWP models M. P. Mittermaier and R. Bullock7. Exploratory use of a satellite cloud mask to verify NWP models Ric Crocker and Marion Mittermaier8. A new index for the verification of accuracy and timeliness of weather warnings Laurence J. Wilson and Andrew Giles9. Expected impacts and value of improvements in weather forecasting on the road transport sector Pertti Nurmi, Adriaan Perrels and Väinö Nurmi10. Verification of marine forecasts using an objective area forecast verification system Michael A. Sharpe11. Comparative skill assessment of consensus and physically based tercile probability seasonal precipitation forecasts for Brazil Caio A. S. Coelho12. Three recommendations for evaluating climate predictions Thomas E. Fricker, Christopher A. T. Ferro and David B. Stephenson

SEDI for ECMWF vs UKMet 6 h precip forecasts

12FROM: North et al 2013

1-SEEPS for UKMet and ECMWF 6h precip fcsts

13

FROM: North et al, 2013

Comparison of physical and statistical tercile precip probability forecast accuracy for Brazil

14From: Coehlo, 2013

Uses “GeneralizedDiscrimination” score(Mason and Wiegel, 2009)

In this case, springAnd summer, theConsensus forecasts“win”

15

FDPs and RDPs

Sydney 2000 FDP

Beijing 2008 FDP/RDP

FROST-14 FDP/RDP-Participants to do own verification-JWGFVR to assist with special data and road verification-JWGFVR to establish compulsory

measures for verification

SCMREX and INCE-CE-Requests for advice, projects

starting

MAP D-PHASE

Severe Weather FDP

Typhoon Landfall FDP-Mainly training sessions at workshops so far

SNOW-V10 RDP

16

Verification of model precipitation forecasts for E. Africa

All GTS data received by ECMWF and NCEP for 2010-11 rainy season.

The full set of results is beingwritten up and will be published by WMO later this year as part of an SWFDPverification training document

RSMC chart and Hydro-est

18

Spatial Verification Method Intercomparison Project

• Falls under “promotion of best practice in verification”

• International comparison of many new spatial verification methods

• Phase 2 in planning stage

– Complex terrain

– MAP D-PHASE / COPS dataset

– Wind and precipitation, timing errors

– Case selection underway

– More information at EMS/ECAM Reading September

– Led by Eric Gilleland (NCAR)

19

Outreach and training• Verification workshops and

tutorials

– On-site, travelling

– Ensemble verification methods

(EMS/ECAM) Sept 8, Reading

– East Africa SWFDP

• EUMETCAL training modules

• SWFDP verification document

• Verification web page

• Sharing of tools

• Proposal for 6th International

Verification Methods Workshop

http://www.cawcr.gov.au/projects/verification/

Proposal for 6th International Verification Methods Workshop

• Invited by Indian Meteorological Department (IMD) and National Center for Medium Range Weather Forecasting (NCMRWF)

• March 13-19, 2014• Similar format to previous workshops (Three day tutorial

– one day off – three day science workshop)• Special emphasis on Monsoon verification and Tropical

Storm verification in both tutorial and workshop.

20

Topics for 6IVMW

• Verification of high impact weather forecasts and warnings, especially tropical cyclones and monsoon events.

• Verification of ensembles and probability forecasts • Spatial forecast verification• Seasonal forecast verification• Climate projection evaluation• Propagation of uncertainty• User issues including communicating verification to

decision makers• Verification tools

21

Seamless verification

22

Seamless forecasts - consistent across space/time scalessingle modelling system or blendedlikely to be probabilistic / ensemble

climatechange

local

point

regional

global

Sp

atia

l sca

le

forecast aggregation timeminutes hours days weeks months years decades

NWP

nowcasts

decadalprediction

seasonalprediction

sub-seasonalprediction

veryshortrange

More thoughts on seamless verification• Principles of all verification:

– Why is it being done? What does the user want to know about the forecast?

• “Attributes of forecast” – reliability, discrimination, accuracy, skill etc.

• Verification doesn’t care about the source of the forecast, or its presentation – that is decided by the forecast user (hopefully)

• It does care about the nature of the predictand (deterministic/continuous, categorical, probabilistic)

• Attributes can be measured for any forecast projection, or averaging period; the meaning is similar.

• Example: Generalized discrimination score23

Near Future Events

• WMO publication of TC document following comments• EMS one-day ensemble verification training workshop• Advice on precipitation verification metrics for WGNE (by

Sept 2013)• JWGFVR-WGTMR joint meeting (Oct 2013)• 6IVMW (March 2014)

24

Final thoughts

• “Good will” participation (beyond advice) in WWRP and THORPEX projects getting harder to provide– Videoconferencing

– Capacity building of “local” scientists

– Include verification component in funded projects

• Tendency towards “Verification within”– May be fine for research users of verification only– Not consistent with “best verification practice” when other users

are considered, e.g. SERA– SOCHI– HyMEX– A change from the original intent of JWGFVR – the

INDEPENDENT verification of products from an RDP/FDP

25

26

Thank you