the challenge to verify operational weather warnings

Eidgenössisches Departement des Innern EDIBundesamt für Meteorologie und Klimatologie MeteoSchweiz

The challenge to verify operational weather warnings

Tanja Weusthoff and Marco Arpagaus

EMS, 14.09.2011

MeteoSwiss official warnings (24h acc. precip.) for 4th September 2011

2 Verification of Weather Warnings | EMS 2011Tanja Weusthoff

Introduction 152 warning regions

5 level

8 different hazards

current project: development of an application for an automatic verification of weather warnings


Introduction

specifications: • develop an automatic verification for operational warnings,

which allows a certain tolerance (space and time)

• use synergies (e.g. with GIN, a common platform for all natural hazards in Switzerland: http://www.gin-info.ch/index.html)

motivation:• replace current manual (subjective) verification in order to

release resources

• prove the quality of official warnings

• single official voice (SOV, since January 2011) distribution via media for level 4 and 5


Challenges

• how to evaluate the usefulness of warnings without knowing the needs of individual users (and their cost/loss)

• how to interprete „tolerant results“ and what should be communicated to the users

Precautions causes Costs

Having no protection results in Losses


Challenges

• representativity of observations

• account for feedback of authorities

• small statistics (rare events)

> 2000 m

< 800 m

e.g. station Magadino / Cadenazzo (203 m asl) is only wind station for two warn regions with complex terrain (308 and 309)


Basic Concept

• event-oriented verification, warnings as binary events

• tolerant in space, time and threshold

• distinguish two types of warnings• SHORT: short-term events (e.g. thunderstorms)• AKKU: accumulated events (e.g. 24h precipitation accumulation)

• event definition and evaluation is (in principle) the same within each group; differentiate between basic verification (strict) and detailed verification

• verification per warning region, summary for specific regions or whole Switzerland


Example: AKKU

Rain level 3-5Snowfall (lowlands and mountains) level 3-5Snowmelt level 3-5Heat wave level 3

• snowmelt cannot be verified due to a lack of observations; • for snow and rain use of radar data (spatial information) and

psychrometer temperature


Event-Definition

• each warning is an event; duration of warning at least as long as accumulation period duration

• observation: consider hourly 24h, 48h and 72h sum; important is first threshold exceedance (t2)

Hit An event is observed during a valid warning and the first threshold exceedance (t2) occurs at least accumulationPeriod/3 hours (i.e. 8h,16h,24h) after the beginning of the warn event (ts) and not later then the end of the warn event (te).Miss A threshold exceedance is observed without an active warning or the first threshold exceedance occurs bevor accumulationPeriod/3 hours after the beginning of the warning (ts). False Alarm A warning has been issued, but no threshold exceedance has been observed.

AKKU

t2 - ti > tv?

tiissue time

WarnEvent (ts to te)

ts t1 t2 te

ObsEvent (t1 to t2)

acc/3


Evaluation A: Basic-verification

Present results as:

• POD

• FAR

• FBI

• (TS)

derived from contingency table

AKKU

Event observed No event

Warning issued A (Hit) B (False Alarm)

No warning C (Miss) D (Correct Rejection)

Evaluation B: detailed verification• in principle like Evaluation A (Hit, Miss, False Alarm)

• introduce additional category

• combination of Hit, Miss und False Alarm see definitions on next slide

Missing-D-problem: what is a „non-event“?


Evaluation B: detailed verification

(1.)

Specifications

Miss + Hit + False Alarm the first threshold exceedance (t2) occurs maximal tshift hours after the end of the warn event (te) or less than accumulatioPeriod/3 (8,16,24h) but more than accumulationPeriod/4 (6,12,18h) after the start of the warning (ts).

tiissue time


ts t2 te

ObsEvent (t1 to t2)

(2.)

AKKU

tiissue time


ts t1 t2te

ObsEvent (t1 to t2)

t2 - te < tshift

acc/4 acc/3


Evaluation B: detailed verification

Presentation of results:

„perfect“ hit

„useful“ combined categories including a hit

„bad“ false alarm, miss

AKKU

„good“

... adapted from DWD


Tolerance

• Allow tolerances in:• threshold: LowHit = 90% of threshold• time: variation of time components (e.g. tshift) • space: if possible, take into account neighbouring warn

regions

• apply tolerances to basic and detailed verification

• SHORT: possibility to request a minimum leadtime tv for the warning (i.e. a hit is only possible if the warning has been issued at least tv (= t1 - ti) hours before the observed event, otherwise it is classified as a miss)

SHORT + AKKU


Application Flow Diagram

obs DataBase hourly data per warn region

warn DataBase warn events

store warn and obs events on a monthly basis

store evaluation results for each event on a monthly basis

present results• per region• single event• …

Java Application AWV• derive obs events• perform event-based

verification for each setting and each region

• aggregate results and calculate scores for specific period


Observations (surface stations): 24 h acc.MeteoSwiss official warnings (24h acc. precip.)

for 4th September 2011

A qualitative example…Level 3 warning:

WarnIssue 04.09.2011 08:56

WarnStart 04.09.2011 12:00

WarnEnd 05.09.2011 12:00

24h acc. precip.

35 mm/24h

50 mm/24h

Bern

Luzern

precipitation analysis for 4th September 2011 (prel.)


Thanks for listening …

the challenge to verify operational weather warnings

Documents