Verification of Precipitation Areas

Beth EbertBureau of Meteorology Research Centre

Melbourne, Australiae.ebert@bom.gov.au

Outline

1. “Eyeball” verification - use of maps

2. QPF verification using gridpoint match-ups

3. Space-time verification of pooled data

4. Entity-based (rain “blob”) verification

5. Summary

1. “Eyeball” verification - some examples

Accumulated rain over eastern Germany and western Poland, 4-8 July 1997

WWRP Sydney 2000 Forecast Demonstration Project

RAINVAL - Operational verification of NWP QPFs

2. QPF verification using (grid)point match-ups

All verification statistics can be applied to spatial estimates when treated as a matched set of forecasts/observations at a set of individual points!

. . . .. .... . . . .. ...

. . . .. ...

. . . .. ... . . . .. ... . . . .. ...

. . . .. .... . . . .. ..

. . . .. .... . . . .. ...

. . . .. ...

. . . .. ... . . . .. ... . . . .. ...

. . . .. ... . . . .. ...

Observed Forecast

Method 1: Analyze observations onto a grid

Observed Forecast

Method 2: Interpolate model forecast to station locations

Q: Which verification approach is better?

A: It depends!

Arguments in favor of grid:• point observations may not represent rain in local area

• gridded analysis of observations better represents the grid-scale values that a model predicts

• spatially uniform sampling

Use to verify gridded forecasts

Arguments in favor of station locations:• observations are “pure” (not smoothed or interpolated)

Use to verify forecasts at point locations or sets of point locations

Note: Verification scores improve with increasing scale!

Preparation of gridded (rain gauge) verification data:

• Real time vs. non-real time

• Quality control to eliminate bad data

• Mapping procedure:– simple gridbox average– objective analysis (Barnes, statistical interpolation,

kriging, splines, etc.)

• Map observations to model grid

• Model intercomparison - map to common grid

• Uncertainty in gridbox values

Continuous statistics quantify errors in forecast rain amount

Score What it measures

Mean difference Forecast bias

Mean absolute error Average error magnitude

RMS error Error magnitude, greater emphasis tooutliers

Correlation coefficient Correspondence of forecast to observedrain pattern

Categorical statistics quantify errors in forecast rain occurrence

Score What it measures

Accuracy Correspondence of rain and no-rainareas

Bias score Tendency to under- or over-forecastrain area (occurrence)

Probability of detection Ability to predict observed rain

False alarm ratio Tendency to predict rain when/ wherenone occurred

Threat score Penalizes both misses and false alarms

Equitable threat score As above, accounts for regime

Hanssen & Kuipers (trueskill) score

Accuracy for events + accuracy fornon-events - 1

Heidke skill score Can use multiple rain categories

Verification of QPFs from NWP models

Vary rain threshold from light to heavy

Equitable threat scoreBias score

Verification of NWP QPFs over Germany

equitable threatw.r.t. chance

equitable threatw.r.t. persistence

Verification of nowcasts in Sydney 2000 FDP

—— Nowcast- - - - Persistence

3. Space-time QPF verification

(a) Pool forecasts and observations in SPACE AND TIME summary statistics

Caution: Results may mask regional and/or seasonal differences

annual

winter

summerModel performance in Australian tropics

(b) Pool forecasts and observations in SPACE but NOT TIME maps of temporal statistics

1.0-1.1

1.1-1.2

1.2-1.5

1.5-2.0

2.0-3.0

3.0-4.0

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-0.9

0.9-1.0

No data

Bias scoreJune 1995-November 1996

(c) Pool forecasts and observations in TIME but NOT SPACE time series of spatial statistics

OBSLAPS 24 hLAPS 36 hLAPS 48 h

1-30 Apr 2001Australian region

Limitations to QPF verification using (grid)point match-ups:

• Some seemingly good verification statistics may result from compensating errors

– too much rain in one part of the domain offset by too little rain in another part of the domain

– interseasonal rainfall variation captured but shorter period variation not captured

• Conservative forecasts are rewarded

• Some rain forecasts look quite good except for the location of the system; unfortunately, traditional verification statistics severely penalize these cases

4. Entity-based QPF verification (rain “blobs”)

Verify the properties of the forecast rain system against the properties of the observed rain system:

• location• rain area• rain intensity (mean, maximum)

Observed Forecast

Define a rain entity by a Contiguous Rain Area (CRA), a region bounded by a user-specified isohyet.

Some possible choices of CRA thresholds are:

1 mm d-1: ~ all rain in system5 mm d-1: “important” rain 20 mm d-1: rain center

Observed Forecast

Determining the location error:

• Horizontally translate the QPF until the total squared error between the forecast and the analysis (observations) is minimized in the shaded region.

• The displacement is the vector difference between the original and final locations of the forecast. Arrow shows optimum shift.

Observed Forecast

CRA error decomposition

The total mean squared error (MSE) can be written as:

MSEtotal = MSEdisplacement + MSEvolume + MSEpattern

The difference between the mean square error before and after translation is the contribution to total error due to displacement,

MSEdisplacement = MSEtotal – MSEshifted

The error component due to volume represents the bias in mean intensity,

where and are the CRA mean forecast and observed values after the shift.

The pattern error accounts for differences in the fine structure of the forecast and observed fields,

MSEpattern = MSEshifted - MSEvolume

2)( XFMSEvolume

XF

Example: Nowcasts from Sydney 2000 FDP

Example: Australian regional NWP model

Rain area Mean rain intensity

North of 25°S

South of 25°S

Maximum rain intensity Rain volume

Displacement error

Event forecast classification

Two most important aspects of a “useful” QPF:

• Location of predicted rain must be close to the observed location

• Predicted maximum rain rate must be “in the ballpark”

Forecast Maximum Rain Rate

TooLittle

Approx.Correct*

TooMuch

Close**Under-estimate

HitOver-estimateDisplacement

of forecastrain pattern Far

MissedEvent

MissedLocation

FalseAlarm

Example: Proposed event forecast criteria for 24h NWP QPFs

Good location: Forecast rain system must be within 2° lat/lon or one effective radius of the rain system, but not farther than 5° from the observed location

Good intensity: Maximum rain rate must be within one category of observed (using rain categories of 1-2, 2-5, 5-10, 10-25, 25-50, 50-100, 100-150, 150-200, >200 mm d-1)

Event forecast classification

Australian 24h QPFs from BoM regional model, July 1995-June 1999 (2066 events)

Error decomposition

Australian 24h QPFs from BoM regional model, July 1995-June 1999 (2066 events)

Advantages of entity-based QPF verification:

• intuitive, quantifies “eyeball” verification

• addresses location errors

• allows decomposition of total error into contributions from location, volume, and pattern errors

• rain event forecasts can be classified as "hits", "misses", etc.

• does not reward conservative forecasts

Disadvantages of entity-based verification:

• more than one way to do pattern matching (i.e., not 100% objective

• forecast must resemble observations sufficiently to enable pattern matching

5. Summary

Spatial QPF success* can be qualitatively and quantitatively measured in many ways, each of which tells only part of the story

*Note: “success” depends on the requirements of the user!!

Objective

Subjective

Point Area

(grid)point match-ups

Precision

Meaningmaps

entities