National Precipitation Verification Unit

Letitia L. Soulliard

RSIS & NOAA/NWS/OCWWS

RFC Verification Workshop

August 15, 2007

Salt Lake City, UT

2

What is the NPVU?

• National• Precipitation• Verification• Unit

• Verifies the QPF from the NGM, NAM, GFS, HPC, Conus RFCs, Conus WFO’s (NDFD), and Puerto Rico (NDFD)

3

Who oversees the NPVU?

• Office of Climate, Water, and Weather ServicesPerformance Branch (Douglas Young)

• Located at :Hydrometeorological Prediction Center/NCEP (Ed

Danaher)

4

What is the History of the NPVU?

• 32km Verification – Started October 2000• Eastern RFCs

Had day 1 totals starting in August 2000 Had 6hr amounts starting October 2000

• Western RFCs CN and CB started in November 2000 NW started in January 2001

• Mosaic of all RFCs Started January 2001

• NDFD Conus WFO’s - Started October 2004 Puerto Rico – Started June 2005

• 4km Verification – Started October 2004

5

How does the NDFD relate to the RFC data?

• It doesn’t

• Different domain areas make it impossible to directly compare

• Both are individually compared to HPC and the models but not to each other

6

Why should you use the NPVU’s products?

• To see if there is an improvement over HPC’s forecast

• To see if HPC is improving upon the models• To see which model might be better for

each season or for each RFC area• To view QPF products for other RFCs and

view continuity issues

7

What are the different NPVU Products found on the web?

• QPF Verification Statistics• Comparison Plots• RFC “Live” Data• Archive Data

RFC QPEHPC QPFOffline request form

8

What is verified by the NPVU and when is it updated?

• 12Z QPF for RFCs, WFO’s and HPC• 00Z QPF for Models

• Updated Once a Month approximately around the 10th for the previous month

• E-mail notification

9

How does the data get from the RFCs to the NPVU?

RFC’s QPF & QPE

AWIPS WANNetworkControlFacility

NPVUFTPIBMSP

Offices ofSystems

Operations SBN

dbnet

10

How are the forecast and observation grids modified in order to be compared?NPVU Receives: To 32km grid by: To 4km grid by:

4km QPE from RFC Spatial Average No Conversion Needed

10km QPF from RFC Area Preservation Technique

Area Preservation Technique

1x1 Degree GFS Area Preservation Technique

Linear Interpolation

32km NAM Change coverage area Linear Interpolation

90km NGM Area Preservation Technique

Linear Interpolation

32km HPC No Conversion Needed Linear Interpolation

5km NDFD Spatial Average Area Preservation Technique

11

What statistical scores are calculated by the NPVU?

NPVUStatistics

32km / 4km

NPVUStatistics

32km / 4km

CategoricalContingency

Table

CategoricalContingency

Table

ContinuosBoth, Obs, ForContinuos

Both, Obs, For

Graphs

BiasPODFARTS

ETSThreshold Distribution

Graphs

BiasPODFARTS

ETSThreshold Distribution

Text

HSS

Text

HSS

Threshold Values (in.)

X >= 0.01X >= 0.10X >= 0.25X >= 0.50X >= 1.00

Threshold Values (in.)

X >= 0.01X >= 0.10X >= 0.25X >= 0.50X >= 1.00

Interval Values (in.)

0.00 <= X < 0.010.01 <= X < 0.100.10 <= X < 0.250.25 <= X < 0.500.50 <= X < 1.00

1.00 <= X

Interval Values (in.)

0.00 <= X < 0.010.01 <= X < 0.100.10 <= X < 0.250.25 <= X < 0.500.50 <= X < 1.00

1.00 <= X

Graphs

MEMAE

RMSEVolume Bias

Interval Distribution

Graphs

MEMAE

RMSEVolume Bias

Interval Distribution

Text

x and of For and Obs Bias of the

Envelope for CategoriesReduction of Variance

Linear CorrelationCoefficient

Text

x and of For and Obs Bias of the

Envelope for CategoriesReduction of Variance

Linear CorrelationCoefficient

12

What do all these numbers really mean?

• Online Help Guide• http://www.hpc.ncep.noaa.gov/npvu/help/

13

What is the difference between the Continuous Interval Types?• Observed – Forecast Pair

Obs: 0.2”For: 0.3”

• Observed OnlyPair is put in A

• Forecast OnlyPair is put in B

• Observed and Forecast (Both)Pair is put in A & B

B0.25 – 0.499

A0.10 – 0.2499

14

How are Categorical Scores Calculated?

• Contingency Table

• High-Low Misses

15

What are some common confusions about the NPVU statistics?

• What verification periods are covered?All verification is done at 6hr intervalsDay 1 is four 6hr periods

• Why create an interval distribution graphic?Important for identifying how significant

the statistics really are

16

How is the Interval Distribution Useful?

17

What constraints are there in the statistics?

• Small sample sizes can result in anomalous scores

• The eastern RFCs differ from the western RFCs due to Mountain Mapper

• Re-mapping can introduce small discrepancies

18

Which RFCs are considered eastern and which are western?

Eastern RFCs

ABRFCLMRFCMARFCMBRFCNCRFCNERFCOHRFCSERFCWGRFC

Western RFCs

CBRFC

CNRFC

NWRFC

APRFC

19

What are the differences between the Eastern RFCs and the Western RFCs?

Eastern RFCs

• 00Z and 12Z Forecast• 18Z forecast

AB, MA, MB and NC RFC

• Forecast out to 24hr Some forecast more (but

not verified)

• QPE are stage III multi-sensor data

Western RFCs

• 12Z forecast ONLY (due to MM restrictions)

• Forecast out to 72hr (and verified)

• QPF is made for points• QPE is gauge data• Both are gridded using

Mountain Mapper

20

What is Mountain Mapper and how does it affect the NPVU's product/process?

• Developed at CBRFC to convert point data into gridded data

• Uses PRISM precipitation Climatology• Since both QPE and QPF are run through this

they are both “biased” toward the climatology where as the models and HPC are not

21

What is the NPVU doing about this Mountain Mapper Issue?

• HPC provided forecast run through Mountain MapperFor the CNRFC, CBRFC, and NWRFC areasSince January 2003

• Statistics have been run for each month, warm and cool season, and annual

• Mountain Mapper has not improved HPC’s forecast as much as expected

22

Thank You

NPVU web site:http://www.hpc.ncep.noaa.gov/npvu

My e-mail letitia.soulliard@noaa.gov