17th Meeting, 06/29/2009

NWS Hydrology Forecast NWS Hydrology Forecast Verification Team:Verification Team:

1717thth Meeting Meeting

06/29/2009 –2 pm EDT06/29/2009 –2 pm EDT

2 17th Meeting, 06/29/2009

OutlineOutline

• Verification in CHPS:– Demo– Next steps

• Final team report: consensus on – Recommended verification metrics and products– RFC verification case studies

Impact of QPF horizon Impact of run-time mods made on the fly

– New Verification Team charter

3 17th Meeting, 06/29/2009

Verification in CHPSVerification in CHPS• 2 classes of verification with different user requirements

– Diagnostic verification to know/improve model performance; done off-line with archived forecasts or hindcasts to compute verification statistics given different conditions (e.g., time periods, above threshold)

– Real-time verification to help forecasters make decision in real-time; done in real-time by

querying and displaying historical analogs using multiple criteria

displaying summary of past diagnostic verification statistics

checking for forecast anomalies

if necessary, run bias-correction program and display result

4 17th Meeting, 06/29/2009

Verification in CHPS: demoVerification in CHPS: demo• Demonstrate analog capability display

Select analogs from a pre-defined set of historical events and display with ‘live’ forecast in FEWS Time Series Display

• Demonstrate summary diagnostic verification displaysDisplay summary diagnostic verification products in

FEWS Map Display: map with 1 metric (e.g. Relative RMSE)

5 17th Meeting, 06/29/2009

Verification in CHPS: analog demoVerification in CHPS: analog demo• Demonstrate analog capability display w/ FEWS Time Series Display

Select analogs from a pre-defined set of historical events and display with ‘live’ forecast

Live forecast

Observations

6 17th Meeting, 06/29/2009

• Demonstrate analog capability display w/ FEWS Time Series Display Select analogs from a pre-defined set of historical events and display

with ‘live’ forecast

Verification in CHPS: analog demoVerification in CHPS: analog demo

Analog 1Analog 2Analog 3

7 17th Meeting, 06/29/2009

Verification in CHPS: analog demoVerification in CHPS: analog demo

What happened

Analog 3 Live forecast

• Demonstrate analog capability display w/ FEWS Time Series Display Select analogs from a pre-defined set of historical events and display

with ‘live’ forecast

8 17th Meeting, 06/29/2009

Verification in CHPS: analogsVerification in CHPS: analogs• Build analog query prototype using multiple criteria

Queries from forecast and/or observed attributes (e.g. forecast amount) on different time and/or space coordinates (e.g. same forecast lead time and location). Could include auxiliary variables (e.g. select flow analogs based on precipitation forcing).

Need to identify appropriate auxiliary information for selecting and interpreting the analogs (e.g. initial soil moisture conditions).

Should we just focus on a restricted range of flows, such as high flows? Of course, potential problems with sample size and uniqueness of major events.

9 17th Meeting, 06/29/2009

Examples of queries for analogsExamples of queries for analogs• Seeking analogs for precipitation: “Give me past

forecasts for the 10 largest events relative to hurricanes for this basin.”

• Seeking analogs for flow: “Give me all past forecasts for which the forecast value at lead hour 6 was within an interval [current 6-hr forecast value ± 20%] and the forecast peak occurred within the next 48 hours”.

10 17th Meeting, 06/29/2009

More complex queries for analogsMore complex queries for analogs• Seeking analogs for temperature: “Give me all past forecasts

with lead time 12 hours whose ensemble mean was within 5% of the live ensemble mean.”

• Seeking analogs for precipitation: “Give me all past forecasts with lead time 6 hours whose PoP was >=0.95, whose ensemble mean was >= 0.5 inches and for which lightening strikes were observed within a 50km radius of the forecast point at the forecast issue time.”

• Seeking analogs for flow: “Give me all past forecasts with lead times of 12-48 hours whose probability of flooding was >=0.95, where the basin-averaged soil-moisture was > x and the immediately prior observed flow exceeded y at the forecast issue time”.

11 17th Meeting, 06/29/2009

Verification in CHPS: map demoVerification in CHPS: map demo• Demonstrate summary diagnostic verification displays w/

FEWS Spatial Display Display verification map with monthly results of Relative RMSE

OctoberJanuaryApril

12 17th Meeting, 06/29/2009

Verification in CHPS: map demoVerification in CHPS: map demo• Demonstrate summary diagnostic verification displays w/

FEWS Spatial Display Display verification map with monthly results of Relative RMSE

Lead Day 1Lead Day 2Lead Day 3

13 17th Meeting, 06/29/2009

Verification in CHPS: next stepsVerification in CHPS: next steps• Perform user analysis to identify functional

requirements and meaningful verification productsTo be done by NWS Verification Team, CAT/CAT2

Teams, Graphics Generator Requirements Team, and SCHs

New RFC verification case studies to work with proposed standards

14 17th Meeting, 06/29/2009

Verification in CHPS: time lineVerification in CHPS: time line• Summer ’09 - Winter ’10

– define user requirements for CHPS VS (HEP w/ RFCs & teams)

– develop analog query prototype (wo/ database querying) (HEP w/ some RFCs)

• Winter ’10 - ’11– prepare functional/technical software design (HSEB w/ HEP)

• Continue to enhance verification science for CHPS VS (HEP)

15 17th Meeting, 06/29/2009

Final team report: consensus?Final team report: consensus?• Recommendations on

– sets of verification metrics and products to be used at all RFCs

– sensitivity analyses on impact of QPF horizon and impact of run-time mods made on the fly

• New future team activities

• Team work to be presented at the HIC meeting on 07/10– approval of second team charter?

16 17th Meeting, 06/29/2009

Sensitivity analysis: QPF horizonSensitivity analysis: QPF horizon• Goal: what is the optimized QPF horizon for hydrologic

forecasts?

• QPF horizon to test: – 0 (no QPF), 6-hr, 12-hr, 18-hr, 24-hr, 30-hr, 36-hr, 48-hr, 72-hr, 96-hr

• Model states to use: – Similar to operational mods except mods that impact future states

– Metadata to store which mods were used in these runs

• What forecast to verify– 6-hr stage forecasts for 7-day window (longer for slow response

basins)

17 17th Meeting, 06/29/2009

Sensitivity analysis: run-time MODsSensitivity analysis: run-time MODs• Goal: do run-time mods made on the fly improve forecasts?

• 4 scenarios – Operational forecasts (w/ all mods)

– Forecasts w/ best available obs. and fcst. inputs wo/ on-the-fly mods

– Forecasts w/ best available obs. inputs (no fcst) w/ all mods

– Forecasts w/ best available obs. inputs (no fcst) wo/ on-the-fly mods

• What forecast to verify– 6-hr stage forecasts for same window as in operations

• Model states: – Carryover from 5 days ago (w/ past mods) + a priori mods (known before

producing any forecast)

18 17th Meeting, 06/29/2009

Next meetingNext meeting

• 18th meeting: early September – Update on comments for team report– Discussion on

RFC verification case studies CHPS Verification Service

19 17th Meeting, 06/29/2009

Thank you!Thank you!

Questions?Questions?

20 17th Meeting, 06/29/2009

Final team report: MetricsFinal team report: Metrics• 4 different levels of information

21 17th Meeting, 06/29/2009

Final team report: AnalysesFinal team report: Analyses• Use different temporal aggregations

6-hr instantaneous flow vs. weekly minimum flow

• Avoid data pooling across different lead times Quality strongly depends on lead time Plot verification statistic as function of lead time

• Perform spatial aggregation carefully Aggregate verification results across basins with similar

hydrologic processes Analyze results for each individual basin and analyze results

plotted on verification maps

22 17th Meeting, 06/29/2009

Final team report: AnalysesFinal team report: Analyses• Analyze forecast performance

w/ time conditioning: by month, by season w/ atmospheric/hydrologic conditioning:

– low/high probability threshold

– absolute thresholds (e.g., PoP, Flood Stage)

Plot also sample size (in future confidence intervals)

• Analyze sources of uncertainty and error Verify forcing input forecasts and output forecasts For extreme events, verify both stage and flow Sensitivity analysis to be set up at all RFCs: 1) impact of

QPF horizon; 2) impact of run-time mods made on the fly