rtma/urma, nws’ mesoscale analysis system, with analysis

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

1

Verification and Validation (V&V) of Forecasters’ Needs for RTMA/URMA, NWS’ Mesoscale Analysis System,

with Analysis and Nowcast V&V FrameworkYoung-Joon (YJ) Kim1, Tatiana Gonzalez1, Eric Guillot1,

Bradley Brokamp2,3 and Mark Tew2

Analysis and Nowcast Branch (AFS11)1

Analysis and Mission Support Division2

Analyze, Forecast, and Support OfficePathways Internship Program3

NWS/NOAASilver Spring, MD

July 28, 2020

2020 UFS Users’ WorkshopVerification, Evaluation, and Post Processing (Parallel Session 1)


2

❏ In order to address the needs of NWS forecasters for improving Analysis and Nowcast systems, we (Analysis and Nowcast Branch of AFS) are collecting field’s input by various means, ranging from community web fora to direct field surveys.

❏ The collected information indicates that the RTMA/URMA (Real-Time Mesoscale Analysis/UnRestricted Mesoscale Analysis) system reveals systematic biases, especially over complex terrain, and urgently need to be improved.

❏ In order to identify the systematic biases and develop the requirements for RTMA/URMA, we are analyzing outstanding weather events reported by the field utilizing the Analysis and Nowcast V&V Framework.

Introduction


3

RTMA/URMA VLab Forum Feedback Statistics(Nov. 2016 ~ Present, Over 100 Postings, Processed by Eric Guillot)

* not a true NWS Mission Service Area

Posts by RegionWestern 49.5%Central 26.2%

Southern 13.6%Eastern 6.8%

HQ 1.9%Alaska 1.0%NCEP 1.0%

Posts by NWS Mission Service AreaPublic Weather 36.4%

Mountain Weather* 24.3%Marine and Coastal Weather 16.8%

Aviation Weather 12.1%Fire Weather 4.6%

Water Resources 3.5%Winter Weather 1.7%

Tropical Weather 0.6%

Posts by Phenomenon/IssueRTMA Quality Control 25.5%

Cold Pools 16.1%Wind 15.3%

RTMA Station Flagging 10.9%

Lakes 11.7%Ocean 7.3%Fires 5.1%

Precipitation 4.4%Clouds 2.2%

Fog 0.7%Hurricane 0.7%

Posts by RTMA/URMA Parameter

Temperature 36.6%Terrain* 29.0%

Wind 15.2%Water Temp 6.2%

Moisture 4.8%QPE 4.1%

Sky Cover 2.1%Station

Locations* 1.4%Visibility 0.7%

More issues are reported in WR (Western Region) and CR (Central Region), over areas of large terrain variations (mountains, valleys, coasts & islands) for temperatures and winds, and are largely associated with observation data quality control & station flagging.

*Most reports are for Version 2.7. New version (v2.8) scheduled to be implemented in July 2020.

+ not an explicit Parameter

FY21 AFS/PMO AOP AFS11 Presentation (3/4/2020)


4

AFS11 RTMA/URMA Survey(5/7-6/12/2020, 172 Submissions, Processed by Eric Guillot)

RTMA/URMA Difficulty Capturing PhenomenonCold pools in complex terrain (e.g.,

mountains, coastlines, islands) 57.0%

High wind events in complex terrain 55.2%Precipitation in complex terrain 42.4%Other (mostly "I don't know" or

"marine/ocean phenomena") 32.0%

High wind events in flat terrain 29.7%Moisture anomalies 27.9%

Ocean and/or lake temperatures 25.0%

RTMA/URMA Technical ProblemsQuality Control (e.g., good data getting thrown out, bad data being assimilated) 58.1%

Data Influence (e.g., station data does not modify the background field enough) 49.4%

Error in Model Background (e.g., NWP models are not accurate enough) 48.8%

Lack of Data (e.g., surface obs datasets not available to be ingested) 41.9%

Functionality (e.g., ability to specify which data is used/unused in analysis) 37.8%

RTMA/URMA Parameters Not Adequate for OpsWind Speed 55.2%Wind Gust 53.5%Sky Cover 32.6%

Min Temperature 31.4%Ceiling 29.7%

Visibility 29.1%QPF 29.1%

Hourly Temperature 28.5%Dew Point Temperature /

Relative Humidity 28.5%Max Temperature 26.7%

● Winds and temperatures (collectively) are reported as the most inadequate RTMA/URMA parameters.

● Errors in quality control and model background, and lack of data are reported as major technical problems.

● Cold pools, high wind and precipitation events in complex terrain are reported as the most difficult phenomena to capture.


5

plus

Source: METviewer TrainingOct. 2018

(Model Evaluation Tools)

Data covering Outstanding Weather Events in the field

Multi-Scale Common Environmental Data

(For Consistent V&V)

Less Rejected Observations through better Quality Control based on Field Input

Selected set of Observations & Analyses

UFS METplus Verification Framework

V&V (Verification & Validation) with AN Framework coupled with UFS METplus (Model Evaluation Tools plus) Framework

Analysis and Nowcast V&V Framework

Common Env Data

RTMA/URMA

V&VTestbed

with Common ICs & BCs (Lateral, Lower & Upper)

NWP (e.g., HRRR)


6

❏ Forecaster at WFO Pueblo, CO, noted that RTMA/URMA low temperatures were too warm in the Rio Grande River Valley in the mornings of 12/1/19.

❏ Temperatures are warmer than station observations in the river valley by as much as 20 deg F!

❏ This case highlights the RTMA/URMA biases in complex terrain, particularly with mountain valley temperatures/cold pools.

Case 1: Cold Pool over Complex TerrainKathleen Torgerson @ WFO Pueblo, CO (CR): Rio Grande Valley, am 12/1/2019

Cold Pool generated by Valley Effect

METAR station (Wolf Creek Pass - KCPW) located near the Rio Grande Valley in Pagosa Springs, CO.

This station’s elevation is nearly 12,000 feet.


7

Temperature Tolerances Number Percentage2 to < 3 deg F 74 43.0%1 to < 2 deg F 66 38.4%

Other (mostly "0" or "depends on the type of sensor") 18 10.5%

3 to < 4 deg F 10 5.8%4 to < 5 deg F 3 1.7%5 to < 6 deg F 1 0.6%

AFS11 Survey on RTMA/URMA

Rio Grande ValleyKCPW

KGUC

KALS

K04V

Case 1: Cold Pool over Complex TerrainKathleen Torgerson @ WFO Pueblo, CO (CR): Rio Grande Valley, am 12/1/2019

URMA MeanMETAR Mean

The cold pool represented in some METAR station data (and also WxUnderground sites) is not represented by URMA mean (nor by METAR mean). Possible reasons include inaccurate model background (HRRR/RAP) and/or inadequate QC, which provides under-represented verification for RTMA/URMA.

cold pool reported


8

❏ Forecaster at WFO Portland, OR, noted that RTMA/URMA wind speeds were too low in the Columbia River Gorge on 10/29/19.

❏ Wind speeds at the Portland International Airport were sustained at 29 knots (~15 m/s), while RTMA/URMA values were 20-25 knots (~10-13 m/s) too low!

❏ This case highlights the RTMA/URMA biases in complex terrain, particularly with winds associated with mountain gap flow.

Case 2: Wind Gust over Complex Terrain Dan Miller @ WFO Portland, OR (WR): Columbia River Gorge, am 10/29/2019

KPDX 45.6 -122.6

Portland International Airport (KPDX) relative to the

Columbia River in Portland, OR. The river flows east to

west, parallel with the airport. Strong easterly winds blow during gap flow events.

Gap Flow generated by Directional Channeling


9

AFS11 Survey on RTMA/URMA

Wind Tolerances Number Percentage3 to < 5 knots (1.5-2.6 m/s) 75 43.6%1 to < 3 knots (0.5-1.5 m/s) 74 43.0%Other (mostly "0" or "N/A") 14 8.1%

5 to < 7 knots 8 4.7%7 to < 9 knots 0 0.0%9 to < 11 knots 1 0.6%

Columbia River Gorge

KDPX

KTTD

KSPB

KHIO

KUAO

Case 2: Wind Gust over Complex Terrain Dan Miller @ WFO Portland, OR (WR): Columbia River Gorge, am 10/29/2019

URMA Mean

METAR Mean

The wind gust represented in some METAR station data is not represented in URMA mean (nor by METAR mean). Possible reasons include inaccurate model background (HRRR/RAP) and/or inadequate QC, which provides under-represented verification for RTMA/URMA.


10

Conclusions

❏ We performed the V&V of the field’s needs on RTMA/URMA utilizing the Analysis & Nowcast V&V Framework for sample cases found from the vLab Forum.

❏ We presented example cases of systematic biases over complex terrain involving cold pool and wind gust events reported by the field.

❏ The biases are considered to be due to a combination of inaccurate model background and/or improper observation data quality control.

❏ We are developing “operational requirements” based on the V&V of these and other cases as well as gap analysis of survey results, which will be validated through the NWS Governance process.

❏ Developed requirements will be delivered to the developers for alleviating the systematic biases of RTMA/URMA.


11

Are these effects fully represented in current model physics, downscaling, & error estimation & observations quality control algorithms, meeting the field’s needs?

Terrain of large variation (e.g., mountains) can have significant effect on temperature and humidity as well on winds.

Gap Analysis: Complex Physical Mechanisms of Flow around Topography - To Be Represented in the Analysis

Wave Breaking

Low-level Flow

Blocking

Windstorm

GapFlowWind

Gust

Wind Stagnation

Wind Stagnation

Valley Effect (Cold Pool)

rtma/urma, nws’ mesoscale analysis system, with analysis

Documents