• The forecasters understood and adjusted for model uncertainty.

Models are not always accurate due to lack of observations and inadequacies of model physics.

• The forecasters estimated uncertainty at 2 stages:

BackgroundWeather forecasters examine current weather conditions upstream of the forecast area. They transform these conditions, according to the principles of atmospheric physics, into the future weather in the forecast area.

Satellite Image

MM5: Numerical Model

Previous research suggests the steps in the forecasting process are:

1. Initialize understanding of synoptic (large) scale weather

2. Build and consolidate a qualitative mental model (QMM) of the local weather

(Trafton, Kirschenbaum, Tsui, Miyamoto, Ballas & Raymond, 2000)

– From quantitative information– Cause & effect relationships of elements in atmosphere

3. Verify and adjust QMM (Pilske, Klinger, Hutton, Crandall, Knight, Klein 1997, Hoffman, 1991)

– Check against other information sources (e.g. numerical models)– Much of information gathering at this stage

4. Project forward to the time of forecast

RESULTS

There are special challenges involved in naval aviation forecasting.

PROCEDURE: Cognitive task analysis of navy forecasters• Collected think aloud verbal protocols. Subjects verbalized thoughts while completing the task.

(Ericsson & Simon, 1984)

• Recorded 4 forecasters’ verbalizations and computer screen images as they each produced a forecast, in February, 2002.

• Transcribed audio recordings. Broke down into statements and coded transcripts for information source (e.g. satellite, numerical models, radar imagery).

• We suggest that these replace building and consolidating a qualitative mental model.

Advantages: Reduces working memory load. It is a practical approach for individuals who are making rapid decisions amid constant interruptions.

Disadvantages: When there is an insufficient match between present situation and “if” condition, adjustment is difficult without the benefit of an integrated mental model.

Question: What value do the forecasters add to numerical model predictions?PROCEDURE: linear regression analysis

- Dependent variable = observed wind- Predictors = Navy forecasters predictions;

Numerical Model predictions

RESULTS:Wind Speed• Correlation between Navy Forecasters’ prediction and the observed wind speed was greater than the correlation between numerical models and observed wind speed.

Conclusions• Navy forecasters have a streamlined forecasting process.

- Use few information sources - Rely on rules of thumb

• Navy forecasters understand model dynamics.

- Evaluate model uncertainty- Use several standard evaluation techniques - Adjust the model prediction

• Navy forecasters add value to numerical model predictions.

Question: What is the impact of time pressure and information overload on weather forecasting?

RESULTS (cont.)

•The forecasters relied heavily on numerical modelsQuestion: What is the forecasters’ understanding of model dynamics?

ReferencesEriccson KA, and Simon HA. 1984. Protocol Analysis: Verbal Reports as Data. Cambridge, MA: MIT

Press.

Hoffman RR. 1991. Human factors psychology in the support of forecasting: The design of advanced meteorological workstations. Weather and Forecasting 6: 98-110.

Pilske R, Klinger D, Hutton R, Crandell B, Knight B, Klein G. 1997. Understanding skilled weather forecasting: Implications for training and design of forecasting tools. Technical Report Al/HR-Cr-1997-003.

Trafton GJ, Kirschenbaum SS, Tsui TL, Miyamoto RT, Ballas JA, Raymond PD. 2000. Turning pictures into numbers: extracting and generating information from complex visualizations. International Journal of Human-Computer Studies 53: 827-850.

Wind Direction• The human forecaster outperformed the numerical model.

Naturalistic Decision-Making: A Cognitive Task Analysis of Naval Weather ForecastingSusan Joslyn, Karla Schweitzer & Earl Hunt

University of Washington

• The variance in actual wind speed accounted for by the human forecast subsumed that accounted for by the numerical model.

Multiple R= .68

.56

.68

.39Human Forecast

ActualNumerical Model

• Statements were coded qualitative or quantitative.

• The forecasters’ mental representation was predominantly qualitative, with the exception of pressure.

Fact statements by parameter: percent qualitative versus percent quantitative

Qualitative

Quantitative

Numerical Model alone=.15

Human Forecast=.46

Observed=1

Forecaster A Forecaster B Forecaster C Forecaster D

Synoptic Scale Analysis

40%

Synoptic Scale Analysis

8%

Synoptic Scale Analysis

50%

Synoptic Scale Analysis

16%

Estimate Specific

Parameters

52%

Estimate Specific

Parameters

65%

Estimate Specific

Parameters

40%

Estimate Specific

Parameters

6%

Write Forecast

9%

Write Forecast

3%

Write Forecast

11%

Write Forecast

78%

Consolidate Mental Model

0%

Check Mental Model

38%

Goal Structure for each forecaster:

Percent of source statements under each goal

• Most of the forecasters used rule of thumb (if-then rule) reasoning.

“If I see a system coming into the coast, I forecast strong southerly winds over Whidbey Island.”

“These systems aren’t normal… it's going to act a little bit different. ” (Forecaster A)

• Forecasts must be posted every six hours while completing other tasks.

• Forecasters are interrupted as many as 12 times while writing a single forecast.

• Forecasters learn about current weather conditions from:

- Surface Charts - Satellite Imagery

- Radar Imagery • Forecasters predict future weather using current observations and numerical models:

- Numerical models are computer programs that make weather predictions which are fairly accurate for large-scale phenomena over a short period of time.

Navy forecasting office

271

2623

108

7

63

Regional Forecasts

Weather Discussion

Surface Observations

Satellite

Radar

Numerical Models

Total number of statements by source

0%

10%

20%

30%

40%

50%

60%

70%

% Closer toActual

% Correct-Variable

Human Forecast versus Numerical Model: Forecast wind direction or variable winds

Direction: % Variable: % closer correct

Human Forecast

Numerical Model

Both/Tie

Numerical Model: MM5

Satellite

Evaluation involves comparing patterns (e.g. position of low) in numerical models and satellite image (actual weather).

Is the model doing a good job?

• All of the forecasters started with the synoptic scale analysis• Most of the forecasters omitted the step of consolidating and verifying the qualitative mental model

- Only Forecaster B (the most experienced forecaster) conformed to the predicted pattern

• Most of the forecasters gathered information in the middle stages of the process

- Surprisingly, Forecaster D gathered most of his information after he began writing his forecast- A strategy to avoid maintaining large quantities of information amid constant interruptions?

This research was supported by the DOD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-10745

Estimating specific parameter values: 41% of model uncertainty statements involved checking the specific values.

Synoptic Scale Analysis: 59% of statements regarding model uncertainty were made as forecasters examined the evolution of large-scale weather patterns over time.

Example: Forecaster D compared the observed pressure to what the model had predicted.

Access current pressure 29.69

Calculate difference 29.69-29.64=.05 between current and

forecast (error)

Access predicted 29.57 pressure for forecast time

Adjust predicted 29.57-.05=29.52 pressure based on current error

Adjust predicted 29.52+.02=29.54 pressure based on model bias