NEW VERIFICATION APPROACHES FOR CONVECTIVE WEATHER FORECASTS

Barbara G. Brown*, Randy R. Bullock, Christopher A. Davis, John Halley Gotway, Michael B. Chapman, Agnes Takacs, Eric Gilleland, Kevin Manning

National Center for Atmospheric Research, Boulder, Colorado

Jennifer L. MahoneyNOAA Research-Forecast Systems Laboratory, Boulder, Colorado

9.4

1. INTRODUCTION

In recent years, the need for alternative veri-fication approaches to evaluate convective fore-casts has become apparent. In particular, theaviation weather community has a strong need forverification measures that are more clearly tied tothe operational usefulness of the forecasts and areable to diagnose the sources of the errors in theforecasts (e.g., Brown et al. 2002). The need forthese alternative approaches also extends to fore-casts of more standard weather elements (e.g.,winds, precipitation) produced by mesoscalenumerical weather prediction (NWP) models,where standard verification approaches penalizethe models’ abilities to capture finer scale features.These penalty effects have been documented, forexample, by Baldwin (2003), Rife et al. (2004), andothers (e.g., Mass et al. 2002). Developing verifica-tion methods that satisfy the needs for operation-ally relevant and diagnostic approaches will bebeneficial for managers who need to understandthe benefits and usefulness of particular types offorecasts; for operational users of the forecastswho need to understand how to make use of theforecasts; and for forecast developers who need toknow how their forecasts can/should be improved.

In response to these needs, several effortsare in progress to develop verification approachesthat are more diagnostic. Development of theseapproaches will eventually lead to verification mea-sures that are operationally meaningful in the con-text of specific forecast users, such as air traffic orwater managers. Some developments in this areaare described in this paper, along with examples ofapplications to several types of forecasts that haverelevance for aviation forecast developers anddecision makers. The focus here is on quantitativeprecipitation forecasts (QPFs) and forecasts ofconvection.

The methodology considered here is basedon an “object-oriented” approach to verification.With the object-oriented approach under develop-ment, forecast and observed precipitation/convec-tive areas are reduced to regions of interest thatcan be compared to one another in a meaningfulway. In the case of most human-generated fore-casts, the forecast objects are pre-defined by theforecaster(s). In contrast, gridded forecasts, whichare the norm for forecasts produced by NWP mod-els, must be converted to objects.

Following a short discussion of the verifica-tion problem in Section 2, a general description ofthe verification approach is presented in Section 3.Several examples of applications of the approachare presented in Section 4, and future work on thistopic is considered in Section 5. Some conclusionsare presented in the final section.

2. MOTIVATION

Standard approaches for verification of spa-tial QPFs and convective forecasts are based onsimple grid overlays in which the forecast grid ismatched to an observation grid or set of observa-tion points. For Yes/No forecasts of convectiveweather, the forecast/observation (Yes/No) pairsare counted, to complete the standard 2x2 verifica-tion contingency table. The counts in this table canbe used to compute a variety of verification mea-sures and skill scores, such as the Probability ofDetection (POD), False Alarm Ratio (FAR), CriticalSuccess Index (CSI), and Equitable Threat Score(ETS) (e.g., Doswell et al. 1990; Wilks 1995). Forcontinuous variables (e.g., precipitation amount),the forecasts and observations are used to esti-mate the mean squared error (MSE), mean abso-lute error (MAE), and other standard verificationmeasures for continuous variables (e.g., Wilks1995; Jolliffe and Stephenson 2003).

An important concern is that this approachdoes not provide information needed to diagnoseparticular problems with the forecasts, to reveal thesteps that can be taken to improve the forecasts orto give meaningful guidance to forecast users.

*Corresponding author address: Barbara G. Brown, NCAR, PO Box 3000, Boulder CO 80307-3000; e-mail: bgb@ucar.edu

Uncertainty and scaling issues associated with theobservations also are of importance (e.g., Casati etal. 2004), but are beyond the scope of this paper;characteristics of convective observations aretreated more fully by Mahoney et al. (2004a,b).

Figure 1 illustrates some of the difficultiesassociated with diagnosing forecast problemsusing standard verification statistics. This figureshows five examples of forecast/observation pairs,with the forecasts and observations represented asareas. For a forecast user, cases a-d clearly dem-onstrate four different types or levels of "good-ness": (a) appears to be a fairly good forecast, justoffset somewhat to the right; (b) is a poorer fore-cast since the location error is much larger than for(a); (c) is a case where the forecast area is muchtoo large and is offset to the right; (d) shows a situ-ation where the forecast is both offset and has thewrong shape. Of the four examples, it appears thatcase (a) is the "best." Given the perceived differ-ences in performance, it is dismaying to note thatall of the first four examples have identical basicverification statistics (POD=0, FAR=1, CSI=0) indi-cating no skill. Thus, the verification approach isinsensitive to differences in location and shapeerrors. Similar insensitivity could be shown to beassociated with timing errors. Moreover, example(e) - which could be considered a very poor fore-cast from a variety of points of view - actually hassome skill (POD, CSI >0), suggesting it is a betterforecast than the one depicted in example (a).

These examples demonstrate that it is diffi-cult to diagnose some of the sources of error in aforecast using traditional verification statisticsalone. While these measures may be useful forproviding an aggregate measure of overall perfor-mance or for tracking performance over time, theytell us little about the sources of errors or theiroperational impacts. For example, each type oferror illustrated in Fig. 1 would lead to different air-craft route impacts, which cannot be distinguishedby examining the overall verification statistics. Thegoal of the work described here is to develop a ver-ification approach that will clearly distinguish thetypes of errors associated with a forecast, includingforecasts like those illustrated in Fig. 1.

3. ALTERNATIVE APPROACHES

3.1 Background

The general goal of the object-oriented verifi-cation approach is to reduce the forecast andobserved convection/precipitation areas to regions

of interest that can be compared to one another ina meaningful way. Ebert and McBride (2000) had asimilar goal in development of their entity-basedverification approach, but they approached theproblem from a somewhat different point of view.The main goals of the Ebert-McBride approach areto identify displacement vectors and to decomposethe error statistics into their sources (e.g., displace-ment errors, pattern errors). An approach devel-oped by Hoffman et al. (1995) also focused ondecomposing the forecast error according to vari-ous attributes.

In general, the object-oriented approachrequires several steps:• Objects are defined in the observation and

forecast fields;• Adjacent objects or objects in close proximity

that naturally “belong” together are merged into composite objects;

• Forecast and observed objects are optimally matched to each other;

• Differences between matched forecast and observed objects are computed for all rele-vant attributes; other verification measures (e.g., the size of the intersection area) are also computed, as well as measures that are meaningful for forecast developers and oper-ational forecast users;

• Verification measures are aggregated across a variety of forecasts to reveal any system-atic errors and to understand the overall per-formance of the forecasts.

Specific details of the application of the object-ori-ented approach depend on the characteristics of

O F O F

O F O F

FO

O F O F

O F O F

O FO F O FO F

O FO F O FO F

FO FO

(a) (b)

(c)

(d)

(e)

O F O F

O F O F

FO

O F O F

O F O F

O FO F O FO F

O FO F O FO F

FO FO

(a) (b)

(c)

(d)

(e)

FIGURE 1. Schematic example of various forecast (F) and observation (O) combinations.

the forecasts and observations. For example, inthe case of forecasts that are already defined asshapes (e.g., human-generated forecasts), theforecast objects essentially are pre-defined and itis only necessary to define the observed objects. Insome cases the observed objects may also be pre-defined.

More details about the application of theobject-oriented approach are presented in the fol-lowing sections. Specific information for particulartypes of forecasts and observations is also pro-vided.

3.2 Gridded forecasts and observations

As suggested in the previous section, themore complex application of the technique involvessituations where both the forecasts and observa-tions are represented by a grid. In this case,objects must be defined for both the forecast gridand the observation grid. However, it is importantto remember that all spatial shapes on a map canbe represented as a gridded field. Thus, the “grid-ded forecasts and observations” situation is themost generic application of the approach.

3.2.1 Characterizing areas of precipitation/convection

The approach for defining the objects, forboth the forecasts and the observations, involvesapplication of a convolution function (essentially asmoothing function) to the spatial grid, followed byuse of a threshold to eliminate areas that are not ofinterest. Our experience with precipitation fieldshas indicated that a circular convolution functionprovides good results in most cases. The convolu-tion filter essentially assigns the average value inthe surrounding circular region to the point in thecenter of the circle. Thresholding the convolvedfield allows object boundaries to be detected. Thecombination of these two steps results in bound-aries that are similar to those that a human woulddraw. It might seem more straightforward to simplyapply a threshold to the raw grid. However, thisapproach would lead to fields that are very spikyand discontinuous and not at all like the regionsthat a human would identify subjectively. Finally,although the field is thresholded, the underlyingvalues (i.e., precipitation intensity or convectionoccurrence/non-occurrence) are retained for fur-ther analysis.

In summary, only two parameters areneeded for this process: the convolution radius and

the masking threshold. Figure 2 shows an exampleof the process applied to precipitation forecastfields from the Weather Research and Forecasting(WRF) NWP system on a 22-km grid scale acrossthe CONUS. Further information about these stepsin the process is presented by Bullock et al. (2004)and Davis et al. (2004).

3.2.2 Merging and matching objects

The purpose of the merging step is to com-bine objects that naturally go together (e.g., adja-cent objects that may appear to be part of thesame weather system), in either the observation orforecast field. Similarly, the purpose of the match-ing step is to optimally match forecast andobserved objects that naturally go together. Bynature, these two steps are interrelated; in fact, inthe current version of the approach, the merging ofobjects follows directly from the process of match-ing objects. However, future enhancements to thesystem are likely to include additional criteria forboth merging and matching of objects. Mergedobjects are referred to as composite objects.

The process of matching objects is based onan application of fuzzy logic (e.g., Yager et al.1987). First, a set of attributes or characteristics ismeasured for each forecast object and eachobserved object. These attributes include the distri-bution of intensity values within the object (repre-sented by various percentile values); the objectarea; the centroid location; the orientation angle;and the object curvature. The attributes for eachpair of forecast and observed objects are thencompared by computing the difference or ratio ofthe attribute values, and other attributes that applyto that pair (e.g., the area of overlap) are also com-puted. Each of the attribute comparison values fora pair of objects is then mapped to a predefined“interest” curve to determine the interest value thatis assigned to that pair of objects for that particularattribute.

An example of one of the interest maps isshown in Fig. 3. For this attribute comparison - theratio of the area of the forecast object to the area ofthe observed object - the interest value is low whenthe ratio is small (i.e., when the forecast shape ismuch smaller than the observed shape); increasesto one when the ratio is near one (i.e., the forecastobject and observed object have about the samesize); and decreases toward zero as the ratiobecomes large (i.e., the forecast object is muchbigger than the observed object).

Finally, the interest values for all of theattributes are combined as a weighted sum todetermine an overall interest value that can beassigned to the pair of objects. These overall inter-est values are used to objectively define pairs ofmatched forecast and observed objects, by com-paring the overall interest values to a threshold.Two (several) objects are merged if they both (all)are matched to the same objects in the other field.That is, if two forecast objects are matched to thesame observed object, the two forecast objects arecombined into a composite object and matched tothe observed object, and vice versa.

Note that only object pairs with an overallinterest value that exceeds the threshold arematched to one another. In some cases an objectmay not have a matching object in the other field,and thus it is an “unmatched” object. Such casesare not unanticipated; essentially these objects arefalse alarms if they are in the forecast field andthey are missed objects if they are in the observedfield.

3.2.3 Verification measures

Many different measures can be computedbased on the results of the object matching pro-cess. In particular, characteristics of each of theattributes defined in the previous section can becompared between the forecast and observedobjects. For example, the location of the centroid(i.e., “center of mass”) of the forecast object rela-tive to the centroid of the observed object may beof interest to answer questions regarding whetherthe forecast is consistently located too far to theeast or west. Characteristics and differences canbe summarized using various statistical measures

OriginalOriginal

ConvolvedConvolved

MaskedMasked

FilteredFiltered

FIGURE 2. Example (in 3 dimensions) of a precipitation forecast field from the WRF model and the steps in the process of identifying objects in the

field: (a) original precipitation field; (b) convolved (smoothed) field; (c) “masked” field after a

threshold has been applied, leaving the areas of interest; (d) original precipitation values underlying

the masked objects.

(a)

(b)

(c)

(d)

FIGURE 3. Example of an interest map, in this case for the ratio of the area of the forecast object to the area of

the observed object.

(e.g., mean, median, extreme percentiles) or distri-bution representations (e.g., histograms, box plots,scatter plots), as appropriate.

3.3 Forecast shapes and gridded observations

The object definition and matching processcan be simplified if the forecasts are alreadydefined as shapes, such as the forecasts and advi-sories that are commonly produced by human fore-casters. Examples of shape forecasts are theConvective Significant Meteorological Advisory (C-SIGMET; NWS 1991) and the Collaborative Con-vective Forecast Product (CCFP). Both of theseforecasts are issued by forecasters at the NationalOceanic and Atmospheric Administration (NOAA)National Centers for Environmental Prediction(NCEP) Aviation Weather Center (AWC). The C-SIGMETs are advisories and 1- and 2-h forecastsof significant convective activity over regions cov-ering an area at least 3,000 mi2 in size, with a mini-mum areal coverage of 40%. Similarly, the CCFPforecasts are 2-, 4-, or 6-h outlooks for convectiveactivity with at least 25% coverage over a minimumarea of 3,000 mi2. The CCFP is produced by AWCforecasters in collaboration with meteorologists atairline meteorology departments, Center WeatherService Units, and other facilities.

Essentially two different approaches can betaken with regard to these types of forecasts: (i)Optimize the “placement” of the forecasts relativeto the gridded observations; or (ii) Define objects inthe observation field and match them to the fore-cast shapes using the fuzzy logic approach. Appli-cations thus far have involved the first approach.The disadvantage of this approach is that itrequires an immense amount of processing timebecause it is necessary to test a large variety oflocations and orientations for the forecast shapes.The advantage of this more straightforward optimi-zation approach is that the matching process maybe somewhat less arbitrary; the goal simply is tofind the best possible score given a particular set offorecast shapes, regardless of whether they clearlymatch the corresponding shapes in the observa-tions. The main advantage of the fuzzy logic shapematching approach in this context is that it requiresmuch less processing. Moreover, the object char-acteristics are clearly defined by the matching pro-cess. Both approaches leave open the possibilityof future enhancements to the verification proce-dure in which additional forecast shapes might be

added or forecast shapes might be altered to makethe forecasts more optimally fit the observations.

4. EXAMPLES

4.1 Gridded forecasts and observations

The main application in this context so farhas been to spatial QPFs produced by the WRFmodel. The forecasts for this example are on anominal 22-km horizontal grid, and are matched toprecipitation observations from the NCEP Stage IVprecipitation analysis (see http://www.emc.ncep.noaa.gov/mmb/ylin/pcpanl/QandA/#STAGEX). These observations combine radarand rain gauge measurements and are producedhourly on a 4-km grid; for this study the observa-tions were re-mapped to the 22-km WRF grid.

Figure 4 shows an example of a set of fore-cast and observed objects for a particular case: the12-h WRF precipitation forecast valid at 0000 UTCon 2 July 2001. Although the geographic map isnot shown, the scale of the forecast essentiallyincludes the whole CONUS. The figure shows acase in which 10 forecast objects were identified,along with 11 observed objects (Fig. 4a and c).However, the merging process led to four compos-ite forecast objects, four composite observedobjects, and two individual observed objects (Fig.4b). Each composite forecast object was matchedto one of the composite observed objects, but thetwo observed objects that were not merged into acomposite object were not matched to a forecastobject. The final panel in Fig. 4 (Fig. 4d) shows theoverlap of the observed shapes on the forecastobjects and the overlap of the forecast objects onthe observed shapes.

Some basic verification statistics for theexample composite shapes are shown in Tables 1and 2. Table 1 shows the intersection area (IA) andsymmetric difference (SD) area values, which aretwo basic indicators of the correspondencebetween the forecast and observed objects. In par-ticular, IA measures the area of overlap betweenthe two objects. The SD is the total area coveredby the forecast and observed objects, less theintersection area. As indicated in Table 1, SD isgenerally much larger than IA, which indicates thedifferences between the forecast and observedobjects are quite large. In one case (compositeobjects A) the forecast and observed objects donot intersect.

Table 2 shows the values of some basicattributes of the forecast and observed objects.

FIGURE 4. Example WRF and Stage 4 case for 12-h WRF precipitation forecasts valid on 2 July 2001 at 0000 UTC: (a) precipitation objects defined by convolution-threshold approach; (b) Merged and matched areas; (c) individual object identifiers, where colors indicate objects that are matched and merged (a black number indicates the object was not matched to any

objects in the other field); and (d) overlap areas between forecast and observed objects.

(a)

(c)

(b)

(d)

Unmatched Bo Ao

Co

DoDf

Cf

BfAf

These attributes measure such characteristics asthe location of the objects (centroid location in the xand y directions), precipitation intensity 0.50th and(0.90th percentiles of the intensity distributions)and overall object size (area). For this simple case,some basic characteristics of the forecasts can beinferred. For example, for all composite objectsexcept one (C), the forecast object is located toofar to the west; in all cases except for one (B), theforecast object is located at least somewhat too farto the north; generally the median forecast intensityis somewhat too large, whereas the 0.90th percen-tile of forecast intensities is too small (implying theforecast did not correctly capture extreme precipi-tation occurrences); and forecast areas C and Dare too small, while forecast area B is somewhattoo large. Finally, it is important to note that two rel-atively small observed objects were not matched toany of the forecast objects; the areas of theseobserved objects were 7 and 121 grid boxes.

Although these results are based on a singlecase, they are indicative of the types of informationthat can be obtained from these types of compari-sons. Figure 5 is an example of one way these sta-tistics can be summarized across a larger set ofcases. For this analysis, WRF precipitation fore-casts for July through August 2001 were analyzed;only objects in the area of the CONUS east of theRocky Mountains were considered, to make theresults more homogeneous. The lines in Fig. 5show the CSI value for matching objects and theaverage SD value for matched objects as a func-tion of lead time. Because the CSI line is relativelyconstant for all lead times, it can be inferred fromthis figure that the ability to create matched fore-cast and observed objects does not diminish withincreasing lead time. In contrast, the average SDvalue increases with lead time, which indicates that

TABLE 1. Verification statistics (intersection area and symmetric difference area) for example presented in

Fig. 4. Intersection area is the area of overlap between the forecast and observed object; symmetric difference is the total area covered by the forecast and observed object, excluding the intersection area. Both values are

in units of gridboxes.

Composite object

Intersection Area (IA)

Symmetric Difference (SD)

Area

A 0 638B 100 488C 66 134D 259 905

TABLE 2. Verification measures/attributes for objects shown in Fig. 4. Area values are in units of grid boxes; intensity is in mm; and centroid location is measured in

grid squares.

Attribute WRF Stage IV Difference

Composite Objects “A”Centroid X 187 197 -10Centroid Y 44 31 13

Intensity (0.50) 4.7 2.5 2.2Intensity (0.90) 8.5 13.9 -5.4

Area 319 319 0Composite Objects “B”

Centroid X 130 144 -14Centroid Y 36 36 0

Intensity (0.50) 4.7 2.0 2.7Intensity (0.90) 8.7 9.7 -1.0

Area 355 333 22Composite Objects “C”

Centroid X 128 121 7Centroid Y 93 90 3

Intensity (0.50) 4.0 2.4 1.6Intensity (0.90) 8.5 11.3 -2.8

Area 126 140 -14Composite Objects “D”

Centroid X 205 215 -10Centroid Y 102 100 2

Intensity (0.50) 3.8 3.8 0Intensity (0.90) 7.4 13.8 -6.4

Area 585 838 -253

FIGURE 5. Symmetric difference (SD) and critical success index (CSI) for matched objects as a function

of lead time (h).

the errors in the forecasts grow with lead time.Additional summary plots are presented in Davis etal. (2004).

4.2 Forecast shapes and gridded observations

Examples of two types of human-generatedforecasts are considered in this application of thegeneral object-matching approach: the CCFP andthe C-SIGMETs. Figures 6 and 7 show examples ofeach type of forecast, along with the resultingobject optimization. In the case of the CCFP, theforecast locations and orientations were optimizeddirectly by searching the grid, rather than bymatching shapes. For the C-SIGMETs, objects inthe observation field were identified and used toprovide an initial guess regarding the optimal loca-tion of a C-SIGMET. Because no additional fore-cast areas were included in the forecasts (i.e., thebias and FAR values could not be changed bymoving the forecast shapes), the optimizationswere based on the value of PODy alone. For theseforecasts, the observations used for verificationrepresent the “convective constrained area” (CCA)estimated from radar and lightning measurements.The CCA is intended to represent the regionaround convection where air traffic is impacted bythe weather, and is expressed as a percent cover-age value across a 3,000 mi2 area; for more infor-mation about computation of the CCA, seeMahoney et al. (2004a,b).

In the case of the CCFP (Fig. 6), the fore-casts are optimized by relatively small movementsof the forecast shapes, as shown in Table 3. Forexample, shape number 2 is optimized by a smallrotation as well as a small change in location in thenorth-south direction. Although these optimizationstrategies do not result in large increases in PODyfor the optimal forecasts, they clearly diagnose thecharacteristics of the errors in the forecasts.

As noted earlier, the C-SIGMET optimizationprocess is somewhat more complex than the pro-cess applied to the CCFP. For the initial step, thecentroid of each C-SIGMET was matched to thecentroid of the best-fit observed CCA object. Fol-lowing this matching, the location and orientation ofthe best-fit forecast shape were optimized by test-ing a variety of possible choices. The green shapesin Fig. 7 show the actual C-SIGMET location andorientation; the gold shapes show the first guess atan optimal location and orientation; and the blueshapes show the final optimization.

The case presented in Fig. 7 shows C-SIG-METs that were initiated at 0200 UTC on 5 July2003. The C-SIGMETs were advected using themotion vector included in the C-SIGMET message,so all of the C-SIGMETs shown in Fig. 7 were validat 0400 UTC. Several small line shapes in Fig. 7were not changed through the optimization pro-cess; these shapes could not be matched to aregion with observed convection.

As was the case for the CCFP example, thechanges in the verification scores achieved by theoptimal forecasts was not large; nevertheless, it isclear that at least in some cases (e.g., observedshape 12 in Fig. 7) the actual location of the fore-cast was not optimal; it appears that the forecastcould have been improved by not advecting the C-SIGMET area as quickly over the 2-h time period.This type of diagnostic information on its own couldprovide useful feedback to the C-SIGMET forecast-ers. It also could be summarized across a set ofcases to characterize overall performance of theforecasts.

4.3 Probabilistic forecasts

The issues that have been considered hereessentially apply to all types of spatial forecasts. Itis important to note that although probabilistic fore-casts provide numerous advantages over the morecommon deterministic types of spatial forecaststhat have traditionally been produced and used,they do not eliminate the need to apply approachesthat are more diagnostic and that are able to indi-cate the sources of the errors in the forecasts. Forexample, traditional verification statistics for proba-bilistic forecasts are very sensitive to displacementerrors. This effect is illustrated here with a simpleexample. In particular, the cartoons in Fig. 8 repre-sent a spatial probability forecast in which five dif-ferent probability values can be forecast, levels 0-4, where level 0 represents the area outside theblue contours).

TABLE 3. CCFP optimization statistics showing displacement and orientation errors for the sample

forecasts in Fig. 6.

Shape number

Rotation (deg)

x-movement

(km)

y-movement

(km)

1 180 40 -402 -5 0 -1203 -95 -160 04 15 -120 805 0 -240 -80

FIGURE 6.Example of CCFP optimization for 2-h CCFP forecasts valid at 2100 UTC on 8 June 2003: (a) original forecasts; and (b) forecasts with optimized location and orientation.

(a)

(b)

FIGURE 7. C-SIGMET optimization example for C-SIGMETs valid at 0400 UTC on 5 July 2003. Original C-SIGMET locations are shown in blue, initial optimal locations in gold, and final optimal locations are

shown in light green. Observed objects are identified using red numbers; black number 12 for that object is presented for clarity only.

12

Figure 9 shows the frequency of use of eachof the forecast probability levels. Note that Fore-casts A and B differ only in the location of the fore-cast regions relative to the observed region ofconvection: the areas representing the forecastprobability levels are the same size for both fore-cast examples. In particular, the forecasts are thesame size and shape and have the same values.However, the ellipses in Forecast A completelyoverlap the convection area, whereas the forecastellipses in Forecast B are displaced to the south-east.

An overall evaluation of the quality of theprobability forecasts is provided by reliability dia-grams and relative operating characteristic (ROC)plots. Together these diagrams measure twoimportant attributes of the capabilities of the fore-casts. In particular, the reliability diagrams show

how consistent the relative frequency of occur-rence of the event is with the forecast probabilityvalue. The ROC measures the forecast’s ability todiscriminate between situations when the event(i.e., convection) occurs and when it does notoccur; it is a plot of the hit rate vs. the false alarmrate for various probability thresholds.

The reliability diagrams for the two forecastsin Fig. 8 are shown in Fig. 10. The lines in this fig-ure indicate that Forecast A is much more reliablethan Forecast B (or at least has the potential to bemore reliable, depending on the calibration of theforecast values). In particular, for Forecast B thefrequency of convection varies only slightly with theprobability level, whereas a great deal of differenti-ation is demonstrated by Forecast A.

The ROC diagrams presented in Figure 11also show that Forecast A has much more skill

than Forecast B. In the ROC diagrams, greater skillis demonstrated by lines that lie closer to the upperleft corner of the diagram, and it is clear that theline for Forecast A is much further to the upper leftthan the line for Forecast B. Another way to con-sider this measure of skill is the area under theROC curve; clearly, the area under the curve forForecast A is much larger than the area under thecurve for Forecast B. These results indicate thatForecast A is much more skillful than Forecast B interms of its ability to correctly classify Yes and Noobservations of convection.

These results suggest that the verificationresults for spatial probabilistic forecasts can bestrongly impacted by displacement errors. Applica-tion of object-oriented verification techniques tothese types of forecasts would make it possible todecompose the errors according to their varioussources (including displacement and shapeerrors). This approach would also make it possibleto diagnose the impacts of various improvementsin placement, orientation, size, and so on.

5. FUTURE WORK

A number of tests and enhancements areplanned for the future. For example, the verificationtechnique will be applied to a wide variety of casesand to a variety of types of forecasts. The newcases will allow more complete testing of the meth-odology. Although it is difficult to objectively test theapproach and unambiguously measure its specificcapabilities, examination of many different cases

A

B

FIGURE 8. Schematic spatial probability forecasts. Blue contours represent different probability forecast levels (0-4). Green ellipse represents the observed

Yes field.

1

2

3

4

0

0

100

200

300

400

500

0 1 2 3 4

Example Forecast Frequencies

Freq

uenc

y

Probability level

FIGURE 9. Relative frequency of use of forecast probability levels in Fig. 8.

FIGURE 10. Reliability diagrams for forecasts shown in Fig. 8. X-axis is the probability level; Y-axis shows the frequency of occurrence of the “event” for each of

the probability levels.

0

0.2

0.4

0.6

0.8

1

0 1 2 3 4

Rel

ativ

e fr

eque

ncy

Probability level

Forecast A

Forecast B

will allow us to evaluate its consistency and rea-sonableness. Sensitivity tests will clarify the robust-ness of the technique relative to the varioussettable parameters. New types of forecasts to beinvestigated include the National ConvectiveWeather Forecast (NCWF) and higher resolutionWRF precipitation forecasts. The new version ofthe NCWF (NCWF-2) is in the form of a probabilityforecast, which will allow us to test the methodol-ogy on actual probabilistic forecasts.

A number of additional steps are needed tomake the approach fully functional as a verificationtool. For example, we will identify a set of generalverification measures that summarize the basicattributes of the forecasts and their quality in the

context of the object-oriented approach. Some ofthe likely measures have been described here, butothers will be defined as we examine additionaldatasets. In addition to the basic statistics, it will benecessary and valuable to define specific mea-sures that are meaningful to particular users andprovide information that is operationally relevant forparticular applications.

Several enhancements to the method havealready been identified. For example, it hasbecome apparent that the methodology can beimproved by including the time dimension in theobject identification and matching process (muchas a human would do). We also may investigatethe possibility of evaluating objects in three dimen-sions. Alternative approaches for matching objectsare also being investigated. For example, we aretesting a binary image matching technique devel-oped by Baddeley (1992) and comparing theresults of this approach to the results of the fuzzylogic approach. In addition, as the verification tech-nique matures, local/regional characteristics (e.g.,orography) may be incorporated into the matchingprocess.

Finally, the approach described here clearlyis highly dependent on the scale of the forecastsand observations. The approach naturally takesthese effects into account through the definition ofthe radius and threshold parameters. Thus, scaleeffects could simply be handled by applying a vari-ety of different parameter combinations and exam-ining the variations in the verification statistics as afunction of the parameters. In addition, we plan toinvestigate more direct approaches such as thewavelet (intensity-scale) decomposition approachrecently developed by Casati et al. (2004).

6. CONCLUSIONS

The object-oriented verification approach forspatial forecasts has the potential for providing newuseful verification information that cannot beobtained using standard approaches. The methoddescribed here is complementary to the approachdeveloped by Ebert and McBride (2000) as well asother approaches that are under development(e.g., Baldwin 2003; Casati et al. 2004). Futurework on this topic will lead to verification informa-tion that is meaningful in a variety of operationalsettings and provides feedback to forecast devel-opers to implement meaningful improvements toforecasting systems.

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Example B: ROC

hit r

ate

(PO

Dy)

false alarm rate (1-PODn)

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1

Example A: ROChi

t rat

e (P

OD

y)

false alarm rate (1-PODn)

FIGURE 11. ROC plots for two forecasts shown in Fig. 8.

ACKNOWLEDGMENTS

This research is in response to requirementsand funding by the Federal Aviation Administration(FAA). The views expressed are those of theauthors and do not necessarily represent the offi-cial policy or position of the U.S. Government.

This work was partially supported by theNational Science Foundation and the U.S. WeatherResearch Program. NCAR is sponsored by theNational Science Foundation.

REFERENCES

Baddeley, A.J., 1992: An error metric for binaryimages. In Robust Computer Vision: Qualityof Vision Algorithms, W. Forstner and S.Ruwiedel, Editors, Wichmann Verlag, 59-78.

Baldwin, M.E., 2003: Development of an Events-Oriented Verification System Using data min-ing and Image Processing Algorithms. 3rdConference on Artificial Intelligence Applica-tions to the Environmental Science. Pre-prints, 83rd AMS Annual Meeting, 9-13February, Long Beach, CA, American Meteo-rological Society (Boston), available at http://ams.confex.com/ams/pdfview.cgi?user-name=57821.

Brown, B.G., J.L. Mahoney, C.A. Davis, R. Bullock,and C.K. Mueller, 2002: Improvedapproaches for measuring the quality of con-vective weather forecasts. Preprints, 16thConference on Probability and Statistics inthe Atmospheric Sciences. Orlando, 13-17May, American Meteorological Society (Bos-ton), 20-25.

Bullock, R., B.G. Brown, C.A. Davis, M. Chapman,K.W. Manning, and R. Morss, 2004: AnObject-Oriented Approach to the Verificationof Quantitative Precipitation Forecasts: PartI—Methodology. 17th Conference on Proba-bility and Statistics in the Atmospheric Sci-ences, Preprints, 84th Annual Meeting,American Meteorological Society, http://www.ametsoc.org/meet/index.html.

Casati, B., G. Ross, and D.B. Stephenson, 2004: Anew intensity-scale approach for the verifica-tion of spatial precipitation forecasts. Meteo-rological Applications, 11,141-154.

Davis, C., B. Brown, and R. Bullock, 2004: Object-based verification of precipitation forecasts.Submitted to Monthly Weather Review.

Ebert, E., and J.L. McBride, 2000: Verification ofprecipitation in weather systems: Determina-tion of systematic errors. Journal of Hydrol-ogy, 239, 179-202.

Doswell, C.A., R. Davies-Jones, and D.L. Keller,1990: On summary measures of skill in rareevent forecasting based on contingencytables. Weather and Forecasting, 5, 576-585.

Hoffman, R.N., Z. Liu, J.-F. Louis, and C. Grassotti,1995: Distortion representation of forecasterrors. Monthly Weather Review, 123, 2758-2770.

Jolliffe, I.T., and D.B. Stephenson, 2003: ForecastVerification. A Practitioner's Guide in Atmo-spheric Science. Wiley and Sons Ltd., 240pp.

Mahoney, J.L., J.E. Hart, and B.G. Brown, 2004a:Defining observation fields for verification ofspatial forecasts of convection. Preprints,17th Conference on Probability and Statisticsin the Atmospheric Sciences, 12-16 January,Seattle, WA, American Meteorological Soci-ety (Boston). Available from www.amet-soc.org.

Mahoney, J.L., S. Seseske, J.E. Hart, M. Kay, andB.G. Brown, 2004b: Defining observationfields for verification of spatial forecasts ofconvection: Part 2. Preprints, 11th Confer-ence on Aviation, Range, and AerospaceMeteorology, 4-8 October, Hyannis, MA,American Meteorological Society (Boston).Available from www.ametsoc.org.

Mass, C.F., D. Ovens, K. Westrick, and B.A. Colle,2002: Does increasing horizontal resolutionproduce more skillful forecasts? Bulletin ofthe American Meteorological Society, 83,407-430.

NWS, 1991: National Weather Service OperationsManual, D-22. National Weather Service.(Available at http://www.nws.noaa.gov).

Rife, D. L., C. A. Davis, Y. Liu, and T. T. Warner,2004: Predictability of low-level winds bymesoscale meteorological models. MonthlyWeather Review, in press.

Wilks, D.S., 1995: Statistical Methods in the Atmo-spheric Sciences. Academic Press, 467 pp.

Yager R.R., S. Ovchinnikov, R.M. Tong, and H.T.Hguyen, 1987: Fuzzy sets and Applications,Selected Papers by L.A. Zadeh. John Wileyand Sons, 685 pp.