OBSERVATIONS ON WAFS GRIDDED FORECASTS

WAFS Workshop on the Use and Visualization of Gridded SIGWX Forecasts14-15 September 2009

Overview Introduction Compatibility of forecasts of the two WAFCs Quality issues: seasonal aspects Quality issues: CB diurnal variation Quality issues: CB associated with TC Verification issues Calibration issues User survey Conclusions

Introduction WAFSOPSG/4: WAFC Provider States, in

coordination with WAFSOPSG members from IATA, IFALPA and WMO, to undertake systematic comparisons of trial gridded forecasts of icing, turbulence and CB clouds of the two models, highlighting characteristics of areas with different values.

The results should lead to the alignment of algorithms used by the two WAFCs.

Introduction To facilitate comparison of the trial gridded

forecasts, HK, China set up website (http://wafs-grid-fc.weather.gov.hk) to display gridded values (without smoothing/interpolation).

The observations were reported in CNS/MET SG/13 meeting in July 2009 and were agreed to be referred to the WAFSOPSG/5 meeting.

(Earlier observations reported in CNS/MET SG/12 meeting in 2008)

Compatibility of forecasts of the two WAFCs

T+24 h Max CAT Potential at FL340 based on 12 UTC 2 July 2009 )

UK US Diff

Difference map shows absolute difference in the two forecasts (little agreement between the two forecasts if difference map appears to be the union of them).

Compatibility of forecasts of the two WAFCs CB horizontal extent

- CB coverage of UK more extensive- Forecast values of US reaching 50-75% (OCNL following Annex 3 spec) over tropics, larger than those given in the UK forecast

ICAO height at CB top- UK still gives more extensive CB areas- UK over-forecast CB for vast areas in the Pacific Ocean

Mean in-cloud turbulence potential (FL180)- UK and US forecast values with similar range- UK forecast shows small values almost everywhere

Compatibility of forecasts of the two WAFCs Max icing potential (FL180)

- Significant differences in the spatial coverage over the tropical and higher latitude regions- WAFS Science Coordination Meeting 2009: “percentage overlap of all (icing) forecasts between UK and US ~ 20%”

Max CAT potential (FL340)- UK gives much less extensive CAT areas (except over certain spots)

Compatibility of forecasts of the two WAFCs Difference maps in most cases appear to be the

union of the forecasts from the two centres, indicating little agreement

Latest CB forecasts of UK

CB top height valid12 UTC 14 Sep 2009

CB horiz extent valid12 UTC 14 Sep 2009

MTSAT-1R infrared satellite imagery, with

deep convection indicated in red, at 08 UTC on 14 Sep 2009

Quality issues: seasonal aspectsmean in-cloud turbulence at FL 180

For both winter & summer: large values are forecast by UK for high latitudes in both hemispheres. They are not seen in the US forecasts.

T+24 h Gridded Forecasts(based on 00 UTC 6 Jan 2009)

T+24 h Gridded Forecasts(based on 00 UTC 2 Jul 2009)

UK US

Quality issues: seasonal aspectsmax icing potential at FL 180

At high latitudes in winter: large values (> 85%) forecast by UK, but much smaller values (mostly < 30%) forecast by US

Scientific paper on icing climatology: non-occurrence of icing at FL180 in the wintertime north of 25N

UK US

T+24 h Gridded Forecasts(based on 00 UTC 2 Jul 2009)

T+24 h Gridded Forecasts(based on 00 UTC 6 Jan 2009)

Quality issues: seasonal aspects max CAT potential at FL340 – 3 days in Jan 2009

- large values are forecast by UK for a number of isolated locations

- stationary “hot spots” associated with mountainous regions

Quality issues: seasonal aspects Some common issues can be identified

when data for different seasons are monitored (e.g. in-cloud turb, icing)

Some issues (e.g. terrain-related) may show up at different locations in different seasons

The above observations are not exhaustive It is imperative for the WAFCs to:

conduct verification for all seasonsexamine model climatological biases

Quality issues: seasonal aspectsClimatology of differences between two WAFCs (May – Aug 2009)

Max icing Max CATMean

in-cloud turb

CB horiz extent

CB top height

? ?

Quality issues: CB diurnal variation

ICAO height at CB top (UK)T+24h gridded forecastsBased on 00 UTC 17 August 2008

T+24h gridded forecastsBased on 06 UTC 17 August 2008

Solar heating

Again the UK forecasts are giving much more extensive CB areas compared to the US forecasts

T+24h gridded forecastsBased on 00 UTC 17 August 2008

T+24h gridded forecastsBased on 06 UTC 17 August 2008

Quality issues: CB diurnal variation ICAO height at CB top (US)

Solar heating

MTSAT-1R infrared satellite imagery, with

deep convection indicated in red,

at 23:30 UTC on 17 August 2008

MTSAT-1R infrared satellite imagery, with

deep convection indicated in red,

at 05:30 UTC on 18 August 2008

• Actual weather as shown by the satellite images

• Weather was generally fine over southern China throughout the period

• Both UK and US over-forecast CB occurrence over this region at 06 UTC (14 H local time), probably due to difficulties of the models in handling CB development induced by daytime solar heating

Quality issues: CB diurnal variation

Quality issues: CB diurnal variation 2009 WAFS Science Coordination Meeting

acknowledged that further work would need to be undertaken by the WAFCs for improving the CB gridded forecasts, including to “look at diurnal variations of forecasts (manual and automatic), particularly in tropics”

Quality issues: CB associated with TC

Satellite imagery clearly shows deep convection associated with tropical cyclone Linfa

Rather difficult to identify the area associated with TC without referring to the satellite imagery

Based on 06 UTC 19 June 2009

UK US

Need TC (and other SIGWX) information in future visualization models

Verification issues APANPIRG called for systematic

SIGWX verification in 2006 WAFSOPSG/4: Evaluation of the

SIGWX forecasts proposed by APANPIRG would be sufficiently addressed in the systematic comparison of WAFS SIGWX forecasts with the gridded forecasts

2009 Science Coordination Meeting

Forecast verified Period

Parameters verified

Data used as “ground truth” for verification

Verification metrics

considered

Criteria for counting hit and false alarm

Icing Nov 2008 – Jan 2009

Maximum & mean icing potential at

FL100 & FL140

CloudSat Icing Potential (CLIP)

product (derived from CloudSat cloud

classification data and numerical model

temperature forecast)

POD+ & FAR^

Icing potential thresholds – 0.1, 0.3 & 0.5

CLIP – trace icing

CAT Nov 2008 – Jan 2009

CAT potential@

Global Aircraft Data Set (GADS)

based on British Airways fleet of Boeing

747-400 aircraft

POD & FAR

CAT potential thresholds – numerous

GADS – DEVG# >= 2 m/s

In-cloud Turbulence Not verified

CB Nov 2008 & Jan 2009

UK - convective

rainfall rate*US - ??

UK Sferics data POD & FAR

Rainfall rate thresholds – 16 levels

0 to 0.0046 kg/m2/sSferics data - ??

+ POD = probability of detection, or hit rate = hit / (hit + miss).^ FAR = false alarm rate = false alarm /(false alarm + correct rejection) . For rare events, correct rejection is very large!@ No indication whether max or mean turb potential was considered, and whether or not forecasts for all flight levels were verified.# DEVG = Derived equivalent vertical gust* Not a product of the CB gridded forecasts

Verification issues In-cloud turbulence – systematic verification yet to be

done. Icing – verification based on another icing product

(CLIP) as ground truth, with “trace icing” criterion Kay et al (2009) on CLIP: “It is important to note that our algorithm only states where icing conditions have the potential to exist rather than stating exactly where within the clouds exactly the icing does or does not exist. ... Therefore our algorithm should be considered as a biased representation of icing conditions in the atmosphere. In the absence of a true measure of icing conditions the amount of bias cannot be known”.

The choice of the ground truth for the verification, as well as the verification method itself, must be agreed by WAFSOPSG. “Trace icing” calibration issue.

Verification issues CAT – verification based on DEVG >= 2 m/s from

aircraft data, with numerous thresholds, rather than ICAO adopted turbulence metrics, i.e. MOD/SEV in PIREP or EDR thresholds. Numerous thresholds calibration issue.

CB – verification based on sferics data, with numerous thresholds, rather than observations of CB (e.g. from satellite). Also lack of information on how CB forecast parameters are derived, and use of smoothing algorithm by UK.

Calibration issues Need to be mindful about users’

perception and expectation based on their experience with the existing SIGWX forecasts

Thresholds for verifying and visualizing gridded forecasts need to be determined in consultation with users

User survey HKO conducted a user survey on the WAFS trial

gridded forecasts in Jul to Aug 2009, 15 returns from IFALPA, pilots, airline management and dispatchers:80% consider gridded products are NOT suitable for

operational use in flight planning and flight documentation;

93% considered gridded products CANNOT replace the SWH and SWM charts.

Specific user comments are summarized below.

User commentsComments User

Why is there so much difference from the London and Washington charts if they are from the same time period?

IFALPA Rep

Have lots of deficiencies: too general; discontinuities; disagreement between centres; loss of useful information; too long forecast time periods etc.

“Potential” appears to have no scientific definition or validity and of little or no assistance to pilot, whereas PROB gives the indication of the likelihood of the turbulence/icing/CAT being encountered.

They must reflect the situation and not be so far forecast that the time period does not apply or that the forecast is so general as to be meaningless.

IFALPAPilot

User commentsComments User

Not suitable for operational use without considerable improvement. The improvement suggested by the HKO would help a lot, particularly if the horizontal "real time" strips can be developed. These would be very useful and very practical for the pilots.

The gridded products could not replace the high-level/medium level at this stage as the information is too general and some of the previous information to be omitted.

IFALPAPilot(cont’d)

User commentsComments User

The scale is too large and the areas of hazard are too wide. If pilot was presented with so much weather, he/she would be unable to fly around it.

Different aircraft types have different rides through turbulence depending on wing loading.

Icing is only a problem at cruise levels in large tropical convective clouds.

Pilots

The technological advance and gridded forecasts could not take the place of the forecaster’s expertise of the MET situation, particularly for SIGWX like CB.

Member of WAFS OPSG

User commentsComments User

Large difference between WAFC London and Washington.

Two WAFCs should align to produce an integrated WAFS.

Very large scale, too big an area, insufficient details for short-haul flights.

It is a rough forecast; user cannot obtain detailed information just by a glance; not precise and detailed in comparison with JMA data.

We do prefer using High-level SIGWX and Medium-level SIGWX charts (the current one). It's simple, easy to understand and contains lots of information.

Airlines

Conclusions Suitability of WAFS gridded forecasts of icing,

turbulence and CB for operational use :

• Issues with compatibility, quality, verification, and calibration need to be satisfactorily addressed before they could be considered suitable for operational use.

• Systematic verification: WAFS output performance indicators (including methodology, thresholds, choice of ground truth) for verifying the gridded forecasts should be systematically developed and agreed.

• Calibration: the thresholds for verifying and visualizing the gridded forecast should be calibrated for compatibility with users’ expectation/perception. Pilots have problems with the use of “potential” for icing and turbulence.

Conclusions Suitability of WAFS gridded forecasts of

icing, turbulence and CB to replace the current SIGWX forecasts (SWH and SWM):Not at this stage (not for operational use)Visualization issues (including thresholds to be

adopted, forecast irregularities e.g. CB top/base heights) need to be resolved

High “at-a-glance” visualization models should be agreed by users

SIGWX features other than icing, turbulence and CB required by users should be included, especially if they are safety-related (e.g. TC, VA)

Thank you

Supplementary information

Observations reported in CNS/MET SG/12, 2008CB horizontal extent

CB coverage is more extensive with generally larger forecast values in the UK forecast compared with that of US. Furthermore, small values of CB horizontal extent are given almost everywhere in the UK forecast

valid 00 UTC 23 Jun (based on 00 UTC 22 June 2008)UK US

Back

Observations reported in CNS/MET SG/12, 2008 ICAO height at CB top

UK forecast generally gives more extensive but lower heights of CB top (generally below 37kft) compared with that of US (above 43 kft)

valid 00 UTC 23 Jun (based on 00 UTC 22 June 2008)

UK US

Back

Observations reported in CNS/MET SG/12, 2008 Mean in-cloud turbulence potential

UK forecast generally gives wider range and higher turbulence potential values compared with that of US. On the other hand, small values of mean in-cloud turbulence potential are given almost everywhere in the US forecast

valid 00 UTC 23 Jun at FL180 (based on 00 UTC 22 June 2008)UK US

Back

Observations reported in CNS/MET SG/12, 2008 Maximum icing potential

At FL180: UK forecast giving generally higher icing potential over the higher latitude regions; US forecast giving generally higher icing potential over the tropical region and the UK forecast giving generally higher icing potential over the higher latitude regions

valid 00 UTC 23 Jun at FL180 (based on 00 UTC 22 June 2008)

UK US

Back

Observations reported in CNS/MET SG/12, 2008 Maximum CAT potential

At FL340: UK forecast generally gives much less extensive CAT areas compared with that of US

valid 00 UTC 23 Jun at FL340 (based on 00 UTC 22 June 2008)

UK US

Back

Back

Bernstein, B.C. and Christine LeBot, 2009: An Inferred Climatology of Icing Conditions Aloft, Including Supercooled Large Drops Part II: Europe, Asia and the Globe, Journal of Applied Meteorology and Climatology, American Meteorological Society

Back

CB horizontal extent at FL340T+24 h Gridded Forecasts (based on 12 UTC 2 July 2009)

UK US Diff

Back

Back

ICAO height at CB topT+24 h Gridded Forecasts (based on 12 UTC 2 July 2009)

UK US Diff

Back

ICAO height at CB topT+24 h Gridded Forecasts (based on 12 UTC 2 July 2009)

Back

UK US

MTSAT-1R IR sat picWith deep convection indicated in redAt 11:30 UTC 3 July 2009

Back

Mean in-cloud turbulence potential at FL 180T+24 h Gridded Forecasts (based on 12 UTC 2 July 2009)

UK US Diff

Back

Back

Maximum icing potential at FL 180T+24 h Gridded Forecasts (based on 12 UTC 2 July 2009)

UK US Diff

Back

Back

Maximum CAT potential at FL 340T+24 h Gridded Forecasts (based on 12 UTC 2 July 2009)

UK US Diff

Back

Back

List of Respondents to WAFS SurveyIFALPA Rep (Captain Miguel Marin, Chairman of ATS Committee)IFALPA Pilot (Captain Brian Greeves, HKALPA)

Pilots: The Guild of Air Pilots and Air Navigators (GAPAN) (Captain Brent

Hawkins, Vice-Chairman, Hong Kong Region) Metrojet (Captain Peter Garraty)

Airlines: Cathay Pacific Airways (Neil Phillips, Manager Line Operations) China Airlines (Lo Wai Lun Kevin, Dispatcher) Dragonair (Andy Jepps, FTU A320; Jim Ashby, Deputy Manager Line Operations) El Al Israel Airlines Limited (c/o Hammond Law, Team Manager) Fedex (Andy Yeung, Manager - Operations) Hong Kong Airport Services Limited (Gary Hor, Flight Dispatch Manager; Jackson Lam, Quality Assurance Manager) Japan Airlines (Stephen Chan, Manager of Flight Operation) Jardine Airport Services Limited (Hammond Law, Team Manager) South African Airways (Ringo Chui, Airport Manager & Vice-chairman of Airline Operators Committee) Back